Itamar Oren

This commit introduces a **bug fix** to the **PAR bootstrap mechanism** by **clearing the `PYTHON_GIL` environment variable** after the Python interpreter has initialized. Previously, `PYTHON_GIL=0`, which is set for free-threading PAR builds to disable the GIL, would leak into child processes. This leakage caused **fatal crashes** when a free-threading PAR spawned a subprocess using a standard, GIL-enabled Python build. By adding `PYTHON_GIL` to the `var_names` list in `sitecustomize.__clear_env()`, the system now ensures **robustness in subprocess execution** by preventing incompatible environment settings from being inherited.

This commit is a **reland** that provides a **bug fix** for **standalone fastzip PARs** where **startup functions** were not executing in child processes spawned via `multiprocessing` or `subprocess.run`. It resolves this by extracting `sitecustomize.py` to a dedicated subdirectory within the PAR's unpack directory and prepending this path to `PYTHONPATH` for child processes. This ensures that the child Python interpreter can correctly load `sitecustomize.py` and execute the necessary startup functions, including a fallback in `__startup__()` to add the PAR zip to `sys.path` if needed. The dedicated subdirectory prevents unintended side effects on other Python processes, significantly improving the robustness and correct initialization of PAR child processes.

This commit **reverts** a previous change (D96249473) that aimed to fix **startup functions not running** correctly within **standalone fastzip multiprocessing children**. This action effectively undoes the prior **bug fix**, indicating that the original solution may have introduced new issues or was not the correct approach. The primary impact is on the `fastzip` module's multiprocessing behavior, potentially reintroducing the problem of startup functions failing to execute in child processes.

This commit delivers a **bug fix** addressing an issue where **startup functions** failed to execute in **multiprocessing child processes** when running **standalone fastzip PAR binaries** in `spawn` mode. The problem stemmed from `sitecustomize.py` and other critical startup components within the PAR zip not being correctly loaded by child interpreters. The solution involves writing a **proxy `sitecustomize.py`** to the unpack directory during `__patch_spawn()`, which is then found via `PYTHONPATH` by child processes. This proxy dynamically adds the PAR zip to `sys.path`, allowing the real `sitecustomize` to be re-imported and execute `__startup()` and all associated startup functions, ensuring proper initialization for multiprocessing applications.

This commit **fixes a Time-of-Check-to-Time-of-Use (TOCTOU) race condition** within the `_reset_tracker_in_child` function in **`prelude/python/tools/make_par/sitecustomize.py`**. It reorders the state reset operations (`tracker._fd = None`, `tracker._pid = None`) to execute **before** the `os.close(fd)` call. This **bug fix** prevents other threads from potentially accessing a stale file descriptor after it has been closed but before the tracker's state is updated. The change improves the **robustness and correctness** of file descriptor management, particularly in multi-threaded or multi-process scenarios involving child processes.

This commit **fixes a bug** in **Python inplace PARs** where child processes spawned via `subprocess.run` would incorrectly lose their critical library path environment variables. Previously, `sitecustomize.py`'s `__clear_env()` cleared `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH`, and `__patch_subprocess_run` failed to restore them, causing child processes to fall back to the platform Python. The **bug fix** modifies `__patch_subprocess_run` in `prelude/python/tools/make_par/sitecustomize.py` to correctly restore these environment variables, aligning its behavior with `__patch_spawn`. This ensures that child processes within PARs consistently execute using the bundled Python interpreter and its associated libraries.

This commit implements a **bug fix** to prevent **deadlocks** in forked child processes that utilize the **`multiprocessing` resource tracker**. It introduces `__patch_resource_tracker_fork()` within `prelude/python/tools/make_par/sitecustomize.py`, which registers an `os.register_at_fork` callback. This callback, executed in child processes, safely closes the inherited resource tracker pipe file descriptor and resets its internal state. By doing so, it prevents the parent process's tracker shutdown from deadlocking on `waitpid`, significantly improving the stability of applications that fork processes and use `multiprocessing`.

This commit **introduces a new capability** by adding an `execution_modifier` parameter to the `constraint_with_aliases` macro. This enhancement to the **configuration system**, specifically within `prelude/configurations/constraint_alias.bzl`, allows callers to explicitly opt into execution modifier resolution for their defined constraints. By providing this parameter, users gain more granular control over how constraints are applied, enabling more flexible and context-aware build target execution. This change improves the expressiveness of constraint definitions by allowing them to react to execution modifiers.

This commit delivers a **bug fix** for a `RecursionError` encountered when a **PAR (Python Archive) file** spawned a child process using `subprocess.run`. The error originated from the parent's `sys.path`, which included `.par` zip files, being copied into the child's `PYTHONPATH`, causing recursive imports during early Python initialization. To resolve this, the `__patch_subprocess_run` function in `prelude/python/tools/make_par/sitecustomize.py` now **filters non-directory entries from `PYTHONPATH`** and refines `PYTHONHOME` handling, mirroring an existing fix in `__patch_spawn_preparation_data`. This change also **refactors** the shared path-resolution logic into a new helper function, `_resolve_path_entries`, thereby enhancing the stability of PAR execution environments.

This commit **refactors** the `_patched_run` function within `prelude/python/tools/make_par/sitecustomize.py` to enhance the **monkeypatching mechanism** for `subprocess.run`. By applying `functools.wraps`, the original function's metadata is now preserved, significantly improving introspection and debugging capabilities. As part of this **maintenance** effort, the return type annotation for `_patched_run` has been corrected, and the `subprocess` module is now lazily imported by inlining its import statement. This change ensures greater reliability and debuggability for code interacting with the patched `subprocess` functionality.

This commit introduces a **patch to `subprocess.run`** within the **Python runtime environment** to improve the reliability of spawning the current Python interpreter. Specifically, it **correctly sets `PYTHONPATH` and `PYTHONHOME`** when `subprocess.run` is used with `sys.executable` from a bundled runtime (e.g., a PAR file). This **enhancement** prevents issues where a mix of interpreter binaries and C extensions from different runtimes could be loaded, ensuring that spawned subprocesses pick up the correct Python environment. The patch is conservative, only applying when `sys.executable` is explicitly called and no `PYTHONPATH` is already provided, significantly increasing the chances of success for such operations.

This commit delivers a **bug fix** addressing an issue where **Python PARs** using a **bundled runtime** failed to correctly initialize child processes when employing the **multiprocessing** module with `spawn` or `forkserver` start methods. Previously, the `PYTHONHOME` environment variable, crucial for CPython's `sys.prefix` calculation, was not properly preserved and restored, causing child processes to load standard library modules from the system Python instead of the bundled runtime. The fix modifies the `__clear_env` function in `prelude/python/tools/make_par/sitecustomize.py` to include `PYTHONHOME` in the list of environment variables managed by the **bootstrap mechanism**. This ensures `PYTHONHOME` is temporarily restored around `spawnv_passfds` calls, guaranteeing that multiprocessing children correctly initialize with the **bundled Python runtime environment**.

This commit delivers a **bug fix** addressing `multiprocessing` failures in **standalone fastzip PARs** when using the `spawn` or `forkserver` start methods. Previously, child processes would fail to import modules because `sys.path` entries, serialized from the parent, contained invalid `/proc/self/fd/<N>` paths that were closed upon exec. The fix **monkeypatches `multiprocessing.spawn.get_preparation_data`** within `sitecustomize.py` to resolve these `/proc/self/fd/<N>` entries to their actual filesystem paths using `os.readlink()` before serialization. This ensures child processes receive valid paths, enabling `zipimport` to correctly load modules from the PAR archive, and is conditionally applied to avoid conflicts with xar PARs.

This commit delivers a **bug fix** for **standalone fastzip PARs** that utilize **`multiprocessing`** with `spawn` or `forkserver` start methods. Previously, child processes failed to import modules from the PAR archive because `sys.path` entries, which were `/proc/self/fd/<N>` paths, became invalid after `exec` due to `FD_CLOEXEC`. The fix monkeypatches `multiprocessing.spawn.get_preparation_data` in `prelude/python/tools/make_par/sitecustomize.py` to resolve these file descriptor paths back to their real filesystem paths using `os.readlink()`. This ensures that child processes receive valid, importable paths to the PAR file, allowing them to correctly load modules and preventing module import failures.

This commit provides a **bug fix** to resolve a **use-after-free** error occurring during **Python 3.14 shutdown** within the **Thrift Python bindings**. The issue stemmed from the `SetMetaClass` function in `thrift/lib/py3/exceptions.pxd` failing to properly increment the reference count of metaclasses, leading to their premature deallocation. By adding a necessary reference increment, this change ensures that metaclass objects remain valid throughout their lifecycle, preventing crashes and improving the stability of Thrift-dependent Python applications during interpreter shutdown.

This commit **updates the Python test suite** by replacing deprecated `unittest` assert methods with their modern equivalents. This **maintenance fix** specifically targets `unittest.assertEquals` within the `bytes_comp` helper function in `thrift/test/py/SerializationTest.py`. By addressing these deprecated calls, the change ensures **compatibility with Python 3.12 and later**, preventing test failures that would occur due to the complete removal of these asserts in newer Python versions. This guarantees the continued reliability of the **Thrift serialization tests**.

This commit **fixes** a regression in the **Python build system** that prevented `DEFAULT_PY_VERSION` from loading correctly within the `pyrefly.bxl` script. It updates the `load` statement in `prelude/python/sourcedb/pyrefly.bxl` to point to the correct configuration file. This **bug fix** ensures the **default Python version** is consistently applied, including new conditional loading logic specifically for open-source builds, thereby restoring proper build behavior for Python projects.

This commit initiates the **migration of Python-related constants** from `fbcode defs` into the **Buck2 prelude's Python configuration**. It begins consolidating constants previously defined in `python_consts.bzl` into the `prelude/python/typing.bzl` file. This **refactoring** effort aims to centralize Python configuration within Buck2's core infrastructure. The immediate change involves commenting out a load statement and adding an OSS-enable comment to the `DEFAULT_PY_VERSION` constant, marking an initial step in this broader **internal infrastructure consolidation**.

This commit **enables `pyc` compilation** for **Python 3.14** by adjusting the version compatibility checks within the build system. It specifically updates the Python build rules in `prelude/python/python_binary.bzl` and `prelude/python/python_library.bzl` to extend support for `pyc` generation to Python 3.14. This **maintenance update** ensures that `python_executable` and `python_library_impl` targets built with Python 3.14 will now correctly produce and leverage compiled bytecode. The change primarily impacts the **Python build infrastructure**, allowing projects using Python 3.14 to benefit from potentially improved startup performance for **Python binaries and libraries**.

This commit performs a **maintenance update** by changing the default Python version constant within `prelude/python/typing.bzl` from 3.10 to **3.12**. This **chore** ensures that the project's Python build and typing configurations align with a more current interpreter version, as Python 3.10 is being deprecated. The change primarily impacts **Python development environments** and build processes that rely on this default, promoting the use of newer language features and improvements.