1# Precompiling 2 3Precompiling is compiling Python source files (`.py` files) into byte code 4(`.pyc` files) at build time instead of runtime. Doing it at build time can 5improve performance by skipping that work at runtime. 6 7Precompiling is disabled by default, so you must enable it using flags or 8attributes to use it. 9 10## Overhead of precompiling 11 12While precompiling helps runtime performance, it has two main costs: 131. Increasing the size (count and disk usage) of runfiles. It approximately 14 double the count of the runfiles because for every `.py` file, there is also 15 a `.pyc` file. Compiled files are generally around the same size as the 16 source files, so it approximately doubles the disk usage. 172. Precompiling requires running an extra action at build time. While 18 compiling itself isn't that expensive, the overhead can become noticable 19 as more files need to be compiled. 20 21## Binary-level opt-in 22 23Binary-level opt-in allows enabling precompiling on a per-target basic. This is 24useful for situations such as: 25 26* Globally enabling precompiling in your `.bazelrc` isn't feasible. This may 27 be because some targets don't work with precompiling, e.g. because they're too 28 big. 29* Enabling precompiling for build tools (exec config targets) separately from 30 target-config programs. 31 32To use this approach, set the {bzl:attr}`pyc_collection` attribute on the 33binaries/tests that should or should not use precompiling. Then change the 34{bzl:flag}`--precompile` default. 35 36The default for the {bzl:attr}`pyc_collection` attribute is controlled by the flag 37{bzl:obj}`--@rules_python//python/config_settings:precompile`, so you 38can use an opt-in or opt-out approach by setting its value: 39* targets must opt-out: `--@rules_python//python/config_settings:precompile=enabled` 40* targets must opt-in: `--@rules_python//python/config_settings:precompile=disabled` 41 42## Pyc-only builds 43 44A pyc-only build (aka "source less" builds) is when only `.pyc` files are 45included; the source `.py` files are not included. 46 47To enable this, set 48{bzl:obj}`--@rules_python//python/config_settings:precompile_source_retention=omit_source` 49flag on the command line or the {bzl:attr}`precompile_source_retention=omit_source` 50attribute on specific targets. 51 52The advantage of pyc-only builds are: 53* Fewer total files in a binary. 54* Imports _may_ be _slightly_ faster. 55 56The disadvantages are: 57* Error messages will be less precise because the precise line and offset 58 information isn't in an pyc file. 59* pyc files are Python major-version specific. 60 61:::{note} 62pyc files are not a form of hiding source code. They are trivial to uncompile, 63and uncompiling them can recover almost the original source. 64::: 65 66## Advanced precompiler customization 67 68The default implementation of the precompiler is a persistent, multiplexed, 69sandbox-aware, cancellation-enabled, json-protocol worker that uses the same 70interpreter as the target toolchain. This works well for local builds, but may 71not work as well for remote execution builds. To customize the precompiler, two 72mechanisms are available: 73 74* The exec tools toolchain allows customizing the precompiler binary used with 75 the {bzl:attr}`precompiler` attribute. Arbitrary binaries are supported. 76* The execution requirements can be customized using 77 `--@rules_python//tools/precompiler:execution_requirements`. This is a list 78 flag that can be repeated. Each entry is a key=value that is added to the 79 execution requirements of the `PyCompile` action. Note that this flag 80 is specific to the rules_python precompiler. If a custom binary is used, 81 this flag will have to be propagated from the custom binary using the 82 `testing.ExecutionInfo` provider; refer to the `py_interpreter_program` an 83 84The default precompiler implementation is an asynchronous/concurrent 85implementation. If you find it has bugs or hangs, please report them. In the 86meantime, the flag `--worker_extra_flag=PyCompile=--worker_impl=serial` can 87be used to switch to a synchronous/serial implementation that may not perform 88as well, but is less likely to have issues. 89 90The `execution_requirements` keys of most relevance are: 91* `supports-workers`: 1 or 0, to indicate if a regular persistent worker is 92 desired. 93* `supports-multiplex-workers`: 1 o 0, to indicate if a multiplexed persistent 94 worker is desired. 95* `requires-worker-protocol`: json or proto; the rules_python precompiler 96 currently only supports json. 97* `supports-multiplex-sandboxing`: 1 or 0, to indicate if sanboxing is of the 98 worker is supported. 99* `supports-worker-cancellation`: 1 or 1, to indicate if requests to the worker 100 can be cancelled. 101 102Note that any execution requirements values can be specified in the flag. 103 104## Known issues, caveats, and idiosyncracies 105 106* Precompiling requires Bazel 7+ with the Pystar rule implementation enabled. 107* Mixing rules_python PyInfo with Bazel builtin PyInfo will result in pyc files 108 being dropped. 109* Precompiled files may not be used in certain cases prior to Python 3.11. This 110 occurs due to Python adding the directory of the binary's main `.py` file, which 111 causes the module to be found in the workspace source directory instead of 112 within the binary's runfiles directory (where the pyc files are). This can 113 usually be worked around by removing `sys.path[0]` (or otherwise ensuring the 114 runfiles directory comes before the repos source directory in `sys.path`). 115* The pyc filename does not include the optimization level (e.g. 116 `foo.cpython-39.opt-2.pyc`). This works fine (it's all byte code), but also 117 means the interpreter `-O` argument can't be used -- doing so will cause the 118 interpreter to look for the non-existent `opt-N` named files. 119* Targets with the same source files and different exec properites will result 120 in action conflicts. This most commonly occurs when a `py_binary` and 121 `py_library` have the same source files. To fix, modify both targets so 122 they have the same exec properties. If this is difficult because unsupported 123 exec groups end up being passed to the Python rules, please file an issue 124 to have those exec groups added to the Python rules. 125