1.. _module-pw_build-python: 2 3------------------- 4Python GN templates 5------------------- 6The Python build is implemented with GN templates defined in 7``pw_build/python.gni``. See the .gni file for complete usage documentation. 8 9.. seealso:: :ref:`docs-python-build` 10 11.. _module-pw_build-pw_python_package: 12 13pw_python_package 14================= 15The main Python template is ``pw_python_package``. Each ``pw_python_package`` 16target represents a Python package. As described in 17:ref:`module-pw_build-python-target`, each ``pw_python_package`` expands to 18several subtargets. In summary, these are: 19 20- ``<name>`` - Represents the files themselves 21- ``<name>.lint`` - Runs static analysis 22- ``<name>.tests`` - Runs all tests for this package 23- ``<name>.install`` - Installs the package 24- ``<name>.wheel`` - Builds a Python wheel 25 26GN permits using abbreviated labels when the target name matches the directory 27name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, Python 28package subtargets are aliased to the directory when the target name is the 29same as the directory. For example, these two labels are equivalent: 30 31.. code-block:: 32 33 //path/to/my_python_package:my_python_package.tests 34 //path/to/my_python_package:tests 35 36The actions in a ``pw_python_package`` (e.g. installing packages and running 37Pylint) are done within a single GN toolchain to avoid duplication in 38multi-toolchain builds. This toolchain can be set with the 39``pw_build_PYTHON_TOOLCHAIN`` GN arg, which defaults to 40``$dir_pw_build/python_toolchain:python``. 41 42Arguments 43--------- 44- ``setup`` - List of setup file paths (setup.py or pyproject.toml & setup.cfg), 45 which must all be in the same directory. 46- ``generate_setup``: As an alternative to ``setup``, generate setup files with 47 the keywords in this scope. ``name`` is required. This follows the same 48 structure as a ``setup.cfg`` file's `declarative config 49 <https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html>`_ 50 For example: 51 52 .. code-block:: 53 54 generate_setup = { 55 metadata = { 56 name = "a_nifty_package" 57 version = "1.2a" 58 } 59 options = { 60 install_requires = [ "a_pip_package" ] 61 } 62 } 63 64- ``sources`` - Python sources files in the package. 65- ``tests`` - Test files for this Python package. 66- ``python_deps`` - Dependencies on other pw_python_packages in the GN build. 67- ``python_test_deps`` - Test-only pw_python_package dependencies. 68- ``other_deps`` - Dependencies on GN targets that are not pw_python_packages. 69- ``inputs`` - Other files to track, such as package_data. 70- ``proto_library`` - A pw_proto_library target to embed in this Python package. 71 ``generate_setup`` is required in place of setup if proto_library is used. See 72 :ref:`module-pw_protobuf_compiler-add-to-python-package`. 73- ``static_analysis`` List of static analysis tools to run; ``"*"`` (default) 74 runs all tools. The supported tools are ``"mypy"`` and ``"pylint"``. 75- ``pylintrc`` - Optional path to a pylintrc configuration file to use. If not 76 provided, Pylint's default rcfile search is used. Pylint is executed 77 from the package's setup directory, so pylintrc files in that directory 78 will take precedence over others. 79- ``mypy_ini`` - Optional path to a mypy configuration file to use. If not 80 provided, mypy's default configuration file search is used. mypy is 81 executed from the package's setup directory, so mypy.ini files in that 82 directory will take precedence over others. 83 84Example 85------- 86This is an example Python package declaration for a ``pw_my_module`` module. 87 88.. code-block:: 89 90 import("//build_overrides/pigweed.gni") 91 92 import("$dir_pw_build/python.gni") 93 94 pw_python_package("py") { 95 setup = [ 96 "pyproject.toml", 97 "setup.cfg", 98 "setup.py", 99 ] 100 sources = [ 101 "pw_my_module/__init__.py", 102 "pw_my_module/alfa.py", 103 "pw_my_module/bravo.py", 104 "pw_my_module/charlie.py", 105 ] 106 tests = [ 107 "alfa_test.py", 108 "charlie_test.py", 109 ] 110 python_deps = [ 111 "$dir_pw_status/py", 112 ":some_protos.python", 113 ] 114 python_test_deps = [ "$dir_pw_build/py" ] 115 pylintrc = "$dir_pigweed/.pylintrc" 116 } 117 118pw_python_script 119================ 120A ``pw_python_script`` represents a set of standalone Python scripts and/or 121tests. These files support all of the arguments of ``pw_python_package`` except 122those ``setup``. These targets can be installed, but this only installs their 123dependencies. 124 125``pw_python_script`` allows creating a 126:ref:`pw_python_action <module-pw_build-python-action>` associated with the 127script. To create an action, pass an ``action`` scope to ``pw_python_script``. 128If there is only a single source file, it serves as the action's ``script`` by 129default. 130 131An action in ``pw_python_script`` can always be replaced with a standalone 132``pw_python_action``, but using the embedded action has some advantages: 133 134- The embedded action target bridges the gap between actions and Python targets. 135 A Python script can be expressed in a single, concise GN target, rather than 136 in two overlapping, dependent targets. 137- The action automatically depends on the ``pw_python_script``. This ensures 138 that the script's dependencies are installed and the action automatically 139 reruns when the script's sources change, without needing to specify a 140 dependency, a step which is easy to forget. 141- Using a ``pw_python_script`` with an embedded action is a simple way to check 142 an existing action's script with Pylint or Mypy or to add tests. 143 144pw_python_group 145=============== 146Represents a group of ``pw_python_package`` and ``pw_python_script`` targets. 147These targets do not add any files. Their subtargets simply forward to those of 148their dependencies. 149 150pw_python_requirements 151====================== 152Represents a set of local and PyPI requirements, with no associated source 153files. These targets serve the role of a ``requirements.txt`` file. 154 155When packages are installed by Pigweed, additional version constraints can be 156provided using the ``pw_build_PIP_CONSTRAINTS`` GN arg. This option should 157contain a list of paths to pass to the ``--constraint`` option of ``pip 158install``. This can be used to synchronize dependency upgrades across a project 159which facilitates reproducibility of builds. 160 161Note using multiple ``pw_python_requirements`` that install different versions 162of the same package will currently cause unpredictable results, while using 163constraints should have correct results (which may be an error indicating a 164conflict). 165 166.. _module-pw_build-python-dist: 167 168--------------------- 169Python Distributables 170--------------------- 171Pigweed also provides some templates to make it easier to bundle Python packages 172for deployment. These templates are found in ``pw_build/python_dist.gni``. See 173the .gni file for complete documentation on building distributables. 174 175pw_python_wheels 176================ 177Collects Python wheels for one or more ``pw_python_package`` targets, plus any 178additional ``pw_python_package`` targets they depend on, directly or indirectly. 179Note that this does not include Python dependencies that come from outside the 180GN build, like packages from PyPI, for example. Those should still be declared 181in the package's ``setup.py`` file as usual. 182 183Arguments 184--------- 185- ``packages`` - List of ``pw_python_package`` targets whose wheels should be 186 included; their dependencies will be pulled in as wheels also. 187- ``directory`` - Output directory for the collected wheels. Defaults to 188 ``$target_out_dir/$target_name``. 189 190Wheel collection under the hood 191------------------------------- 192The ``.wheel`` subtarget of every ``pw_python_package`` generates a wheel 193(``.whl``) for the Python package. The ``pw_python_wheels`` template figures 194out which wheels to collect by traversing the ``pw_python_package_wheels`` 195`GN metadata 196<https://gn.googlesource.com/gn/+/HEAD/docs/reference.md#var_metadata>`_ key, 197which lists the output directory for each wheel. 198 199pw_python_zip_with_setup 200======================== 201Generates a ``.zip`` archive suitable for deployment outside of the project's 202developer environment. The generated ``.zip`` contains Python wheels 203(``.whl`` files) for one or more ``pw_python_package`` targets, plus wheels for 204any additional ``pw_python_package`` targets in the GN build they depend on, 205directly or indirectly. Dependencies from outside the GN build, such as packages 206from PyPI, must be listed in packages' ``setup.py`` or ``setup.cfg`` files as 207usual. 208 209The ``.zip`` also includes simple setup scripts for Linux, 210MacOS, and Windows. The setup scripts automatically create a Python virtual 211environment and install the whole collection of wheels into it using ``pip``. 212 213Optionally, additional files and directories can be included in the archive. 214One common example of an additional file to include is a README file with setup 215and usage instructions for the distributable. A simple ready-to-use README file 216is available at ``pw_build/py_dist/README.md``. 217 218Arguments 219--------- 220- ``packages`` - A list of `pw_python_package` targets whose wheels should be 221 included; their dependencies will be pulled in as wheels also. 222- ``inputs`` - An optional list of extra files to include in the generated 223 ``.zip``, formatted the same way as the ``inputs`` argument to ``pw_zip`` 224 targets. 225- ``dirs`` - An optional list of directories to include in the generated 226 ``.zip``, formatted the same was as the ``dirs`` argument to ``pw_zip`` 227 targets. 228 229Example 230------- 231 232.. code-block:: 233 234 import("//build_overrides/pigweed.gni") 235 236 import("$dir_pw_build/python_dist.gni") 237 238 pw_python_zip_with_setup("my_tools") { 239 packages = [ ":some_python_package" ] 240 inputs = [ "$dir_pw_build/python_dist/README.md > /${target_name}/" ] 241 } 242 243pw_create_python_source_tree 244============================ 245 246Generates a directory of Python packages from source files suitable for 247deployment outside of the project developer environment. The resulting directory 248contains only files mentioned in each package's ``setup.cfg`` file. This is 249useful for bundling multiple Python packages up into a single package for 250distribution to other locations like `<http://pypi.org>`_. 251 252Arguments 253--------- 254 255- ``packages`` - A list of :ref:`module-pw_build-pw_python_package` targets to be installed into 256 the build directory. Their dependencies will be pulled in as wheels also. 257 258- ``include_tests`` - If true, copy Python package tests to a ``tests`` subdir. 259 260- ``extra_files`` - A list of extra files that should be included in the output. 261 The format of each item in this list follows this convention: 262 263 .. code-block:: text 264 265 //some/nested/source_file > nested/destination_file 266 267 - Source and destination file should be separated by ``>``. 268 269 - The source file should be a GN target label (starting with ``//``). 270 271 - The destination file will be relative to the generated output 272 directory. Parent directories are automatically created for each file. If a 273 file would be overwritten an error is raised. 274 275- ``generate_setup_cfg`` - If included, create a merged ``setup.cfg`` for all 276 python Packages using a ``common_config_file`` as a base. That file should 277 contain the required fields in the ``metadata`` and ``options`` sections as 278 shown in 279 `Configuring setup() using setup.cfg files <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html>`_. 280 ``append_git_sha_to_version`` and ``append_date_to_version`` will optionally 281 append the current git SHA or date to the package version string after a ``+`` 282 sign. 283 284 .. code-block:: 285 286 generate_setup_cfg = { 287 common_config_file = "pypi_common_setup.cfg" 288 append_git_sha_to_version = true 289 append_date_to_version = true 290 } 291 292Example 293------- 294 295:octicon:`file;1em` ./pw_env_setup/BUILD.gn 296 297.. code-block:: 298 299 import("//build_overrides/pigweed.gni") 300 301 import("$dir_pw_build/python_dist.gni") 302 303 pw_create_python_source_tree("build_python_source_tree") { 304 packages = [ 305 ":some_python_package", 306 ":another_python_package", 307 ] 308 include_tests = true 309 extra_files = [ 310 "//README.md > ./README.md", 311 "//some_python_package/py/BUILD.bazel > some_python_package/BUILD.bazel", 312 "//another_python_package/py/BUILD.bazel > another_python_package/BUILD.bazel", 313 ] 314 generate_setup_cfg = { 315 common_config_file = "pypi_common_setup.cfg" 316 append_git_sha_to_version = true 317 append_date_to_version = true 318 } 319 } 320 321:octicon:`file-directory;1em` ./out/obj/pw_env_setup/build_python_source_tree/ 322 323.. code-block:: text 324 325 $ tree ./out/obj/pw_env_setup/build_python_source_tree/ 326 ├── README.md 327 ├── setup.cfg 328 ├── some_python_package 329 │ ├── BUILD.bazel 330 │ ├── __init__.py 331 │ ├── py.typed 332 │ ├── some_source_file.py 333 │ └── tests 334 │ └── some_source_test.py 335 └── another_python_package 336 ├── BUILD.bazel 337 ├── __init__.py 338 ├── another_source_file.py 339 ├── py.typed 340 └── tests 341 └── another_source_test.py 342