Lines Matching +full:- +full:python +full:- +full:pip
1 .. _docs-python-build:
4 Pigweed's GN Python Build
8 - :bdg-ref-primary-line:`module-pw_build-python` for detailed template usage.
9 - :bdg-ref-primary-line:`module-pw_build` for other GN templates available
11 - :bdg-ref-primary-line:`docs-build-system` for a high level guide and
14 Pigweed uses a custom GN-based build system to manage its Python code. The
15 Pigweed Python build supports packaging, installation and distribution of
16 interdependent local Python packages. It also provides for fast, incremental
18 with :ref:`module-pw_watch`) or in continuous integration.
20 Pigweed's Python code is exclusively managed by GN, but the GN-based build may
22 setup uses GN to set up the initial Python environment, regardless of the final
23 build system. As needed, non-GN projects can declare just their Python packages
28 In addition to compiler commands a Pigweed GN build will execute Python scripts
31 :ref:`module-pw_build-pw_python_action` GN template which will ultimately run
32 ``python``. Running Python on it's own by default will make any Python packages
35 workstation. To get around this the Python community uses `virtual environments
36 <https://docs.python.org/3/library/venv.html>`_ (venvs) that expose a specific
37 set of Python packages separate from the host system.
40 :ref:`pw_python_actions <module-pw_build-pw_python_action>` throughout the build
41 graph. Once created, all required third-party Python packages needed for the
43 the venv. Of course if a new third-party package dependency is added it will be
45 with the :ref:`module-pw_build-pw_python_venv` template if desired, but only one
56 out[GN Build Dir<br/>fa:fa-folder out]
58 out -->|ninja -C out| createvenvs
61 createvenvs --> pyactions1
62 createvenvs --> pyactions2
64 subgraph pyactions1[Python venv 1]
66 venv1(fa:fa-folder out/python-venv  )
69 venv1 --> a1
70 venv1 --> a2
73 subgraph pyactions2[Python venv 2]
75 venv2(fa:fa-folder out/another-venv  )
78 venv2 --> a3
79 venv2 --> a4
87 :bdg-ref-primary-line:`docs-python-build-python-gn-venv` on how to define
90 Having a static venv containing only third-party dependencies opens the flood
91 gates for python scripts to run. If the venv only contains third-party
92 dependencies you may be wondering how you can import your own in-tree Python
93 packages. Python code run in the build may still import any in-tree Python
94 packages created with :ref:`module-pw_build-pw_python_package`
96 provided. Having that Python dependency defined in GN allows the
97 :ref:`module-pw_build-pw_python_action`
99 <https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH>`_ so that given
101 dependency for any Python action or package is missing.
103 .. admonition:: Benefits of Python ``venvs`` in GN
106 - Using venvs to execute Python in GN provides reproducible builds with fixed
107 third-party dependencies.
108 - Using ``PYTHONPATH`` coupled with ``python_deps`` to import in-tree Python
112 Managing Python Requirements
115 .. _docs-python-build-python-gn-venv:
117 Build Time Python Virtualenv
118 ----------------------------
119 Pigweed's GN Python build infrastructure relies on `Python virtual environments
120 <https://docs.python.org/3/library/venv.html>`_ for executing Python code. This
122 Python constraints where all Python tests, linting and
123 :ref:`module-pw_build-pw_python_action` targets are executed.
125 There must be at least one venv for Python defined in GN. There can be multiple
131 .. code-block::
136 Additional :ref:`module-pw_build-pw_python_venv` targets can be created as
137 needed. The :ref:`module-pw_build-pw_python_action` template can take an
138 optional ``venv`` argument to specify which Python venv it should run
142 .. _docs-python-build-python-gn-requirements-files:
144 Third-party Python Requirements and Constraints
145 -----------------------------------------------
146 Your project may have third party Python dependencies you wish to install into
148 to add Python package dependencies:
153 :ref:`module-pw_build-pw_python_package` template. This is the best option
154 if your in-tree Python package requires an external Python package.
156 2. Create a standard Python ``requirements.txt`` file in your project and add it
160 packages from pypi.org, the local file system and git repos. See `pip's
162 <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_ for more
167 .. code-block::
174 See the :ref:`docs-python-build-python-gn-structure` section below for a full
179 Every project should ideally inherit Pigweed's third party Python package
180 version. This is accomplished via `Python constraints files
181 <https://pip.pypa.io/en/stable/user_guide/#constraints-files>`_. Constraints
182 control which versions of packages get installed by ``pip`` if that package is
183 installed. To inherit Pigweed's Python constraints include ``constraint.list``
187 .. code-block::
194 In-tree ``pw_python_package`` Requirements
195 ------------------------------------------
202 To ensure the requirements of in-tree :ref:`module-pw_build-pw_python_package`
203 targets are installed :ref:`module-pw_build-pw_python_venv` introduces the
204 ``source_packages`` argument. This is a list of in-tree ``pw_python_package``
208 all in-tree packages and any in-tree transitive dependencies is then written to
216 adds ``-c`` lines for constraint files.
219 The pip documentation on the `Requirements File Format
220 <https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format>`_
223 :start-after: [downstream-project-venv]
224 :end-before: [downstream-project-venv]
226 .. code-block::
227 …:caption: :octicon:`file;1em` out/python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/gene…
230 # Auto-generated requirements.txt from the following packages:
244 -c ../../../../../../../pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list
254 pip-tools>=6.12.3
255 prompt-toolkit>=3.0.26
266 types-pygments
267 types-pyserial>=3.5,<4.0
268 types-pyyaml
269 types-setuptools
270 types-six
278 ``pip-compile`` command from `the pip-tools package
279 <https://pypi.org/project/pip-tools>`_ to fully expand and pin each package with
281 single Python requirements file for replicating this ``pw_python_venv``
283 exact versions of each required Python package.
287 by the :ref:`module-pw_build-pw_python_zip_with_setup` template when
288 producing a self contained zip of in-tree and third party Python packages.
291 :ref:`module-pw_build-pw_python_venv` target:
294 .. code-block::
295 …:caption: :octicon:`file;1em` out/python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/comp…
299 # This file is autogenerated by pip-compile with Python 3.11
302 # pip-compile --allow-unsafe --generate-hashes
303 …# --output-file=python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/compiled_requirem…
304 # --resolver=backtracking
305 # python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/generated_requirements.txt
308 --hash=sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41 \
309 --hash=sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128
311 …# -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_set…
314 --hash=sha256:0e0e3709d64fbffd3037e4ff403580550f14471fd3eaae9fa11cc9a5c7901153 \
315 --hash=sha256:a3cf9f02c53dd259144a7e8f3ccd75d67c9a8c716ef183e0c1f291bc5d7bb3cf
317 …# -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_set…
322 ``pip_generate_hashes`` arg to the :ref:`module-pw_build-pw_python_venv`
325 Caching Python Packages for Offline Installation
326 ------------------------------------------------
328 .. _docs-python-build-downloading-packages:
332 The :ref:`module-pw_build-pw_python_venv` target adds an optional sub target
333 that will download all Python packages from remote servers into a local
345 .. code-block:: bash
348 ninja -C out \
349 $(gn ls out --as=output \
353 ``pip_download_log.txt`` with verbose logs from running ``pip download``.
355 .. code-block::
356 :caption: :octicon:`file-directory;1em` Vendor wheels output directory
357 :name: vendor-wheel-output
359 out/python/gen/pw_build/py/gn_tests/downstream_tools_build_venv.vendor_wheels/
362 ├── appdirs-1.4.4-py2.py3-none-any.whl
363 ├── astroid-2.14.2-py3-none-any.whl
364 ├── backcall-0.2.0-py2.py3-none-any.whl
365 ├── black-23.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
367 …├── websockets-10.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manyl…
368 ├── wheel-0.40.0-py3-none-any.whl
369 ├── wrapt-1.14.1.tar.gz
370 └── yapf-0.31.0-py2.py3-none-any.whl
372 Note the above output has both Python wheel ``.whl`` and source distribution
373 ``.tar.gz`` files. The ``.whl`` may contain Python packages with precompiled C
375 ``cp311-cp311-manylinux_2_17_x86_64.whl``. These binary packages are selected by
376 the ``pip download`` command based on the host machine python version, OS, and
380 If you need to cache Python packages for multiple platforms the
381 ``.vendor_wheels`` target will need to be run for each combination of Python
386 - cp311, manylinux_2_17_x86_64
387 - cp311, manylinux2014_x86_64
388 - cp311, macosx_11_0_arm64
389 - cp311, macosx_10_9_x86_64
390 - cp311, win_amd64
391 - cp311, win32
393 Plus all of the above duplicated for Python 3.10 and 3.9 (``cp310`` and
402 .. code-block::
406 This will invoke `pip download
407 <https://pip.pypa.io/en/stable/cli/pip_download/>`_ for each combination of
408 platform, architecture and Python version. This can take a significant amount of
412 :start-after: [wheel-platform-args]
413 :end-before: [wheel-platform-args]
416 The set of Python packages that will be downloaded is determined by the
418 current host OS and Python version. `pip-tools
419 <https://pypi.org/project/pip-tools>`_ does not expand requirements for
421 …<https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#platform-specific-depen…
424 .. code-block::
429 If pip-tools is run on Linux then the above packages will not appear in
433 .. _docs-python-build-installing-offline:
438 this as the default pip install location in two different ways.
442 Setting these args in the ``//.gn`` file will add the relevant pip command line
445 .. code-block::
447 # Adds --no-index forcing pip to not reach out to the internet (pypi.org) to
452 # List of paths to folders containing Python wheels (*.whl) or source tar
453 # files (*.tar.gz). Pip will check each of these directories when looking for
460 # Optional: Adds '--no-cache-dir' forcing pip to ignore any previously cached
461 # Python packages. On most systems this is located in ~/.cache/pip/
464 Using a ``.pip.conf`` File
466 1. Create a ``//pip.conf`` file containing:
468 .. code-block::
469 :caption: :octicon:`file;1em` //pip.conf
470 :name: pip-conf-file
474 no-index = True
476 find-links =
480 This tells pip to not search pypi.org for packages and only look in
483 are relative to the ``pip.conf`` file.
488 .. code-block:: bash
490 export PIP_CONFIG_FILE="${PW_PROJECT_ROOT}/pip.conf"
492 With that environment var set all invocations of pip will apply the config file
496 The ``pip`` `documentation on Configuration
497 <https://pip.pypa.io/en/stable/topics/configuration/>`_.
499 .. _docs-python-build-python-gn-structure:
501 GN File Structure for Python Code
503 Here is a full example of what is required to build Python packages using
505 content following. See also :ref:`docs-python-build-structure` below for details
506 on the structure of Python packages.
508 .. code-block::
509 :caption: :octicon:`file-directory;1em` Top level GN file hierarchy
510 :name: gn-python-file-tree
540 - :octicon:`file-directory;1em` project_root/
542 - :octicon:`file;1em` .gn
544 .. code-block::
551 # Inherit Pigweed Python constraints
569 There are some additional gn args to control how pip installations are
573 :start-after: [default-pip-gn-args]
574 :end-before: [default-pip-gn-args]
576 - :octicon:`file;1em` BUILDCONFIG.gn
578 .. code-block::
586 - :octicon:`file-directory;1em` build_overrides / :octicon:`file;1em` pigweed.gni
588 .. code-block::
598 - :octicon:`file;1em` BUILD.gn
600 .. code-block::
604 import("$dir_pw_build/python.gni")
609 # Lists all the targets build by default with e.g. `ninja -C out`.
612 ":python.lint",
613 ":python.tests",
617 # This group is built during bootstrap to setup the interactive Python
619 pw_python_group("python") {
621 # Generate and pip install _all_python_packages
626 # In-tree Python packages
631 # Pigweed Python packages to include
641 # The default venv for Python actions in GN
648 path = "$root_build_dir/python-venv"
652 # Ensure all third party Python dependencies are installed into this venv.
658 # This template collects all python packages and their dependencies into a
659 # single super Python package for installation into the bootstrapped virtual
664 name = "project-tools"
671 # Install the project-tools super Python package into the bootstrapped
672 # Python venv.
677 .. _docs-python-build-structure:
679 Pigweed Module Structure for Python Code
681 Pigweed Python code is structured into standard Python packages. This makes it
682 simple to package and distribute Pigweed Python packages with common Python
685 Like all Pigweed source code, Python packages are organized into Pigweed
686 modules. A module's Python package is nested under a ``py/`` directory (see
687 :ref:`Pigweed Module Stucture <docs-module-structure>`).
689 .. code-block::
690 :caption: :octicon:`file-directory;1em` Example layout of a Pigweed Python package.
691 :name: python-file-tree
710 check in ensures that all Python files are listed in a ``BUILD.gn``.
712 Pigweed prefers to define Python packages using ``setup.cfg`` files. In the
716 .. code-block::
718 :name: pyproject-toml-stub
720 [build-system]
722 build-backend = 'setuptools.build_meta'
725 used for the given Python package. In Pigweed's case it always specifies using
730 - ``setup.cfg`` examples at `Configuring setup() using setup.cfg files`_
731 - ``pyproject.toml`` background at `Build System Support - How to use it?`_
733 .. _module-pw_build-python-target:
736 -------------------------
737 The key abstraction in the Python build is the ``pw_python_package``.
738 A ``pw_python_package`` represents a Python package as a GN target. It is
740 in :ref:`module-pw_build-python`.
744 - a ``setup.cfg`` and ``pyproject.toml`` file,
745 - source files,
746 - test files,
747 - dependencies on other ``pw_python_package`` targets.
750 subtarget represents different functionality in the Python build.
752 - ``<name>`` - Represents the Python files in the build, but does not take any
754 - ``<name>.tests`` - Runs all tests for this package.
756 - ``<name>.tests.<test_file>`` - Runs the specified test.
758 - ``<name>.lint`` - Runs static analysis tools on the Python code. This is a
761 - ``<name>.lint.mypy`` - Runs Mypy on all Python files, if enabled.
762 - ``<name>.lint.pylint`` - Runs Pylint on all Python files, if enabled.
763 - ``<name>.lint.ruff`` - Runs ruff on all Python files, if enabled.
765 - ``<name>.install`` - Installs the package in a Python virtual environment.
766 - ``<name>.wheel`` - Builds a Python wheel for this package.
768 To avoid unnecessary duplication, all Python actions are executed in the default
773 Tests for a Python package are listed in its ``pw_python_package`` target.
775 Python package. The build will run it when the test, the package, or one of its
782 and ``ruff_toml`` files to use on a per-package basis. The configuration files
794 .. literalinclude:: pw_build/python.gni
795 :start-after: [python-static-analysis-tools]
796 :end-before: [python-static-analysis-tools]
800 ``//pw_build/python.gni`` below:
802 .. literalinclude:: pw_build/python.gni
803 :start-after: [default-mypy-args]
804 :end-before: [default-mypy-args]
806 Building Python wheels
809 distributing Python packages. The Pigweed Python build supports creating wheels
812 <https://pypa-build.readthedocs.io/en/stable/>`_ tool.
815 :ref:`module-pw_build-pw_python_distribution` records the location of the
818 Wheels for a Python package and its transitive dependencies can be collected
820 :ref:`module-pw_build-python-dist`.
825 target (see :ref:`module-pw_protobuf_compiler`). Python protobuf modules are
826 generated as standalone Python packages by default. Protocol buffers may also be
827 nested within existing Python packages. In this case, the Python package in the
828 source tree is incomplete; the final Python package, including protobufs, is
848 ----------
855 Python is an interpreted language, but it shares most build automation concerns
856 with other languages. Pigweed uses Python extensively and must address these
860 ------------------
861 The Python programming langauge does not have an official build automation
862 system. However, there are numerous Python-focused build automation tools with
863 varying degrees of adoption. See the `Python Wiki
864 <https://wiki.python.org/moin/ConfigurationAndBuildTools>`_ for examples.
866 A few Python tools have become defacto standards, including `setuptools
868 <https://pypi.org/project/wheel/>`_, and `pip <https://pypi.org/project/pip/>`_.
869 These essential tools address key aspects of Python packaging and distribution,
872 provide more general build automation for Python.
875 Python and other languages used by Pigweed, including protocol buffers.
878 ----------
879 Pigweed's use of Python is different from many other projects. Pigweed is a
880 multi-language, modular project. It serves both as a library or middleware and
883 This section describes Python build automation challenges encountered by
888 Pigweed is organized into distinct modules. In Python, each module is a separate
892 The basic Python packaging tools lack dependency tracking for local packages.
894 ``pip`` is not aware of local packages until they are installed. Packages must
906 by contributing to the long-term resilience of a codebase. Despite their
909 bug-prone codebases.
911 There are lots of great Python libraries for testing, such as
912 `unittest <https://docs.python.org/3/library/unittest.html>`_ and
914 write and execute individual Python tests, but are not well suited for managing
924 :bdg-ref-primary-line:`docs-automated-analysis` for info on other static
927 Various static analysis tools exist for Python. Two widely used, powerful tools
928 are `Pylint <https://www.pylint.org/>`_ and `Mypy <http://mypy-lang.org/>`_.
937 These tools do not have built-in support for incremental runs or dependency
948 `Protocol buffers <https://developers.google.com/protocol-buffers>`_ are an
952 The protobuf compiler ``protoc`` generates Python modules from ``.proto`` files.
959 ------------
961 flexible development experience for its customers. Pigweed's high-level goals
963 Python build.
965 - Integrate seamlessly with the other Pigweed build tools.
966 - Easy to use independently, even if primarily using a different build system.
967 - Support standard packaging and distribution with setuptools, wheel, and pip.
968 - Correctly manage interdependent local Python packages.
969 - Out-of-the-box support for writing and running tests.
970 - Preconfigured, trivial-to-run static analysis integration for Pylint and Mypy.
971 - Fast, dependency-aware incremental rebuilds and test execution, suitable for
972 use with :ref:`module-pw_watch`.
973 - Seamless protocol buffer support.
976 ---------------
977 Existing Python tools may be effective for Python codebases, but their utility
978 is more limited in a multi-language project like Pigweed. The cost of bringing
983 natively support Python, adding support is straightforward with GN templates.
985 GN has strong multi-toolchain and multi-language capabilities. In GN, it is
987 example, C++, Go, and Python targets can depend on the same protobuf
993 clean syntax. This makes it feasible to use GN only for Python while building
996 Given these considerations, GN is an ideal choice for Pigweed's Python build.
999 …ild System Support - How to use it?: https://setuptools.readthedocs.io/en/latest/build_meta.html?h…