Lines Matching +full:build +full:- +full:packages
1 .. _docs-python-build:
4 Pigweed's GN Python Build
9 - :bdg-ref-primary-line:`module-pw_build-python` for detailed template usage.
10 - :bdg-ref-primary-line:`module-pw_build` for other GN templates available
12 - :bdg-ref-primary-line:`docs-build-system` for a high level guide and
13 background information on Pigweed's build system as a whole.
15 Pigweed uses a custom GN-based build system to manage its Python code. The
16 Pigweed Python build supports packaging, installation and distribution of
17 interdependent local Python packages. It also provides for fast, incremental
19 with :ref:`module-pw_watch`) or in continuous integration.
21 Pigweed's Python code is exclusively managed by GN, but the GN-based build may
22 be used alongside CMake, Bazel, or any other build system. Pigweed's environment
24 build system. As needed, non-GN projects can declare just their Python packages
29 In addition to compiler commands a Pigweed GN build will execute Python scripts
32 :ref:`module-pw_build-pw_python_action` GN template which will ultimately run
33 ``python``. Running Python on it's own by default will make any Python packages
35 lead to flaky builds when different packages are installed on each developer
38 set of Python packages separate from the host system.
40 When a Pigweed GN build starts a single venv is created for use by all
41 :ref:`pw_python_actions <module-pw_build-pw_python_action>` throughout the build
42 graph. Once created, all required third-party Python packages needed for the
44 the venv. Of course if a new third-party package dependency is added it will be
46 with the :ref:`module-pw_build-pw_python_venv` template if desired, but only one
57 out[GN Build Dir<br/>fa:fa-folder out]
59 out -->|ninja -C out| createvenvs
62 createvenvs --> pyactions1
63 createvenvs --> pyactions2
67 venv1(fa:fa-folder out/python-venv  )
70 venv1 --> a1
71 venv1 --> a2
76 venv2(fa:fa-folder out/another-venv  )
79 venv2 --> a3
80 venv2 --> a4
86 … <https://cs.opensource.google/pigweed/pigweed/+/main:pw_env_setup/BUILD.gn?q=pigweed_build_venv>`_
87 if a project does not specify it's own build venv. See
88 :bdg-ref-primary-line:`docs-python-build-python-gn-venv` on how to define
91 Having a static venv containing only third-party dependencies opens the flood
92 gates for python scripts to run. If the venv only contains third-party
93 dependencies you may be wondering how you can import your own in-tree Python
94 packages. Python code run in the build may still import any in-tree Python
95 packages created with :ref:`module-pw_build-pw_python_package`
98 :ref:`module-pw_build-pw_python_action`
100 <https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH>`_ so that given
101 package can be imported. This has the benefit of the build failing if a
107 - Using venvs to execute Python in GN provides reproducible builds with fixed
108 third-party dependencies.
109 - Using ``PYTHONPATH`` coupled with ``python_deps`` to import in-tree Python
110 packages enforces dependency correctness.
116 .. _docs-python-build-python-gn-venv:
118 Build Time Python Virtualenv
119 ----------------------------
120 Pigweed's GN Python build infrastructure relies on a single build-only venv for
123 run. All :ref:`module-pw_build-pw_python_action` targets are executed within
124 this build venv.
126 The default build venv is specified via a GN arg and is best set in the root
127 ``.gn`` or ``BUILD.gn`` file. For example:
129 .. code-block::
133 .. _docs-python-build-python-gn-requirements-files:
135 Third-party Python Requirements and Constraints
136 -----------------------------------------------
138 the bootstrapped environment and in the GN build venv. There are two main ways
144 :ref:`module-pw_build-pw_python_package` template. This is the best option
145 if your in-tree Python package requires an external Python package.
151 packages from pypi.org, the local file system and git repos. See `pip's
153 <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_ for more
156 The GN arg can be set in your project's root ``.gn`` or ``BUILD.gn`` file.
158 .. code-block::
165 See the :ref:`docs-python-build-python-gn-structure` section below for a full
172 <https://pip.pypa.io/en/stable/user_guide/#constraints-files>`_. Constraints
173 control which versions of packages get installed by ``pip`` if that package is
178 .. code-block::
185 .. _docs-python-build-python-gn-structure:
189 Here is a full example of what is required to build Python packages using
190 Pigweed's GN build system. A brief file hierarchy is shown here with file
191 content following. See also :ref:`docs-python-build-structure` below for details
192 on the structure of Python packages.
194 .. code-block::
195 :caption: :octicon:`file-directory;1em` Top level GN file hierarchy
196 :name: gn-python-file-tree
203 ├── BUILD.gn
206 │ ├── BUILD.gn
227 - :octicon:`file-directory;1em` project_root/
229 - :octicon:`file;1em` .gn
231 .. code-block::
250 # Default gn build virtualenv target.
254 - :octicon:`file;1em` BUILDCONFIG.gn
256 .. code-block::
264 - :octicon:`file-directory;1em` build_overrides / :octicon:`file;1em` pigweed.gni
266 .. code-block::
276 - :octicon:`file;1em` BUILD.gn
278 .. code-block::
287 # Lists all the targets build by default with e.g. `ninja -C out`.
304 # In-tree Python packages
309 # Pigweed Python packages to include
320 # Set this gn arg in a declare_args block in this file 'BUILD.gn' or in '.gn' to
326 path = "$root_build_dir/python-venv"
331 # This works by checking the setup.cfg files for all packages listed here and
332 # installing the packages listed in the [options].install_requires field.
336 # This template collects all python packages and their dependencies into a
340 packages = _all_python_packages
342 name = "project-tools"
349 # Install the project-tools super Python package into the bootstrapped
352 packages = [ ":generate_project_python_distribution" ]
355 .. _docs-python-build-structure:
359 Pigweed Python code is structured into standard Python packages. This makes it
360 simple to package and distribute Pigweed Python packages with common Python
363 Like all Pigweed source code, Python packages are organized into Pigweed
365 :ref:`Pigweed Module Stucture <docs-module-structure>`).
367 .. code-block::
368 :caption: :octicon:`file-directory;1em` Example layout of a Pigweed Python package.
369 :name: python-file-tree
373 │ ├── BUILD.gn
388 The ``BUILD.gn`` declares this package in GN. For upstream Pigweed, a presubmit
389 check in ensures that all Python files are listed in a ``BUILD.gn``.
391 Pigweed prefers to define Python packages using ``setup.cfg`` files. In the
395 .. code-block::
397 :name: setup-py-stub
402 .. code-block::
404 :name: pyproject-toml-stub
406 [build-system]
408 build-backend = 'setuptools.build_meta'
410 The stub ``setup.py`` file is there to support running ``pip install --editable``.
412 Each ``pyproject.toml`` file is required to specify which build system should be
418 - ``setup.cfg`` examples at `Configuring setup() using setup.cfg files`_
419 - ``pyproject.toml`` background at `Build System Support - How to use it?`_
422 .. _module-pw_build-python-target:
425 -------------------------
426 The key abstraction in the Python build is the ``pw_python_package``.
429 in :ref:`module-pw_build-python`.
433 - a ``setup.py`` file,
434 - source files,
435 - test files,
436 - dependencies on other ``pw_python_package`` targets.
439 subtarget represents different functionality in the Python build.
441 - ``<name>`` - Represents the Python files in the build, but does not take any
443 - ``<name>.tests`` - Runs all tests for this package.
445 - ``<name>.tests.<test_file>`` - Runs the specified test.
447 - ``<name>.lint`` - Runs static analysis tools on the Python code. This is a
450 - ``<name>.lint.mypy`` - Runs Mypy on all Python files, if enabled.
451 - ``<name>.lint.pylint`` - Runs Pylint on all Python files, if enabled.
453 - ``<name>.install`` - Installs the package in a Python virtual environment.
454 - ``<name>.wheel`` - Builds a Python wheel for this package.
463 Python package. The build will run it when the test, the package, or one of its
471 use on a per-package basis. The configuration files may also be provided in the
475 Packages may opt out of static analysis as necessary.
482 :start-after: [default-mypy-args]
483 :end-before: [default-mypy-args]
488 distributing Python packages. The Pigweed Python build supports creating wheels
489 for individual packages and groups of packages. Building the ``.wheel``
490 subtarget creates a ``.whl`` file for the package using the PyPA's `build
491 <https://pypa-build.readthedocs.io/en/stable/>`_ tool.
494 :ref:`module-pw_build-pw_python_distribution` records the location of the
499 :ref:`module-pw_build-python-dist`.
503 The Pigweed GN build supports protocol buffers with the ``pw_proto_library``
504 target (see :ref:`module-pw_protobuf_compiler`). Python protobuf modules are
505 generated as standalone Python packages by default. Protocol buffers may also be
506 nested within existing Python packages. In this case, the Python package in the
512 The ``pw_python_package`` target in the ``BUILD.gn`` duplicates much of the
518 Pigweed packages containing protobufs are generated in full or in part. These
519 packages may use generated setup files, since they are always packaged or
520 installed from the build output directory.
527 ----------
531 Build systems automate these auxiliary tasks of software development, making it
532 possible to build larger, more complex systems quickly and robustly.
534 Python is an interpreted language, but it shares most build automation concerns
539 ------------------
540 The Python programming langauge does not have an official build automation
541 system. However, there are numerous Python-focused build automation tools with
549 but are not intended for general build automation. Tools like `PyBuilder
551 provide more general build automation for Python.
553 The `Bazel <http://bazel.build/>`_ build system has first class support for
557 ----------
559 multi-language, modular project. It serves both as a library or middleware and
562 This section describes Python build automation challenges encountered by
569 <https://pypi.org/>`_ packages.
571 The basic Python packaging tools lack dependency tracking for local packages.
573 its dependencies, but ``pip`` is not aware of local packages until they are
574 installed. Packages must be installed with their dependencies taken into
585 by contributing to the long-term resilience of a codebase. Despite their
588 bug-prone codebases.
603 :bdg-ref-primary-line:`docs-automated-analysis` for info on other static
607 are `Pylint <https://www.pylint.org/>`_ and `Mypy <http://mypy-lang.org/>`_.
616 These tools do not have built-in support for incremental runs or dependency
627 `Protocol buffers <https://developers.google.com/protocol-buffers>`_ are an
635 protobufs with existing packages awkward.
638 ------------
640 flexible development experience for its customers. Pigweed's high-level goals
642 Python build.
644 - Integrate seamlessly with the other Pigweed build tools.
645 - Easy to use independently, even if primarily using a different build system.
646 - Support standard packaging and distribution with setuptools, wheel, and pip.
647 - Correctly manage interdependent local Python packages.
648 - Out-of-the-box support for writing and running tests.
649 - Preconfigured, trivial-to-run static analysis integration for Pylint and Mypy.
650 - Fast, dependency-aware incremental rebuilds and test execution, suitable for
651 use with :ref:`module-pw_watch`.
652 - Seamless protocol buffer support.
655 ---------------
657 is more limited in a multi-language project like Pigweed. The cost of bringing
658 up and maintaining an additional build automation system for a single language
661 Pigweed uses GN as its primary build system for all languages. While GN does not
664 GN has strong multi-toolchain and multi-language capabilities. In GN, it is
667 declaration. When using GN for multiple languages, Ninja schedules build steps
668 for all languages together, resulting in faster total build times.
670 Not all Pigweed users build with GN. Of Pigweed's three supported build systems,
675 Given these considerations, GN is an ideal choice for Pigweed's Python build.
678 …ild System Support - How to use it?: https://setuptools.readthedocs.io/en/latest/build_meta.html?h…