Lines Matching +full:build +full:- +full:packages
1 .. _docs-python-build:
4 Pigweed's Python build
6 Pigweed uses a custom GN-based build system to manage its Python code. The
7 Pigweed Python build supports packaging, installation, and distribution of
8 interdependent local Python packages. It also provides for fast, incremental
10 with :ref:`module-pw_watch`) or in continuous integration.
12 Pigweed's Python code is exclusively managed by GN, but the GN-based build may
13 be used alongside CMake, Bazel, or any other build system. Pigweed's environment
15 build system. As needed, non-GN projects can declare just their Python packages
23 Build systems automate these auxiliary tasks of software development, making it
24 possible to build larger, more complex systems quickly and robustly.
26 Python is an interpreted language, but it shares most build automation concerns
32 The Python programming langauge does not have an official build automation
33 system. However, there are numerous Python-focused build automation tools with
41 but are not intended for general build automation. Tools like `PyBuilder
43 provide more general build automation for Python.
45 The `Bazel <http://bazel.build/>`_ build system has first class support for
51 multi-language, modular project. It serves both as a library or middleware and
54 This section describes Python build automation challenges encountered by
58 ------------
61 <https://pypi.org/>`_ packages.
63 The basic Python packaging tools lack dependency tracking for local packages.
65 its dependencies, but ``pip`` is not aware of local packages until they are
66 installed. Packages must be installed with their dependencies taken into
74 -------
77 by contributing to the long-term resilience of a codebase. Despite their
80 bug-prone codebases.
91 ---------------
93 are `Pylint <https://www.pylint.org/>`_ and `Mypy <http://mypy-lang.org/>`_.
102 These tools do not have built-in support for incremental runs or dependency
112 ----------------
113 `Protocol buffers <https://developers.google.com/protocol-buffers>`_ are an
121 protobufs with existing packages awkward.
126 flexible development experience for its customers. Pigweed's high-level goals
128 Python build.
130 - Integrate seamlessly with the other Pigweed build tools.
131 - Easy to use independently, even if primarily using a different build system.
132 - Support standard packaging and distribution with setuptools, wheel, and pip.
133 - Correctly manage interdependent local Python packages.
134 - Out-of-the-box support for writing and running tests.
135 - Preconfigured, trivial-to-run static analysis integration for Pylint and Mypy.
136 - Fast, dependency-aware incremental rebuilds and test execution, suitable for
137 use with :ref:`module-pw_watch`.
138 - Seamless protocol buffer support.
143 Build automation tool
144 ---------------------
146 is more limited in a multi-language project like Pigweed. The cost of bringing
147 up and maintaining an additional build automation system for a single language
150 Pigweed uses GN as its primary build system for all languages. While GN does not
153 GN has strong multi-toolchain and multi-language capabilities. In GN, it is
156 declaration. When using GN for multiple languages, Ninja schedules build steps
157 for all languages together, resulting in faster total build times.
159 Not all Pigweed users build with GN. Of Pigweed's three supported build systems,
164 Given these considerations, GN is an ideal choice for Pigweed's Python build.
166 .. _docs-python-build-structure:
169 ----------------
170 Pigweed Python code is structured into standard Python packages. This makes it
171 simple to package and distribute Pigweed Python packages with common Python
174 Like all Pigweed source code, Python packages are organized into Pigweed
176 :ref:`docs-module-structure`).
178 .. code-block::
180 :name: python-file-tree
184 │ ├── BUILD.gn
199 The ``BUILD.gn`` declares this package in GN. For upstream Pigweed, a presubmit
200 check in ensures that all Python files are listed in a ``BUILD.gn``.
202 Pigweed prefers to define Python packages using ``setup.cfg`` files. In the
206 .. code-block::
208 :name: setup-py-stub
213 .. code-block::
215 :name: pyproject-toml-stub
217 [build-system]
219 build-backend = 'setuptools.build_meta'
221 The stub ``setup.py`` file is there to support running ``pip install --editable``.
223 Each ``pyproject.toml`` file is required to specify which build system should be
229 - ``setup.cfg`` examples at `Configuring setup() using setup.cfg files`_
230 - ``pyproject.toml`` background at `Build System Support - How to use it?`_
233 .. _module-pw_build-python-target:
236 -------------------------
237 The key abstraction in the Python build is the ``pw_python_package``.
240 in :ref:`module-pw_build-python`.
244 - a ``setup.py`` file,
245 - source files,
246 - test files,
247 - dependencies on other ``pw_python_package`` targets.
250 subtarget represents different functionality in the Python build.
252 - ``<name>`` - Represents the Python files in the build, but does not take any
254 - ``<name>.tests`` - Runs all tests for this package.
256 - ``<name>.tests.<test_file>`` - Runs the specified test.
258 - ``<name>.lint`` - Runs static analysis tools on the Python code. This is a
261 - ``<name>.lint.mypy`` - Runs Mypy on all Python files, if enabled.
262 - ``<name>.lint.pylint`` - Runs Pylint on all Python files, if enabled.
264 - ``<name>.install`` - Installs the package in a Python virtual environment.
265 - ``<name>.wheel`` - Builds a Python wheel for this package.
274 Python package. The build will run the it when the test, the package, or one
282 use on a per-package basis. The configuration files may also be provided in the
286 Packages may opt out of static analysis as necessary.
288 Installing packages in a virtual environment
290 Python packages declared in the Python build may be installed in a specified
292 default venv to use may be specified using a GN build arg. The venv may be
293 overridden for individual targets. The Python build tracks installation status
294 of packages based on which venv is in use.
296 The Python build examines the ``VIRTUAL_ENV`` environment variable to determine
297 the current venv. If the selected virtual environment is active, packages are
299 installing the packages.
307 Python packages defined entirely in tree are installed with the ``--editable``
308 option. Partially or fully generated packages are installed without that option.
312 Wheels are the standard format for distributing Python packages. The Pigweed
313 Python build supports creating wheels for individual packages and groups of
314 packages. Building the ``.wheel`` subtarget creates a ``.whl`` file for the
315 package using the PyPA's `build <https://pypa-build.readthedocs.io/en/stable/>`_
323 :ref:`module-pw_build-python-dist`.
327 The Pigweed GN build supports protocol buffers with the ``pw_proto_library``
328 target (see :ref:`module-pw_protobuf_compiler`). Python protobuf modules are
329 generated as standalone Python packages by default. Protocol buffers may also be
330 nested within existing Python packages. In this case, the Python package in the
335 -------------------
336 The ``pw_python_package`` target in the ``BUILD.gn`` duplicates much of the
342 Pigweed packages containing protobufs are generated in full or in part. These
343 packages may use generated setup files, since they are always be packaged or
344 installed from the build output directory.
349 - :ref:`module-pw_build-python`
350 - :ref:`module-pw_build`
351 - :ref:`docs-build-system`
354 …ild System Support - How to use it?: https://setuptools.readthedocs.io/en/latest/build_meta.html?h…