Lines Matching +full:pip +full:- +full:requirements +full:- +full:file
1 .. _docs-python-build:
8 - :bdg-ref-primary-line:`module-pw_build-python` for detailed template usage.
9 - :bdg-ref-primary-line:`module-pw_build` for other GN templates available
11 - :bdg-ref-primary-line:`docs-build-system` for a high level guide and
14 Pigweed uses a custom GN-based build system to manage its Python code. The
18 with :ref:`module-pw_watch`) or in continuous integration.
20 Pigweed's Python code is exclusively managed by GN, but the GN-based build may
23 build system. As needed, non-GN projects can declare just their Python packages
31 :ref:`module-pw_build-pw_python_action` GN template which will ultimately run
40 :ref:`pw_python_actions <module-pw_build-pw_python_action>` throughout the build
41 graph. Once created, all required third-party Python packages needed for the
43 the venv. Of course if a new third-party package dependency is added it will be
45 with the :ref:`module-pw_build-pw_python_venv` template if desired, but only one
56 out[GN Build Dir<br/>fa:fa-folder out]
58 out -->|ninja -C out| createvenvs
61 createvenvs --> pyactions1
62 createvenvs --> pyactions2
66 venv1(fa:fa-folder out/python-venv  )
69 venv1 --> a1
70 venv1 --> a2
75 venv2(fa:fa-folder out/another-venv  )
78 venv2 --> a3
79 venv2 --> a4
87 :bdg-ref-primary-line:`docs-python-build-python-gn-venv` on how to define
90 Having a static venv containing only third-party dependencies opens the flood
91 gates for python scripts to run. If the venv only contains third-party
92 dependencies you may be wondering how you can import your own in-tree Python
93 packages. Python code run in the build may still import any in-tree Python
94 packages created with :ref:`module-pw_build-pw_python_package`
97 :ref:`module-pw_build-pw_python_action`
99 <https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH>`_ so that given
106 - Using venvs to execute Python in GN provides reproducible builds with fixed
107 third-party dependencies.
108 - Using ``PYTHONPATH`` coupled with ``python_deps`` to import in-tree Python
112 Managing Python Requirements
115 .. _docs-python-build-python-gn-venv:
118 ----------------------------
123 :ref:`module-pw_build-pw_python_action` targets are executed.
129 ``.gn`` or ``BUILD.gn`` file. For example:
131 .. code-block::
136 Additional :ref:`module-pw_build-pw_python_venv` targets can be created as
137 needed. The :ref:`module-pw_build-pw_python_action` template can take an
142 .. _docs-python-build-python-gn-requirements-files:
144 Third-party Python Requirements and Constraints
145 -----------------------------------------------
150 **Adding Requirements Files**
152 1. Add a ``install_requires`` entry to a ``setup.cfg`` file defined in a
153 :ref:`module-pw_build-pw_python_package` template. This is the best option
154 if your in-tree Python package requires an external Python package.
156 2. Create a standard Python ``requirements.txt`` file in your project and add it
159 Requirements files support a wide range of install locations including
160 packages from pypi.org, the local file system and git repos. See `pip's
161 Requirements File documentation
162 <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_ for more
165 The GN arg can be set in your project's root ``.gn`` or ``BUILD.gn`` file.
167 .. code-block::
170 # Project specific requirements
171 "//tools/requirements.txt",
174 See the :ref:`docs-python-build-python-gn-structure` section below for a full
181 <https://pip.pypa.io/en/stable/user_guide/#constraints-files>`_. Constraints
182 control which versions of packages get installed by ``pip`` if that package is
184 from the ``pw_env_setup`` module from in your top level ``.gn`` file. Additonal
187 .. code-block::
194 In-tree ``pw_python_package`` Requirements
195 ------------------------------------------
196 A given venv inherits a project's requirements and constraint files by default
202 To ensure the requirements of in-tree :ref:`module-pw_build-pw_python_package`
203 targets are installed :ref:`module-pw_build-pw_python_venv` introduces the
204 ``source_packages`` argument. This is a list of in-tree ``pw_python_package``
206 ``pw_python_package``'s ``setup.cfg`` file is read to pull the
208 all in-tree packages and any in-tree transitive dependencies is then written to
216 adds ``-c`` lines for constraint files.
219 The pip documentation on the `Requirements File Format
220 <https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format>`_
223 :start-after: [downstream-project-venv]
224 :end-before: [downstream-project-venv]
226 .. code-block::
227 …:caption: :octicon:`file;1em` out/python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/gene…
230 # Auto-generated requirements.txt from the following packages:
244 -c ../../../../../../../pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list
254 pip-tools>=6.12.3
255 prompt-toolkit>=3.0.26
266 types-pygments
267 types-pyserial>=3.5,<4.0
268 types-pyyaml
269 types-setuptools
270 types-six
277 The above ``generated_requirements.txt`` file is then fed into the
278 ``pip-compile`` command from `the pip-tools package
279 <https://pypi.org/project/pip-tools>`_ to fully expand and pin each package with
281 single Python requirements file for replicating this ``pw_python_venv``
282 elsewhere. Each ``pw_python_venv`` will get this single file containing the
287 by the :ref:`module-pw_build-pw_python_zip_with_setup` template when
288 producing a self contained zip of in-tree and third party Python packages.
291 :ref:`module-pw_build-pw_python_venv` target:
294 .. code-block::
295 …:caption: :octicon:`file;1em` out/python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/comp…
299 # This file is autogenerated by pip-compile with Python 3.11
302 # pip-compile --allow-unsafe --generate-hashes
303 …# --output-file=python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/compiled_requirem…
304 # --resolver=backtracking
308 --hash=sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41 \
309 --hash=sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128
311 …# -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_set…
314 --hash=sha256:0e0e3709d64fbffd3037e4ff403580550f14471fd3eaae9fa11cc9a5c7901153 \
315 --hash=sha256:a3cf9f02c53dd259144a7e8f3ccd75d67c9a8c716ef183e0c1f291bc5d7bb3cf
317 …# -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_set…
322 ``pip_generate_hashes`` arg to the :ref:`module-pw_build-pw_python_venv`
326 ------------------------------------------------
328 .. _docs-python-build-downloading-packages:
332 The :ref:`module-pw_build-pw_python_venv` target adds an optional sub target
345 .. code-block:: bash
348 ninja -C out \
349 $(gn ls out --as=output \
353 ``pip_download_log.txt`` with verbose logs from running ``pip download``.
355 .. code-block::
356 :caption: :octicon:`file-directory;1em` Vendor wheels output directory
357 :name: vendor-wheel-output
362 ├── appdirs-1.4.4-py2.py3-none-any.whl
363 ├── astroid-2.14.2-py3-none-any.whl
364 ├── backcall-0.2.0-py2.py3-none-any.whl
365 ├── black-23.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
367 …├── websockets-10.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manyl…
368 ├── wheel-0.40.0-py3-none-any.whl
369 ├── wrapt-1.14.1.tar.gz
370 └── yapf-0.31.0-py2.py3-none-any.whl
375 ``cp311-cp311-manylinux_2_17_x86_64.whl``. These binary packages are selected by
376 the ``pip download`` command based on the host machine python version, OS, and
386 - cp311, manylinux_2_17_x86_64
387 - cp311, manylinux2014_x86_64
388 - cp311, macosx_11_0_arm64
389 - cp311, macosx_10_9_x86_64
390 - cp311, win_amd64
391 - cp311, win32
402 .. code-block::
406 This will invoke `pip download
407 <https://pip.pypa.io/en/stable/cli/pip_download/>`_ for each combination of
412 :start-after: [wheel-platform-args]
413 :end-before: [wheel-platform-args]
417 ``compiled_requirements.txt`` file. This file can only be generated for the
418 current host OS and Python version. `pip-tools
419 <https://pypi.org/project/pip-tools>`_ does not expand requirements for
421 …<https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#platform-specific-depen…
422 example ipython defines these two requirements:
424 .. code-block::
429 If pip-tools is run on Linux then the above packages will not appear in
433 .. _docs-python-build-installing-offline:
438 this as the default pip install location in two different ways.
442 Setting these args in the ``//.gn`` file will add the relevant pip command line
445 .. code-block::
447 # Adds --no-index forcing pip to not reach out to the internet (pypi.org) to
453 # files (*.tar.gz). Pip will check each of these directories when looking for
460 # Optional: Adds '--no-cache-dir' forcing pip to ignore any previously cached
461 # Python packages. On most systems this is located in ~/.cache/pip/
464 Using a ``.pip.conf`` File
466 1. Create a ``//pip.conf`` file containing:
468 .. code-block::
469 :caption: :octicon:`file;1em` //pip.conf
470 :name: pip-conf-file
474 no-index = True
476 find-links =
477 file://third_party/python_packages/universal
478 file://third_party/python_packages/linux/cp311
480 This tells pip to not search pypi.org for packages and only look in
483 are relative to the ``pip.conf`` file.
486 this file.
488 .. code-block:: bash
490 export PIP_CONFIG_FILE="${PW_PROJECT_ROOT}/pip.conf"
492 With that environment var set all invocations of pip will apply the config file
496 The ``pip`` `documentation on Configuration
497 <https://pip.pypa.io/en/stable/topics/configuration/>`_.
499 .. _docs-python-build-python-gn-structure:
501 GN File Structure for Python Code
504 Pigweed's GN build system. A brief file hierarchy is shown here with file
505 content following. See also :ref:`docs-python-build-structure` below for details
508 .. code-block::
509 :caption: :octicon:`file-directory;1em` Top level GN file hierarchy
510 :name: gn-python-file-tree
540 - :octicon:`file-directory;1em` project_root/
542 - :octicon:`file;1em` .gn
544 .. code-block::
554 # Project specific constraints file
559 # Project specific requirements
560 "//tools/requirements.txt",
569 There are some additional gn args to control how pip installations are
573 :start-after: [default-pip-gn-args]
574 :end-before: [default-pip-gn-args]
576 - :octicon:`file;1em` BUILDCONFIG.gn
578 .. code-block::
586 - :octicon:`file-directory;1em` build_overrides / :octicon:`file;1em` pigweed.gni
588 .. code-block::
598 - :octicon:`file;1em` BUILD.gn
600 .. code-block::
609 # Lists all the targets build by default with e.g. `ninja -C out`.
621 # Generate and pip install _all_python_packages
626 # In-tree Python packages
642 # Set this gn arg in a declare_args block in this file 'BUILD.gn' or in '.gn' to
648 path = "$root_build_dir/python-venv"
650 requirements = pw_build_PIP_REQUIREMENTS
664 name = "project-tools"
671 # Install the project-tools super Python package into the bootstrapped
677 .. _docs-python-build-structure:
687 :ref:`Pigweed Module Stucture <docs-module-structure>`).
689 .. code-block::
690 :caption: :octicon:`file-directory;1em` Example layout of a Pigweed Python package.
691 :name: python-file-tree
713 above file tree the ``pyproject.toml`` file is only a stub with the following
716 .. code-block::
717 :caption: :octicon:`file;1em` pyproject.toml
718 :name: pyproject-toml-stub
720 [build-system]
722 build-backend = 'setuptools.build_meta'
724 Each ``pyproject.toml`` file is required to specify which build system should be
730 - ``setup.cfg`` examples at `Configuring setup() using setup.cfg files`_
731 - ``pyproject.toml`` background at `Build System Support - How to use it?`_
733 .. _module-pw_build-python-target:
736 -------------------------
740 in :ref:`module-pw_build-python`.
744 - a ``setup.cfg`` and ``pyproject.toml`` file,
745 - source files,
746 - test files,
747 - dependencies on other ``pw_python_package`` targets.
752 - ``<name>`` - Represents the Python files in the build, but does not take any
754 - ``<name>.tests`` - Runs all tests for this package.
756 - ``<name>.tests.<test_file>`` - Runs the specified test.
758 - ``<name>.lint`` - Runs static analysis tools on the Python code. This is a
761 - ``<name>.lint.mypy`` - Runs Mypy on all Python files, if enabled.
762 - ``<name>.lint.pylint`` - Runs Pylint on all Python files, if enabled.
763 - ``<name>.lint.ruff`` - Runs ruff on all Python files, if enabled.
765 - ``<name>.install`` - Installs the package in a Python virtual environment.
766 - ``<name>.wheel`` - Builds a Python wheel for this package.
774 Adding a new test is simple: write the test file and list it in its accompanying
782 and ``ruff_toml`` files to use on a per-package basis. The configuration files
795 :start-after: [python-static-analysis-tools]
796 :end-before: [python-static-analysis-tools]
803 :start-after: [default-mypy-args]
804 :end-before: [default-mypy-args]
811 subtarget creates a ``.whl`` file for the package using the PyPA's `build
812 <https://pypa-build.readthedocs.io/en/stable/>`_ tool.
815 :ref:`module-pw_build-pw_python_distribution` records the location of the
820 :ref:`module-pw_build-python-dist`.
825 target (see :ref:`module-pw_protobuf_compiler`). Python protobuf modules are
834 information in the ``setup.cfg`` file. In many cases, it would be possible to
835 generate a ``setup.cfg`` file rather than including it in the source
848 ----------
860 ------------------
862 system. However, there are numerous Python-focused build automation tools with
868 <https://pypi.org/project/wheel/>`_, and `pip <https://pypi.org/project/pip/>`_.
878 ----------
880 multi-language, modular project. It serves both as a library or middleware and
894 ``pip`` is not aware of local packages until they are installed. Packages must
906 by contributing to the long-term resilience of a codebase. Despite their
909 bug-prone codebases.
924 :bdg-ref-primary-line:`docs-automated-analysis` for info on other static
928 are `Pylint <https://www.pylint.org/>`_ and `Mypy <http://mypy-lang.org/>`_.
937 These tools do not have built-in support for incremental runs or dependency
942 support using a single configuration file for an entire run, which poses a
948 `Protocol buffers <https://developers.google.com/protocol-buffers>`_ are an
958 Requirements section in Rationale
959 ------------
961 flexible development experience for its customers. Pigweed's high-level goals
962 and the `challenges`_ described above inform these requirements for the Pigweed
965 - Integrate seamlessly with the other Pigweed build tools.
966 - Easy to use independently, even if primarily using a different build system.
967 - Support standard packaging and distribution with setuptools, wheel, and pip.
968 - Correctly manage interdependent local Python packages.
969 - Out-of-the-box support for writing and running tests.
970 - Preconfigured, trivial-to-run static analysis integration for Pylint and Mypy.
971 - Fast, dependency-aware incremental rebuilds and test execution, suitable for
972 use with :ref:`module-pw_watch`.
973 - Seamless protocol buffer support.
976 ---------------
978 is more limited in a multi-language project like Pigweed. The cost of bringing
985 GN has strong multi-toolchain and multi-language capabilities. In GN, it is
999 …ild System Support - How to use it?: https://setuptools.readthedocs.io/en/latest/build_meta.html?h…