Lines Matching +full:build +full:- +full:docs
1 .. _blog-08-bazel-docgen:
6 *Published on 2025-02-03 by Kayce Basques*
10 * :ref:`blog-08-bazel-docgen-why`
11 * :ref:`blog-08-bazel-docgen-compare`
12 * :ref:`blog-08-bazel-docgen-good`
13 * :ref:`blog-08-bazel-docgen-challenges`
14 * :ref:`blog-08-bazel-docgen-next`
19 post merely discusses how we migrated the build for our own docs site
22 .. _blog-08-bazel-docgen-why:
24 ---------------------------------
26 ---------------------------------
27 .. _hermetic: https://bazel.build/basics/hermeticity
29 Back in October 2023, Pigweed adopted Bazel as its :ref:`primary build system
30 <seed-0111>`. For the first 12 months or so, the team focused on critical
32 cross-platform builds. Most of that work is done now so we shifted our focus
33 to the last major part of Pigweed not using Bazel: our docs.
37 "Hermeticity" essentially means that the build system runs in an isolated,
38 reproducible environment to guarantee that the build always produces the
42 .. _blog-08-bazel-docgen-compare:
44 -------------------------------------------------
45 Comparing the new Bazel build to the old GN build
46 -------------------------------------------------
47 We eventually ended up with this architecture for the new Bazel-based docs
48 build system:
54 Doxygen --> Breathe
55 Breathe --> reST
56 reST --> Sphinx
57 Rust --> Sphinx
58 Python --> Sphinx
60 The GN build had roughly the same architecture, but the architecture is
61 much more explicit and well-defined now.
65 * C/C++ API references. We auto-generate our C/C++ API references with
67 into our main docs files, which are authored in **reST** (reStructuredText).
68 Example: :ref:`pw_unit_test C++ API reference <module-pw_unit_test-cpp>`
70 * Rust API references. We auto-generate our Rust API references with
71 ``rustdoc`` and upload these docs to their own subsite.
74 * reST and Sphinx. We author most of our docs in **reST** (reStructuredText)
76 docs system. Example: :ref:`Tour of Pigweed <showcase-sense-tutorial-intro>`
78 * Python API references. We auto-generate our Python API references with
79 Sphinx's ``autodoc`` feature. Example: :ref:`pw_rpc Python client API <module-pw_rpc-py>`
88 migration was my first time working with Bazel in-depth and was the largest
94 .. _blog-08-bazel-docgen-compare-doxygen:
98 In the GN build we needed a custom script to run Doxygen.
101 then ran Doxygen non-hermetically. I.e. Doxygen had access to
105 In the Bazel build all of our Doxygen logic resides within the ``MODULE.bazel``
106 file at the root of our repo and the ``BUILD.bazel`` files distributed
113 modules <https://bazel.build/external/module>`_ (the future of external
121 .. _blog-08-bazel-docgen-compare-rust:
125 In the GN build there is no equivalent to this step. We have always generated
128 Bazel. Previously our docs builder would generate the Rust API references with
129 Bazel, then use GN to build the rest of the docs, then upload the two
130 disconnected outputs to production. Now, the docs builder just runs a single
131 Bazel command and everything is generated together. Long-term, this will
132 probably make it easier to integrate the Rust docs more thoroughly with the
135 .. _blog-08-bazel-docgen-compare-sphinx:
137 Building the reStructuredText docs
139 .. inclusive-language: disable
140 .. _Sphinx: https://www.sphinx-doc.org/en/master/
141 .. inclusive-language: enable
143 This is the heart of our docs system. We author our docs in `reStructuredText
148 In the GN build, we basically had to implement all core docs workflows
150 the docs with Sphinx, another for locally previewing the docs, etc.
155 In the Bazel build, we no longer need any of this custom code.
156 `rules_python <https://rules-python.readthedocs.io/en/latest/>`_
157 provides almost all of our core docs workflows now.
158 See :ref:`blog-08-bazel-docgen-good-sources` and
159 :ref:`blog-08-bazel-docgen-good-rules_python` for more details.
161 .. _blog-08-bazel-docgen-compare-verify:
167 manually verify that the Bazel build was producing the same outputs as the GN
168 build. I ended up automating the verification workflow like this:
170 #. Build the docs with the old GN-based system.
172 #. Build them again with the new Bazel-based system.
186 each GN-generated HTML file and its Bazel-built equivalent.
188 #. Visually diff the screenshots with `pixelmatch-py
189 <https://github.com/whtsky/pixelmatch-py>`_.
191 .. _blog-08-bazel-docgen-good:
193 --------------
195 --------------
196 We kicked off the migration project in mid-September 2024 and started using
197 Bazel in production in mid-January 2025. If we were in a rush, we probably
199 help I got from others, it was about 120 hours of work. I.e. one full-time
203 .. _blog-08-bazel-docgen-good-sources:
205 Built-in support for reorganizing sources
207 Our docs are stored alongside the rest of Pigweed's code in a single
208 repository. To make it easier to keep the docs in-sync with code changes, each
211 .. code-block:: text
220 └── docs
227 .. code-block:: text
234 └── docs
243 be the root docs directory. All ``*.rst`` (reST) files should be at or
244 below the root docs directory.
246 In the old GN-based system we had to hack together this reorganization
247 logic ourselves. Bazel has built-in support for source reorganization via
250 .. _blog-08-bazel-docgen-good-rules_python:
254 We now get almost all of :ref:`our core docs workflows <contrib-docs-build>`
258 because our docs system now needs much less custom code.
260 .. _blog-08-bazel-docgen-good-speed:
264 Currently, building the docs from scratch in Bazel is about 27% faster than
265 building them from scratch in GN. However, there's still one major docs feature
266 being migrated over to Bazel so it's not an apples-to-apples comparison yet.
268 .. _blog-08-bazel-docgen-challenges:
270 ----------
272 ----------
275 .. _blog-08-bazel-docgen-challenges-incremental:
280 file, and it takes 30-60 seconds to regenerate the docs. Unacceptable! Bazel
284 .. _blog-08-bazel-docgen-challenges-skylib:
289 searched the Bazel docs, but couldn't find a built-in mechanism for this basic
291 <https://bazel.build/reference/be/general#genrule>`_. During code review, I
293 <https://github.com/bazelbuild/bazel-skylib/blob/main/rules/copy_directory.bzl>`_.
295 Bazel docs.
297 .. _blog-08-bazel-docgen-challenges-deps:
302 Pigweed, all of Pigweed is built and tested in 10-100 different environments
309 The :ref:`rules_python <blog-08-bazel-docgen-good-rules_python>` features that
318 .. _blog-08-bazel-docgen-challenges-graphs:
320 Explicit build graphs were time consuming
323 sources and dependencies in the docs build rules, like this:
325 .. code-block:: py
328 name = "docs",
333 "docs.rst",
342 .. code-block:: py
345 name = "docs",
352 .. _blog-08-bazel-docgen-challenges-starlark:
356 .. _Starlark: https://github.com/bazelbuild/starlark?tab=readme-ov-file#starlark
363 doesn't allow it. For example, to build out a dict in Python, I sometimes
366 .. code-block:: py
375 .. code-block:: py
381 .. _blog-08-bazel-docgen-next:
383 -----------
385 -----------
387 down the old GN-based build.
390 find me in the ``#docs`` channel of `Pigweed's Discord