1.. _docs-blog-06-better-cpp-toolchains: 2 3=================================================================== 4Pigweed Blog #6: Shaping a better future for Bazel C/C++ toolchains 5=================================================================== 6*Published on 2024-12-11 by Armando Montanez* 7 8---------------- 9Pigweed �� Bazel 10---------------- 11Not too long ago, the Pigweed team announced our transition to a Bazel-first 12ecosystem. You can read more about the decision in :ref:`seed-0111`, but the key 13takeaway is that we're committed to making sure Bazel has a good story for 14embedded software development. There's a surprising amount of history behind 15this, and I thought it'd be helpful to unpack some of our journey. 16 17This blog post is the first in a series on getting Bazel to work well for 18embedded firmware projects, so keep an eye out for others that will come along. 19 20---------------------------------------- 21First things first, we need a toolchain! 22---------------------------------------- 23Pigweed traditionally prioritized our `GN <https://gn.googlesource.com/gn/>`__ 24build, and getting Bazel to work for embedded devices presented very different 25challenges. One of the first major differences between Bazel and GN is the 26experience of compiling a simple ``Hello, World!`` example. In Bazel, this "just 27works" out of the box for most major operating systems. With GN, you can't start 28to compile C/C++ code without first defining the details of your toolchain such 29as which binaries to use, how to construct the arguments/flags passed to the 30tools, and which specific flags to pass, so we were forced to learn those 31intricacies right out the gate. 32 33Problems with Bazel's default toolchain 34======================================= 35.. _hermetic: https://bazel.build/basics/hermeticity 36 37For a long time, Pigweed relied on Bazel's default toolchain, primarily because 38Pigweed's Bazel build mostly just existed for interoperability with Google’s 39internal codebase. Bazel’s default C/C++ toolchain is not `hermetic`_, and its 40behaviors are auto-generated in a way that is very unintuitive to inspect and 41debug. Because we weren't defining our own toolchain, we regularly hit several 42major issues: 43 44* Builds used binutils/gcc rather than our preferred llvm/clang toolchain. 45* CI builds and local builds were inconsistent; just because builds passed 46 locally didn't mean the same build would pass in automated builds. 47* All flags had to be passed in via the command line or pushed into a 48 ``.bazelrc`` file. 49* There was no clear path for scalably supporting embedded MCU device builds. 50 51We knew at some point we'd have to configure our own toolchain, but we also knew 52it wouldn’t be easy. 53 54------------------------- 55Why is it this difficult? 56------------------------- 57Bazel toolchains are quite powerful, and with great power comes great 58complexity. Traditionally, a C/C++ toolchain in Bazel is expected to be declared 59in `Starlark <https://bazel.build/rules/language>`__ rather than ``BUILD`` 60files, which makes them a little harder to find and read. Also, there’s 61documentation fragmentation around Bazel’s toolchain API that can make it 62difficult to understand how all the moving pieces of toolchains interact. For 63years there was a distinct lack of good examples to learn from, too. Setting up 64a simple, custom toolchain isn’t necessarily a lot of typing, but it’s 65surprisingly difficult because of the sheer amount of Bazel implementation 66details and behaviors you must first understand. 67 68We looked into declaring a custom C++ toolchain in the early days of Pigweed’s 69Bazel build, but quickly realized it would be quite a chore. What we didn’t 70realize was that this would turn out to be a problem that would take much design 71and discussion over the course of multiple years. 72 73------------------ 74The journey begins 75------------------ 76The early origins of this story are really thanks to Nathaniel Brough, who was 77one of Pigweed’s first community contributors and an original pioneer of 78Pigweed’s Bazel build. Nat put together 79`a Bazel toolchain definition <https://github.com/bazelembedded/rules_cc_toolchain>`__ 80that Pigweed adopted and used until late 2023. This allowed us to build Pigweed 81using a consistent version of clang, but wasn’t quite perfect. The initial 82toolchain Nat put together got us off the ground, but came with a few 83limitations: 84 85* Configurability was limited. Pigweed didn’t have quite as much control over 86 flags as we wanted. 87* The declared toolchain configuration was quite rigid, so it was difficult to 88 adjust or adapt to different sysroots or compilers. 89* The most significant limitation, though, was that it wasn't configurable 90 enough to scalably support targeting many different embedded MCUs in a flexible way. 91 92Nat went back to the drawing board, and started drafting out 93`modular_cc_toolchains <https://github.com/bazelembedded/modular_cc_toolchains>`__. 94This proposal promised much more modular building blocks that would provide more 95direct access to the underlying constructs exposed by Bazel for defining a 96modular toolchain. 97 98--------------------------------------------- 99Pigweed’s first attempt at modular toolchains 100--------------------------------------------- 101In mid-2023 I was passed the ball for conclusively solving Pigweed’s Bazel C/C++ 102toolchain problems. I had previously spent a lot of time in Pigweed’s GN build 103maintaining toolchain integration, so I was relatively familiar with what went 104well and what didn’t. I dove into Bazel with a naive vision: make declaring a 105toolchain as simple as listing a handful of tools and an assortment of ordered, 106groupable flags: 107 108.. code-block:: py 109 110 pw_cc_toolchain( 111 name = "host_toolchain", 112 cc = "@llvm_clang//:bin/clang", 113 cxx = "@llvm_clang//:bin/clang++", 114 ld = "@llvm_clang//:bin/clang++", 115 flags = [ 116 ":cpp_version", 117 ":size_optimized", 118 ] 119 ) 120 121 pw_cc_flags( 122 name = "cpp_version", 123 copts = ["-std=c++17"], 124 ) 125 126 pw_cc_flags( 127 name = "size_optimized", 128 copts = ["-Os"], 129 linkopts = ["-Os"], 130 ) 131 132This approach brought a few major improvements: 133 134* Pigweed (and downstream projects) could now declare toolchains quite easily. 135* We were able to make Bazel use the clang/llvm toolchain binaries that we host 136 in `CIPD <https://chromium.googlesource.com/chromium/src/+/67.0.3396.27/docs/cipd.md>`__. 137 138The big drawback with this approach was that it obscured a lot of Bazel’s 139toolchain complexity in a way that limited Pigweed’s ability to cleanly and 140modularly introduce fixes. 141 142------------------ 143Making it official 144------------------ 145With the learnings from my first attempt, I went back over Nat’s work for 146`modular_cc_toolchains <https://github.com/bazelembedded/modular_cc_toolchains>`__, 147and set out authoring :ref:`seed-0113`. There were some discussions on this API, 148and the upstream Bazel owners of 149`Bazel's C/C++ rules (rules_cc) <https://github.com/bazelbuild/rules_cc>`__ 150expressed interest in the work too. Eventually, the SEED was approved, and 151landed largely as described. This toolchain API checked the critical boxes for 152Pigweed: we could now declare toolchains in a scalable, modular way! 153 154The only major remaining wart was handling of 155`toolchain features <https://bazel.build/docs/cc-toolchain-config-reference#features>`__ 156and `action names <https://bazel.build/docs/cc-toolchain-config-reference#actions>`__\: 157Pigweed didn’t try to innovate in this area as a first pass. The primary 158reasoning behind this was we quickly learned that part of our advice would be to 159recommend against a large proliferation of toolchain features. Still there was 160room for improvement, especially since specifying action names on a flag set was 161a little unwieldy: 162 163.. code-block:: py 164 165 load( 166 "@pw_toolchain//cc_toolchain:defs.bzl", 167 "pw_cc_flag_set", 168 "ALL_CPP_COMPILER_ACTIONS", 169 "ALL_C_COMPILER_ACTIONS", 170 ) 171 172 pw_cc_flag_set( 173 name = "werror", 174 # These symbols have to be `load()`ed since they're string lists. 175 actions = ALL_CPP_COMPILER_ACTIONS + ALL_C_COMPILER_ACTIONS, 176 flags = [ 177 "-Werror", 178 ], 179 ) 180 181.. inclusive-language: disable 182 183This work caught the eye of Matt Stark, who was interested in building out 184toolchains for ChromeOS. He noticed these shortcomings, and put in a lot of work 185to make these types type-safe by changing them to also be provided through build 186rules: 187 188.. inclusive-language: enable 189 190.. code-block:: py 191 192 load("@pw_toolchain//cc_toolchain:defs.bzl", "pw_cc_flag_set") 193 194 pw_cc_flag_set( 195 name = "werror", 196 # Much nicer! 197 actions = [ 198 "@pw_toolchain//actions:all_c_compiler_actions", 199 "@pw_toolchain//actions:all_cpp_compiler_actions", 200 ], 201 flags = [ 202 "-Werror", 203 ], 204 ) 205 206.. todo-check: disable 207 208There were a few other changes along the way to support these kinds of 209expressions, but by and large the toolchain API has served Pigweed quite well; 210it allowed us to finally close out many bugs strewn about the codebase that said 211things along the lines of “TODO: someday this should live in a toolchain”. We 212even used it when setting up initial Bazel support for the 213`Raspberry Pi Pico SDK <https://github.com/raspberrypi/pico-sdk/>`__, and 214the only changes required were 215`a few tweaks to the toolchain template build files <http://pwrev.dev/194591>`__ 216to get it working on Windows for the first time. 217 218.. todo-check: enable 219 220.. inclusive-language: disable 221 222------------------------ 223Making it SUPER official 224------------------------ 225As Matt finished out his improvements to Pigweed’s toolchain rules, he posed the 226question we’d previously considered: could this work just live in 227`rules_cc <https://github.com/bazelbuild/rules_cc>`__, the source of truth for 228Bazel’s C/C++ rules? We were optimistic, and reached out again to the owners. 229The owners of rules_cc enthusiastically gave us the green light, and Matt took 230Pigweed’s Bazel C/C++ toolchain constructs and began the process of upstreaming 231the work to rules_cc. There have been some changes along the way (particularly 232with naming), but they’ve been part of an effort to be more forward-looking 233about guiding the future of the underlying constructs. 234 235.. inclusive-language: enable 236 237.. _module-pw_toolchain_bazel: 238 239.. _module-pw_toolchain_bazel-get-started: 240 241----------- 242Try it out! 243----------- 244These rules were initially launched in 245`rules_cc v0.0.10 <https://github.com/bazelbuild/rules_cc/releases/tag/0.0.10>`__, 246and have since received a slew of updates, improvements, and most importantly 247documentation/examples. Today, we’re ready to more broadly encourage projects to 248try out the new work. We hope that these rules will become the preferred 249foundations for declaring C/C++ toolchains in Bazel. We’re excited to see how 250the wider Bazel community expands on these foundational building blocks! 251 252If you’d like to give these rules a spin, check out the following resources and examples: 253 254* `Creating C++ Toolchains Easily - Bazelcon Presentation by Matt Stark & Armando Montanez <https://www.youtube.com/watch?v=PVFU5kFyr8Y>`__ 255* `rules_cc toolchain API <https://github.com/bazelbuild/rules_cc/blob/main/docs/toolchain_api.md>`__ 256* `rules_cc living rule-based toolchain example <https://github.com/bazelbuild/rules_cc/tree/main/examples/rule_based_toolchain>`__ 257* `Raspberry Pi Pico SDK Bazel toolchain <https://github.com/raspberrypi/pico-sdk/blob/6587f5cc9a91ca7fef7ccf56420d465b88d8d398/bazel/toolchain/BUILD.bazel>`__ 258* `Pigweed’s host clang toolchain <https://cs.opensource.google/pigweed/pigweed/+/main:pw_toolchain/host_clang/BUILD.bazel>`__ 259 260-------------- 261Special thanks 262-------------- 263This work would not have been possible without Nat and Matt’s contributions. Nat 264spent a lot of time collaborating with Pigweed to really kickstart the Bazel 265effort, and Matt’s enthusiasm for finishing out Pigweed’s fledgling toolchain 266API and getting it pushed into rules_cc quickly has been inspiring! A lot of 267work went into solving this problem, and community contributions were a critical 268part of the journey. Also, a very special thanks to Ivo List, who reviewed many 269CLs as part of moving the toolchain rules into rules_cc. 270 271This has been an amazing journey, and I'm excited for a better future for 272C/C++ toolchains in Bazel! 273