1.. _docs-get-started-upstream: 2 3============================================= 4Get started with upstream Pigweed development 5============================================= 6This guide will walk you through the typical upstream development workflow. 7"Upstream development" means that you're contributing to the main Pigweed 8repository, ``https://pigweed.googlesource.com/pigweed/pigweed``. See 9:ref:`docs-get-started` if you're looking for instructions on how to use 10Pigweed in your own project. 11 12.. note:: 13 14 This guide needs updating for developing upstream with Bazel. The workflows 15 are a bit simpler since bootstrap isn't needed. See 16 https://pwbug.dev/398929253 for updates. 17 18.. _prerequisites: 19 20Prerequisites 21============= 22If you haven't already, :ref:`docs-first-time-setup`. 23 24------------- 25Express setup 26------------- 27If you'd like to skip the detailed explanations, below is the shorter version 28of getting setup for Pigweed. If you run into trouble, ensure you've followed 29all the steps to :ref:`docs-first-time-setup`. The express setup 30configures Pigweed's watcher for three targets to give a taste of Pigweed: 31 32#. **Host** - Mac, Linux, or Windows. Builds and runs tests 33#. **Device/STM32F429** - Build only; Optionally, the STM32F429I-DISC1 kit to 34 follow along later in the guide to run tests directly on said device(s) 35#. **Docs** - Builds the Pigweed docs 36 37To get setup: 38 39#. Make sure you have Git and Python installed and on your path. 40 41#. Clone Pigweed and bootstrap the environment (compiler setup & more). **Be 42 patient, this step downloads ~1GB of LLVM, GCC, and other tooling**. 43 44 .. code-block:: bash 45 46 $ cd ~ 47 $ git clone https://pigweed.googlesource.com/pigweed/pigweed 48 ... 49 $ cd pigweed 50 $ source ./bootstrap.sh (On Linux & Mac) 51 $ bootstrap.bat (On Windows) 52 ... 53 54 .. tip:: 55 56 If you use the `Fish shell <https://fishshell.com/>`_ run `source 57 ./bootstrap.fish` instead. 58 59#. Configure the GN build. 60 61 .. code-block:: bash 62 63 $ gn gen out 64 Done. Made 1047 targets from 91 files in 114ms 65 66#. Start the watcher. The watcher will invoke Ninja to build all the targets 67 68 .. code-block:: bash 69 70 $ pw watch 71 72 ▒█████▄ █▓ ▄███▒ ▒█ ▒█ ░▓████▒ ░▓████▒ ▒▓████▄ 73 ▒█░ █░ ░█▒ ██▒ ▀█▒ ▒█░ █ ▒█ ▒█ ▀ ▒█ ▀ ▒█ ▀█▌ 74 ▒█▄▄▄█░ ░█▒ █▓░ ▄▄░ ▒█░ █ ▒█ ▒███ ▒███ ░█ █▌ 75 ▒█▀ ░█░ ▓█ █▓ ░█░ █ ▒█ ▒█ ▄ ▒█ ▄ ░█ ▄█▌ 76 ▒█ ░█░ ░▓███▀ ▒█▓▀▓█░ ░▓████▒ ░▓████▒ ▒▓████▀ 77 78 20200707 17:24:06 INF Starting Pigweed build watcher 79 20200707 17:24:06 INF Will build [1/1]: out 80 20200707 17:24:06 INF Attaching filesystem watcher to $HOME/wrk/pigweed/... 81 20200707 17:24:06 INF Triggering initial build... 82 ... 83 84#. **Congratulations, you're ready to go!** Now take Pigweed for a spin by 85 making a test fail. 86 87#. With the watcher running in a separate window, edit 88 ``pw_status/status_test.cc`` to make an expectation fail; for example, add 89 ``EXPECT_EQ(0, 1);`` in a test. 90 91#. Save the file. Observe the watcher rebuild & retest, and fail. Restore the 92 test if you feel like it. 93 94#. Open the generated docs in ``out/docs/gen/docs/html/index.html`` in your 95 browser. 96 97#. Edit ``docs/getting_started.rst`` (this file!) and make any change. Save. 98 See the watcher rebuild the docs. Reload your browser, and see the changes. 99 100See below for equivalent Windows commands, and for more details on what each 101part does. 102 103**Note:** After running bootstrap once, use ``source ./activate.sh`` (or 104``activate.bat`` on Windows) to re-activate the environment without 105re-bootstrapping. 106 107--------- 108Bootstrap 109--------- 110Once you satisfied the prerequisites, you will be able to clone Pigweed and 111run the bootstrap that initializes the Pigweed virtual environment. The 112bootstrap may take several minutes to complete, so please be patient. 113 114**Linux & macOS** 115 116.. code-block:: bash 117 118 $ git clone https://pigweed.googlesource.com/pigweed/pigweed pigweed 119 $ cd pigweed 120 $ source ./bootstrap.sh 121 122**Windows** 123 124.. code-block:: batch 125 126 :: Run git commands from the shell you set up to use with Git during install. 127 > git clone https://pigweed.googlesource.com/pigweed/pigweed %HOMEPATH%\pigweed 128 > cd %HOMEPATH%\pigweed 129 > bootstrap.bat 130 131Below is a real-time demo with roughly what you should expect to see as output: 132 133.. image:: https://storage.googleapis.com/pigweed-media/pw_env_setup_demo.gif 134 :width: 800 135 :alt: build example using pw watch 136 137Congratulations, you are now set up to start using Pigweed! 138 139.. _activate-pigweed-environment: 140 141--------------------------------- 142Activate your Pigweed environment 143--------------------------------- 144After going through the initial setup process, your current terminal will be in 145the Pigweed development environment that provides all the tools you should need 146to develop on Pigweed. If you leave that session, you can activate the 147environment in a new session with the following command: 148 149**Linux & macOS** 150 151.. code-block:: bash 152 153 $ source ./activate.sh 154 155**Windows** 156 157.. code-block:: batch 158 159 > activate.bat 160 161Some major changes may require triggering the bootstrap again, so if you run 162into host tooling changes after a pull it may be worth re-running bootstrap. 163 164---------------------- 165Build Pigweed for host 166---------------------- 167Pigweed's primary build system is GN/Ninja based. There are CMake and Bazel 168builds in-development, but they are incomplete and don't have feature parity 169with the GN build. We strongly recommend you stick to the GN build system. 170 171GN (Generate Ninja) just does what it says on the tin; GN generates 172`Ninja <https://ninja-build.org/>`_ build files. 173 174The default GN configuration generates build files that allow you to build host 175binaries, device binaries, and upstream documentation all in one Ninja 176invocation. 177 178Run GN as seen below: 179 180.. code-block:: bash 181 182 $ gn gen out 183 184.. note:: 185 ``out`` is simply the directory the build files are saved to. Unless 186 this directory is deleted or you desire to do a clean build, there's no need 187 to run GN again; just rebuild using Ninja directly. 188 189.. warning:: 190 Unless your build directory (the ``out`` in ``gn gen out``) is exactly one 191 directory away from the project root directory (the Pigweed repo root in this 192 case), there will be issues finding source files while debugging and while 193 generating coverage reports. This is due an issue in upstream LLVM reordering 194 debug and coverage path mappings (tracked by 195 `b/278898014 <https://issuetracker.google.com/278898014>`_ and 196 `b/278906020 <https://issuetracker.google.com/278906020>`_). **Stick to 197 simple, single directory build directories for now.** 198 199Now that we have build files, it's time to build Pigweed! 200 201Now you *could* manually invoke the host build using ``ninja -C out`` every 202time you make a change, but that's tedious. Instead, let's use ``pw_watch``. 203 204Go ahead and start ``pw_watch``: 205 206.. code-block:: bash 207 208 $ pw watch 209 210When ``pw_watch`` starts up, it will automatically build the directory we 211generated in ``out``. Additionally, ``pw_watch`` watches source code files for 212changes, and triggers a Ninja build whenever it notices a file has been saved. 213You might be surprised how much time it can save you! 214 215With ``pw watch`` running, try modifying 216``pw_status/public/pw_status/status.h`` and watch the build re-trigger when you 217save the file. 218 219See below for a demo of this in action: 220 221.. image:: https://storage.googleapis.com/pigweed-media/pw_watch_build_demo.gif 222 :width: 800 223 :alt: build example using pw watch 224 225------------------ 226Running unit tests 227------------------ 228Fun fact, you've been running the unit tests already! Ninja builds targeting 229the host automatically build and run the unit tests. Unit tests err on the side 230of being quiet in the success case, and only output test results when there's a 231failure. 232 233To see a test failure, modify ``pw_status/status_test.cc`` to fail by changing 234one of the strings in the "KnownString" test. 235 236.. image:: https://storage.googleapis.com/pigweed-media/pw_watch_test_demo.gif 237 :width: 800 238 :alt: example test failure using pw watch 239 240Running tests as part of the build isn't particularly expensive because GN 241caches passing tests. Each time you build, only the tests that are affected 242(whether directly or transitively) by the code changes since the last build 243will be re-built and re-run. 244 245Try running the ``pw_status`` test manually: 246 247.. code-block:: bash 248 249 $ ./out/pw_strict_host_{clang,gcc}_debug/obj/pw_status/test/status_test 250 251Depending on your host OS, the compiler will default to either ``clang`` or 252``gcc``. 253 254--------------------- 255Building for a device 256--------------------- 257A Pigweed "target" is a build configuration that includes a toolchain, default 258library configurations, and more to result in binaries that run natively on the 259target. With the default build invocation, you're already building for a device 260target (the STMicroelectronics STM32F429I-DISC1) in parallel with the host 261build! 262 263If you want to build JUST for the device, you can kick of watch with: 264 265.. code-block:: bash 266 267 $ pw watch stm32f429i 268 269This is equivalent to the following Ninja invocation: 270 271.. code-block:: bash 272 273 $ ninja -C out stm32f429i 274 275------------------------- 276Running tests on a device 277------------------------- 278While tests run automatically on the host, it takes a few more steps to get 279tests to run automatically on a device, too. Even though we've verified tests 280pass on the host, it's crucial to verify the same with on-device testing. We've 281encountered some unexpected bugs that can only be found by running the unit 282tests directly on the device. 283 2841. Connect device(s) 285==================== 286Connect any number of STM32F429I-DISC1 boards to your computer using the mini 287USB port on the board (**not** the micro USB). Pigweed will automatically 288detect the boards and distribute the tests across the devices. More boards = 289faster tests! Keep in mind that you may have to make some environment specific 290updates to ensure you have permissions to use the USB device. For example, on 291Linux you may need to update your udev rules and ensure you're in the plugdev 292and dialout groups. 293 294.. image:: https://storage.googleapis.com/pigweed-media/stm32f429i-disc1_connected.jpg 295 :width: 800 296 :alt: development boards connected via USB 297 2982. Launch test server 299===================== 300To allow Ninja to run tests on an arbitrary number of devices, Ninja will send 301test requests to a server running in the background. Launch the server in 302another window using the command below (remember, you'll need to activate the 303Pigweed environment first). 304 305.. code-block:: bash 306 307 $ stm32f429i_disc1_test_server 308 309**Note:** If you attach or detach any more boards to your workstation you'll 310need to relaunch this server. 311 3123. Configure GN 313=============== 314Tell GN to use the testing server by enabling a build arg specific to the 315stm32f429i-disc1 target. 316 317.. code-block:: bash 318 319 $ gn args out 320 # Append this line to the file that opens in your editor to tell GN to run 321 # on-device unit tests. 322 pw_use_test_server = true 323 324Done! 325===== 326Whenever you make code changes and trigger a build, all the affected unit tests 327will be run across the attached boards! 328 329See the demo below for an example of what this all looks like put together: 330 331.. image:: https://storage.googleapis.com/pigweed-media/pw_watch_on_device_demo.gif 332 :width: 800 333 :alt: pw watch running on-device tests 334 335-------------------------- 336Building the documentation 337-------------------------- 338In addition to the markdown documentation, Pigweed has a collection of 339information-rich RST files that are used to generate HTML documentation. All 340the docs are hosted at https://pigweed.dev/, and are built as a part of the 341default build invocation. This makes it easier to make changes and see how they 342turn out. Once built, you can find the rendered HTML documentation at 343``out/docs/gen/docs/html``. 344 345You can explicitly build just the documentation with the command below. 346 347.. code-block:: bash 348 349 $ ninja -C out docs 350 351This concludes the introduction to developing for upstream Pigweed. 352 353--------------------------- 354Building tests individually 355--------------------------- 356Sometimes it's faster to incrementally build a single test target rather than 357waiting for the whole world to build and all tests to run. GN has a built-in 358tool, ``gn outputs``, that will translate a GN build step into a Ninja build 359step. In order to build and run the right test, it's important to explicitly 360specify which target to build the test under (e.g. host, SM32F529I-DISC1). 361This can be done by appending the GN path to the target toolchain in parenthesis 362after the desired GN build step label as seen in the example below. 363 364.. code-block:: none 365 366 $ gn outputs out "//pw_status:status_test.run(//targets/host/pigweed_internal:pw_strict_host_clang_debug)" 367 pw_strict_host_clang_debug/obj/pw_status/status_test.run.pw_pystamp 368 369 $ ninja -C out pw_strict_host_clang_debug/obj/pw_status/status_test.run.pw_pystamp 370 ninja: Entering directory `out' 371 [4/4] ACTION //pw_status:status_test.run(//targets/host/pigweed_internal:pw_strict_host_clang_debug) 372 373The ``.run`` following the test target name is a sub-target created as part of 374the ``pw_test`` GN template. If you remove ``.run``, the test will build but 375not attempt to run. 376 377In macOS and Linux, ``xargs`` can be used to turn this into a single command: 378 379.. code-block:: bash 380 381 $ gn outputs out "//pw_status:status_test.run(//targets/host/pigweed_internal:pw_strict_host_clang_debug)" | xargs ninja -C out 382 383---------- 384Next steps 385---------- 386 387Contribute a patch 388================== 389Ready to submit a patch upstream? See our :ref:`docs-contributing` guide for our 390patch submission process. 391 392Quickstarts 393=========== 394Visit :ref:`docs-get-started` to learn how to set up a new Bazel-based 395project, how to add Pigweed to an existing Bazel-based project, and more. 396 397Other modules 398============= 399If you'd like to see more of what Pigweed has to offer, dive into the 400:ref:`docs-module-guides`. 401 402The :ref:`docs-kudzu` repo demonstrates how to use Pigweed in your own project. 403Note that there are many ways to leverage Pigweed and Kudzu is just one 404approach. 405 406Editor setup 407============ 408Check out the :ref:`module-pw_ide` for setting up editor configurations or run 409the following for a quick setup: 410 411.. code-block:: bash 412 413 pw ide sync 414 415Hackaday Supercon talk about Pigweed 416==================================== 417We gave a talk at Hackaday's 2021 supercon, `Give Pigweed a Whirl 418<https://hackaday.com/2021/01/13/remoticon-video-pigweed-brings-embedded-unit-testing-library-integration-to-commandline/>`_ 419 420We've made improvements since we gave the talk; for example, we now have RTOS 421primitives. 422 423Get help 424======== 425Dropping into our `chat room <https://discord.gg/M9NSeTA>`_ is the most 426immediate way to get help from the Pigweed team. 427
