1.. _docs-automated-analysis: 2 3================== 4Automated analysis 5================== 6 7The correctness and style of Pigweed's source code is continuously verified 8using a suite of automated tools. We also make it easy to use the same tools 9to verify the code of projects using Pigweed. 10 11------- 12Summary 13------- 14On presubmit or in CI we verify Pigweed using: 15 16* pylint 17* mypy 18* clang-tidy 19* AddressSanitizer (asan) 20* ThreadSanitizer (tsan) 21* UndefinedBehaviorSanitizer (ubsan) 22* OSS-Fuzz 23 24The rest of this document discusses these tools and their configuration in 25greater detail, and how to use them in your own project. 26 27-------------- 28Analysis tools 29-------------- 30 31Static analysis 32=============== 33 34PyLint 35------ 36`PyLint`_ is a customizable Python linter. Pigweed complies with almost all 37the default checks; see `.pylintrc`_ for details. PyLint detects problems such 38as overly broad catch statements, unused arguments/variables, and mutable 39default parameter values. 40 41For upstream Pigweed, PyLint can be run with ``ninja python.lint.pylint`` or 42``ninja python.lint``. It's also included in a variety of presubmit steps, 43like ``static_analysis`` and ``python_checks.gn_python_check``. See the 44`Enabling analysis for your project`_ section to learn how to run PyLint on 45your Pigweed-based project. 46 47.. _PyLint: https://pylint.org/ 48.. _.pylintrc: https://cs.pigweed.dev/pigweed/+/main:.pylintrc 49 50Mypy 51---- 52Python 3 allows for `type annotations`_ for variables, function arguments, and 53return values. Most, but not all, of Pigweed's Python code has type 54annotations, and these annotations have caught real bugs in code that didn't 55yet have unit tests. `Mypy`_ is an analysis tool that enforces these 56annotations. 57 58Mypy helps find bugs like when a string is passed into a function that expects 59a list of strings---since both are iterables this bug might otherwise be hard 60to track down. 61 62Mypy can be run with ``ninja python.lint.mypy`` or ``ninja python.lint``. It's 63also included in a variety of presubmit steps, like ``static_analysis`` and 64``python_checks.gn_python_check``. 65 66.. _type annotations: https://docs.python.org/3/library/typing.html 67.. _Mypy: http://mypy-lang.org/ 68 69clang-tidy 70---------- 71`clang-tidy`_ is a C++ "linter" and static analysis tool. It identifies 72bug-prone patterns (e.g., use after move), non-idiomatic usage (e.g., creating 73``std::unique_ptr`` with ``new`` rather than ``std::make_unique``), and 74performance issues (e.g., unnecessary copies of loop variables). 75 76While powerful, clang-tidy defines a very large number of checks, many of which 77are special-purpose (e.g., only applicable to FPGA HLS code, or code using the 78`Abseil`_ library) or have high false positive rates. Pigweed enables over 50 79checks which are relevant to an embedded C/C++ library and have good 80signal-to-noise ratios. The full list of Pigweed's checks is in `.clang-tidy`_. 81 82We do not currently enable the `Clang Static Analyzers`_ because they suffer 83from false positives, and their findings are time-consuming to manually verify. 84 85clang-tidy can be run with ``ninja static_analysis`` or ``pw presubmit --step 86static_analysis`` or ``bazelisk build --config=clang-tidy //...``. Note that as 87a static analysis tool, clang-tidy will not produce any runnable binaries: it 88simply analyzes the source files. 89 90.. _clang-tidy: https://clang.llvm.org/extra/clang-tidy/ 91.. _Abseil: https://abseil.io/ 92.. _.clang-tidy: https://cs.pigweed.dev/pigweed/+/main:.clang-tidy 93.. _Clang Static Analyzers: https://clang-analyzer.llvm.org/available_checks.html 94 95 96Clang sanitizers 97================ 98We run all of Pigweed's unit tests with the additional instrumentation 99described in this section. For more detail about these sanitizers, see the 100`Github documentation`_. 101 102* asan: `AddressSanitizer`_ detects memory errors such as out-of-bounds access 103 and use-after-free. 104* tsan: `ThreadSanitizer`_ detects data races. 105* ubsan: `UndefinedBehaviorSanitizer`_ is a fast undefined behavior detector. 106 We use the default ``-fsanitize=undefined`` option. 107 108.. note:: 109 Pigweed does not currently support `MemorySanitizer`_ (msan). See 110 :bug:`234876100` for details. 111 112The exact configurations we use for these sanitizers are in 113`pw_toolchain/host_clang/BUILD.gn <https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/host_clang/BUILD.gn>`_. 114You can see the current status of the sanitizer builds in the `Pigweed CI 115console`_, as ``pigweed-linux-san-*``. 116 117Unlike clang-tidy, the clang sanitizers are runtime instrumentation: the 118instrumented binary needs to be run for issues to be detected. 119 120.. _Github documentation: https://github.com/google/sanitizers 121.. _AddressSanitizer: https://clang.llvm.org/docs/AddressSanitizer.html 122.. _MemorySanitizer: https://clang.llvm.org/docs/MemorySanitizer.html 123.. _Pigweed CI console: https://ci.chromium.org/p/pigweed/g/pigweed/console 124.. _ThreadSanitizer: https://clang.llvm.org/docs/ThreadSanitizer.html 125.. _UndefinedBehaviorSanitizer: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html 126 127 128Fuzzers 129======= 130`Fuzz testing`_ detects errors in software by providing it with randomly 131generated inputs. We use `OSS-fuzz`_ to continuously uncover potential 132vulnerabilities in Pigweed. `Dashboard with Pigweed's latest results`_. See 133the :ref:`module-pw_fuzzer` module documentation for more details. 134 135.. _Dashboard with Pigweed's latest results: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#pigweed 136.. _Fuzz testing: https://en.wikipedia.org/wiki/Fuzzing 137.. _OSS-fuzz: https://github.com/google/oss-fuzz 138 139.. _Enabling analysis for your project: 140 141---------------------------------- 142Enabling analysis for your project 143---------------------------------- 144 145GN 146== 147 148PyLint and Mypy 149--------------- 150PyLint and Mypy can be configured to run every time your project is built by 151adding ``python.lint`` to your default build group. (You can also add one or both 152individually using ``python.lint.mypy`` and ``python.lint.pylint``.) Likewise, 153these can be added to individual presubmit steps (`examples`_). You can also 154directly include the `python_checks.gn_python_lint`_ presubmit step. 155 156.. _examples: https://cs.opensource.google/search?q=file:pigweed_presubmit.py%20%22python.lint%22&sq=&ss=pigweed%2Fpigweed 157.. _python_checks.gn_python_lint: https://cs.pigweed.dev/pigweed/+/main:pw_presubmit/py/pw_presubmit/python_checks.py?q=file:python_checks.py%20gn_python_lint&ss=pigweed%2Fpigweed 158 159clang-tidy 160---------- 161`pw_toolchain/static_analysis_toolchain.gni`_ provides the 162``pw_static_analysis_toolchain`` template that can be used to create a build 163group performing static analysis. See :ref:`module-pw_toolchain` documentation 164for more details. This group can then be added as a presubmit step using 165pw_presubmit. 166 167You can place a ``.clang-tidy`` file at the root of your repository to control 168which checks are executed. See the `clang documentation`_ for a discussion of how 169the tool chooses which ``.clang-tidy`` files to apply when run on a particular 170source file. 171 172.. _pw_toolchain/static_analysis_toolchain.gni: https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/static_analysis_toolchain.gni 173.. _clang documentation: https://clang.llvm.org/extra/clang-tidy/ 174 175Clang sanitizers 176---------------- 177There are two ways to enable sanitizers for your build. 178 179GN args on debug toolchains 180^^^^^^^^^^^^^^^^^^^^^^^^^^^ 181If you are already building your tests with one of the following toolchains (or 182a toolchain derived from one of them): 183 184* ``pw_toolchain_host_clang.debug`` 185* ``pw_toolchain_host_clang.speed_optimized`` 186* ``pw_toolchain_host_clang.size_optimized`` 187 188you can enable the clang sanitizers simply by setting the gn arg 189``pw_toolchain_SANITIZERS`` to the desired subset of 190``["address", "thread", "undefined"]``. 191 192Example 193....... 194If your project defines a toolchain ``host_clang_debug`` that is derived from 195one of the above toolchains, and you'd like to run the ``pw_executable`` target 196``sample_binary`` defined in the ``BUILD.gn`` file in ``examples/sample`` with 197asan, you would run, 198 199.. code-block:: bash 200 201 gn gen out --args='pw_toolchain_SANITIZERS=["address"]' 202 ninja -C out host_clang_debug/obj/example/sample/bin/sample_binary 203 out/host_clang_debug/obj/example/sample/bin/sample_binary 204 205Sanitizer toolchains 206^^^^^^^^^^^^^^^^^^^^ 207Otherwise, instead of using ``gn args`` you can build your tests with the 208appropriate toolchain from the following list (or a toolchain derived from one 209of them): 210 211* ``pw_toolchain_host_clang.asan`` 212* ``pw_toolchain_host_clang.ubsan`` 213* ``pw_toolchain_host_clang.tsan`` 214 215See the :ref:`module-pw_toolchain` module documentation for more 216about Pigweed toolchains. 217 218Bazel 219===== 220 221.. _docs-automated-analysis-clang-sanitizers: 222 223Clang sanitizers 224---------------- 225If you're using Pigweed's own host toolchain configuration, you can enable 226AddressSanitizer by building with the appropriate flag: 227 228.. code-block:: sh 229 230 bazelisk build --@pigweed//pw_toolchain/host_clang:asan //... 231 232If you're building your own toolchain, you can add 233``@pigweed//pw_toolchain/cc/args:asan`` to it. 234 235clang-tidy 236---------- 237We recommend using `bazel_clang_tidy 238<https://github.com/erenon/bazel_clang_tidy>`__ to run clang-tidy from Bazel. 239 240If you're using Pigweed's own host toolchain configuration, see the 241:ref:`module-pw_toolchain-bazel-clang-tidy` section for information on how to 242enable clang-tidy in your build. 243 244 245Fuzzers 246======= 247See the :ref:`module-pw_fuzzer` module documentation. 248