1.. _module-pw_fuzzer-guides-using_fuzztest: 2 3======================================== 4pw_fuzzer: Adding Fuzzers Using FuzzTest 5======================================== 6.. pigweed-module-subpage:: 7 :name: pw_fuzzer 8 9.. note:: 10 11 `FuzzTest`_ is currently only supported on Linux and MacOS using Clang. 12 13.. _module-pw_fuzzer-guides-using_fuzztest-toolchain: 14 15---------------------------------------- 16Step 0: Set up FuzzTest for your project 17---------------------------------------- 18.. note:: 19 20 This workflow only needs to be done once for a project. 21 22FuzzTest and its dependencies are not included in Pigweed and need to be added. 23 24See the following: 25 26* :ref:`module-pw_third_party_abseil_cpp-using_upstream` 27* :ref:`module-pw_third_party_fuzztest-using_upstream` 28* :ref:`module-pw_third_party_googletest-using_upstream` 29* :ref:`module-pw_third_party_re2-using_upstream` 30 31.. tab-set:: 32 33 .. tab-item:: GN 34 :sync: gn 35 36 You may not want to use upstream GoogleTest all the time. For example, it 37 may not be supported on your target device. In this case, you can limit it 38 to a specific toolchain used for fuzzing. For example: 39 40 .. code-block:: 41 42 import("$dir_pw_toolchain/host/target_toolchains.gni") 43 44 my_toolchains = { 45 ... 46 clang_fuzz = { 47 name = "my_clang_fuzz" 48 forward_variables_from(pw_toolchain_host.clang_fuzz, "*", ["name"]) 49 pw_unit_test_MAIN = "$dir_pw_fuzzer:fuzztest_main" 50 pw_unit_test_BACKEND = "$dir_pw_fuzzer:gtest" 51 } 52 ... 53 } 54 55 .. tab-item:: CMake 56 :sync: cmake 57 58 FuzzTest is enabled by setting several CMake variables. The easiest way to 59 set these is to extend your ``toolchain.cmake`` file. 60 61 For example: 62 63 .. code-block:: 64 65 include(my_project_toolchain.cmake) 66 67 set(dir_pw_third_party_fuzztest 68 "path/to/fuzztest" 69 CACHE STRING "" FORCE 70 ) 71 set(dir_pw_third_party_googletest 72 "path/to/googletest" 73 CACHE STRING "" FORCE 74 ) 75 set(pw_unit_test_BACKEND 76 "pw_third_party.fuzztest" 77 CACHE STRING "" FORCE 78 ) 79 80 .. tab-item:: Bazel 81 :sync: bazel 82 83 FuzzTest provides a build configuration that can be imported into your 84 ``.bazelrc`` file. Add the following: 85 86 .. code-block:: 87 88 # Include FuzzTest build configurations. 89 try-import %workspace%/third_party/fuzztest/fuzztest.bazelrc 90 build:fuzztest --@pigweed//targets:fuzztest_config=//pw_fuzzer:fuzztest 91 92---------------------------------------- 93Step 1: Write a unit test for the target 94---------------------------------------- 95 96As noted previously, the very first step is to identify one or more target 97behavior that would benefit from testing. See `FuzzTest Use Cases`_ for more 98details on how to identify this code. 99 100Once identified, it is useful to start from a unit test. You may already have a 101unit test writtern, but if not it is likely still be helpful to write one first. 102Many developers are more familiar with writing unit tests, and there are 103detailed guides available. See for example the `GoogleTest documentation`_. 104 105This guide will use code from ``//pw_fuzzer/examples/fuzztest/``. This code 106includes the following object as an example of code that would benefit from 107fuzzing for undefined behavior and from roundtrip fuzzing. 108 109.. note:: 110 111 To keep the example simple, this code uses the standard library. As a result, 112 this code may not work with certain devices. 113 114.. literalinclude:: ../examples/fuzztest/metrics.h 115 :language: cpp 116 :linenos: 117 :start-after: [pwfuzzer_examples_fuzztest-metrics_h] 118 :end-before: [pwfuzzer_examples_fuzztest-metrics_h] 119 120Unit tests for this class might attempt to deserialize previously serialized 121objects and to deserialize invalid data: 122 123.. literalinclude:: ../examples/fuzztest/metrics_unittest.cc 124 :language: cpp 125 :linenos: 126 :start-after: [pwfuzzer_examples_fuzztest-metrics_unittest] 127 :end-before: [pwfuzzer_examples_fuzztest-metrics_unittest] 128 129-------------------------------------------- 130Step 2: Convert your unit test to a function 131-------------------------------------------- 132 133Examine your unit tests and identify any places you have fixed values that could 134vary. Turn your unit test into a function that takes those values as parameters. 135Since fuzzing may not occur on all targets, you should preserve your unit test 136by calling the new function with the previously fixed values. 137 138.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 139 :language: cpp 140 :linenos: 141 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest1] 142 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest1] 143 144.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 145 :language: cpp 146 :linenos: 147 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest3] 148 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest3] 149 150Note that in ``ArbitrarySerializeAndDeserialize`` we no longer assume the 151marshalling will always be successful, and exit early if it is not. You may need 152to make similar modifications to your unit tests if constraints on parameters 153are not expressed by `domains`__ as described below. 154 155.. __: `FuzzTest Domain Reference`_ 156 157-------------------------------------------- 158Step 3: Add a FUZZ_TEST macro invocation 159-------------------------------------------- 160 161Now, include ``"fuzztest/fuzztest.h"`` and pass the test suite name and your 162function name to the ``FUZZ_TEST`` macro. Call ``WithDomains`` on the returned 163object to specify the input domain for each parameter of the function. For 164example: 165 166.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 167 :language: cpp 168 :linenos: 169 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest2] 170 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest2] 171 172.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 173 :language: cpp 174 :linenos: 175 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest4] 176 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest4] 177 178You may know of specific values that are "interesting", i.e. that represent 179boundary conditions, involve, special handling, etc. To guide the fuzzer towards 180these code paths, you can include them as `seeds`_. However, as noted in 181the comments of the examples, it is recommended to include a unit test with the 182original parameters to ensure the code is tested on target devices. 183 184FuzzTest provides more detailed documentation on these topics. For example: 185 186* Refer to `The FUZZ_TEST Macro`_ reference for more details on how to use this 187 macro. 188 189* Refer to the `FuzzTest Domain Reference`_ for details on all the different 190 types of domains supported by FuzzTest and how they can be combined. 191 192* Refer to the `Test Fixtures`_ reference for how to create fuzz tests from unit 193 tests that use GoogleTest fixtures. 194 195------------------------------------ 196Step 4: Add the fuzzer to your build 197------------------------------------ 198Next, indicate that the unit test includes one or more fuzz tests. 199 200.. tab-set:: 201 202 .. tab-item:: GN 203 :sync: gn 204 205 The ``pw_fuzz_test`` template can be used to add the necessary FuzzTest 206 dependency and generate test metadata. 207 208 For example, consider the following ``BUILD.gn``: 209 210 .. literalinclude:: ../examples/fuzztest/BUILD.gn 211 :linenos: 212 :start-after: [pwfuzzer_examples_fuzztest-gn] 213 :end-before: [pwfuzzer_examples_fuzztest-gn] 214 215 .. tab-item:: CMake 216 :sync: cmake 217 218 Unit tests can support fuzz tests by simply adding a dependency on 219 FuzzTest. 220 221 For example, consider the following ``CMakeLists.txt``: 222 223 .. literalinclude:: ../examples/fuzztest/CMakeLists.txt 224 :linenos: 225 :start-after: [pwfuzzer_examples_fuzztest-cmake] 226 :end-before: [pwfuzzer_examples_fuzztest-cmake] 227 228 .. tab-item:: Bazel 229 :sync: bazel 230 231 Unit tests can support fuzz tests by simply adding a dependency on 232 FuzzTest. 233 234 For example, consider the following ``BUILD.bazel``: 235 236 .. literalinclude:: ../examples/fuzztest/BUILD.bazel 237 :linenos: 238 :start-after: [pwfuzzer_examples_fuzztest-bazel] 239 :end-before: [pwfuzzer_examples_fuzztest-bazel] 240 241------------------------ 242Step 5: Build the fuzzer 243------------------------ 244.. tab-set:: 245 246 .. tab-item:: GN 247 :sync: gn 248 249 Build using ``ninja`` on a target that includes your fuzzer with a 250 :ref:`fuzzing toolchain<module-pw_fuzzer-guides-using_fuzztest-toolchain>`. 251 252 Pigweed includes a ``//:fuzzers`` target that builds all tests, including 253 those with fuzzers, using a fuzzing toolchain. You may wish to add a 254 similar top-level to your project. For example: 255 256 .. code-block:: 257 258 group("fuzzers") { 259 deps = [ ":pw_module_tests.run($dir_pigweed/targets/host:host_clang_fuzz)" ] 260 } 261 262 .. tab-item:: CMake 263 :sync: cmake 264 265 Build using ``cmake`` with the FuzzTest and GoogleTest variables set. For 266 example: 267 268 .. code-block:: 269 270 cmake ... \ 271 -Ddir_pw_third_party_fuzztest=path/to/fuzztest \ 272 -Ddir_pw_third_party_googletest=path/to/googletest \ 273 -Dpw_unit_test_BACKEND=pw_third_party.fuzztest 274 275 276 .. tab-item:: Bazel 277 :sync: bazel 278 279 By default, ``bazel`` will simply omit the fuzz tests and build unit 280 tests. To build these tests as fuzz tests, specify the ``fuzztest`` 281 config. For example: 282 283 .. code-block:: sh 284 285 bazel build //... --config=fuzztest 286 287---------------------------------- 288Step 6: Running the fuzzer locally 289---------------------------------- 290.. TODO: b/281138993 - Add tooling to make it easier to find and run fuzzers. 291 292.. tab-set:: 293 294 .. tab-item:: GN 295 :sync: gn 296 297 When building. Most toolchains will simply omit the fuzz tests and build 298 and run unit tests. A 299 :ref:`fuzzing toolchain<module-pw_fuzzer-guides-using_fuzztest-toolchain>` 300 will include the fuzzers, but only run them for a limited time. This makes 301 them suitable for automated testing as in CQ. 302 303 If you used the top-level ``//:fuzzers`` described in the previous 304 section, you can find available fuzzers using the generated JSON test 305 metadata file: 306 307 .. code-block:: sh 308 309 jq '.[] | select(contains({tags: ["fuzztest"]}))' \ 310 out/host_clang_fuzz/obj/pw_module_tests.testinfo.json 311 312 To run a fuzz with different options, you can pass additional flags to the 313 fuzzer binary. This binary will be in a subdirectory related to the 314 toolchain. For example: 315 316 .. code-block:: sh 317 318 out/host_clang_fuzz/obj/my_module/test/metrics_test \ 319 --fuzz=MetricsTest.Roundtrip 320 321 Additional `sanitizer flags`_ may be passed uisng environment variables. 322 323 .. tab-item:: CMake 324 :sync: cmake 325 326 When built with FuzzTest and GoogleTest, the fuzzer binaries can be run 327 directly from the CMake build directory. By default, the fuzzers will only 328 run for a limited time. This makes them suitable for automated testing as 329 in CQ. To run a fuzz with different options, you can pass additional flags 330 to the fuzzer binary. 331 332 For example: 333 334 .. code-block:: sh 335 336 build/my_module/metrics_test --fuzz=MetricsTest.Roundtrip 337 338 .. tab-item:: Bazel 339 :sync: bazel 340 341 By default, ``bazel`` will simply omit the fuzz tests and build and run 342 unit tests. To build these tests as fuzz tests, specify the "fuzztest" 343 config. For example: 344 345 .. code-block:: sh 346 347 bazel test //... --config=fuzztest 348 349 This will build the tests as fuzz tests, but only run them for a limited 350 time. This makes them suitable for automated testing as in CQ. 351 352 To run a fuzz with different options, you can use ``run`` and pass 353 additional flags to the fuzzer binary. For example: 354 355 .. code-block:: sh 356 357 bazel run //my_module:metrics_test --config=fuzztest \ 358 --fuzz=MetricsTest.Roundtrip 359 360Running the fuzzer should produce output similar to the following: 361 362.. code-block:: 363 364 [.] Sanitizer coverage enabled. Counter map size: 21290, Cmp map size: 262144 365 Note: Google Test filter = MetricsTest.Roundtrip 366 [==========] Running 1 test from 1 test suite. 367 [----------] Global test environment set-up. 368 [----------] 1 test from MetricsTest 369 [ RUN ] MetricsTest.Roundtrip 370 [*] Corpus size: 1 | Edges covered: 131 | Fuzzing time: 504.798us | Total runs: 1.00e+00 | Runs/secs: 1 371 [*] Corpus size: 2 | Edges covered: 133 | Fuzzing time: 934.176us | Total runs: 3.00e+00 | Runs/secs: 3 372 [*] Corpus size: 3 | Edges covered: 134 | Fuzzing time: 2.384383ms | Total runs: 5.30e+01 | Runs/secs: 53 373 [*] Corpus size: 4 | Edges covered: 137 | Fuzzing time: 2.732274ms | Total runs: 5.40e+01 | Runs/secs: 54 374 [*] Corpus size: 5 | Edges covered: 137 | Fuzzing time: 7.275553ms | Total runs: 2.48e+02 | Runs/secs: 248 375 376.. TODO: b/282560789 - Add guides/improve_fuzzers.rst 377.. TODO: b/281139237 - Add guides/continuous_fuzzing.rst 378.. ---------- 379.. Next steps 380.. ---------- 381.. Once you have a working fuzzer, the next steps are to: 382 383.. * `Run it continuously on a fuzzing infrastructure <continuous_fuzzing>`_. 384.. * `Measure its code coverage and improve it <improve_fuzzers>`_. 385 386.. _FuzzTest: https://github.com/google/fuzztest 387.. _FuzzTest Domain Reference: https://github.com/google/fuzztest/blob/main/doc/domains-reference.md 388.. _FuzzTest Use Cases: https://github.com/google/fuzztest/blob/main/doc/use-cases.md 389.. _GoogleTest documentation: https://google.github.io/googletest/ 390.. _Test Fixtures: https://github.com/google/fuzztest/blob/main/doc/fixtures.md 391.. _The FUZZ_TEST Macro: https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md 392.. _sanitizer flags: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags 393.. _seeds: https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#initial-seeds 394