1# Reproducing libFuzzer and AFL crashes 2 3*** note 4**Requirements:** For Windows, you must convert the forward slashes (/) to 5backslashes (\\) in the commands below and use `set` command instead of `export` 6to set the environment variable (step 4). Note that these commands are intended 7to be used with cmd.exe, not PowerShell. Also, you may find [these tips] on how 8to debug an ASAN instrumented binary helpful. 9*** 10 11[TOC] 12 13## Crashes reported as Reproducible 14 15The majority of the bugs reported by ClusterFuzz have **Reproducible** label. 16That means there is a testcase that can be used to reliably reproduce the crash. 17 181. Download the testcase from ClusterFuzz. If you are CCed on an issue filed by 19 ClusterFuzz, a link to it is next to "Reproducer testcase" in the bug 20 description. 21 22 For the rest of this walkthrough, we call the path of this 23 file: `$TESTCASE_PATH` and the fuzz target you want to reproduce a 24 crash on: `$FUZZER_NAME` (provided as "Fuzz Target" in the bug 25 description). 26 272. Generate gn build configuration: 28 29``` 30gn args out/fuzz 31``` 32 33 This will open up an editor. Copy the gn configuration parameters from the 34 values provided in `GN Config` section in the ClusterFuzz testcase report. 35 36 373. Build the fuzzer: 38 39``` 40autoninja -C out/fuzz $FUZZER_NAME 41``` 42 434. Set the `*SAN_OPTIONS` environment variable as provided in the 44 `Crash Stacktrace` section in the testcase report. 45 Here is an example value of `ASAN_OPTIONS` that is similar to its value on 46 ClusterFuzz: 47 48``` 49export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=1 50``` 51 525. Run the fuzz target: 53 54``` 55out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH 56``` 57 58If you see an un-symbolized stacktrace, please see the instructions [here]. 59 60[File a bug] if you run into any issues. 61 62## Crashes reported as Unreproducible 63 64ClusterFuzz generally does not report issues that it cannot reliably reproduce, 65unless the following condition is met. If a certain crash is occurring often 66enough, such a crash might be reported with **Unreproducible** label and an 67explicit clarification that there is no convenient way to reproduce it. There 68are two ways to work with such crashes. 69 701. Try a speculative fix based on the stacktrace. Once the fix is landed, wait a 71 couple days and then check Crash Statistics section on the ClusterFuzz 72 testcase report page. If the fix works out, you will see that the crash is 73 not happening anymore. If the crash does not occur again for a little while, 74 ClusterFuzz will automatically close the issue as Verified. 75 762. (libFuzzer only) Try to reproduce the whole fuzzing session. This workflow is 77 very similar to the one described above for the **Reproducible** crashes. The 78 only differences are: 79 80 * On step 1, instead of downloading a single testcase, you need to download 81 corpus backup. This can be done using the following command: 82``` 83gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip . 84``` 85 86 * Alternatively, you can navigate to the following URL in your browser and 87 download the `latest.zip` file: 88``` 89https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME 90``` 91 92 * Create an empty directory and unpack the corpus into it. 93 * Follow steps 2-4 in the **Reproducible** section above. 94 * On step 5, use the following command: 95 96``` 97out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP 98``` 99 100 * Wait and hope that the fuzzer will crash. 101 102Waiting for a crash to occur may take some time (up to 1hr), but if it happens, 103you will be able to test the fix locally and/or somehow debug the issue. 104 105## Minimizing a crash input (optional) 106 107ClusterFuzz does crash input minimization automatically, and a typical crash 108report has two testcases available for downloading: 109 110* An original testcase that has triggered the crash; 111* A minimized testcase that is smaller than the original but triggers the same 112 crash. 113 114If you would like to further minimize a testcase, run the fuzz target with the 115two additional arguments: 116 117* `-minimize_crash=1` 118* `-exact_artifact_path=<output_filename_for_minimized_testcase>` 119 120The full command would be: 121 122``` 123out/fuzz/$FUZZER_NAME -minimize_crash=1 -exact_artifact_path=<minimized_testcase_path> $TESTCASE_PATH 124``` 125 126This might be useful for large testcases that make it hard to identify a root 127cause of a crash. You can leave the minimization running locally for a while 128(e.g. overnight) for better results. 129 130 131[File a bug]: https://bugs.chromium.org/p/chromium/issues/entry?components=Tools%3EStability%3ElibFuzzer&comment=What%20problem%20are%20you%20seeing 132[here]: getting_started.md#symbolizing-a-stacktrace 133[these tips]: https://github.com/google/sanitizers/wiki/AddressSanitizerWindowsPort#debugging 134