1# syzbot 2 3`syzbot` system continuously fuzzes main Linux kernel branches and automatically 4reports found bugs to kernel mailing lists. 5[syzbot dashboard](https://syzkaller.appspot.com) shows current statuses of 6bugs. All `syzbot`-reported bugs are also CCed to 7[syzkaller-bugs mailing list](https://groups.google.com/forum/#!forum/syzkaller-bugs). 8Direct all questions to `syzkaller@googlegroups.com`. 9 10## Bug status tracking 11 12`syzbot` needs to know when a bug is fixed in order to (1) verify that it is 13in fact fixed and (2) be able to report other similarly-looking crashes 14(while a bug is considered open all similarly-looking crashes are merged into 15the existing bug). To understand when a bug is fixed `syzbot` needs to know 16what commit fixes the bug; once `syzbot` knows the commit it will track when 17the commit reaches all kernel builds on all tracked branches. Only when the 18commit reaches all builds, the bug is considered closed (new similarly-looking 19crashes create a new bug). 20 21## Communication with syzbot 22 23If you fix a bug reported by `syzbot`, please add the provided `Reported-by` 24tag to the commit. You can also communicate with `syzbot` by replying to its 25emails. The commands are: 26 27- to attach a fixing commit to the bug (if you forgot to add `Reported-by` tag): 28``` 29#syz fix: exact-commit-title 30```` 31It's enough that the commit is merged into any tree or you are reasonably sure 32about its final title, in particular, you don't need to wait for the commit to 33be merged into upstream tree. `syzbot` only needs to know the title by which 34it will appear in tested trees. In case of an error or a title change, you can 35override the commit simply by sending another `#syz fix` command. 36- to mark the bug as a duplicate of another `syzbot` bug: 37``` 38#syz dup: exact-subject-of-another-report 39``` 40- to undo a previous dup command and turn it into an independent bug again: 41``` 42#syz undup 43``` 44- to mark the bug as a one-off invalid report (e.g. induced by a previous memory corruption): 45``` 46#syz invalid 47``` 48**Note**: if the crash happens again, it will cause creation of a new bug report. 49 50**Note**: all commands must start from beginning of the line. 51 52**Note**: please keep `syzkaller-bugs@googlegroups.com` mailing list in CC. 53It serves as a history of what happened with each bug report. 54 55## Testing patches 56 57`syzbot` can test patches for bugs *with reproducers*. This can be used for 58testing of fix patches, or just for debugging (i.e. adding additional checks to 59code and testing with them), or to check if the bug still happens. To test on 60a particular git tree and branch reply with: 61``` 62#syz test: git://repo/address.git branch 63``` 64or alternatively, to test on exact commit reply with: 65``` 66#syz test: git://repo/address.git commit-hash 67``` 68If you also provide a patch with the email, `syzbot` will apply it on top of the 69tree before testing. The patch can be provided either inline in email text or as 70a text attachment (which is more reliable if your email client messes with 71whitespaces). 72 73If you don't provide a patch, `syzbot` will test the tree as is. 74This is useful if this is your own tree which already contains the patch, 75or to check if the bug is already fixed by some recent commit. 76 77After sending an email you should get a reply email with results within an hour. 78 79Note: you may send the request only to `syzbot` email address, as patches sent 80to some mailing lists (e.g. netdev, netfilter-devel) will trigger patchwork. 81 82## syzkaller reproducers 83 84`syzbot` aims at providing stand-alone C reproducers for all reported bugs. 85However, sometimes it can't extract a reproducer at all, or can only extract a 86syzkaller reproducer. syzkaller reproducers are programs in a special syzkaller 87notation and they can be executed on the target system with a little bit more 88effort. See [this](https://github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md) 89for instructions. 90 91A syskaller program can also give you an idea as to what syscalls with what 92arguments were executed (note that some calls can actually be executed in 93parallel). 94 95A syzkaller program can be converted to an almost equivalent C source using `syz-prog2c` utility. `syz-prog2c` has lots of flags in common with [syz-execprog](https://github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md), e.g. `-threaded`/`-collide` which control if the syscalls are executed sequentially or in parallel. An example invocation: 96 97``` 98syz-prog2c -prog repro.syz.txt -threaded -collide -repeat -procs=8 -sandbox=namespace -tun -tmpdir -waitrepeat 99``` 100 101However, note that if `syzbot` did not provide a C reproducer, it wasn't able to trigger the bug using the C program (though, it can be just because the bug is triggered by a subtle race condition). 102 103## Crash does not reproduce? 104 105If the provided reproducer does not work for you, most likely it is related to the 106fact that you have slightly different setup than `syzbot`. `syzbot` has obtained 107the provided crash report on the provided reproducer on a freshly-booted 108machine, so the reproducer worked for it somehow. 109 110`syzbot` uses GCE VMs for testing, but *usually* it is not important. 111 112If the reproducer exits quickly, try to run it several times, or in a loop. 113There can be some races involved. 114 115Exact compilers used by `syzbot` can be found here: 116- [gcc 7.1.1 20170620](https://storage.googleapis.com/syzkaller/gcc-7.tar.gz) (245MB) 117- [gcc 8.0.1 20180301](https://storage.googleapis.com/syzkaller/gcc-8.0.1-20180301.tar.gz) (286MB) 118- [gcc 8.0.1 20180412](https://storage.googleapis.com/syzkaller/gcc-8.0.1-20180412.tar.gz) (33MB) 119- [clang 7.0.0 (trunk 329060)](https://storage.googleapis.com/syzkaller/clang-kmsan-329060.tar.gz) (44MB) 120- [clang 7.0.0 (trunk 334104)](https://storage.googleapis.com/syzkaller/clang-kmsan-334104.tar.gz) (44MB) 121 122A qemu-suitable Debian/wheezy image can be found [here](https://storage.googleapis.com/syzkaller/wheezy.img) (1GB, compression somehow breaks it), root ssh key for it is [here](https://storage.googleapis.com/syzkaller/wheezy.img.key). 123A reference `qemu` command line to run it is as follows: 124``` 125qemu-system-x86_64 -hda wheezy.img -net user,hostfwd=tcp::10022-:22 \ 126 -net nic -nographic -enable-kvm -m 2G -smp 4 -cpu host \ 127 -kernel arch/x86/boot/bzImage \ 128 -append "root=/dev/sda console=ttyS0 earlyprintk=serial rodata=n \ 129 oops=panic panic_on_warn=1 panic=86400 kvm-intel.nested=1" 130``` 131And then you can ssh into it using: 132``` 133ssh -p 10022 -i wheezy.id_rsa root@localhost 134``` 135 136## No reproducer at all? 137 138Reproducers are best-effort. `syzbot` always tries to create reproducers, and 139once it has one it adds it to the bug. If there is no reproducer referenced in a 140bug, a reproducer does not exist. There are multiple reasons why `syzbot` can 141fail to create a reproducer: some crashes are caused by subtle races and are 142very hard to reproduce in general; some crashes are caused by global accumulated 143state in kernel (e.g. lockdep reports); some crashes are caused by 144non-reproducible coincidences (e.g. an integer `0x12345` happened to reference an 145existing IPC object) and there is long tail of other reasons. 146 147## Moderation queue 148 149Bugs with reproducers are automatically reported to kernel mailing lists. 150Bugs without reproducers are first staged in moderation queue to filter out 151invalid, unactionable or duplicate reports. Staged bugs are shown on dashboard 152in [moderation](https://syzkaller.appspot.com/#upstream-moderation2) section 153and mailed to 154[syzkaller-upstream-moderation](https://groups.google.com/forum/#!forum/syzkaller-upstream-moderation) 155mailing list. Staged bugs accept all commands supported for reported bugs 156(`fix`, `dup`, `invalid`) with a restriction that reported and staged bugs 157can't be `dup`-ed onto each other in any direction. Additionally, staged bugs 158accept upstream command: 159``` 160#syz upstream 161``` 162which sends the bug to kernel mailing lists. 163 164## KMSAN bugs 165 166`KMSAN` is a dynamic, compiler-based tool (similar to `KASAN`) that detects 167uses of uninitialized values. As compared to (now deleted) `KMEMCHECK` which 168simply detected loads of non-stored-to memory, `KMSAN` tracks precise 169propagation of uninitialized values through memory and registers and only flags 170actual eventual uses of uninitialized values. For example, `KMSAN` will detect 171a branch on or a `copy_to_user()` of values that transitively come from 172uninitialized memory created by heap/stack allocations. This ensures 173/theoretical/ absense of both false positives and false negatives (with some 174implementation limitations of course). 175 176`KMSAN` is not upstream yet, though, we want to upstream it later. For now, 177it lives in [github.com/google/kmsan](https://github.com/google/kmsan) and is 178based on a reasonably fresh upstream tree. As the result, any patch testing 179requests for `KMSAN` bugs need to go to `KMSAN` tree 180(`https://github.com/google/kmsan.git` repo, `master` branch). Also note that 181`KMSAN` requires `clang` compiler. 182 183Report explanation. The first call trace points to the `use` of the uninit value 184(which is usually a branching or copying it to userspace). Then there are 0 or 185more "Uninit was stored to memory at:" stacks which denote how the unint value 186travelled through memory. Finally there is a "Uninit was created at:" 187section which points either to a heap allocation or a stack variable which 188is the original source of uninitialized-ness. 189 190## No custom patches 191 192While `syzbot` can test patches that fix bugs, it does not support applying 193custom patches during fuzzing. It always tests vanilla unmodified git trees. 194There are several reasons for this: 195 196- custom patches may not apply tomorrow 197- custom patches may not apply to all of the tested git trees 198- it's hard to communicate exact state of the code with bug reports (not just hash anymore) 199- line numbers won't match in reports (which always brings suspecion as to the quality of reports) 200- custom patches can also introduce bugs, and even if they don't a developer may (rightfully) 201 question validity of and may not want to spend time on reports obtained 202 with a number of out-of-tree patches 203- order of patch application generatelly matters, and at some point patches 204 need to be removed, there is nobody to manage this 205 206We've experimented with application of custom patches in the past and it lead 207to unrecoverable mess. If you want `syzbot` to pick up patches sooner, 208ask tree maintainers for priority handling. 209 210## Kernel configs 211 212Kernel configs, sysctls and command line arguments that `syzbot` uses are available in [/dashboard/config](/dashboard/config). 213 214## Is syzbot code available? 215 216Yes, it is [here](https://github.com/google/syzkaller/tree/master/dashboard/app). 217