1# Fuzzy Fastboot 2 3Fuzzy Fastboot (FF) is a standalone automated conformance and penetration tester for 4validating device-side fastboot protocol implementations. 5The tool is completely generic, and uses a simple extensible XML 6configuration file to auto-generate device-specific tests for any device. 7Any Android device that uses the fastboot protocol should have fuzzy fastboot run on it prior to 8release to find implementation bugs, make sure it conforms to the fastboot spec, 9and that it safely handles malicious inputs. 10 11 12## Background 13The [fastboot protocol](../README.md) provides an easy way to manage low level 14aspects of the device directly from bootloader. However, with great power comes 15great responsibility. An improper or insecure fastboot implementation can 16open the possibility for critical security exploits on the bootloader via fastboot 17commands. Furthermore, an untrustworthy or insecure bootloader means nothing that is 18either directly or indirectly bootstrapped by the bootloader can be trusted (including Android). 19By checking a bootloader's conformance to the fastboot spec, as well as make sure 20nefarious/malformed input is properly and gracefully handled, easy exploits of a 21device's bootloaders can be mitigated. 22 23Additionally, since the fastboot tool itself must support a myriad of fastboot 24implementations, it is important to make sure an implementation is conforming to 25avoid potential incompatibilities with the fastboot command line tool itself. 26Thus, Fuzzy Fastboot also checks for proper conformance to the spec. 27 28## Overview 29Fuzzy Fastboot is written in C++ and uses [Google Test](https://github.com/google/googletest) 30for the underlying test framework. This means that Fuzzy Fastboot supports all of 31gtest's command line flags and options. 32 33Additionally, by using gtest it makes it extremely easy to add additional C++ based 34tests to Fuzzy Fastboot. However, in most cases the optional device specific 35XML configuration file that is provided to Fuzzy Fastboot supports the necessary 36features and hooks for testing device specific commands/features 37without touching the underlying C++. 38 39### Generic Tests 40Without a provided device XML configuration, Fuzzy Fastboot can only perform 41some basic tests that are generic to any fastboot device. These generic tests are 42divided into several test suite categories: 43 441. **USBFunctionality** - Test USB communication 452. **Conformance** - Test the device properly handles well-formed fastboot commands 463. **UnlockPermissions** - Test commands only allowed in the unlocked state work 474. **LockPermissions** - Test commands only not allowed in the locked state are rejected 485. **Fuzz** - Test malicious and/or ill-formed commands are properly and gracefully handled 49 50 51### XML Generated Tests 52With a provided XML device configuration, Fuzzy Fastboot will be able to generate 53many more additional tests cases. 54 55The device config XML has five element pairs all inside a root level `<config>`: 56 57#### `<getvar>` Element 58Inside the `<getvar></getvar>` element pairs, one should list all the device's getvar 59variables, with an associated ECMAScript regex you wish the returned variable to match on. 60Each tested variable should appear in a `<var key="key" assert="regex"/>` format. 61For example: 62```xml 63<getvar> 64 <var key="product" assert="superphone2000"/> 65 <var key="secure" assert="no|yes"/> 66 <var key="battery-voltage" assert="[34][[:digit:]]{3}"/> 67 <!-- etc... --> 68</getvar> 69``` 70 71#### `<partitions>` Element 72Inside the `<partitions></partitions>` element pairs, one should list all the device's 73partitions. Each device partition has should be put inside a `<part/>` element. 74The `<part/>` element supports the following attributes: 75 76 77| Attribute | Value | Purpose | Default | 78|-----------|----------------|---------------------------------------------------------------------------------------------|----------| 79| value | Partition name | The name of the partition | Required | 80| slots | "yes" or "no" | Is this partition is slotted | "no" | 81| test | "yes" or "no" | Is Fuzzy Fastboot is allowed to generate tests that overwrite this partition | Required | 82| hashable | "yes" or "no" | Is this partition hashable with the hash command specified in `<checksum>` | "yes" | 83| parsed | "yes" or "no" | Does the bootloader parse this partition, such as look for a header, look for magic, etc... | "no" | 84 85For example: 86```xml 87<!-- All the device partitions should be listed here --> 88<partitions> 89 <part value="boot" slots="yes" test="yes" hashable="yes" parsed="yes"/> 90 <part value="modem" slots="yes" test="yes" hashable="yes"/> 91 <part value="userdata" slots="no" test="yes" hashable="no"/> 92 <!-- etc... --> 93</partitions> 94``` 95 96#### `<packed>` Element 97Most devices have pseudo partitions, such as a `bootloader` partition, 98that in reality is composed of several real partitions. 99When one of these pseudo partitions is flashed, the bootloader 100will internally expand the image into the individual images for each underlying 101partition. These pseudo partitions should be listed inside a `<part></part>` 102element pair. Each element `<part>` has a mandatory attribute `value`, 103which lists the name of this pseudo partition, and a `slots` attribute, 104which can be yes or no if this pseudo partition is slotted. 105Additionally, inside the `<part></part>` element pair, one should list 106all the real partition that make up this pseudo partition inside of 107`<child>PART_NAME</child>` element pairs. 108An example is should below: 109 110```xml 111<!-- All the device packed partitions should be listed here --> 112<packed> 113 <part value="bootloader" slots="yes"> 114 <!-- We list the real partitions it is composed of --> 115 <child>foo1</child> 116 <child>foo2</child> 117 <child>bar3</child> 118 <!-- We list tests, expect defaults to 'okay' --> 119 <test packed="bootloader.img" unpacked="unpacked"/> 120 <test packed="bootloader_garbage.img" expect="fail"/> 121 </part> 122</packed> 123``` 124 125You might notice there are additional `<test/>` elements as well contained inside of 126a `<part></part>` pair. This is because Fuzzy Fastboot allows (and recommends) one to specify 127valid and invalid test packed images for flashing this particular pseudo partition. 128Additionally, one should specify a folder with all the partitions' images 129that the packed image unpacks to. If your device supports hashing partitions, this 130will allow Fuzzy Fastboot to validate the images are unpacking correctly, and 131the correct slots are being flashed. 132 133Each `<test/>` element has the following supported attributes: 134 135| Attribute | Value | Purpose | Default | 136|-----------|---------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------| 137| packed | The name of the packed test image | The image uploaded to the device. It is searched for in dir if --search_path=dir | Required | 138| unpacked | The name of the directory containing the unpacked version of packed | Searched for in dir if --search_path=dir. This folder should have the all the images that packed unpacks to. The name of each of the images should be the name of the real partition it is flashed to. | Required if expect != "fail" | 139| expect | "okay" or "fail" | If uploading a invalid or garbage image the bootloader should reject use "fail" otherwise "okay" | "okay" | 140 141 142#### `<oem>` Element 143Vendors can extend the fastboot protocol with oem commands. This allows vendors 144to support device/vendor specific features/commands over the fastboot protocol. 145Fuzzy Fastboot allows testing these oem commands as well. 146 147Oem commands are specefied in `<command></command>` element pairs. Each command 148element supports the following attributes: 149 150 151| Attribute | Value | Purpose | Default | 152|-------------|----------------------|---------------------------------------------------------------|----------| 153| value | The oem command name | Ex: if value="foo", the oem command will start with "oem foo" | Required | 154| permissions | "none" or "unlocked" | Whether the bootloader must be "unlocked" to perform command | "none" | 155 156An example is should below: 157```xml 158<oem> 159 <command value="self_destruct" permissions="unlocked"> 160 <!-- This will test that "oem self_destruct now" returns 'okay' --> 161 <test value="now" expect="okay"/> 162 <!-- This will test that "oem self_destruct yesterday" returns 'fail' --> 163 <test value="yesterday" expect="fail" /> 164 </command> 165 166 <command value="foobar" permissions="unlocked"> 167 <!-- FF will first stage test_image.img before running 'oem foobar use_staged' --> 168 <test value="use_staged" expect="okay" input="test_image.img" /> 169 <!-- FF will run 'oem foobar send_response', upload data from device, then run the validator script --> 170 <test value="send_response" expect="fail" validate="python validator.py"/> 171 </command> 172<oem/> 173``` 174 175Again you will notice that one can, and should, specify tests to run with `<test/>` elements. 176The test elements support the following attributes: 177 178 179| Attribute | Value | Purpose | Default | 180|-----------|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------| 181| value | The oem command argument | Ex: if value="bar", and the oem command name was "foo", the full command will be "oem foo bar" | Empty String (no argument) | 182| expect | "okay" or "fail" | Whether the bootloader should accept or reject this command | "okay" | 183| input | A image filename | Some oem commands require staging files before the command is executed | Empty String (no argument) | 184| validate | A program/script to run to validate the response | Some oem commands will stage data that can be downloaded afterwards and should be validated to be correct. Fuzzy Fastboot will launch the validation program with the first arg the oem command executed, the second arg the path to the downloaded image. Ex: "python validate.py'. If the program has a non-zero return code, the validation is marked as failed and anything from the launched programs stderr is logged in the test failure. | Empty String (no argument) | 185| assert | A Regular expression | In the "okay" or "fail" response, Fuzzy Fastboot will assert the response matches this regular expression. | Empty String (no argument) | 186| output | The name of the saved file | This is the name of the saved output file passed to the validation script. It is saved in whatever location is specified by the --output_path argument | out.img | 187 188 189#### `<checksum/>` Element 190If the bootloader supports hashing partitions (implementing this is strongly recommended), Fuzzy Fastboot can 191use it to do a bunch more testing. Make sure this hash is a cryptographically secure hash, as a non-secure one 192might reveal secrets about the partitions' contents. 193 194The checksum element has no children and only two required attributes: 195 196- **value** - The command that hashes a partition. Note that the name of the partition will be appended to the end of the command. For example, if `value="oem hash"`, hashing the partition `bar` would be issued with `oem hash bar`. 197- **parser** - How the hash is returned is up to the vendor's implementation. It could be part of the `OKAY` response, or be encoded in `INFO` responses. Thus, the parser attribute is used to specify a program/script that will extract the hash. The first argument to the program will be the be the response from `OKAY`, the second argument will be all the `INFO` responses joined by newlines. The extracted hash should be sent back to Fuzzy Fastboot as a string written to stderr, and a return code of 0 to signal the parsing was successful. In the case of failure, return a non-zero return code, an optionally an associated error message written to stderr. 198 199 200 201## Full Example XML Configuration 202Here is a basic example configuration. This can also be found in the 'example' folder 203as well as the associated python scripts 'checksum_parser.py' (used to extract partition hash), 204and 'validator.py' (used to validate an oem command that returns data). 205```xml 206<?xml version="1.0"?> 207<config> 208<!-- All the device getvar variables should be listed here --> 209<getvar> 210 <var key="product" assert="superphone2000"/> 211 <var key="secure" assert="no|yes"/> 212</getvar> 213 214<!-- All the device partitions should be listed here --> 215<partitions> 216 <part value="boot" slots="yes" test="yes" hashable="yes" parsed="yes"/> 217 <part value="modem" slots="yes" test="yes" hashable="yes"/> 218 <part value="userdata" slots="no" test="yes" hashable="no"/> 219 220 <!-- Bootloader partitions --> 221 <part value="foo1" slots="yes" test="no" hashable="yes"/> 222 <part value="foo2" slots="yes" test="no" hashable="yes"/> 223 <part value="bar3" slots="yes" test="no" hashable="yes"/> 224</partitions> 225 226<!-- All the device packed partitions should be listed here --> 227<packed> 228 <part value="bootloader" slots="yes"> 229 <!-- We list the real partitions it is composed of --> 230 <child>foo1</child> 231 <child>foo2</child> 232 <child>bar3</child> 233 <!-- We list tests, expect defaults to 'okay' --> 234 <test packed="bootloader.img" unpacked="unpacked"/> 235 <test packed="bootloader_garbage.img" expect="fail"/> 236 </part> 237</packed> 238 239<!-- All the oem commands should be listed here --> 240<oem> 241 <!-- The 'oem self_destruct' command requires an unlocked bootloader --> 242 <command value="self_destruct" permissions="unlocked"> 243 <!-- This will test that "oem self_destruct now" returns 'okay' --> 244 <test value="now" expect="okay"/> 245 <test value="yesterday" expect="fail" /> 246 </command> 247 248 <!-- Test a fictional 'oem get' command --> 249 <command value="get" permissions="none"> 250 <test value="batch_id" expect="okay" assert="[[:digit:]]+"/> 251 <test value="device_color" expect="okay" assert="green|blue"/> 252 <test value="build_num" expect="okay" assert="[\w\-.]+"/> 253 <test value="garbage" expect="fail" assert="Invalid var '[\w ]+'"/> 254 </command> 255 256 <!-- Some oem commands might require staging or downloading data, or both --> 257 <command value="foobar" permissions="unlocked"> 258 <!-- FF will first stage test_image.img before running 'oem foobar use_staged' --> 259 <test value="use_staged" expect="okay" input="test_image.img" /> 260 <!-- FF will run 'oem foobar send_response', upload data from device, then run the validator script --> 261 <test value="send_response" expect="fail" validate="python validator.py"/> 262 </command> 263</oem> 264 265<!-- If there is a custom oem checksum command to hash partitions, add it here --> 266<checksum value="oem sha1sum"/> 267</config> 268 269``` 270 271## Running Fuzzy Fastboot 272Fuzzy Fastboot is built with the fastboot tool itself. It will appear in `out/host/linux-x86/testcases/fuzzy_fastboot/x86_64`. 273 274### Command Line Arguments 275- **--config=**: Specify the name of the configuration XML file. If omitted, only the generic tests will be available. 276- **--search_path=**: Specify the path where Fuzzy Fastboot will look for files referenced in the XML. This includes all the test images and the referenced programs/scripts. This is also where the --config is searched for. If this argument is omitted it defaults to the current directory. 277- **--output_path**: Some oem tests can download an image to the host for validation. This is the location where that image is stored. This deafults to '/tmp'. 278- **--serial_port**: Many devices have a UART or serial log, that reports logging information. Fuzzy Fastboot can include this logging information in the backtraces it generates. This can make debugging far easier. If your device has this, it can be specified with the path to the tty device. Ex: "/dev/ttyUSB0". 279- **--gtest_***: Any valid gtest argument (they all start with 'gtest_') 280- **-h**: Print gtest's help message 281 282 283## Using Fuzzy Fastboot on my Device 284All Fuzzy Fastboot tests should pass on your device. No test should be able to 285crash the bootloader. Invalid input MUST be handled gracefully. Using "asserts" 286or panicking on invalid or malformed input is not an acceptable way to handle 287these tests, as ungraceful forced termination of the bootloader can expose 288vulnerabilities and leave the device in a bad state. 289 290The following is the recommended workflow for using Fuzzy Fastboot on a new device: 291 292### Step 1: Pass the generic Conformance tests 293Begin with just the generic tests (i.e. no XML file). In particular, make sure all 294the conformance tests are passing before you move on. All other tests require that 295the basic generic conformance tests all pass for them to be valid. The conformance 296tests can be run with `./fuzzy_fastboot --gtests_filter=Conformance.*`. 297 298#### Understanding and Fixing Failed Tests 299Whenever a test fails, it will print out to the console the reason for failure 300and the lines and file where the error happened. At the end of each failure 301block, there will usually be a message that Fuzzy Fastboot reports to gtest 302explaining what went wrong. An example is shown below: 303 304``` 305Expected equality of these values: 306 resp 307 Which is: "no" 308 unlock ? "yes" : "no" 309 Which is: "yes" 310getvar:unlocked response was not 'no' or 'yes': no 311system/core/fastboot/fuzzy_fastboot/fixtures.cpp:227: Failure 312Expected: SetLockState(UNLOCKED) doesn't generate new fatal failures in the current thread. 313 Actual: it does. 314[THERE WILL BE A MESSAGE HERE EXPLAINING WHY IT FAILED] 315``` 316 317In most cases this message at the bottom is all that is needed to figure out why it failed. 318If this is not enough information, below this, gtest will also print out a human readable 319backtrace of the underlying fastboot commands leading up the failure in this test. 320Here is an example: 321``` 322<<<<<<<< TRACE BEGIN >>>>>>>>> 323[WRITE 0ms](15 bytes): "getvar:unlocked" 324[READ 20ms](6 bytes): "OKAYno" 325<<<<<<<< TRACE END >>>>>>>>> 326``` 327One can easily see the reason for the failure was the test expected the device to 328be unlocked. 329 330If it is still unclear why the failure is happening, the last thing to do is look 331at what line number and file is generating the error. Gtest will always print this out. 332You can then manually look through Fuzzy Fastboot's test source code, and figure out 333what went wrong. 334 335 336### Step 2: Pass all the other generic tests 337Run all the other generic tests (still no XML file). A list of all of them can be 338printed out with: "./fuzzy_fastboot --gtest_list_tests". As before, "--gtest_filter" 339can be used to select certain tests to run, once you figure out which ones failed. 340 341One particular set of tests to watch out for are the ones that involve USB resets. 342USB resets effectively unplug and replug the device in software. Fuzzy Fastboot, 343expects USB resets to cancel whatever transaction is currently going on. 344This is also how Fuzzy Fastboot attempts to recover from errors when the device is 345unresponsive. 346 347### Step 3: Create a device XML configuration 348Without a device specific configuration file, Fuzzy Fastboot will have poor test 349coverage of your device. The vast majority of tests are auto-generated via the XML 350configuration file. Use the guide above to generate a configuration for your device. 351Make sure to be as thorough as possible, and list everything in the configuration 352that can be tested. Finally, make sure that the packed pseudo partitions and 353oem commands all have provided test cases. Be sure to test both the positive case 354(i.e. with valid input), as well as the opposite. Make sure the failure tests 355have good coverage by thinking about all the ways invalid and malicious inputs 356could be formed. These means creating images with malformed headers, illegal chars, 357and other evil inputs. 358 359Now run fuzzy_fastboot with the supplied configuration file. If you do "--gtest_list_tests", 360you should see a ton more tests that were autogenerated by Fuzzy Fastboot. 361As before, run these tests till everything passes. Again, use "--gtest_filter" 362to select specific tests to run once you know what fail, 363as running the whole things with a large configuration can take over 30 minutes. 364See the gtest documentation, for nifty tricks and command line options. 365 366### Step 4: Figure out what Fuzzy Fastboot can't/isn't testing 367While Fuzzy Fastboot with a XML configuration file, should provide good test coverage. 368Think about what device specific things are not being tested, and test them manually. 369In particular, things that if not handled carefully could create security exploits. 370Don't be lazy here, as you already put in the time to get this far. 371 372### Step 5: Celebrate 373You're done :). Now you can be more confident that your implementation is sound, and 374have piece of mind knowing you are protecting the users' security and data by 375running these tests. Don't get too complacent. If the bootloader's source code 376is modified in a way that could introduce bugs or security issues. Make sure to 377test again. You might have to add to your existing configuration file. 378 379## Limitations and Warnings 380- Currently this only works on Linux (even if it builds on Mac) 381- Only fastboot over USB is currently supported 382- Passing these tests does not mean there are not bugs/security issues. For example, a buffer overrun might not always trigger a crash or have any noticeable side effects. 383- **Be extremely careful of the Fuzzy Fastboot tests you are running. Know exactly what the tests do you are about to run before you run them. It is very possible to brick a device with many of these tests.** 384 385## Fuzzy Fastboot Missing Features TODO's 386The following are missing features that should eventually be added 387- *Sparse Image Tests*: Currently there are no tests that tests sparse images. Both well-formed and malicious images need to be tested. 388- *Unlocking/Locking Critical*: Currently there are no tests that tests that locking/unlocking critical functionality. 389- *Saved Test Log*: Fuzzy Fastboot should be able to create a failure log for every failing test and save it to a file. This file should include the test information, the reason it failed, and the fastboot command trace (with the serial console logs). Currently it just prints it to the console at the end of every test. 390- *Host Side Hashing*: One should be able to provide the hashing algorithm to the Fuzzy Fastboot, so it can be checked to agree with what the device is reporting. 391 392 393## Author 394Aaron Wisner - awisner@google.com 395