1<a name="faft" /> 2 3# FAFT (Fully Automated Firmware Test) 4 5_Last updated: 2011/11/08_ 6 7- [FAFT (Fully Automated Firmware Test)](#faft) 8 - [FAFT Related Documents](#faft-related-documents) 9 - [Introduction](#introduction) 10 - [Backgrounds](#backgrounds) 11 - [State of the Art](#state-of-the-art) 12 - [VBoot_Reference Library Tests](#vboot-reference-library-tests) 13 - [U-Boot vbexport/vboot Tests](#u-boot-vbexport-vboot-tests) 14 - [SAFT (Semi Automated Firmware Test)](#saft-semi-automated-firmware-test) 15 - [Factory Regression Testing Using Servo](#factory-regression-testing-using-servo) 16 - [U-Boot Test System ](#u-boot-test-system) 17 - [FAFT Implementation Details](#faft-implementation-details) 18 - [Test Environment](#test-environment) 19 - [Test Harness](#test-harness) 20 - [Software Architecture](#software-architecture) 21 - [Test Case Example](#test-case-example) 22 - [FAFT Scopes](#faft-scopes) 23 - [Scope 1: Enhance SAFT in Automated Way on Autotest Harness](#scope-1) 24 - [Scope 2: Test the Layer of VBExport APIs](#scope-2) 25 - [Scope 3: Verify Recovery, Factory Install, Firmware AU, RMA, and Many Complicated Flows](#scope-3) 26 - [Scope 4: Monkey Test during Firmware/EC update](#scope-4) 27 - [Other Ideas](#other-ideas) 28 29<a name="faft-related-documents" /> 30 31## FAFT Related Documents 32 33- FAFT Setup Instructions: [http://goto/faft-setup](http://goto/faft-setup) 34- FAFT Test Coverage: [http://goto/faft-coverage](http://goto/faft-coverage) 35- FAFT Comparing Existing Manual Test Cases: [http://goto/faft-comparison](http://goto/faft-comparison) 36 37<a name="introduction" /> 38 39## Introduction 40 41Security is one of the selling points of Chrome OS netbooks. Verified boot provides a solution to Chrome OS security model. In the verified boot design, firmware is the foundation of this secured castle. In addition, part of the firmware is marked as read-only, that means we are unable to patch security holes via autoupdate. So firmware testing is very important; however, our current firmware is lack of automated tests. It only relies on the manual tests by developers, our test team, and dogfooders. It seems to be a risk. 42 43This document proposes a solution to the test problem. The advantages of FAFT: 44- Fully automatic, no human involved; 45- Real test of physical hardware, like USB plug-in, Ctrl-D key press; 46- High test coverage of complicated verified boot flows; 47- Easy to integrate to our existing test infrastructure, i.e. deploy to test lab, run remotely, continue test, etc. 48 49<a name="backgrounds" /> 50 51## Backgrounds 52 53This document uses [U-Boot](http://en.wikipedia.org/wiki/Das_U-Boot) as an example to discuss the FAFT test approach. It can be also applied to [coreboot](http://en.wikipedia.org/wiki/Coreboot) and any proprietary BIOS in a similar way. 54 55The U-Boot software stack looks like the following graph: [cl/70339149](https://critique.corp.google.com/#review/70339149) 56 57 58 59Most of the verified boot logic and crypto algorithms are inside the vboot_reference library. The main control logic in U-Boot is simple, just like reading some headers from SPI and filling them in a data structure which is then passed to vboot_reference. 60 61The underlying U-Boot drivers are abstracted into VBoot Export APIs and some (not too many) APIs which the main control logic is used. VBoot export APIs are the most, e.g., TPM, display, keyboard, disk, system related, etc. The main control logic used APIs are not many, e.g., SPI, GPIO, etc. The latter APIs are not included in VBoot Export APIs because some x86 legacy reasons. They may be merged in the future. 62 63The firmware boot path is like the following graph: 64 65 66 67Our tests are mostly focused on VBoot Bootstub and VBoot Main Firmware since they are complicated and many decision branches. U-Boot initialization and kernel booting are simple and single path decisions. So full system regression tests are enough. 68 69<a name="state-of-the-art" /> 70 71## State of the Art 72 73<a name="vboot-reference-library-tests" /> 74 75### VBoot_Reference Library Tests 76 77 78Source: src/platform/vboot_reference/tests 79 80vboot_reference provides a lot of tests varying from crypto algorithms to vboot main logic and control flows. These tests stub the underlying APIs as all success functions. So it is unable to validate the correctness of underlying BIOS platform and the hardware behaviors. 81 82 83 84<a name="u-boot-vbexport-vboot-tests" /> 85 86### U-Boot vbexport/vboot Tests 87 88Source: src/third_party/u-boot/files/common/cmd_vb*_test.c 89 90Almost all the U-Boot drivers are abstracted into couple of major APIs: VBExport APIs and main logic used driver APIs. U-Boot provides a CLI for manual diagnostic. We wrote some commands to test these APIs. For example, for testing the disk read/write functions, a test command is to write some patterns to the disk and read them back to verify. However, some of these tests needs human involved, like inserting an USB stick to check if it is detectable. 91 92 93 94<a name="saft-semi-automated-firmware-test" /> 95 96### SAFT (Semi Automated Firmware Test) 97 98Source: src/platform/saft 99 100First of all, SAFT only works on x86 and hasn’t ported to ARM yet. 101 102SAFT assumes the whole firmware as a black-box and tests the whole system from firmware, kernel, to rootfs. For example, it corrupts firmware RW A/B to see if it enters recovery mode success. However, SAFT also needs human involved in many cases. 103 104 105 106<a name="factory-regression-testing-using-servo" /> 107 108### Factory Regression Testing Using Servo 109 110Design Doc: [https://docs.google.com/a/google.com/document/d/1XZgX4_v-Ps7YBRnjZmCFYjrjyBZQz5YmRrg7YWBFfV0/edit?hl=en_US](https://docs.google.com/a/google.com/document/d/1XZgX4_v-Ps7YBRnjZmCFYjrjyBZQz5YmRrg7YWBFfV0/edit?hl=en_US) 111 112This work is an on-going project to ensure the whole factory install flow work fine. It downloads a factory bundle from buildbot, setup TFTP and Omaha servers. It uses a mini-Servo board to control a Chrome OS device, like switching dev switch, resetting device, sending Ctrl-D key, etc. This work is done by Rong Chang and the factory team to ensure partners get a good factory bundle. 113 114Since using a Servo board can automate almost all the human behaviors. The test can be done fully automated. The proposed FAFT in this document is an extension of this work. 115 116<a name="u-boot-test-system" /> 117 118### U-Boot Test System 119 120Design Doc: [https://docs.google.com/a/google.com/document/d/1JorndqNOrEOXgdMRuNBPk1o73XKl1JwCwLkDSOQy7wU/edit?hl=en_US](https://docs.google.com/a/google.com/document/d/1JorndqNOrEOXgdMRuNBPk1o73XKl1JwCwLkDSOQy7wU/edit?hl=en_US) 121 122Simon proposed a U-Boot native test system to split the U-Boot components into platform code and generic U-Boot code + Verified Bood code. By mocking the platform code. we can test the functions of VBoot library and it integration in a platform-independent way. On the other side, the architecture-specific platform code can be tested in a separated way. 123 124<a name="faft-implementation-details" /> 125 126## FAFT Implementation Details 127 128<a name="test-environment" /> 129 130### Test Environment 131 132We need a [Servo board](https://sites.google.com/a/google.com/chromeos-partner/hardware-control-and-debug/servo) to connect a DUT such that we can: 133 134- flash a new firmware image; 135- run regression tests; 136- simulate lid close, power cycles, gpios, and key presses; 137- get console output via UART; 138- simulate USB plug in/out, by either muxing a physical USB stick or emulating a USB storage gadget in the host machine. 139 140 141 142<a name="test-harness" /> 143 144### Test Harness 145 146We use Autotest as our test harness and run FAFT as server tests. The host uses Ethernet to connect a DUT to push the test commands to it. Since Autotest is widely used in the Chrome OS projects. We can benefit in: 147 148- easy to integrate to buildbot; 149- easy to run remotely; 150- easy to deploy to the test lab. 151 152Some existing works using Servo board test on top on Autotest can be found in: 153 154- src/third_party/autotest/files/server/site_tests/platform_CloseOpenLid 155- src/third_party/autotest/files/server/site_tests/platform_LongPressPower 156 157FAFT is also based on the existing ServoTest approach and does some enhancements. 158 159<a name="software-architecture" /> 160 161### Software Architecture 162 163Here is our proposed FAFT framework. The red pieces are missing and need to be implemented in this project. 164 165 166 167All the FAFT test cases are server side tests, meaning that all the actions are controlled by Host. It is flexible to keep states after sequential reboot. However, we still want to get the DUT state and run some command on it, like getting current firmware version, setting the fw_try_b flag, etc. These actions should be completed in the DUT. So we setup a FAFT Client for this purpose and use RPC to communicate with it. 168 169<a name="test-case-example" /> 170 171### Test Case Example 172 173Here is an example to demonstrate how easy it is to write a FAFT test case. It test the firmware like: 1741. FW A normal boot -> enable devsw + set try_fwb -> reboot -> press ctrl-d 1752. FW B developer boot -> do nothing -> reboot -> press ctrl-d 1763. FW A developer boot -> disable devsw -> reboot -> do nothing 1774. FW A normal boot -> finished 178 179``` 180class firmware_DevModeTryB(FAFTSequence): 181 def run_once(self, host=None): 182 self.register_faft_sequence(( 183 # ------------------------- step 1 ------------------------- 184 {‘state_checker’: (self.faft_util.crossystem_checker, 185 {‘devsw_boot’: ‘0’, ‘mainfw_type’: ‘normal’, ‘mainfw_act’: ‘A’}), 186 ‘userspace_action’: self.enable_devmode_and_set_try_fwb, 187 ‘firmware_action’: self.wait_and_press_ctrl_d}, 188 # ------------------------- step 2 ------------------------- 189 {‘state_checker’, (self.faft_util.crossystem_checker, 190 {‘devsw_boot’: ‘1’, ‘mainfw_type’: ‘developer’, ‘mainfw_act’: ‘B’}), 191 ‘userspace_action’: None, 192 ‘firmware_action’: self.wait_and_press_ctrl_d}, 193 # ------------------------- step 3 ------------------------- 194 {‘state_checker’, (self.faft_util.crossystem_checker, 195 {‘devsw_boot’: ‘1’, ‘mainfw_type’: ‘developer’, ‘mainfw_act’: ‘A’}), 196 ‘userspace_action’: self.servo.disable_development_mode, 197 ‘firmware_action’: None}, 198 # ------------------------- step 4 ------------------------- 199 {‘state_checker’, (self.faft_util.crossystem_checker, 200 {‘devsw_boot’: ‘0’, ‘mainfw_type’: ‘normal’, ‘mainfw_act’: ‘A’}), 201 } 202 )) 203 self.run_faft_sequence() 204 205 def enable_devmode_and_set_try_fwb(self): 206 self.faft_client.crossystem.set_try_fwb() 207 self.servo.enable_development_mode() 208 209 def wait_and_press_ctrl_d(self): 210 time.sleep(10) 211 self.servo.ctrl_d() 212``` 213 214<a name="faft-scopes" /> 215 216## FAFT Scopes 217 218There are several scopes about FAFT. 219 220<a name="scope-1" /> 221 222### Scope 1: Enhance SAFT in Automated Way on Autotest Harness 223 224SAFT provides a complete set of libraries to access system info and corrupt/restore firmware and kernel. It also has a set of test cases and expected results. For example, in the SAFT main program, `saft_utility.py`: 225 226``` 227TEST_STATE_SEQUENCE = ( 228 ('1:1:1:0:3', FST.set_try_fw_b), # Step 0 229 ('1:2:1:0:3', None), 230 ('1:1:1:0:3', FST.corrupt_firmware, 'a'), 231 ('1:2:1:0:3', FST.restore_firmware, 'a'), 232 ('1:1:1:0:3', FST.corrupt_firmware, ('a', 'b')), 233 ('5:0:1:1:3', FST.restore_firmware, ('a', 'b')), # Step 5 234 ('1:1:1:0:3', FST.corrupt_kernel, 'a'), 235 ('1:1:1:0:5', FST.corrupt_kernel, 'b'), 236 ('6:0:1:1:3', FST.restore_kernel, ('a', 'b')), 237 ('1:1:1:0:3', FST.request_recovery_boot), 238 ('8:0:1:1:3', FST.prepare_tpm_tests), # Step 10 239 ('1:1:1:0:5', FST.move_kernel_backward, 'b'), 240 ('6:0:1:1:3', FST.jump_kernels_forward), 241 ('1:1:1:0:3', FST.move_firmware_backward, 'a'), 242 ('1:2:1:0:3', FST.move_firmware_backward, 'b'), 243 ('5:0:1:1:3', FST.jump_firmwares_forward), # Step 15 244 ('1:1:1:0:3', FST.request_recovery_boot), 245 ('8:0:1:1:3', FST.terminate_tpm_tests), 246 ('*:*:*:*:*', cgpt_st.test_loop), 247 ('1:1:1:0:3', FST.revert_firmware), 248 ('1:1:1:0:3', None), # Step 20 249 ) 250``` 251 252It is good that we don’t need to reimplement the same work. FAFT still uses the original SAFT test cases and libraries in the first step and port them to Autotest harness. FAFT does a bit modification to makes all test cases automated by using the powerful Servo board functions. 253 254Current SAFT is a sequence of actions and expected results to test all firmware / kernel / recovery / tpm behaviors as a whole. SAFT does tests in this way because we want to simply the human actions, not easy to be confused. Since FAFT can simulate all human actions. We can separate the test sequence more isolated and one for a special test purpose. 255 256A demo of porting SAFT test cases on FAFT: [https://video.google.com/a/?pli=1#/Play/contentId=60bdda4552e4c6a2](https://video.google.com/a/?pli=1#/Play/contentId=60bdda4552e4c6a2) 257 258This demo presents the feasibility of the integration of SAFT library and FAFT framework. We still have a lot of work to make all the features completed. The tasks like: 259- Create a separated FAFT client and wrap all existing SAFT libraries into it; 260- Setup RPC to invoke FAFT client instead of using SSH tunnel like the demo; 261- Enhance the functions of FAFT server side utilities; 262- Make FAFT framework more generalized to fit any possible test cases; 263- Keep the state and backup firmware/kernel in the server side instead of USB; 264- Automatic setup DUT disk and USB image when given a buildbot image. 265 266<a name="scope-2" /> 267 268### Scope 2: Test the Layer of VBExport APIs 269 270In order to increase the test coverage and verify the underlying BIOS functions and drivers, a good entry is to re-run all existing vbexport / vboot tests in an automated way using our test framework!. See more detail in the State of the Art Section. We can catch bugs easily if any drivers changed or the BIOS source rebased. 271 272 273 274<a name="scope-3" /> 275 276### Scope 3: Verify Recovery, Factory Install, Firmware AU, RMA, and Many Complicated Flows 277 278The normal boot and developer boot can be verified by SAFT. However, we still missed the test of the whole Recovery process. The whole recovery sequence is complicated: 2791. DD the recovery shim image to a USB stick; 2802. Hold Recovery button and power on DUT; 2813. Press SPACE; 2824. Insert a recovery shim; 2835. Wait for the complete of recovery process; 2846. Reboot; 2857. Boot to normal Chrome OS. 286 287This sequence can be fully automated by FAFT. For example, in step 1, we require an USB stick plugged to the Servo board by default. We first mux it to connect to the host machine such that we can write the recovery shim image to it. In step 4, we then mux the USB stick to the DUT. Another approach is to emulate an USB storage gadget in the host machine, it would be more flexible to control its functionalities by software. 288 289Moreover, there are many recovery mode variants, like normal recovery/devloper recovery, physical recovery button/keyboard simulated button, etc. It’d be easy to verify all variants by writing different test cases. 290 291Besides, factory install shim, firmware AU, reset shim, RMA, etc, are more complicated and have many variants. Factory install process currently will be tested by an automated script written by the factory team. Firmware AU still relies on manual tests, see the [test matrix](https://docs.google.com/a/google.com/spreadsheet/ccc?key=0AoHv3bO2oLIjdGQ5cnJ6eWc1ZnpjYzlaWFA4T0xlVVE&hl=en_US&authkey=CJWF8-gP#gid=8). Reset shim and RMA has no test yet. Actually, all these complicated flows can be tested in FAFT. 292 293<a name="scope-4" /> 294 295### Scope 4: Monkey Test during Firmware/EC update 296 297Firmware/EC update is a risky thing and may brick the machine. For example, the bug [http://crosbug.com/p/4419](http://crosbug.com/p/4419) reported, during the EC update, a user presses Power Button at the same time that bricks the machine. It is because the powerd triggers the shutdown process immediately and the system goes to off, but it cannot be turned on again by power button (until a cold boot). Note that the EC is still in update mode now. 298 299 300To catch bugs like that, we need a monkey test during firmware/EC update, like randomly switching GPIO, pressing Power Button, closing LID, pressing keyboard, plugging USB/SD disk, etc. 301 302<a name="other-ideas" /> 303 304## Other Ideas 305 306- Run FAFT on every build, integrated with buildbot 307- Stress test to ensure the stability of hardware and drivers 308- Cold reboot test 309- Use HDMI capture device or camera + image recognition to verify the display output 310
