1# How to run FAFT (Fully Automated Firmware Test) {#faft-how-to-run} 2 3_[go/faft-running](https://goto.google.com/faft-running)_ 4 5- [How to run FAFT (Fully Automated Firmware Test)](#faft-how-to-run) 6 - [FAFT Overview](#faft-overview) 7 - [Hardware Setup](#hardware-setup) 8 - [ServoV4 Type-A with servo micro](#servov4-typea-micro) 9 - [ServoV4 Type-C](#servov4-typec) 10 - [ServoV4 Type-C with servo micro](#servov4-typec-micro) 11 - [(Deprecated) ServoV2](#servov2-deprecated) 12 - [Installing Test Image onto USB Stick](#image-onto-usb) 13 - [Running Tests](#faft-running-tests) 14 - [Setup Confirmation](#setup-confirmation) 15 - [Sample Commands](#sample-commands) 16 - [Frequently Asked Questions (FAQ)](#faq) 17 18## FAFT Overview {#faft-overview} 19 20[FAFT] (Fully Automated Firmware Tests) is a collection of tests and related 21infrastructure that exercise and verify capabilities of Chrome OS. 22The features tested by FAFT are implemented through low-level software 23(firmware/BIOS) and hardware. FAFT evolved from SAFT 24(Semi-Automated Firmware Tests) and you can locate tests in the [FAFT suite] 25in the Autotest tree as directories with the prefix `firmware_`. 26 27The founding principles of FAFT are: 28 29- Fully automated, no human intervention required 30- Real test of physical hardware, like USB plug-in, Ctrl-D key press 31- High test coverage of complicated verified boot flows 32- Easy to integrate with existing test infrastructure (e.g. test lab, continuous testing, etc). 33 34To access some of these low-level capabilities, the tests require a 35[servod] instance running and executing controls with the help of physical 36[servo] board ([servo v2], [servo v4] with [servo micro] or [servo v4 Type-C]) 37 38The servo board is connected directly to the DUT (Device Under Test) to enable 39access to low-level hardware interfaces, as well as staging areas for backup 40software (on a USB drive). 41 42The [FAFT framework] runs the tests with a tool called [test that] and it is 43based on a client-server architecture, where the client runs on the DUT and 44the server runs on the host machine. 45 46The tests may corrupt various states in the EC, firmware, and kernel to verify 47recovery processes. In these cases you can almost always use FAFT to restore 48the system to its original state. 49The FAFT suite of tests can be invoked locally or remotely. 50This document describes how to set up the local configuration only. 51 52The Chrome OS firmware controls, among other things, the initial setup of the 53system hardware during the boot process. They are necessarily complicated, 54providing reliability against various corruption scenarios and security to 55ensure trusted software is controlling the system. Currently, the purpose of 56FAFT is to exercise EC firmware and BIOS firmware functionality and performance. 57 58## Hardware Setup {#hardware-setup} 59 60### ServoV4 Type-A with Micro {#servov4-typea-micro} 61 62The hardware configuration for running FAFT on a servo v4 Type-A 63with servo micro includes: 64 65- A test controller (your host workstation with a working chroot environment) 66- The test device (a device / DUT that can boot Chrome OS) 67- A servo board 68- Related cables and components 69 - servo-micro cable 70 - USB type-A to USB micro cable for DUT connection (~ 2' in length) 71 - USB type-A to USB micro cable for test controller connection (~ 4' - 6' in length) 72 - Ethernet cable 73 - USB drive (flashed with the appropriate OS image) 74 75Figure 1 shows a diagram of how to connect the latest debug boards, 76servoV4 Type-A and servo micro, to the test controller, DUT, and network. 77It is important to ensure the DUT is powered off 78before plugging in cables and components to the servo. 79 80 81 82**Figure 1.Diagram of hardware configuration for a ServoV4 Type-A with servo micro.** 83 84Details of servoV4 Type-A with micro connections: 85 861. Connect one end (micro USB) of the servo micro to servoV4 using a micro USB to USB cable. 872. Connect the servo micro to the debug header on the chrome device. 883. Connect the USB type A cable of the servoV4 to the DUT. 894. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown in the diagram. 905. Connect the micro USB port of the servo to the host machine (typically your workstation). 916. Connect an Ethernet cable to the Ethernet jack of the servo that goes to the a network reachable from the network that your host machine is on. 92 93### ServoV4 Type-C {#servov4-typec} 94 95The hardware configuration for running FAFT with a servo v4 type-C includes: 96 97- A test controller (your host workstation with a working chroot environment) 98- The test device (a device / DUT that can boot Chrome OS) 99- A servo board 100- Related cables and components 101 - USB type-A to USB micro cable for test controller connection (~ 4' - 6' in length) 102 - Ethernet cable 103 - USB drive (flashed with the appropriate OS image) 104 105Figure 2 shows a diagram of how to connect a servoV4 Type-C, to the test 106controller, DUT, and network. It is important to ensure the DUT is powered off 107before plugging in cables and components to the servo and DUT. 108 109 110 111**Figure 2.Diagram of hardware configuration for a ServoV4 Type-C.** 112 113Details of servoV4 Type-C connections in Figure 2: 114 1151. Connect the USB Type-C cable of the servoV4 to the DUT. 1162. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown in the diagram. 1173. Connect the micro USB port of the servo to the host machine (typically your workstation). 1184. Connect an Ethernet cable to the Ethernet jack of the servo that goes to the a network reachable from the network that your host machine is on. 119 120### ServoV4 Type-C with servo micro {#servov4-typec-micro} 121 122Make sure to use the following servo type and configuration 123for running the faft_pd suite or the faft_cr50 suite (note: the cr50 suite 124requires special images so is not runnable outside of Google). This setup 125requires servod to be in "DUAL_V4" mode. You should generally only use this 126setup for faft_pd and faft_cr50, faft_ec and faft_bios do not expect servod to 127be in DUAL_V4 mode. 128 129 130 131**Figure 3.Diagram of hardware configuration for a ServoV4 Type-C with servo micro.** 132 133Details about FAFT PD's ServoV4 Type-C + servo micro setup (Figure 3): 134 135- The suite should only be run on devices released in 2019 and forward. 136- The charger connected to the servo must have support for 5V, 12V, and 20V. 137- The servo v4 and servo micro cable must be updated to their latest FW: 138 - Servo_v4: servo_v4_v2.3.30-b35860984 139 - servo micro: servo_micro_v2.3.30-b35960984 140 141To check or upgrade the FW on the servo v4 and servo micro, respectively, before kicking off the FAFT PD suite: 142 143- Have the servo v4 connected to your workstation/labstation along with the servo micro connected to the servo. 144- Run the following commands on chroot one after the other: 145 - sudo servo_updater -b servo_v4 146 - sudo servo_updater -b servo_micro 147 148### (Deprecated) ServoV2 {#servov2-deprecated} 149 150(Deprecated) The following photo shows the details how to connect the older, 151deprecated servo v2 board to the test controller, test device, and network. 152 153 154 155**Figure 4.Diagram of hardware configuration for a ServoV2 board.** 156 157Details of servo v2 connections: 158 1591. Connect one end(ribbon cable) of the flex cable to servoV2 and the other end to the debug header on the chrome device. 1602. Connect DUT_HUB_IN(micro USB port) of the servo to the DUT. 1613. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown in the photo. 1624. Connect the micro USB port of the servo to the host machine(workstation or a labstation). 1635. Connect an Ethernet cable to the Ethernet jack of the servo. 164 165### Installing Test Image onto USB Stick {#image-onto-usb} 166 167After the hardware components are correctly connected, 168prepare and install a test Chromium OS image: 169 1701. Build the binary (chromiumos_test_image.bin) with build_image test, or fetch the file from a buildbot. 1712. Load the test image onto a USB drive (use cros flash). 1723. Insert the USB drive into the servo board, as shown in the photo. 1734. Install the test image onto the internal disk by booting from the USB drive and running chromeos-install. 174 175## Running Tests {#faft-running-tests} 176 177### Setup Confirmation {#setup-confirmation} 178 179To run FAFT you use the `test_that` tool, which does not automatically start a 180`servod` process for communicating with the servo board. Running FAFT is easiest 181with `servod` and `test_that` running in separate terminals inside the SDK, 182using either multiple SDK instances (`cros_sdk --enter --no-ns-pid`) or a tool 183such as `screen` inside an SDK instance. Before running any tests, go into 184chroot: 185 1861. (chroot 1) Run `$ sudo servod --board=$BOARD` where `$BOARD` is the code name of the board you are testing. For example: `$ sudo servod --board=eve` 1871. Go into a second chroot 1881. (chroot 2) Run the `firmware_FAFTSetup` test to verify basic functionality and ensure that your setup is correct. 1891. If test_that is in `/usr/bin`, the syntax is `$ /usr/bin/test_that --board=$BOARD $DUT_IP firmware_FAFTSetup` 190 191It is important to note that this syntax will work only if the correct packages 192for the DUT have been built. To build the packages, which usually takes 193a few hours, run the following from chroot: 194 195(chroot) `$ ./build_packages --board=$BOARD` where `$BOARD` is the code name of the board under test 196 197If packages have not been built, the command won't work unless a path to the 198autotest directory is included in the command as follows: 199 200(chroot) `$ test_that --autotest_dir ~/trunk/src/third_party/autotest/files/ --args="servo_host=localhost servo_port=9999" -b $BOARD $IP $TEST_NAME` 201 202### Sample Commands {#sample-commands} 203 204A few sample invocations of launching tests against a DUT: 205 206Running FAFT test with test case name 207 208- `$ /usr/bin/test_that --board=$BOARD $DUT_IP f:.*DevMode/control` 209 210Some tests can be run in either normal mode or dev mode, specify the control file 211 212- `$ /usr/bin/test_that --board=$BOARD $DUT_IP f:.*TryFwB/control.dev` 213 214FAFT can install Chrome OS image from the USB when image filename is specified 215 216- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "image=$IMAGE_FILE" f:.*RecoveryButton/control.normal` 217 218To update the firmware using the shellball in the image, specify the argument firmware_update=1 219 220- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "image=$IMAGE_FILE firmware_update=1" f:.*RecoveryButton/control.normal` 221 222Run the entire faft_bios suite 223 224- `$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_bios` 225 226Run the entire faft_ec suite 227 228- `$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_ec` 229 230Run the entire faft_pd suite 231 232- `$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_pd` 233 234To run servod in a different host, specify the servo_host and servo_port arguments. 235 236- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "servo_host=$SERVO_HOST servo_port=$SERVO_PORT" suite:faft_lv1` 237 238To run multiple servo boards on the same servo host (labstation), use serial and port number. 239 240- `$ sudo servod --board=$BOARD --port $port_number --serial $servo_serial_number` 241- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "servo_host=localhost servo_port=$port_number faft_iterations=5000" f:.*firmware_ConsecutiveBoot/control` 242 243## Frequently Asked Questions (FAQ) {#faq} 244 245Q: All of my FAFT tests are failing. What should I do? 246 247- A1: Run `firmware_FAFTSetup` as a single test. Once it fails, check the log and determine which step failed and why. 248- A2: Check that the servo has all the wired connections and a USB drive with the valid OS plugged in. A missing USB drive is guaranteed to make `firmware_FAFTSetup` fail. 249 250Q: A few of my FAFT tests failed, but most tests are passing. What should I do? 251 252- A1: Re-run the failed tests and try to isolate between flaky infrastructure, an actual firmware bug, or non-firmware bugs. 253- A2: See if you were running FAFT without the AC charger connected. The DUT's battery may have completely drained during the middle of the FAFT suite. 254 255Q: I still need help. Who can help me? 256 257- A: Try joining the [FAFT-users chromium.org mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/faft-users) and asking for help. Be sure to include logs and test details in your request for help. 258 259Q: I got an error while running FAFT: `AutoservRunError: command execution error: sudo -n which flash_ec` . What's wrong? 260 261- A: Run `sudo emerge chromeos-ec` inside your chroot. 262 263Q: All tests are failing to run, saying that python was not found. 264 What's wrong? 265 266- A: This happens when the stateful partition that holds Python is wiped by a 267 powerwash. 268 269 It is usually caused by the stateful filesystem becoming corrupted, since 270 Chrome OS performs a powerwash instead of running `fsck` like a standard 271 Linux distribution would. 272 273Q: What causes filesystem corruption? 274 275- A1: Most cases of corruption are triggered by a test performing an EC reset, 276 because the current sync logic in Autotest doesn't fully guarantee that all 277 writes have been completed, especially on USB storage devices. 278 279- A2: If the outer stateful partition (`/mnt/stateful_partition`) becomes full, 280 the inner loop-mounted DM device (`/mnt/stateful_partition/encrypted`) 281 will encounter write errors, likely corrupting the filesystem. 282 283 Note: Running out of space only tends to happens when running FAFT tests that 284 leave the DUT running from the USB disk, and only if the image's 285 [stateful partition is too small]. 286 287Q: Can I compare the results obtained with a Type-C servo to those obtained with a Type-A servo + micro? 288 289- A: When running tests with a Type-C servo, it is recommended to to rerun a failure using the Type-A setup to do a fast check prior to digging deeper, i.e. before connecting a USB analyzer or probing the signals. 290 291[FAFT suite]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/server/site_tests/ 292[servo]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/README.md#Power-Measurement 293[servo v2]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_v2.md 294[servo v4]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_v4.md 295[servo micro]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_micro.md 296[servo v4 Type-C]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_v4.md#Type_C-Version 297[stateful partition is too small]: https://crrev.com/c/1935408 298[FAFT]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/docs/faft-design-doc.md 299[FAFT framework]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/docs/faft-code.md 300[servod]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servod.md 301[test that]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/docs/test-that.md 302
