1# How to run FAFT (Fully Automated Firmware Test) {#faft-how-to-run} 2 3- [How to run FAFT (Fully Automated Firmware Test)](#faft-how-to-run) 4 - [FAFT Overview](#faft-overview) 5 - [Hardware Setup](#hardware-setup) 6 - [ServoV4 Type-A with servo micro](#servov4-typea-micro) 7 - [ServoV4 Type-C](#servov4-typec) 8 - [ServoV4 Type-C with servo micro](#servov4-typec-micro) 9 - [(Deprecated) ServoV2](#servov2-deprecated) 10 - [Installing Test Image onto USB Stick](#image-onto-usb) 11 - [Running Tests](#faft-running-tests) 12 - [Setup Confirmation](#setup-confirmation) 13 - [Sample Commands](#sample-commands) 14 - [Frequently Asked Questions (FAQ)](#faq) 15 16## FAFT Overview {#faft-overview} 17 18FAFT (Fully Automated Firmware Tests) is a collection of tests and related 19infrastructure that exercise and verify capabilities of Chrome OS. 20The features tested by FAFT are implemented through low-level software 21(firmware/BIOS) and hardware. FAFT evolved from SAFT 22(Semi-Automated Firmware Tests) and you can locate tests in the [FAFT suite] 23in the Autotest tree as directories with the prefix `firmware_`. 24 25The founding principles of FAFT are: 26 27- Fully automated, no human intervention required 28- Real test of physical hardware, like USB plug-in, Ctrl-D key press 29- High test coverage of complicated verified boot flows 30- Easy to integrate with existing test infrastructure (e.g. test lab, continuous testing, etc). 31 32To access some of these low-level capabilities, the tests require a 33[servo] board ([servo v2], [servo v4] with [servo micro] or [servo v4 Type-C]) 34 35The servo board is connected directly to the DUT (Device Under Test) to enable 36access to low-level hardware interfaces, as well as staging areas for backup 37software (on a USB drive). 38The tests may corrupt various states in the EC, firmware, and kernel to verify 39recovery processes. In these cases you can almost always use FAFT to restore 40the system to its original state. 41The FAFT suite of tests can be invoked locally or remotely. 42This document describes how to set up the local configuration only. 43 44The Chrome OS firmware controls, among other things, the initial setup of the 45system hardware during the boot process. They are necessarily complicated, 46providing reliability against various corruption scenarios and security to 47ensure trusted software is controlling the system. Currently, the purpose of 48FAFT is to exercise EC firmware and BIOS firmware functionality and performance. 49 50## Hardware Setup {#hardware-setup} 51 52### ServoV4 Type-A with Micro {#servov4-typea-micro} 53 54The hardware configuration for running FAFT on a servo v4 Type-A 55with servo micro includes: 56 57- A test controller (your host workstation with a working chroot environment) 58- The test device (a device / DUT that can boot Chrome OS) 59- A servo board 60- Related cables and components 61 - servo-micro cable 62 - USB type-A to USB micro cable for DUT connection (~ 2' in length) 63 - USB type-A to USB micro cable for test controller connection (~ 4' - 6' in length) 64 - Ethernet cable 65 - USB drive (flashed with the appropriate OS image) 66 67Figure 1 shows a diagram of how to connect the latest debug boards, 68servoV4 Type-A and servo micro, to the test controller, DUT, and network. 69It is important to ensure the DUT is powered off 70before plugging in cables and components to the servo. 71 72Note: Do not use a type C servo 73(but using a type A servo v4 with a converter to type C is fine. 74A pure type C setup will put the security chip into debug mode). 75 76 77 78**Figure 1.Diagram of hardware configuration for a ServoV4 Type-A with servo micro.** 79 80Details of servoV4 Type-A with micro connections: 81 821. Connect one end (micro USB) of the servo micro to servoV4 using a micro USB to USB cable. 832. Connect the servo micro to the debug header on the chrome device. 843. Connect the USB type A cable of the servoV4 to the DUT. 854. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown in the diagram. 865. Connect the micro USB port of the servo to the host machine (typically your workstation). 876. Connect an Ethernet cable to the Ethernet jack of the servo that goes to the a network reachable from the network that your host machine is on. 88 89### ServoV4 Type-C {#servov4-typec} 90 91The hardware configuration for running FAFT with a servo v4 type-C includes: 92 93- A test controller (your host workstation with a working chroot environment) 94- The test device (a device / DUT that can boot Chrome OS) 95- A servo board 96- Related cables and components 97 - USB type-A to USB micro cable for test controller connection (~ 4' - 6' in length) 98 - Ethernet cable 99 - USB drive (flashed with the appropriate OS image) 100 101Figure 2 shows a diagram of how to connect a servoV4 Type-C, to the test 102controller, DUT, and network. It is important to ensure the DUT is powered off 103before plugging in cables and components to the servo and DUT. 104 105 106 107**Figure 2.Diagram of hardware configuration for a ServoV4 Type-C.** 108 109Details of servoV4 Type-C connections in Figure 2: 110 1111. Connect the USB Type-C cable of the servoV4 to the DUT. 1122. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown in the diagram. 1133. Connect the micro USB port of the servo to the host machine (typically your workstation). 1144. Connect an Ethernet cable to the Ethernet jack of the servo that goes to the a network reachable from the network that your host machine is on. 115 116### ServoV4 Type-C with servo micro {#servov4-typec-micro} 117 118Make sure to use the following servo type and configuration 119for running the FAFT PD suite 120 121 122 123**Figure 3.Diagram of hardware configuration for a ServoV4 Type-C with servo micro.** 124 125Details about FAFT PD's ServoV4 Type-C + servo micro setup (Figure 3): 126 127- The suite should only be run on devices released in 2019 and forward. 128- The charger connected to the servo must have support for 5V, 12V, and 20V. 129- The servo v4 and servo micro cable must be updated to their latest FW: 130 - Servo_v4: servo_v4_v2.3.30-b35860984 131 - servo micro: servo_micro_v2.3.30-b35960984 132 133To check or upgrade the FW on the servo v4 and servo micro, respectively, before kicking off the FAFT PD suite: 134 135- Have the servo v4 connected to your workstation/labstation along with the servo micro connected to the servo. 136- Run the following commands on chroot one after the other: 137 - sudo servo_updater -b servo_v4 138 - sudo servo_updater -b servo_micro 139 140### (Deprecated) ServoV2 {#servov2-deprecated} 141 142(Deprecated) The following photo shows the details how to connect the older, 143deprecated servo v2 board to the test controller, test device, and network. 144 145 146 147**Figure 4.Diagram of hardware configuration for a ServoV2 board.** 148 149Details of servo v2 connections: 150 1511. Connect one end(ribbon cable) of the flex cable to servoV2 and the other end to the debug header on the chrome device. 1522. Connect DUT_HUB_IN(micro USB port) of the servo to the DUT. 1533. Prepare a USB flash drive with valid Chrome OS image and plug into the USB port of the servo as shown in the photo. 1544. Connect the micro USB port of the servo to the host machine(workstation or a labstation). 1555. Connect an Ethernet cable to the Ethernet jack of the servo. 156 157### Installing Test Image onto USB Stick {#image-onto-usb} 158 159After the hardware components are correctly connected, 160prepare and install a test Chromium OS image: 161 1621. Build the binary (chromiumos_test_image.bin) with build_image test, or fetch the file from a buildbot. 1632. Load the test image onto a USB drive (use cros flash). 1643. Insert the USB drive into the servo board, as shown in the photo. 1654. Install the test image onto the internal disk by booting from the USB drive and running chromeos-install. 166 167## Running Tests {#faft-running-tests} 168 169### Setup Confirmation {#setup-confirmation} 170 171To run FAFT you use the test_that tool, which does not automatically start a 172servod process for communicating with the servo board. Before running any tests, go into chroot: 173 1741. (chroot 1) Run `$ sudo servod --board=$BOARD --no-ns-pid` where `$BOARD` is the code name of the board you are testing. For example: `$ sudo servod --board=eve` 1751. Go into a second chroot 1761. (chroot 2) Run the `firmware_FAFTSetup` test to verify basic functionality and ensure that your setup is correct. 1771. If test_that is in `/usr/bin`, the syntax is `$ /usr/bin/test_that --board=$BOARD $DUT_IP firmware_FAFTSetup` 178 179It is important to note that this syntax will work only if the correct packages 180for the DUT have been built. To build the packages, which usually takes 181a few hours, run the following from chroot: 182 183`$ ./build_packages --board=$BOARD` where `$BOARD` is the code name of the board under test 184 185If packages have not been built, the command won't work unless a path to the 186autotest directory is included in the command as follows: 187 188`$ test_that --autotest_dir ~/trunk/src/third_party/autotest/files/ --args="servo_host=localhost servo_port=9999" -b $BOARD $IP $TEST_NAME` 189 190### Sample Commands {#sample-commands} 191 192A few sample invocations of launching tests against a DUT: 193 194Running FAFT test with test case name 195 196- `$ /usr/bin/test_that --board=$BOARD $DUT_IP f:.*DevMode/control` 197 198Some tests can be run in either normal mode or dev mode, specify the control file 199 200- `$ /usr/bin/test_that --board=$BOARD $DUT_IP f:.*TryFwB/control.dev` 201 202FAFT can install Chrome OS image from the USB when image filename is specified 203 204- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "image=$IMAGE_FILE" f:.*RecoveryButton/control.normal` 205 206To update the firmware using the shellball in the image, specify the argument firmware_update=1 207 208- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "image=$IMAGE_FILE firmware_update=1" f:.*RecoveryButton/control.normal` 209 210Run the entire faft_bios suite 211 212- `$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_bios` 213 214Run the entire faft_ec suite 215 216- `$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_ec` 217 218Run the entire faft_pd suite 219 220- `$ /usr/bin/test_that --board=$BOARD $DUT_IP suite:faft_pd` 221 222To run servod in a different host, specify the servo_host and servo_port arguments. 223 224- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "servo_host=$SERVO_HOST servo_port=$SERVO_PORT" suite:faft_lv1` 225 226To run multiple servo boards on the same servo host (labstation), use serial and port number. 227 228- `$ sudo servod --board=$BOARD --port $port_number --serial $servo_serial_number` 229- `$ /usr/bin/test_that --board=$BOARD $DUT_IP --args "servo_host=localhost servo_port=$port_number faft_iterations=5000" f:.*firmware_ConsecutiveBoot/control` 230 231## Frequently Asked Questions (FAQ) {#faq} 232 233Q: All of my FAFT tests are failing. What should I do? 234 235- A1: Run `firmware_FAFTSetup` as a single test. Once it fails, check the log and determine which step failed and why. 236- A2: Check that the servo has all the wired connections and a USB drive with the valid OS plugged in. A missing USB drive is guaranteed to make `firmware_FAFTSetup` fail. 237 238Q: A few of my FAFT tests failed, but most tests are passing. What should I do? 239 240- A1: Re-run the failed tests and try to isolate between flaky infrastructure, an actual firmware bug, or non-firmware bugs. 241- A2: See if you were running FAFT without the AC charger connected. The DUT's battery may have completely drained during the middle of the FAFT suite. 242 243Q: I still need help. Who can help me? 244 245- A: Try joining the [FAFT-users chromium.org mailing list](https://groups.google.com/a/chromium.org/forum/#!forum/faft-users) and asking for help. Be sure to include logs and test details in your request for help. 246 247Q: I got an error while running FAFT: `AutoservRunError: command execution error: sudo -n which flash_ec` . What's wrong? 248 249- A: Run `sudo emerge chromeos-ec` inside your chroot. 250 251[FAFT suite]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/server/site_tests/ 252[servo]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/README.md#Power-Measurement 253[servo v2]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_v2.md 254[servo v4]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_v4.md 255[servo micro]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_micro.md 256[servo v4 Type-C]: https://chromium.googlesource.com/chromiumos/third_party/hdctools/+/refs/heads/master/docs/servo_v4.md#Type_C-Version 257
