Lines Matching +full:state +full:- +full:of +full:- +full:the +full:- +full:art
5 _Self-link: [go/faft-design-doc](https://goto.google.com/faft-design-doc)_
9 - [FAFT (Fully Automated Firmware Test)](#faft)
10 - [FAFT Related Documents](#faft-related-documents)
11 - [Introduction](#introduction)
12 - [Backgrounds](#backgrounds)
13 - [State of the Art](#state-of-the-art)
14 - [VBoot_Reference Library Tests](#vboot-reference-library-tests)
15 - [U-Boot vbexport/vboot Tests](#u-boot-vbexport-vboot-tests)
16 - [SAFT (Semi Automated Firmware Test)](#saft-semi-automated-firmware-test)
17 - [Factory Regression Testing Using Servo](#factory-regression-testing-using-servo)
18 - [U-Boot Test System ](#u-boot-test-system)
19 - [FAFT Implementation Details](#faft-implementation-details)
20 - [Test Environment](#test-environment)
21 - [Test Harness](#test-harness)
22 - [Software Architecture](#software-architecture)
23 - [Test Case Example](#test-case-example)
24 - [FAFT Scopes](#faft-scopes)
25 - [Scope 1: Enhance SAFT in Automated Way on Autotest Harness](#scope-1)
26 - [Scope 2: Test the Layer of VBExport APIs](#scope-2)
27 …- [Scope 3: Verify Recovery, Factory Install, Firmware AU, RMA, and Many Complicated Flows](#scope…
28 - [Scope 4: Monkey Test during Firmware/EC update](#scope-4)
29 - [Other Ideas](#other-ideas)
31 <a name="faft-related-documents" />
35 - [FAFT Setup Instructions](https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ref…
36 - FAFT Test Coverage (Google internal only): [http://goto/faft-coverage](http://goto/faft-coverage)
37 - FAFT Comparing Existing Manual Test Cases (Google internal only): [http://goto/faft-comparison](h…
43 …of the selling points of ChromeOS netbooks. Verified boot provides a solution to ChromeOS security…
45 This document proposes a solution to the test problem. The advantages of FAFT:
46 - Fully automatic, no human involved;
47 - Real test of physical hardware, like USB plug-in, Ctrl-D key press;
48 - High test coverage of complicated verified boot flows;
49 - Easy to integrate to our existing test infrastructure, i.e. deploy to test lab, run remotely, con…
55 This document uses [U-Boot](http://en.wikipedia.org/wiki/Das_U-Boot) as an example to discuss the F…
57 The U-Boot software stack looks like the following graph:
59 
61 Most of the verified boot logic and crypto algorithms are inside the vboot_reference library. The m…
63 The underlying U-Boot drivers are abstracted into VBoot Export APIs and some (not too many) APIs wh…
65 The firmware boot path is like the following graph:
67 
69 …VBoot Main Firmware since they are complicated and many decision branches. U-Boot initialization a…
71 <a name="state-of-the-art" />
73 ## State of the Art
75 <a name="vboot-reference-library-tests" />
82 …of tests varying from crypto algorithms to vboot main logic and control flows. These tests stub th…
84 
86 <a name="u-boot-vbexport-vboot-tests" />
88 ### U-Boot vbexport/vboot Tests
90 Source: src/third_party/u-boot/files/common/cmd_vb*_test.c
92 …the U-Boot drivers are abstracted into couple of major APIs: VBExport APIs and main logic used dri…
94 
96 <a name="saft-semi-automated-firmware-test" />
102 First of all, SAFT only works on x86 and hasn’t ported to ARM yet.
104 SAFT assumes the whole firmware as a black-box and tests the whole system from firmware, kernel, to…
106 
108 <a name="factory-regression-testing-using-servo" />
112 …om/document/d/1XZgX4_v-Ps7YBRnjZmCFYjrjyBZQz5YmRrg7YWBFfV0/edit?hl=en_US](https://docs.google.com/…
114 …-going project to ensure the whole factory install flow work fine. It downloads a factory bundle f…
116 …automate almost all the human behaviors. The test can be done fully automated. The proposed FAFT i…
118 <a name="u-boot-test-system" />
120 ### U-Boot Test System
124 …-Boot native test system to split the U-Boot components into platform code and generic U-Boot code…
126 <a name="faft-implementation-details" />
130 <a name="test-environment" />
136 - flash a new firmware image;
137 - run regression tests;
138 - simulate lid close, power cycles, gpios, and key presses;
139 - get console output via UART;
140 - simulate USB plug in/out, by either muxing a physical USB stick or emulating a USB storage gadget…
142 
144 <a name="test-harness" />
148 …FAFT as server tests. The host uses Ethernet to connect a DUT to push the test commands to it. Sin…
150 - easy to integrate to buildbot;
151 - easy to run remotely;
152 - easy to deploy to the test lab.
156 - src/third_party/autotest/files/server/site_tests/platform_CloseOpenLid
157 - src/third_party/autotest/files/server/site_tests/platform_LongPressPower
159 FAFT is also based on the existing ServoTest approach and does some enhancements.
161 <a name="software-architecture" />
165 Here is our proposed FAFT framework. The red pieces are missing and need to be implemented in this …
167 
169 …the FAFT test cases are server side tests, meaning that all the actions are controlled by Host. It…
171 <a name="test-case-example" />
175 Here is an example to demonstrate how easy it is to write a FAFT test case. It test the firmware li…
176 1. FW A normal boot -> enable devsw + set try_fwb -> reboot -> press ctrl-d
177 2. FW B developer boot -> do nothing -> reboot -> press ctrl-d
178 3. FW A developer boot -> disable devsw -> reboot -> do nothing
179 4. FW A normal boot -> finished
185 # ------------------------- step 1 -------------------------
190 # ------------------------- step 2 -------------------------
195 # ------------------------- step 3 -------------------------
200 # ------------------------- step 4 -------------------------
216 <a name="faft-scopes" />
222 <a name="scope-1" />
226 …t of libraries to access system info and corrupt/restore firmware and kernel. It also has a set of…
254 …the same work. FAFT still uses the original SAFT test cases and libraries in the first step and po…
256 …of actions and expected results to test all firmware / kernel / recovery / tpm behaviors as a whol…
258 A demo of porting SAFT test cases on FAFT: [https://video.google.com/a/?pli=1#/Play/contentId=60bdd…
260 …esents the feasibility of the integration of SAFT library and FAFT framework. We still have a lot …
261 - Create a separated FAFT client and wrap all existing SAFT libraries into it;
262 - Setup RPC to invoke FAFT client instead of using SSH tunnel like the demo;
263 - Enhance the functions of FAFT server side utilities;
264 - Make FAFT framework more generalized to fit any possible test cases;
265 - Keep the state and backup firmware/kernel in the server side instead of USB;
266 - Automatic setup DUT disk and USB image when given a buildbot image.
268 <a name="scope-2" />
270 ### Scope 2: Test the Layer of VBExport APIs
272 …the test coverage and verify the underlying BIOS functions and drivers, a good entry is to re-run …
274 
276 <a name="scope-3" />
280 The normal boot and developer boot can be verified by SAFT. However, we still missed the test of th…
281 1. DD the recovery shim image to a USB stick;
285 5. Wait for the complete of recovery process;
289 …the Servo board by default. We first mux it to connect to the host machine such that we can write …
293 …the factory team. Firmware AU still relies on manual tests, see the [test matrix](https://docs.goo…
295 <a name="scope-4" />
299 …the machine. For example, the bug [http://crosbug.com/p/4419](http://crosbug.com/p/4419) reported,…
304 <a name="other-ideas" />
308 - Run FAFT on every build, integrated with buildbot
309 - Stress test to ensure the stability of hardware and drivers
310 - Cold reboot test
311 - Use HDMI capture device or camera + image recognition to verify the display output