1Copyright (c) 2010 The Chromium OS Authors. All rights reserved. 2Use of this source code is governed by a BSD-style license that can be 3found in the LICENSE file. 4 5 6This document describes the steps to go through in order to run ChromeOS 7hardware qualification on a device under test. 8 9================================================================================ 10Glossary 11================================================================================ 12 13- $: command line prompt 14- $HOME: home directory of current user. 15- AC: alternating current, implies device is not powered by battery. 16- DUT: device under test 17- Semi-Automated Test: test that runs with automation but requires manual 18 intervention. 19 20================================================================================ 21Test Setup 22================================================================================ 23 24- Setup a Linux machine to serve as the Autotest server. The Autotest 25 server requires Python, Wireless access to the DUT and basic Linux 26 shell utilities. The setup has been tested on Ubuntu 9.10 available 27 for download at http://www.ubuntu.com/getubuntu/download/. 28 29 30- Create an installation directory on the Autotest server for the 31 ChromeOS hardware qualification package. The rest of the 32 instructions assume that you're installing the package in the 33 current user home directory ($HOME/). 34 35 36- Contact your Google technical support person and download the Chrome 37 OS hardware qualification package chromeos-hwqual-TAG.tar.bz2 for 38 your device in $HOME/. 39 40 41- Install the package on the server: 42 43 $ cd $HOME/ && tar xjf chromeos-hwqual-TAG.tar.bz2 44 45 46- Install the ChromeOS test image on the DUT. The USB test image is 47 available in: 48 49 $HOME/chromeos-hwqual-TAG/chromeos-hwqual-usb.img 50 51 Here are sample steps to install the test image. 52 53 - Plug a USB storage device into the Autotest server. Note that all 54 data on your USB stick will be destroyed. 55 56 - Unmount any mounts on the USB device. 57 58 - Copy the USB image to a USB storage device by executing: 59 60 $ sudo dd if=$HOME/chromeos-hwqual-TAG/chromeos-hwqual-usb.img \ 61 of=/dev/sdX 62 63 - where /dev/sdX is your USB device. 64 65 - Plug the USB device into the DUT and boot from it. 66 67 - Log in to ChromeOS. Start the ChromeOS shell by pressing Ctrl-Alt-T. 68 Install ChromeOS on the DUT: 69 70 crosh> install 71 72 73- Cold boot the DUT -- turn the DUT off and then back on. This ensures 74 a consistent starting point for the qualification tests and allows 75 the system to collect cold boot performance metrics. Make sure you 76 don't boot from USB. 77 78 79- Connect the DUT to the network and note its IP address <DUT_IP>. The 80 IP address is displayed at the bottom of the network selection 81 menu. Unless specified explicitly, the test setup works correctly 82 through either wireless or wired network connections. 83 84 85- Add the DUT root private key to ssh-agent on the Autotest server: 86 87 $ ssh-add $HOME/chromeos-hwqual-TAG/testing_rsa 88 89- If ssh-add fails saying that it cannot connect to your authentication agent, 90 retry the command after running: 91 92 $ eval `ssh-agent -s` 93 94- These commands allow the Autotest server to connect and login as root on the 95 DUT. 96 97 98- Make sure you can ssh as root to the DUT from the Autotest 99 server. The command below should print 0. 100 101 $ ssh root@<DUT_IP> true; echo $? 102 103================================================================================ 104Automated and Semi-Automated Test Runs 105================================================================================ 106 107- Unless otherwise noted, all tests can be performed on an AC-powered DUT. 108 109- Go to the Autotest server directory and clean up previous test results. 110 111 $ cd $HOME/chromeos-hwqual-TAG/autotest/ 112 $ rm -rf results.* 113 114 115- Run the fully automated client-side tests: 116 117 $ ./server/autoserv -r results.auto -m <DUT_IP> \ 118 -c client/site_tests/suite_HWQual/control.auto 119 120 121- Plug high-speed high-capacity storage devices in all USB and SD Card 122 slots and run the external storage test: 123 124 $ ./server/autoserv -r results.external_devices -m <DUT_IP> \ 125 -c client/site_tests/suite_HWQual/control.external_drives 126 127 128- Run the system suspend/resume stability test: 129 130 $ ./server/autoserv -r results.suspend_resume -m <DUT_IP> \ 131 -c client/site_tests/suite_HWQual/control.suspend_resume 132 133 134- If the DUT has video out ports, run the Video Out semi-automated 135 test by following the instructions specified in the control file 136 (control.video_out) and then executing: 137 138 $ ./server/autoserv -r results.video_out.${PORT} -m <DUT_IP> \ 139 -c client/site_tests/suite_HWQual/control.video_out 140 141- Where PORT is the name of each video port you are testing. For 142 example, if the DUT has one HDMI and one VGA out port, run: 143 144 $ ./server/autoserv -r results.video_out.hdmi1 -m <DUT_IP> \ 145 -c client/site_tests/suite_HWQual/control.video_out 146 147 $ ./server/autoserv -r results.video_out.vga1 -m <DUT_IP> \ 148 -c client/site_tests/suite_HWQual/control.video_out 149 150 151- Run the graphics tearing test: 152 153 $ ./server/autoserv -r results.teartest -m <DUT_IP> \ 154 -c client/site_tests/suite_HWQual/control.teartest 155 156- Run audio test, with built-in speakers and microphone 157 158 $ ./server/autoserv -r results.audio -m <DUT_IP> \ 159 -c client/site_tests/suite_HWQual/control.audio 160 161- Plug-in headphone and microphone, run audio test again 162 163 $ ./server/autoserv -r results.audio_ext -m <DUT_IP> \ 164 -c client/site_tests/suite_HWQual/control.audio 165 166- Run the Keyboard test : 167 (Wait several seconds after running the test. Then strike the "Search" key, 168 e.g. the key above Left Shift and below Tab) 169 170 $ ./server/autoserv -r result.keyboard -m <DUT_IP> \ 171 -c client/site_tests/suite_HWQual/control.keyboard 172 173- Run the DUT on AC. Probe the AC driver: 174 175 $ ./server/autoserv -r result.probe_ac -m <DUT_IP> \ 176 -c client/site_tests/suite_HWQual/control.probe_ac 177 178- Run the DUT on battery. Probe the battery driver: 179 (If you just unplugged AC, please wait for a second before running 180 the test for kernel updating power status.) 181 182 $ ./server/autoserv -r result.probe_bat -m <DUT_IP> \ 183 -c client/site_tests/suite_HWQual/control.probe_bat 184 185- Run the DUT on AC. Plug a power draw USB dongle in each USB port. 186 Run the max power draw test: 187 188 $ ./server/autoserv -r results.max_power_draw.ac -m <DUT_IP> \ 189 -c client/site_tests/suite_HWQual/control.max_power_draw 190 191- Run the DUT on battery. Plug a power draw USB dongle in each USB 192 port. Run the max power draw test: 193 194 $ ./server/autoserv -r results.max_power_draw.batt -m <DUT_IP> \ 195 -c client/site_tests/suite_HWQual/control.max_power_draw 196 197- Run the DUT on AC. Run the power settings test: 198 199 $ ./server/autoserv -r results.power_x86_setting.ac -m <DUT_IP> \ 200 -c client/site_tests/suite_HWQual/control.power_x86_settings 201 202- Run the DUT on battery. Run the power settings test: 203 204 $ ./server/autoserv -r results.power_x86_setting.batt -m <DUT_IP> \ 205 -c client/site_tests/suite_HWQual/control.power_x86_settings 206 207- Make sure the remaining battery charge is less than 5%. Note that the test 208 will check and fail quickly if the initial battery charge is more than 5%. 209 Run the DUT on AC. Run the battery charge test: 210 211 $ ./server/autoserv -r results.battery_charge_time -m <DUT_IP> \ 212 -c client/site_tests/suite_HWQual/control.battery_charge_time 213 214- Make sure that the battery is fully charged. Note that the test will not 215 check if the battery is fully charged before running. Run the DUT on 216 battery. Run the battery load test by first following the manual 217 instructions specified in the control file (control.battery_load) 218 and then executing: 219 220 $ ./server/autoserv -r results.battery_load -m <DUT_IP> \ 221 -c client/site_tests/suite_HWQual/control.battery_load 222 223================================================================================ 224Reviewing Automated and Semi-Automated Test Results 225================================================================================ 226 227- Autotest logs progress and performance data in results.* directories 228 specified through the '-r' autoserv option. The easy way to see a summary 229 of your test results is to use the 'generate_test_report' 230 script installed under $HOME/chromeos-hwqual-TAG/: 231 232 $ ../generate_test_report results.* 233 234 This will display a table with test status and perfomance data for 235 all result directories. 236 237- If deeper investigation into a failure is required, you can review 238 the debug information stored in results: 239 240 $ ls */*/debug/. 241 242================================================================================ 243Manual Test Runs 244================================================================================ 245 246- Perform the manual tests specified in 247 $HOME/chromeos-hwqual-TAG/manual/testcases.csv. 248 249- Please note that some tests cannot be tested as they rely on 250 features not yet implemented. They are being included as a preview for 251 manual tests that will be required. Such tests will have 252 "NotImplemented" in their "LABELS" column. 253 254- Update this spreadsheet with a column of pass/fail results with any notes 255 which may be useful. 256 257================================================================================ 258Reporting Results 259================================================================================ 260 261- Upon completing automatic, semi-automatic, and manual test runs, you should 262 report your results to a Google technical support contact to verify the 263 tests were run correctly and to help diagnose failures. Verify you have 264 updated your manual test spreadsheet as described above and copy it into 265 your autotest output directory: 266 267 $ cd $HOME/chromeos-hwqual-TAG/autotest/ 268 $ cp ../manual/testcases.csv . 269 270- Package all results into a tar file: 271 272 $ tar cjf chromeos-hwqual-results-TAG-DATE.tar.bz2 results.* testcases.csv 273 274- Send the resulting chromeos-hwqual-results file to your technical support 275 contact. 276