1## Introduction 2 3`test_that` is the supported mechanism to run autotests against ChromeOS 4devices at your desk. `test_that` replaces an older script, `run_remote_tests`. 5 6Features for testing a local device: 7 - CTRL+C kills `test_that` and all its autoserv children. Orphaned processes 8 are no longer left behind. 9 - Tests that require binary autotest dependencies will just work, because 10 test_that always runs from the sysroot location. 11 - Running emerge after python-only test changes is no longer necessary. 12 test_that uses autotest_quickmerge to copy your python changes to the 13 sysroot. 14 - Tests are generally specified to `test_that` by the NAME field of their 15 control file. Matching tests by filename is supported using f:[file 16 pattern] 17 18In addition to running tests against local device, `test_that` can be used to 19launch jobs in the ChromeOS Hardware Lab (or against a local Autotest instance 20or a Moblab). This feature is only supported for infrastructure-produced builds 21that were uploaded to google storage. 22 23### Example uses (inside the chroot) 24 25Run the test(s) named stub\_Pass: 26 27``` 28$ test_that -b ${board} ${host} stub_Pass 29``` 30 31Run the test(s) named stub\_Pass.suspend: 32 33``` 34$ test_that -b ${board} ${host} stub_Pass.suspend 35``` 36 37Run the bvt-inline suite against dut: 38 39``` 40$ test_that -b ${board} ${host} suite:bvt-inline 41``` 42 43Run all tests whose names match the regular expression `^login_.*$`. Note that 44even though these tests have binary dependencies, there is no longer a need to 45specify extra flags. 46 47``` 48$ test_that -b ${board} ${host} e:login_.* 49``` 50 51Run all tests whose control file filename matches the regular expression 52`^.*control.stub$`: 53 54``` 55$ test_that -b ${board} ${host} f:.*control.stub 56``` 57 58### Running jobs in the lab 59 60`test_that` now allows you to run jobs in the test lab. The usage is similar to 61running tests against a specified host. The keyword :lab: is used as 62test\_that's REMOTE argument, and the -i/--build argument is required, and takes 63a trybot, paladin, or canary build number. To learn how to build a trybot image 64with a new test that you're iterating on, see "dynamic suite" codelab or the 65[Chromium OS Remote 66Trybots](https://sites.google.com/a/chromium.org/dev/chromium-os/build/using-remote-trybots) 67guide. 68 69For instance: 70 71``` 72$ test_that -b lumpy -i lumpy-paladin/R38-6009.0.0-rc4 :lab: stub_Pass 73``` 74 75This will kick off a suite in the lab that consists of just 1 job, stub\_Pass, 76to run in this case on board lumpy using the image 77lumpy-paladin/R38-6009.0.0-rc4. The lab's scheduler will take responsibility 78for finding a suitable set of hosts, provisioning them to the correct image, 79and running the tests. `test_that` will return after the suite finishes running, 80with a suite run report. 81 82You can specify multiple tests or test-matching expressions in the same way as 83before: 84 85``` 86$ test_that -b lumpy -i ${latest_image} :lab: stub_Pass dummy_Fail 87$ test_that -b lumpy -i ${latest_image} :lab: e:login_.* 88``` 89 90Kicking off a run in the lab should be useful whenever you need to run a 91particular test on a board or image that you do not have readily available 92locally.For occasional runs of ad-hoc suites in the lab, this will also avoid 93the need to create a suite control file and wait for it to end up in an image. 94 95You can also kick off a suite, for example with: 96 97``` 98test_that -b peach_pit :lab: suite:pyauto_perf -i 'peach_pit-release/R32-4763.0.0' 99``` 100 101That told me that my job ID was 5196037. I could follow along by going to 102http://cautotest/afe/#tab_id=view_job&object_id=5195962. 103 104Things to note about running in the lab: 105 106 - This feature will only work on builds that produced autotest test artifacts. 107 If your build doesn't have such artifacts, you will see a 108 [confusing error](https://crbug.com/354556). The easiest way today to 109 guarantee that hwtest artifacts are produced is to make sure that your tryjob 110 is launched with the --hwtest flag. Once [this bug](https://crbug.com/299838) 111 is fixed that will no longer be the case. 112 - By default, jobs will be scheduled in the `suites` machine pool. That can be 113 overridden with the `-p` flag. 114 - This will only work with images newer than Sept 20, 2013 (specifically, builds 115 that contain Ifa73d7de7aac9c6efebd5f559708623804ad3691). 116 117 118### Running jobs against a local Autotest setup or MobLab 119 120`test_that` allows you to run jobs against a local Autotest setup or a 121MobLab instance. This usage is similar to running tests in the lab. The argument 122--web allows you to specify the web address of the Autotest instance you want to 123run tests within. 124 125For instance: 126``` 127$ test_that -b lumpy -i lumpy-paladin/R38-6009.0.0-rc4 --web 100.96.51.136 :lab: 128stub_Pass 129``` 130 131This will kick off the stub_Pass test on a lumpy device on the Autotest 132instance located at 100.96.51.136 133