1# Running ART Tests with Atest / Trade Federation 2 3ART Testing has early support for execution in the [Trade 4Federation](https://source.android.com/devices/tech/test_infra/tradefed) 5("TradeFed") test harness, in particular via the 6[Atest](https://source.android.com/compatibility/tests/development/atest) 7command line tool. 8 9Atest conveniently takes care of building tests and their dependencies (using 10Soong, the Android build system) and executing them using Trade Federation. 11 12See also [README.md](README.md) for a general introduction to ART run-tests and 13gtests. 14 15## ART run-tests 16 17### Running ART run-tests on device 18 19ART run-tests are defined in sub-directories of `test/` starting with a number 20(e.g. `test/001-HelloWorld`). Each ART run-test is identified in the build 21system by a Soong module name following the `art-run-test-`*`<test-directory>`* 22format (e.g. `art-run-test-001-HelloWorld`). 23 24You can run a specific ART run-test on device by passing its Soong module name 25to Atest: 26```bash 27atest art-run-test-001-HelloWorld 28``` 29 30To run all ART run-tests in a single command, the currently recommended way is 31to use [test mapping](#test-mapping) (see below). 32 33You can nonetheless run all supported ART run-tests with a single Atest command, 34using its support for wildcards: 35```bash 36atest art-run-test-\* 37``` 38 39Note: Many ART run-tests are failing with the TradeFed harness as of March 2021, 40so the above Atest command will likely report many tests failures. The ART team 41is actively working on this issue. 42 43## ART gtests 44 45### Running ART gtests on device 46 47There are three ways to run ART gtests on device: 481. by building "standalone" ART gtests and running them against the active ART 49 APEX on the device; 502. by installing the Testing ART APEX (i.e. manually "updating" the ART APEX on 51 device); or 523. by setting up a `chroot` environment on the device, and "activating" the 53 Testing ART APEX in that environment. 54 55The first approach can be used to test the ART APEX presently residing on a 56device (either the original one, located in the "system" partition, or an 57updated package, present in the "data" partition). 58 59The second and third approaches make use of the Testing ART APEX 60(`com.android.art.testing.apex`), and were previously the only options to run 61ART gtests on device, because of build- and link-related limitations (the ART 62gtests had to be part of the ART APEX package itself to be able to build and run 63properly). 64 65### Running standalone ART gtests on device 66 67Standalone ART gtests are defined as Soong modules `art_standalone_*_tests`. You 68can run them individually with Atest, e.g: 69 70```bash 71atest art_standalone_cmdline_tests 72``` 73 74You can also run all of them with a single Atest command, using its support for 75wildcards: 76 77```bash 78atest art_standalone_\*_tests 79``` 80 81The previous commands build the corresponding ART gtests and their dependencies, 82dynamically link them against local ART APEX libraries (in the source tree), and 83run them on device against the active ART APEX. 84 85### Running ART gtests on device by installing the Testing ART APEX 86 87You can run ART gtests on device with the ART APEX installation strategy by 88using the following `atest` command: 89 90```bash 91atest ArtGtestsTargetInstallApex 92``` 93 94This command: 951. builds the Testing ART APEX from the Android source tree (including the ART 96 gtests); 972. installs the Testing ART APEX using `adb install`; 983. reboots the device; 994. runs the tests; and 1005. uninstalls the module. 101 102You can run the tests of a single ART gtest C++ class using the 103`ArtGtestsTargetInstallApex:`*`<art-gtest-c++-class>`* syntax, e.g.: 104```bash 105atest ArtGtestsTargetInstallApex:JniInternalTest 106``` 107 108This syntax also supports the use of wildcards, e.g.: 109```bash 110atest ArtGtestsTargetInstallApex:*Test* 111``` 112 113You can also use Trade Federation options to run a subset of ART gtests, e.g.: 114```bash 115atest ArtGtestsTargetInstallApex -- \ 116 --module ArtGtestsTargetInstallApex --test '*JniInternalTest*' 117``` 118 119You can also pass option `--gtest_filter` to the gtest binary to achieve a 120similar effect: 121```bash 122atest ArtGtestsTargetInstallApex -- \ 123 --test-arg com.android.tradefed.testtype.GTest:native-test-flag:"--gtest_filter=*JniInternalTest*" 124``` 125 126### Running ART gtests on device using a `chroot` environment 127 128You can run ART gtests on device with the chroot-based strategy by using the 129following command: 130 131```bash 132atest ArtGtestsTargetChroot 133``` 134 135This sequence: 1361. builds the Testing ART APEX from the Android source tree (including the ART 137 gtests) and all the necessary dependencies for the `chroot` environment; 1382. sets up a `chroot` environment on the device; 1393. "activates" the Testing ART APEX (and other APEXes that it depends on) in the 140 `chroot` environment; 1414. runs the tests within the `chroot` environment; and 1425. cleans up the environment (deactivates the APEXes and removes the `chroot` 143 environment). 144 145### Running ART gtests on host 146 147You first need to build the boot classpath and boot image on host: 148 149```bash 150m art-host-tests 151``` 152 153Then you can use `atest --host` to run host gtests, e.g: 154 155```bash 156atest --host art_runtime_tests 157``` 158 159## Test Mapping 160 161ART Testing supports the execution of tests via [Test 162Mapping](https://source.android.com/compatibility/tests/development/test-mapping). 163The tests declared in ART's [TEST_MAPPING](../TEST_MAPPING) file are executed 164during pre-submit testing (when an ART changelist in Gerrit is verified by 165Treehugger) and/or post-submit testing (when a given change is merged in the 166Android code base), depending on the "test group" where a test is declared. 167 168### Running tests via Test Mapping with Atest 169 170It is possible to run tests via test mapping locally using Atest. 171 172To run all the tests declared in ART's `TEST_MAPPING` file, use the following 173command from the Android source tree top-level directory: 174```bash 175atest --test-mapping art:all 176``` 177In the previous command, `art` is the (relative) path to the directory 178containing the `TEST_MAPPING` file listing the tests to run, while `all` means 179that tests declared in all [test 180groups](https://source.android.com/compatibility/tests/development/test-mapping#defining_test_groups) 181shall be run. 182 183To only run tests executed during pre-submit testing, use: 184```bash 185atest --test-mapping art:presubmit 186``` 187