1# Android Instrumentation Tests 2 3Instrumentation tests are Java tests based on 4[`android.app.Instrumentation`](https://developer.android.com/reference/android/app/Instrumentation.html). 5They run on a device. 6 7[TOC] 8 9## Writing an instrumentation test 10 11Currently, an instrumentation test is just a JUnit3-style test based on 12[android.test.InstrumentationTestCase](https://developer.android.com/reference/android/test/InstrumentationTestCase.html). 13(This will change starting in [Android N](https://en.wikipedia.org/wiki/Android_Nougat).) 14 15Writing an instrumentation test case can be simple, e.g. 16 17```java 18package org.chromium.sample.test; 19 20import android.test.InstrumentationTestCase; 21 22public class MyInstrumentationTest extends InstrumentationTestCase { 23 24 // Note that, because it's a JUnit3-style test, the test method *must* 25 // start with "test". 26 public void testTheFirst() { 27 bool writingInstrumentationTestsCanBeEasy = true; 28 29 // InstrumentationTestCase inherits the assert* methods through 30 // junit.framework.TestCase. 31 assertTrue(writingInstrumentationTestsCanBeEasy); 32 } 33 34 public void testTheSecond() { 35 bool writingInstrumentationTestsIsAlwaysEasy = false; 36 assertFalse(writingInstrumentationTestsIsAlwaysEasy); 37 } 38} 39``` 40 41After writing a test, you can run it by: 42 43 - Adding the file to the relevant gn target if the entire file is new. 44Typically, the "relevant gn target" is simply the target containing the 45other files in the same directory. 46 - Rebuild. 47 - Run the test using the process described [here](/testing/android/docs/todo.md). 48 49## Instrumentation test features 50 51In many cases, Chromium has extended the instrumentation test framework 52classes to implement additional features. 53 54### Tracing 55 56Enabling tracing during a test run allows all the function calls involved to be 57observed in a visual display (using Chrome's built-in chrome://tracing feature). 58To run a test with tracing, add the `--trace-output` flag to the command used to 59call the instrumentation test (either running the test_runner.py script, or a 60generated binary such as `run_chrome_public_test_apk`). The `--trace-output` flag 61takes a filename, which, after the test run, will contain a JSON file readable 62by chrome://tracing. 63 64By default, the trace includes only certain function calls important to the test 65run, both within the Python test runner framework and the Java code running on 66the device. For a more detailed look, add the (no-argument) `--trace-all` flag. 67This causes every function called on the Python side to be added to the trace. 68 69### Annotations 70 71Instrumentation tests in Chromium use a wide variety of annotations to control 72and manipulate test execution. Some of these are implemented in Chromium, while 73others are pulled in from outside. They include: 74 75#### Test Batching 76 77The [`@Batch("group_name")`](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/Batch.java) 78annotation is used to run all tests with the same batch group name in the same 79instrumentation invocation. In other words, the browser process is not 80restarted between these tests, and so any changes to global state, like 81launching an Activity, will persist between tests within a batch group. The 82benefit of this is that these tests run significantly faster - the per-test cost 83of restarting the process can be as high as 10 seconds (usually around 2 84seconds), and that doesn't count the cost of starting an Activity like 85ChromeTabbedActivity. 86 87#### Size annotations 88 89Size annotations are used primarily by the test runner to determine the length 90of time to wait before considering a test hung (i.e., its timeout duration). 91 92Several of the annotations are Android APIs from 93[android.test.suitebuilder.annotation](https://developer.android.com/reference/android/test/suitebuilder/annotation/package-summary.html) 94(prior to [Android N](https://en.wikipedia.org/wiki/Android_Nougat)) or 95[androidx.test.filters](https://developer.android.com/reference/androidx/test/filters/package-summary.html) 96(starting in Android N). These are all fairly self-explanatory: 97 98 - [`@SmallTest`](https://developer.android.com/reference/androidx/test/filters/SmallTest.html) (timeout: **10 seconds**) 99 - [`@MediumTest`](https://developer.android.com/reference/androidx/test/filters/MediumTest.html) (timeout: **30 seconds**) 100 - [`@LargeTest`](https://developer.android.com/reference/androidx/test/filters/LargeTest.html) (timeout: **2 minutes**) 101 102A few additional size annotations are provided in 103[//base](https://chromium.googlesource.com/chromium/src/+/main/base): 104 105 - [`@EnormousTest`](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/EnormousTest.java) 106(timeout: **5 minutes**) Typically used for tests that require WiFi. 107 - [`@IntegrationTest`](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/IntegrationTest.java) 108(timeout: **10 minutes**) Used for tests that run against real services. 109 - [`@Manual`](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/Manual.java) 110(timeout: **10 hours**) Used for manual tests. 111 112Beware that the timeout durations for these annotations are subject to 113change, though they rarely do. These values are defined 114[here](https://chromium.googlesource.com/chromium/src/+/main/build/android/pylib/local/device/local_device_instrumentation_test_run.py#20). 115 116#### Annotations that disable tests 117 118There are several annotations that control whether or not a test runs. 119Some are conditional, others are not. 120 121##### Unconditional disabling 122 123[**@DisabledTest**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/DisabledTest.java) 124unconditionally disables a test. 125```java 126@DisabledTest( 127 // Describes why the test is disabled. Typically includes a crbug link. 128 message = "" 129) 130``` 131 132##### Conditional disabling 133 134There are two primary annotation categories that conditionally disable tests: 135**@DisableIf** and **@Restriction**. The **@DisableIf** annotations are intended 136to temporarily disable a test in certain scenarios where it *should* work but 137doesn't. In contrast, the **@Restriction** annotation is intended to 138permanently limit a test to specific configurations. It signifies that the test 139was not, is not, and will not be intended to run beyond those configurations. 140In both cases, conditional disabling manifests as a skipped test. 141 142[**@DisableIf.Build**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/DisableIf.java#25) 143allows for conditional test disabling based on values in 144[`android.os.Build`](https://developer.android.com/reference/android/os/Build.html): 145 146```java 147@DisableIf.Build( 148 149 // Describes why the test is disabled. 150 message = "", 151 152 // Disables the test on SDK levels that match the given conditions. 153 // Checks against Build.VERSION.SDK_INT. 154 sdk_is_greater_than = 0, 155 sdk_is_less_than = Integer.MAX_VALUE, 156 157 // Disables the test on devices that support the given ABI 158 // (e.g. "arm64-v8a"). Checks against: 159 // - Build.SUPPORTED_ABIS on L+ 160 // - Build.CPU_ABI and Build.CPU_ABI2 otherwise 161 supported_abis_includes = "", 162 163 // Disables the test on devices with hardware that matches the given 164 // value. Checks against Build.HARDWARE. 165 hardware_is = "", 166 167 // Disables the test on devices with product names that contain the 168 // given value. Checks against Build.PRODUCT. 169 product_name_includes = "", 170 171) 172``` 173 174[**@DisableIf.Device**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/DisableIf.java#40) 175allows for conditional test disabling based on whether 176a device is a phone, a tablet, or a "large tablet" as determined by 177[org.chromium.ui.base.DeviceFormFactor](https://chromium.googlesource.com/chromium/src/+/main/ui/android/java/src/org/chromium/ui/base/DeviceFormFactor.java). 178This is available to tests in 179[//ui](https://chromium.googlesource.com/chromium/src/+/main/ui/) 180or code that uses //ui. 181 182```java 183@DisableIf.Device( 184 // Disables the test on devices that match the given type(s) as described 185 // above. 186 type = {} 187) 188``` 189 190[**@Restriction**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/Restriction.java) 191currently allows for conditional test disabling based on device 192type, device performance, internet connectivity, whether Google Play Services is 193up to date, and whether the build was an official one. 194 195```java 196@Restriction( 197 // Possible values include: 198 // 199 // base: 200 // - Restriction.RESTRICTION_TYPE_LOW_END_DEVICE 201 // Restricts the test to low-end devices as determined by SysUtils.isLowEndDevice(). 202 // 203 // - Restriction.RESTRICTION_TYPE_NON_LOW_END_DEVICE 204 // Restricts the test to non-low-end devices as determined by SysUtils.isLowEndDevice(). 205 // 206 // - Restriction.RESTRICTION_TYPE_INTERNET 207 // Restricts the test to devices that have an internet connection. 208 // 209 // chrome: 210 // - ChromeRestriction.RESTRICTION_TYPE_GOOGLE_PLAY_SERVICES 211 // Restricts the test to devices with up-to-date versions of Google Play Services. 212 // 213 // - ChromeRestriction.RESTRICTION_TYPE_OFFICIAL_BUILD 214 // Restricts the test to official builds as determined by ChromeVersionInfo.isOfficialBuild(). 215 // 216 // ui: 217 // - UiRestriction.RESTRICTION_TYPE_PHONE 218 // Restricts the test to phones as determined by DeviceFormFactor. 219 // 220 // - UiRestriction.RESTRICTION_TYPE_TABLET 221 // Restricts the test to tablets as determined by DeviceFormFactor. 222 value = {} 223) 224``` 225 226[**@MinAndroidSdkLevel**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/MinAndroidSdkLevel.java) 227is similar to **@Restriction** in purpose in that it's 228intended to permanently limit a test to only recent versions of Android. 229 230```java 231@MinAndroidSdkLevel( 232 // The minimum SDK level at which this test should be executed. Checks 233 // against Build.VERSION.SDK_INT. 234 value = 0 235) 236``` 237 238#### Annotations that affect how a test is run 239 240Several annotations affect how a test is run in interesting or nontrivial ways. 241 242[**@CommandLineFlags.Add**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/CommandLineFlags.java#46) 243and 244[**@CommandLineFlags.Remove**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/CommandLineFlags.java#58) 245manipulate Chrome's 246command-line flags on a per-test basis (i.e., the flags handled by 247[`org.chromium.base.CommandLine`](https://chromium.googlesource.com/chromium/src/+/main/base/android/java/src/org/chromium/base/CommandLine.java) and 248[`base::CommandLine`](https://chromium.googlesource.com/chromium/src/+/main/base/command_line.h)). 249 250```java 251@CommandLineFlags.Add( 252 // The flags to add to the command line for this test. These can be 253 // anything and typically should include the leading dashes (e.g. "--foo"). 254 value = {} 255) 256 257@CommandLineFlags.Remove( 258 // The flags to remove from the command line for this test. These can only 259 // be flags added via @CommandLineFlags.Add. Flags already present in the 260 // command-line file on the device are only present in the native 261 // CommandLine object and cannot be manipulated. 262 value = {} 263) 264``` 265 266#### Feature annotations 267 268[**@Feature**](https://chromium.googlesource.com/chromium/src/+/main/base/test/android/javatests/src/org/chromium/base/test/util/Feature.java) 269has been used inconsistently in Chromium to group tests across 270test cases according to the feature they're testing. 271 272```java 273@Feature( 274 // The features associated with this test. These can be anything. 275 value = {} 276) 277``` 278 279@Feature doesn't have an inherent function, but it can be used to filter tests 280via the test runner's 281[-A/--annotation](/testing/android/docs/todo.md) and 282[-E/--exclude-annotation](/testing/android/docs/todo.md) flags. For example, 283this would run only the tests with @Feature annotations containing at least 284"Sync" in `chrome_public_test_apk`: 285 286```bash 287out/Debug/bin/run_chrome_public_test_apk -A Feature=Sync 288``` 289 290## Advanced 291 292 - [Creating a component for use in instrumentation tests.](/testing/android/docs/components_for_testing.md) 293