1<!-- 2 Copyright 2012 The Android Open Source Project 3 4 Licensed under the Apache License, Version 2.0 (the "License"); 5 you may not use this file except in compliance with the License. 6 You may obtain a copy of the License at 7 8 http://www.apache.org/licenses/LICENSE-2.0 9 10 Unless required by applicable law or agreed to in writing, software 11 distributed under the License is distributed on an "AS IS" BASIS, 12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 See the License for the specific language governing permissions and 14 limitations under the License. 15--> 16 17# Getting Started 18 19## Using the console 20 21TF is based around an interactive console. You can fire up the console by going to the 22`tools/tradefederation/` directory and running 23 24 $ ./tradefed.sh 25 26You should end up at a `tf >` prompt. 27 28The console is self-documenting. Try entering "help" 29 30 tf >help 31 Enter 'q' or 'exit' to exit 32 Enter 'kill' to attempt to forcibly exit, by shutting down adb 33 34 Enter 'help list' for help with 'list' commands 35 [...] 36 37As the help text suggests, the help menus are organized hierarchically 38 39 tf >help list 40 l(?:ist)? help: 41 i[nvocations] List all invocation threads 42 d[evices] List all detected or known devices 43 [...] 44 45The majority of commands have a convenient short form, which the help text displays. The 46`l(?:ist)?` is a regular expression. As an example, here are the four equivalent ways to list 47invocations: 48* `list invocations` 49* `list i` 50* `l invocations` 51* `l i` 52 53 54## Running a config/command 55 56This is documented by the `help run` command in the console. 57 58As a reminder, a command is a config along with all of its command-line arguments. A command *must* 59begin with the name of the respective config. 60 61As a quick example, you could run the calculator unit tests like so: 62 63 $./tradefed.sh 64 tf >run instrument --package com.android.calculator2.tests 65 66As a shortcut, if you specify any arguments to `tradefed.sh`, it will attempt to execute them as if 67they were typed on the commandline. So the short version of the above would be 68 69 $./tradefed.sh run instrument --package com.android.calculator2.tests 70 71In both of these cases, the name of the config is "instrument", and 72"--class com.android.calculator2.tests" is a command-line argument. The command that is being run 73is "instrument --class com.android.calculator2.tests". 74 75TF can run both configs that are compiled in (such as the "instrument" config above), as well as 76configs that exist as xml files on the local filesystem. You can see a list of compiled-in configs 77with the `list configs` console command. Furthermore, you can investigate any config (compiled-in 78or local) by passing the "--help" or "--help-all" command-line arguments. The "--help" argument 79will only show "important" arguments, and "--help-all" will show all arguments, regardless of 80whether they've been marked as "important" or not. To take the final step, you can tell TF to print 81the contents of any config (compiled-in or local) with the `dump config <configname>` console 82command. 83 84### So, let's say you want to run the calculator instrumentation tests, but don't know where to start. 85 86You could try something like this sequence of steps. First, look for a config that looks like it 87might do what you want: 88 89 tf >list configs 90 Use 'run command --help <configuration_name>' to get list of options for a configuration 91 Use 'dump config <configuration_name>' to display the configuration's XML content. 92 93 Available configurations include: 94 [...] 95 instrument: Runs a single Android instrumentation test on an existing device 96 [...] 97 98Now that you've found something reasonable-looking, see what options it takes. The `list configs` output suggests trying `run command instrument --help` 99 100 tf >run command --help instrument 101 'instrument' configuration: Runs a single Android instrumentation test on an existing device 102 103 Printing help for only the important options. To see help for all options, use the --help-all flag 104 [...] 105 'instrumentation' test options: 106 -p, --package The manifest package name of the Android test application to run. 107 108As the message suggests, if you need more options, use the "--help-all" flag instead of "--help". In this case, we've got all we need. You could figure out the package by checking with `runtest`, or reading testdefs.xml directly. We use `runtest -n` to simply show what would be run without actually running it: 109 110 $runtest -n calculator 111 adb root 112 ONE_SHOT_MAKEFILE="packages/apps/Calculator/Android.mk" make -j4 -C "/srv/xsdg/master2" files 113 adb sync 114 adb shell am instrument -w com.android.calculator2.tests/android.test.InstrumentationTestRunner 115 116The argument to `am instrument` that comes before the slash is the manifest package. `android.test.InstrumentationTestRunner` is the default runner, so no need to set it if that's the 117right one. Otherwise, using "--help-all" will tell you about the "--runner" argument, which you can 118use to specify an alternate runner. Ok, so at this point, we've got the following command, which 119you'll recognize from above 120 121 tf >run instrument --package com.android.calculator2.tests 122 123 124## Interacting with a device 125 126### Generic device behavior in TF 127 128The running version of a command is called in `invocation`. First and foremost, every invocation 129requires a device before it can run. In addition to physical Android devices, TF can run tests with 130a mock device (by specifying the "-n" argument for the command), or with the Android emulator (by 131specifying the "-e" argument"). 132 133The primary console command to figure out what devices are up to is `list devices`: 134 135 $./tradefed.sh 136 06-07 17:03:22 I/: Detected new device 016B756E03018007 137 06-07 17:03:22 I/: Detected new device 1700614743c14397 138 06-07 17:03:22 I/: Detected new device 3531C342606300EC 139 tf >l d 140 Serial State Product Variant Build Battery 141 016B756E03018007 Available tuna toro MASTER 100 142 1700614743c14397 Available stingray stingray MASTER 100 143 3531C342606300EC Available herring crespo4g MASTER 92 144 145As far as the invocations are concerned, there are three device states: available, unavailable, and 146allocated. An `Available` device is ready to be allocated for an invocation. An `Unavailable` 147device is not ready for allocation, for any of a variety of reasons — TF may have deemed to the 148device as unstable, the device may be critically low on storage, or something else may be amiss. 149Finally, an `Allocated` device is a device that is already being used by an invocation. 150 151When you start TF, all detected physical devices will be checked for responsiveness with a simple 152shell command. If the command completes successfully, the device will be listed as Available. If 153the command fails, the device state will be shown as Unavailable. Thereafter, a device will typically bounce between the Available and Allocated states as invocation requirements dictate. 154 155Finally, once invocations are already underway, you can see what's going on with the `list 156invocations` command 157 158 tf >run instrument --package com.android.calculator2.tests 159 06-07 17:18:31 I/TestInvocation: Starting invocation for 'stub' on build '0' on device 1700614743c14397 160 [...] 161 tf >l d 162 Serial State Product Variant Build Battery 163 1700614743c14397 Allocated stingray stingray MASTER 100 164 3531C342606300EC Available herring crespo4g JRN11 93 165 016B756E03018007 Available tuna toro MASTER 100 166 167 tf >l i 168 Command Id Exec Time Device State 169 1 0m:02 1700614743c14397 running stub on build 0 170 171 172### Running invocations on specific devices 173 174TF supports a number of filtering mechanisms for specifying which device or devices to use for a 175particular invocation. Since the filtering mechanisms are run before a command turns into an 176invocation, you can find all of the filtering options in the help for any config: 177 178tf >run instrument --help-all 179[...] 180 device_requirements options: 181 -s, --serial run this test on a specific device with given serial number(s). 182 --exclude-serial run this test on any device except those with this serial number(s). 183 --product-type run this test on device with this product type(s). May also filter by variant using product:variant. 184 --property run this test on device with this property value. Expected format <propertyname>=<propertyvalue>. 185 -e, --[no-]emulator force this test to run on emulator. Default: false. 186 -d, --[no-]device force this test to run on a physical device, not an emulator. Default: false. 187 --[no-]new-emulator allocate a placeholder emulator. Should be used when config intends to launch an emulator Default: false. 188 -n, --[no-]null-device 189 do not allocate a device for this test. Default: false. 190 --min-battery only run this test on a device whose battery level is at least the given amount. Scale: 0-100 191 --max-battery only run this test on a device whose battery level is strictly less than the given amount. Scale: 0-100 192[...] 193 194The built-in help should be pretty self-explanatory. All of the filtering options excluding "-n", 195"-e", and "-d" may be specified as many times as needed. So, for instance, to run an invocation 196using any Verizon Galaxy Nexus, you could do the following: 197 198 tf >run instrument --package com.android.calculator2.tests --product-type tuna:toro 199 200As another example, to run on a GSM device with a SIM, you could do the following: 201 202 tf >run instrument --package com.android.calculator2.tests --property gsm.sim.state=READY 203 204The filtering works by exclusion from the pool of Available devices, so the "--serial" option simply 205excludes devices that aren't in the list of required serials, and --exclude-serial excludes devices 206that *are* in its list. As such, an argument like --exclude-serial XXX --serial XXX will simply 207make the respective command un-runnable — it will never match any device, since all devices are 208excluded. 209 210 211## Logging 212 213There are a few different aspects to logging in TF. First and foremost, TF has a built-in logging 214infrastructure that's based on DDMLib's Log class. For the common case, where the log tag is just 215the classname of the current class, you can use our CLog convenience shim. In short, if you might 216have originally done this: 217 218 class ClassName { 219 private static final LOG_TAG = "ClassName"; 220 [...] 221 Log.v(LOG_TAG, "This is a simple verbose log message"); 222 Log.w(LOG_TAG, String.format("This warning message brought to you by the number %d", 17)); 223 224You can now accomplish the same thing with the shim like this: 225 226 class ClassName { 227 [...] 228 CLog.v("This is a simple verbose log message"); 229 CLog.w("This warning message brought to you by the number %d", 17); 230 231Each Invocation has its own ThreadGroup. Any host-side logging that happens inside of that thread 232group is associated with the Invocation, and will be reported as that invocation's "host_log" after 233the Invocation completes. 234 235Device logging is performed as part of TradeFed's device wrapper. We keep a buffer of up to 20 MB 236that captures log data as the device churns it out. In particular, we are not limited by the size 237of the on-device logcat buffer. 238 239The next important piece is the ITestInvocationListener. This is one of the components of an 240Invocation that handles results reporting. Each reporter has the option to implement the #testLog 241method, which will be used to pass logfiles to that result reporter. Among the files that are 242passed by TF itself will be the aforementioned host_log, as well as the device logcat for the device 243associated with the Invocation. 244 245<!-- 246FIXME: discuss test result reporting, retrieving builds, doing things continuously, target prep and 247FIXME: flashing builds, extending tradefed (like CTS). 248--> 249 250