1# Benchmarking in AndroidX
2
3[TOC]
4
5The public documentation at
6[d.android.com/benchmark](http://d.android.com/benchmark) explains how to use
7the library - this page focuses on specifics to writing libraries in the
8AndroidX repo, and our continuous testing / triage process.
9
10This page is for MICRO benchmarks measuring CPU performance of small sections of
11code. If you're looking for measuring startup or jank, see the guide for
12MACRObenchmarks [here](/docs/macrobenchmarking.md).
13
14### Writing the benchmark
15
16Benchmarks are just regular instrumentation tests! Just use the
17[`BenchmarkRule`](https://developer.android.com/reference/kotlin/androidx/benchmark/junit4/BenchmarkRule)
18provided by the library:
19
20<section class="tabs">
21
22#### Kotlin {.new-tab}
23
24```kotlin
25@RunWith(AndroidJUnit4::class)
26class ViewBenchmark {
27    @get:Rule
28    val benchmarkRule = BenchmarkRule()
29
30    @Test
31    fun simpleViewInflate() {
32        val context = InstrumentationRegistry
33                .getInstrumentation().targetContext
34        val inflater = LayoutInflater.from(context)
35        val root = FrameLayout(context)
36
37        benchmarkRule.measure {
38            inflater.inflate(R.layout.test_simple_view, root, false)
39        }
40    }
41}
42```
43
44#### Java {.new-tab}
45
46```java
47@RunWith(AndroidJUnit4.class)
48public class ViewBenchmark {
49    @Rule
50    public BenchmarkRule mBenchmarkRule = new BenchmarkRule();
51
52    @Test
53    public void simpleViewInflate() {
54        Context context = InstrumentationRegistry
55                .getInstrumentation().getTargetContext();
56        final BenchmarkState state = mBenchmarkRule.getState();
57        LayoutInflater inflater = LayoutInflater.from(context);
58        FrameLayout root = new FrameLayout(context);
59
60        while (state.keepRunning()) {
61            inflater.inflate(R.layout.test_simple_view, root, false);
62        }
63    }
64}
65```
66
67</section>
68
69## Project structure
70
71As in the public documentation, benchmarks in the AndroidX repo are test-only
72library modules. Differences for AndroidX repo:
73
741.  Module *must* apply `id("androidx.benchmark")` in the plugin block
751.  Module *should* live in `integration-tests` group directory to follow
76    convention
771.  Module name *should* end with `-benchmark` in `settings.gradle` to follow
78    convention
791.  Module *should not* contain non-benchmark tests to avoid wasting resources
80    in benchmark postsubmit
81
82Applying the benchmark plugin give you benefits from the AndroidX plugin:
83
84*   Inclusion in microbenchmark CI runs
85*   AOT Compilation of module (local and CI) for stability
86*   Disable ANR avoidance in local runs (so you always get method traces)
87
88But note that these can be detrimental for non-benchmark code.
89
90### I'm lazy and want to start quickly
91
92Start by copying one of the following non-Compose projects:
93
94*   [navigation-benchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:navigation/navigation-benchmark/)
95*   [recyclerview-benchmark](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:recyclerview/recyclerview-benchmark/)
96
97Many Compose libraries already have benchmark modules:
98
99*   [Compose UI Benchmarks](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/ui/ui/benchmark/)
100*   [Compose Runtime Benchmarks](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/runtime/runtime/compose-runtime-benchmark/)
101*   [Compose Material Benchmarks](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/material/material/benchmark/)
102*   [Wear Compose Material Benchmarks](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:wear/compose/compose-material/benchmark/)
103
104## Profiling
105
106See the
107[public profiling guide](https://developer.android.com/studio/profile/benchmark#profiling)
108for more details.
109
110Jetpack benchmark supports capturing profiling information by setting
111instrumentation arguments. Stack sampling and method tracing can be performed
112either from CLI or Studio invocation.
113
114### Set Arguments in Gradle
115
116Args can be set in your benchmark's `build.gradle`, which will affect both
117Studio / command-line gradlew runs. Runs from Studio will link result traces
118that can be opened directly from the IDE.
119
120```
121android {
122    defaultConfig {
123        // must be one of: 'None', 'StackSampling', or 'MethodTracing'
124        testInstrumentationRunnerArgument 'androidx.benchmark.profiling.mode', 'StackSampling'
125    }
126}
127```
128
129### Set Arguments on Command Line
130
131Args can also be passed from CLI. Here's an example which runs the
132`androidx.compose.material.benchmark.CheckboxesInRowsBenchmark#draw` method with
133`StackSampling` profiling:
134
135```
136./gradlew compose:material:material-benchmark:cC \
137    -P android.testInstrumentationRunnerArguments.androidx.benchmark.profiling.mode=StackSampling \
138    -P android.testInstrumentationRunnerArguments.class=androidx.compose.material.benchmark.CheckboxesInRowsBenchmark#draw
139```
140
141The command output will tell you where to look for the file on your host
142machine:
143
144```
14504:33:49 I/Benchmark: Benchmark report files generated at
146/androidx-main/out/ui/ui/integration-tests/benchmark/build/outputs/connected_android_test_additional_output
147```
148
149To inspect the captured trace, open the appropriate `*.trace` file in that
150directory with Android Studio, using `File > Open`.
151
152NOTE For stack sampling, it's recommended to profile on Android Q(API 29) or
153higher, as this enables the benchmark library to use
154[Simpleperf](https://android.googlesource.com/platform/system/extras/+/master/simpleperf/doc/)
155when capturing samples.
156
157For more information on the `StackSampling` and `MethodTracing` profiling modes,
158see the
159[Studio Profiler recording configuration docs](https://developer.android.com/studio/profile/record-traces#configurations),
160specifically "Sample C/C++ Functions" (called "Callstack sample" in recent
161versions), and Java Method Tracing.
162
163![Sample flame chart](benchmarking_images/profiling_flame_chart.png "Sample flame chart")
164
165### Advanced: Connected Studio Profiler
166
167Profiling for allocations requires Studio to capture, and a debuggable build. Do
168not commit the following changes.
169
170First, set your benchmark to be debuggable in your benchmark module's
171`androidTest/AndroidManifest.xml`:
172
173```
174  <application
175    ...
176    android:debuggable="false"
177    tools:ignore="HardcodedDebugMode"/>
178```
179
180Note that switching to the debug variant will likely not work, as Studio will
181fail to find the benchmark as a test source.
182
183Next select `ConnectedAllocation` in your benchmark module's `build.gradle`:
184
185```
186android {
187    defaultConfig {
188        // --- Local only, don't commit this! ---
189        // pause for manual profiler connection before/after a single run of
190        // the benchmark loop, after warmup
191        testInstrumentationRunnerArgument 'androidx.benchmark.profiling.mode', 'ConnectedAllocation'
192    }
193}
194```
195
196Run `File > Sync Project with Gradle Files`, or sync if Studio asks you. Now any
197benchmark runs in that project will permit debuggable, and pause before and
198after the test, to allow you to connect a profiler and start recording, and then
199stop recording.
200
201#### Running and Profiling
202
203After the benchmark test starts, you have about 20 seconds to connect the
204profiler:
205
2061.  Click the profiler tab at the bottom
2071.  Click the plus button in the top left, `<device name>`, `<process name>`
2081.  Click the memory section, and right click the window, and select `Record
209    allocations`.
2101.  Approximately 20 seconds later, right click again and select `Stop
211    recording`.
212
213If timed correctly, you'll have started and stopped collection around the single
214run of your benchmark loop, and see all allocations in detail with call stacks
215in Studio.
216
217## Minification / R8
218
219As many Android apps don't yet enable R8, the default for microbenchmarks in
220AndroidX is to run with R8 disabled to measure worst-case performance. It may
221still be useful to run your microbenchmarks with R8 enabled locally however, and
222that is supported experimentally. To do this in your microbench module, set the
223**androidTest** minification property:
224
225```
226android {
227    buildTypes.release.androidTest.enableMinification = true
228}
229```
230
231Then, if you see any errors from classes not found at runtime, you can add
232proguard rules
233[here](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/benchmark-utils/proguard-rules.pro),
234or in a similar place for your module.
235