1# MACRObenchmarking in AndroidX
2
3[TOC]
4
5<!-- Copied from macrobenchmark docs -->
6
7<table>
8    <tr>
9      <td><strong>Macrobenchmark</strong></td>
10      <td><strong>Benchmark</strong></td>
11    </tr>
12    <tr>
13        <td>Measure high-level entry points (Activity launch / Scrolling a list)</td>
14        <td>Measure individual functions</td>
15    </tr>
16    <tr>
17        <td>Out-of-process test of full app</td>
18        <td>In-process test of CPU work</td>
19    </tr>
20    <tr>
21        <td>Slow iteration speed (Often more than a minute)</td>
22        <td>Fast iteration speed (Often less than 10 seconds)</td>
23    </tr>
24    <tr>
25        <td>Configure compilation with <a href="https://developer.android.com/reference/androidx/benchmark/macro/CompilationMode">CompilationMode</a></td>
26        <td>Always fully AOT (<code>speed</code>) compiled.</td>
27    </tr>
28    <tr>
29        <td>Relatively lower stability measurements</td>
30        <td>Higher stability measurements</td>
31    </tr>
32</table>
33
34The
35[public documentation](https://developer.android.com/studio/profile/macrobenchmark)
36for macrobenchmark explains how to use the library. This page focuses on
37specifics to writing library macrobenchmarks in the AndroidX repo. If you're
38looking for measuring CPU perf of individual functions, see the guide for
39MICRObenchmarks [here](/docs/benchmarking.md).
40
41### Writing the benchmark
42
43Benchmarks are just regular instrumentation tests! Just use the
44[`MacrobenchmarkRule`](https://developer.android.com/reference/kotlin/androidx/benchmark/macro/junit4/MacrobenchmarkRule)
45provided by the library:
46
47<section class="tabs">
48
49#### Kotlin {.new-tab}
50
51```kotlin
52    @get:Rule
53    val benchmarkRule = MacrobenchmarkRule()
54
55    @Test
56    fun startup() = benchmarkRule.measureRepeated(
57        packageName = "mypackage.myapp",
58        metrics = listOf(StartupTimingMetric()),
59        startupMode = StartupMode.COLD,
60        iterations = 10
61    ) { // this = MacrobenchmarkScope
62        pressHome()
63        val intent = Intent()
64        intent.setPackage("mypackage.myapp")
65        intent.setAction("mypackage.myapp.myaction")
66        startActivityAndWait(intent)
67    }
68```
69
70#### Java {.new-tab}
71
72```java
73    @Rule
74    public MacrobenchmarkRule mBenchmarkRule = MacrobenchmarkRule()
75
76    @Test
77    public void startup() {
78        mBenchmarkRule.measureRepeated(
79                "mypackage.myapp",
80                Collections.singletonList(new StartupTimingMetric()),
81                StartupMode.COLD,
82                /* iterations = */ 10,
83                scope -> {
84                    scope.pressHome();
85                    Intent intent = Intent();
86                    intent.setPackage("mypackage.myapp");
87                    intent.setAction("mypackage.myapp.myaction");
88                    scope.startActivityAndWait(intent);
89                    return Unit.INSTANCE;
90                }
91        );
92    }
93```
94
95</section>
96
97## Project structure
98
99As in the public documentation, macrobenchmarks in the AndroidX repo are
100comprised of an app, and a separate macrobenchmark test module. In the AndroidX
101repository, there are additional requirements:
102
1031.  Macrobenchmark test module path in `settings.gradle` **must** end with
104    `macrobenchmark` to run in CI.
105
1061.  Macrobenchmark target module path in `settings.gradle` **should** end with
107    `macrobenchmark-target` to follow convention.
108
1091.  Macrobenchmark modules **must** use project dependencies where available (so
110    `implementation(project(":activity:activity-ktx"))` rather than
111    `implementation("androidx.activity:activity-ktx:1.5.0")`). This prevents
112    accidentally testing against out-of-date versions, and increases coverage of
113    lower level libraries.
114
1151.  Each library group **must** declare its own in-group macrobenchmark test and
116    app module, with out using these modules for anything else (e.g. samples).
117    We want to be intentional about which changes affect measurements. More than
118    one of either is allowed, which is sometimes necessary to compare different
119    startup behaviors, see e.g.
120    `:emoji2:integration-tests:init-<disabled/enabled>-macrobenchmark-target`.
121    Note that comparing multiple app variants are not currently supported by CI.
122
123Compose Macrobenchmark Examples:
124
125*   [`:compose:integration-tests:macrobenchmark-target`](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/macrobenchmark-target/)
126
127*   [`:compose:integration-tests:macrobenchmark`](https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:compose/integration-tests/macrobenchmark/)
128
129Note: Compose macrobenchmarks are ideally duplicated with View system
130counterparts, defined in `:benchmark:integration-tests:macrobenchmark-target`.
131This is how we compare performance of the two systems.
132
133### Setup checklist
134
135<table>
136    <tr>
137      <td><strong>Did you setup...</strong></td>
138      <td><strong>Required setup</strong></td>
139    </tr>
140    <tr>
141        <td>Two modules in <code>settings.gradle</code></td>
142        <td>Both the macrobenchmark and target must be added for your group</td>
143    </tr>
144    <tr>
145        <td>The module name for the benchmark (<code>com.android.test</code>) module</td>
146        <td>Must end with <code>macrobenchmark</code></td>
147    </tr>
148    <tr>
149        <td>The module name for the app (<code>com.android.app</code>) module</td>
150        <td>Must end with <code>macrobenchmark-target</code></td>
151    </tr>
152    <tr>
153        <td>Name the test class in a discoverable way</td>
154        <td>Test classes should have standalone names for easy discovery in the
155          web UI. E.g EmojiStartupTest instead of StartupTest.</td>
156    </tr>
157</table>
158