• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# Assembly Tests
2
3The Benchmark library provides a number of functions whose primary
4purpose in to affect assembly generation, including `DoNotOptimize`
5and `ClobberMemory`. In addition there are other functions,
6such as `KeepRunning`, for which generating good assembly is paramount.
7
8For these functions it's important to have tests that verify the
9correctness and quality of the implementation. This requires testing
10the code generated by the compiler.
11
12This document describes how the Benchmark library tests compiler output,
13as well as how to properly write new tests.
14
15
16## Anatomy of a Test
17
18Writing a test has two steps:
19
20* Write the code you want to generate assembly for.
21* Add `// CHECK` lines to match against the verified assembly.
22
23Example:
24```c++
25
26// CHECK-LABEL: test_add:
27extern "C" int test_add() {
28    extern int ExternInt;
29    return ExternInt + 1;
30
31    // CHECK: movl ExternInt(%rip), %eax
32    // CHECK: addl %eax
33    // CHECK: ret
34}
35
36```
37
38#### LLVM Filecheck
39
40[LLVM's Filecheck](https://llvm.org/docs/CommandGuide/FileCheck.html)
41is used to test the generated assembly against the `// CHECK` lines
42specified in the tests source file. Please see the documentation
43linked above for information on how to write `CHECK` directives.
44
45#### Tips and Tricks:
46
47* Tests should match the minimal amount of output required to establish
48correctness. `CHECK` directives don't have to match on the exact next line
49after the previous match, so tests should omit checks for unimportant
50bits of assembly. ([`CHECK-NEXT`](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-next-directive)
51can be used to ensure a match occurs exactly after the previous match).
52
53* The tests are compiled with `-O3 -g0`. So we're only testing the
54optimized output.
55
56* The assembly output is further cleaned up using `tools/strip_asm.py`.
57This removes comments, assembler directives, and unused labels before
58the test is run.
59
60* The generated and stripped assembly file for a test is output under
61`<build-directory>/test/<test-name>.s`
62
63* Filecheck supports using [`CHECK` prefixes](https://llvm.org/docs/CommandGuide/FileCheck.html#cmdoption-check-prefixes)
64to specify lines that should only match in certain situations.
65The Benchmark tests use `CHECK-CLANG` and `CHECK-GNU` for lines that
66are only expected to match Clang or GCC's output respectively. Normal
67`CHECK` lines match against all compilers. (Note: `CHECK-NOT` and
68`CHECK-LABEL` are NOT prefixes. They are versions of non-prefixed
69`CHECK` lines)
70
71* Use `extern "C"` to disable name mangling for specific functions. This
72makes them easier to name in the `CHECK` lines.
73
74
75## Problems Writing Portable Tests
76
77Writing tests which check the code generated by a compiler are
78inherently non-portable. Different compilers and even different compiler
79versions may generate entirely different code. The Benchmark tests
80must tolerate this.
81
82LLVM Filecheck provides a number of mechanisms to help write
83"more portable" tests; including [matching using regular expressions](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-pattern-matching-syntax),
84allowing the creation of [named variables](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-variables)
85for later matching, and [checking non-sequential matches](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-dag-directive).
86
87#### Capturing Variables
88
89For example, say GCC stores a variable in a register but Clang stores
90it in memory. To write a test that tolerates both cases we "capture"
91the destination of the store, and then use the captured expression
92to write the remainder of the test.
93
94```c++
95// CHECK-LABEL: test_div_no_op_into_shr:
96extern "C" void test_div_no_op_into_shr(int value) {
97    int divisor = 2;
98    benchmark::DoNotOptimize(divisor); // hide the value from the optimizer
99    return value / divisor;
100
101    // CHECK: movl $2, [[DEST:.*]]
102    // CHECK: idivl [[DEST]]
103    // CHECK: ret
104}
105```
106
107#### Using Regular Expressions to Match Differing Output
108
109Often tests require testing assembly lines which may subtly differ
110between compilers or compiler versions. A common example of this
111is matching stack frame addresses. In this case regular expressions
112can be used to match the differing bits of output. For example:
113
114```c++
115int ExternInt;
116struct Point { int x, y, z; };
117
118// CHECK-LABEL: test_store_point:
119extern "C" void test_store_point() {
120    Point p{ExternInt, ExternInt, ExternInt};
121    benchmark::DoNotOptimize(p);
122
123    // CHECK: movl ExternInt(%rip), %eax
124    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
125    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
126    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
127    // CHECK: ret
128}
129```
130
131## Current Requirements and Limitations
132
133The tests require Filecheck to be installed along the `PATH` of the
134build machine. Otherwise the tests will be disabled.
135
136Additionally, as mentioned in the previous section, codegen tests are
137inherently non-portable. Currently the tests are limited to:
138
139* x86_64 targets.
140* Compiled with GCC or Clang
141
142Further work could be done, at least on a limited basis, to extend the
143tests to other architectures and compilers (using `CHECK` prefixes).
144
145Furthermore, the tests fail for builds which specify additional flags
146that modify code generation, including `--coverage` or `-fsanitize=`.
147
148