Lines Matching +full:cross +full:- +full:ios +full:- +full:build
1 test-suite Guide
5 ----------
8 from an LLVM build:
11 % <path to llvm build>/bin/llvm-lit --version
22 % pip install svn+https://llvm.org/svn/llvm-project/llvm/trunk/utils/lit
23 % lit --version
27 2. Check out the `test-suite` module with:
30 % git clone https://github.com/llvm/llvm-test-suite.git test-suite
33 3. Create a build directory and use CMake to configure the suite. Use the
35 to choose a typical build configuration:
38 % mkdir test-suite-build
39 % cd test-suite-build
40 % cmake -DCMAKE_C_COMPILER=<path to llvm build>/bin/clang \
41 -C../test-suite/cmake/caches/O3.cmake \
42 ../test-suite
45 4. Build the benchmarks:
49 Scanning dependencies of target timeit-target
50 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
51 [ 0%] Linking C executable timeit-target
58 % llvm-lit -v -j 1 -o results.json .
59 -- Testing: 474 tests, 1 threads --
60 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
61 …********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESUL…
67 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
77 % test-suite/utils/compare.py results.json
79 % test-suite/utils/compare.py results_a.json results_b.json
84 ---------
86 The test-suite contains benchmark and test programs. The programs come with
91 The test-suite is divided into several directories:
93 - `SingleSource/`
98 - `MultiSource/`
103 - `MicroBenchmarks/`
105 Programs using the [google-benchmark](https://github.com/google/benchmark)
109 - `External/`
112 distributed with the test-suite. The most prominent members of this
114 See [External Suites](#external-suites).
116 - `Bitcode/`
120 - `CTMark/`
133 -------------
135 The test-suite has configuration options to customize building and running the
139 % cd test-suite-build
141 % cmake -LH
143 % cmake -LAH
148 - `CMAKE_C_FLAGS`
154 - `CMAKE_C_COMPILER`
161 - `CMAKE_BUILD_TYPE`
163 Select a build type like `OPTIMIZE` or `DEBUG` selecting a set of predefined
168 - `TEST_SUITE_RUN_UNDER`
171 cross-compiled tests within a simulator tool.
173 - `TEST_SUITE_BENCHMARKING_ONLY`
179 - `TEST_SUITE_SUBDIRS`
181 Semicolon-separated list of directories to include. This can be used to only
182 build parts of the test-suite or to include external suites. This option
186 - `TEST_SUITE_COLLECT_STATS`
188 Collect internal LLVM statistics. Appends `-save-stats=obj` when invoking the
191 - `TEST_SUITE_RUN_BENCHMARKS`
194 collect build statistics like compile time and code size.
196 - `TEST_SUITE_USE_PERF`
199 comes with the test-suite. The `perf` is usually available on linux systems.
201 - `TEST_SUITE_SPEC2000_ROOT`, `TEST_SUITE_SPEC2006_ROOT`, `TEST_SUITE_SPEC2017_ROOT`, ...
209 - `-GNinja`
211 Generate build files for the ninja build tool.
213 - `-Ctest-suite/cmake/caches/<cachefile.cmake>`
215 Use a CMake cache. The test-suite comes with several CMake caches which
216 predefine common or tricky build configurations.
220 --------------------------------
223 produced when invoking lit with the `-o filename.json` flag.
227 - Basic Usage:
230 % test-suite/utils/compare.py baseline.json
231 Warning: 'test-suite :: External/SPEC/CINT2006/403.gcc/403.gcc.test' has No metrics!
251 - Show compile_time or text segment size metrics:
254 % test-suite/utils/compare.py -m compile_time baseline.json
255 % test-suite/utils/compare.py -m size.__text baseline.json
258 - Compare two result files and filter short running tests:
261 % test-suite/utils/compare.py --filter-short baseline.json experiment.json
265 SingleSour.../Benchmarks/Linpack/linpack-pc 5.16 4.30 -16.5%
266 MultiSourc...erolling-dbl/LoopRerolling-dbl 7.01 7.86 12.2%
267 SingleSour...UnitTests/Vectorizer/gcc-loops 3.89 3.54 -9.0%
271 - Merge multiple baseline and experiment result files by taking the minimum
275 % test-suite/utils/compare.py base0.json base1.json base2.json vs exp0.json exp1.json exp2.json
287 ---------------
291 - placing (or linking) them into the `test-suite/test-suite-externals/xxx` directory (example: `tes…
292 - using a configuration option such as `-D TEST_SUITE_SPEC2000_ROOT=path/to/speccpu2000`
295 `test-suite/External/SPEC/README`.
303 -------------
305 You can build custom suites using the test-suite infrastructure. A custom suite
307 picked up automatically if placed into a subdirectory of the test-suite or when
311 % cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite
316 ---------------------------
321 using `llvm-profdata` so they can be used by the second compilation run.
326 % cmake -DTEST_SUITE_PROFILE_GENERATE=ON \
327 -DTEST_SUITE_RUN_TYPE=train \
328 ../test-suite
330 % llvm-lit .
332 % cmake -DTEST_SUITE_PROFILE_GENERATE=OFF \
333 -DTEST_SUITE_PROFILE_USE=ON \
334 -DTEST_SUITE_RUN_TYPE=ref \
337 % llvm-lit -o result.json .
343 Cross Compilation and External Devices
344 --------------------------------------
348 CMake allows to cross compile to a different target via toolchain files. More
351 - [https://llvm.org/docs/lnt/tests.html#cross-compiling](https://llvm.org/docs/lnt/tests.html#cross…
353 - [https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html](https://cmake.org/cmake/help…
355 Cross compilation from macOS to iOS is possible with the
356 `test-suite/cmake/caches/target-target-*-iphoneos-internal.cmake` CMake cache
357 files; this requires an internal iOS SDK.
361 There are two ways to run the tests in a cross compilation setting:
363 - Via SSH connection to an external device: The `TEST_SUITE_REMOTE_HOST` option
373 % cmake -G Ninja -D CMAKE_C_COMPILER=path/to/clang \
374 -C ../test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake \
375 -D TEST_SUITE_REMOTE_HOST=mydevice \
376 ../test-suite
379 % llvm-lit -j1 -o result.json .
382 - You can specify a simulator for the target machine with the
387 Running the test-suite via LNT
388 ------------------------------
390 The LNT tool can run the test-suite. Use this when submitting test results to
392 …ttps://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite](https://llvm.org/docs/lnt/tests.html#ll…
395 Running the test-suite via Makefiles (deprecated)
396 -------------------------------------------------
398 **Note**: The test-suite comes with a set of Makefiles that are considered
403 [test-suite Makefile Guide](TestSuiteMakefileGuide).