Lines Matching +full:test +full:- +full:suite
2 LLVM test-suite Guide
11 This document describes the features of the Makefile-based LLVM
12 test-suite as well as the cmake based replacement. This way of interacting
13 with the test-suite is deprecated in favor of running the test-suite using LNT,
15 Guide's :ref:`test-suite Quickstart <test-suite-quickstart>` section for more
18 Test suite Structure
21 The ``test-suite`` module contains a number of programs that can be
29 subset of the available tests or programs. This makes test run times
31 test failures. To run some test only on a subset of programs, simply
33 there. Alternatively, you can run a different test using the ``TEST``
37 In addition for testing correctness, the ``test-suite`` directory also
43 ``test-suite`` tests are divided into three types of tests: MultiSource,
46 - ``test-suite/SingleSource``
48 The SingleSource directory contains test programs that are only a
53 - ``test-suite/MultiSource``
59 - ``test-suite/External``
67 external programs is configured by the test-suite ``configure``
81 The tests in the test suite have no such feature at this time. If the
82 test passes, only warnings and other miscellaneous output will be
83 generated. If a test fails, a large <program> FAILED message will be
84 displayed. This will help you separate benign warnings from actual test
87 Running the test suite via CMake
90 To run the test suite, you need to use the following steps:
92 #. The test suite uses the lit test runner to run the test-suite,
95 .. code-block:: bash
97 % svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
99 % sudo python setup.py install # Or without sudo, install in virtual-env.
103 writing lit.egg-info/PKG-INFO
105 % lit --version
108 #. Check out the ``test-suite`` module with:
110 .. code-block:: bash
112 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
114 #. Use CMake to configure the test suite in a new directory. You cannot build
115 the test suite in the source tree.
117 .. code-block:: bash
119 % mkdir test-suite-build
120 % cd test-suite-build
121 % cmake ../test-suite
125 .. code-block:: bash
128 Scanning dependencies of target timeit-target
129 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
130 [ 0%] Linking C executable timeit-target
131 [ 0%] Built target timeit-target
132 Scanning dependencies of target fpcmp-host
134 [ 0%] Built target fpcmp-host
135 Scanning dependencies of target timeit-host
137 [ 0%] Built target timeit-host
142 .. code-block:: bash
144 % lit -v -j 1 . -o results.json
145 -- Testing: 474 tests, 1 threads --
146 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
147 …********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESUL…
153 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
156 Running the test suite via Makefiles (deprecated)
161 the test suite creates temporary files during execution.
163 To run the test suite, you need to use the following steps:
166 #. Check out the ``test-suite`` module with:
168 .. code-block:: bash
170 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
172 This will get the test suite into ``llvm/projects/test-suite``.
176 #. Configure and build ``llvm-gcc``.
178 #. Install ``llvm-gcc`` somewhere.
180 #. *Re-configure* ``llvm`` from the top level of each build tree (LLVM
181 object directory tree) in which you want to run the test suite, just
184 During the *re-configuration*, you must either: (1) have ``llvm-gcc``
186 just-built ``llvm-gcc`` is installed using
187 ``--with-llvmgccdir=$LLVM_GCC_DIR``.
189 You must also tell the configure machinery that the test suite is
192 .. code-block:: bash
194 % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]
197 *installed* llvm-gcc, not its src or obj directory.]
199 #. You can now run the test suite from your build tree as follows:
201 .. code-block:: bash
203 % cd $LLVM_OBJ_ROOT/projects/test-suite
207 you have the suite checked out and configured, you don't need to do it
208 again (unless the test code or configure script changes).
211 --------------------------
213 In order to run the External tests in the ``test-suite`` module, you
214 must specify *--with-externals*. This must be done during the
215 *re-configuration* step (see above), and the ``llvm`` re-configuration
216 must recognize the previously-built ``llvm-gcc``. If any of these is
219 * *--with-externals*
221 * *--with-externals=<directory>*
241 -----------------------
243 In addition to the regular "whole program" tests, the ``test-suite``
245 ways. If the variable TEST is defined on the ``gmake`` command line, the
246 test system will include a Makefile named
247 ``TEST.<value of TEST variable>.Makefile``. This Makefile can modify
250 For example, the LLVM nightly tester uses ``TEST.nightly.Makefile`` to
251 create the nightly test reports. To run the nightly tests, run
252 ``gmake TEST=nightly``.
254 There are several TEST Makefiles available in the tree. Some of them are
257 writing your own TEST Makefile for any optimization or analysis passes
260 Generating test output
261 ----------------------
270 Somewhat better is running ``gmake TEST=sometest test``, which runs the
271 specified test and usually adds per-program summaries to the output
272 (depending on which sometest you use). For example, the ``nightly`` test
273 explicitly outputs TEST-PASS or TEST-FAIL for every test after each
279 exact contents of the report are dependent on which ``TEST`` you are
282 running with ``TEST=<type>``). The ``report`` also generate a file
284 test run.
286 Writing custom tests for the test suite
287 ---------------------------------------
289 Assuming you can run the test suite, (e.g.
290 "``gmake TEST=nightly report``" should work), it is really easy to run
301 Following this, you can set up a test and a report that collects these
303 "``test-suite/TEST.XXX.Makefile``" fragment (where XXX is the name of
304 your test) and a "``test-suite/TEST.XXX.report``" file that indicates
306 various levels of sophistication included with the test suite, and the
310 "libcalls" test as an example. It can be run like this:
312 .. code-block:: bash
314 % cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level
315 % make TEST=libcalls report
330 Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
331 Prolangs-C/agrep/agrep | 33 | 12 |
332 Prolangs-C/allroots/allroots | * | * |
333 Prolangs-C/assembler/assembler | 47 | * |
334 Prolangs-C/bison/mybison | 74 | * |
337 This basically is grepping the -stats output and displaying it in a
338 table. You can also use the "TEST=libcalls report.html" target to get
341 The source for this is in ``test-suite/TEST.libcalls.*``. The format is
342 pretty simple: the Makefile indicates how to run the test (in this case,
343 "``opt -simplify-libcalls -stats``"), and the report contains one line