Lines Matching +full:run +full:- +full:tests
18 infrastructure, the tools needed to use it, and how to add and run
19 tests.
28 If you intend to run the :ref:`test-suite <test-suite-overview>`, you will also
29 need a development version of zlib (zlib1g-dev is known to work on several Linux
35 The LLVM testing infrastructure contains two major categories of tests:
36 regression tests and whole programs. The regression tests are contained
38 to always pass -- they should be run before every commit.
40 The whole programs tests are referred to as the "LLVM test suite" (or
41 "test-suite") and are in the ``test-suite`` module in subversion. For
42 historical reasons, these tests are also referred to as the "nightly
43 tests" in places, which is less ambiguous than "test-suite" and remains
44 in use although we run them much more often than nightly.
46 Regression tests
47 ----------------
49 The regression tests are small pieces of code that test a specific
51 written in depends on the part of LLVM being tested. These tests are driven by
60 ``test-suite``
61 --------------
64 can be compiled and linked into a stand-alone program that can be
73 In addition to compiling and executing programs, whole program tests
78 The test-suite is located in the ``test-suite`` Subversion module.
80 Debugging Information tests
81 ---------------------------
83 The test suite contains tests to check quality of debugging information.
86 These tests are compiled and run under a debugger. The debugger output
89 ``debuginfo-tests`` Subversion module.
94 The tests are located in two separate Subversion modules. The
95 regressions tests are in the main "llvm" module under the directory
96 ``llvm/test`` (so you get these tests for free with the main LLVM tree).
97 Use ``make check-all`` to run the regression tests after building LLVM.
100 is in the ``test-suite`` module. See :ref:`test-suite Quickstart
101 <test-suite-quickstart>` for more information on running these tests.
103 Regression tests
104 ----------------
106 To run all of the LLVM regression tests use the check-llvm target:
108 .. code-block:: bash
110 % make check-llvm
113 can run the LLVM and Clang tests simultaneously using:
115 .. code-block:: bash
117 % make check-all
119 To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make
122 .. code-block:: bash
124 % make check LIT_ARGS="-v --vg --vg-leak"
128 To run individual tests or subsets of tests, you can use the ``llvm-lit``
129 script which is built as part of LLVM. For example, to run the
130 ``Integer/BitPacked.ll`` test by itself you can run:
132 .. code-block:: bash
134 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
136 or to run all of the ARM CodeGen tests:
138 .. code-block:: bash
140 % llvm-lit ~/llvm/test/CodeGen/ARM
142 For more information on using the :program:`lit` tool, see ``llvm-lit --help``
145 Debugging Information tests
146 ---------------------------
148 To run debugging information tests simply checkout the tests inside
151 .. code-block:: bash
154 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
156 These tests are already set up to run as part of clang regression tests.
161 The LLVM regression tests are driven by :program:`lit` and are located in the
164 This directory contains a large array of small tests that exercise
166 The directory is broken into several sub-directories, each focused on a
169 Writing new regression tests
170 ----------------------------
177 In order for the regression tests to work, each directory of tests must
179 how to run the tests. This file is just Python code and thus is very
180 flexible, but we've standardized it for the LLVM regression tests. If
181 you're adding a directory of tests, just copy ``lit.local.cfg`` from
183 specifies which files to look in for tests. Any directory that contains
187 Each test file must contain lines starting with "RUN:" that tell :program:`lit`
188 how to run it. If there are no RUN lines, :program:`lit` will issue an error
191 RUN lines are specified in the comments of the test program using the
192 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
194 executes to run the test case. The syntax of the RUN lines is similar to a
197 script, they are not. RUN lines are interpreted by :program:`lit`.
199 as many RUN lines as needed.
201 :program:`lit` performs substitution on each RUN line to replace LLVM tool names
206 Each RUN line is executed on its own, distinct from other lines unless
207 its last character is ``\``. This continuation character causes the RUN
210 ending in ``\`` are concatenated until a RUN line that doesn't end in
211 ``\`` is found. This concatenated set of RUN lines then constitutes one
216 Below is an example of legal RUN lines in a ``.ll`` file:
218 .. code-block:: llvm
220 ; RUN: llvm-as < %s | llvm-dis > %t1
221 ; RUN: llvm-dis < %s.bc-13 > %t2
222 ; RUN: diff %t1 %t2
224 As with a Unix shell, the RUN lines permit pipelines and I/O
228 your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
233 In general, you should strive to keep your RUN lines as simple as possible,
234 using them only to run tools that generate textual output you can then examine.
236 the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN
237 lines is deprecated - please do not send or commit patches that use it.]*
239 Put related tests into a single file rather than having a separate file per
244 -----------
246 If your test requires extra files besides the file containing the ``RUN:``
262 .. code-block:: llvm
266 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
270 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
271 ; CHECK-DAG: "Compiler V1"
272 ; CHECK-DAG: "Compiler V2"
273 ; CHECK-DAG: "Compiler V3"
287 actually participate in the test besides holding the ``RUN:`` lines.
291 Some existing tests use ``RUN: true`` in extra files instead of just
295 Fragile tests
296 -------------
302 .. code-block:: console
309 $ opt -S /path/to/example.ll
318 .. code-block:: llvm
320 ; RUN: opt -S %s | FileCheck
323 ; CHECK-NOT: load
329 To make your tests robust, always use ``opt ... < %s`` in the RUN line.
332 Platform-Specific Tests
333 -----------------------
335 Whenever adding tests that require the knowledge of a specific platform,
336 either related to code generated, specific output or back-end features,
338 run on different architectures (and don't even compile all back-ends),
341 The first problem is to check for target-specific output, for example sizes
344 * Tests containing Windows paths will fail on Linux and vice-versa.
345 * Tests that check for ``x86_64`` somewhere in the text will fail anywhere else.
346 * Tests where the debug information calculates the size of types and structures.
348 Also, if the test rely on any behaviour that is coded in any back-end, it must
349 go in its own directory. So, for instance, code generator tests for ARM go
351 ``lit`` configuration file that ensure all tests in that directory will
352 only run if a specific back-end is compiled and available.
356 .. code-block:: python
362 Other platform-specific tests are those that depend on a specific feature
363 of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
365 For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
368 .. code-block:: llvm
370 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
371 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
372 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
376 .. code-block:: llvm
385 So, if you're testing for a behaviour that you know is platform-specific or
386 depends on special features of sub-architectures, you must add the specific
390 REQUIRES and REQUIRES-ANY directive
393 Some tests can be enabled only in specific situation - like having
396 .. code-block:: llvm
403 ``REQUIRES-ANY`` means at least one must be satisfied.
405 List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
409 -------------
412 RUN lines:
446 **LLVM-specific substitutions:**
460 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
462 integer offset. This can be used in tests with multiple RUN lines, which
466 **Clang-specific substitutions:**
475 Invokes the CL-compatible Clang driver.
478 Invokes the G++-compatible Clang driver.
486 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
487 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
494 -------
503 % llvm-lit "-Dllc=llc -verify-machineinstrs"
506 Enable the execution of long running tests.
513 --------------
515 To make RUN line writing easier, there are several helper programs. These
516 helpers are in the PATH when running tests, so you can just call them using
521 Zero result codes become 1. Non-zero result codes become 0.
537 .. code-block:: llvm
543 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
551 the last RUN: line. This has two side effects:
559 .. _test-suite-overview:
561 ``test-suite`` Overview
564 The ``test-suite`` module contains a number of programs that can be
565 compiled and executed. The ``test-suite`` includes reference outputs for
569 ``test-suite`` tests are divided into three types of tests: MultiSource,
572 - ``test-suite/SingleSource``
579 - ``test-suite/MultiSource``
585 - ``test-suite/External``
591 tests, but only the Makefiles that know how to properly compile these
593 ``--test-externals`` option to include these tests in the results.
595 .. _test-suite-quickstart:
597 ``test-suite`` Quickstart
598 -------------------------
600 The modern way of running the ``test-suite`` is focused on testing and
604 For more information on using LNT to execute the ``test-suite``, please
608 ``test-suite`` Makefiles
609 ------------------------
611 Historically, the ``test-suite`` was executed using a complicated setup
618 For more information on the ``test-suite`` Makefile setup, please see