Lines Matching +full:test +full:- +full:suite
28 If you intend to run the :ref:`test-suite <test-suite-overview>`, you will also
29 need a development version of zlib (zlib1g-dev is known to work on several Linux
37 inside the LLVM repository itself under ``llvm/test`` and are expected
38 to always pass -- they should be run before every commit.
40 The whole programs tests are referred to as the "LLVM test suite" (or
41 "test-suite") and are in the ``test-suite`` module in subversion. For
43 tests" in places, which is less ambiguous than "test-suite" and remains
47 ----------------
49 The regression tests are small pieces of code that test a specific
53 are located in the ``llvm/test`` directory.
55 Typically when a bug is found in LLVM, a regression test containing just
60 ``test-suite``
61 --------------
63 The test suite contains whole programs, which are pieces of code which
64 can be compiled and linked into a stand-alone program that can be
78 The test-suite is located in the ``test-suite`` Subversion module.
81 ---------------------------
83 The test suite contains tests to check quality of debugging information.
84 The test are written in C based languages or in LLVM assembly language.
88 test suite for more information . This test suite is located in the
89 ``debuginfo-tests`` Subversion module.
96 ``llvm/test`` (so you get these tests for free with the main LLVM tree).
97 Use ``make check-all`` to run the regression tests after building LLVM.
99 The more comprehensive test suite that includes whole programs in C and C++
100 is in the ``test-suite`` module. See :ref:`test-suite Quickstart
101 <test-suite-quickstart>` for more information on running these tests.
104 ----------------
106 To run all of the LLVM regression tests use the check-llvm target:
108 .. code-block:: bash
110 % make check-llvm
115 .. code-block:: bash
117 % make check-all
122 .. code-block:: bash
124 % make check LIT_ARGS="-v --vg --vg-leak"
128 To run individual tests or subsets of tests, you can use the ``llvm-lit``
130 ``Integer/BitPacked.ll`` test by itself you can run:
132 .. code-block:: bash
134 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
138 .. code-block:: bash
140 % llvm-lit ~/llvm/test/CodeGen/ARM
142 For more information on using the :program:`lit` tool, see ``llvm-lit --help``
146 ---------------------------
149 clang/test directory.
151 .. code-block:: bash
153 % cd clang/test
154 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
158 Regression test structure
162 ``llvm/test`` directory.
166 The directory is broken into several sub-directories, each focused on a
170 ----------------------------
172 The regression test structure is very simple, but does require some
174 and is written to a file, ``test/lit.site.cfg`` in the build directory.
175 The ``llvm/test`` Makefile does this work for you.
187 Each test file must contain lines starting with "RUN:" that tell :program:`lit`
189 while running a test.
191 RUN lines are specified in the comments of the test program using the
194 executes to run the test case. The syntax of the RUN lines is similar to a
214 test case) fails too.
218 .. code-block:: llvm
220 ; RUN: llvm-as < %s | llvm-dis > %t1
221 ; RUN: llvm-dis < %s.bc-13 > %t2
235 The recommended way to examine output to figure out if the test passes is using
237 lines is deprecated - please do not send or commit patches that use it.]*
240 test. Check if there are files already covering your feature and consider
244 -----------
246 If your test requires extra files besides the file containing the ``RUN:``
250 For example, consider ``test/Linker/ident.ll``. The directory structure is
253 test/
262 .. code-block:: llvm
266 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s
270 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2}
271 ; CHECK-DAG: "Compiler V1"
272 ; CHECK-DAG: "Compiler V2"
273 ; CHECK-DAG: "Compiler V3"
287 actually participate in the test besides holding the ``RUN:`` lines.
296 -------------
298 It is easy to write a fragile test that would fail spuriously if the tool being
302 .. code-block:: console
309 $ opt -S /path/to/example.ll
318 .. code-block:: llvm
320 ; RUN: opt -S %s | FileCheck
323 ; CHECK-NOT: load
327 This test will fail if placed into a ``download`` directory.
332 Platform-Specific Tests
333 -----------------------
336 either related to code generated, specific output or back-end features,
338 run on different architectures (and don't even compile all back-ends),
341 The first problem is to check for target-specific output, for example sizes
344 * Tests containing Windows paths will fail on Linux and vice-versa.
348 Also, if the test rely on any behaviour that is coded in any back-end, it must
350 into ``test/CodeGen/ARM`` and so on. Those directories contain a special
352 only run if a specific back-end is compiled and available.
354 For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is:
356 .. code-block:: python
358 config.suffixes = ['.ll', '.c', '.cpp', '.test']
362 Other platform-specific tests are those that depend on a specific feature
363 of a specific sub-architecture, for example only to Intel chips that support ``AVX2``.
365 For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture
368 .. code-block:: llvm
370 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2
371 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1
372 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2
376 .. code-block:: llvm
385 So, if you're testing for a behaviour that you know is platform-specific or
386 depends on special features of sub-architectures, you must add the specific
387 triple, test with the specific FileCheck and put it into the specific
390 REQUIRES and REQUIRES-ANY directive
393 Some tests can be enabled only in specific situation - like having
396 .. code-block:: llvm
398 ; This test will be only enabled in the build with asserts
403 ``REQUIRES-ANY`` means at least one must be satisfied.
405 List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
409 -------------
418 File path to the test case's source. This is suitable for passing on the
421 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s``
424 Directory path to the test case's source.
426 Example: ``/home/user/llvm/test/MC/ELF``
429 File path to a temporary file name that could be used for this test case.
430 The file name won't conflict with other test cases. You can append to it
434 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
439 Example: ``/home/user/llvm.build/test/MC/ELF/Output``
446 **LLVM-specific substitutions:**
460 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
463 reference test file's line numbers.
466 **Clang-specific substitutions:**
475 Invokes the CL-compatible Clang driver.
478 Invokes the G++-compatible Clang driver.
485 the desired ABI. For example, if the test suite is running with the
486 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to
487 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without
490 To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``.
494 -------
503 % llvm-lit "-Dllc=llc -verify-machineinstrs"
513 --------------
521 Zero result codes become 1. Non-zero result codes become 0.
523 Sometimes it is necessary to mark a test case as "expected fail" or
524 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
525 on a line near the top of the file. This signals that the test case
526 should succeed if the test fails. Such test cases are counted separately
528 in the comments of the test program followed by a colon and one or more
530 fail everywhere), or a part of a target triple (indicating the test
532 (for example, ``loadable_module``). If there is a match, the test is
533 expected to fail. If not, the test is expected to succeed. To XFAIL
537 .. code-block:: llvm
542 the lines of the test case for ones that contain a pattern that matches
543 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
544 that is related to the test case. The number after "PR" specifies the
547 a test fails.
553 (a) it prevents special interpretation of lines that are part of the test
554 program, not the instructions to the test case, and
556 (b) it speeds things up for really big test cases by avoiding
559 .. _test-suite-overview:
561 ``test-suite`` Overview
564 The ``test-suite`` module contains a number of programs that can be
565 compiled and executed. The ``test-suite`` includes reference outputs for
569 ``test-suite`` tests are divided into three types of tests: MultiSource,
572 - ``test-suite/SingleSource``
574 The SingleSource directory contains test programs that are only a
579 - ``test-suite/MultiSource``
585 - ``test-suite/External``
593 ``--test-externals`` option to include these tests in the results.
595 .. _test-suite-quickstart:
597 ``test-suite`` Quickstart
598 -------------------------
600 The modern way of running the ``test-suite`` is focused on testing and
604 For more information on using LNT to execute the ``test-suite``, please
608 ``test-suite`` Makefiles
609 ------------------------
611 Historically, the ``test-suite`` was executed using a complicated setup
618 For more information on the ``test-suite`` Makefile setup, please see
619 the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.