1lit - LLVM Integrated Tester 2============================ 3 4SYNOPSIS 5-------- 6 7:program:`lit` [*options*] [*tests*] 8 9DESCRIPTION 10----------- 11 12:program:`lit` is a portable tool for executing LLVM and Clang style test 13suites, summarizing their results, and providing indication of failures. 14:program:`lit` is designed to be a lightweight testing tool with as simple a 15user interface as possible. 16 17:program:`lit` should be run with one or more *tests* to run specified on the 18command line. Tests can be either individual test files or directories to 19search for tests (see :ref:`test-discovery`). 20 21Each specified test will be executed (potentially in parallel) and once all 22tests have been run :program:`lit` will print summary information on the number 23of tests which passed or failed (see :ref:`test-status-results`). The 24:program:`lit` program will execute with a non-zero exit code if any tests 25fail. 26 27By default :program:`lit` will use a succinct progress display and will only 28print summary information for test failures. See :ref:`output-options` for 29options controlling the :program:`lit` progress display and output. 30 31:program:`lit` also includes a number of options for controlling how tests are 32executed (specific features may depend on the particular test format). See 33:ref:`execution-options` for more information. 34 35Finally, :program:`lit` also supports additional options for only running a 36subset of the options specified on the command line, see 37:ref:`selection-options` for more information. 38 39Users interested in the :program:`lit` architecture or designing a 40:program:`lit` testing implementation should see :ref:`lit-infrastructure`. 41 42GENERAL OPTIONS 43--------------- 44 45.. option:: -h, --help 46 47 Show the :program:`lit` help message. 48 49.. option:: -j N, --threads=N 50 51 Run ``N`` tests in parallel. By default, this is automatically chosen to 52 match the number of detected available CPUs. 53 54.. option:: --config-prefix=NAME 55 56 Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for 57 test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`. 58 59.. option:: --param NAME, --param NAME=VALUE 60 61 Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty 62 string if not given). The meaning and use of these parameters is test suite 63 dependent. 64 65.. _output-options: 66 67OUTPUT OPTIONS 68-------------- 69 70.. option:: -q, --quiet 71 72 Suppress any output except for test failures. 73 74.. option:: -s, --succinct 75 76 Show less output, for example don't show information on tests that pass. 77 78.. option:: -v, --verbose 79 80 Show more information on test failures, for example the entire test output 81 instead of just the test result. 82 83.. option:: --no-progress-bar 84 85 Do not use curses based progress bar. 86 87.. _execution-options: 88 89EXECUTION OPTIONS 90----------------- 91 92.. option:: --path=PATH 93 94 Specify an additional ``PATH`` to use when searching for executables in tests. 95 96.. option:: --vg 97 98 Run individual tests under valgrind (using the memcheck tool). The 99 ``--error-exitcode`` argument for valgrind is used so that valgrind failures 100 will cause the program to exit with a non-zero status. 101 102 When this option is enabled, :program:`lit` will also automatically provide a 103 "``valgrind``" feature that can be used to conditionally disable (or expect 104 failure in) certain tests. 105 106.. option:: --vg-arg=ARG 107 108 When :option:`--vg` is used, specify an additional argument to pass to 109 :program:`valgrind` itself. 110 111.. option:: --vg-leak 112 113 When :option:`--vg` is used, enable memory leak checks. When this option is 114 enabled, :program:`lit` will also automatically provide a "``vg_leak``" 115 feature that can be used to conditionally disable (or expect failure in) 116 certain tests. 117 118.. option:: --time-tests 119 120 Track the wall time individual tests take to execute and includes the results 121 in the summary output. This is useful for determining which tests in a test 122 suite take the most time to execute. Note that this option is most useful 123 with ``-j 1``. 124 125.. _selection-options: 126 127SELECTION OPTIONS 128----------------- 129 130.. option:: --max-tests=N 131 132 Run at most ``N`` tests and then terminate. 133 134.. option:: --max-time=N 135 136 Spend at most ``N`` seconds (approximately) running tests and then terminate. 137 138.. option:: --shuffle 139 140 Run the tests in a random order. 141 142ADDITIONAL OPTIONS 143------------------ 144 145.. option:: --debug 146 147 Run :program:`lit` in debug mode, for debugging configuration issues and 148 :program:`lit` itself. 149 150.. option:: --show-suites 151 152 List the discovered test suites as part of the standard output. 153 154.. option:: --repeat=N 155 156 Run each test ``N`` times. Currently this is primarily useful for timing 157 tests, other results are not collated in any reasonable fashion. 158 159EXIT STATUS 160----------- 161 162:program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS 163results. Otherwise, it will exit with the status 0. Other exit codes are used 164for non-test related failures (for example a user error or an internal program 165error). 166 167.. _test-discovery: 168 169TEST DISCOVERY 170-------------- 171 172The inputs passed to :program:`lit` can be either individual tests, or entire 173directories or hierarchies of tests to run. When :program:`lit` starts up, the 174first thing it does is convert the inputs into a complete list of tests to run 175as part of *test discovery*. 176 177In the :program:`lit` model, every test must exist inside some *test suite*. 178:program:`lit` resolves the inputs specified on the command line to test suites 179by searching upwards from the input path until it finds a :file:`lit.cfg` or 180:file:`lit.site.cfg` file. These files serve as both a marker of test suites 181and as configuration files which :program:`lit` loads in order to understand 182how to find and run the tests inside the test suite. 183 184Once :program:`lit` has mapped the inputs into test suites it traverses the 185list of inputs adding tests for individual files and recursively searching for 186tests in directories. 187 188This behavior makes it easy to specify a subset of tests to run, while still 189allowing the test suite configuration to control exactly how tests are 190interpreted. In addition, :program:`lit` always identifies tests by the test 191suite they are in, and their relative path inside the test suite. For 192appropriately configured projects, this allows :program:`lit` to provide 193convenient and flexible support for out-of-tree builds. 194 195.. _test-status-results: 196 197TEST STATUS RESULTS 198------------------- 199 200Each test ultimately produces one of the following six results: 201 202**PASS** 203 204 The test succeeded. 205 206**XFAIL** 207 208 The test failed, but that is expected. This is used for test formats which allow 209 specifying that a test does not currently work, but wish to leave it in the test 210 suite. 211 212**XPASS** 213 214 The test succeeded, but it was expected to fail. This is used for tests which 215 were specified as expected to fail, but are now succeeding (generally because 216 the feature they test was broken and has been fixed). 217 218**FAIL** 219 220 The test failed. 221 222**UNRESOLVED** 223 224 The test result could not be determined. For example, this occurs when the test 225 could not be run, the test itself is invalid, or the test was interrupted. 226 227**UNSUPPORTED** 228 229 The test is not supported in this environment. This is used by test formats 230 which can report unsupported tests. 231 232Depending on the test format tests may produce additional information about 233their status (generally only for failures). See the :ref:`output-options` 234section for more information. 235 236.. _lit-infrastructure: 237 238LIT INFRASTRUCTURE 239------------------ 240 241This section describes the :program:`lit` testing architecture for users interested in 242creating a new :program:`lit` testing implementation, or extending an existing one. 243 244:program:`lit` proper is primarily an infrastructure for discovering and running 245arbitrary tests, and to expose a single convenient interface to these 246tests. :program:`lit` itself doesn't know how to run tests, rather this logic is 247defined by *test suites*. 248 249TEST SUITES 250~~~~~~~~~~~ 251 252As described in :ref:`test-discovery`, tests are always located inside a *test 253suite*. Test suites serve to define the format of the tests they contain, the 254logic for finding those tests, and any additional information to run the tests. 255 256:program:`lit` identifies test suites as directories containing ``lit.cfg`` or 257``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are 258initially discovered by recursively searching up the directory hierarchy for 259all the input files passed on the command line. You can use 260:option:`--show-suites` to display the discovered test suites at startup. 261 262Once a test suite is discovered, its config file is loaded. Config files 263themselves are Python modules which will be executed. When the config file is 264executed, two important global variables are predefined: 265 266**lit** 267 268 The global **lit** configuration object (a *LitConfig* instance), which defines 269 the builtin test formats, global configuration parameters, and other helper 270 routines for implementing test configurations. 271 272**config** 273 274 This is the config object (a *TestingConfig* instance) for the test suite, 275 which the config file is expected to populate. The following variables are also 276 available on the *config* object, some of which must be set by the config and 277 others are optional or predefined: 278 279 **name** *[required]* The name of the test suite, for use in reports and 280 diagnostics. 281 282 **test_format** *[required]* The test format object which will be used to 283 discover and run tests in the test suite. Generally this will be a builtin test 284 format available from the *lit.formats* module. 285 286 **test_source_root** The filesystem path to the test suite root. For out-of-dir 287 builds this is the directory that will be scanned for tests. 288 289 **test_exec_root** For out-of-dir builds, the path to the test suite root inside 290 the object directory. This is where tests will be run and temporary output files 291 placed. 292 293 **environment** A dictionary representing the environment to use when executing 294 tests in the suite. 295 296 **suffixes** For **lit** test formats which scan directories for tests, this 297 variable is a list of suffixes to identify test files. Used by: *ShTest*. 298 299 **substitutions** For **lit** test formats which substitute variables into a test 300 script, the list of substitutions to perform. Used by: *ShTest*. 301 302 **unsupported** Mark an unsupported directory, all tests within it will be 303 reported as unsupported. Used by: *ShTest*. 304 305 **parent** The parent configuration, this is the config object for the directory 306 containing the test suite, or None. 307 308 **root** The root configuration. This is the top-most :program:`lit` configuration in 309 the project. 310 311 **on_clone** The config is actually cloned for every subdirectory inside a test 312 suite, to allow local configuration on a per-directory basis. The *on_clone* 313 variable can be set to a Python function which will be called whenever a 314 configuration is cloned (for a subdirectory). The function should takes three 315 arguments: (1) the parent configuration, (2) the new configuration (which the 316 *on_clone* function will generally modify), and (3) the test path to the new 317 directory being scanned. 318 319 **pipefail** Normally a test using a shell pipe fails if any of the commands 320 on the pipe fail. If this is not desired, setting this variable to false 321 makes the test fail only if the last command in the pipe fails. 322 323TEST DISCOVERY 324~~~~~~~~~~~~~~ 325 326Once test suites are located, :program:`lit` recursively traverses the source 327directory (following *test_source_root*) looking for tests. When :program:`lit` 328enters a sub-directory, it first checks to see if a nested test suite is 329defined in that directory. If so, it loads that test suite recursively, 330otherwise it instantiates a local test config for the directory (see 331:ref:`local-configuration-files`). 332 333Tests are identified by the test suite they are contained within, and the 334relative path inside that suite. Note that the relative path may not refer to 335an actual file on disk; some test formats (such as *GoogleTest*) define 336"virtual tests" which have a path that contains both the path to the actual 337test file and a subpath to identify the virtual test. 338 339.. _local-configuration-files: 340 341LOCAL CONFIGURATION FILES 342~~~~~~~~~~~~~~~~~~~~~~~~~ 343 344When :program:`lit` loads a subdirectory in a test suite, it instantiates a 345local test configuration by cloning the configuration for the parent direction 346--- the root of this configuration chain will always be a test suite. Once the 347test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file 348in the subdirectory. If present, this file will be loaded and can be used to 349specialize the configuration for each individual directory. This facility can 350be used to define subdirectories of optional tests, or to change other 351configuration parameters --- for example, to change the test format, or the 352suffixes which identify test files. 353 354TEST RUN OUTPUT FORMAT 355~~~~~~~~~~~~~~~~~~~~~~ 356 357The :program:`lit` output for a test run conforms to the following schema, in 358both short and verbose modes (although in short mode no PASS lines will be 359shown). This schema has been chosen to be relatively easy to reliably parse by 360a machine (for example in buildbot log scraping), and for other tools to 361generate. 362 363Each test result is expected to appear on a line that matches: 364 365.. code-block:: none 366 367 <result code>: <test name> (<progress info>) 368 369where ``<result-code>`` is a standard test result such as PASS, FAIL, XFAIL, 370XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and 371REGRESSED are also allowed. 372 373The ``<test name>`` field can consist of an arbitrary string containing no 374newline. 375 376The ``<progress info>`` field can be used to report progress information such 377as (1/300) or can be empty, but even when empty the parentheses are required. 378 379Each test result may include additional (multiline) log information in the 380following format: 381 382.. code-block:: none 383 384 <log delineator> TEST '(<test name>)' <trailing delineator> 385 ... log message ... 386 <log delineator> 387 388where ``<test name>`` should be the name of a preceding reported test, ``<log 389delineator>`` is a string of "*" characters *at least* four characters long 390(the recommended length is 20), and ``<trailing delineator>`` is an arbitrary 391(unparsed) string. 392 393The following is an example of a test run output which consists of four tests A, 394B, C, and D, and a log message for the failing test C: 395 396.. code-block:: none 397 398 PASS: A (1 of 4) 399 PASS: B (2 of 4) 400 FAIL: C (3 of 4) 401 ******************** TEST 'C' FAILED ******************** 402 Test 'C' failed as a result of exit code 1. 403 ******************** 404 PASS: D (4 of 4) 405 406LIT EXAMPLE TESTS 407~~~~~~~~~~~~~~~~~ 408 409The :program:`lit` distribution contains several example implementations of 410test suites in the *ExampleTests* directory. 411 412SEE ALSO 413-------- 414 415valgrind(1) 416