1============== 2Testing libc++ 3============== 4 5.. contents:: 6 :local: 7 8Getting Started 9=============== 10 11libc++ uses LIT to configure and run its tests. 12 13The primary way to run the libc++ tests is by using ``make check-cxx``. 14 15However since libc++ can be used in any number of possible 16configurations it is important to customize the way LIT builds and runs 17the tests. This guide provides information on how to use LIT directly to 18test libc++. 19 20Please see the `Lit Command Guide`_ for more information about LIT. 21 22.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html 23 24Usage 25----- 26 27After building libc++, you can run parts of the libc++ test suite by simply 28running ``llvm-lit`` on a specified test or directory. If you're unsure 29whether the required libraries have been built, you can use the 30`check-cxx-deps` target. For example: 31 32.. code-block:: bash 33 34 $ cd <monorepo-root> 35 $ make -C <build> check-cxx-deps # If you want to make sure the targets get rebuilt 36 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests 37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic 39 40Sometimes you'll want to change the way LIT is running the tests. Custom options 41can be specified using the `--param=<name>=<val>` flag. The most common option 42you'll want to change is the standard dialect (ie -std=c++XX). By default the 43test suite will select the newest C++ dialect supported by the compiler and use 44that. However if you want to manually specify the option like so: 45 46.. code-block:: bash 47 48 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std 49 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param=std=c++03 # Run the tests in C++03 50 51Occasionally you'll want to add extra compile or link flags when testing. 52You can do this as follows: 53 54.. code-block:: bash 55 56 $ <build>/bin/llvm-lit -sv libcxx/test --param=compile_flags='-Wcustom-warning' 57 $ <build>/bin/llvm-lit -sv libcxx/test --param=link_flags='-L/custom/library/path' 58 59Some other common examples include: 60 61.. code-block:: bash 62 63 # Specify a custom compiler. 64 $ <build>/bin/llvm-lit -sv libcxx/test/std --param=cxx_under_test=/opt/bin/g++ 65 66 # Disable warnings in the test suite 67 $ <build>/bin/llvm-lit -sv libcxx/test --param=enable_warnings=False 68 69 # Use UBSAN when running the tests. 70 $ <build>/bin/llvm-lit -sv libcxx/test --param=use_sanitizer=Undefined 71 72Using a custom site configuration 73--------------------------------- 74 75By default, the libc++ test suite will use a site configuration that matches 76the current CMake configuration. It does so by generating a ``lit.site.cfg`` 77file in the build directory from one of the configuration file templates in 78``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around 79``llvm/utils/lit/lit.py``) to that file. So when you're running 80``<build>/bin/llvm-lit``, the generated ``lit.site.cfg`` file is always loaded 81instead of ``libcxx/test/lit.cfg.py``. If you want to use a custom site 82configuration, simply point the CMake build to it using 83``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration 84will be used instead. That file can use CMake variables inside it to make 85configuration easier. 86 87 .. code-block:: bash 88 89 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config> 90 $ make -C <build> check-cxx-deps 91 $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file 92 93 94LIT Options 95=========== 96 97:program:`lit` [*options*...] [*filenames*...] 98 99Command Line Options 100-------------------- 101 102To use these options you pass them on the LIT command line as ``--param NAME`` 103or ``--param NAME=VALUE``. Some options have default values specified during 104CMake's configuration. Passing the option on the command line will override the 105default. 106 107.. program:: lit 108 109.. option:: cxx_under_test=<path/to/compiler> 110 111 Specify the compiler used to build the tests. 112 113.. option:: stdlib=<stdlib name> 114 115 **Values**: libc++, libstdc++, msvc 116 117 Specify the C++ standard library being tested. The default is libc++ if this 118 option is not provided. This option is intended to allow running the libc++ 119 test suite against other standard library implementations. 120 121.. option:: std=<standard version> 122 123 **Values**: c++03, c++11, c++14, c++17, c++2a 124 125 Change the standard version used when building the tests. 126 127.. option:: cxx_headers=<path/to/headers> 128 129 Specify the c++ standard library headers that are tested. By default the 130 headers in the source tree are used. 131 132.. option:: cxx_library_root=<path/to/lib/> 133 134 Specify the directory of the libc++ library to be tested. By default the 135 library folder of the build directory is used. 136 137 138.. option:: cxx_runtime_root=<path/to/lib/> 139 140 Specify the directory of the libc++ library to use at runtime. This directory 141 is not added to the linkers search path. This can be used to compile tests 142 against one version of libc++ and run them using another. The default value 143 for this option is `cxx_library_root`. 144 145.. option:: use_system_cxx_lib=<bool> 146 147 **Default**: False 148 149 Enable or disable testing against the installed version of libc++ library. 150 This impacts whether the ``with_system_cxx_lib`` Lit feature is defined or 151 not. The ``cxx_library_root`` and ``cxx_runtime_root`` parameters should 152 still be used to specify the path of the library to link to and run against, 153 respectively. 154 155.. option:: debug_level=<level> 156 157 **Values**: 0, 1 158 159 Enable the use of debug mode. Level 0 enables assertions and level 1 enables 160 assertions and debugging of iterator misuse. 161 162.. option:: use_sanitizer=<sanitizer name> 163 164 **Values**: Memory, MemoryWithOrigins, Address, Undefined 165 166 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 167 building libc++ then that sanitizer will be used by default. 168 169.. option:: llvm_unwinder 170 171 Enable the use of LLVM unwinder instead of libgcc. 172 173.. option:: builtins_library 174 175 Path to the builtins library to use instead of libgcc. 176 177 178Writing Tests 179------------- 180 181When writing tests for the libc++ test suite, you should follow a few guidelines. 182This will ensure that your tests can run on a wide variety of hardware and under 183a wide variety of configurations. We have several unusual configurations such as 184building the tests on one host but running them on a different host, which add a 185few requirements to the test suite. Here's some stuff you should know: 186 187- All tests are run in a temporary directory that is unique to that test and 188 cleaned up after the test is done. 189- When a test needs data files as inputs, these data files can be saved in the 190 repository (when reasonable) and referenced by the test as 191 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or 192 directories will be made available to the test in the temporary directory 193 where it is run. 194- You should never hardcode a path from the build-host in a test, because that 195 path will not necessarily be available on the host where the tests are run. 196- You should try to reduce the runtime dependencies of each test to the minimum. 197 For example, requiring Python to run a test is bad, since Python is not 198 necessarily available on all devices we may want to run the tests on (even 199 though supporting Python is probably trivial for the build-host). 200 201Benchmarks 202========== 203 204Libc++ contains benchmark tests separately from the test of the test suite. 205The benchmarks are written using the `Google Benchmark`_ library, a copy of which 206is stored in the libc++ repository. 207 208For more information about using the Google Benchmark library see the 209`official documentation <https://github.com/google/benchmark>`_. 210 211.. _`Google Benchmark`: https://github.com/google/benchmark 212 213Building Benchmarks 214------------------- 215 216The benchmark tests are not built by default. The benchmarks can be built using 217the ``cxx-benchmarks`` target. 218 219An example build would look like: 220 221.. code-block:: bash 222 223 $ cd build 224 $ cmake [options] <path to libcxx sources> 225 $ make cxx-benchmarks 226 227This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 228built against the just-built libc++. The compiled tests are output into 229``build/benchmarks``. 230 231The benchmarks can also be built against the platforms native standard library 232using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 233is useful for comparing the performance of libc++ to other standard libraries. 234The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 235``<test>.native.out`` otherwise. 236 237Also See: 238 239 * :ref:`Building Libc++ <build instructions>` 240 * :ref:`CMake Options` 241 242Running Benchmarks 243------------------ 244 245The benchmarks must be run manually by the user. Currently there is no way 246to run them as part of the build. 247 248For example: 249 250.. code-block:: bash 251 252 $ cd build/benchmarks 253 $ make cxx-benchmarks 254 $ ./algorithms.libcxx.out # Runs all the benchmarks 255 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 256 257For more information about running benchmarks see `Google Benchmark`_. 258