1<a id="top"></a> 2# Known limitations 3 4Over time, some limitations of Catch2 emerged. Some of these are due 5to implementation details that cannot be easily changed, some of these 6are due to lack of development resources on our part, and some of these 7are due to plain old 3rd party bugs. 8 9 10## Implementation limits 11### Sections nested in loops 12 13If you are using `SECTION`s inside loops, you have to create them with 14different name per loop's iteration. The recommended way to do so is to 15incorporate the loop's counter into section's name, like so: 16 17```cpp 18TEST_CASE( "Looped section" ) { 19 for (char i = '0'; i < '5'; ++i) { 20 SECTION(std::string("Looped section ") + i) { 21 SUCCEED( "Everything is OK" ); 22 } 23 } 24} 25``` 26 27or with a `DYNAMIC_SECTION` macro (that was made for exactly this purpose): 28 29```cpp 30TEST_CASE( "Looped section" ) { 31 for (char i = '0'; i < '5'; ++i) { 32 DYNAMIC_SECTION( "Looped section " << i) { 33 SUCCEED( "Everything is OK" ); 34 } 35 } 36} 37``` 38 39### Tests might be run again if last section fails 40 41If the last section in a test fails, it might be run again. This is because 42Catch2 discovers `SECTION`s dynamically, as they are about to run, and 43if the last section in test case is aborted during execution (e.g. via 44the `REQUIRE` family of macros), Catch2 does not know that there are no 45more sections in that test case and must run the test case again. 46 47 48### MinGW/CygWin compilation (linking) is extremely slow 49 50Compiling Catch2 with MinGW can be exceedingly slow, especially during 51the linking step. As far as we can tell, this is caused by deficiencies 52in its default linker. If you can tell MinGW to instead use lld, via 53`-fuse-ld=lld`, the link time should drop down to reasonable length 54again. 55 56 57## Features 58This section outlines some missing features, what is their status and their possible workarounds. 59 60### Thread safe assertions 61Catch2's assertion macros are not thread safe. This does not mean that 62you cannot use threads inside Catch's test, but that only single thread 63can interact with Catch's assertions and other macros. 64 65This means that this is ok 66```cpp 67 std::vector<std::thread> threads; 68 std::atomic<int> cnt{ 0 }; 69 for (int i = 0; i < 4; ++i) { 70 threads.emplace_back([&]() { 71 ++cnt; ++cnt; ++cnt; ++cnt; 72 }); 73 } 74 for (auto& t : threads) { t.join(); } 75 REQUIRE(cnt == 16); 76``` 77because only one thread passes the `REQUIRE` macro and this is not 78```cpp 79 std::vector<std::thread> threads; 80 std::atomic<int> cnt{ 0 }; 81 for (int i = 0; i < 4; ++i) { 82 threads.emplace_back([&]() { 83 ++cnt; ++cnt; ++cnt; ++cnt; 84 CHECK(cnt == 16); 85 }); 86 } 87 for (auto& t : threads) { t.join(); } 88 REQUIRE(cnt == 16); 89``` 90 91Because C++11 provides the necessary tools to do this, we are planning 92to remove this limitation in the future. 93 94### Process isolation in a test 95Catch does not support running tests in isolated (forked) processes. While this might in the future, the fact that Windows does not support forking and only allows full-on process creation and the desire to keep code as similar as possible across platforms, mean that this is likely to take significant development time, that is not currently available. 96 97### Running multiple tests in parallel 98Catch's test execution is strictly serial. If you find yourself with a test suite that takes too long to run and you want to make it parallel, there are 2 feasible solutions 99 * You can split your tests into multiple binaries and then run these binaries in parallel. 100 * You can have Catch list contained test cases and then run the same test binary multiple times in parallel, passing each instance list of test cases it should run. 101 102Both of these solutions have their problems, but should let you wring parallelism out of your test suite. 103 104## 3rd party bugs 105This section outlines known bugs in 3rd party components (this means compilers, standard libraries, standard runtimes). 106 107### Visual Studio 2017 -- raw string literal in assert fails to compile 108There is a known bug in Visual Studio 2017 (VC 15), that causes compilation error when preprocessor attempts to stringize a raw string literal (`#` preprocessor is applied to it). This snippet is sufficient to trigger the compilation error: 109```cpp 110#define CATCH_CONFIG_MAIN 111#include "catch.hpp" 112 113TEST_CASE("test") { 114 CHECK(std::string(R"("\)") == "\"\\"); 115} 116``` 117 118Catch provides a workaround, it is possible to disable stringification of original expressions by defining `CATCH_CONFIG_DISABLE_STRINGIFICATION`: 119```cpp 120#define CATCH_CONFIG_FAST_COMPILE 121#define CATCH_CONFIG_DISABLE_STRINGIFICATION 122#include "catch.hpp" 123 124TEST_CASE("test") { 125 CHECK(std::string(R"("\)") == "\"\\"); 126} 127``` 128 129_Do note that this changes the output somewhat_ 130``` 131catchwork\test1.cpp(6): 132PASSED: 133 CHECK( Disabled by CATCH_CONFIG_DISABLE_STRINGIFICATION ) 134with expansion: 135 ""\" == ""\" 136``` 137 138### Visual Studio 2015 -- Alignment compilation error (C2718) 139 140VS 2015 has a known bug, where `declval<T>` can cause compilation error 141if `T` has alignment requirements that it cannot meet. 142 143 144A workaround is to explicitly specialize `Catch::is_range` for given 145type (this avoids code path that uses `declval<T>` in a SFINAE context). 146 147 148### Visual Studio 2015 -- Wrong line number reported in debug mode 149VS 2015 has a known bug where `__LINE__` macro can be improperly expanded under certain circumstances, while compiling multi-file project in Debug mode. 150 151A workaround is to compile the binary in Release mode. 152 153### Clang/G++ -- skipping leaf sections after an exception 154Some versions of `libc++` and `libstdc++` (or their runtimes) have a bug with `std::uncaught_exception()` getting stuck returning `true` after rethrow, even if there are no active exceptions. One such case is this snippet, which skipped the sections "a" and "b", when compiled against `libcxxrt` from master 155```cpp 156#define CATCH_CONFIG_MAIN 157#include <catch.hpp> 158 159TEST_CASE("a") { 160 CHECK_THROWS(throw 3); 161} 162 163TEST_CASE("b") { 164 int i = 0; 165 SECTION("a") { i = 1; } 166 SECTION("b") { i = 2; } 167 CHECK(i > 0); 168} 169``` 170 171If you are seeing a problem like this, i.e. a weird test paths that trigger only under Clang with `libc++`, or only under very specific version of `libstdc++`, it is very likely you are seeing this. The only known workaround is to use a fixed version of your standard library. 172 173### Clang/G++ -- `Matches` string matcher always returns false 174This is a bug in `libstdc++-4.8`, where all matching methods from `<regex>` return false. Since `Matches` uses `<regex>` internally, if the underlying implementation does not work, it doesn't work either. 175 176Workaround: Use newer version of `libstdc++`. 177 178 179### libstdc++, `_GLIBCXX_DEBUG` macro and random ordering of tests 180 181Running a Catch2 binary compiled against libstdc++ with `_GLIBCXX_DEBUG` 182macro defined with `--order rand` will cause a debug check to trigger and 183abort the run due to self-assignment. 184[This is a known bug inside libstdc++](https://stackoverflow.com/questions/22915325/avoiding-self-assignment-in-stdshuffle/23691322) 185 186Workaround: Don't use `--order rand` when compiling against debug-enabled 187libstdc++. 188