1.. _module-pw_compilation_testing: 2 3====================== 4pw_compilation_testing 5====================== 6The pw_compilation_testing module provides for negative compilation (NC) 7testing. Negative compilation tests ensure that code that should not compile 8does not compile. Negative compilation testing is helpful in a variety of 9scenarios, for example: 10 11- Testing for compiler errors, such as ``[[nodiscard]]`` checks. 12- Testing that a template cannot be instantiated with certain types. 13- Testing that a ``static_assert`` statement is triggered as expected. 14- For a ``constexpr`` function, testing that a ``PW_ASSERT`` is triggered as 15 expected. 16 17Negative compilation tests are only supported in GN currently. Negative 18compilation tests are not currently supported in GN on Windows due to 19`b/241565082 <https://issues.pigweed.dev/241565082>`_. 20 21.. warning:: 22 23 This module is in an early, experimental state. Do not use it unless you have 24 consulted with the Pigweed team. 25 26--------------------------------- 27Negative compilation test example 28--------------------------------- 29.. code-block:: cpp 30 31 #include "gtest/gtest.h" 32 #include "pw_compilation_testing/negative_compilation.h" 33 34 template <int kValue> 35 struct MyStruct { 36 static_assert(kValue % 2 == 0, "wrong number!"); 37 38 constexpr int MultiplyOdd(int runtime_value) const { 39 PW_ASSERT(runtime_value % 2 == 0); 40 return kValue * runtime_value; 41 } 42 }; 43 44 [[maybe_unused]] MyStruct<16> this_one_works; 45 46 // NC tests cannot be compiled, so they are created in preprocessor #if or 47 // #elif blocks. These NC tests check that a static_assert statement fails if 48 // the code is compiled. 49 #if PW_NC_TEST(NegativeOddNumber) 50 PW_NC_EXPECT("wrong number!"); 51 [[maybe_unused]] MyStruct<-1> illegal; 52 #elif PW_NC_TEST(PositiveOddNumber) 53 PW_NC_EXPECT("wrong number!"); 54 [[maybe_unused]] MyStruct<5> this_is_illegal; 55 #endif // PW_NC_TEST 56 57 struct Foo { 58 // Negative compilation tests can go anywhere in a source file. 59 #if PW_NC_TEST(IllegalValueAsClassMember) 60 PW_NC_EXPECT("wrong number!"); 61 MyStruct<12> also_illegal; 62 #endif // PW_NC_TEST 63 }; 64 65 TEST(MyStruct, MultiplyOdd) { 66 MyStruct<5> five; 67 EXPECT_EQ(five.MultiplyOdd(3), 15); 68 69 // This NC test checks that a specific PW_ASSERT() fails when expected. 70 // This only works in an NC test if the PW_ASSERT() fails while the compiler 71 // is executing constexpr code. The test code is used in a constexpr 72 // statement to force compile-time evaluation. 73 #if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber) 74 [[maybe_unused]] constexpr auto fail = [] { 75 PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);"); 76 MyStruct<3> my_struct; 77 return my_struct.MultiplyOdd(4); // Even number, PW_ASSERT should fail. 78 }(); 79 #endif // PW_NC_TEST 80 } 81 82------------------------------------ 83Creating a negative compilation test 84------------------------------------ 85- Declare a ``pw_cc_negative_compilation_test()`` GN target or set 86 ``negative_compilation_test = true`` in a ``pw_test()`` target. 87- Add the test to the build in a toolchain with negative compilation testing 88 enabled (``pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true``). 89- In the test source files, add 90 ``#include "pw_compilation_testing/negative_compilation.h"``. 91- Use the ``PW_NC_TEST(TestName)`` macro in a ``#if`` statement. 92- Immediately after the ``PW_NC_TEST(TestName)``, provide one or more 93 Python-style regular expressions with the ``PW_NC_EXPECT()`` macro, one per 94 line. 95- Execute the tests by running the build. 96 97To simplify parsing, all ``PW_NC_TEST()`` statements must fit on a single line. 98``PW_NC_EXPECT()`` statements may span multiple lines, but must contain a single 99regular expression as a string literal. The string may be comprised of multiple 100implicitly concatenated string literals. The ``PW_NC_EXPECT()`` statement cannot 101contain anything else except for ``//``-style comments. 102 103Test assertions 104=============== 105Negative compilation tests must have at least one assertion about the 106compilation output. The assertion macros must be placed immediately after the 107line with the ``PW_NC_TEST()`` or the test will fail. 108 109.. c:macro:: PW_NC_EXPECT(regex_string_literal) 110 111 When negative compilation tests are run, checks the compilation output for the 112 provided regular expression. The argument to the ``PW_NC_EXPECT()`` statement 113 must be a string literal. The literal is interpreted character-for-character 114 as a Python raw string literal and compiled as a Python `re 115 <https://docs.python.org/3/library/re.html>`_ regular expression. 116 117 For example, ``PW_NC_EXPECT("something (went|has gone) wrong!")`` searches the 118 failed compilation output with the Python regular expression 119 ``re.compile("something (went|has gone) wrong!")``. 120 121.. c:macro:: PW_NC_EXPECT_GCC(regex_string_literal) 122 123 Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with GCC. 124 125.. c:macro:: PW_NC_EXPECT_CLANG(regex_string_literal) 126 127 Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with Clang. 128 129.. admonition:: Test expectation tips 130 :class: tip 131 132 Be as specific as possible, but avoid compiler-specific error text. Try 133 matching against the following: 134 135 - ``static_assert`` messages. 136 - Contents of specific failing lines of source code: 137 ``PW_NC_EXPECT("PW_ASSERT\(!empty\(\));")``. 138 - Comments on affected lines: ``PW_NC_EXPECT("// Cannot construct from 139 nullptr")``. 140 - Function names: ``PW_NC_EXPECT("SomeFunction\(\).*private")``. 141 142 Do not match against the following: 143 144 - Source file paths. 145 - Source line numbers. 146 - Compiler-specific wording of error messages, except when necessary. 147 148------ 149Design 150------ 151The basic flow for negative compilation testing is as follows. 152 153- The user defines negative compilation tests in preprocessor ``#if`` blocks 154 using the ``PW_NC_TEST()`` and :c:macro:`PW_NC_EXPECT` macros. 155- The build invokes the ``pw_compilation_testing.generator`` script. The 156 generator script: 157 158 - finds ``PW_NC_TEST()`` statements and extracts a list of test cases, 159 - finds all associated :c:macro:`PW_NC_EXPECT` statements, and 160 - generates build targets for each negative compilation tests, 161 passing the test information and expectations to the targets. 162 163- The build compiles the test source file with all tests disabled. 164- The build invokes the negative compilation test targets, which run the 165 ``pw_compilation_testing.runner`` script. The test runner script: 166 167 - invokes the compiler, setting a preprocessor macro that enables the ``#if`` 168 block for the test. 169 - captures the compilation output, and 170 - checks the compilation output for the :c:macro:`PW_NC_EXPECT` expressions. 171 172- If compilation failed, and the output matches the test case's 173 :c:macro:`PW_NC_EXPECT` expressions, the test passes. 174- If compilation succeeded or the :c:macro:`PW_NC_EXPECT` expressions did not 175 match the output, the test fails. 176 177Existing frameworks 178=================== 179Pigweed's negative compilation tests were inspired by Chromium's `no-compile 180tests <https://www.chromium.org/developers/testing/no-compile-tests/>`_ 181tests and a similar framework used internally at Google. Pigweed's negative 182compilation testing framework improves on these systems in a few respects: 183 184- Trivial integration with unit tests. Negative compilation tests can easily be 185 placed alongside other unit tests instead of in separate files. 186- Safer, more natural macro-based API for test declarations. Other systems use 187 ``#ifdef`` macro checks to define test cases, which fail silently when there 188 are typos. Pigweed's framework uses function-like macros, which provide a 189 clean and natural API, catch typos, and ensure the test is integrated with the 190 NC test framework. 191- More readable, flexible test assertions. Other frameworks place assertions in 192 comments after test names, while Pigweed's framework uses function-like 193 macros. Pigweed also supports compiler-specific assertions. 194- Assertions are required. This helps ensure that compilation fails for the 195 expected reason and not for an accidental typo or unrelated issue. 196