1.. _module-pw_compilation_testing: 2 3====================== 4pw_compilation_testing 5====================== 6.. pigweed-module:: 7 :name: pw_compilation_testing 8 9The pw_compilation_testing module provides for negative compilation (NC) 10testing. Negative compilation tests ensure that code that should not compile 11does not compile. Negative compilation testing is helpful in a variety of 12scenarios, for example: 13 14- Testing for compiler errors, such as ``[[nodiscard]]`` checks. 15- Testing that a template cannot be instantiated with certain types. 16- Testing that a ``static_assert`` statement is triggered as expected. 17- For a ``constexpr`` function, testing that a ``PW_ASSERT`` is triggered as 18 expected. 19 20Negative compilation tests are only supported in GN currently. Negative 21compilation tests are not currently supported in GN on Windows due to 22`b/241565082 <https://issues.pigweed.dev/241565082>`_. 23 24.. warning:: 25 26 This module is in an early, experimental state. Do not use it unless you have 27 consulted with the Pigweed team. 28 29--------------------------------- 30Negative compilation test example 31--------------------------------- 32.. code-block:: cpp 33 34 #include "pw_unit_test/framework.h" 35 #include "pw_compilation_testing/negative_compilation.h" 36 37 template <int kValue> 38 struct MyStruct { 39 static_assert(kValue % 2 == 0, "wrong number!"); 40 41 constexpr int MultiplyOdd(int runtime_value) const { 42 PW_ASSERT(runtime_value % 2 == 0); 43 return kValue * runtime_value; 44 } 45 }; 46 47 [[maybe_unused]] MyStruct<16> this_one_works; 48 49 // NC tests cannot be compiled, so they are created in preprocessor #if or 50 // #elif blocks. These NC tests check that a static_assert statement fails if 51 // the code is compiled. 52 #if PW_NC_TEST(NegativeOddNumber) 53 PW_NC_EXPECT("wrong number!"); 54 [[maybe_unused]] MyStruct<-1> illegal; 55 #elif PW_NC_TEST(PositiveOddNumber) 56 PW_NC_EXPECT("wrong number!"); 57 [[maybe_unused]] MyStruct<5> this_is_illegal; 58 #endif // PW_NC_TEST 59 60 struct Foo { 61 // Negative compilation tests can go anywhere in a source file. 62 #if PW_NC_TEST(IllegalValueAsClassMember) 63 PW_NC_EXPECT("wrong number!"); 64 MyStruct<12> also_illegal; 65 #endif // PW_NC_TEST 66 }; 67 68 TEST(MyStruct, MultiplyOdd) { 69 MyStruct<5> five; 70 EXPECT_EQ(five.MultiplyOdd(3), 15); 71 72 // This NC test checks that a specific PW_ASSERT() fails when expected. 73 // This only works in an NC test if the PW_ASSERT() fails while the compiler 74 // is executing constexpr code. The test code is used in a constexpr 75 // statement to force compile-time evaluation. 76 #if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber) 77 [[maybe_unused]] constexpr auto fail = [] { 78 PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);"); 79 MyStruct<3> my_struct; 80 return my_struct.MultiplyOdd(4); // Even number, PW_ASSERT should fail. 81 }(); 82 #endif // PW_NC_TEST 83 } 84 85 // PW_NC_TESTs can be conditionally executed using preprocessor conditionals. 86 #if PW_CXX_STANDARD_IS_SUPPORTED(20) 87 #if PW_NC_TEST(RequiresSomeCpp20Feature) 88 [[maybe_unused]] constinit MyStruct<4> constinit_works; 89 #endif // PW_NC_TEST 90 #endif // PW_CXX_STANDARD_IS_SUPPORTED(20) 91 92------------------------------------ 93Creating a negative compilation test 94------------------------------------ 95- Declare a ``pw_cc_negative_compilation_test()`` GN target or set 96 ``negative_compilation_test = true`` in a ``pw_test()`` target. 97- Add the test to the build in a toolchain with negative compilation testing 98 enabled (``pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true``). 99- In the test source files, add 100 ``#include "pw_compilation_testing/negative_compilation.h"``. 101- Use the ``PW_NC_TEST(TestName)`` macro in a ``#if`` statement. 102- Immediately after the ``PW_NC_TEST(TestName)``, provide one or more 103 Python-style regular expressions with the ``PW_NC_EXPECT()`` macro, one per 104 line. 105- Execute the tests by running the build. 106 107To simplify parsing, all ``PW_NC_TEST()`` statements must fit on a single line 108and cannot have any other code before or after them. ``PW_NC_EXPECT()`` 109statements may span multiple lines, but must contain a single regular expression 110as a string literal. The string may be comprised of multiple implicitly 111concatenated string literals. The ``PW_NC_EXPECT()`` statement cannot contain 112anything else except for ``//``-style comments. 113 114Test assertions 115=============== 116Negative compilation tests must have at least one assertion about the 117compilation output. The assertion macros must be placed immediately after the 118line with the ``PW_NC_TEST()`` or the test will fail. 119 120.. c:macro:: PW_NC_EXPECT(regex_string_literal) 121 122 When negative compilation tests are run, checks the compilation output for the 123 provided regular expression. The argument to the ``PW_NC_EXPECT()`` statement 124 must be a string literal. The literal is interpreted character-for-character 125 as a Python raw string literal and compiled as a Python `re 126 <https://docs.python.org/3/library/re.html>`_ regular expression. 127 128 For example, ``PW_NC_EXPECT("something (went|has gone) wrong!")`` searches the 129 failed compilation output with the Python regular expression 130 ``re.compile("something (went|has gone) wrong!")``. 131 132.. c:macro:: PW_NC_EXPECT_GCC(regex_string_literal) 133 134 Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with GCC. 135 136.. c:macro:: PW_NC_EXPECT_CLANG(regex_string_literal) 137 138 Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with Clang. 139 140.. admonition:: Test expectation tips 141 :class: tip 142 143 Be as specific as possible, but avoid compiler-specific error text. Try 144 matching against the following: 145 146 - ``static_assert`` messages. 147 - Contents of specific failing lines of source code: 148 ``PW_NC_EXPECT("PW_ASSERT\(!empty\(\));")``. 149 - Comments on affected lines: ``PW_NC_EXPECT("// Cannot construct from 150 nullptr")``. 151 - Function names: ``PW_NC_EXPECT("SomeFunction\(\).*private")``. 152 153 Do not match against the following: 154 155 - Source file paths. 156 - Source line numbers. 157 - Compiler-specific wording of error messages, except when necessary. 158 159------ 160Design 161------ 162The basic flow for negative compilation testing is as follows. 163 164- The user defines negative compilation tests in preprocessor ``#if`` blocks 165 using the ``PW_NC_TEST()`` and :c:macro:`PW_NC_EXPECT` macros. 166- The build invokes the ``pw_compilation_testing.generator`` script. The 167 generator script: 168 169 - finds ``PW_NC_TEST()`` statements and extracts a list of test cases, 170 - finds all associated :c:macro:`PW_NC_EXPECT` statements, and 171 - generates build targets for each negative compilation tests, 172 passing the test information and expectations to the targets. 173 174- The build compiles the test source file with all tests disabled. 175- The build invokes the negative compilation test targets, which run the 176 ``pw_compilation_testing.runner`` script. The test runner script: 177 178 - invokes the compiler, setting a preprocessor macro that enables the ``#if`` 179 block for the test. 180 - captures the compilation output, and 181 - checks the compilation output for the :c:macro:`PW_NC_EXPECT` expressions. 182 183- If compilation failed, and the output matches the test case's 184 :c:macro:`PW_NC_EXPECT` expressions, the test passes. 185- If compilation succeeded or the :c:macro:`PW_NC_EXPECT` expressions did not 186 match the output, the test fails. 187 188Existing frameworks 189=================== 190Pigweed's negative compilation tests were inspired by Chromium's `no-compile 191tests <https://www.chromium.org/developers/testing/no-compile-tests/>`_ 192tests and a similar framework used internally at Google. Pigweed's negative 193compilation testing framework improves on these systems in a few respects: 194 195- Trivial integration with unit tests. Negative compilation tests can easily be 196 placed alongside other unit tests instead of in separate files. 197- Safer, more natural macro-based API for test declarations. Other systems use 198 ``#ifdef`` macro checks to define test cases, which fail silently when there 199 are typos. Pigweed's framework uses function-like macros, which provide a 200 clean and natural API, catch typos, and ensure the test is integrated with the 201 NC test framework. 202- More readable, flexible test assertions. Other frameworks place assertions in 203 comments after test names, while Pigweed's framework uses function-like 204 macros. Pigweed also supports compiler-specific assertions. 205- Assertions are required. This helps ensure that compilation fails for the 206 expected reason and not for an accidental typo or unrelated issue. 207