1Test suite 2########## 3 4Sparse has a number of test cases in its validation directory. The test-suite 5script aims at making automated checking of these tests possible. It works by 6embedding tags in C comments in the test cases. 7 8Tag's syntax 9============ 10 11``check-name:`` *name* 12 13 Name of the test. This is the only mandatory tag. 14 15``check-description:`` *description ...* 16 17 A description of what the test checks. 18 19``check-command:`` *command arg ...* 20 21 There are different kinds of tests. Some can validate the sparse 22 preprocessor, while others will use sparse, cgcc, or even other backends 23 of the library. check-command allows you to give a custom command to 24 run the test-case. 25 The ``$file`` string is special. It will be expanded to the file name at 26 run time. 27 It defaults to ``sparse $file``. 28 29``check-arch-ignore:`` *arch[|...]* 30 31``check-arch-only:`` *arch[|...]* 32 33 Ignore the test if the current architecture (as returned by ``uname -m``) 34 matches or not one of the archs given in the pattern. 35 36``check-assert:`` *condition* 37 38 Ignore the test if the given condition is false when evaluated as a 39 static assertion (``_Static_assert``). 40 41``check-cpp-if:`` *condition* 42 43 Ignore the test if the given condition is false when evaluated 44 by sparse's pre-processor. 45 46``check-exit-value:`` *value* 47 48 The expected exit value of check-command. It defaults to 0. 49 50``check-timeout:`` *timeout* 51 52 The maximum expected duration of check-command, in seconds. 53 It defaults to 1. 54 55``check-output-start`` / ``check-output-end`` 56 57 The expected output (stdout and stderr) of check-command lies between 58 those two tags. It defaults to no output. 59 60``check-output-ignore`` / ``check-error-ignore`` 61 62 Don't check the expected output (stdout or stderr) of check-command 63 (useful when this output is not comparable or if you're only interested 64 in the exit value). By default this check is done. 65 66``check-known-to-fail`` 67 68 Mark the test as being known to fail. 69 70``check-output-contains:`` *pattern* 71 72 Check that the output (stdout) contains the given pattern. 73 Several such tags can be given, in which case the output 74 must contains all the patterns. 75 76``check-output-excludes:`` *pattern* 77 78 Similar than the above one, but with opposite logic. 79 Check that the output (stdout) doesn't contain the given pattern. 80 Several such tags can be given, in which case the output 81 must contains none of the patterns. 82 83``check-output-pattern(``\ *nbr*\ ``):`` *pattern* 84 85``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern* 86 87 Similar to the contains/excludes above, but with full control 88 of the number of times the pattern should occur in the output. 89 If *min* or *max* is ``-`` the corresponding check is ignored. 90 91``check-output-match(``\ *start*\ ``):`` *pattern* 92 93 Check that in the output (stdout) all lines starting with the 94 first pattern also contains the second pattern. This should be 95 reserved for matching IR instructions since the '.$size' suffix 96 is ignored in the first pattern but is expected to be followed 97 by a space character. 98 99``check-output-returns:`` *value* 100 101 Check that in the output (stdout) all IR return instructions 102 have the given value. 103 104Using test-suite 105================ 106 107The test-suite script is called through the check target of the Makefile. It 108will try to check every test case it finds (``find validation -name '*.c'``). 109It can be called to check a single test with:: 110 111 $ cd validation 112 $ ./test-suite single preprocessor/preprocessor1.c 113 TEST Preprocessor #1 (preprocessor/preprocessor1.c) 114 preprocessor/preprocessor1.c passed ! 115 116 117Writing a test 118============== 119 120The test-suite comes with a format command to make a test easier to write:: 121 122 test-suite format [-a] [-l] [-f] file [name [cmd]] 123 124`name:` check-name value 125 If no name is provided, it defaults to the file name. 126 127`cmd:` check-command value 128 If no cmd is provided, it defaults to ``sparse $file``. 129 130The output of the test-suite format command can be redirected into the 131test case to create a test-suite formatted file.:: 132 133 $ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c 134 $ cat !$ 135 cat bad-assignment.c 136 /* 137 * check-name: bad assignment 138 * 139 * check-command: sparse $file 140 * check-exit-value: 1 141 * 142 * check-output-start 143 bad-assignment.c:3:6: error: Expected ; at end of statement 144 bad-assignment.c:3:6: error: got \ 145 * check-output-end 146 */ 147 148The same effect without the redirection can be achieved by using the ``-a`` 149option. 150 151You can define the check-command you want to use for the test.:: 152 153 $ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \ 154 "sparse -E \$file" 155 $ cat !$ 156 cat validation/preprocessor2.c 157 /* 158 * This one we happen to get right. 159 * 160 * It should result in a simple 161 * 162 * a + b 163 * 164 * for a proper preprocessor. 165 */ 166 #define TWO a, b 167 168 #define UNARY(x) BINARY(x) 169 #define BINARY(x, y) x + y 170 171 UNARY(TWO) 172 /* 173 * check-name: Preprocessor #2 174 * 175 * check-command: sparse -E $file 176 * check-exit-value: 0 177 * 178 * check-output-start 179 180 a + b 181 * check-output-end 182 */ 183