1Motivation for metadata extraction 2================================== 3 4Exporting documentation 5----------------------- 6 7This allow us to build browsable documentation for the testcases, e.g. a 8catalogue of test information that would be searchable etc. At this point there 9is a single page generated from the extracted data that tries to outline the 10intent. 11 12 13Propagating test requirements 14----------------------------- 15 16Some subtests require different hardware resources/software versions/etc. the 17test execution framework needs to consume these so that it can locate proper 18hardware, install proper software, etc. 19 20Some examples of requirements are: 21 22* Test needs at least 1GB of RAM. 23 24* Test needs a block device at least 512MB in size 25 26* Test needs a NUMA machine with two memory nodes and at least 300 free pages on each node 27 28* Test needs i2c eeprom connected on a i2c bus 29 30* Test needs two serial ports connected via null-modem cable 31 32 33With this information extracted from the tests the testrunner can then map the 34requirements on the available machines in a lab and select a proper machine for 35the particular (sub)set of testcases as well as supply a particular test with 36additional information needed for the test, such as address of the i2c device, 37paths to the serial devices, etc. In the case of virtual machines the test could 38also dynamically prepare the correct environment for the test on demand. 39 40 41Parallel test execution 42----------------------- 43 44An LTP testrun on a modern hardware wastes most of the machine resources 45because the testcases are running sequentially. However in order to execute 46tests in parallel we need to know which system resources are utilized by a 47given test, as obviously we cannot run two tests that monopolize the same 48resource. In some cases we would also need to partition the system resource 49accordingly, e.g. if we have two memory stress tests running at the same time 50we will need to cap each of these tests on half of the available memory, or 51make sure that sum of the memory used by these two tests is not greater an 52available memory. 53 54Examples of such tests are: 55 56* Tests that mess with global system state 57 - system time (e.g. settimeofday() test and leap second test) 58 - SysV SHM 59 - ... 60 61* Tests that use block device 62 63* Tests that work with a particular hardware resource 64 - i2c eeprom test 65 - serial port tests 66 - ... 67 68Exporting test runtime/timeout to the testrunner 69------------------------------------------------ 70 71Currently most of the testrunners usually do not know for how long is the test 72supposed to run, this means that we have to guess some upper limit on how long 73a test is supposed to run. The value is usually twice of the maximal runtime 74for all testcases or whole suite or even larger. This means that we are wasting 75time in the case that the test ends up stuck and we could have failed it much 76sooner in most of the cases. This becomes quite important for a kernel 77regression tests that do crash the host, if the information that the test is 78supposed to crash a kernel under a minute is exported to the testrunner we can 79reboot the machine much faster in an event of a crash. 80 81Getting rid of runtest files 82---------------------------- 83 84This would also allow us to get rid of the unflexible and hard to maintain 85runtest files. Once this system is in place we will have a list of all tests 86along with their respective metadata - which means that we will be able to 87generate subsets of the test easily on the fly. 88 89In order to achieve this we need two things: 90 91Each test will describe which syscall/functionality it tests in the metadata. 92Then we could define groups of tests based on that. I.e. instead of having 93syscall runtest file we would ask the testrunner to run all test that have a 94defined which syscall they test, or whose filename matches a particular syscall name. 95 96Secondly we will have to store the test variants in the test metadata instead 97of putting them in a file that is unrelated to the test. 98 99For example: 100 101* To run CVE related test we would select testcases with CVE tag 102 103* To run IPC test we will define a list of IPC syscalls and run all syscall 104 test that are in the list 105 106* And many more... 107 108 109Implementation 110============== 111 112The docparser is implemented as a minimal C tokenizer that can parse and 113extract code comments and C structures. The docparser then runs over all C 114sources in the testcases directory and if tst\_test structure is present in the 115source it's parsed and the result is included in the resulting metadata. 116 117During parsing the metadata is stored in a simple key/value storage that more 118or less follows C structure layout, i.e. can include hash, array, and string. 119Once the parsing is finished the result is filtered so that only interesting 120fields of the tst\_test structure are included and then converted into JSON 121output. 122 123This process produces one big JSON file with metadata for all tests, that 124is then installed along with the testcases. This would then be used by the 125testrunner. 126 127The test requirements are stored in the tst\_test structure either as 128bitflags, integers or arrays of strings: 129 130```c 131struct tst_test test = { 132 ... 133 /* tests needs to run with UID=0 */ 134 .needs_root = 1, 135 136 /* 137 * Tests needs a block device at least 1024MB in size and also 138 * mkfs.ext4 installed. 139 */ 140 .needs_device = 1, 141 .dev_min_size = 1024, 142 .dev_fs_type = ext4, 143 144 /* Indicates that the test is messing with system wall clock */ 145 .restore_wallclock = 1, 146 147 /* Tests needs uinput either compiled in or loaded as a module */ 148 .needs_drivers = (const char *[]) { 149 "uinput", 150 NULL 151 }, 152 153 /* Tests needs enabled kernel config flags */ 154 .needs_kconfigs = (const char *[]) { 155 "CONFIG_X86_INTEL_UMIP=y", 156 NULL 157 }, 158 159 /* Additional array of key value pairs */ 160 .tags = (const struct tst_tag[]) { 161 {"linux-git", "43a6684519ab"}, 162 {"CVE", "2017-2671"}, 163 {NULL, NULL} 164 } 165}; 166``` 167 168The test documentation is stored in a special comment such as: 169 170``` 171/*\ 172 * Test description 173 * 174 * This is a test description. 175 * Consisting of several lines. 176 */ 177``` 178 179Which will yield following JSON output: 180 181```json 182 "testcaseXY": { 183 "needs_root": "1", 184 "needs_device": "1", 185 "dev_min_size": "1024", 186 "dev_fs_type": "ext4", 187 "restore_wallclock": "1", 188 "needs_drivers": [ 189 "uinput", 190 ], 191 "needs_kconfigs": [ 192 "CONFIG_X86_INTEL_UMIP=y", 193 ], 194 "tags": [ 195 [ 196 "linux-git", 197 "43a6684519ab" 198 ], 199 [ 200 "CVE", 201 "2017-2671" 202 ], 203 ], 204 "doc": [ 205 "Test description", 206 "", 207 "This is a test description.", 208 "Consisting of several lines." 209 ], 210 "fname": "testcases/kernel/syscalls/foo/testcaseXY.c" 211 }, 212``` 213 214The final JSON file is JSON object of test descriptions indexed by a test name 215with a header describing the testsuite: 216 217```json 218{ 219 "testsuite": "Linux Test Project", 220 "testsuite_short": "LTP", 221 "url": "https://github.com/linux-test-project/ltp/", 222 "scm_url_base": "https://github.com/linux-test-project/ltp/tree/master/", 223 "timeout": 300, 224 "version": "20200930", 225 "tests": { 226 "testcaseXY": { 227 ... 228 }, 229 ... 230 } 231} 232``` 233 234Open Points 235=========== 236 237There are stil some loose ends. Mostly it's not well defined where to put 238things and how to format them. 239 240* Some of the hardware requirements are already listed in the tst\_test. Should 241 we put all of them there? 242 243* What would be the format for test documentation and how to store things such 244 as test variants there? 245 246So far this proof of concept generates a metadata file. I guess that we need 247actual consumers which will help to settle things down, I will try to look into 248making use of this in the runltp-ng at least as a reference implementation. 249