1# Test library design document 2 3## High-level picture 4 5 library process 6 +----------------------------+ 7 | main | 8 | tst_run_tcases | 9 | do_setup | 10 | for_each_variant | 11 | for_each_filesystem | test process 12 | fork_testrun ------------->+--------------------------------------------+ 13 | waitpid | | testrun | 14 | | | do_test_setup | 15 | | | tst_test->setup | 16 | | | run_tests | 17 | | | tst_test->test(i) or tst_test->test_all | 18 | | | do_test_cleanup | 19 | | | tst_test->cleanup | 20 | | | exit(0) | 21 | do_exit | +--------------------------------------------+ 22 | do_cleanup | 23 | exit(ret) | 24 +----------------------------+ 25 26## Test lifetime overview 27 28When a test is executed the very first thing to happen is that we check for 29various test prerequisites. These are described in the tst\_test structure and 30range from simple '.require\_root' to a more complicated kernel .config boolean 31expressions such as: "CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y". 32 33If all checks are passed the process carries on with setting up the test 34environment as requested in the tst\_test structure. There are many different 35setup steps that have been put into the test library again ranging from rather 36simple creation of a unique test temporary directory to a bit more complicated 37ones such as preparing, formatting, and mounting a block device. 38 39The test library also intializes shrared memory used for IPC at this step. 40 41Once all the prerequisites are checked and test environment has been prepared 42we can move on executing the testcase itself. The actual test is executed in a 43forked process, however there are a few hops before we get there. 44 45First of all there are test variants, which means that the test is re-executed 46several times with a slightly different setting. This is usually used to test a 47family of similar syscalls, where we test each of these syscalls exactly the 48same, but without re-executing the test binary itself. Test variants are 49implemented as a simple global variable counter that gets increased on each 50iteration. In a case of syscall tests we switch between which syscall to call 51based on the global counter. 52 53Then there is all\_filesystems flag which is mostly the same as test variants 54but executes the test for each filesystem supported by the system. Note that we 55can get cartesian product between test variants and all filesystems as well. 56 57In a pseudo code it could be expressed as: 58 59``` 60for test_variants: 61 for all_filesystems: 62 fork_testrun() 63``` 64 65Before we fork() the test process the test library sets up a timeout alarm and 66also a heartbeat signal handlers and also sets up an alarm(2) accordingly to 67the test timeout. When a test times out the test library gets SIGALRM and the 68alarm handler mercilessly kills all forked children by sending SIGKILL to the 69whole process group. The heartbeat handler is used by the test process to reset 70this timer for example when the test functions run in a loop. 71 72With that done we finally fork() the test process. The test process firstly 73resets signal handlers and sets its pid to be a process group leader so that we 74can slaughter all children if needed. The test library proceeds with suspending 75itself in waitpid() syscall and waits for the child to finish at this point. 76 77The test process goes ahead and calls the test setup() function if present in 78the tst\_test structure. It's important that we execute all test callbacks 79after we have forked the process, that way we cannot crash the test library 80process. The setup can also cause the test to exit prematurely by either direct 81or indirect (SAFE\_MACROS()) call to tst\_brk(). In this case the 82fork\_testrun() function exits, but the loops for test variants or filesystems 83carries on. 84 85All that is left to be done is to actually execute the tests, what happnes now 86depends on the -i and -I command line parameters that can request that the 87run() or run\_all() callbacks are executed N times or for N seconds. Again the 88test can exit at any time by direct or indirect call to tst\_brk(). 89 90Once the test is finished all that is left for the test process is the test 91cleanup(). So if a there is a cleanup() callback in the tst\_test structure 92it's executed. The cleanup() callback runs in a special context where the 93tst\_brk(TBROK, ...) calls are converted into tst\_res(TWARN, ...) calls. This 94is because we found out that carrying on with partially broken cleanup is 95usually better option than exiting it in the middle. 96 97The test cleanup() is also called by the tst\_brk() handler in order to cleanup 98before exiting the test process, hence it must be able to cope even with 99partial test setup. Usually it suffices to make sure to clean up only 100resources that already have been set up and to do that in an inverse order that 101we did in setup(). 102 103Once the test process exits or leaves the run() or run\_all() function the test 104library wakes up from the waitpid() call, and checks if the test process 105exitted normally. 106 107Once the testrun is finished the test library does a cleanup() as well to clean 108up resources set up in the test library setup(), reports test results and 109finally exits the process. 110 111### Test library and fork()-ing 112 113Things are a bit more complicated when fork()-ing is involved, however the test 114results are stored in a page of a shared memory and incremented by atomic 115operations, hence the results are stored right after the test reporting 116function returns from the test library and the access is, by definition, 117race-free as well. 118 119On the other hand the test library, apart from sending a SIGKILL to the whole 120process group on timeout, does not track grandchildren. 121 122This especially means that: 123 124- The test exits once the main test process exits. 125 126- While the test results are, by the design, propagated to the test library 127 we may still miss a child that gets killed by a signal or exits unexpectedly. 128 129The test writer should, because of this, take care for reaping these proceses 130properly, in most cases this could be simply done by calling 131tst\_reap\_children() to collect and dissect deceased. 132 133Also note that tst\_brk() does exit only the current process, so if a child 134process calls tst\_brk() the counters are incremented and only the process 135exits. 136 137### Test library and exec() 138 139The piece of mapped memory to store the results to is not preserved over 140exec(2), hence to use the test library from a binary started by an exec() it 141has to be remaped. In this case the process must to call tst\_reinit() before 142calling any other library functions. In order to make this happen the program 143environment carries LTP\_IPC\_PATH variable with a path to the backing file on 144tmpfs. This also allows us to use the test library from shell testcases. 145 146### Test library and process synchronization 147 148The piece of mapped memory is also used as a base for a futex-based 149synchronization primitives called checkpoints. And as said previously the 150memory can be mapped to any process by calling the tst\_reinit() function. As a 151matter of a fact there is even a tst\_checkpoint binary that allows us to use 152the checkpoints from shell code as well. 153