1LTP Test Writing Guidelines 2=========================== 3 4This document describes LTP guidelines and is intended for anybody who want to 5write or modify a LTP testcase. It's not a definitive guide and it's not, by 6any means, a substitute for common sense. 7 8NOTE: See also 9 https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API], 10 https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API], 11 https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines[LTP Library API Writing Guidelines]. 12 13Rules and recommendations which are "machine checkable" should be 14tagged with an ID like +LTP-XXX+. There will be a corresponding entry 15in 16https://github.com/linux-test-project/ltp/tree/master/doc/rules.tsv[doc/rules.tsv]. When 17you run 'make check' or 'make check-test' it will display these IDs as 18a reference. 19 201. Guide to clean and understandable code 21~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 22 23For testcases it's required that the source code is as easy to follow as 24possible. When a test starts to fail the failure has to be analyzed, clean 25test codebase makes this task much easier and quicker. 26 27Here are some hints on how to write clean and understandable code, a few of 28these points are further discussed below: 29 30* First of all *Keep things simple* 31 32* Keep function and variable names short but descriptive 33 34* Keep functions reasonably short and focused on a single task 35 36* Do not overcomment 37 38* Be consistent 39 40* Avoid deep nesting 41 42* DRY 43 441.1 Keep things simple 45~~~~~~~~~~~~~~~~~~~~~~ 46 47For all it's worth keep the testcases simple or better as simple as possible. 48 49The kernel and libc are tricky beasts and the complexity imposed by their 50interfaces is quite high. Concentrate on the interface you want to test and 51follow the UNIX philosophy. 52 53It's a good idea to make the test as self-contained as possible too, ideally 54tests should not depend on tools or libraries that are not widely available. 55 56Do not reinvent the wheel! 57 58* Use LTP standard interface 59 60* Do not add custom PASS/FAIL reporting functions 61 62* Do not write Makefiles from scratch, use LTP build system instead 63 64* Etc. 65 661.2 Keep functions and variable names short 67~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 68 69Choosing a good name for an API functions or even variables is a difficult 70task do not underestimate it. 71 72There are a couple of customary names for different things that help people to 73understand code, for example: 74 75* For loop variables are usually named with a single letter 'i', 'j', ... 76 77* File descriptors 'fd' or 'fd_foo'. 78 79* Number of bytes stored in file are usually named as 'size' or 'len' 80 81* Etc. 82 831.3 Do not overcomment 84~~~~~~~~~~~~~~~~~~~~~~ 85 86Comments can sometimes save you day but they can easily do more harm than 87good. There has been several cases where comments and actual implementation 88drifted slowly apart which yielded into API misuses and hard to find bugs. 89Remember there is only one thing worse than no documentation, wrong 90documentation. 91 92Ideally everybody should write code that is obvious, which unfortunately isn't 93always possible. If there is a code that requires to be commented keep it 94short and to the point. These comments should explain *why* and not *how* 95things are done. 96 97Never ever comment the obvious. 98 99In case of LTP testcases it's customary to add an asciidoc formatted comment 100paragraph with highlevel test description at the beginning of the file right 101under the GPL SPDX header. This helps other people to understand the overall 102goal of the test before they dive into the technical details. It's also 103exported into generated documentation hence it should mostly explain what is 104tested. 105 1061.4 DRY (Code duplication) 107~~~~~~~~~~~~~~~~~~~~~~~~~~ 108 109Copy & paste is a good servant but very poor master. If you are about to copy a 110large part of the code from one testcase to another, think what would happen if 111you find bug in the code that has been copied all around the tree. What about 112moving it to a library instead? 113 114The same goes for short but complicated parts, whenever you are about to copy & 115paste a syscall wrapper that packs arguments accordingly to machine 116architecture or similarly complicated code, put it into a header instead. 117 1182 Coding style 119~~~~~~~~~~~~~~ 120 1212.1 C coding style 122^^^^^^^^^^^^^^^^^^ 123 124LTP adopted Linux kernel coding style: 125https://www.kernel.org/doc/html/latest/process/coding-style.html 126 127If you aren't familiar with its rules please read it, it's a well written 128introduction. 129 130Run `make check` in the test's directory and/or use `make check-$TCID`, 131it uses (among other checks) our vendored version of 132https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl[checkpatch.pl] 133script from kernel git tree. 134 135NOTE: If `make check` does not report any problems, the code still may be wrong 136 as all tools used for checking only look for common mistakes. 137 1382.1.1 LTP-004: Test executable symbols are marked static 139^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 140 141Test executables should not export symbols unnecessarily. This means 142that all top-level variables and functions should be marked with the 143static keyword. The only visible symbols should be those included from 144shared object files. 145 1462.2 Shell coding style 147^^^^^^^^^^^^^^^^^^^^^^ 148 149When writing testcases in shell write in *portable shell* only, it's a good 150idea to try to run the test using alternative shell (alternative to bash, for 151example dash) too. 152 153*Portable shell* means Shell Command Language as defined by POSIX with a 154exception of few widely used extensions, namely 'local' keyword used inside of 155functions and '-o' and '-a' test parameters (that are marked as obsolete in 156POSIX). 157 158You can either try to run the testcases on Debian which has '/bin/sh' pointing 159to 'dash' by default or install 'dash' on your favorite distribution and use 160it to run the tests. If your distribution lacks 'dash' package you can always 161compile it from http://gondor.apana.org.au/~herbert/dash/files/[source]. 162 163Run `make check` in the test's directory and/or use `make check-$TCID.sh`, 164it uses (among other checks) our vendored version of 165https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl[checkbashism.pl] 166from Debian, that is used to check for non-portable shell code. 167 168NOTE: If `make check` does not report any problems, the code still may be wrong 169 as `checkbashisms.pl` used for checking only looks for common mistakes. 170 171Here are some common sense style rules for shell 172 173* Keep lines under 80 chars 174 175* Use tabs for indentation 176 177* Keep things simple, avoid unnecessary subshells 178 179* Don't do confusing things (i.e. don't name your functions like common shell 180 commands, etc.) 181 182* Quote variables 183 184* Be consistent 185 1863 Backwards compatibility 187~~~~~~~~~~~~~~~~~~~~~~~~~ 188 189LTP test should be as backward compatible as possible. Think of an enterprise 190distributions with long term support (more than five years since the initial 191release) or of an embedded platform that needs to use several years old 192toolchain supplied by the manufacturer. 193 194Therefore LTP test for more current features should be able to cope with older 195systems. It should at least compile fine and if it's not appropriate for the 196configuration it should return 'TCONF'. 197 198There are several types of checks we use: 199 200The *configure script* is usually used to detect availability of a function 201declarations in system headers. It's used to disable tests at compile time or 202to enable fallback definitions. 203 204Checking the *errno* value is another type of runtime check. Most of the 205syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented 206or was disabled upon kernel compilation. 207 208LTP has kernel version detection that can be used to disable tests at runtime, 209unfortunately kernel version does not always corresponds to a well defined 210feature set as distributions tend to backport hundreds of patches while the 211kernel version stays the same. Use with caution. 212 213Lately we added kernel '.config' parser, a test can define a boolean 214expression of kernel config variables that has to be satisfied in order for a 215test to run. This is mostly used for kernel namespaces at the moment. 216 217Sometimes it also makes sense to define a few macros instead of creating 218configure test. One example are Linux specific POSIX clock ids in 219'include/lapi/posix_clocks.h'. 220 2213.1 Dealing with messed up legacy code 222~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 223 224LTP still contains a lot of old and messy code and we are cleaning it up as 225fast as we can but despite the decade of efforts there is still a lot. If you 226start modifying old or a messy testcase and your changes are more complicated 227than simple typo fixes you should convert the test into a new library first. 228 229It's also much easier to review the changes if you split them into a smaller 230logical groups. The same goes for moving files. If you need a rename or move 231file do it in a separate patch. 232 2334 License 234~~~~~~~~~ 235 236Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or 237any later version). 238 239Use `SPDX-License-Identifier: GPL-2.0-or-later` 240 2415 LTP Structure 242~~~~~~~~~~~~~~~ 243 244The structure of LTP is quite simple. Each test is a binary written either in 245portable shell or C. The test gets a configuration via environment variables 246and/or command line parameters, it prints additional information into the 247stdout and reports overall success/failure via the exit value. 248 249Tests are generally placed under the 'testcases/' directory. Everything that 250is a syscall or (slightly confusingly) libc syscall wrapper goes under 251'testcases/kernel/syscalls/'. 252 253Then there is 'testcases/open_posix_testsuite/' which is a well maintained fork 254of the upstream project that has been dead since 2005 and also a number of 255directories with tests for more specific features. 256 2575.1 Runtest Files 258^^^^^^^^^^^^^^^^^ 259 260The list of tests to be executed is stored in runtest files under the 261'runtest/' directory. The default set of runtest files to be executed is 262stored in 'scenario_groups/default'. When you add a test you should add 263corresponding entries into some runtest file(s) as well. 264 265For syscall tests (these placed under 'testcases/kernel/syscalls/') use 266'runtest/syscalls' file, for kernel related tests for memory management we 267have 'runtest/mm', etc. 268 269IMPORTANT: The runtest files should have one entry per a test. Creating a 270 wrapper that runs all your tests and adding it as a single test 271 into runtest file is strongly discouraged. 272 2735.2 Datafiles 274^^^^^^^^^^^^^ 275 276If your test needs datafiles to work, these should be put into a subdirectory 277named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to 278do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the 279'datafiles/Makefile'). 280 281You can obtain path to datafiles via $TST_DATAROOT provided by test.sh 282'$TST_DATAROOT/...' 283or via C function 'tst_dataroot()' provided by libltp: 284 285[source,c] 286------------------------------------------------------------------------------- 287const char *dataroot = tst_dataroot(); 288------------------------------------------------------------------------------- 289 290Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...', 291but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used 292when running testcases directly in git tree as well as from install 293location. 294 295The path is constructed according to these rules: 296 2971. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID' 2982. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles' 299 (where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()') 3003. else return '$CWD/datafiles' 301 302See 'testcases/commands/file/' for example. 303 3045.3 Subexecutables 305^^^^^^^^^^^^^^^^^^ 306 307If you test needs to execute a binary, place it in the same directory as the 308testcase and name the file starting with '${test_binary_name}_'. Once the 309test is executed by the framework, the path to the directory with all LTP 310binaries is added to the '$PATH' and you can execute it just by its name. 311 312TIP: If you need to execute such test from the LTP tree, you can add path to 313 current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'. 314 3156 Test Contribution Checklist 316------------------------------ 317 318NOTE: See also 319 https://github.com/linux-test-project/ltp/wiki/Maintainer-Patch-Review-Checklist[Maintainer Patch Review Checklist]. 320 3211. Test compiles and runs fine (check with `-i 10` too) 3222. `make check` does not emit any warnings for the test you are working on 323 (hint: run it in the test's directory and/or use `make check-$TCID`) 3243. The runtest entries are in place 3254. Test binaries are added into corresponding '.gitignore' files 3265. Patches apply over the latest git 327 3286.1 About .gitignore files 329~~~~~~~~~~~~~~~~~~~~~~~~~~ 330 331There are numerous '.gitignore' files in the LTP tree. Usually there is a 332'.gitignore' file per a group of tests. The reason for this setup is simple. 333It's easier to maintain a '.gitignore' file per directory with tests, rather 334than having single file in the project root directory. This way, we don't have 335to update all the gitignore files when moving directories, and they get deleted 336automatically when a directory with tests is removed. 337