LTP Shell Test API ================== NOTE: See also https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[Test Writing Guidelines], https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API]. 1 Writing a testcase in shell ----------------------------- LTP supports testcases to be written in a portable shell too. There is a shell library modeled closely to the C interface at 'testcases/lib/tst_test.sh'. WARNING: All identifiers starting with 'TST_' or 'tst_' are reserved for the test library. 1.1 Basic test interface ~~~~~~~~~~~~~~~~~~~~~~~~ [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # This is a basic test for true shell builtin TST_TESTFUNC=do_test . tst_test.sh do_test() { true ret=$? if [ $ret -eq 0 ]; then tst_res TPASS "true returned 0" else tst_res TFAIL "true returned $ret" fi } tst_run ------------------------------------------------------------------------------- TIP: To execute this test the 'tst_test.sh' library must be in '$PATH'. If you are executing the test from a git checkout you can run it as 'PATH="$PATH:../../lib" ./foo01.sh' The shell library expects test setup, cleanup and the test function executing the test in the '$TST_SETUP', '$TST_CLEANUP' and '$TST_TESTFUNC' variables. Both '$TST_SETUP' and '$TST_CLEANUP' are optional. The '$TST_TESTFUNC' may be called several times if more than one test iteration was requested by passing right command line options to the test. The '$TST_CLEANUP' may be called even in the middle of the setup and must be able to clean up correctly even in this situation. The easiest solution for this is to keep track of what was initialized and act accordingly in the cleanup. WARNING: Similar to the C library, calling 'tst_brk' in the $TST_CLEANUP does not exit the test and 'TBROK' is converted to 'TWARN'. Notice also the 'tst_run' shell API function called at the end of the test that actually starts the test. WARNING: cleanup function is called only after 'tst_run' has been started. Calling 'tst_brk' in shell libraries, e.g. 'tst_test.sh' or 'tst_net.sh' does not trigger calling it. [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Example test with tests in separate functions TST_TESTFUNC=test TST_CNT=2 . tst_test.sh test1() { tst_res TPASS "Test $1 passed" } test2() { tst_res TPASS "Test $1 passed" } tst_run # output: # foo 1 TPASS: Test 1 passed # foo 2 TPASS: Test 2 passed ------------------------------------------------------------------------------- If '$TST_CNT' is set, the test library looks if there are functions named '$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are found they are executed one by one. The test number is passed to it in the '$1'. [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Example test with tests in a single function TST_TESTFUNC=do_test TST_CNT=2 . tst_test.sh do_test() { case $1 in 1) tst_res TPASS "Test $1 passed";; 2) tst_res TPASS "Test $1 passed";; esac } tst_run # output: # foo 1 TPASS: Test 1 passed # foo 2 TPASS: Test 2 passed ------------------------------------------------------------------------------- Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed to it in the '$1'. [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Example test with tests in a single function, using $TST_TEST_DATA and # $TST_TEST_DATA_IFS TST_TESTFUNC=do_test TST_TEST_DATA="foo:bar:d dd" TST_TEST_DATA_IFS=":" . tst_test.sh do_test() { tst_res TPASS "Test $1 passed with data '$2'" } tst_run # output: # foo 1 TPASS: Test 1 passed with data 'foo' # foo 2 TPASS: Test 1 passed with data 'bar' # foo 3 TPASS: Test 1 passed with data 'd dd' ------------------------------------------------------------------------------- It's possible to pass data for function with '$TST_TEST_DATA'. Optional '$TST_TEST_DATA_IFS' is used for splitting, default value is space. [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT TST_TESTFUNC=do_test TST_CNT=2 TST_TEST_DATA="foo bar" . tst_test.sh do_test() { case $1 in 1) tst_res TPASS "Test $1 passed with data '$2'";; 2) tst_res TPASS "Test $1 passed with data '$2'";; esac } tst_run # output: # foo 1 TPASS: Test 1 passed with data 'foo' # foo 2 TPASS: Test 2 passed with data 'foo' # foo 3 TPASS: Test 1 passed with data 'bar' # foo 4 TPASS: Test 2 passed with data 'bar' ------------------------------------------------------------------------------- '$TST_TEST_DATA' can be used with '$TST_CNT'. If '$TST_TEST_DATA_IFS' not specified, space as default value is used. Of course, it's possible to use separate functions. 1.2 Library environment variables and functions for shell ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Similarily to the C library various checks and preparations can be requested simply by setting right '$TST_NEEDS_FOO'. [options="header"] |============================================================================= | Variable name | Action done | 'TST_NEEDS_ROOT' | Exit the test with 'TCONF' unless executed under root. | | Alternatively the 'tst_require_root' command can be used. | 'TST_NEEDS_TMPDIR' | Create test temporary directory and cd into it. | 'TST_NEEDS_DEVICE' | Prepare test temporary device, the path to testing device is stored in '$TST_DEVICE' variable. The option implies 'TST_NEEDS_TMPDIR'. | 'TST_NEEDS_CMDS' | String with command names that has to be present for the test (see below). | 'TST_NEEDS_MODULE' | Test module name needed for the test (see below). | 'TST_NEEDS_DRIVERS' | Checks kernel drivers support for the test. | 'TST_NEEDS_KCONFIGS' | Checks kernel kconfigs support for the test (see below). | 'TST_NEEDS_KCONFIGS_IFS' | Used for splitting '$TST_NEEDS_KCONFIGS' variable, default value is comma, it only supports single character. | 'TST_TIMEOUT' | Maximum timeout set for the test in sec. Must be int >= 1, or -1 (special value to disable timeout), default is 300. Variable is meant be set in tests, not by user. It's an equivalent of `tst_test.timeout` in C, can be set via 'tst_set_timeout(timeout)' after test has started. |============================================================================= [options="header"] |============================================================================= | Function name | Action done | 'tst_set_timeout(timeout)' | Maximum timeout set for the test in sec. See 'TST_TIMEOUT' variable. |============================================================================= NOTE: Network tests (see testcases/network/README.md) use additional variables and functions in 'tst_net.sh'. Checking for presence of commands +++++++++++++++++++++++++++++++++ [source,sh] ------------------------------------------------------------------------------- #!/bin/sh ... TST_NEEDS_CMDS="modinfo modprobe" . tst_test.sh ... ------------------------------------------------------------------------------- Setting '$TST_NEEDS_CMDS' to a string listing required commands will check for existence each of them and exits the test with 'TCONF' on first missing. Alternatively the 'tst_require_cmds()' function can be used to do the same on runtime, since sometimes we need to the check at runtime too. 'tst_check_cmds()' can be used for requirements just for a particular test as it doesn't exit (it issues 'tst_res TCONF'). Expected usage is: [source,sh] ------------------------------------------------------------------------------- #!/bin/sh TST_TESTFUNC=do_test . tst_test.sh do_test() { tst_check_cmds cmd || return cmd --foo ... } tst_run ------------------------------------------------------------------------------- Locating kernel modules +++++++++++++++++++++++ The LTP build system can build kernel modules as well, setting '$TST_NEEDS_MODULE' to module name will cause the library to look for the module in a few possible paths. If module was found the path to it will be stored into '$TST_MODPATH' variable, if module wasn't found the test will exit with 'TCONF'. Alternatively the 'tst_require_module()' function can be used to do the same at runtime. 1.3 Optional command line parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Optional test command line parameters TST_OPTS="af:" TST_USAGE=usage TST_PARSE_ARGS=parse_args TST_TESTFUNC=do_test . tst_test.sh ALTERNATIVE=0 MODE="foo" usage() { cat << EOF usage: $0 [-a] [-f ] OPTIONS -a Enable support for alternative foo -f Specify foo or bar mode EOF } parse_args() { case $1 in a) ALTERNATIVE=1;; f) MODE="$2";; esac } do_test() { ... } tst_run ------------------------------------------------------------------------------- The 'getopts' string for optional parameters is passed in the '$TST_OPTS' variable. There are a few default parameters that cannot be used by a test, these can be listed with passing help '-h' option to any test. The function that prints the usage is passed in '$TST_USAGE', the help for the options implemented in the library is appended when usage is printed. Lastly the function '$PARSE_ARGS' is called with the option name in the '$1' and, if option has argument, its value in the '$2'. [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Optional test positional parameters TST_POS_ARGS=3 TST_USAGE=usage TST_TESTFUNC=do_test . tst_test.sh usage() { cat << EOF usage: $0 [min] [max] [size] EOF } min="$1" max="$2" size="$3" do_test() { ... } tst_run ------------------------------------------------------------------------------- You can also request a number of positional parameters by setting the '$TST_POS_ARGS' variable. If you do, these will be available as they were passed directly to the script in '$1', '$2', ..., '$n'. 1.4 Useful library functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Retrieving configuration variables ++++++++++++++++++++++++++++++++++ You may need to retrieve configuration values such as PAGESIZE, there is 'getconf' but as some system may not have it, you are advised to use 'tst_getconf' instead. Note that it implements subset of 'getconf' system variables used by the testcases only. [source,sh] ------------------------------------------------------------------------------- # retrieve PAGESIZE pagesize=`tst_getconf PAGESIZE` ------------------------------------------------------------------------------- Sleeping for subsecond intervals ++++++++++++++++++++++++++++++++ Albeit there is a sleep command available basically everywhere not all implementations can support sleeping for less than one second. And most of the time sleeping for a second is too much. Therefore LTP includes 'tst_sleep' that can sleep for defined amount of seconds, milliseconds or microseconds. [source,sh] ------------------------------------------------------------------------------- # sleep for 100 milliseconds tst_sleep 100ms ------------------------------------------------------------------------------- Retry a function call multiple times ++++++++++++++++++++++++++++++++++++ Sometimes an LTP test needs to retry a function call multiple times because the system is not ready to process it successfully on the first try. The LTP library has useful tools to handle the call retry automatically. 'TST_RETRY_FUNC()' will keep retrying for up to 1 second. If you want a custom time limit use 'TST_RETRY_FN_EXP_BACKOFF()'. Both methods return the value returned by the last 'FUNC' call. The delay between retries starts at 1 microsecond and doubles after each call. The retry loop ends when the function call succeeds or when the next delay exceeds the specified time (1 second for 'TST_RETRY_FUNC()'). The maximum delay is multiplied by TST_TIMEOUT_MUL. The total cumulative delay may be up to twice as long as the adjusted maximum delay. The C version of 'TST_RETRY_FUNC()' is a macro which takes two arguments: * 'FUNC' is the complete function call with arguments which should be retried multiple times. * 'SUCCESS_CHECK' is a macro or function which will validate 'FUNC' return value. 'FUNC' call was successful if 'SUCCESS_CHECK(ret)' evaluates to non-zero. Both retry methods clear 'errno' before every 'FUNC' call so your 'SUCCESS_CHECK' can look for specific error codes as well. The LTP library also includes predefined 'SUCCESS_CHECK' macros for the most common call conventions: * 'TST_RETVAL_EQ0()' - The call was successful if 'FUNC' returned 0 or NULL * 'TST_RETVAL_NOTNULL()' - The call was successful if 'FUNC' returned any value other than 0 or NULL. * 'TST_RETVAL_GE0()' - The call was successful if 'FUNC' returned value >= 0. [source,c] ------------------------------------------------------------------------------- /* Keep trying for 1 second */ TST_RETRY_FUNC(FUNC, SUCCESS_CHECK) /* Keep trying for up to 2*N seconds */ TST_RETRY_FN_EXP_BACKOFF(FUNC, SUCCESS_CHECK, N) ------------------------------------------------------------------------------- The shell version of 'TST_RETRY_FUNC()' is simpler and takes slightly different arguments: * 'FUNC' is a string containing the complete function or program call with arguments. * 'EXPECTED_RET' is a single expected return value. 'FUNC' call was successful if the return value is equal to EXPECTED_RET. [source,sh] ------------------------------------------------------------------------------- # Keep trying for 1 second TST_RETRY_FUNC "FUNC arg1 arg2 ..." "EXPECTED_RET" # Keep trying for up to 2*N seconds TST_RETRY_FN_EXP_BACKOFF "FUNC arg1 arg2 ..." "EXPECTED_RET" "N" ------------------------------------------------------------------------------- Checking for integers +++++++++++++++++++++ [source,sh] ------------------------------------------------------------------------------- # returns zero if passed an integer parameter, non-zero otherwise tst_is_int "$FOO" ------------------------------------------------------------------------------- Checking for integers and floating point numbers ++++++++++++++++++++++++++++++++++++++++++++++++ [source,sh] ------------------------------------------------------------------------------- # returns zero if passed an integer or floating point number parameter, # non-zero otherwise tst_is_num "$FOO" ------------------------------------------------------------------------------- Obtaining random numbers ++++++++++++++++++++++++ There is no '$RANDOM' in portable shell, use 'tst_random' instead. [source,sh] ------------------------------------------------------------------------------- # get random integer between 0 and 1000 (including 0 and 1000) tst_random 0 1000 ------------------------------------------------------------------------------- Formatting device with a filesystem +++++++++++++++++++++++++++++++++++ The 'tst_mkfs' helper will format device with the filesystem. [source,sh] ------------------------------------------------------------------------------- # format test device with ext2 tst_mkfs ext2 $TST_DEVICE # default params are $TST_FS_TYPE $TST_DEVICE tst_mkfs # optional parameters tst_mkfs ext4 /dev/device -T largefile ------------------------------------------------------------------------------- Mounting and unmounting filesystems +++++++++++++++++++++++++++++++++++ The 'tst_mount' and 'tst_umount' helpers are a safe way to mount/umount a filesystem. The 'tst_mount' mounts '$TST_DEVICE' of '$TST_FS_TYPE' (optional) to '$TST_MNTPOINT' (defaults to mntpoint), optionally using the '$TST_MNT_PARAMS'. The '$TST_MNTPOINT' directory is created if it didn't exist prior to the function call. If the path passed (optional, must be absolute path, defaults to '$TST_MNTPOINT') to the 'tst_umount' is not mounted (present in '/proc/mounts') it's noop. Otherwise it retries to umount the filesystem a few times on failure. This is a workaround since there are daemons dumb enough to probe all newly mounted filesystems, and prevents them from being umounted shortly after they were mounted. ROD and ROD_SILENT ++++++++++++++++++ These functions supply the 'SAFE_MACROS' used in C although they work and are named differently. [source,sh] ------------------------------------------------------------------------------- ROD_SILENT command arg1 arg2 ... # is shorthand for: command arg1 arg2 ... > /dev/null 2>&1 if [ $? -ne 0 ]; then tst_brk TBROK "..." fi ROD command arg1 arg2 ... # is shorthand for: ROD arg1 arg2 ... if [ $? -ne 0 ]; then tst_brk TBROK "..." fi ------------------------------------------------------------------------------- WARNING: Keep in mind that output redirection (to a file) happens in the caller rather than in the ROD function and cannot be checked for write errors by the ROD function. As a matter of a fact doing +ROD echo a > /proc/cpuinfo+ would work just fine since the 'ROD' function will only get the +echo a+ part that will run just fine. [source,sh] ------------------------------------------------------------------------------- # Redirect output to a file with ROD ROD echo foo \> bar ------------------------------------------------------------------------------- Note the '>' is escaped with '\', this causes that the '>' and filename are passed to the 'ROD' function as parameters and the 'ROD' function contains code to split '$@' on '>' and redirects the output to the file. EXPECT_PASS{,_BRK} and EXPECT_FAIL{,_BRK} +++++++++++++++++++++++++++++++++++++++++ [source,sh] ------------------------------------------------------------------------------- EXPECT_PASS command arg1 arg2 ... [ \> file ] EXPECT_FAIL command arg1 arg2 ... [ \> file ] ------------------------------------------------------------------------------- 'EXPECT_PASS' calls 'tst_res TPASS' if the command exited with 0 exit code, and 'tst_res TFAIL' otherwise. 'EXPECT_FAIL' does vice versa. Output redirection rules are the same as for the 'ROD' function. In addition to that, 'EXPECT_FAIL' always redirects the command's stderr to '/dev/null'. There are also 'EXPECT_PASS_BRK' and 'EXPECT_FAIL_BRK', which works the same way except breaking a test when unexpected action happen. It's possible to detect whether expected value happened: [source,sh] ------------------------------------------------------------------------------- if ! EXPECT_PASS command arg1 2\> /dev/null; then continue fi ------------------------------------------------------------------------------- tst_kvcmp +++++++++ This command compares the currently running kernel version given conditions with syntax similar to the shell test command. [source,sh] ------------------------------------------------------------------------------- # Exit the test if kernel version is older or equal to 2.6.8 if tst_kvcmp -le 2.6.8; then tst_brk TCONF "Kernel newer than 2.6.8 is needed" fi # Exit the test if kernel is newer than 3.8 and older than 4.0.1 if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1" fi ------------------------------------------------------------------------------- [options="header"] |======================================================================= | expression | description | -eq kver | Returns true if kernel version is equal | -ne kver | Returns true if kernel version is not equal | -gt kver | Returns true if kernel version is greater | -ge kver | Returns true if kernel version is greater or equal | -lt kver | Returns true if kernel version is lesser | -le kver | Returns true if kernel version is lesser or equal | -a | Does logical and between two expressions | -o | Does logical or between two expressions |======================================================================= The format for kernel version has to either be with one dot e.g. '2.6' or with two dots e.g. '4.8.1'. .tst_fs_has_free [source,sh] ------------------------------------------------------------------------------- #!/bin/sh ... # whether current directory has 100MB free space at least. if ! tst_fs_has_free . 100MB; then tst_brkm TCONF "Not enough free space" fi ... ------------------------------------------------------------------------------- The 'tst_fs_has_free' shell interface returns 0 if the specified free space is satisfied, 1 if not, and 2 on error. The second argument supports suffixes kB, MB and GB, the default unit is Byte. .tst_retry [source,sh] ------------------------------------------------------------------------------- #!/bin/sh ... # Retry ping command three times tst_retry "ping -c 1 127.0.0.1" if [ $? -ne 0 ]; then tst_resm TFAIL "Failed to ping 127.0.0.1" else tst_resm TPASS "Successfully pinged 127.0.0.1" fi ... ------------------------------------------------------------------------------- The 'tst_retry' function allows you to retry a command after waiting small amount of time until it succeeds or until given amount of retries has been reached (default is three attempts). 1.5 Restarting daemons ~~~~~~~~~~~~~~~~~~~~~~ Restarting system daemons is a complicated task for two reasons. * There are different init systems (SysV init, systemd, etc...) * Daemon names are not unified between distributions (apache vs httpd, cron vs crond, various syslog variations) To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that provides functions to start/stop/query daemons as well as variables that store correct daemon name. .Supported operations |============================================================================== | start_daemon() | Starts daemon, name is passed as first parameter. | stop_daemon() | Stops daemon, name is passed as first parameter. | restart_daemon() | Restarts daemon, name is passed as first parameter. | status_daemon() | Detect daemon status (exit code: 0: running, 1: not running). |============================================================================== .Variables with detected names |============================================================================== | CROND_DAEMON | Cron daemon name (cron, crond). | SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog). |============================================================================== .Cron daemon restart example [source,sh] ------------------------------------------------------------------------------- #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Cron daemon restart example TCID=cron01 TST_COUNT=1 . test.sh . daemonlib.sh ... restart_daemon $CROND_DAEMON ... tst_exit ------------------------------------------------------------------------------- 1.6 Access to the checkpoint interface ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The shell library provides an implementation of the checkpoint interface compatible with the C version. All 'TST_CHECKPOINT_*' functions are available. In order to initialize checkpoints '$TST_NEEDS_CHECKPOINTS' must be set to '1' before the inclusion of 'tst_test.sh': [source,sh] ------------------------------------------------------------------------------- #!/bin/sh TST_NEEDS_CHECKPOINTS=1 . tst_test.sh ------------------------------------------------------------------------------- Since both the implementations are compatible, it's also possible to start a child binary process from a shell test and synchronize with it. This process must have checkpoints initialized by calling 'tst_reinit()'. 1.7 Parsing kernel .config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The shell library provides an implementation of the kconfig parsing interface compatible with the C version. It's possible to pass kernel kconfig list for tst_require_kconfigs API with '$TST_NEEDS_KCONFIGS'. Optional '$TST_NEEDS_KCONFIGS_IFS' is used for splitting, default value is comma. ------------------------------------------------------------------------------- #!/bin/sh TST_NEEDS_KCONFIGS="CONFIG_EXT4_FS, CONFIG_QUOTACTL=y" . tst_test.sh -------------------------------------------------------------------------------