1# Test generator and test runner tools 2 3 4## 5 6Tests are being generated during building. 7 8* `CTS_TEST_SELECT_OPTION` - options, passed to test-runner.rb. 9Useful for defining `--exclude-tag` and `--include-tag` options. 10* `PANDA_CTS_JOBS_NUMBER` - amount of parallel jobs for test execution. Default is 8. 11 12## Test generator 13 14Options: 15 16``` 17Usage: generate-cts.rb [options] 18 -t, --template FILE Path to template yaml file to generate tests (required) 19 -s, --schema FILE Path to json schema for template yaml (required) 20 -k, --skip Skip yaml schema validation 21 -o, --output DIR Path to directory where tests will be generated (required) 22 -h, --help Prints this help 23``` 24 25Usage example: 26 27``` 28 ${PANDA_SRC_ROOT}/tests/cts-generator/generate-cts.rb \ 29 -t ${PANDA_SRC_ROOT}/tests/cts-generator/cts-template/template.yaml \ 30 -s ${PANDA_SRC_ROOT}/tests/cts-generator/cts-template/yaml-schema.json \ 31 -o cts-generated 32``` 33 34This command will generate CTS tests using provided template file. Template is validated using schema file. 35 36## Test runner 37 38Options: 39 40``` 41Usage: test-runner.rb [options] 42 -p, --panda-build DIR Path to panda build directory (required) 43 -t, --test-dir DIR Path to test directory to search tests recursively, or path to single test (required) 44 -v, --verbose LEVEL Set verbose level 1..5 45 --verbose-verifier Allow verifier to produce extended checking log 46 --aot-mode Perform AOT compilation on test sources 47 --timeout SECONDS Set process timeout, default is 30 seconds 48 --dump-timeout SECONDS Set process completion timeout, default is 30 seconds 49 --enable-core-dump Enable core dumps 50 --verify-tests Run verifier against positive tests (option for test checking) 51 --with-quickener Run quickener tool after assembly 52 --global-timeout SECONDS Set testing timeout, default is 0 (ulimited) 53 -a, --run-all Run all tests, ignore "runner-option: ignore" tag in test definition 54 --run-ignored Run ignored tests, which have "runner-option: ignore" tag in test definition 55 --reporter TYPE Reporter for test results (default 'log', available: 'log', 'jtr', 'allure') 56 --report-dir DIR Where to put results, applicable for 'jtr' and 'allure' logger 57 --verifier-config PATH Path to verifier config file 58 -e, --exclude-tag TAG Exclude tags for tests 59 -o, --panda-options OPTION Panda options 60 -i, --include-tag TAG Include tags for tests 61 -b, --bug_id BUGID Include tests with specified bug ids 62 -j, --jobs N Amount of concurrent jobs for test execution (default 8) 63 --prlimit OPTS Run panda via prlimit with options 64 --plugins PLUGINS Paths to runner plugins 65 -h, --help Prints this help 66``` 67 68Usage example: 69 70``` 71${PANDA_SRC_ROOT}/tests/cts-generator/test-runner.rb 72 -t cts-generated \ 73 -p ${PANDA_BUILD_ROOT} \ 74 -e release -e debug \ 75 -i sanitizer-fail,wrong-tag 76``` 77 78This command will start all tests in `cts-generated` directory. Tests which have runner options `ignore` will be ignored. 79Tests that have `release` and `sanitizer-fail` will be excluded. 80 81To run all tests, add `-a` options. 82 83To run only tests with `ignore` runner option, add `--run-ignored` options. 84 85## Tips 86 87### How to specify options for cmake cts-generator target? 88 89`CTS_TEST_SELECT_OPTION` variable can be used. 90 91``` 92cmake -DCTS_TEST_SELECT_OPTION="-b 1316 -a" ../panda 93``` 94 95Run all tests marked with bugid 1316. 96 97### How to generate all test not using cmake/make 98 99``` 100cd ${ROOT_PATH}/tests/cts-generator 101./generate-cts.rb \ 102 -t ./cts-template/template.yaml \ 103 -s ./cts-template/yaml-schema.json \ 104 -o cts-generated 105``` 106 107### How to run all tests 108 109All test can be executed using `make cts-generated` command, test with `ignore` runner options will be ignored by test runner. 110If you want to run all tests, you can do the following: 111 112``` 113cmake ${ROOT_PATH} -DCTS_TEST_SELECT_OPTION="--run-all" 114make cts-generator 115``` 116 117Also you can start test-runner.rb directly: 118 119``` 120cd ${BUILD_DIR}/tests/cts-generator 121./test-runner.rb 122 -p ${BUILD_DIR} \ 123 -t ./cts-generated/ \ 124 -v 2 -j 8 \ 125 -i release \ 126 -e sanitizer-fail 127``` 128 129Tests with `release` tag will be included to test execution, with `sanitizer-fail` will be excluded. 130 131### How to run test with specified bug id runner-option? 132 133Example: 134``` 135test-runner.rb \ 136 -t ./cts-generated/ \ 137 -p ${BUILD_DIR} \ 138 -b 977 \ 139 -v 2 \ 140 -a 141``` 142 143Please note that `-a` options (`--run-all`) is defined, otherwise tests will be excluded, if they have `ignore` runner option. 144 145### How run panda via prlimit? 146 147Example: 148``` 149test-runner.rb \ 150 -t ./cts-generated/ \ 151 -p ${BUILD_DIR} \ 152 --prlimit='--stack=8000000:8000000 --nproc=512' 153``` 154 155 156### What should I do with failed/passed tests after bug-fixing? 157 1581. Run all tests using regular build or using `tests` or `cts-geerator` targets. 1592. Run tests related to bug. 160 ``` 161 cmake -DCTS_TEST_SELECT_OPTION="-b 1316 -a" ../panda 162 make cts-generator 163 ``` 1643. If all test passed, congrats! Now you can enable tests for regular 165execution. Update all tests that have `bugid: [number]` and `ignore: true` - remove bugid relation and `ignore` tag 166to allow test runner to execute test. 167 1684. If some tests failed, you can update them, if there are problems with tests. If tests are correct, continue with bugfixing and repeat all steps. 169