1# Universal test runner 2 3## Prerequisites 4- Panda build 5- Python3 with required libs (`tqdm`, `dataclasses`, `python-dotenv`, etc). Make sure that `scripts/install-deps-ubuntu` has run with option `-i=test` 6- Suite `ets-es-checked` requires [node and some packages](#ets-es-checked-dependencies) 7 8## Quick run 9 10It is possible to run tests either using script `runner.sh` or `main.py` directly. 11 12### Shell script 13Script `runner.sh` activates the virtual environment with all required libraries 14installed by `scripts/install-deps-ubuntu -i=test` and then runs test(s). 15This way is preferable. 16 17After install script finishes you can run 18 19```bash 20export PROJECT=/path/to/panda 21export BUILD=/path/to/panda/build 22 23$PROJECT/tests/tests-u-runner/runner.sh <test-suite-name> --build-dir $BUILD 24``` 25 26List of possible values for `<test-suite-name>` is below. 27 28### Python script 29You can run `main.py` directly. In order to do so you have to activate 30the virtual environment `$HOME/.venv-panda` manually or propose all required 31libraries in your working environment. Then `main.py` will run test(s) for you. 32 33```bash 34export PROJECT=/path/to/panda 35export BUILD=/path/to/panda/build 36 37python3 $PROJECT/tests/tests-u-runner/main.py <test-suite-name> --build-dir $BUILD 38``` 39 40### Supported ETS test suites 41 42- `--parser` - parser (aka regression) 43 Tests folders: 44 - `ets_frontend/ets2panda/test/parser/ets` 45 - `ets_frontend/ets2panda/test/compiler/ets` 46- `--ets-func-tests` - tests for ArkTS: standard library and ets func tests 47 Tests folder: `plugins/ets/tests/ets_func_tests/` 48 Chapters: 49 - `all` 50 - `algorithms-dir` 51 - `escompat-dir` 52 - `regression-dir` 53 - `spec-dir` 54 - `std-containers-dir` 55 - `std-core-dir` 56 - `std-math-dir` 57 - `std-time-dir` 58 - `chapter-1-of-4` 59 - `chapter-2-of-4` 60 - `chapter-3-of-4` 61 - `chapter-4-of-4` 62- `--ets-es-checked` - `ETS tests that cross validate results with ts` 63 `--ets-gc-stress` - `GC stress tests` 64- `--ets-runtime` - `ETS runtime tests` 65 Tests folder: `ets_frontend/ets2panda/test/runtime/ets` 66- `--ets-cts` - CTS language specification tests 67 Tests folder: `plugins/ets/tests/ets-templates` 68 Chapters: 69 - `all` 70 - `chapter02` - `02.lexical_elements` 71 - `chapter03` - `03.types` 72 - `chapter04` - `04.names_declarations_and_scopes` 73 - `chapter05` - `05.generics` 74 - `chapter06` - `06.contexts_and_conversions` 75 - `chapter07` - `07.expressions` 76 - `chapter08` - `08.statements` 77 - `chapter09` - `09.classes` 78 - `chapter10` - `10.interfaces` 79 - `chapter11` - `11.enumerations` 80 - `chapter12` - `12.errors_handling` 81 - `chapter13` - `13.modules_and_compilation_units` 82 - `chapter15` - `15.semantic_rules` 83 - `chapter17` - `17.experimental_features` 84 - `chapters02-05` 85 - `chapters06-09` 86 - `chapters10-12` 87 - `chapters13-17` 88- `--ets-sdk` - tests for ArkTS SDK 89 Tests folder: `plugins/ets/tests/ets_sdk/` 90 91 92### Supported JS test suites 93 94- `--parser` - parser (aka regression) 95- `--hermes` - Hermes JS runtime tests. To run tests from the Hermes suite, specify environment variables `HERMES_REVISION` and `HERMES_URL` in the `.env` file. 96- `--test262` - Test262 JS parser and runtime tests. To run tests from the test262 suite, specify environment variables `TEST262_REVISION` and `TEST262_URL` in the `.env` file. 97 98Additionally, a test suite can be specified with the option `--test-suite`: for example, for test262 `--test-suite test262`, 99for ets-func-tests `--test-suite ets_func_tests` 100 101### ArkTS important notes 102 103The full command for running all stdlib tests and ets_func_tests would be: 104 105`$PROJECT/tests/tests-u-runner/runner.sh --ets-func-tests --build-dir $BUILD` 106 107or 108 109`python3 $PROJECT/tests/tests-u-runner/main.py --ets-func-tests --build-dir $BUILD` 110 111ArkTS STDLIB tests are generated automatically from Jinja2 template. By default, runner starts test code generation automatically. 112As well test generator can be run as a standalone command, the corresponding procedure is described in plugins/ets/tests/stdlib-templates/readme.md 113 114## Reproducing CI test run 115 116In case of fail on CI runner will show its options - use it to run test locally. 117 118For example, to run the test262 test suite with AOT FULL INLINE the build options could be following: 119 120`$PROJECT/tests/tests-u-runner/runner.sh --test262 --gc-type=g1-gc --aot --aot-args='--compiler-inline-full-intrinsics=true' --aot-args='--compiler-memory-size-limit=4294967296' --build-dir $BUILD` 121 122## Yaml configuration files 123 124Any option can be set through yaml configuration file as `--config $YAML_CONFIG_FILE`. It can be mentioned several times 125and the data are merged according to following rules: 126- If some keys are repeated the latest value is taken (from the last specified config file) 127- If for a key different types are specified the first-met type is taken 128- If for a key different lists are specified they are merged and repeated values are excluded. 129 130The folder `cfg` contains several example config files which should be edited to set the real `$panda` and `$panda_build` paths, 131and any custom paths. 132 133To see full list of supported options use option `--generate-config $YAML_CONFIG_FILE`: 134 135`$PROJECT/tests/tests-u-runner/runner.sh --generate-config $YAML_CONFIG_FILE <test-suite-name> --build-dir $BUILD` 136 137The generated file will contain all supported options with default values. 138 139> **Note**: if an option is set both in yaml config file and in command line interface the latter value will 140> be applied. So cli value has a higher priority. 141 142## Important runner options 143 144Note: after `/` the option name from config file is specified. 145 146- `--build-dir`/`general.build` - the path to the compiled project. Referenced as $BUILD in this readme. 147- `--test-root`/`general.test-root` - the folder where test suite is located. It must exist before runner starts. By default, Hermes and Test262 test suites are downloaded to `/tmp/<test-suite-name>` folder. Then they are unpacking, transforming and copying to `<panda-build>\<test-suite-name>` folder 148- `--list-root`/`general.list-root` - the folder where test lists are located. It must exist before runner starts. By default, it's the test suite plugin. 149- `--work-dir`/`general.work-dir` - path to the working temp folder with gen, intermediate and report folders. 150where tests are generated from templates. For Hermes and Test262 it's a path where tests transformed for run are copied. 151 152## Test lists 153 154Runner supports following kinds of test lists: 155 156- **excluded** test lists - tests specified here are excluded from execution. "Don't try to run" 157- **ignored** test lists - tests specified here run, but failures are ignored. Such lists are named "known failures list" or KFL as well. 158 159Test list name usually has the following format: `<test-suite-name>[-<additional info>]-<kind>[-<OS>][-<architecture>][-<configuration>][-<sanitizer>][-<opt-level>][-<debug-info>][-REPEATS][-<build-type>].txt` 160 161- `kind` is one of `excluded` or `ignored` 162- `OS` is one of `LIN`, `WIN`, `MAC`. If an operating system is set explicitly, the test list is applied only to this OS. If none is set, the list is applied to any OS. 163- `architecture` is one of `ARM32`, `ARM64`, `AMD32`, `AMD64`. If an architecture is set explicitly, the test list is applied only to this architecture. If none is set, the list is applied to any architecture. 164- `configuration` is one of `INT`, `AOT`, `AOT-FULL`, `IRTOC`, `LLVM`, `JIT` or other used value for `interpreter-type` option. If a configuration is set explicitly, the test list is applied only to this configuration. If none is set, the list is applied to any configuration. 165- `sanitizer` is one of `ASAN` or `TSAN`. If a sanitizer is set explicitly, the test list is applied only to this build configuration. If none is set, the list is applied to any configuration. 166- `opt-level` is `OLx`, where `x` is opt-level, usually 0 or 2. 167- `debug-info` is `DI` when `--es2panda-debug-info` option is specified. 168- `REPEATS` is set if the test list should apply to runs with the option `--jit-repeats` sets number of repeats more than 1. 169- `build-type` is one of `DEBUG`, `RELEASE`, `FAST-VERIFY`. If a build type is set explicitly, the test list is applied only to this build type. If none is set, the list is applied to any build type. 170 171Examples: 172 173- `test262-flaky-ignored-JIT.txt` - list of ignored tests from Test262 suite, applied only in JIT configuration. `flaky` is additional info, it's only for more clear description what tests are collected there. 174- `hermes-excluded.txt` - list of excluded tests from Hermes suite, applied in any configuration. 175- `parser-js-ignored.txt` - list of ignored tests from JS Parser suite, applied in any configuration. 176- `ets-func-tests-ignored.txt` - list of ignored tests in `ets-func-tests` test suite 177- `ets-cts-FastVerify-ignored-OL2.txt` - list of ignored tests for `ets-cts` test suite and for opt-level=2. `FastVerify` is additional info. 178 179In any test list the test is specified as path to the test file relative from the `test_root`: Examples: 180 181- array-large.js 182- built-ins/Date/UTC/fp-evaluation-order.js 183- tests/stdlib/std/math/sqrt-negative-01.ets 184 185Test file specified in the option `--test-file` should be set in this format too. 186By default, ignored or excluded lists are located in correspondent project folder (for example for ArkTS should be `runtime_core/static_core/tests/tests-u-runner/test-lists`) 187 188All test lists are loaded automatically from the specified `LIST_ROOT` and based on following options: 189- architecture: 190 - from cli one of: `--test-list-arch=ARCH`, where ARCH is one of `amd64`, `arm32`, `arm64` 191 - from config file: `test-lists.architecture: ARCH`. Values are the same. 192- sanitizer: 193 - from cli on of: `--test-list-san=SAN`, where SAN in one of `asan` or `tsan` 194 - from config file: `test-lists.sanitizer: SAN`. Values are the same 195 196> **Note**: these options just specifies what test lists to load and do not affect on how and where to start the runner 197> itself and binaries used within. 198 199## Utility runner options: 200 201- `--skip-test-lists`/`test-lists.skip-test-lists: True` - do not use ignored or excluded lists, run all available tests, report all found failures 202- `--test-list TEST_LIST`/`test-lists.explicit-list: TEST_LIST` - run tests ONLY listed in TEST\_LIST file. 203- `--test-file TEST_FILE`/`test-lists.explicit-file: TEST_FILE` - run ONLY ONE specified test. **Attention** - not test suite, but the single test from the suite. 204- `--update-excluded`/`test-lists.update-excluded: True` - regenerates excluded test lists 205- `--update-expected`/`test-lists.update-expected: True` - regenerates expected test lists (applied only for JS Parser test suite) 206- `--report-format`/`report.report-format` - specifies in what format to generate failure reports. By default, `md`. Possible value: `html`. As well reports in the plain text format with `.log` extension are always generated. 207- `--filter FILTER`/`test-lists.filter: FILTER` - test filter regexp 208- `--show-progress`/`general.show-progress: True` - show progress bar during test execution 209- `--time-report`/`time-report.enable: True` - generates report with grouping tests by execution time. 210 211## Detailed report 212 213Detailed report shows test statistics for every folder 214- `--detailed-report`/`report.detailed-report: True` - if it's specified the report is generated 215- `--detailed-report-file FILE`/`report.detailed-report-file: FILE` - specifies file/path where the report should be saved to 216 217## ArkTS Specification coverage report 218 219ArkTS Specification coverage report shows test statistics for every specification chapter or section 220- `--spec-report`/`report.spec-report: True` - if it's specified the two report files are generated 221- `--spec-report-file FILE`/`report.spec-report-file: FILE` - specifies file/path to output the markdown-formatted report 222- `--spec-report-yaml FILE`/`report.spec-report-yaml: FILE` - specifies file/path to output the YAML-formatted report 223- `--spec-file FILE`/`report.spec-file: FILE` - specifies file/path to the specification PDF file 224 225## Verbose and logging options: 226 227- `--verbose`, `-v` - Enable verbose output. Possible values one of: 228 - `all` - the most detailed output, 229 - `short` - test status and output. 230 - if none specified (by default): only test status for new failed tests 231 - in config file use `general.verbose` property with the save values. 232- `--verbose-filter` - Filter for what kind of tests to output stdout and stderr. 233 Supported values: 234 - `all` - for all executed tests both passed and failed. 235 - `ignored` - for new failures and tests from ignored test lists including both passed and failed. ' 236 - `new` - only for new failures. Default value. 237 - in config file use `general.verbose-filter` property with the same values. 238 239## Generation options: 240 241- `--generate-only`/`general.generate-only` - only generate tests without running them. Tests are run as usual without this option. 242- `--force-generate`/`ets.force-generate` - force ETS tests generation from templates 243 244## Timeout options: 245 246All timeouts are set in seconds 247 248- `--es2panda-timeout`/`es2panda.timeout` - es2panda translator timeout 249- `--paoc-timeout`/`ark_aot.timeout` - paoc compiler timeout 250- `--timeout`/`ark.timeout` - ark runtime timeout 251 252## Configuration options: 253 254- `--aot`/`ark_aot.enable: True` - use AOT compilation 255- `--jit`/`ark.jit.enable` - use JIT in interpreter 256- `--interpreter-type`/`ark.interpreter-type` - use explicitly specified interpreter for es2panda. Popular values: `irtoc`, `llvm`, `cpp`. 257 258## JIT specific option 259- `--jit-preheat-repeats` - compound option to set additional parameters for JIT preheat actions. 260 Example how to set and supported values (they are default as well): `"num_repeats=30,compiler_threshold=20"`. 261 Works only with `--jit` options. If to use it with default values just set the empty string: `--jit-preheat-repeats=""` 262- in config file use `ark.jit.num_repeats` and `ark.jit.compiler_threshold` with explicit values. 263 264## Other options: 265 266To get runner all options use: `main.py --help` or `runner.sh --help` 267 268``` 269 --no-run-gc-in-place enable --run-gc-in-place mode 270 --gc-type GC_TYPE Type of garbage collector 271 --heap-verifier Heap verifier options 272 --no-bco disable bytecodeopt 273 --arm64-compiler-skip 274 use skiplist for tests failing on aarch64 in AOT or JIT mode 275 --arm64-qemu launch all binaries in qemu aarch64 276 --arm32-qemu launch all binaries in qemu arm 277 --aot-args AOT_ARGS Additional arguments that will passed to ark_aot 278``` 279 280## Execution time report 281 282It is possible to collect statistics how long separate tests work. In the result report tests are grouped by execution time. 283 284The grouping edges are set in seconds. For example, the value `1 5 10` specifies 4 groups - less than 1 second, from 1 second to 5 seconds, from 5 seconds to 10 seconds and from 10 seconds and more. For the last group the report contains real durations. 285 286- Specify the option `--time-report`/`time-report.enable: True` 287- Specify the option `--time-edges TIME_EDGES` in the format "1 5 10" 288or in yaml config file `time-report.time-edges: [1, 5, 10]` in the list of integers format. 289- After test run the short report will be output to the console 290- And full report will be created at the `<report_root>/time_report.txt` 291 292 293## Linter and type checks 294 295A script running linter and type checks starts as: 296 297`$PROJECT/tests/tests-u-runner/linter.sh` 298 299It performs checks by running `pylint` and `mypy`. 300 301For `pylint` settings see `.pylintrc` file. For `mypy` settings see `mypy.ini` file. 302 303## ETS ES checked dependencies 304- ruby (installed by default with `$PROJECT/scripts/install-deps-ubuntu -i=dev`) 305- node and ts-node, to install them see commands below 306 307```bash 308sudo apt-get -y install npm 309sudo npm install -g n 310sudo n install 21.4.0 311cd $PROJECT/tests/tests-u-runner/tools/generate-es-checked 312npm install 313``` 314 315## Custom suite 316If you want to run arbitrary set of ETS tests with URunner you can use a custom suite like (minimal configuration): 317 318`$PROJECT/tests/tests-u-runner/runner.sh $PROJECT --ets-custom --build-dir $BUILD --custom-suite SUITE_NAME --custom-test-root TEST_ROOT --custom-list-root LIST_ROOT` 319 320where: 321- SUITE_NAME - a tag used to name the suite 322- TEST_ROOT - a path to the folder with tests (*.ets files or templates) 323- LIST_ROOT - a path to the folder with ignored/excluded lists. Note: SUITE_NAME is used as a <test-suite-name> in the name of the test list (see #Test lists for details) 324 325In yaml configuration file these options can be specified as: 326```yaml 327 custom: 328 suite-name: SUITE_NAME 329 test-root: TEST_ROOT 330 list-root: LIST_ROOT 331``` 332 333### Custom generator 334It is possible to specify a custom generator. It should be executable file accepting following parameters: 335 336`custom_generator --source SOURCE_PATH --target TARGET_PATH <other-options>` 337 338Here: 339- SOURCE_PATH - path where template files are located. URunner fills this parameter with TEST_ROOT specified above. 340- TARGET_PATH - path where generated tests should be put. URunner fills this parameter with value from --work-dir option ($WorkDir/gen) 341- other-options - other options that might be required by the generator (optional) 342 343To specify generator and its options use: 344- `--custom-generator PATH`/`custom.generator: PATH` - path to the executable file-generator 345- `--custom-generator-option`/`custom.generator-options: [OPTIONS]` - multiple value option (can be specified several times) 346