1# Universal test runner, version 2 2 3## Description 4Universal test runner, version 2, for Open Harmony. 5 6## Prerequisites 7- Panda build 8- Python3 version at least 3.8. 3.10 is recommended. 9- Make sure to run `sudo static_core/scripts/install-deps-ubuntu -i=test` to create a ~/.venv-panda virtual environment with 10all required python libraries (`tqdm`, `dataclasses`, `python-dotenv`, etc). 11- Suite `ets-es-checked` requires [node and some packages](#sts-es-checked-dependencies) 12 13## Quick run 14 15It is possible to run tests either using script `runner.sh` or `main.py` directly. 16 17### Before the first start 18- Create at your home the file `.urunner.env` 19- Specify there following environment variables: 20```bash 21ARKCOMPILER_RUNTIME_CORE_PATH=<your path to the folder with cloned repository arkcompiler_runtime_core> 22ARKCOMPILER_ETS_FRONTEND_PATH=<your path to the folder with cloned repository arkcompiler_ets_frontend> 23PANDA_BUILD=<your path to build folder> 24WORK_DIR=<your path to temporary folder where all intermediate files and reports will be kept> 25``` 26 27### Shell script 28Script `runner.sh` activates the virtual environment with all required libraries 29installed by `sudo static_core/scripts/install-deps-ubuntu -i=test` and then runs test(s). 30This way is preferable. 31 32After install script finishes you can run 33 34```bash 35urunner2> ./runner.sh <workflow-name> <test-suite-name> <other-options> 36``` 37 38### Python script 39You can run `main.py` directly. In order to do so you have to activate 40the virtual environment `$HOME/.venv-panda` manually or propose all required 41libraries in your working environment. Then `main.py` will run test(s) for you. 42 43```bash 44urunner>source ~/.venv-panda/bin/activate 45urunner>python3 main.py <workflow-name> <test-suite-name> <other-options> 46urunner>deactivate 47``` 48 49## Reproducing CI test run 50 51In case of fail on CI runner will show its options and used configurations files - use it to run test locally. 52 53 54## Yaml configuration files 55 56A lot of options can be set through yaml configuration file as `--config $YAML_CONFIG_FILE`. 57It can be mentioned several times and the data are merged according to following rules: 58- If some keys are repeated the latest value is taken (from the last specified config file) 59- If for a key different types are specified the first-met type is taken 60- If for a key different lists are specified they are merged and repeated values are excluded. 61 62The folder `cfg` contains several example config files which should be edited to set the real paths. 63 64To see full list of supported options use CLI option `--generate-config $YAML_CONFIG_FILE`: 65 66`urunner> ./runner.sh --generate-config $YAML_CONFIG_FILE --build-dir $BUILD` 67 68The generated file will contain all supported options with default values. 69 70> **Note**: if an option is set both in yaml config file and in command line interface the latter value will 71> be applied. So CLI value has a higher priority. 72 73## Important runner options 74 75Note: after `/` the option name from config file is specified. 76 77- `--build-dir`/`general.build` - the path to the compiled project. Referenced as $BUILD in this readme. 78- `--work-dir`/`general.work-dir` - path to the working temp folder with gen, intermediate and report folders. As well 79here tests are generated from templates. 80- `--test-root`/`custom.test-root` - the folder where test suite is located. It must exist before runner starts. 81- `--list-root`/`custom.list-root` - the folder where test lists are located. It must exist before runner starts. 82 83 84## Test lists 85 86Runner supports following kinds of test lists: 87 88- **excluded** test lists - tests specified here are excluded from execution. "Don't try to run" 89- **ignored** test lists - tests specified here run, but failures are ignored. Such lists are named "known failures list" or KFL as well. 90 91Test list name usually has the following format: `<test-suite-name>[-<additional info>]-<kind>[-<OS>][-<architecture>][-<configuration>][-<interpreter>][-<sanitizer>][-<opt-level>][-REPEATS][-<gc-type>][-<build-type>].txt` 92 93- `kind` is one of `excluded` or `ignored` 94- `OS` is one of `LIN`, `WIN`, `MAC`. If an operating system is set explicitly, the test list is applied only to this OS. If none is set, the list is applied to any OS. 95- `architecture` is one of `ARM32`, `ARM64`, `AMD32`, `AMD64`. If an architecture is set explicitly, the test list is applied only to this architecture. If none is set, the list is applied to any architecture. 96- `configuration` is one of `INT`, `AOT`, `AOT-FULL`, `IRTOC`, `LLVM`, `JIT`. If a configuration is set explicitly, the test list is applied only to this configuration. If none is set, the list is applied to any configuration. 97- `interpreter` used value for `interpreter-type` option. `DEFAULT` can be used to specify specific only for default interpreter. If an interpreter is set explicitly, the test list is applied only to this interpreter. If none is set, the list is applied to any interpreter. 98- `sanitizer` is one of `ASAN` or `TSAN`. If a sanitizer is set explicitly, the test list is applied only to this build configuration. If none is set, the list is applied to any configuration. 99- `opt-level` is `OLx`, where `x` is opt-level, usually 0 or 2. 100- `REPEATS` is set if the test list should apply to runs with the option `--jit-repeats` sets number of repeats more than 1. 101- `gc-type` is the value of option `--gc-type`. Usually, one of `G1-GC`, `GEN-GC`, `STW`. 102- `build-type` is one of `DEBUG`, `RELEASE`, `FAST-VERIFY`. If a build type is set explicitly, the test list is applied only to this build type. If none is set, the list is applied to any build type. 103 104Examples: 105 106- `test262-flaky-ignored-JIT.txt` - list of ignored tests from Test262 suite, applied only in JIT configuration. `flaky` is additional info, it's only for more clear description what tests are collected there. 107- `hermes-excluded.txt` - list of excluded tests from Hermes suite, applied in any configuration. 108- `parser-js-ignored.txt` - list of ignored tests from JS Parser suite, applied in any configuration. 109- `ets-func-tests-ignored.txt` - list of ignored tests in `ets-func-tests` test suite 110- `ets-cts-FastVerify-ignored-OL2.txt` - list of ignored tests for `ets-cts` test suite and for opt-level=2. `FastVerify` is additional info. 111 112In any test list the test is specified as path to the test file relative from the `test_root`: Examples: 113 114- array-large.js 115- built-ins/Date/UTC/fp-evaluation-order.js 116- tests/stdlib/std/math/sqrt-negative-01.ets 117 118Test file specified in the option `--test-file`/`test-lists.explicit-file` should be set in this format too. 119By default, ignored or excluded lists are located near tests themselves. 120 121All test lists are loaded automatically from the specified `LIST_ROOT` and based on following options: 122- architecture: 123 - from cli one of: `--test-list-arch=ARCH`, where ARCH is one of `amd64`, `arm32`, `arm64` 124 - from config file: `test-lists.architecture: ARCH`. Values are the same. 125- sanitizer: 126 - from cli on of: `--test-list-san=SAN`, where SAN in one of `asan` or `tsan` 127 - from config file: `test-lists.sanitizer: SAN`. Values are the same 128 129> **Note**: these options just specifies what test lists to load and do not affect on how and where to start the runner 130> itself and binaries used within. 131 132## Utility runner options: 133 134- `--skip-test-lists`/`test-lists.skip-test-lists: True` - do not use ignored or excluded lists, run all available tests, report all found failures 135- `--test-list TEST_LIST`/`test-lists.explicit-list: TEST_LIST` - run tests ONLY listed in TEST\_LIST file. 136- `--test-file TEST_FILE`/`test-lists.explicit-file: TEST_FILE` - run ONLY ONE specified test. **Attention** - not test suite, but the single test from the suite. 137- `--update-excluded`/`test-lists.update-excluded: True` - regenerates excluded test lists 138- `--update-expected`/`test-lists.update-expected: True` - regenerates expected test lists (applied only for JS Parser test suite) 139- `--report-format`/`general.report-format` - specifies in what format to generate failure reports. By default, `md`. Possible value: `html`. As well reports in the plain text format with `.log` extension are always generated. 140- `--filter FILTER`/`test-lists.filter: FILTER` - test filter regexp 141- `--show-progress`/`general.show-progress: True` - show progress bar during test execution 142- `--time-report`/`time-report.enable: True` - generates report with grouping tests by execution time. 143 144## Detailed report 145 146Detailed report shows test statistics for every folder 147- `--detailed-report`/`general.detailed-report: True` - if it's specified the report is generated 148- `general.detailed-report-file: FILE` - specifies file/path where the report should be saved to 149 150## Verbose and logging options: 151 152- `--verbose`, `-v` - Enable verbose output. Possible values one of: 153 - `all` - the most detailed output, 154 - `short` - test status and output. 155 - if none specified (by default): only test status for new failed tests 156 - in config file use `general.verbose` property with the save values. 157- `--verbose-filter` - Filter for what kind of tests to output stdout and stderr. 158 Supported values: 159 - `all` - for all executed tests both passed and failed. 160 - `ignored` - for new failures and tests from ignored test lists including both passed and failed. ' 161 - `new` - only for new failures. Default value. 162 - in config file use `general.verbose-filter` property with the same values. 163 164## Generation options: 165 166- `--generate-only`/`general.generate-only` - only generate tests without running them. Tests are run as usual without this option. 167- `--force-generate`/`ets.force-generate` - force ETS tests generation from templates 168 169## Other options: 170 171To get runner all options use: `main.py --help` or `runner.sh --help` 172 173## Execution time report 174 175It is possible to collect statistics how long separate tests work. In the result report tests are grouped by execution time. 176 177The grouping edges are set in seconds. For example, the value `1 5 10` specifies 4 groups - less than 1 second, from 1 second to 5 seconds, from 5 seconds to 10 seconds and from 10 seconds and more. For the last group the report contains real durations. 178 179- Specify the option `--time-report`/`time-report.enable: True` 180- Specify in yaml config file `time-report.time-edges: [1, 5, 10]` where value is a list of integers. 181- After test run the short report will be output to the console 182- And full report will be created at the `<work-dir/report>/time_report.txt` 183 184 185## Linter and type checks 186 187A script running linter and type checks starts as: 188 189`urunner>./linter.sh` 190 191It performs checks by running `pylint` and `mypy`. 192 193For `pylint` settings see `.pylintrc` file. For `mypy` settings see `mypy.ini` file. 194 195## Custom suite 196If you want to run arbitrary set of ETS tests with URunner you can use a custom suite like (minimal configuration): 197 198`urunner>./runner.sh --config suite.yaml --build-dir $BUILD` 199 200suite.yaml configuration file with these options can be looked as follows: 201```yaml 202 custom: 203 suite-name: SUITE_NAME 204 test-root: TEST_ROOT 205 list-root: LIST_ROOT 206 extension: EXTENSION 207``` 208 209where: 210- SUITE_NAME - a tag used to name the suite 211- TEST_ROOT - a path to the folder with tests (test files or templates) 212- LIST_ROOT - a path to the folder with ignored/excluded lists. Note: SUITE_NAME is used as a <test-suite-name> in the 213name of the test list (see #Test lists for details) 214- EXTENSION - extension what test files should be selected. For example: `ets` 215 216### Custom generator 217It is possible to specify a custom generator. It should be executable file accepting following parameters: 218 219`custom_generator --source SOURCE_PATH --target TARGET_PATH <other-options>` 220 221Here: 222- SOURCE_PATH - path where template files are located. URunner fills this parameter with TEST_ROOT specified above. 223- TARGET_PATH - path where generated tests should be put. URunner fills this parameter with value from --work-dir option ($WorkDir/gen) 224- other-options - other options that might be required by the generator (optional) 225 226To specify generator and its options use: 227- `custom.generator: PATH` - path to the executable file-generator 228- `custom.generator-options: [OPTIONS]` - multiple value option (can be specified several times) 229 230## Steps 231 232If set option `customScheme` in the yaml configuration file like: 233```yaml 234test-suites: 235 - customScheme 236``` 237It's possible to specify steps what should be executed: 238```yaml 239steps: 240 es2abc: # step name, user-defined value 241 order: 10 # order of execution the step with the less number is executed earlier 242 executable-path: ${general.build}/arkcompiler/ets_frontend/es2abc # the binary to execute 243 extra: # a dictionary of user-defined options 244 opt-level: 2 # both the option name and value are defined by user 245 output-file: "${general.work-dir}/intermediate/${test-id}.abc" # possible types of value: int, bool, str 246 env: # a dictionary of key-value pairs to add to environment variables before the binary to run 247 LD_LIBRARY_PATH: # sample variable name 248 - "${general.build}/thirdparty/icu:" # the variable value can be anything, if it's a list, the items joined without separator 249 - "${general.build}/arkcompiler/ets_runtime" 250 args: # a list of options to transfer to the binary 251 - "--opt-level=${steps.es2abc.extra.opt-level}" 252 - "--output=${steps.es2abc.extra.output-file}" 253 - "${general.work-dir}/gen/${test-id}" 254 timeout: 30 # is seconds 255``` 256The values can contain macros detected by `${}`. The macro's name is yaml path to the wished property. 257 258The special macro `${test-id}` refers to the test file name (or path to the test file if some folder hierarchy is used). 259It will be expanded during the binary executing. 260