Name |
Date |
Size |
#Lines |
LOC |
||
---|---|---|---|---|---|---|
.. | - | - | ||||
android/ | 03-May-2024 | - | 518 | 374 | ||
HistogramWriter.cpp | D | 03-May-2024 | 8.9 KiB | 271 | 199 | |
HistogramWriter.h | D | 03-May-2024 | 1.7 KiB | 60 | 39 | |
README.md | D | 03-May-2024 | 4 KiB | 84 | 62 | |
TestSuite.cpp | D | 03-May-2024 | 60.1 KiB | 1,978 | 1,595 | |
TestSuite.h | D | 03-May-2024 | 5.8 KiB | 212 | 169 | |
TestSuite_unittest.cpp | D | 03-May-2024 | 6.9 KiB | 244 | 176 |
README.md
1# ANGLE Test Harness 2 3The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the 4[Chromium test harness][BaseTest]. It features: 5 6 * splitting a test set into shards 7 * catching and reporting crashes and timeouts 8 * outputting to the Chromium [JSON test results format][JSONFormat] 9 * multi-process execution 10 11## Command-Line Arguments 12 13The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the 14following additional command-line arguments: 15 16 * `--batch-size` limits the number of tests to run in each batch 17 * `--batch-timeout` limits the amount of time spent in each batch 18 * `--bot-mode` enables multi-process execution and test batching 19 * `--debug-test-groups` dumps the test config categories when using `bot-mode` 20 * `--filter-file` allows passing a larger `gtest_filter` via a file 21 * `--histogram-json-file` outputs a [formatted JSON file][HistogramSet] for perf dashboards 22 * `--max-processes` limits the number of simuntaneous processes 23 * `--results-directory` specifies a directory to write test results to 24 * `--results-file` specifies a location for the JSON test result output 25 * `--shard-count` and `--shard-index` control the test sharding 26 * `--test-timeout` limits the amount of time spent in each test 27 * `--flaky-retries` allows for tests to fail a fixed number of times and still pass 28 * `--disable-crash-handler` forces off OS-level crash handling 29 * `--isolated-outdir` specifies a test artifacts directory 30 * `--max-failures` specifies a count of failures after which the harness early exits. 31 32`--isolated-script-test-output` and `--isolated-script-perf-test-output` mirror `--results-file` 33and `--histogram-json-file` respectively. 34 35As well as the custom command-line arguments we support a few standard GoogleTest arguments: 36 37 * `gtest_filter` works as it normally does with GoogleTest 38 * `gtest_also_run_disabled_tests` works as it normally does as well 39 40Other GoogleTest arguments are not supported although they may work. 41 42## Implementation Notes 43 44 * The test harness only requires `angle_common` and `angle_util`. 45 * It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms. 46 * It uses rapidjson to read and write JSON files. 47 * Test execution is not currently deterministic in multi-process mode. 48 49## Normal Mode vs Bot Mode 50 51The test runner has two main modes of operation: normal and *bot mode*. 52 53During normal mode: 54 55 * Tests are executed single-process and single-thread. 56 * The test runner executes the GoogleTest Run function. 57 * We use a `TestEventListener` to record test results for our output JSON file. 58 * A *watchdog thread* will force a fast exit if no test results get recorded after a timeout. 59 * Crashes are handled via ANGLE's test crash handling code. 60 61During bot mode: 62 63 * Tests are run in multiple processes depending on the system processor count. 64 * A server process records the child processes' stdout and stderr. 65 * The server terminates a child process if there's no progress after a timeout. 66 * The server sorts work into batches according to the back-end configuration. 67 * This prevents driver errors from using multiple back-ends in the same process. 68 * Batches are striped to help split up slow groups of tests. 69 * The server passes test batches to child processes via a `gtest_filter` file. 70 * Bot mode does not work on Android or Fuchsia. 71 72See the source code for more details: [TestSuite.h](TestSuite.h) and [TestSuite.cpp](TestSuite.cpp). 73 74## Potential Areas of Improvement 75 76 * Deterministic test execution. 77 * Using sockets to communicate with test children. Similar to dEQP's test harness. 78 * Closer integration with ANGLE's test expectations and system config libraries. 79 * Supporting a GoogleTest-free integration. 80 81[BaseTest]: https://chromium.googlesource.com/chromium/src/+/refs/heads/master/base/test/ 82[JSONFormat]: https://chromium.googlesource.com/chromium/src/+/master/docs/testing/json_test_results_format.md 83[HistogramSet]: https://chromium.googlesource.com/catapult/+/HEAD/docs/histogram-set-json-format.md 84