1# Src-side test specifications 2 3Src-side test specifications enable developers to quickly add tests running on 4specific bots on V8's continuous infrastructure (CI) or tryserver. Features to 5be tested must live behind runtime flags, which are mapped to named testing 6variants specified [here](https://chromium.googlesource.com/v8/v8/+/master/tools/testrunner/local/variants.py). 7Changes to src-side test specifications go through CQ like any other CL and 8require tests added for specific trybots to pass. 9 10The test specifications are defined in a V8-side python-literal file 11`infra/testing/builders.pyl`. 12 13The structure of the file is: 14``` 15{ 16 <buildername>: { 17 'tests': [ 18 { 19 'name': <test-spec name>, 20 'suffix': <step suffix>, 21 'variant': <variant name>, 22 'shards': <number of shards>, 23 'test_args': <list of flags>, 24 'swarming_task_attrs': {...}, 25 'swarming_dimensions': {...}, 26 }, 27 ... 28 ], 29 'swarming_task_attrs': {...}, 30 'swarming_dimensions': {...}, 31 }, 32 ... 33} 34``` 35The `<buildername>` is a string name of the builder to execute the tests. 36`<test-spec name>` is a label defining a test specification matching the 37[infra-side](https://chromium.googlesource.com/chromium/tools/build/+/refs/heads/master/recipes/recipe_modules/v8/testing.py). 38The optional `suffix` will be appended to test-step names for disambiguation. 39The optional `variant` is a testing variant specified 40[here](https://chromium.googlesource.com/v8/v8/+/master/tools/testrunner/local/variants.py). 41The optional `shards` (default 1) can be provided to increase the swarming 42shards for long-running tests. 43The optional `test_args` is a list of string flags that will be passed to the 44V8 test driver. 45The optional `swarming_task_attrs` is a dict allowing to override the defaults 46for `priority`, `expiration` and `hard_timeout`. 47The optional `swarming_dimensions` is a dict allowing to override the defaults 48for `cpu`, `cores` and `os`. 49Both `swarming_task_attrs` and `swarming_dimensions` can be defined per builder 50and per test, whereas the latter takes precedence. 51 52Example: 53``` 54{ 55 'v8_linux64_rel_ng_triggered': { 56 'tests': [ 57 { 58 'name': 'v8testing', 59 'suffix': 'stress', 60 'variant': 'nooptimization', 61 'shards': 2, 62 'test_args': ['--gc-stress'], 63 'swarming_dimensions': {'os': 'Ubuntu-14.4'}, 64 }, 65 ], 66 'swarming_properties': {'priority': 35}, 67 'swarming_dimensions': {'os': 'Ubuntu'}, 68 }, 69} 70``` 71 72## Guidelines 73 74Please keep trybots and continuous bots in sync. E.g. add the same configuration 75for the release and debug CI bots and the corresponding trybot (where 76applicable). E.g. 77 78``` 79tryserver.v8: 80 v8_linux64_rel_ng_triggered 81client.v8: 82 V8 Linux64 83 V8 Linux64 - debug 84``` 85 86Please only add tests that are expected to pass, or skip failing tests via 87status file for the selected testing variants only. If you want to add FYI tests 88(i.e. not closing the tree and not blocking CQ) you can do so for the following 89set of bots: 90 91``` 92tryserver.v8: 93 v8_linux64_fyi_rel_ng_triggered 94client.v8: 95 V8 Linux64 - fyi 96 V8 Linux64 - debug - fyi 97``` 98