1# Centralized Suites 2 3Centralized Suites are a way to centrally and modularly define groupings of 4tests that together capture a certification goal when run against a device (e.g. 5FSI, CQ, Release). Centralized Suites consist of: 6 7* **Suites** - which contain tests 8* **SuiteSets** - which contain other Suites and SuiteSets 9 10This README details the process of authoring Suites and SuiteSets. For more 11details on the motivation and design behind Centralized Suites see the design 12doc at [go/cros-centralized-suites]. 13 14## Authoring Centralized Suites 15 16Centralized Suites are defined in Starlark across two repositories: [config] and 17[config-internal]. Only Suites with private Tests and SuiteSets with private 18Suites/SuiteSets should live in the private config-internal repo, the rest 19should live in the public config repo. See [Centralized Suite Build Process] for 20more details on how Centralized Suites interact across these two repos. 21 22To define a new Suite/SuiteSet in either of these repos, do the following: 23 241. Create a owners specific subdirectory 25 26 * Create a subdirectory in the [suite_sets] directory and add an `OWNERS` 27 file to it with individuals/teams that should sign off on changes to the 28 Suite/SuiteSet 29 30 * If a subdirectory with the same owners already exists, use that 31 subdirectory instead of creating a new one 32 331. Define your Suite/SuiteSet 34 35 * In your specific subdirectory, define your Suite/SuiteSet as demonstrated 36 in [example_suites.star] and [example_suite_sets.star] 37 381. Add Suite/SuiteSet to compilation list 39 40 * If you defined a Suite, import and add it `compiled_suites` list in the 41 [compiled_suites.star] file 42 43 * If you defined a SuiteSet, import and add it to the `compiled_suite_sets` 44 list in the [compiled_suite_sets.star] file 45 461. Validate your Suite/SuiteSet 47 48 * Run [generate.sh] script 49 50 ``` 51 ./generate.sh 52 ``` 53 54 * Fix any linting errors it returns 55 56 * Validate your Suite/SuiteSet is included in the generated proto file 57 581. Upload a CL with your changes 59 * See this [reference CL] for an example of what a change might look like 60 61## Migrating from Legacy Suites 62 63If you want to migrate a pre-existing legacy suite (where the suites are 64attributes in the tests themselves) to a Centralized Suite, you can use the 65[suite_migration.py script] as follows: 66 671. Complete step 1 of [Authoring Centralized Suites] 68 691. Build the test metadata so it is up-to-date 70 ``` 71 cros_sdk '$CROS_WORKON_SRCROOT'/src/third_party/autotest/files/contrib/suite_migration.py --update-metadata 72 ``` 73 741. Migrate the suite of interest 75 ``` 76 cros_sdk '$CROS_WORKON_SRCROOT'/src/third_party/autotest/files/contrib/suite_migration.py --suite <suite_name> --output <output_starlark_path> 77 ``` 78 791. Fill out missing metadata 80 81 * The starlark file generated by the script will have TODO messages for all 82 the missing metadata; fill in the relevant information 83 841. Complete steps 3-5 of [Authoring Centralized Suites] 85 86 87## Centralized Suite Build Process 88 89As of January 30, 2023, Centralized Suites are built per board by combining the 90information across the [config] and [config-internal] repos. Specifically, the 91logic is as follows: 92 931. Load the Centralized Suites from the [config] repo 94 951. If the [config-internal] repo is present (will not be available to public 96builders) do: 97 98 1. Load the Centralized Suites from the [config-internal] repo 99 100 1. Merge Centralized Suites with the same name: 101 102 * Suites will be merged by unioning of the test lists 103 104 * SuiteSets will be merged by unioning the Suite and SuiteSet lists 105 1061. Filter tests from Suites that are not relevant to the Board 107 108<!-- Insert links below --> 109[go/cros-centralized-suites]: https://docs.google.com/document/d/1Qr7oW2kCeCsxsyu2vWBSXjwCHlhMEaSWSme7zoALnWc/edit?resourcekey=0-U91BRPcjbxKGdR_TpJsHBA&tab=t.0#heading=h.7zgnj8bwqfld 110[config]: https://chromium.googlesource.com/chromiumos/config/+/refs/heads/main/test/suite_sets/ 111[config-internal]: https://chrome-internal.googlesource.com/chromeos/config-internal/+/refs/heads/main/test/suite_sets/ 112[example_suites.star]: https://chromium.googlesource.com/chromiumos/config/+/refs/heads/main/test/suite_sets/suite_sets/example/example_suites.star 113[example_suite_sets.star]: https://chromium.googlesource.com/chromiumos/config/+/refs/heads/main/test/suite_sets/suite_sets/example/example_suite_sets.star 114[reference CL]: https://chromium-review.googlesource.com/c/chromiumos/config/+/5171723 115[Centralized Suite Build Process]: #centralized-suite-build-process 116[suite_migration.py script]: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/main/contrib/suite_migration.py 117[Authoring Centralized Suites]: #authoring-centralized-suites 118[test/suite_sets/suite_sets]: suite_sets 119[compiled_suites.star]: compiled_suites.star 120[compiled_suite_sets.star]: compiled_suite_sets.star 121[generate.sh]: generate.sh 122