1Skia Automated Testing 2====================== 3 4Overview 5-------- 6 7Skia uses [Swarming](https://github.com/luci/luci-py/blob/master/appengine/swarming/doc/Design.md) 8to do the heavy lifting for our automated testing. It farms out tasks, which may 9consist of compiling code, running tests, or any number of other things, to our 10bots, which are virtual or real machines living in our local lab, Chrome Infra's 11lab, or in GCE. 12 13The [Skia Task Scheduler](http://go/skia-task-scheduler) determines what tasks 14should run on what bots at what time. See the link for a detailed explanation of 15how relative task priorities are derived. A *task* corresponds to a single 16Swarming task. A *job* is composed of a directed acyclic graph of one or more 17*tasks*. The job is complete when all of its component tasks have succeeded 18or is considered a failure when any of its component tasks fails. The scheduler 19may automatically retry tasks within its set limits. Jobs are not retried. 20Multiple jobs may share the same task, for example, tests on two different 21Android devices which use the same compiled code. 22 23Each Skia repository has an `infra/bots/tasks.json` file which defines the jobs 24and tasks for the repo. Most jobs will run at every commit, but it is possible 25to specify nightly and weekly jobs as well. For convenience, most repos also 26have a `gen_tasks.go` which will generate `tasks.json`. You will need to 27[install Go](https://golang.org/doc/install). From the repository root: 28 29 $ go get -u go.skia.org/infra/... 30 $ go run infra/bots/gen_tasks.go 31 32It is necessary to run `gen_tasks.go` every time it is changed or every time an 33[asset](https://skia.googlesource.com/skia/+/master/infra/bots/assets/README.md) 34has changed. There is also a test mode which simply verifies that the `tasks.json` 35file is up to date: 36 37 $ go run infra/bots/gen_tasks.go --test 38 39 40 41Try Jobs 42-------- 43 44Skia's trybots allow testing and verification of changes before they land in the 45repo. You need to have permission to trigger try jobs; if you need permission, 46ask a committer. After uploading your CL to [Gerrit](https://skia-review.googlesource.com/), 47you may trigger a try job for any job listed in `tasks.json`, either via the 48Gerrit UI, using `git cl try`, eg. 49 50 git cl try -B skia.primary -b Some-Tryjob-Name 51 52or using `bin/try`, a small wrapper for `git cl try` which helps to choose try jobs. 53From a Skia checkout: 54 55 bin/try --list 56 57You can also search using regular expressions: 58 59 bin/try "Test.*GTX660.*Release" 60 61 62Status View 63------------ 64 65The status view shows a table with tasks, grouped by test type and platform, 66on the X-axis and commits on the Y-axis. The cells are colored according to 67the status of the task for each commit: 68 69* green: success 70* orange: failure 71* purple: exception (infrastructure issue) 72* black border, no fill: task in progress 73* blank: no task has started yet for a given revision 74 75Commits are listed by author, and the branch on which the commit was made is 76shown on the very left. A purple result will override an orange result. 77 78For more detail, you can click on an individual cell to get a summary of the 79task. You can also click one of the white bars at the top of each column to see 80a summary of recent tasks with the same name. 81 82The status page has several filters which can be used to show only a subset of 83task specs: 84 85* Interesting: Task specs which have both successes and failures within the 86 visible commit window. 87* Failures: Task specs which have failures within the visible commit window. 88* Comments: Task specs which have comments. 89* Failing w/o comment: task specs which have failures within the visible commit 90 window but have no comments. 91* All: Display all tasks. 92* Search: Enter a search string. Substrings and regular expressions may be 93 used, per the Javascript String Match() rules: 94 http://www.w3schools.com/jsref/jsref_match.asp 95 96<a name="adding-new-jobs"></a> 97Adding new jobs 98--------------- 99 100If you would like to add jobs to build or test new configurations, please file a 101[New Bot Request][new bot request]. 102 103If you know that the new jobs will need new hardware or you aren't sure which 104existing bots should run the new jobs, assign to jcgregorio. Once the Infra team 105has allocated the hardware, we will assign back to you to complete the process. 106 107Generally it's possible to copy an existing job and make changes to accomplish 108what you want. You will need to add the new job to 109[infra/bots/jobs.json][jobs json]. In some cases, you will need to make changes 110to recipes: 111 112* If there are new GN flags or compiler options: 113 [infra/bots/recipe_modules/flavor/gn_flavor.py][gn flavor py] 114* If there are modifications to dm flags: [infra/bots/recipes/test.py][test py] 115* If there are modifications to nanobench flags: 116 [infra/bots/recipes/perf.py][perf py] 117 118After modifying any of the above files, run `make train` in the infra/bots 119directory to update generated files. Upload the CL, then run `git cl try -B 120skia.primary -b <job name>` to run the new job. (After commit, the new job will 121appear in the PolyGerrit UI after the next successful run of the 122Housekeeper-Nightly-UpdateMetaConfig task.) 123 124If you need to do something more complicated, or if you are not sure how to add 125and configure the new jobs, please ask for help from borenet, benjaminwagner, or 126mtklein. 127 128[new bot request]: 129 https://bugs.chromium.org/p/skia/issues/entry?template=New+Bot+Request 130[jobs json]: https://skia.googlesource.com/skia/+/master/infra/bots/jobs.json 131[gn flavor py]: 132 https://skia.googlesource.com/skia/+/master/infra/bots/recipe_modules/flavor/gn_flavor.py 133[test py]: 134 https://skia.googlesource.com/skia/+/master/infra/bots/recipes/test.py 135[perf py]: 136 https://skia.googlesource.com/skia/+/master/infra/bots/recipes/perf.py 137