1Skia Automated Testing 2====================== 3 4Overview 5-------- 6 7Skia uses [Swarming](https://github.com/luci/luci-py/blob/master/appengine/swarming/doc/Design.md) 8to do the heavy lifting for our automated testing. It farms out tasks, which may 9consist of compiling code, running tests, or any number of other things, to our 10bots, which are virtual or real machines living in our local lab, Chrome Infra's 11lab, or in GCE. 12 13The [Skia Task Scheduler](http://go/skia-task-scheduler) determines what tasks 14should run on what bots at what time. See the link for a detailed explanation of 15how relative task priorities are derived. A *task* corresponds to a single 16Swarming task. A *job* is composed of a directed acyclic graph of one or more 17*tasks*. The job is complete when all of its component tasks have succeeded 18or is considered a failure when any of its component tasks fails. The scheduler 19may automatically retry tasks within its set limits. Jobs are not retried. 20Multiple jobs may share the same task, for example, tests on two different 21Android devices which use the same compiled code. 22 23Each Skia repository has an infra/bots/tasks.json file which defines the jobs 24and tasks for the repo. Most jobs will run at every commit, but it is possible 25to specify nightly and weekly jobs as well. For convenience, most repos also 26have a gen_tasks.go which will generate tasks.json. You will need to 27[install Go](https://golang.org/doc/install). From the repository root: 28 29 $ go get -u go.skia.org/infra/... 30 $ go run infra/bots/gen_tasks.go 31 32It is necessary to run gen_tasks.go every time it is changed or every time an 33[asset](https://skia.googlesource.com/skia/+/master/infra/bots/assets/README.md) 34has changed. There is also a test mode which simply verifies that the tasks.json 35file is up to date: 36 37 $ go run infra/bots/gen_tasks.go --test 38 39 40 41Try Jobs 42-------- 43 44It is useful to know how your change will perform before it is submitted. After 45uploading your CL to [Gerrit](https://skia-review.googlesource.com/), you may 46trigger a try job for any job listed in tasks.json: 47 48 $ git cl try -B <bucket name> -b <job name> 49 50The bucket name refers to the [Buildbucket](https://chromium.googlesource.com/infra/infra/+/master/appengine/cr-buildbucket/README.md) 51bucket to which the request will be submitted. Most public Skia repos use the 52"skia.primary" bucket, and most private Skia repos use the "skia.internal" 53bucket. 54 55 56Status View 57------------ 58 59The status view shows a table with tasks, grouped by test type and platform, 60on the X-axis and commits on the Y-axis. The cells are colored according to 61the status of the task for each commit: 62 63* green: success 64* orange: failure 65* purple: exception (infrastructure issue) 66* black border, no fill: task in progress 67* blank: no task has started yet for a given revision 68 69Commits are listed by author, and the branch on which the commit was made is 70shown on the very left. A purple result will override an orange result. 71 72For more detail, you can click on an individual cell to get a summary of the 73task. You can also click one of the white bars at the top of each column to see 74a summary of recent tasks with the same name. 75 76The status page has several filters which can be used to show only a subset of 77task specs: 78 79* Interesting: Task specs which have both successes and failures within the 80 visible commit window. 81* Failures: Task specs which have failures within the visible commit window. 82* Comments: Task specs which have comments. 83* Failing w/o comment: task specs which have failures within the visible commit 84 window but have no comments. 85* All: Display all tasks. 86* Search: Enter a search string. Substrings and regular expressions may be 87 used, per the Javascript String Match() rules: 88 http://www.w3schools.com/jsref/jsref_match.asp 89 90 91