1Skia Gold 2========= 3 4Overview 5-------- 6 7Gold is a web application that compares the images produced by our bots against 8known baseline images. 9Key features: 10 11* Baselines are managed in Gold outside of Git, but in lockstep with Git commits. 12* Each commit creates >500k images. 13* Deviations from the baseline are triaged after a CL lands and images are 14 triaged as either `positive` or `negative`. 'Positive' means the diff is 15 considered acceptable. 'Negative' means the diff is considered unacceptable 16 and requires a fix. 17 If a CL causes Skia to break it is reverted or an additional CL is landed to 18 fix the problem. 19* We test across a range of dimensions, e.g.: 20 21 - OS (Windows, Linux, Mac, Android, iOS) 22 - Architectures (Intel, ARM) 23 - Backends (CPU, OpenGL, Vulkan etc.) 24 - etc. 25 26* Written in Go, Polymer and deployed on the Google Cloud. The code 27 is in the [Skia Infra Repository](https://github.com/google/skia-buildbot). 28 29Recommended Workflows 30--------------------- 31### How to best use Gold for commonly faced problems ### 32 33These instructions will refer to various views which are accessible via the left 34navigation on [gold.skia.org](https://gold.skia.org/). 35View access is public, triage access is granted to 36Skia contributors. You must be logged in to triage. 37 38Problem #1: As sheriff, I need to triage and “assign” many incoming new images. 39------------------------------------------------------------------------------- 40Solution today: 41 42* Access the By Blame view to see digests needing triage and associated 43 owners/CLs 44 + Only untriaged digests will be shown by default 45 + Blame is not sorted in any particular order 46 + Digests are clustered by runs and the most minimal set of blame 47 48<img src=BlameView.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 49 50* Select digests for triage 51 + Digests will be listed in order with largest difference first 52 + Click to open the digest view with detailed information 53 54<img src=Digests.png style="margin-left:40px" align="left" width="780"/> <br clear="left"> 55 56* Open bugs for identified owner(s) 57 + The digest detail view has a link to open a bug from the UI 58 + Via the Gold UI or when manually entering a bug, copy the full URL of 59 single digest into a bug report 60 + The URL reference to the digest in Issue Tracker will link the bug to 61 the digest in Gold 62 63<img src="IssueHighlight.png" style="margin-left:60px" align="left" width="720" border=1/> <br clear="left"> 64 65<br> 66 67Future improvements: 68 69* Smarter, more granular blamelist 70 71<br> 72 73Problem #2: As a developer, I need to land a CL that may change many images. 74---------------------------------------------------------------------------- 75To find your results: 76 77* Immediately following commit, access the By Blame view to find untriaged 78 digest groupings associated with your ID 79* Click on one of the clusters including your CL to triage 80* Return to the By Blame view to walk through all untriaged digests involving 81 your change 82* Note: It is not yet implemented in the UI but possible to filter the view 83 by CL. Delete hashes in the URL to only include the hash for your CL. 84 85<img src=BlameView.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 86 87To rebaseline images: 88 89* Access the Ignores view and create a new, short-interval (hours) ignore for 90 the most affected configuration(s) 91 92<img src=Ignores.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 93 94 95* Click on the Ignore to bring up a search view filtered by the affected 96 configuration(s) 97* Mark untriaged images as positive (or negative if appropriate) 98* Follow one of two options for handling former positives: 99 + Leave former positives as-is and let them fall off with time if there is 100 low risk of recurrence 101 + Mark former positives as negative if needed to verify the change moving 102 forward 103 104Future improvements: 105 106* Trybot support prior to commit, with view limited to your CL 107* Pre-triage prior to commit that will persist when the CL lands 108 109<br> 110 111Problem #3: As a developer or infrastructure engineer, I need to add a new or updated config. 112--------------------------------------------------------------------------------------------- 113(ie: new bot, test mode, environment change) 114 115Solution today: 116 117* Follow the process for rebaselining images: 118 + Wait for the bot/test/config to be committed and show up in the Gold UI 119 + Access the Ignores view and create a short-interval ignore for the 120 configuration(s) 121 + Triage the ignores for that config to identify positive images 122 + Delete the ignore 123 124Future improvements: 125 126* Introduction of a new or updated test can make use of try jobs and pre-triage. 127* New configs may be able to use these features as well. 128 129<br> 130 131Problem #4: As a developer, I need to analyze the details of a particular image digest. 132--------------------------------------------------------------------------------------- 133Solution: 134 135* Access the By Test view 136 137<img src=ByTest.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 138 139* Click the magnifier to filter by configuration 140* Access the Cluster view to see the distribution of digest results 141 + Use control-click to select and do a direct compare between data points 142 + Click on configurations under “parameters” to highlight data points and 143 compare 144 145<img src=ClusterConfig.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 146 147* Access the Grid view to see NxN diffs 148 149<img src=Grid.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 150 151* Access the Dot diagram to see history of commits for the trace 152 + Each dot represents a commit 153 + Each line represents a configuration 154 + Dot colors distinguish between digests 155 156<img src=DotDiagram.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 157 158<br> 159 160Future improvements: 161 162* Large diff display of image vs image 163 164<br> 165 166Problem #5: As a developer, I need to find results for a particular configuration. 167---------------------------------------------------------------------------------- 168Solution: 169 170* Access the Search view 171* Select any parameters desired to search across tests 172 173<img src=Search.png style="margin-left:30px" align="left" width="800"/> <br clear="left"> 174
