1# ANGLE Wrangling 2 3As an ANGLE Sheriff. Your job is to: 4 5 1. Keep the [ANGLE Try Waterfall](https://ci.chromium.org/p/chromium/g/tryserver.chromium.angle/builders) in good 6 working order. 7 1. Control and monitor the [ANGLE auto-rollers](#the-auto-rollers). 8 1. Keep the [ANGLE Standalone Testers](README.md) in good working order. 9 1. Keep the [SwANGLE Try Waterfall](https://luci-milo.appspot.com/p/chromium/g/tryserver.chromium.swangle/builders) in good 10 working order. 11 1. Monitor and respond to ANGLE's [Perf alerts](https://groups.google.com/u/0/a/chromium.org/g/angle-perf-alerts) 12 13If you're not an ANGLE team member, you can contact us on the public ANGLE project 14[Google group](https://groups.google.com/forum/#!forum/angleproject). 15 16**Note**: Please review and if needed update the [wrangler schedule]. 17 18**Note**: It's highly recommend that all wranglers install the [Chromium Flake Linker][Flaker] 19extension for inspecting bot builds. It'll save you a lot of time. 20 21**Note**: If you need to suppress test failures (e.g. to keep an auto-roller unblocked), see 22[Handling Test Failures](../doc/TestingAndProcesses.md). 23 24[Flaker]: https://chrome.google.com/webstore/detail/flake-linker/boamnmbgmfnobomddmenbaicodgglkhc 25 26[wrangler schedule]: https://rotations.corp.google.com/rotation/5080504293392384 27 28## Task 1: Monitor ANGLE CI and Try Testers 29 30Your first job is to keep the ANGLE builders green and unblocked. 31 32There are two consoles for ANGLE continuous integration builders: 33 34 1. Standalone ANGLE: https://ci.chromium.org/p/angle/g/ci/console 35 1. Chromium + ANGLE integration: https://ci.chromium.org/p/chromium/g/chromium.angle/console 36 37We expect these waterfalls to be perfectly green at all times. As of writing 38several builders are red or flaky. Oart of your job as wrangler is to 39eliminate all sources of breaks and flakiness. We can fix flakiness by 40suppressing tests that are not worth fixing, reverting problematic CLs, or 41finding other solutions. 42 43When you encouter red builds or flakiness, please [file an ANGLE bug] 44(http://anglebug.com/new) and set the label: `Hotlist-Wrangler`. 45 46[Click here to see a ANGLE wrangler bug hot list.][WranglerBugs] 47 48In addition to the CI builders, we have a console for try jobs on the ANGLE CV (change verifier): 49 50 * https://ci.chromium.org/ui/p/angle/g/try/builders 51 52Some failures are expected on this waterfall as developers test WIP changes. 53Please watch for persistent sources of flakiness and failure and take action 54as appropriate by filing bugs, reverting CLs, or taking other action. 55 56[WranglerBugs]:https://bugs.chromium.org/p/angleproject/issues/list?q=Hotlist%3DWrangler&can=2 57 58If you find a point of failure that is unrelated to ANGLE, please [file a 59Chromium bug](http://crbug.com/new). Set the bug label 60`Hotlist-PixelWrangler`. Ensure you cc the current ANGLE and Chrome GPU 61wranglers, which you can find by consulting 62[build.chromium.org](https://ci.chromium.org/p/chromium/g/main/console). 63For more information see [Filing Chromium Bug Reports](#filing-chromium-bug-reports) below. 64 65Also follow [Chromium bugs in the `Internals>GPU>ANGLE` component][ChromiumANGLEBugs] 66to be alerted to reports of ANGLE-related breakage in Chrome. 67 68[ChromiumANGLEBugs]:https://bugs.chromium.org/p/chromium/issues/list?q=component%3AInternals%3EGPU%3EANGLE&can=2 69 70**NOTE: When all builds seem to be purple or otherwise broken:** 71 72This could be a major infrastructure outage. File a high-priority bug using 73[g.co/bugatrooper](http://g.co/bugatrooper). 74 75### Filing Chromium Bug Reports 76 77The GPU Pixel Wrangler is responsible for *Chromium* bugs. Please file 78Chromium issues with the Label `Hotlist-PixelWrangler` for bugs outside of 79the ANGLE project. 80 81*IMPORTANT* info to include in Chromium bug reports: 82 83 * Links to all first failing builds (e.g. first windows failure, first mac failure, etc). 84 * Related regression ranges. See below on how to determine the ANGLE regression range. 85 * Relevant error messages. 86 * Set the **Components** to one or more value, such as (start typing "Internals" and you'll see choices): 87 * `Internals>GPU` for general GPU bugs 88 * `Internals>GPU>Testing` for failures that look infrastructure-related 89 * `Internals>GPU>ANGLE` for ANGLE-related Chromium bugs 90 * `Internals>Skia` for Skia-specific bugs 91 * Cc relevant sheriffs or blame suspects, as well as yourself or the current ANGLE Wrangler. 92 * Set the `Hotlist-PixelWrangler` Label. 93 94### How to determine the ANGLE regression range on Chromium bots: 95 96 1. Open the first failing and last passing builds. 97 1. For test failures: record `parent_got_angle_revision` in both builds. 98 1. For compile failures record `got_angle_revision`. 99 1. Create a regression link with this URL template: 100 `https://chromium.googlesource.com/angle/angle.git/+log/<last passing revision>..<first failing revision>` 101 102## <a name="the-auto-rollers"></a>Task 3: The Auto-Rollers 103 104The [ANGLE auto-roller](https://autoroll.skia.org/r/angle-chromium-autoroll) automatically updates 105Chrome with the latest ANGLE changes. 106 107 1. **Roller health**: You will be cc'ed on all rolls. Please check failed rolls to verify there is no blocking 108 breakage. 109 1. **Chrome Branching**: You are responsible for pausing the roller 24h before branch days, and resuming afterwards. 110 See the [Chrome Release Schedule](https://chromiumdash.appspot.com/schedule). 111 112We also use additional auto-rollers to roll third party libraries, and Chromium, into ANGLE once per day: 113 114 * [SwiftShader into ANGLE](https://autoroll.skia.org/r/swiftshader-angle-autoroll) 115 * [vulkan-deps into ANGLE](https://autoroll.skia.org/r/vulkan-deps-angle-autoroll) 116 * [VK-GL-CTS into ANGLE](https://autoroll.skia.org/r/vk-gl-cts-angle-autoroll?tab=status) 117 * [Chromium into ANGLE](https://autoroll.skia.org/r/chromium-angle-autoroll) 118 119Please ensure these rollers are also healthy and unblocked. You can trigger manual rolls using the 120dashboards to land high-priority changes, for example Chromium-side test expectation updates or 121suppressions. When a roll fails, stop the roller, determine if the root cause is a problem with 122ANGLE or with the upstream repo, and file an issue with an appropriate next step. 123 124The autoroller configurations live in the [skia/buildbot repository](https://skia.googlesource.com/buildbot/) 125in the [autoroll/config](https://skia.googlesource.com/buildbot/+/main/autoroll/config) folder. 126 127**NOTE: vulkan-deps consists of several related Vulkan dependencies:** 128 129vulkan-deps houses Vulkan-Tools, Vulkan-Loader, Vulkan-ValidationLayers, Vulkan-Headers and other 130related repos. If the roll fails, you will have to determine the correct upstream repo and file 131an issue upstream. For more info on vulkan-deps see the 132[README](https://chromium.googlesource.com/vulkan-deps/+/refs/heads/main/README.md). 133 134Occasionally, a vulkan-deps AutoRoll CL will get an error in the `presubmit` bot. For example, 135see: https://chromium-review.googlesource.com/c/angle/angle/+/3198390 where the 136`export_targets.py` script had trouble with the `loader_windows.h` file. The `export_targets.py` 137script sometimes has difficulty with headers. If you cannot see an obvious problem, create a CL 138that adds the header to `IGNORED_INCLUDES` in `export_targets.py`. 139 140## Task 4: ANGLE Standalone Testing 141 142See more detailed instructions on by following [this link](README.md). 143 144## Task 5: Monitor SwANGLE CI and Try Testers 145 146The most important task here is to keep the 2 SwANGLE bots on the ANGLE CV healthy: 147[linux-swangle-try-tot-angle-x64](https://luci-milo.appspot.com/p/chromium/builders/try/linux-swangle-try-tot-angle-x64) 148and 149[win-swangle-try-tot-angle-x86](https://luci-milo.appspot.com/p/chromium/builders/try/win-swangle-try-tot-angle-x86). 150As well as the 2 SwANGLE bots used for ANGLE rolls on Chromium CV, 151[linux-swangle-try-x64](https://luci-milo.appspot.com/p/chromium/builders/try/linux-swangle-try-x64) 152and 153[win-swangle-try-x86](https://luci-milo.appspot.com/p/chromium/builders/try/win-swangle-try-x86). 154 155Same instructions as for [Task 1](#task-1_monitor-angle-ci-and-try-testers) apply here. 156Some failures on these bots may be due to SwiftShader changes, however. 157The possible ways to handle these failures are: 1581. If possible, suppress the failing tests in ANGLE, opening a bug to investigate these later. 1591. If it is clear that an ANGLE CL caused a regression, 160 consider whether reverting it or suppressing the failures is a better course of action. 1611. If a SwiftShader CL is suspected, and the breakage is too severe to be suppressed, 162 (a lot of tests fail in multiple suites), 163 it is possible to revert the responsible SwiftShader roll into Chromium 164 and open a SwiftShader [bug](http://go/swiftshaderbugs). SwiftShader rolls into Chromium 165 should fail afterwards, but if the bad roll manages to reland, 166 the [autoroller](https://autoroll.skia.org/r/swiftshader-chromium-autoroll) needs to be stopped. 167 168A lower priority task here is to keep healthy all the SwANGLE 169[CI](https://luci-milo.appspot.com/p/chromium/g/chromium.swangle/builders) and 170[Try](https://luci-milo.appspot.com/p/chromium/g/tryserver.chromium.swangle/builders) bots. 171 172## Task 6: Monitor and respond to ANGLE's perf alerts 173 174Any large regressions should be triaged with a new ANGLE bug linked to any suspected CLs that may 175have caused performance to regress. If it's a known/expected regression, the bug can be closed as 176such. The tests are very flaky right now, so a WontFix resolution is often appropriate. 177