1 2--- 3title: "Blink layout tests" 4linkTitle: "Blink layout tests" 5 6--- 7 8How to land Skia changes that change Blink layout test results. 9See https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/web_tests.md for more 10detail on running the Blink layout tests. 11 12General tips about layout tests 13------------------------------- 14* Layout tests come in two flavors: "compare 2 html pages" and "compare html page and .png file" 15 When rebaselining, most of the effort comes from regenerating the .png files for the second 16 kind. The first kind will be something like `third_party/blink/web_tests/.../something.html` 17 and have a companion file `.../something-expected.html` as a sibling file. 18 ([example](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/fast/forms/text/input-appearance-autocomplete-very-long-value.html;drc=f68d6358bed8ebfc88a0198d6cda50256620c71d); 19 [companion html](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/fast/forms/text/input-appearance-autocomplete-very-long-value-expected.html;drc=f68d6358bed8ebfc88a0198d6cda50256620c71d)) 20 The second type won't have the companion html file, but might have a companion .png file, or 21 multiple .png files in other directories when the html should render differently 22 on other platforms or settings. 23 ([example](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/dark-mode/images/opt-out-svg-gradient.html;drc=44ad10338113aab1779d81df359aca34da89daf3); 24 [expected png](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/virtual/dark-mode-default/dark-mode/images/opt-out-svg-gradient-expected.png;l=1;drc=ec59d7b96e81ccc0e3dc497697e23304d7259b09)) 25 For this second type, using <https://cs.chromium.org> is a good way to find these if you need to 26 look at rebaselining history or something 27 28* Layout tests (of both kinds) can be given fuzzy matching by adding a meta HTML tag to the test 29 file. 30 `<meta name="fuzzy" content="maxDifference=0-4; totalPixels=0-100" />` 31 32* Some non-layout tests (also called pixel tests) will fail as a result of rendering changes because 33 they have their own checked-in images. Look at the logs of failing tests, as these will hopefully 34 output base64 encoded pngs of the expected and actual image. Open up a browser tab, use Dev Tools 35 to create an `<img src="[base64]" />` with the actual base64 data and right-click to save the 36 image as the new expected data. 37 ([example](https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/ui/views/accessibility/accessibility_focus_highlight_browsertest.cc;l=238;drc=a48632411d7e7263e8fd4d273d24a80f668b73ec); 38 [expected png](https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/data/accessibility/focus_highlight_appearance.png;l=1;drc=1e2dbf6a77e2f7264da0097a3cd4158c249a75b8)) 39 40* Some tests compare [Skia and PyCairo](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/html/canvas/tools/README.md). 41 Since Skia makes different choices to Cairo, it's best to increase fuzzy tolerance for these. Look 42 for a `fuzzy` entry in the .yaml file that generates the tests and then regenerate things (or just 43 use find and replace ��). 44 45* Failing CQ tests usually have a "Show Reproduction Instructions" for running locally. This can help verify 46 fuzzy tolerances. Be sure to use [--gtest_filter](https://github.com/google/googletest/blob/main/docs/advanced.md#running-a-subset-of-the-tests) 47 to limit what you are testing for faster iteration time. 48 49Changes that affect a large number of test results 50-------------------------------------------------- 51Where a 'large number' or 'many' means more than about 20, it's a bit of a process to get 52things rebaselined in Chromium. 53 541. Add a staging define to the Skia code that allows a client to (at compile time) opt-in to 55 the old code path. If only Chromium needs rebaselining, it's probably easier to set it up 56 like `if !defined(SK_USE_LEGACY_xxx)`. If this needs to be staged across multiple clients, 57 `if defined(SK_USE_NEW_xxx)` is better to let clients "opt-in" one at a time. 58 ([example CL](https://crrev.com/c/6316987)) 592. Tell Chromium to use the old code path using a staging define in their `SkUserConfig.h` 60 (or their `//skia/BUILD.gn` if it impacts only specific builds). 61 ([example CL](https://crrev.com/c/6316987)) 623. Land and wait for the autoroller to roll Skia into Chromium. 63 ([example CL](http://review.skia.org/953516) [autoroll CL](https://crrev.com/c/6324680)) 644. Create Chromium CL to use the new code path (by removing the define) and update expectations. 65 Follow [the rebaselining steps](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/web_test_expectations.md#How-to-rebaseline) to update layout tests that use a reference image. 66 For other types of tests (including the .html and -expected.html type), observe the above tips 67 to manually update them. To update the images, you may have to repeat the flow of "sync", 68 "run tryjobs" and "rebaseline images from them" a few times due to other simultaneous changes or 69 flaky tests. Feel free to add some `<meta name="fuzzy"` tags to any of the flaky tests. 70 71 ([example CL](https://crrev.com/c/6328778)) 725. Remove staging define from Skia. 73 ([example CL](http://review.skia.org/960516)) 74 75Changes that affect a small number of layout test results 76--------------------------------------------------------- 77Changes affecting fewer than ~20 layout tests can be rebaselined without a staging define using 78these steps: 79 801. Prepare your Skia change. Run the `Chromium-Canary` tryjob and take note of which layout tests 81 will turn red. 822. Ahead of the Skia auto roll including your change, manually push a change to the 83 Blink LayoutTests/TestExpectations [file](https://chromium.googlesource.com/chromium/src/+/main/third_party/blink/web_tests/TestExpectations), flagging tests expected to fail as a result of your change as 84 follows: 85 ``` 86 foo/bar/test-name.html [ Failure Pass ] # Needs rebaseline 87 ``` 88 There's a section `Skia roll test suppressions` to use (to avoid conflicts with other changes). 893. Check in your code to the Skia repo. 904. Wait for the Skia roll to land successfully. 915. In your Chromium checkout, create a new branch 92 (e.g. `git co main && gclient sync -D && git cl new-branch update-expectations`). 93 Follow [the rebaselining steps](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/web_test_expectations.md#How-to-rebaseline) 94 and remove the suppressions from `TestExpectations`. 95
)){kind=link}
)){kind=link}