Lines Matching +full:coverage +full:- +full:results
1 # Regres - SwiftShader automated testing
5 Regres is a collection of tools to perform [dEQP](https://github.com/KhronosGroup/VK-GL-CTS)
6 presubmit and continuous integration testing and code coverage evaluation for
11 * [Presubmit testing](#presubmit-testing) - An automatic Vulkan
13 * [Continuous integration testing](#daily-run-continuous-integration-testing) -
15 This nightly run also produces code coverage information which can be viewed at
16 …[swiftshader-regres.github.io/swiftshader-coverage](https://swiftshader-regres.github.io/swiftshad…
17 * [Local dEQP test runner](#local-dEQP-test-runner) Provides a local tool for
25 Regres monitors changes that have been [put up for review with Gerrit](https://swiftshader-review.g…
29 Any differences in results are reported as a review comment on the change
30 [[example]](https://swiftshader-review.googlesource.com/c/SwiftShader/+/46369/5#message-4f09ea3e6d0…
47 land', which is determined by the change's `Kokoro-Presubmit`, `Code-Review` and
48 `Presubmit-Ready` Gerrit labels.
53 …`<swiftshader>/tests/regres/ci-tests.json`](https://cs.opensource.google/swiftshader/SwiftShader/+…
54 As new functionally is being implemented, the test lists in `ci-tests.json` may
55 reference known-passing test lists updated by the [daily run](#daily-run-continuous-integration-tes…
60 …wiftshader>/tests/regres/full-tests.json`](https://cs.opensource.google/swiftshader/SwiftShader/+/…
65 Test: <dEQP-test-pattern>
68 `<dEQP-test-pattern>` can be a single dEQP test name, or you can use wildcards
73 [For example](https://swiftshader-review.googlesource.com/c/SwiftShader/+/26574):
78 Test: dEQP-VK.glsl.operator.bool_compare.*
79 Test: dEQP-VK.glsl.operator.binary_operator.equal.*
80 Test: dEQP-VK.glsl.operator.binary_operator.not_equal.*
82 Change-Id: I9d33444d67792274d8027b7d1632235533cfc079
85 ## Daily-run continuous integration testing
87 …wiftshader>/tests/regres/full-tests.json`](https://cs.opensource.google/swiftshader/SwiftShader/+/…
89 [[example]](https://swiftshader-review.googlesource.com/c/SwiftShader/+/46448).
91 The daily run also performs code coverage instrumentation per dEQP test,
92 automatically uploading the results of all the dEQP tests to the viewer at
93 [swiftshader-regres.github.io/swiftshader-coverage](https://swiftshader-regres.github.io/swiftshade…
97 Regres also provides a multi-threaded, [process sandboxed](#process-sandboxing),
98 local dEQP test runner with a wild-card / regex based test name matcher.
102 …ftshader/SwiftShader/+/master:tests/regres/run_testlist.sh) `--deqp-vk=<path to deqp-vk> [--filter…
111 -limit int
113 -no-results
114 disable generation of results.json file
115 -output string
116 path to an output JSON results file (default "results.json")
117 -shuffle
119 -test-list string
120 path to a test list file (default "vk-master-PASS.txt")
123 …oogle/swiftshader/SwiftShader/+/master:tests/regres/run_testlist.sh) with `--help` to see all avai…
136 …the system's memory using [linux resource limits](https://man7.org/linux/man-pages/man2/getrlimit.…
159 4. The built dEQP binaries are used to test the change. The full test results
161 5. If the parent change's test results aren't already cached, then steps 3 and
163 6. The results of the two changes are diffed, and the results of the diff are
170 coverage instrumenting.
171 4. The build dEQP binaries are used to test the change. The full test results
175 …for review, along with a summary of test result changes [[example]](https://swiftshader-review.goo…
178 6. If the build included code coverage instrumentation, then the coverage
179 results are collated from all test runs, processed and compressed, and
180 …oaded to [github.com/swiftshader-regres/swiftshader-coverage](https://github.com/swiftshader-regre…
181 …ly reflected at [swiftshader-regres.github.io/swiftshader-coverage](https://swiftshader-regres.git…
182 This process is [described in more detail below](#code-coverage).
183 7. Stages 3 - 5 are repeated for both the LLVM and Subzero backends.
189 the test results of the parent can be calculated once and stored. A tested
194 cache-key for storing and retrieving data. Both the test lists and version of
200 …use of the Vulkan API by loading the [Vulkan Loader](https://github.com/KhronosGroup/Vulkan-Loader)
208 test results are independent of the system's Vulkan setup.
213 ### Code coverage
215 The [daily run](#daily-run-continuous-integration-testing) produces code
216 coverage information that can be examined for each individual dEQP test at
217 [swiftshader-regres.github.io/swiftshader-coverage](https://swiftshader-regres.github.io/swiftshade…
222 #### Per-test generation
224 Code coverage instrumentation is generated with
225 [clang's `--coverage`](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html)
230 variable value which dictates where the process writes its raw coverage profile
231 file. Each process gets a different path so that we can emit coverage from
236 …ang.llvm.org/docs/SourceBasedCodeCoverage.html#creating-coverage-reports) for processing coverage …
238 * `llvm-profdata` indexes the raw `.profraw` coverage profile file and emits a
240 * `llvm-cov` further processes the `.profdata` file into something human
243 `llvm-cov` provides many options, including emitting an pretty HTML file, but is
244 remarkably slow at producing easily machine-parsable data. Fortunately the core
245 of `llvm-cov` is [a few hundreds of lines of code](https://github.com/llvm/llvm-project/tree/master…
246 replaces `llvm-cov` with ["`turbo-cov`"](https://cs.opensource.google/swiftshader/SwiftShader/+/mas…
253 If you used 1 bit for each source line, per-line source coverage for all dEQP
257 around 10 MiB (~1100x reduction in size), and supports sub-line coverage scopes.
261 Code coverage information is described in spans.
264 line-column pair:
285 `dEQP-VK.fragment_shader_interlock.basic.discard.ssbo.sample_unordered.4xaa.sample_shading.16x16`
290 dEQP-VK <-- root group name
297 ╰ 16x16 <-- leaf test name
301 structure coverage data, as tests of the same group are likely to have similar
302 coverage spans.
305 non-leaf nodes, and tests as leaf nodes.
337 represent a test, and non-leaf nodes (`a`, `b`, `c`, `d`, `e`, `g`, `i`, `k`,
341 coverage spans with every leaf node (test) in this tree.
417 together. To further reduce coverage data, the whole graph is scanned for common
424 All spans, span-groups and strings are stored in de-duplicated tables, and are
431 The coverage data is encoded into JSON for parsing by the web page.
437 The zlib-compressed JSON coverage data is decompressed using
439 [vanilla JavaScript](https://github.com/swiftshader-regres/swiftshader-coverage/blob/gh-pages/index…
441 [`codemirror`](https://codemirror.net/) is used to perform coverage span and C++