1# Web Platform Tests 2 3The tests here are drivers for running the [Web Platform Tests][]. 4 5See [`test/fixtures/wpt/README.md`][] for a hash of the last 6updated WPT commit for each module being covered here. 7 8See the json files in [the `status` folder](./status) for prerequisites, 9expected failures, and support status for specific tests in each module. 10 11Currently there are still some Web Platform Tests titled `test-whatwg-*` 12under `test/parallel` that have not been migrated to be run with the 13WPT harness and have automatic updates. There are also a few 14`test-whatwg-*-custom-*` tests that may need to be upstreamed. 15This folder covers the tests that have been migrated. 16 17<a id="add-tests"></a> 18## How to add tests for a new module 19 20### 1. Create a status file 21 22For example, to add the URL tests, add a `test/wpt/status/url.json` file. 23 24In the beginning, it's fine to leave an empty object `{}` in the file if 25it's not yet clear how compliant the implementation is, 26the requirements and expected failures can be figured out in a later step 27when the tests are run for the first time. 28 29See [Format of a status JSON file](#status-format) for details. 30 31### 2. Pull the WPT files 32 33Use the [git node wpt][] command to download the WPT files into 34`test/fixtures/wpt`. For example, to add URL tests: 35 36```text 37$ cd /path/to/node/project 38$ git node wpt url 39``` 40 41### 3. Create the test driver 42 43For example, for the URL tests, add a file `test/wpt/test-url.js`: 44 45```js 46'use strict'; 47 48require('../common'); 49const { WPTRunner } = require('../common/wpt'); 50 51const runner = new WPTRunner('url'); 52 53// Set Node.js flags required for the tests. 54runner.setFlags(['--expose-internals']); 55 56// Set a script that will be executed in the worker before running the tests. 57runner.setInitScript(` 58 const { internalBinding } = require('internal/test/binding'); 59 const { DOMException } = internalBinding('messaging'); 60 global.DOMException = DOMException; 61`); 62 63runner.runJsTests(); 64``` 65 66This driver is capable of running the tests located in `test/fixtures/wpt/url` 67with the WPT harness while taking the status file into account. 68 69### 4. Run the tests 70 71Run the test using `tools/test.py` and see if there are any failures. 72For example, to run all the URL tests under `test/fixtures/wpt/url`: 73 74```text 75$ tools/test.py wpt/test-url 76``` 77 78To run a specific test in WPT, for example, `url/url-searchparams.any.js`, 79pass the file name as argument to the corresponding test driver: 80 81```text 82node test/wpt/test-url.js url-searchparams.any.js 83``` 84 85If there are any failures, update the corresponding status file 86(in this case, `test/wpt/status/url.json`) to make the test pass. 87 88For example, to mark `url/url-searchparams.any.js` as expected to fail, 89add this to `test/wpt/status/url.json`: 90 91```json 92 "url-searchparams.any.js": { 93 "fail": "explain why the test fails, ideally with links" 94 } 95``` 96 97See [Format of a status JSON file](#status-format) for details. 98 99### 5. Commit the changes and submit a Pull Request 100 101See [the contributing guide](../../CONTRIBUTING.md). 102 103## How to update tests for a module 104 105The tests can be updated in a way similar to how they are added. 106Run Step 2 and Step 4 of [adding tests for a new module](#add-tests). 107 108The [git node wpt][] command maintains the status of the local 109WPT subset, if no files are updated after running it for a module, 110the local subset is up to date and there is no need to update them 111until they are changed in the upstream. 112 113## How it works 114 115Note: currently this test suite only supports `.js` tests. There is 116ongoing work in the upstream to properly split out the tests into files 117that can be run in a shell environment like Node.js. 118 119### Getting the original test files and harness from WPT 120 121The original files and harness from WPT are downloaded and stored in 122`test/fixtures/wpt`. 123 124The [git node wpt][] command automate this process while maintaining a map 125containing the hash of the last updated commit for each module in 126`test/fixtures/wpt/versions.json` and [`test/fixtures/wpt/README.md`][]. 127It also maintains the LICENSE file in `test/fixtures/wpt`. 128 129### Loading and running the tests 130 131Given a module, the `WPTRunner` class in [`test/common/wpt`](../common/wpt.js) 132loads: 133 134* `.js` test files (for example, `test/common/wpt/url/*.js` for `url`) 135* Status file (for example, `test/wpt/status/url.json` for `url`) 136* The WPT harness 137 138Then, for each test, it creates a worker thread with the globals and mocks, 139sets up the harness result hooks, loads the metadata in the test (including 140loading extra resources), and runs all the tests in that worker thread, 141skipping tests that cannot be run because of lack of dependency or 142expected failures. 143 144<a id="status-format"></a> 145## Format of a status JSON file 146 147```text 148{ 149 "something.scope.js": { // the file name 150 // Optional: If the requirement is not met, this test will be skipped 151 "requires": ["small-icu"], // supports: "small-icu", "full-icu" 152 153 // Optional: the test will be skipped with the reason printed 154 "skip": "explain why we cannot run a test that's supposed to pass", 155 156 // Optional: the test will be skipped with the reason printed 157 "fail": "explain why we the test is expected to fail" 158 } 159} 160``` 161 162A test may have to be skipped because it depends on another irrelevant 163Web API, or certain harness has not been ported in our test runner yet. 164In that case it needs to be marked with `skip` instead of `fail`. 165 166[Web Platform Tests]: https://github.com/web-platform-tests/wpt 167[`test/fixtures/wpt/README.md`]: ../fixtures/wpt/README.md 168[git node wpt]: https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt 169