1<!-- 2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al. 3 4SPDX-License-Identifier: curl 5--> 6 7# curl test suite file format 8 9The curl test suite's file format is simple and extendable, closely resembling 10XML. All data for a single test case resides in a single ASCII file. Labels 11mark the beginning and the end of all sections, and each label must be written 12in its own line. Comments are either XML-style (enclosed with `<!--` and 13`-->`) or shell script style (beginning with `#`) and must appear on their own 14lines and not alongside actual test data. Most test data files are 15syntactically valid XML, although a few files are not (lack of support for 16character entities and the preservation of CR/LF characters at the end of 17lines are the biggest differences). 18 19Each test case source exists as a file matching the format 20`tests/data/testNUM`, where `NUM` is the unique test number, and must begin 21with a `testcase` tag, which encompasses the remainder of the file. 22 23# Preprocessing 24 25When a test is to be executed, the source file is first preprocessed and 26variables are substituted by their respective contents and the output 27version of the test file is stored as `log/testNUM`. That version is what will 28be read and used by the test servers. 29 30## Base64 Encoding 31 32In the preprocess stage, a special instruction can be used to have runtests.pl 33base64 encode a certain section and insert in the generated output file. This 34is in particular good for test cases where the test tool is expected to pass 35in base64 encoded content that might use dynamic information that is unique 36for this particular test invocation, like the server port number. 37 38To insert a base64 encoded string into the output, use this syntax: 39 40 %b64[ data to encode ]b64% 41 42The data to encode can then use any of the existing variables mentioned below, 43or even percent-encoded individual bytes. As an example, insert the HTTP 44server's port number (in ASCII) followed by a space and the hexadecimal byte 459a: 46 47 %b64[%HTTPPORT %9a]b64% 48 49## Hexadecimal decoding 50 51In the preprocess stage, a special instruction can be used to have runtests.pl 52generate a sequence of binary bytes. 53 54To insert a sequence of bytes from a hex encoded string, use this syntax: 55 56 %hex[ %XX-encoded data to decode ]hex% 57 58For example, to insert the binary octets 0, 1 and 255 into the test file: 59 60 %hex[ %00%01%FF ]hex% 61 62## Repeat content 63 64In the preprocess stage, a special instruction can be used to have runtests.pl 65generate a repetitive sequence of bytes. 66 67To insert a sequence of repeat bytes, use this syntax to make the `<string>` 68get repeated `<number>` of times. The number has to be 1 or larger and the 69string may contain `%HH` hexadecimal codes: 70 71 %repeat[<number> x <string>]% 72 73For example, to insert the word hello 100 times: 74 75 %repeat[100 x hello]% 76 77## Conditional lines 78 79Lines in the test file can be made to appear conditionally on a specific 80feature (see the "features" section below) being set or not set. If the 81specific feature is present, the following lines will be output, otherwise it 82outputs nothing, until a following else or `endif` clause. Like this: 83 84 %if brotli 85 Accept-Encoding 86 %endif 87 88It can also check for the inverse condition, so if the feature is *not* set by 89the use of an exclamation mark: 90 91 %if !brotli 92 Accept-Encoding: not-brotli 93 %endif 94 95You can also make an "else" clause to get output for the opposite condition, 96like: 97 98 %if brotli 99 Accept-Encoding: brotli 100 %else 101 Accept-Encoding: nothing 102 %endif 103 104**Note** that there can be no nested conditions. You can only do one 105conditional at a time and you can only check for a single feature in it. 106 107# Variables 108 109When the test is preprocessed, a range of "variables" in the test file will be 110replaced by their content at that time. 111 112Available substitute variables include: 113 114- `%CLIENT6IP` - IPv6 address of the client running curl 115- `%CLIENTIP` - IPv4 address of the client running curl 116- `%CURL` - Path to the curl executable 117- `%FILE_PWD` - Current directory, on Windows prefixed with a slash 118- `%FTP6PORT` - IPv6 port number of the FTP server 119- `%FTPPORT` - Port number of the FTP server 120- `%FTPSPORT` - Port number of the FTPS server 121- `%FTPTIME2` - Timeout in seconds that should be just sufficient to receive a 122 response from the test FTP server 123- `%FTPTIME3` - Even longer than `%FTPTIME2` 124- `%GOPHER6PORT` - IPv6 port number of the Gopher server 125- `%GOPHERPORT` - Port number of the Gopher server 126- `%GOPHERSPORT` - Port number of the Gophers server 127- `%HOST6IP` - IPv6 address of the host running this test 128- `%HOSTIP` - IPv4 address of the host running this test 129- `%HTTP6PORT` - IPv6 port number of the HTTP server 130- `%HTTPPORT` - Port number of the HTTP server 131- `%HTTP2PORT` - Port number of the HTTP/2 server 132- `%HTTPSPORT` - Port number of the HTTPS server 133- `%HTTPSPROXYPORT` - Port number of the HTTPS-proxy 134- `%HTTPTLS6PORT` - IPv6 port number of the HTTP TLS server 135- `%HTTPTLSPORT` - Port number of the HTTP TLS server 136- `%HTTPUNIXPATH` - Path to the Unix socket of the HTTP server 137- `%SOCKSUNIXPATH` - Absolute Path to the Unix socket of the SOCKS server 138- `%IMAP6PORT` - IPv6 port number of the IMAP server 139- `%IMAPPORT` - Port number of the IMAP server 140- `%MQTTPORT` - Port number of the MQTT server 141- `%TELNETPORT` - Port number of the telnet server 142- `%NOLISTENPORT` - Port number where no service is listening 143- `%POP36PORT` - IPv6 port number of the POP3 server 144- `%POP3PORT` - Port number of the POP3 server 145- `%POSIX_PWD` - Current directory somewhat mingw friendly 146- `%PROXYPORT` - Port number of the HTTP proxy 147- `%PWD` - Current directory 148- `%RTSP6PORT` - IPv6 port number of the RTSP server 149- `%RTSPPORT` - Port number of the RTSP server 150- `%SMBPORT` - Port number of the SMB server 151- `%SMBSPORT` - Port number of the SMBS server 152- `%SMTP6PORT` - IPv6 port number of the SMTP server 153- `%SMTPPORT` - Port number of the SMTP server 154- `%SOCKSPORT` - Port number of the SOCKS4/5 server 155- `%SRCDIR` - Full path to the source dir 156- `%SSHPORT` - Port number of the SCP/SFTP server 157- `%SSHSRVMD5` - MD5 of SSH server's public key 158- `%SSHSRVSHA256` - SHA256 of SSH server's public key 159- `%SSH_PWD` - Current directory friendly for the SSH server 160- `%TESTNUMBER` - Number of the test case 161- `%TFTP6PORT` - IPv6 port number of the TFTP server 162- `%TFTPPORT` - Port number of the TFTP server 163- `%USER` - Login ID of the user running the test 164- `%VERSION` - the full version number of the tested curl 165 166# `<testcase>` 167 168Each test is always specified entirely within the `testcase` tag. Each test 169case is split up in four main sections: `info`, `reply`, `client` and 170`verify`. 171 172- **info** provides information about the test case 173 174- **reply** is used for the server to know what to send as a reply for the 175requests curl sends 176 177- **client** defines how the client should behave 178 179- **verify** defines how to verify that the data stored after a command has 180been run ended up correctly 181 182Each main section has a number of available subsections that can be specified, 183that will be checked/used if specified. 184 185## `<info>` 186 187### `<keywords>` 188A newline-separated list of keywords describing what this test case uses and 189tests. Try to use already used keywords. These keywords will be used for 190statistical/informational purposes and for choosing or skipping classes of 191tests. Keywords must begin with an alphabetic character, `-`, `[` or `{` and 192may actually consist of multiple words separated by spaces which are treated 193together as a single identifier. 194 195When using curl built with Hyper, the keywords must include HTTP or HTTPS for 196'hyper mode' to kick in and make line ending checks work for tests. 197## `<reply>` 198 199### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"] [nonewline="yes"] [crlf="yes"]>` 200 201data to be sent to the client on its request and later verified that it 202arrived safely. Set `nocheck="yes"` to prevent the test script from verifying 203the arrival of this data. 204 205If the data contains `swsclose` anywhere within the start and end tag, and 206this is an HTTP test, then the connection will be closed by the server after 207this response is sent. If not, the connection will be kept persistent. 208 209If the data contains `swsbounce` anywhere within the start and end tag, the 210HTTP server will detect if this is a second request using the same test and 211part number and will then increase the part number with one. This is useful 212for auth tests and similar. 213 214`sendzero=yes` means that the (FTP) server will "send" the data even if the 215size is zero bytes. Used to verify curl's behavior on zero bytes transfers. 216 217`base64=yes` means that the data provided in the test-file is a chunk of data 218encoded with base64. It is the only way a test case can contain binary 219data. (This attribute can in fact be used on any section, but it doesn't make 220much sense for other sections than "data"). 221 222`hex=yes` means that the data is a sequence of hex pairs. It will get decoded 223and used as "raw" data. 224 225`nonewline=yes` means that the last byte (the trailing newline character) 226should be cut off from the data before sending or comparing it. 227 228`crlf=yes` forces *header* newlines to become CRLF even if not written so in 229the source file. Note that this makes runtests.pl parse and "guess" what is a 230header and what is not in order to apply the CRLF line endings appropriately. 231 232For FTP file listings, the `<data>` section will be used *only* if you make 233sure that there has been a CWD done first to a directory named `test-[NUM]` 234where `NUM` is the test case number. Otherwise the ftp server can't know from 235which test file to load the list content. 236 237### `<dataNUM [crlf="yes"]>` 238 239Send back this contents instead of the <data> one. The `NUM` is set by: 240 241 - The test number in the request line is >10000 and this is the remainder 242 of [test case number]%10000. 243 - The request was HTTP and included digest details, which adds 1000 to `NUM` 244 - If an HTTP request is NTLM type-1, it adds 1001 to `NUM` 245 - If an HTTP request is NTLM type-3, it adds 1002 to `NUM` 246 - If an HTTP request is Basic and `NUM` is already >=1000, it adds 1 to `NUM` 247 - If an HTTP request is Negotiate, `NUM` gets incremented by one for each 248 request with Negotiate authorization header on the same test case. 249 250Dynamically changing `NUM` in this way allows the test harness to be used to 251test authentication negotiation where several different requests must be sent 252to complete a transfer. The response to each request is found in its own data 253section. Validating the entire negotiation sequence can be done by specifying 254a `datacheck` section. 255 256### `<connect>` 257The connect section is used instead of the 'data' for all CONNECT 258requests. The remainder of the rules for the data section then apply but with 259a connect prefix. 260 261### `<socks>` 262Address type and address details as logged by the SOCKS proxy. 263 264### `<datacheck [mode="text"] [nonewline="yes"] [crlf="yes"]>` 265if the data is sent but this is what should be checked afterwards. If 266`nonewline=yes` is set, runtests will cut off the trailing newline from the 267data before comparing with the one actually received by the client. 268 269Use the `mode="text"` attribute if the output is in text mode on platforms 270that have a text/binary difference. 271 272### `<datacheckNUM [nonewline="yes"] [mode="text"] [crlf="yes"]>` 273The contents of numbered `datacheck` sections are appended to the non-numbered 274one. 275 276### `<size>` 277number to return on a ftp SIZE command (set to -1 to make this command fail) 278 279### `<mdtm>` 280what to send back if the client sends a (FTP) `MDTM` command, set to -1 to 281have it return that the file doesn't exist 282 283### `<postcmd>` 284special purpose server-command to control its behavior *after* the 285reply is sent 286For HTTP/HTTPS, these are supported: 287 288`wait [secs]` - Pause for the given time 289 290### `<servercmd>` 291Special-commands for the server. 292 293The first line of this file will always be set to `Testnum [number]` by the 294test script, to allow servers to read that to know what test the client is 295about to issue. 296 297#### For FTP/SMTP/POP/IMAP 298 299- `REPLY [command] [return value] [response string]` - Changes how the server 300 responds to the [command]. [response string] is evaluated as a perl string, 301 so it can contain embedded \r\n, for example. There's a special [command] 302 named "welcome" (without quotes) which is the string sent immediately on 303 connect as a welcome. 304- `REPLYLF` (like above but sends the response terminated with LF-only and not 305 CRLF) 306- `COUNT [command] [num]` - Do the `REPLY` change for `[command]` only `[num]` 307 times and then go back to the built-in approach 308- `DELAY [command] [secs]` - Delay responding to this command for the given 309 time 310- `RETRWEIRDO` - Enable the "weirdo" RETR case when multiple response lines 311 appear at once when a file is transferred 312- `RETRNOSIZE` - Make sure the RETR response doesn't contain the size of the 313 file 314- `RETRSIZE [size]` - Force RETR response to contain the specified size 315- `NOSAVE` - Don't actually save what is received 316- `SLOWDOWN` - Send FTP responses with 0.01 sec delay between each byte 317- `PASVBADIP` - makes PASV send back an illegal IP in its 227 response 318- `CAPA [capabilities]` - Enables support for and specifies a list of space 319 separated capabilities to return to the client for the IMAP `CAPABILITY`, 320 POP3 `CAPA` and SMTP `EHLO` commands 321- `AUTH [mechanisms]` - Enables support for SASL authentication and specifies 322 a list of space separated mechanisms for IMAP, POP3 and SMTP 323- `STOR [msg]` respond with this instead of default after `STOR` 324 325#### For HTTP/HTTPS 326 327- `auth_required` if this is set and a POST/PUT is made without auth, the 328 server will NOT wait for the full request body to get sent 329- `idle` - do nothing after receiving the request, just "sit idle" 330- `stream` - continuously send data to the client, never-ending 331- `writedelay: [msecs]` delay this amount between reply packets 332- `skip: [num]` - instructs the server to ignore reading this many bytes from 333 a PUT or POST request 334- `rtp: part [num] channel [num] size [num]` - stream a fake RTP packet for 335 the given part on a chosen channel with the given payload size 336- `connection-monitor` - When used, this will log `[DISCONNECT]` to the 337 `server.input` log when the connection is disconnected. 338- `upgrade` - when an HTTP upgrade header is found, the server will upgrade to 339 http2 340- `swsclose` - instruct server to close connection after response 341- `no-expect` - don't read the request body if Expect: is present 342 343#### For TFTP 344`writedelay: [secs]` delay this amount between reply packets (each packet 345 being 512 bytes payload) 346 347## `<client>` 348 349### `<server>` 350What server(s) this test case requires/uses. Available servers: 351 352- `dict` 353- `file` 354- `ftp` 355- `ftp-ipv6` 356- `ftps` 357- `gopher` 358- `gopher-ipv6` 359- `gophers` 360- `http` 361- `http/2` 362- `http-ipv6` 363- `http-proxy` 364- `https` 365- `https-proxy` 366- `httptls+srp` 367- `httptls+srp-ipv6` 368- `http-unix` 369- `imap` 370- `mqtt` 371- `none` 372- `pop3` 373- `rtsp` 374- `rtsp-ipv6` 375- `scp` 376- `sftp` 377- `smb` 378- `smtp` 379- `socks4` 380- `socks5` 381- `socks5unix` 382- `telnet` 383- `tftp` 384 385Give only one per line. This subsection is mandatory (use `none` if no servers 386are required). Servers that require a special server certificate can have the 387PEM certificate file name (found in the `certs` directory) appended to the 388server name separated by a space. 389 390### `<features>` 391A list of features that MUST be present in the client/library for this test to 392be able to run. If a required feature is not present then the test will be 393SKIPPED. 394 395Alternatively a feature can be prefixed with an exclamation mark to indicate a 396feature is NOT required. If the feature is present then the test will be 397SKIPPED. 398 399Features testable here are: 400 401- `alt-svc` 402- `bearssl` 403- `c-ares` 404- `cookies` 405- `crypto` 406- `debug` 407- `DoH` 408- `getrlimit` 409- `GnuTLS` 410- `GSS-API` 411- `h2c` 412- `HSTS` 413- `HTTP-auth` 414- `http/2` 415- `hyper` 416- `idn` 417- `ipv6` 418- `Kerberos` 419- `large_file` 420- `ld_preload` 421- `libssh2` 422- `libssh` 423- `oldlibssh` (versions before 0.9.4) 424- `libz` 425- `manual` 426- `mbedtls` 427- `Mime` 428- `netrc` 429- `NSS` 430- `NTLM` 431- `OpenSSL` 432- `parsedate` 433- `proxy` 434- `PSL` 435- `rustls` 436- `Schannel` 437- `sectransp` 438- `shuffle-dns` 439- `socks` 440- `SPNEGO` 441- `SSL` 442- `SSLpinning` 443- `SSPI` 444- `threaded-resolver` 445- `TLS-SRP` 446- `TrackMemory` 447- `typecheck` 448- `Unicode` 449- `unittest` 450- `unix-sockets` 451- `verbose-strings` 452- `wakeup` 453- `win32` 454- `wolfssh` 455- `wolfssl` 456- `xattr` 457 458as well as each protocol that curl supports. A protocol only needs to be 459specified if it is different from the server (useful when the server is 460`none`). 461 462### `<killserver>` 463Using the same syntax as in `<server>` but when mentioned here these servers 464are explicitly KILLED when this test case is completed. Only use this if there 465is no other alternatives. Using this of course requires subsequent tests to 466restart servers. 467 468### `<precheck>` 469A command line that if set gets run by the test script before the test. If an 470output is displayed by the command or if the return code is non-zero, the test 471will be skipped and the (single-line) output will be displayed as reason for 472not running the test. 473 474### `<postcheck>` 475A command line that if set gets run by the test script after the test. If 476the command exists with a non-zero status code, the test will be considered 477to have failed. 478 479### `<tool>` 480Name of tool to invoke instead of "curl". This tool must be built and exist 481either in the `libtest/` directory (if the tool name starts with `lib`) or in 482the `unit/` directory (if the tool name starts with `unit`). 483 484### `<name>` 485Brief test case description, shown when the test runs. 486 487### `<setenv>` 488 variable1=contents1 489 variable2=contents2 490 491Set the given environment variables to the specified value before the actual 492command is run. They are cleared again after the command has been run. 493 494### `<command [option="no-output/no-include/force-output/binary-trace"] [timeout="secs"][delay="secs"][type="perl/shell"]>` 495Command line to run. 496 497Note that the URL that gets passed to the server actually controls what data 498that is returned. The last slash in the URL must be followed by a number. That 499number (N) will be used by the test-server to load test case N and return the 500data that is defined within the `<reply><data></data></reply>` section. 501 502If there's no test number found above, the HTTP test server will use the 503number following the last dot in the given hostname (made so that a CONNECT 504can still pass on test number) so that "foo.bar.123" gets treated as test case 505123. Alternatively, if an IPv6 address is provided to CONNECT, the last 506hexadecimal group in the address will be used as the test number! For example 507the address "[1234::ff]" would be treated as test case 255. 508 509Set `type="perl"` to write the test case as a perl script. It implies that 510there's no memory debugging and valgrind gets shut off for this test. 511 512Set `type="shell"` to write the test case as a shell script. It implies that 513there's no memory debugging and valgrind gets shut off for this test. 514 515Set `option="no-output"` to prevent the test script to slap on the `--output` 516argument that directs the output to a file. The `--output` is also not added 517if the verify/stdout section is used. 518 519Set `option="force-output"` to make use of `--output` even when the test is 520otherwise written to verify stdout. 521 522Set `option="no-include"` to prevent the test script to slap on the 523`--include` argument. 524 525Set `option="binary-trace"` to use `--trace` instead of `--trace-ascii` for 526tracing. Suitable for binary-oriented protocols such as MQTT. 527 528Set `timeout="secs"` to override default server logs advisor read lock 529timeout. This timeout is used by the test harness, once that the command has 530completed execution, to wait for the test server to write out server side log 531files and remove the lock that advised not to read them. The "secs" parameter 532is the not negative integer number of seconds for the timeout. This `timeout` 533attribute is documented for completeness sake, but is deep test harness stuff 534and only needed for singular and specific test cases. Avoid using it. 535 536Set `delay="secs"` to introduce a time delay once that the command has 537completed execution and before the `<postcheck>` section runs. The "secs" 538parameter is the not negative integer number of seconds for the delay. This 539'delay' attribute is intended for specific test cases, and normally not 540needed. 541 542### `<file name="log/filename" [nonewline="yes"]>` 543This creates the named file with this content before the test case is run, 544which is useful if the test case needs a file to act on. 545 546If `nonewline="yes"` is used, the created file will have the final newline 547stripped off. 548 549### `<stdin [nonewline="yes"]>` 550Pass this given data on stdin to the tool. 551 552If `nonewline` is set, we will cut off the trailing newline of this given data 553before comparing with the one actually received by the client 554 555## `<verify>` 556### `<errorcode>` 557numerical error code curl is supposed to return. Specify a list of accepted 558error codes by separating multiple numbers with comma. See test 237 for an 559example. 560 561### `<strip>` 562One regex per line that is removed from the protocol dumps before the 563comparison is made. This is useful to remove dependencies on dynamically 564changing protocol data such as port numbers or user-agent strings. 565 566### `<strippart>` 567One perl op per line that operates on the protocol dump. This is pretty 568advanced. Example: `s/^EPRT .*/EPRT stripped/`. 569 570### `<protocol [nonewline="yes"][crlf="yes"]>` 571 572the protocol dump curl should transmit, if `nonewline` is set, we will cut off 573the trailing newline of this given data before comparing with the one actually 574sent by the client The `<strip>` and `<strippart>` rules are applied before 575comparisons are made. 576 577`crlf=yes` forces the newlines to become CRLF even if not written so in the 578test. 579 580### `<proxy [nonewline="yes"][crlf="yes"]>` 581 582The protocol dump curl should transmit to an HTTP proxy (when the http-proxy 583server is used), if `nonewline` is set, we will cut off the trailing newline 584of this given data before comparing with the one actually sent by the client 585The `<strip>` and `<strippart>` rules are applied before comparisons are made. 586 587### `<stderr [mode="text"] [nonewline="yes"]>` 588This verifies that this data was passed to stderr. 589 590Use the mode="text" attribute if the output is in text mode on platforms that 591have a text/binary difference. 592 593If `nonewline` is set, we will cut off the trailing newline of this given data 594before comparing with the one actually received by the client 595 596### `<stdout [mode="text"] [nonewline="yes"] [crlf="yes"]>` 597This verifies that this data was passed to stdout. 598 599Use the mode="text" attribute if the output is in text mode on platforms that 600have a text/binary difference. 601 602If `nonewline` is set, we will cut off the trailing newline of this given data 603before comparing with the one actually received by the client 604 605`crlf=yes` forces the newlines to become CRLF even if not written so in the 606test. 607 608### `<file name="log/filename" [mode="text"]>` 609The file's contents must be identical to this after the test is complete. Use 610the mode="text" attribute if the output is in text mode on platforms that have 611a text/binary difference. 612 613### `<file1>` 6141 to 4 can be appended to 'file' to compare more files. 615 616### `<file2>` 617 618### `<file3>` 619 620### `<file4>` 621 622### `<stripfile>` 623One perl op per line that operates on the output file or stdout before being 624compared with what is stored in the test file. This is pretty 625advanced. Example: "s/^EPRT .*/EPRT stripped/" 626 627### `<stripfile1>` 6281 to 4 can be appended to `stripfile` to strip the corresponding <fileN> 629content 630 631### `<stripfile2>` 632 633### `<stripfile3>` 634 635### `<stripfile4>` 636 637### `<upload>` 638the contents of the upload data curl should have sent 639 640### `<valgrind>` 641disable - disables the valgrind log check for this test 642