1<html> 2<head> 3<title>pcre2test specification</title> 4</head> 5<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6<h1>pcre2test man page</h1> 7<p> 8Return to the <a href="index.html">PCRE2 index page</a>. 9</p> 10<p> 11This page is part of the PCRE2 HTML documentation. It was generated 12automatically from the original man page. If there is any nonsense in it, 13please consult the man page, in case the conversion went wrong. 14<br> 15<ul> 16<li><a name="TOC1" href="#SEC1">SYNOPSIS</a> 17<li><a name="TOC2" href="#SEC2">PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a> 18<li><a name="TOC3" href="#SEC3">INPUT ENCODING</a> 19<li><a name="TOC4" href="#SEC4">COMMAND LINE OPTIONS</a> 20<li><a name="TOC5" href="#SEC5">DESCRIPTION</a> 21<li><a name="TOC6" href="#SEC6">COMMAND LINES</a> 22<li><a name="TOC7" href="#SEC7">MODIFIER SYNTAX</a> 23<li><a name="TOC8" href="#SEC8">PATTERN SYNTAX</a> 24<li><a name="TOC9" href="#SEC9">SUBJECT LINE SYNTAX</a> 25<li><a name="TOC10" href="#SEC10">PATTERN MODIFIERS</a> 26<li><a name="TOC11" href="#SEC11">SUBJECT MODIFIERS</a> 27<li><a name="TOC12" href="#SEC12">THE ALTERNATIVE MATCHING FUNCTION</a> 28<li><a name="TOC13" href="#SEC13">DEFAULT OUTPUT FROM pcre2test</a> 29<li><a name="TOC14" href="#SEC14">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a> 30<li><a name="TOC15" href="#SEC15">RESTARTING AFTER A PARTIAL MATCH</a> 31<li><a name="TOC16" href="#SEC16">CALLOUTS</a> 32<li><a name="TOC17" href="#SEC17">NON-PRINTING CHARACTERS</a> 33<li><a name="TOC18" href="#SEC18">SAVING AND RESTORING COMPILED PATTERNS</a> 34<li><a name="TOC19" href="#SEC19">SEE ALSO</a> 35<li><a name="TOC20" href="#SEC20">AUTHOR</a> 36<li><a name="TOC21" href="#SEC21">REVISION</a> 37</ul> 38<br><a name="SEC1" href="#TOC1">SYNOPSIS</a><br> 39<P> 40<b>pcre2test [options] [input file [output file]]</b> 41<br> 42<br> 43<b>pcre2test</b> is a test program for the PCRE2 regular expression libraries, 44but it can also be used for experimenting with regular expressions. This 45document describes the features of the test program; for details of the regular 46expressions themselves, see the 47<a href="pcre2pattern.html"><b>pcre2pattern</b></a> 48documentation. For details of the PCRE2 library function calls and their 49options, see the 50<a href="pcre2api.html"><b>pcre2api</b></a> 51documentation. 52</P> 53<P> 54The input for <b>pcre2test</b> is a sequence of regular expression patterns and 55subject strings to be matched. There are also command lines for setting 56defaults and controlling some special actions. The output shows the result of 57each match attempt. Modifiers on external or internal command lines, the 58patterns, and the subject lines specify PCRE2 function options, control how the 59subject is processed, and what output is produced. 60</P> 61<P> 62There are many obscure modifiers, some of which are specifically designed for 63use in conjunction with the test script and data files that are distributed as 64part of PCRE2. All the modifiers are documented here, some without much 65justification, but many of them are unlikely to be of use except when testing 66the libraries. 67</P> 68<br><a name="SEC2" href="#TOC1">PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a><br> 69<P> 70Different versions of the PCRE2 library can be built to support character 71strings that are encoded in 8-bit, 16-bit, or 32-bit code units. One, two, or 72all three of these libraries may be simultaneously installed. The 73<b>pcre2test</b> program can be used to test all the libraries. However, its own 74input and output are always in 8-bit format. When testing the 16-bit or 32-bit 75libraries, patterns and subject strings are converted to 16-bit or 32-bit 76format before being passed to the library functions. Results are converted back 77to 8-bit code units for output. 78</P> 79<P> 80In the rest of this document, the names of library functions and structures 81are given in generic form, for example, <b>pcre2_compile()</b>. The actual 82names used in the libraries have a suffix _8, _16, or _32, as appropriate. 83<a name="inputencoding"></a></P> 84<br><a name="SEC3" href="#TOC1">INPUT ENCODING</a><br> 85<P> 86Input to <b>pcre2test</b> is processed line by line, either by calling the C 87library's <b>fgets()</b> function, or via the <b>libreadline</b> or <b>libedit</b> 88library. In some Windows environments character 26 (hex 1A) causes an immediate 89end of file, and no further data is read, so this character should be avoided 90unless you really want that action. 91</P> 92<P> 93The input is processed using using C's string functions, so must not 94contain binary zeros, even though in Unix-like environments, <b>fgets()</b> 95treats any bytes other than newline as data characters. An error is generated 96if a binary zero is encountered. By default subject lines are processed for 97backslash escapes, which makes it possible to include any data value in strings 98that are passed to the library for matching. For patterns, there is a facility 99for specifying some or all of the 8-bit input characters as hexadecimal pairs, 100which makes it possible to include binary zeros. 101</P> 102<br><b> 103Input for the 16-bit and 32-bit libraries 104</b><br> 105<P> 106When testing the 16-bit or 32-bit libraries, there is a need to be able to 107generate character code points greater than 255 in the strings that are passed 108to the library. For subject lines, backslash escapes can be used. In addition, 109when the <b>utf</b> modifier (see 110<a href="#optionmodifiers">"Setting compilation options"</a> 111below) is set, the pattern and any following subject lines are interpreted as 112UTF-8 strings and translated to UTF-16 or UTF-32 as appropriate. 113</P> 114<P> 115For non-UTF testing of wide characters, the <b>utf8_input</b> modifier can be 116used. This is mutually exclusive with <b>utf</b>, and is allowed only in 16-bit 117or 32-bit mode. It causes the pattern and following subject lines to be treated 118as UTF-8 according to the original definition (RFC 2279), which allows for 119character values up to 0x7fffffff. Each character is placed in one 16-bit or 12032-bit code unit (in the 16-bit case, values greater than 0xffff cause an error 121to occur). 122</P> 123<P> 124UTF-8 (in its original definition) is not capable of encoding values greater 125than 0x7fffffff, but such values can be handled by the 32-bit library. When 126testing this library in non-UTF mode with <b>utf8_input</b> set, if any 127character is preceded by the byte 0xff (which is an invalid byte in UTF-8) 1280x80000000 is added to the character's value. This is the only way of passing 129such code points in a pattern string. For subject strings, using an escape 130sequence is preferable. 131</P> 132<br><a name="SEC4" href="#TOC1">COMMAND LINE OPTIONS</a><br> 133<P> 134<b>-8</b> 135If the 8-bit library has been built, this option causes it to be used (this is 136the default). If the 8-bit library has not been built, this option causes an 137error. 138</P> 139<P> 140<b>-16</b> 141If the 16-bit library has been built, this option causes it to be used. If only 142the 16-bit library has been built, this is the default. If the 16-bit library 143has not been built, this option causes an error. 144</P> 145<P> 146<b>-32</b> 147If the 32-bit library has been built, this option causes it to be used. If only 148the 32-bit library has been built, this is the default. If the 32-bit library 149has not been built, this option causes an error. 150</P> 151<P> 152<b>-ac</b> 153Behave as if each pattern has the <b>auto_callout</b> modifier, that is, insert 154automatic callouts into every pattern that is compiled. 155</P> 156<P> 157<b>-AC</b> 158As for <b>-ac</b>, but in addition behave as if each subject line has the 159<b>callout_extra</b> modifier, that is, show additional information from 160callouts. 161</P> 162<P> 163<b>-b</b> 164Behave as if each pattern has the <b>fullbincode</b> modifier; the full 165internal binary form of the pattern is output after compilation. 166</P> 167<P> 168<b>-C</b> 169Output the version number of the PCRE2 library, and all available information 170about the optional features that are included, and then exit with zero exit 171code. All other options are ignored. If both -C and -LM are present, whichever 172is first is recognized. 173</P> 174<P> 175<b>-C</b> <i>option</i> 176Output information about a specific build-time option, then exit. This 177functionality is intended for use in scripts such as <b>RunTest</b>. The 178following options output the value and set the exit code as indicated: 179<pre> 180 ebcdic-nl the code for LF (= NL) in an EBCDIC environment: 181 0x15 or 0x25 182 0 if used in an ASCII environment 183 exit code is always 0 184 linksize the configured internal link size (2, 3, or 4) 185 exit code is set to the link size 186 newline the default newline setting: 187 CR, LF, CRLF, ANYCRLF, ANY, or NUL 188 exit code is always 0 189 bsr the default setting for what \R matches: 190 ANYCRLF or ANY 191 exit code is always 0 192</pre> 193The following options output 1 for true or 0 for false, and set the exit code 194to the same value: 195<pre> 196 backslash-C \C is supported (not locked out) 197 ebcdic compiled for an EBCDIC environment 198 jit just-in-time support is available 199 pcre2-16 the 16-bit library was built 200 pcre2-32 the 32-bit library was built 201 pcre2-8 the 8-bit library was built 202 unicode Unicode support is available 203</pre> 204If an unknown option is given, an error message is output; the exit code is 0. 205</P> 206<P> 207<b>-d</b> 208Behave as if each pattern has the <b>debug</b> modifier; the internal 209form and information about the compiled pattern is output after compilation; 210<b>-d</b> is equivalent to <b>-b -i</b>. 211</P> 212<P> 213<b>-dfa</b> 214Behave as if each subject line has the <b>dfa</b> modifier; matching is done 215using the <b>pcre2_dfa_match()</b> function instead of the default 216<b>pcre2_match()</b>. 217</P> 218<P> 219<b>-error</b> <i>number[,number,...]</i> 220Call <b>pcre2_get_error_message()</b> for each of the error numbers in the 221comma-separated list, display the resulting messages on the standard output, 222then exit with zero exit code. The numbers may be positive or negative. This is 223a convenience facility for PCRE2 maintainers. 224</P> 225<P> 226<b>-help</b> 227Output a brief summary these options and then exit. 228</P> 229<P> 230<b>-i</b> 231Behave as if each pattern has the <b>info</b> modifier; information about the 232compiled pattern is given after compilation. 233</P> 234<P> 235<b>-jit</b> 236Behave as if each pattern line has the <b>jit</b> modifier; after successful 237compilation, each pattern is passed to the just-in-time compiler, if available. 238</P> 239<P> 240<b>-jitfast</b> 241Behave as if each pattern line has the <b>jitfast</b> modifier; after 242successful compilation, each pattern is passed to the just-in-time compiler, if 243available, and each subject line is passed directly to the JIT matcher via its 244"fast path". 245</P> 246<P> 247<b>-jitverify</b> 248Behave as if each pattern line has the <b>jitverify</b> modifier; after 249successful compilation, each pattern is passed to the just-in-time compiler, if 250available, and the use of JIT for matching is verified. 251</P> 252<P> 253<b>-LM</b> 254List modifiers: write a list of available pattern and subject modifiers to the 255standard output, then exit with zero exit code. All other options are ignored. 256If both -C and any -Lx options are present, whichever is first is recognized. 257</P> 258<P> 259<b>-LP</b> 260List properties: write a list of recognized Unicode properties to the standard 261output, then exit with zero exit code. All other options are ignored. If both 262-C and any -Lx options are present, whichever is first is recognized. 263</P> 264<P> 265<b>-LS</b> 266List scripts: write a list of recognized Unicode script names to the standard 267output, then exit with zero exit code. All other options are ignored. If both 268-C and any -Lx options are present, whichever is first is recognized. 269</P> 270<P> 271<b>-pattern</b> <i>modifier-list</i> 272Behave as if each pattern line contains the given modifiers. 273</P> 274<P> 275<b>-q</b> 276Do not output the version number of <b>pcre2test</b> at the start of execution. 277</P> 278<P> 279<b>-S</b> <i>size</i> 280On Unix-like systems, set the size of the run-time stack to <i>size</i> 281mebibytes (units of 1024*1024 bytes). 282</P> 283<P> 284<b>-subject</b> <i>modifier-list</i> 285Behave as if each subject line contains the given modifiers. 286</P> 287<P> 288<b>-t</b> 289Run each compile and match many times with a timer, and output the resulting 290times per compile or match. When JIT is used, separate times are given for the 291initial compile and the JIT compile. You can control the number of iterations 292that are used for timing by following <b>-t</b> with a number (as a separate 293item on the command line). For example, "-t 1000" iterates 1000 times. The 294default is to iterate 500,000 times. 295</P> 296<P> 297<b>-tm</b> 298This is like <b>-t</b> except that it times only the matching phase, not the 299compile phase. 300</P> 301<P> 302<b>-T</b> <b>-TM</b> 303These behave like <b>-t</b> and <b>-tm</b>, but in addition, at the end of a run, 304the total times for all compiles and matches are output. 305</P> 306<P> 307<b>-version</b> 308Output the PCRE2 version number and then exit. 309</P> 310<br><a name="SEC5" href="#TOC1">DESCRIPTION</a><br> 311<P> 312If <b>pcre2test</b> is given two filename arguments, it reads from the first and 313writes to the second. If the first name is "-", input is taken from the 314standard input. If <b>pcre2test</b> is given only one argument, it reads from 315that file and writes to stdout. Otherwise, it reads from stdin and writes to 316stdout. 317</P> 318<P> 319When <b>pcre2test</b> is built, a configuration option can specify that it 320should be linked with the <b>libreadline</b> or <b>libedit</b> library. When this 321is done, if the input is from a terminal, it is read using the <b>readline()</b> 322function. This provides line-editing and history facilities. The output from 323the <b>-help</b> option states whether or not <b>readline()</b> will be used. 324</P> 325<P> 326The program handles any number of tests, each of which consists of a set of 327input lines. Each set starts with a regular expression pattern, followed by any 328number of subject lines to be matched against that pattern. In between sets of 329test data, command lines that begin with # may appear. This file format, with 330some restrictions, can also be processed by the <b>perltest.sh</b> script that 331is distributed with PCRE2 as a means of checking that the behaviour of PCRE2 332and Perl is the same. For a specification of <b>perltest.sh</b>, see the 333comments near its beginning. See also the #perltest command below. 334</P> 335<P> 336When the input is a terminal, <b>pcre2test</b> prompts for each line of input, 337using "re>" to prompt for regular expression patterns, and "data>" to prompt 338for subject lines. Command lines starting with # can be entered only in 339response to the "re>" prompt. 340</P> 341<P> 342Each subject line is matched separately and independently. If you want to do 343multi-line matches, you have to use the \n escape sequence (or \r or \r\n, 344etc., depending on the newline setting) in a single line of input to encode the 345newline sequences. There is no limit on the length of subject lines; the input 346buffer is automatically extended if it is too small. There are replication 347features that makes it possible to generate long repetitive pattern or subject 348lines without having to supply them explicitly. 349</P> 350<P> 351An empty line or the end of the file signals the end of the subject lines for a 352test, at which point a new pattern or command line is expected if there is 353still input to be read. 354</P> 355<br><a name="SEC6" href="#TOC1">COMMAND LINES</a><br> 356<P> 357In between sets of test data, a line that begins with # is interpreted as a 358command line. If the first character is followed by white space or an 359exclamation mark, the line is treated as a comment, and ignored. Otherwise, the 360following commands are recognized: 361<pre> 362 #forbid_utf 363</pre> 364Subsequent patterns automatically have the PCRE2_NEVER_UTF and PCRE2_NEVER_UCP 365options set, which locks out the use of the PCRE2_UTF and PCRE2_UCP options and 366the use of (*UTF) and (*UCP) at the start of patterns. This command also forces 367an error if a subsequent pattern contains any occurrences of \P, \p, or \X, 368which are still supported when PCRE2_UTF is not set, but which require Unicode 369property support to be included in the library. 370</P> 371<P> 372This is a trigger guard that is used in test files to ensure that UTF or 373Unicode property tests are not accidentally added to files that are used when 374Unicode support is not included in the library. Setting PCRE2_NEVER_UTF and 375PCRE2_NEVER_UCP as a default can also be obtained by the use of <b>#pattern</b>; 376the difference is that <b>#forbid_utf</b> cannot be unset, and the automatic 377options are not displayed in pattern information, to avoid cluttering up test 378output. 379<pre> 380 #load <filename> 381</pre> 382This command is used to load a set of precompiled patterns from a file, as 383described in the section entitled "Saving and restoring compiled patterns" 384<a href="#saverestore">below.</a> 385<pre> 386 #loadtables <filename> 387</pre> 388This command is used to load a set of binary character tables that can be 389accessed by the tables=3 qualifier. Such tables can be created by the 390<b>pcre2_dftables</b> program with the -b option. 391<pre> 392 #newline_default [<newline-list>] 393</pre> 394When PCRE2 is built, a default newline convention can be specified. This 395determines which characters and/or character pairs are recognized as indicating 396a newline in a pattern or subject string. The default can be overridden when a 397pattern is compiled. The standard test files contain tests of various newline 398conventions, but the majority of the tests expect a single linefeed to be 399recognized as a newline by default. Without special action the tests would fail 400when PCRE2 is compiled with either CR or CRLF as the default newline. 401</P> 402<P> 403The #newline_default command specifies a list of newline types that are 404acceptable as the default. The types must be one of CR, LF, CRLF, ANYCRLF, 405ANY, or NUL (in upper or lower case), for example: 406<pre> 407 #newline_default LF Any anyCRLF 408</pre> 409If the default newline is in the list, this command has no effect. Otherwise, 410except when testing the POSIX API, a <b>newline</b> modifier that specifies the 411first newline convention in the list (LF in the above example) is added to any 412pattern that does not already have a <b>newline</b> modifier. If the newline 413list is empty, the feature is turned off. This command is present in a number 414of the standard test input files. 415</P> 416<P> 417When the POSIX API is being tested there is no way to override the default 418newline convention, though it is possible to set the newline convention from 419within the pattern. A warning is given if the <b>posix</b> or <b>posix_nosub</b> 420modifier is used when <b>#newline_default</b> would set a default for the 421non-POSIX API. 422<pre> 423 #pattern <modifier-list> 424</pre> 425This command sets a default modifier list that applies to all subsequent 426patterns. Modifiers on a pattern can change these settings. 427<pre> 428 #perltest 429</pre> 430This line is used in test files that can also be processed by <b>perltest.sh</b> 431to confirm that Perl gives the same results as PCRE2. Subsequent tests are 432checked for the use of <b>pcre2test</b> features that are incompatible with the 433<b>perltest.sh</b> script. 434</P> 435<P> 436Patterns must use '/' as their delimiter, and only certain modifiers are 437supported. Comment lines, #pattern commands, and #subject commands that set or 438unset "mark" are recognized and acted on. The #perltest, #forbid_utf, and 439#newline_default commands, which are needed in the relevant pcre2test files, 440are silently ignored. All other command lines are ignored, but give a warning 441message. The <b>#perltest</b> command helps detect tests that are accidentally 442put in the wrong file or use the wrong delimiter. For more details of the 443<b>perltest.sh</b> script see the comments it contains. 444<pre> 445 #pop [<modifiers>] 446 #popcopy [<modifiers>] 447</pre> 448These commands are used to manipulate the stack of compiled patterns, as 449described in the section entitled "Saving and restoring compiled patterns" 450<a href="#saverestore">below.</a> 451<pre> 452 #save <filename> 453</pre> 454This command is used to save a set of compiled patterns to a file, as described 455in the section entitled "Saving and restoring compiled patterns" 456<a href="#saverestore">below.</a> 457<pre> 458 #subject <modifier-list> 459</pre> 460This command sets a default modifier list that applies to all subsequent 461subject lines. Modifiers on a subject line can change these settings. 462</P> 463<br><a name="SEC7" href="#TOC1">MODIFIER SYNTAX</a><br> 464<P> 465Modifier lists are used with both pattern and subject lines. Items in a list 466are separated by commas followed by optional white space. Trailing whitespace 467in a modifier list is ignored. Some modifiers may be given for both patterns 468and subject lines, whereas others are valid only for one or the other. Each 469modifier has a long name, for example "anchored", and some of them must be 470followed by an equals sign and a value, for example, "offset=12". Values cannot 471contain comma characters, but may contain spaces. Modifiers that do not take 472values may be preceded by a minus sign to turn off a previous setting. 473</P> 474<P> 475A few of the more common modifiers can also be specified as single letters, for 476example "i" for "caseless". In documentation, following the Perl convention, 477these are written with a slash ("the /i modifier") for clarity. Abbreviated 478modifiers must all be concatenated in the first item of a modifier list. If the 479first item is not recognized as a long modifier name, it is interpreted as a 480sequence of these abbreviations. For example: 481<pre> 482 /abc/ig,newline=cr,jit=3 483</pre> 484This is a pattern line whose modifier list starts with two one-letter modifiers 485(/i and /g). The lower-case abbreviated modifiers are the same as used in Perl. 486</P> 487<br><a name="SEC8" href="#TOC1">PATTERN SYNTAX</a><br> 488<P> 489A pattern line must start with one of the following characters (common symbols, 490excluding pattern meta-characters): 491<pre> 492 / ! " ' ` - = _ : ; , % & @ ~ 493</pre> 494This is interpreted as the pattern's delimiter. A regular expression may be 495continued over several input lines, in which case the newline characters are 496included within it. It is possible to include the delimiter as a literal within 497the pattern by escaping it with a backslash, for example 498<pre> 499 /abc\/def/ 500</pre> 501If you do this, the escape and the delimiter form part of the pattern, but 502since the delimiters are all non-alphanumeric, the inclusion of the backslash 503does not affect the pattern's interpretation. Note, however, that this trick 504does not work within \Q...\E literal bracketing because the backslash will 505itself be interpreted as a literal. If the terminating delimiter is immediately 506followed by a backslash, for example, 507<pre> 508 /abc/\ 509</pre> 510then a backslash is added to the end of the pattern. This is done to provide a 511way of testing the error condition that arises if a pattern finishes with a 512backslash, because 513<pre> 514 /abc\/ 515</pre> 516is interpreted as the first line of a pattern that starts with "abc/", causing 517pcre2test to read the next line as a continuation of the regular expression. 518</P> 519<P> 520A pattern can be followed by a modifier list (details below). 521</P> 522<br><a name="SEC9" href="#TOC1">SUBJECT LINE SYNTAX</a><br> 523<P> 524Before each subject line is passed to <b>pcre2_match()</b>, 525<b>pcre2_dfa_match()</b>, or <b>pcre2_jit_match()</b>, leading and trailing white 526space is removed, and the line is scanned for backslash escapes, unless the 527<b>subject_literal</b> modifier was set for the pattern. The following provide a 528means of encoding non-printing characters in a visible way: 529<pre> 530 \a alarm (BEL, \x07) 531 \b backspace (\x08) 532 \e escape (\x27) 533 \f form feed (\x0c) 534 \n newline (\x0a) 535 \r carriage return (\x0d) 536 \t tab (\x09) 537 \v vertical tab (\x0b) 538 \nnn octal character (up to 3 octal digits); always 539 a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode 540 \o{dd...} octal character (any number of octal digits} 541 \xhh hexadecimal byte (up to 2 hex digits) 542 \x{hh...} hexadecimal character (any number of hex digits) 543</pre> 544The use of \x{hh...} is not dependent on the use of the <b>utf</b> modifier on 545the pattern. It is recognized always. There may be any number of hexadecimal 546digits inside the braces; invalid values provoke error messages. 547</P> 548<P> 549Note that \xhh specifies one byte rather than one character in UTF-8 mode; 550this makes it possible to construct invalid UTF-8 sequences for testing 551purposes. On the other hand, \x{hh} is interpreted as a UTF-8 character in 552UTF-8 mode, generating more than one byte if the value is greater than 127. 553When testing the 8-bit library not in UTF-8 mode, \x{hh} generates one byte 554for values less than 256, and causes an error for greater values. 555</P> 556<P> 557In UTF-16 mode, all 4-digit \x{hhhh} values are accepted. This makes it 558possible to construct invalid UTF-16 sequences for testing purposes. 559</P> 560<P> 561In UTF-32 mode, all 4- to 8-digit \x{...} values are accepted. This makes it 562possible to construct invalid UTF-32 sequences for testing purposes. 563</P> 564<P> 565There is a special backslash sequence that specifies replication of one or more 566characters: 567<pre> 568 \[<characters>]{<count>} 569</pre> 570This makes it possible to test long strings without having to provide them as 571part of the file. For example: 572<pre> 573 \[abc]{4} 574</pre> 575is converted to "abcabcabcabc". This feature does not support nesting. To 576include a closing square bracket in the characters, code it as \x5D. 577</P> 578<P> 579A backslash followed by an equals sign marks the end of the subject string and 580the start of a modifier list. For example: 581<pre> 582 abc\=notbol,notempty 583</pre> 584If the subject string is empty and \= is followed by whitespace, the line is 585treated as a comment line, and is not used for matching. For example: 586<pre> 587 \= This is a comment. 588 abc\= This is an invalid modifier list. 589</pre> 590A backslash followed by any other non-alphanumeric character just escapes that 591character. A backslash followed by anything else causes an error. However, if 592the very last character in the line is a backslash (and there is no modifier 593list), it is ignored. This gives a way of passing an empty line as data, since 594a real empty line terminates the data input. 595</P> 596<P> 597If the <b>subject_literal</b> modifier is set for a pattern, all subject lines 598that follow are treated as literals, with no special treatment of backslashes. 599No replication is possible, and any subject modifiers must be set as defaults 600by a <b>#subject</b> command. 601</P> 602<br><a name="SEC10" href="#TOC1">PATTERN MODIFIERS</a><br> 603<P> 604There are several types of modifier that can appear in pattern lines. Except 605where noted below, they may also be used in <b>#pattern</b> commands. A 606pattern's modifier list can add to or override default modifiers that were set 607by a previous <b>#pattern</b> command. 608<a name="optionmodifiers"></a></P> 609<br><b> 610Setting compilation options 611</b><br> 612<P> 613The following modifiers set options for <b>pcre2_compile()</b>. Most of them set 614bits in the options argument of that function, but those whose names start with 615PCRE2_EXTRA are additional options that are set in the compile context. For the 616main options, there are some single-letter abbreviations that are the same as 617Perl options. There is special handling for /x: if a second x is present, 618PCRE2_EXTENDED is converted into PCRE2_EXTENDED_MORE as in Perl. A third 619appearance adds PCRE2_EXTENDED as well, though this makes no difference to the 620way <b>pcre2_compile()</b> behaves. See 621<a href="pcre2api.html"><b>pcre2api</b></a> 622for a description of the effects of these options. 623<pre> 624 allow_empty_class set PCRE2_ALLOW_EMPTY_CLASS 625 allow_lookaround_bsk set PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK 626 allow_surrogate_escapes set PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES 627 alt_bsux set PCRE2_ALT_BSUX 628 alt_circumflex set PCRE2_ALT_CIRCUMFLEX 629 alt_verbnames set PCRE2_ALT_VERBNAMES 630 anchored set PCRE2_ANCHORED 631 auto_callout set PCRE2_AUTO_CALLOUT 632 bad_escape_is_literal set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL 633 /i caseless set PCRE2_CASELESS 634 dollar_endonly set PCRE2_DOLLAR_ENDONLY 635 /s dotall set PCRE2_DOTALL 636 dupnames set PCRE2_DUPNAMES 637 endanchored set PCRE2_ENDANCHORED 638 escaped_cr_is_lf set PCRE2_EXTRA_ESCAPED_CR_IS_LF 639 /x extended set PCRE2_EXTENDED 640 /xx extended_more set PCRE2_EXTENDED_MORE 641 extra_alt_bsux set PCRE2_EXTRA_ALT_BSUX 642 firstline set PCRE2_FIRSTLINE 643 literal set PCRE2_LITERAL 644 match_line set PCRE2_EXTRA_MATCH_LINE 645 match_invalid_utf set PCRE2_MATCH_INVALID_UTF 646 match_unset_backref set PCRE2_MATCH_UNSET_BACKREF 647 match_word set PCRE2_EXTRA_MATCH_WORD 648 /m multiline set PCRE2_MULTILINE 649 never_backslash_c set PCRE2_NEVER_BACKSLASH_C 650 never_ucp set PCRE2_NEVER_UCP 651 never_utf set PCRE2_NEVER_UTF 652 /n no_auto_capture set PCRE2_NO_AUTO_CAPTURE 653 no_auto_possess set PCRE2_NO_AUTO_POSSESS 654 no_dotstar_anchor set PCRE2_NO_DOTSTAR_ANCHOR 655 no_start_optimize set PCRE2_NO_START_OPTIMIZE 656 no_utf_check set PCRE2_NO_UTF_CHECK 657 ucp set PCRE2_UCP 658 ungreedy set PCRE2_UNGREEDY 659 use_offset_limit set PCRE2_USE_OFFSET_LIMIT 660 utf set PCRE2_UTF 661</pre> 662As well as turning on the PCRE2_UTF option, the <b>utf</b> modifier causes all 663non-printing characters in output strings to be printed using the \x{hh...} 664notation. Otherwise, those less than 0x100 are output in hex without the curly 665brackets. Setting <b>utf</b> in 16-bit or 32-bit mode also causes pattern and 666subject strings to be translated to UTF-16 or UTF-32, respectively, before 667being passed to library functions. 668<a name="controlmodifiers"></a></P> 669<br><b> 670Setting compilation controls 671</b><br> 672<P> 673The following modifiers affect the compilation process or request information 674about the pattern. There are single-letter abbreviations for some that are 675heavily used in the test files. 676<pre> 677 bsr=[anycrlf|unicode] specify \R handling 678 /B bincode show binary code without lengths 679 callout_info show callout information 680 convert=<options> request foreign pattern conversion 681 convert_glob_escape=c set glob escape character 682 convert_glob_separator=c set glob separator character 683 convert_length set convert buffer length 684 debug same as info,fullbincode 685 framesize show matching frame size 686 fullbincode show binary code with lengths 687 /I info show info about compiled pattern 688 hex unquoted characters are hexadecimal 689 jit[=<number>] use JIT 690 jitfast use JIT fast path 691 jitverify verify JIT use 692 locale=<name> use this locale 693 max_pattern_length=<n> set the maximum pattern length 694 memory show memory used 695 newline=<type> set newline type 696 null_context compile with a NULL context 697 parens_nest_limit=<n> set maximum parentheses depth 698 posix use the POSIX API 699 posix_nosub use the POSIX API with REG_NOSUB 700 push push compiled pattern onto the stack 701 pushcopy push a copy onto the stack 702 stackguard=<number> test the stackguard feature 703 subject_literal treat all subject lines as literal 704 tables=[0|1|2|3] select internal tables 705 use_length do not zero-terminate the pattern 706 utf8_input treat input as UTF-8 707</pre> 708The effects of these modifiers are described in the following sections. 709</P> 710<br><b> 711Newline and \R handling 712</b><br> 713<P> 714The <b>bsr</b> modifier specifies what \R in a pattern should match. If it is 715set to "anycrlf", \R matches CR, LF, or CRLF only. If it is set to "unicode", 716\R matches any Unicode newline sequence. The default can be specified when 717PCRE2 is built; if it is not, the default is set to Unicode. 718</P> 719<P> 720The <b>newline</b> modifier specifies which characters are to be interpreted as 721newlines, both in the pattern and in subject lines. The type must be one of CR, 722LF, CRLF, ANYCRLF, ANY, or NUL (in upper or lower case). 723</P> 724<br><b> 725Information about a pattern 726</b><br> 727<P> 728The <b>debug</b> modifier is a shorthand for <b>info,fullbincode</b>, requesting 729all available information. 730</P> 731<P> 732The <b>bincode</b> modifier causes a representation of the compiled code to be 733output after compilation. This information does not contain length and offset 734values, which ensures that the same output is generated for different internal 735link sizes and different code unit widths. By using <b>bincode</b>, the same 736regression tests can be used in different environments. 737</P> 738<P> 739The <b>fullbincode</b> modifier, by contrast, <i>does</i> include length and 740offset values. This is used in a few special tests that run only for specific 741code unit widths and link sizes, and is also useful for one-off tests. 742</P> 743<P> 744The <b>info</b> modifier requests information about the compiled pattern 745(whether it is anchored, has a fixed first character, and so on). The 746information is obtained from the <b>pcre2_pattern_info()</b> function. Here are 747some typical examples: 748<pre> 749 re> /(?i)(^a|^b)/m,info 750 Capture group count = 1 751 Compile options: multiline 752 Overall options: caseless multiline 753 First code unit at start or follows newline 754 Subject length lower bound = 1 755 756 re> /(?i)abc/info 757 Capture group count = 0 758 Compile options: <none> 759 Overall options: caseless 760 First code unit = 'a' (caseless) 761 Last code unit = 'c' (caseless) 762 Subject length lower bound = 3 763</pre> 764"Compile options" are those specified by modifiers; "overall options" have 765added options that are taken or deduced from the pattern. If both sets of 766options are the same, just a single "options" line is output; if there are no 767options, the line is omitted. "First code unit" is where any match must start; 768if there is more than one they are listed as "starting code units". "Last code 769unit" is the last literal code unit that must be present in any match. This is 770not necessarily the last character. These lines are omitted if no starting or 771ending code units are recorded. The subject length line is omitted when 772<b>no_start_optimize</b> is set because the minimum length is not calculated 773when it can never be used. 774</P> 775<P> 776The <b>framesize</b> modifier shows the size, in bytes, of the storage frames 777used by <b>pcre2_match()</b> for handling backtracking. The size depends on the 778number of capturing parentheses in the pattern. 779</P> 780<P> 781The <b>callout_info</b> modifier requests information about all the callouts in 782the pattern. A list of them is output at the end of any other information that 783is requested. For each callout, either its number or string is given, followed 784by the item that follows it in the pattern. 785</P> 786<br><b> 787Passing a NULL context 788</b><br> 789<P> 790Normally, <b>pcre2test</b> passes a context block to <b>pcre2_compile()</b>. If 791the <b>null_context</b> modifier is set, however, NULL is passed. This is for 792testing that <b>pcre2_compile()</b> behaves correctly in this case (it uses 793default values). 794</P> 795<br><b> 796Specifying pattern characters in hexadecimal 797</b><br> 798<P> 799The <b>hex</b> modifier specifies that the characters of the pattern, except for 800substrings enclosed in single or double quotes, are to be interpreted as pairs 801of hexadecimal digits. This feature is provided as a way of creating patterns 802that contain binary zeros and other non-printing characters. White space is 803permitted between pairs of digits. For example, this pattern contains three 804characters: 805<pre> 806 /ab 32 59/hex 807</pre> 808Parts of such a pattern are taken literally if quoted. This pattern contains 809nine characters, only two of which are specified in hexadecimal: 810<pre> 811 /ab "literal" 32/hex 812</pre> 813Either single or double quotes may be used. There is no way of including 814the delimiter within a substring. The <b>hex</b> and <b>expand</b> modifiers are 815mutually exclusive. 816</P> 817<br><b> 818Specifying the pattern's length 819</b><br> 820<P> 821By default, patterns are passed to the compiling functions as zero-terminated 822strings but can be passed by length instead of being zero-terminated. The 823<b>use_length</b> modifier causes this to happen. Using a length happens 824automatically (whether or not <b>use_length</b> is set) when <b>hex</b> is set, 825because patterns specified in hexadecimal may contain binary zeros. 826</P> 827<P> 828If <b>hex</b> or <b>use_length</b> is used with the POSIX wrapper API (see 829<a href="#posixwrapper">"Using the POSIX wrapper API"</a> 830below), the REG_PEND extension is used to pass the pattern's length. 831</P> 832<br><b> 833Specifying wide characters in 16-bit and 32-bit modes 834</b><br> 835<P> 836In 16-bit and 32-bit modes, all input is automatically treated as UTF-8 and 837translated to UTF-16 or UTF-32 when the <b>utf</b> modifier is set. For testing 838the 16-bit and 32-bit libraries in non-UTF mode, the <b>utf8_input</b> modifier 839can be used. It is mutually exclusive with <b>utf</b>. Input lines are 840interpreted as UTF-8 as a means of specifying wide characters. More details are 841given in 842<a href="#inputencoding">"Input encoding"</a> 843above. 844</P> 845<br><b> 846Generating long repetitive patterns 847</b><br> 848<P> 849Some tests use long patterns that are very repetitive. Instead of creating a 850very long input line for such a pattern, you can use a special repetition 851feature, similar to the one described for subject lines above. If the 852<b>expand</b> modifier is present on a pattern, parts of the pattern that have 853the form 854<pre> 855 \[<characters>]{<count>} 856</pre> 857are expanded before the pattern is passed to <b>pcre2_compile()</b>. For 858example, \[AB]{6000} is expanded to "ABAB..." 6000 times. This construction 859cannot be nested. An initial "\[" sequence is recognized only if "]{" followed 860by decimal digits and "}" is found later in the pattern. If not, the characters 861remain in the pattern unaltered. The <b>expand</b> and <b>hex</b> modifiers are 862mutually exclusive. 863</P> 864<P> 865If part of an expanded pattern looks like an expansion, but is really part of 866the actual pattern, unwanted expansion can be avoided by giving two values in 867the quantifier. For example, \[AB]{6000,6000} is not recognized as an 868expansion item. 869</P> 870<P> 871If the <b>info</b> modifier is set on an expanded pattern, the result of the 872expansion is included in the information that is output. 873</P> 874<br><b> 875JIT compilation 876</b><br> 877<P> 878Just-in-time (JIT) compiling is a heavyweight optimization that can greatly 879speed up pattern matching. See the 880<a href="pcre2jit.html"><b>pcre2jit</b></a> 881documentation for details. JIT compiling happens, optionally, after a pattern 882has been successfully compiled into an internal form. The JIT compiler converts 883this to optimized machine code. It needs to know whether the match-time options 884PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT are going to be used, because 885different code is generated for the different cases. See the <b>partial</b> 886modifier in "Subject Modifiers" 887<a href="#subjectmodifiers">below</a> 888for details of how these options are specified for each match attempt. 889</P> 890<P> 891JIT compilation is requested by the <b>jit</b> pattern modifier, which may 892optionally be followed by an equals sign and a number in the range 0 to 7. 893The three bits that make up the number specify which of the three JIT operating 894modes are to be compiled: 895<pre> 896 1 compile JIT code for non-partial matching 897 2 compile JIT code for soft partial matching 898 4 compile JIT code for hard partial matching 899</pre> 900The possible values for the <b>jit</b> modifier are therefore: 901<pre> 902 0 disable JIT 903 1 normal matching only 904 2 soft partial matching only 905 3 normal and soft partial matching 906 4 hard partial matching only 907 6 soft and hard partial matching only 908 7 all three modes 909</pre> 910If no number is given, 7 is assumed. The phrase "partial matching" means a call 911to <b>pcre2_match()</b> with either the PCRE2_PARTIAL_SOFT or the 912PCRE2_PARTIAL_HARD option set. Note that such a call may return a complete 913match; the options enable the possibility of a partial match, but do not 914require it. Note also that if you request JIT compilation only for partial 915matching (for example, jit=2) but do not set the <b>partial</b> modifier on a 916subject line, that match will not use JIT code because none was compiled for 917non-partial matching. 918</P> 919<P> 920If JIT compilation is successful, the compiled JIT code will automatically be 921used when an appropriate type of match is run, except when incompatible 922run-time options are specified. For more details, see the 923<a href="pcre2jit.html"><b>pcre2jit</b></a> 924documentation. See also the <b>jitstack</b> modifier below for a way of 925setting the size of the JIT stack. 926</P> 927<P> 928If the <b>jitfast</b> modifier is specified, matching is done using the JIT 929"fast path" interface, <b>pcre2_jit_match()</b>, which skips some of the sanity 930checks that are done by <b>pcre2_match()</b>, and of course does not work when 931JIT is not supported. If <b>jitfast</b> is specified without <b>jit</b>, jit=7 is 932assumed. 933</P> 934<P> 935If the <b>jitverify</b> modifier is specified, information about the compiled 936pattern shows whether JIT compilation was or was not successful. If 937<b>jitverify</b> is specified without <b>jit</b>, jit=7 is assumed. If JIT 938compilation is successful when <b>jitverify</b> is set, the text "(JIT)" is 939added to the first output line after a match or non match when JIT-compiled 940code was actually used in the match. 941</P> 942<br><b> 943Setting a locale 944</b><br> 945<P> 946The <b>locale</b> modifier must specify the name of a locale, for example: 947<pre> 948 /pattern/locale=fr_FR 949</pre> 950The given locale is set, <b>pcre2_maketables()</b> is called to build a set of 951character tables for the locale, and this is then passed to 952<b>pcre2_compile()</b> when compiling the regular expression. The same tables 953are used when matching the following subject lines. The <b>locale</b> modifier 954applies only to the pattern on which it appears, but can be given in a 955<b>#pattern</b> command if a default is needed. Setting a locale and alternate 956character tables are mutually exclusive. 957</P> 958<br><b> 959Showing pattern memory 960</b><br> 961<P> 962The <b>memory</b> modifier causes the size in bytes of the memory used to hold 963the compiled pattern to be output. This does not include the size of the 964<b>pcre2_code</b> block; it is just the actual compiled data. If the pattern is 965subsequently passed to the JIT compiler, the size of the JIT compiled code is 966also output. Here is an example: 967<pre> 968 re> /a(b)c/jit,memory 969 Memory allocation (code space): 21 970 Memory allocation (JIT code): 1910 971 972</PRE> 973</P> 974<br><b> 975Limiting nested parentheses 976</b><br> 977<P> 978The <b>parens_nest_limit</b> modifier sets a limit on the depth of nested 979parentheses in a pattern. Breaching the limit causes a compilation error. 980The default for the library is set when PCRE2 is built, but <b>pcre2test</b> 981sets its own default of 220, which is required for running the standard test 982suite. 983</P> 984<br><b> 985Limiting the pattern length 986</b><br> 987<P> 988The <b>max_pattern_length</b> modifier sets a limit, in code units, to the 989length of pattern that <b>pcre2_compile()</b> will accept. Breaching the limit 990causes a compilation error. The default is the largest number a PCRE2_SIZE 991variable can hold (essentially unlimited). 992<a name="posixwrapper"></a></P> 993<br><b> 994Using the POSIX wrapper API 995</b><br> 996<P> 997The <b>posix</b> and <b>posix_nosub</b> modifiers cause <b>pcre2test</b> to call 998PCRE2 via the POSIX wrapper API rather than its native API. When 999<b>posix_nosub</b> is used, the POSIX option REG_NOSUB is passed to 1000<b>regcomp()</b>. The POSIX wrapper supports only the 8-bit library. Note that 1001it does not imply POSIX matching semantics; for more detail see the 1002<a href="pcre2posix.html"><b>pcre2posix</b></a> 1003documentation. The following pattern modifiers set options for the 1004<b>regcomp()</b> function: 1005<pre> 1006 caseless REG_ICASE 1007 multiline REG_NEWLINE 1008 dotall REG_DOTALL ) 1009 ungreedy REG_UNGREEDY ) These options are not part of 1010 ucp REG_UCP ) the POSIX standard 1011 utf REG_UTF8 ) 1012</pre> 1013The <b>regerror_buffsize</b> modifier specifies a size for the error buffer that 1014is passed to <b>regerror()</b> in the event of a compilation error. For example: 1015<pre> 1016 /abc/posix,regerror_buffsize=20 1017</pre> 1018This provides a means of testing the behaviour of <b>regerror()</b> when the 1019buffer is too small for the error message. If this modifier has not been set, a 1020large buffer is used. 1021</P> 1022<P> 1023The <b>aftertext</b> and <b>allaftertext</b> subject modifiers work as described 1024below. All other modifiers are either ignored, with a warning message, or cause 1025an error. 1026</P> 1027<P> 1028The pattern is passed to <b>regcomp()</b> as a zero-terminated string by 1029default, but if the <b>use_length</b> or <b>hex</b> modifiers are set, the 1030REG_PEND extension is used to pass it by length. 1031</P> 1032<br><b> 1033Testing the stack guard feature 1034</b><br> 1035<P> 1036The <b>stackguard</b> modifier is used to test the use of 1037<b>pcre2_set_compile_recursion_guard()</b>, a function that is provided to 1038enable stack availability to be checked during compilation (see the 1039<a href="pcre2api.html"><b>pcre2api</b></a> 1040documentation for details). If the number specified by the modifier is greater 1041than zero, <b>pcre2_set_compile_recursion_guard()</b> is called to set up 1042callback from <b>pcre2_compile()</b> to a local function. The argument it 1043receives is the current nesting parenthesis depth; if this is greater than the 1044value given by the modifier, non-zero is returned, causing the compilation to 1045be aborted. 1046</P> 1047<br><b> 1048Using alternative character tables 1049</b><br> 1050<P> 1051The value specified for the <b>tables</b> modifier must be one of the digits 0, 10521, 2, or 3. It causes a specific set of built-in character tables to be passed 1053to <b>pcre2_compile()</b>. This is used in the PCRE2 tests to check behaviour 1054with different character tables. The digit specifies the tables as follows: 1055<pre> 1056 0 do not pass any special character tables 1057 1 the default ASCII tables, as distributed in 1058 pcre2_chartables.c.dist 1059 2 a set of tables defining ISO 8859 characters 1060 3 a set of tables loaded by the #loadtables command 1061</pre> 1062In tables 2, some characters whose codes are greater than 128 are identified as 1063letters, digits, spaces, etc. Tables 3 can be used only after a 1064<b>#loadtables</b> command has loaded them from a binary file. Setting alternate 1065character tables and a locale are mutually exclusive. 1066</P> 1067<br><b> 1068Setting certain match controls 1069</b><br> 1070<P> 1071The following modifiers are really subject modifiers, and are described under 1072"Subject Modifiers" below. However, they may be included in a pattern's 1073modifier list, in which case they are applied to every subject line that is 1074processed with that pattern. These modifiers do not affect the compilation 1075process. 1076<pre> 1077 aftertext show text after match 1078 allaftertext show text after captures 1079 allcaptures show all captures 1080 allvector show the entire ovector 1081 allusedtext show all consulted text 1082 altglobal alternative global matching 1083 /g global global matching 1084 jitstack=<n> set size of JIT stack 1085 mark show mark values 1086 replace=<string> specify a replacement string 1087 startchar show starting character when relevant 1088 substitute_callout use substitution callouts 1089 substitute_extended use PCRE2_SUBSTITUTE_EXTENDED 1090 substitute_literal use PCRE2_SUBSTITUTE_LITERAL 1091 substitute_matched use PCRE2_SUBSTITUTE_MATCHED 1092 substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH 1093 substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY 1094 substitute_skip=<n> skip substitution <n> 1095 substitute_stop=<n> skip substitution <n> and following 1096 substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET 1097 substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY 1098</pre> 1099These modifiers may not appear in a <b>#pattern</b> command. If you want them as 1100defaults, set them in a <b>#subject</b> command. 1101</P> 1102<br><b> 1103Specifying literal subject lines 1104</b><br> 1105<P> 1106If the <b>subject_literal</b> modifier is present on a pattern, all the subject 1107lines that it matches are taken as literal strings, with no interpretation of 1108backslashes. It is not possible to set subject modifiers on such lines, but any 1109that are set as defaults by a <b>#subject</b> command are recognized. 1110</P> 1111<br><b> 1112Saving a compiled pattern 1113</b><br> 1114<P> 1115When a pattern with the <b>push</b> modifier is successfully compiled, it is 1116pushed onto a stack of compiled patterns, and <b>pcre2test</b> expects the next 1117line to contain a new pattern (or a command) instead of a subject line. This 1118facility is used when saving compiled patterns to a file, as described in the 1119section entitled "Saving and restoring compiled patterns" 1120<a href="#saverestore">below.</a> 1121If <b>pushcopy</b> is used instead of <b>push</b>, a copy of the compiled 1122pattern is stacked, leaving the original as current, ready to match the 1123following input lines. This provides a way of testing the 1124<b>pcre2_code_copy()</b> function. 1125The <b>push</b> and <b>pushcopy </b> modifiers are incompatible with compilation 1126modifiers such as <b>global</b> that act at match time. Any that are specified 1127are ignored (for the stacked copy), with a warning message, except for 1128<b>replace</b>, which causes an error. Note that <b>jitverify</b>, which is 1129allowed, does not carry through to any subsequent matching that uses a stacked 1130pattern. 1131</P> 1132<br><b> 1133Testing foreign pattern conversion 1134</b><br> 1135<P> 1136The experimental foreign pattern conversion functions in PCRE2 can be tested by 1137setting the <b>convert</b> modifier. Its argument is a colon-separated list of 1138options, which set the equivalent option for the <b>pcre2_pattern_convert()</b> 1139function: 1140<pre> 1141 glob PCRE2_CONVERT_GLOB 1142 glob_no_starstar PCRE2_CONVERT_GLOB_NO_STARSTAR 1143 glob_no_wild_separator PCRE2_CONVERT_GLOB_NO_WILD_SEPARATOR 1144 posix_basic PCRE2_CONVERT_POSIX_BASIC 1145 posix_extended PCRE2_CONVERT_POSIX_EXTENDED 1146 unset Unset all options 1147</pre> 1148The "unset" value is useful for turning off a default that has been set by a 1149<b>#pattern</b> command. When one of these options is set, the input pattern is 1150passed to <b>pcre2_pattern_convert()</b>. If the conversion is successful, the 1151result is reflected in the output and then passed to <b>pcre2_compile()</b>. The 1152normal <b>utf</b> and <b>no_utf_check</b> options, if set, cause the 1153PCRE2_CONVERT_UTF and PCRE2_CONVERT_NO_UTF_CHECK options to be passed to 1154<b>pcre2_pattern_convert()</b>. 1155</P> 1156<P> 1157By default, the conversion function is allowed to allocate a buffer for its 1158output. However, if the <b>convert_length</b> modifier is set to a value greater 1159than zero, <b>pcre2test</b> passes a buffer of the given length. This makes it 1160possible to test the length check. 1161</P> 1162<P> 1163The <b>convert_glob_escape</b> and <b>convert_glob_separator</b> modifiers can be 1164used to specify the escape and separator characters for glob processing, 1165overriding the defaults, which are operating-system dependent. 1166<a name="subjectmodifiers"></a></P> 1167<br><a name="SEC11" href="#TOC1">SUBJECT MODIFIERS</a><br> 1168<P> 1169The modifiers that can appear in subject lines and the <b>#subject</b> 1170command are of two types. 1171</P> 1172<br><b> 1173Setting match options 1174</b><br> 1175<P> 1176The following modifiers set options for <b>pcre2_match()</b> or 1177<b>pcre2_dfa_match()</b>. See 1178<a href="pcreapi.html"><b>pcreapi</b></a> 1179for a description of their effects. 1180<pre> 1181 anchored set PCRE2_ANCHORED 1182 endanchored set PCRE2_ENDANCHORED 1183 dfa_restart set PCRE2_DFA_RESTART 1184 dfa_shortest set PCRE2_DFA_SHORTEST 1185 no_jit set PCRE2_NO_JIT 1186 no_utf_check set PCRE2_NO_UTF_CHECK 1187 notbol set PCRE2_NOTBOL 1188 notempty set PCRE2_NOTEMPTY 1189 notempty_atstart set PCRE2_NOTEMPTY_ATSTART 1190 noteol set PCRE2_NOTEOL 1191 partial_hard (or ph) set PCRE2_PARTIAL_HARD 1192 partial_soft (or ps) set PCRE2_PARTIAL_SOFT 1193</pre> 1194The partial matching modifiers are provided with abbreviations because they 1195appear frequently in tests. 1196</P> 1197<P> 1198If the <b>posix</b> or <b>posix_nosub</b> modifier was present on the pattern, 1199causing the POSIX wrapper API to be used, the only option-setting modifiers 1200that have any effect are <b>notbol</b>, <b>notempty</b>, and <b>noteol</b>, 1201causing REG_NOTBOL, REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to 1202<b>regexec()</b>. The other modifiers are ignored, with a warning message. 1203</P> 1204<P> 1205There is one additional modifier that can be used with the POSIX wrapper. It is 1206ignored (with a warning) if used for non-POSIX matching. 1207<pre> 1208 posix_startend=<n>[:<m>] 1209</pre> 1210This causes the subject string to be passed to <b>regexec()</b> using the 1211REG_STARTEND option, which uses offsets to specify which part of the string is 1212searched. If only one number is given, the end offset is passed as the end of 1213the subject string. For more detail of REG_STARTEND, see the 1214<a href="pcre2posix.html"><b>pcre2posix</b></a> 1215documentation. If the subject string contains binary zeros (coded as escapes 1216such as \x{00} because <b>pcre2test</b> does not support actual binary zeros in 1217its input), you must use <b>posix_startend</b> to specify its length. 1218</P> 1219<br><b> 1220Setting match controls 1221</b><br> 1222<P> 1223The following modifiers affect the matching process or request additional 1224information. Some of them may also be specified on a pattern line (see above), 1225in which case they apply to every subject line that is matched against that 1226pattern, but can be overridden by modifiers on the subject. 1227<pre> 1228 aftertext show text after match 1229 allaftertext show text after captures 1230 allcaptures show all captures 1231 allvector show the entire ovector 1232 allusedtext show all consulted text (non-JIT only) 1233 altglobal alternative global matching 1234 callout_capture show captures at callout time 1235 callout_data=<n> set a value to pass via callouts 1236 callout_error=<n>[:<m>] control callout error 1237 callout_extra show extra callout information 1238 callout_fail=<n>[:<m>] control callout failure 1239 callout_no_where do not show position of a callout 1240 callout_none do not supply a callout function 1241 copy=<number or name> copy captured substring 1242 depth_limit=<n> set a depth limit 1243 dfa use <b>pcre2_dfa_match()</b> 1244 find_limits find heap, match and depth limits 1245 find_limits_noheap find match and depth limits 1246 get=<number or name> extract captured substring 1247 getall extract all captured substrings 1248 /g global global matching 1249 heap_limit=<n> set a limit on heap memory (Kbytes) 1250 jitstack=<n> set size of JIT stack 1251 mark show mark values 1252 match_limit=<n> set a match limit 1253 memory show heap memory usage 1254 null_context match with a NULL context 1255 null_replacement substitute with NULL replacement 1256 null_subject match with NULL subject 1257 offset=<n> set starting offset 1258 offset_limit=<n> set offset limit 1259 ovector=<n> set size of output vector 1260 recursion_limit=<n> obsolete synonym for depth_limit 1261 replace=<string> specify a replacement string 1262 startchar show startchar when relevant 1263 startoffset=<n> same as offset=<n> 1264 substitute_callout use substitution callouts 1265 substitute_extedded use PCRE2_SUBSTITUTE_EXTENDED 1266 substitute_literal use PCRE2_SUBSTITUTE_LITERAL 1267 substitute_matched use PCRE2_SUBSTITUTE_MATCHED 1268 substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH 1269 substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY 1270 substitute_skip=<n> skip substitution number n 1271 substitute_stop=<n> skip substitution number n and greater 1272 substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET 1273 substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY 1274 zero_terminate pass the subject as zero-terminated 1275</pre> 1276The effects of these modifiers are described in the following sections. When 1277matching via the POSIX wrapper API, the <b>aftertext</b>, <b>allaftertext</b>, 1278and <b>ovector</b> subject modifiers work as described below. All other 1279modifiers are either ignored, with a warning message, or cause an error. 1280</P> 1281<br><b> 1282Showing more text 1283</b><br> 1284<P> 1285The <b>aftertext</b> modifier requests that as well as outputting the part of 1286the subject string that matched the entire pattern, <b>pcre2test</b> should in 1287addition output the remainder of the subject string. This is useful for tests 1288where the subject contains multiple copies of the same substring. The 1289<b>allaftertext</b> modifier requests the same action for captured substrings as 1290well as the main matched substring. In each case the remainder is output on the 1291following line with a plus character following the capture number. 1292</P> 1293<P> 1294The <b>allusedtext</b> modifier requests that all the text that was consulted 1295during a successful pattern match by the interpreter should be shown, for both 1296full and partial matches. This feature is not supported for JIT matching, and 1297if requested with JIT it is ignored (with a warning message). Setting this 1298modifier affects the output if there is a lookbehind at the start of a match, 1299or, for a complete match, a lookahead at the end, or if \K is used in the 1300pattern. Characters that precede or follow the start and end of the actual 1301match are indicated in the output by '<' or '>' characters underneath them. 1302Here is an example: 1303<pre> 1304 re> /(?<=pqr)abc(?=xyz)/ 1305 data> 123pqrabcxyz456\=allusedtext 1306 0: pqrabcxyz 1307 <<< >>> 1308 data> 123pqrabcxy\=ph,allusedtext 1309 Partial match: pqrabcxy 1310 <<< 1311</pre> 1312The first, complete match shows that the matched string is "abc", with the 1313preceding and following strings "pqr" and "xyz" having been consulted during 1314the match (when processing the assertions). The partial match can indicate only 1315the preceding string. 1316</P> 1317<P> 1318The <b>startchar</b> modifier requests that the starting character for the match 1319be indicated, if it is different to the start of the matched string. The only 1320time when this occurs is when \K has been processed as part of the match. In 1321this situation, the output for the matched string is displayed from the 1322starting character instead of from the match point, with circumflex characters 1323under the earlier characters. For example: 1324<pre> 1325 re> /abc\Kxyz/ 1326 data> abcxyz\=startchar 1327 0: abcxyz 1328 ^^^ 1329</pre> 1330Unlike <b>allusedtext</b>, the <b>startchar</b> modifier can be used with JIT. 1331However, these two modifiers are mutually exclusive. 1332</P> 1333<br><b> 1334Showing the value of all capture groups 1335</b><br> 1336<P> 1337The <b>allcaptures</b> modifier requests that the values of all potential 1338captured parentheses be output after a match. By default, only those up to the 1339highest one actually used in the match are output (corresponding to the return 1340code from <b>pcre2_match()</b>). Groups that did not take part in the match 1341are output as "<unset>". This modifier is not relevant for DFA matching (which 1342does no capturing) and does not apply when <b>replace</b> is specified; it is 1343ignored, with a warning message, if present. 1344</P> 1345<br><b> 1346Showing the entire ovector, for all outcomes 1347</b><br> 1348<P> 1349The <b>allvector</b> modifier requests that the entire ovector be shown, 1350whatever the outcome of the match. Compare <b>allcaptures</b>, which shows only 1351up to the maximum number of capture groups for the pattern, and then only for a 1352successful complete non-DFA match. This modifier, which acts after any match 1353result, and also for DFA matching, provides a means of checking that there are 1354no unexpected modifications to ovector fields. Before each match attempt, the 1355ovector is filled with a special value, and if this is found in both elements 1356of a capturing pair, "<unchanged>" is output. After a successful match, this 1357applies to all groups after the maximum capture group for the pattern. In other 1358cases it applies to the entire ovector. After a partial match, the first two 1359elements are the only ones that should be set. After a DFA match, the amount of 1360ovector that is used depends on the number of matches that were found. 1361</P> 1362<br><b> 1363Testing pattern callouts 1364</b><br> 1365<P> 1366A callout function is supplied when <b>pcre2test</b> calls the library matching 1367functions, unless <b>callout_none</b> is specified. Its behaviour can be 1368controlled by various modifiers listed above whose names begin with 1369<b>callout_</b>. Details are given in the section entitled "Callouts" 1370<a href="#callouts">below.</a> 1371Testing callouts from <b>pcre2_substitute()</b> is described separately in 1372"Testing the substitution function" 1373<a href="#substitution">below.</a> 1374</P> 1375<br><b> 1376Finding all matches in a string 1377</b><br> 1378<P> 1379Searching for all possible matches within a subject can be requested by the 1380<b>global</b> or <b>altglobal</b> modifier. After finding a match, the matching 1381function is called again to search the remainder of the subject. The difference 1382between <b>global</b> and <b>altglobal</b> is that the former uses the 1383<i>start_offset</i> argument to <b>pcre2_match()</b> or <b>pcre2_dfa_match()</b> 1384to start searching at a new point within the entire string (which is what Perl 1385does), whereas the latter passes over a shortened subject. This makes a 1386difference to the matching process if the pattern begins with a lookbehind 1387assertion (including \b or \B). 1388</P> 1389<P> 1390If an empty string is matched, the next match is done with the 1391PCRE2_NOTEMPTY_ATSTART and PCRE2_ANCHORED flags set, in order to search for 1392another, non-empty, match at the same point in the subject. If this match 1393fails, the start offset is advanced, and the normal match is retried. This 1394imitates the way Perl handles such cases when using the <b>/g</b> modifier or 1395the <b>split()</b> function. Normally, the start offset is advanced by one 1396character, but if the newline convention recognizes CRLF as a newline, and the 1397current character is CR followed by LF, an advance of two characters occurs. 1398</P> 1399<br><b> 1400Testing substring extraction functions 1401</b><br> 1402<P> 1403The <b>copy</b> and <b>get</b> modifiers can be used to test the 1404<b>pcre2_substring_copy_xxx()</b> and <b>pcre2_substring_get_xxx()</b> functions. 1405They can be given more than once, and each can specify a capture group name or 1406number, for example: 1407<pre> 1408 abcd\=copy=1,copy=3,get=G1 1409</pre> 1410If the <b>#subject</b> command is used to set default copy and/or get lists, 1411these can be unset by specifying a negative number to cancel all numbered 1412groups and an empty name to cancel all named groups. 1413</P> 1414<P> 1415The <b>getall</b> modifier tests <b>pcre2_substring_list_get()</b>, which 1416extracts all captured substrings. 1417</P> 1418<P> 1419If the subject line is successfully matched, the substrings extracted by the 1420convenience functions are output with C, G, or L after the string number 1421instead of a colon. This is in addition to the normal full list. The string 1422length (that is, the return from the extraction function) is given in 1423parentheses after each substring, followed by the name when the extraction was 1424by name. 1425<a name="substitution"></a></P> 1426<br><b> 1427Testing the substitution function 1428</b><br> 1429<P> 1430If the <b>replace</b> modifier is set, the <b>pcre2_substitute()</b> function is 1431called instead of one of the matching functions (or after one call of 1432<b>pcre2_match()</b> in the case of PCRE2_SUBSTITUTE_MATCHED). Note that 1433replacement strings cannot contain commas, because a comma signifies the end of 1434a modifier. This is not thought to be an issue in a test program. 1435</P> 1436<P> 1437Specifying a completely empty replacement string disables this modifier. 1438However, it is possible to specify an empty replacement by providing a buffer 1439length, as described below, for an otherwise empty replacement. 1440</P> 1441<P> 1442Unlike subject strings, <b>pcre2test</b> does not process replacement strings 1443for escape sequences. In UTF mode, a replacement string is checked to see if it 1444is a valid UTF-8 string. If so, it is correctly converted to a UTF string of 1445the appropriate code unit width. If it is not a valid UTF-8 string, the 1446individual code units are copied directly. This provides a means of passing an 1447invalid UTF-8 string for testing purposes. 1448</P> 1449<P> 1450The following modifiers set options (in additional to the normal match options) 1451for <b>pcre2_substitute()</b>: 1452<pre> 1453 global PCRE2_SUBSTITUTE_GLOBAL 1454 substitute_extended PCRE2_SUBSTITUTE_EXTENDED 1455 substitute_literal PCRE2_SUBSTITUTE_LITERAL 1456 substitute_matched PCRE2_SUBSTITUTE_MATCHED 1457 substitute_overflow_length PCRE2_SUBSTITUTE_OVERFLOW_LENGTH 1458 substitute_replacement_only PCRE2_SUBSTITUTE_REPLACEMENT_ONLY 1459 substitute_unknown_unset PCRE2_SUBSTITUTE_UNKNOWN_UNSET 1460 substitute_unset_empty PCRE2_SUBSTITUTE_UNSET_EMPTY 1461</pre> 1462See the 1463<a href="pcre2api.html"><b>pcre2api</b></a> 1464documentation for details of these options. 1465</P> 1466<P> 1467After a successful substitution, the modified string is output, preceded by the 1468number of replacements. This may be zero if there were no matches. Here is a 1469simple example of a substitution test: 1470<pre> 1471 /abc/replace=xxx 1472 =abc=abc= 1473 1: =xxx=abc= 1474 =abc=abc=\=global 1475 2: =xxx=xxx= 1476</pre> 1477Subject and replacement strings should be kept relatively short (fewer than 256 1478characters) for substitution tests, as fixed-size buffers are used. To make it 1479easy to test for buffer overflow, if the replacement string starts with a 1480number in square brackets, that number is passed to <b>pcre2_substitute()</b> as 1481the size of the output buffer, with the replacement string starting at the next 1482character. Here is an example that tests the edge case: 1483<pre> 1484 /abc/ 1485 123abc123\=replace=[10]XYZ 1486 1: 123XYZ123 1487 123abc123\=replace=[9]XYZ 1488 Failed: error -47: no more memory 1489</pre> 1490The default action of <b>pcre2_substitute()</b> is to return 1491PCRE2_ERROR_NOMEMORY when the output buffer is too small. However, if the 1492PCRE2_SUBSTITUTE_OVERFLOW_LENGTH option is set (by using the 1493<b>substitute_overflow_length</b> modifier), <b>pcre2_substitute()</b> continues 1494to go through the motions of matching and substituting (but not doing any 1495callouts), in order to compute the size of buffer that is required. When this 1496happens, <b>pcre2test</b> shows the required buffer length (which includes space 1497for the trailing zero) as part of the error message. For example: 1498<pre> 1499 /abc/substitute_overflow_length 1500 123abc123\=replace=[9]XYZ 1501 Failed: error -47: no more memory: 10 code units are needed 1502</pre> 1503A replacement string is ignored with POSIX and DFA matching. Specifying partial 1504matching provokes an error return ("bad option value") from 1505<b>pcre2_substitute()</b>. 1506</P> 1507<br><b> 1508Testing substitute callouts 1509</b><br> 1510<P> 1511If the <b>substitute_callout</b> modifier is set, a substitution callout 1512function is set up. The <b>null_context</b> modifier must not be set, because 1513the address of the callout function is passed in a match context. When the 1514callout function is called (after each substitution), details of the the input 1515and output strings are output. For example: 1516<pre> 1517 /abc/g,replace=<$0>,substitute_callout 1518 abcdefabcpqr 1519 1(1) Old 0 3 "abc" New 0 5 "<abc>" 1520 2(1) Old 6 9 "abc" New 8 13 "<abc>" 1521 2: <abc>def<abc>pqr 1522</pre> 1523The first number on each callout line is the count of matches. The 1524parenthesized number is the number of pairs that are set in the ovector (that 1525is, one more than the number of capturing groups that were set). Then are 1526listed the offsets of the old substring, its contents, and the same for the 1527replacement. 1528</P> 1529<P> 1530By default, the substitution callout function returns zero, which accepts the 1531replacement and causes matching to continue if /g was used. Two further 1532modifiers can be used to test other return values. If <b>substitute_skip</b> is 1533set to a value greater than zero the callout function returns +1 for the match 1534of that number, and similarly <b>substitute_stop</b> returns -1. These cause the 1535replacement to be rejected, and -1 causes no further matching to take place. If 1536either of them are set, <b>substitute_callout</b> is assumed. For example: 1537<pre> 1538 /abc/g,replace=<$0>,substitute_skip=1 1539 abcdefabcpqr 1540 1(1) Old 0 3 "abc" New 0 5 "<abc> SKIPPED" 1541 2(1) Old 6 9 "abc" New 6 11 "<abc>" 1542 2: abcdef<abc>pqr 1543 abcdefabcpqr\=substitute_stop=1 1544 1(1) Old 0 3 "abc" New 0 5 "<abc> STOPPED" 1545 1: abcdefabcpqr 1546</pre> 1547If both are set for the same number, stop takes precedence. Only a single skip 1548or stop is supported, which is sufficient for testing that the feature works. 1549</P> 1550<br><b> 1551Setting the JIT stack size 1552</b><br> 1553<P> 1554The <b>jitstack</b> modifier provides a way of setting the maximum stack size 1555that is used by the just-in-time optimization code. It is ignored if JIT 1556optimization is not being used. The value is a number of kibibytes (units of 15571024 bytes). Setting zero reverts to the default of 32KiB. Providing a stack 1558that is larger than the default is necessary only for very complicated 1559patterns. If <b>jitstack</b> is set non-zero on a subject line it overrides any 1560value that was set on the pattern. 1561</P> 1562<br><b> 1563Setting heap, match, and depth limits 1564</b><br> 1565<P> 1566The <b>heap_limit</b>, <b>match_limit</b>, and <b>depth_limit</b> modifiers set 1567the appropriate limits in the match context. These values are ignored when the 1568<b>find_limits</b> or <b>find_limits_noheap</b> modifier is specified. 1569</P> 1570<br><b> 1571Finding minimum limits 1572</b><br> 1573<P> 1574If the <b>find_limits</b> modifier is present on a subject line, <b>pcre2test</b> 1575calls the relevant matching function several times, setting different values in 1576the match context via <b>pcre2_set_heap_limit()</b>, 1577<b>pcre2_set_match_limit()</b>, or <b>pcre2_set_depth_limit()</b> until it finds 1578the smallest value for each parameter that allows the match to complete without 1579a "limit exceeded" error. The match itself may succeed or fail. An alternative 1580modifier, <b>find_limits_noheap</b>, omits the heap limit. This is used in the 1581standard tests, because the minimum heap limit varies between systems. If JIT 1582is being used, only the match limit is relevant, and the other two are 1583automatically omitted. 1584</P> 1585<P> 1586When using this modifier, the pattern should not contain any limit settings 1587such as (*LIMIT_MATCH=...) within it. If such a setting is present and is 1588lower than the minimum matching value, the minimum value cannot be found 1589because <b>pcre2_set_match_limit()</b> etc. are only able to reduce the value of 1590an in-pattern limit; they cannot increase it. 1591</P> 1592<P> 1593For non-DFA matching, the minimum <i>depth_limit</i> number is a measure of how 1594much nested backtracking happens (that is, how deeply the pattern's tree is 1595searched). In the case of DFA matching, <i>depth_limit</i> controls the depth of 1596recursive calls of the internal function that is used for handling pattern 1597recursion, lookaround assertions, and atomic groups. 1598</P> 1599<P> 1600For non-DFA matching, the <i>match_limit</i> number is a measure of the amount 1601of backtracking that takes place, and learning the minimum value can be 1602instructive. For most simple matches, the number is quite small, but for 1603patterns with very large numbers of matching possibilities, it can become large 1604very quickly with increasing length of subject string. In the case of DFA 1605matching, <i>match_limit</i> controls the total number of calls, both recursive 1606and non-recursive, to the internal matching function, thus controlling the 1607overall amount of computing resource that is used. 1608</P> 1609<P> 1610For both kinds of matching, the <i>heap_limit</i> number, which is in kibibytes 1611(units of 1024 bytes), limits the amount of heap memory used for matching. 1612</P> 1613<br><b> 1614Showing MARK names 1615</b><br> 1616<P> 1617The <b>mark</b> modifier causes the names from backtracking control verbs that 1618are returned from calls to <b>pcre2_match()</b> to be displayed. If a mark is 1619returned for a match, non-match, or partial match, <b>pcre2test</b> shows it. 1620For a match, it is on a line by itself, tagged with "MK:". Otherwise, it 1621is added to the non-match message. 1622</P> 1623<br><b> 1624Showing memory usage 1625</b><br> 1626<P> 1627The <b>memory</b> modifier causes <b>pcre2test</b> to log the sizes of all heap 1628memory allocation and freeing calls that occur during a call to 1629<b>pcre2_match()</b> or <b>pcre2_dfa_match()</b>. In the latter case, heap memory 1630is used only when a match requires more internal workspace that the default 1631allocation on the stack, so in many cases there will be no output. No heap 1632memory is allocated during matching with JIT. For this modifier to work, the 1633<b>null_context</b> modifier must not be set on both the pattern and the 1634subject, though it can be set on one or the other. 1635</P> 1636<br><b> 1637Setting a starting offset 1638</b><br> 1639<P> 1640The <b>offset</b> modifier sets an offset in the subject string at which 1641matching starts. Its value is a number of code units, not characters. 1642</P> 1643<br><b> 1644Setting an offset limit 1645</b><br> 1646<P> 1647The <b>offset_limit</b> modifier sets a limit for unanchored matches. If a match 1648cannot be found starting at or before this offset in the subject, a "no match" 1649return is given. The data value is a number of code units, not characters. When 1650this modifier is used, the <b>use_offset_limit</b> modifier must have been set 1651for the pattern; if not, an error is generated. 1652</P> 1653<br><b> 1654Setting the size of the output vector 1655</b><br> 1656<P> 1657The <b>ovector</b> modifier applies only to the subject line in which it 1658appears, though of course it can also be used to set a default in a 1659<b>#subject</b> command. It specifies the number of pairs of offsets that are 1660available for storing matching information. The default is 15. 1661</P> 1662<P> 1663A value of zero is useful when testing the POSIX API because it causes 1664<b>regexec()</b> to be called with a NULL capture vector. When not testing the 1665POSIX API, a value of zero is used to cause 1666<b>pcre2_match_data_create_from_pattern()</b> to be called, in order to create a 1667match block of exactly the right size for the pattern. (It is not possible to 1668create a match block with a zero-length ovector; there is always at least one 1669pair of offsets.) 1670</P> 1671<br><b> 1672Passing the subject as zero-terminated 1673</b><br> 1674<P> 1675By default, the subject string is passed to a native API matching function with 1676its correct length. In order to test the facility for passing a zero-terminated 1677string, the <b>zero_terminate</b> modifier is provided. It causes the length to 1678be passed as PCRE2_ZERO_TERMINATED. When matching via the POSIX interface, 1679this modifier is ignored, with a warning. 1680</P> 1681<P> 1682When testing <b>pcre2_substitute()</b>, this modifier also has the effect of 1683passing the replacement string as zero-terminated. 1684</P> 1685<br><b> 1686Passing a NULL context, subject, or replacement 1687</b><br> 1688<P> 1689Normally, <b>pcre2test</b> passes a context block to <b>pcre2_match()</b>, 1690<b>pcre2_dfa_match()</b>, <b>pcre2_jit_match()</b> or <b>pcre2_substitute()</b>. 1691If the <b>null_context</b> modifier is set, however, NULL is passed. This is for 1692testing that the matching and substitution functions behave correctly in this 1693case (they use default values). This modifier cannot be used with the 1694<b>find_limits</b>, <b>find_limits_noheap</b>, or <b>substitute_callout</b> 1695modifiers. 1696</P> 1697<P> 1698Similarly, for testing purposes, if the <b>null_subject</b> or 1699<b>null_replacement</b> modifier is set, the subject or replacement string 1700pointers are passed as NULL, respectively, to the relevant functions. 1701</P> 1702<br><a name="SEC12" href="#TOC1">THE ALTERNATIVE MATCHING FUNCTION</a><br> 1703<P> 1704By default, <b>pcre2test</b> uses the standard PCRE2 matching function, 1705<b>pcre2_match()</b> to match each subject line. PCRE2 also supports an 1706alternative matching function, <b>pcre2_dfa_match()</b>, which operates in a 1707different way, and has some restrictions. The differences between the two 1708functions are described in the 1709<a href="pcre2matching.html"><b>pcre2matching</b></a> 1710documentation. 1711</P> 1712<P> 1713If the <b>dfa</b> modifier is set, the alternative matching function is used. 1714This function finds all possible matches at a given point in the subject. If, 1715however, the <b>dfa_shortest</b> modifier is set, processing stops after the 1716first match is found. This is always the shortest possible match. 1717</P> 1718<br><a name="SEC13" href="#TOC1">DEFAULT OUTPUT FROM pcre2test</a><br> 1719<P> 1720This section describes the output when the normal matching function, 1721<b>pcre2_match()</b>, is being used. 1722</P> 1723<P> 1724When a match succeeds, <b>pcre2test</b> outputs the list of captured substrings, 1725starting with number 0 for the string that matched the whole pattern. 1726Otherwise, it outputs "No match" when the return is PCRE2_ERROR_NOMATCH, or 1727"Partial match:" followed by the partially matching substring when the 1728return is PCRE2_ERROR_PARTIAL. (Note that this is the 1729entire substring that was inspected during the partial match; it may include 1730characters before the actual match start if a lookbehind assertion, \K, \b, 1731or \B was involved.) 1732</P> 1733<P> 1734For any other return, <b>pcre2test</b> outputs the PCRE2 negative error number 1735and a short descriptive phrase. If the error is a failed UTF string check, the 1736code unit offset of the start of the failing character is also output. Here is 1737an example of an interactive <b>pcre2test</b> run. 1738<pre> 1739 $ pcre2test 1740 PCRE2 version 10.22 2016-07-29 1741 1742 re> /^abc(\d+)/ 1743 data> abc123 1744 0: abc123 1745 1: 123 1746 data> xyz 1747 No match 1748</pre> 1749Unset capturing substrings that are not followed by one that is set are not 1750shown by <b>pcre2test</b> unless the <b>allcaptures</b> modifier is specified. In 1751the following example, there are two capturing substrings, but when the first 1752data line is matched, the second, unset substring is not shown. An "internal" 1753unset substring is shown as "<unset>", as for the second data line. 1754<pre> 1755 re> /(a)|(b)/ 1756 data> a 1757 0: a 1758 1: a 1759 data> b 1760 0: b 1761 1: <unset> 1762 2: b 1763</pre> 1764If the strings contain any non-printing characters, they are output as \xhh 1765escapes if the value is less than 256 and UTF mode is not set. Otherwise they 1766are output as \x{hh...} escapes. See below for the definition of non-printing 1767characters. If the <b>aftertext</b> modifier is set, the output for substring 17680 is followed by the the rest of the subject string, identified by "0+" like 1769this: 1770<pre> 1771 re> /cat/aftertext 1772 data> cataract 1773 0: cat 1774 0+ aract 1775</pre> 1776If global matching is requested, the results of successive matching attempts 1777are output in sequence, like this: 1778<pre> 1779 re> /\Bi(\w\w)/g 1780 data> Mississippi 1781 0: iss 1782 1: ss 1783 0: iss 1784 1: ss 1785 0: ipp 1786 1: pp 1787</pre> 1788"No match" is output only if the first match attempt fails. Here is an example 1789of a failure message (the offset 4 that is specified by the <b>offset</b> 1790modifier is past the end of the subject string): 1791<pre> 1792 re> /xyz/ 1793 data> xyz\=offset=4 1794 Error -24 (bad offset value) 1795</PRE> 1796</P> 1797<P> 1798Note that whereas patterns can be continued over several lines (a plain ">" 1799prompt is used for continuations), subject lines may not. However newlines can 1800be included in a subject by means of the \n escape (or \r, \r\n, etc., 1801depending on the newline sequence setting). 1802</P> 1803<br><a name="SEC14" href="#TOC1">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a><br> 1804<P> 1805When the alternative matching function, <b>pcre2_dfa_match()</b>, is used, the 1806output consists of a list of all the matches that start at the first point in 1807the subject where there is at least one match. For example: 1808<pre> 1809 re> /(tang|tangerine|tan)/ 1810 data> yellow tangerine\=dfa 1811 0: tangerine 1812 1: tang 1813 2: tan 1814</pre> 1815Using the normal matching function on this data finds only "tang". The 1816longest matching string is always given first (and numbered zero). After a 1817PCRE2_ERROR_PARTIAL return, the output is "Partial match:", followed by the 1818partially matching substring. Note that this is the entire substring that was 1819inspected during the partial match; it may include characters before the actual 1820match start if a lookbehind assertion, \b, or \B was involved. (\K is not 1821supported for DFA matching.) 1822</P> 1823<P> 1824If global matching is requested, the search for further matches resumes 1825at the end of the longest match. For example: 1826<pre> 1827 re> /(tang|tangerine|tan)/g 1828 data> yellow tangerine and tangy sultana\=dfa 1829 0: tangerine 1830 1: tang 1831 2: tan 1832 0: tang 1833 1: tan 1834 0: tan 1835</pre> 1836The alternative matching function does not support substring capture, so the 1837modifiers that are concerned with captured substrings are not relevant. 1838</P> 1839<br><a name="SEC15" href="#TOC1">RESTARTING AFTER A PARTIAL MATCH</a><br> 1840<P> 1841When the alternative matching function has given the PCRE2_ERROR_PARTIAL 1842return, indicating that the subject partially matched the pattern, you can 1843restart the match with additional subject data by means of the 1844<b>dfa_restart</b> modifier. For example: 1845<pre> 1846 re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 1847 data> 23ja\=ps,dfa 1848 Partial match: 23ja 1849 data> n05\=dfa,dfa_restart 1850 0: n05 1851</pre> 1852For further information about partial matching, see the 1853<a href="pcre2partial.html"><b>pcre2partial</b></a> 1854documentation. 1855<a name="callouts"></a></P> 1856<br><a name="SEC16" href="#TOC1">CALLOUTS</a><br> 1857<P> 1858If the pattern contains any callout requests, <b>pcre2test</b>'s callout 1859function is called during matching unless <b>callout_none</b> is specified. This 1860works with both matching functions, and with JIT, though there are some 1861differences in behaviour. The output for callouts with numerical arguments and 1862those with string arguments is slightly different. 1863</P> 1864<br><b> 1865Callouts with numerical arguments 1866</b><br> 1867<P> 1868By default, the callout function displays the callout number, the start and 1869current positions in the subject text at the callout time, and the next pattern 1870item to be tested. For example: 1871<pre> 1872 --->pqrabcdef 1873 0 ^ ^ \d 1874</pre> 1875This output indicates that callout number 0 occurred for a match attempt 1876starting at the fourth character of the subject string, when the pointer was at 1877the seventh character, and when the next pattern item was \d. Just 1878one circumflex is output if the start and current positions are the same, or if 1879the current position precedes the start position, which can happen if the 1880callout is in a lookbehind assertion. 1881</P> 1882<P> 1883Callouts numbered 255 are assumed to be automatic callouts, inserted as a 1884result of the <b>auto_callout</b> pattern modifier. In this case, instead of 1885showing the callout number, the offset in the pattern, preceded by a plus, is 1886output. For example: 1887<pre> 1888 re> /\d?[A-E]\*/auto_callout 1889 data> E* 1890 --->E* 1891 +0 ^ \d? 1892 +3 ^ [A-E] 1893 +8 ^^ \* 1894 +10 ^ ^ 1895 0: E* 1896</pre> 1897If a pattern contains (*MARK) items, an additional line is output whenever 1898a change of latest mark is passed to the callout function. For example: 1899<pre> 1900 re> /a(*MARK:X)bc/auto_callout 1901 data> abc 1902 --->abc 1903 +0 ^ a 1904 +1 ^^ (*MARK:X) 1905 +10 ^^ b 1906 Latest Mark: X 1907 +11 ^ ^ c 1908 +12 ^ ^ 1909 0: abc 1910</pre> 1911The mark changes between matching "a" and "b", but stays the same for the rest 1912of the match, so nothing more is output. If, as a result of backtracking, the 1913mark reverts to being unset, the text "<unset>" is output. 1914</P> 1915<br><b> 1916Callouts with string arguments 1917</b><br> 1918<P> 1919The output for a callout with a string argument is similar, except that instead 1920of outputting a callout number before the position indicators, the callout 1921string and its offset in the pattern string are output before the reflection of 1922the subject string, and the subject string is reflected for each callout. For 1923example: 1924<pre> 1925 re> /^ab(?C'first')cd(?C"second")ef/ 1926 data> abcdefg 1927 Callout (7): 'first' 1928 --->abcdefg 1929 ^ ^ c 1930 Callout (20): "second" 1931 --->abcdefg 1932 ^ ^ e 1933 0: abcdef 1934 1935</PRE> 1936</P> 1937<br><b> 1938Callout modifiers 1939</b><br> 1940<P> 1941The callout function in <b>pcre2test</b> returns zero (carry on matching) by 1942default, but you can use a <b>callout_fail</b> modifier in a subject line to 1943change this and other parameters of the callout (see below). 1944</P> 1945<P> 1946If the <b>callout_capture</b> modifier is set, the current captured groups are 1947output when a callout occurs. This is useful only for non-DFA matching, as 1948<b>pcre2_dfa_match()</b> does not support capturing, so no captures are ever 1949shown. 1950</P> 1951<P> 1952The normal callout output, showing the callout number or pattern offset (as 1953described above) is suppressed if the <b>callout_no_where</b> modifier is set. 1954</P> 1955<P> 1956When using the interpretive matching function <b>pcre2_match()</b> without JIT, 1957setting the <b>callout_extra</b> modifier causes additional output from 1958<b>pcre2test</b>'s callout function to be generated. For the first callout in a 1959match attempt at a new starting position in the subject, "New match attempt" is 1960output. If there has been a backtrack since the last callout (or start of 1961matching if this is the first callout), "Backtrack" is output, followed by "No 1962other matching paths" if the backtrack ended the previous match attempt. For 1963example: 1964<pre> 1965 re> /(a+)b/auto_callout,no_start_optimize,no_auto_possess 1966 data> aac\=callout_extra 1967 New match attempt 1968 --->aac 1969 +0 ^ ( 1970 +1 ^ a+ 1971 +3 ^ ^ ) 1972 +4 ^ ^ b 1973 Backtrack 1974 --->aac 1975 +3 ^^ ) 1976 +4 ^^ b 1977 Backtrack 1978 No other matching paths 1979 New match attempt 1980 --->aac 1981 +0 ^ ( 1982 +1 ^ a+ 1983 +3 ^^ ) 1984 +4 ^^ b 1985 Backtrack 1986 No other matching paths 1987 New match attempt 1988 --->aac 1989 +0 ^ ( 1990 +1 ^ a+ 1991 Backtrack 1992 No other matching paths 1993 New match attempt 1994 --->aac 1995 +0 ^ ( 1996 +1 ^ a+ 1997 No match 1998</pre> 1999Notice that various optimizations must be turned off if you want all possible 2000matching paths to be scanned. If <b>no_start_optimize</b> is not used, there is 2001an immediate "no match", without any callouts, because the starting 2002optimization fails to find "b" in the subject, which it knows must be present 2003for any match. If <b>no_auto_possess</b> is not used, the "a+" item is turned 2004into "a++", which reduces the number of backtracks. 2005</P> 2006<P> 2007The <b>callout_extra</b> modifier has no effect if used with the DFA matching 2008function, or with JIT. 2009</P> 2010<br><b> 2011Return values from callouts 2012</b><br> 2013<P> 2014The default return from the callout function is zero, which allows matching to 2015continue. The <b>callout_fail</b> modifier can be given one or two numbers. If 2016there is only one number, 1 is returned instead of 0 (causing matching to 2017backtrack) when a callout of that number is reached. If two numbers (<n>:<m>) 2018are given, 1 is returned when callout <n> is reached and there have been at 2019least <m> callouts. The <b>callout_error</b> modifier is similar, except that 2020PCRE2_ERROR_CALLOUT is returned, causing the entire matching process to be 2021aborted. If both these modifiers are set for the same callout number, 2022<b>callout_error</b> takes precedence. Note that callouts with string arguments 2023are always given the number zero. 2024</P> 2025<P> 2026The <b>callout_data</b> modifier can be given an unsigned or a negative number. 2027This is set as the "user data" that is passed to the matching function, and 2028passed back when the callout function is invoked. Any value other than zero is 2029used as a return from <b>pcre2test</b>'s callout function. 2030</P> 2031<P> 2032Inserting callouts can be helpful when using <b>pcre2test</b> to check 2033complicated regular expressions. For further information about callouts, see 2034the 2035<a href="pcre2callout.html"><b>pcre2callout</b></a> 2036documentation. 2037</P> 2038<br><a name="SEC17" href="#TOC1">NON-PRINTING CHARACTERS</a><br> 2039<P> 2040When <b>pcre2test</b> is outputting text in the compiled version of a pattern, 2041bytes other than 32-126 are always treated as non-printing characters and are 2042therefore shown as hex escapes. 2043</P> 2044<P> 2045When <b>pcre2test</b> is outputting text that is a matched part of a subject 2046string, it behaves in the same way, unless a different locale has been set for 2047the pattern (using the <b>locale</b> modifier). In this case, the 2048<b>isprint()</b> function is used to distinguish printing and non-printing 2049characters. 2050<a name="saverestore"></a></P> 2051<br><a name="SEC18" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br> 2052<P> 2053It is possible to save compiled patterns on disc or elsewhere, and reload them 2054later, subject to a number of restrictions. JIT data cannot be saved. The host 2055on which the patterns are reloaded must be running the same version of PCRE2, 2056with the same code unit width, and must also have the same endianness, pointer 2057width and PCRE2_SIZE type. Before compiled patterns can be saved they must be 2058serialized, that is, converted to a stream of bytes. A single byte stream may 2059contain any number of compiled patterns, but they must all use the same 2060character tables. A single copy of the tables is included in the byte stream 2061(its size is 1088 bytes). 2062</P> 2063<P> 2064The functions whose names begin with <b>pcre2_serialize_</b> are used 2065for serializing and de-serializing. They are described in the 2066<a href="pcre2serialize.html"><b>pcre2serialize</b></a> 2067documentation. In this section we describe the features of <b>pcre2test</b> that 2068can be used to test these functions. 2069</P> 2070<P> 2071Note that "serialization" in PCRE2 does not convert compiled patterns to an 2072abstract format like Java or .NET. It just makes a reloadable byte code stream. 2073Hence the restrictions on reloading mentioned above. 2074</P> 2075<P> 2076In <b>pcre2test</b>, when a pattern with <b>push</b> modifier is successfully 2077compiled, it is pushed onto a stack of compiled patterns, and <b>pcre2test</b> 2078expects the next line to contain a new pattern (or command) instead of a 2079subject line. By contrast, the <b>pushcopy</b> modifier causes a copy of the 2080compiled pattern to be stacked, leaving the original available for immediate 2081matching. By using <b>push</b> and/or <b>pushcopy</b>, a number of patterns can 2082be compiled and retained. These modifiers are incompatible with <b>posix</b>, 2083and control modifiers that act at match time are ignored (with a message) for 2084the stacked patterns. The <b>jitverify</b> modifier applies only at compile 2085time. 2086</P> 2087<P> 2088The command 2089<pre> 2090 #save <filename> 2091</pre> 2092causes all the stacked patterns to be serialized and the result written to the 2093named file. Afterwards, all the stacked patterns are freed. The command 2094<pre> 2095 #load <filename> 2096</pre> 2097reads the data in the file, and then arranges for it to be de-serialized, with 2098the resulting compiled patterns added to the pattern stack. The pattern on the 2099top of the stack can be retrieved by the #pop command, which must be followed 2100by lines of subjects that are to be matched with the pattern, terminated as 2101usual by an empty line or end of file. This command may be followed by a 2102modifier list containing only 2103<a href="#controlmodifiers">control modifiers</a> 2104that act after a pattern has been compiled. In particular, <b>hex</b>, 2105<b>posix</b>, <b>posix_nosub</b>, <b>push</b>, and <b>pushcopy</b> are not allowed, 2106nor are any 2107<a href="#optionmodifiers">option-setting modifiers.</a> 2108The JIT modifiers are, however permitted. Here is an example that saves and 2109reloads two patterns. 2110<pre> 2111 /abc/push 2112 /xyz/push 2113 #save tempfile 2114 #load tempfile 2115 #pop info 2116 xyz 2117 2118 #pop jit,bincode 2119 abc 2120</pre> 2121If <b>jitverify</b> is used with #pop, it does not automatically imply 2122<b>jit</b>, which is different behaviour from when it is used on a pattern. 2123</P> 2124<P> 2125The #popcopy command is analogous to the <b>pushcopy</b> modifier in that it 2126makes current a copy of the topmost stack pattern, leaving the original still 2127on the stack. 2128</P> 2129<br><a name="SEC19" href="#TOC1">SEE ALSO</a><br> 2130<P> 2131<b>pcre2</b>(3), <b>pcre2api</b>(3), <b>pcre2callout</b>(3), 2132<b>pcre2jit</b>, <b>pcre2matching</b>(3), <b>pcre2partial</b>(d), 2133<b>pcre2pattern</b>(3), <b>pcre2serialize</b>(3). 2134</P> 2135<br><a name="SEC20" href="#TOC1">AUTHOR</a><br> 2136<P> 2137Philip Hazel 2138<br> 2139Retired from University Computing Service 2140<br> 2141Cambridge, England. 2142<br> 2143</P> 2144<br><a name="SEC21" href="#TOC1">REVISION</a><br> 2145<P> 2146Last updated: 27 July 2022 2147<br> 2148Copyright © 1997-2022 University of Cambridge. 2149<br> 2150<p> 2151Return to the <a href="index.html">PCRE2 index page</a>. 2152</p> 2153