1<html> 2<head> 3<title>pcre2test specification</title> 4</head> 5<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6<h1>pcre2test man page</h1> 7<p> 8Return to the <a href="index.html">PCRE2 index page</a>. 9</p> 10<p> 11This page is part of the PCRE2 HTML documentation. It was generated 12automatically from the original man page. If there is any nonsense in it, 13please consult the man page, in case the conversion went wrong. 14<br> 15<ul> 16<li><a name="TOC1" href="#SEC1">SYNOPSIS</a> 17<li><a name="TOC2" href="#SEC2">PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a> 18<li><a name="TOC3" href="#SEC3">INPUT ENCODING</a> 19<li><a name="TOC4" href="#SEC4">COMMAND LINE OPTIONS</a> 20<li><a name="TOC5" href="#SEC5">DESCRIPTION</a> 21<li><a name="TOC6" href="#SEC6">COMMAND LINES</a> 22<li><a name="TOC7" href="#SEC7">MODIFIER SYNTAX</a> 23<li><a name="TOC8" href="#SEC8">PATTERN SYNTAX</a> 24<li><a name="TOC9" href="#SEC9">SUBJECT LINE SYNTAX</a> 25<li><a name="TOC10" href="#SEC10">PATTERN MODIFIERS</a> 26<li><a name="TOC11" href="#SEC11">SUBJECT MODIFIERS</a> 27<li><a name="TOC12" href="#SEC12">THE ALTERNATIVE MATCHING FUNCTION</a> 28<li><a name="TOC13" href="#SEC13">DEFAULT OUTPUT FROM pcre2test</a> 29<li><a name="TOC14" href="#SEC14">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a> 30<li><a name="TOC15" href="#SEC15">RESTARTING AFTER A PARTIAL MATCH</a> 31<li><a name="TOC16" href="#SEC16">CALLOUTS</a> 32<li><a name="TOC17" href="#SEC17">NON-PRINTING CHARACTERS</a> 33<li><a name="TOC18" href="#SEC18">SAVING AND RESTORING COMPILED PATTERNS</a> 34<li><a name="TOC19" href="#SEC19">SEE ALSO</a> 35<li><a name="TOC20" href="#SEC20">AUTHOR</a> 36<li><a name="TOC21" href="#SEC21">REVISION</a> 37</ul> 38<br><a name="SEC1" href="#TOC1">SYNOPSIS</a><br> 39<P> 40<b>pcre2test [options] [input file [output file]]</b> 41<br> 42<br> 43<b>pcre2test</b> is a test program for the PCRE2 regular expression libraries, 44but it can also be used for experimenting with regular expressions. This 45document describes the features of the test program; for details of the regular 46expressions themselves, see the 47<a href="pcre2pattern.html"><b>pcre2pattern</b></a> 48documentation. For details of the PCRE2 library function calls and their 49options, see the 50<a href="pcre2api.html"><b>pcre2api</b></a> 51documentation. 52</P> 53<P> 54The input for <b>pcre2test</b> is a sequence of regular expression patterns and 55subject strings to be matched. There are also command lines for setting 56defaults and controlling some special actions. The output shows the result of 57each match attempt. Modifiers on external or internal command lines, the 58patterns, and the subject lines specify PCRE2 function options, control how the 59subject is processed, and what output is produced. 60</P> 61<P> 62There are many obscure modifiers, some of which are specifically designed for 63use in conjunction with the test script and data files that are distributed as 64part of PCRE2. All the modifiers are documented here, some without much 65justification, but many of them are unlikely to be of use except when testing 66the libraries. 67</P> 68<br><a name="SEC2" href="#TOC1">PCRE2's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a><br> 69<P> 70Different versions of the PCRE2 library can be built to support character 71strings that are encoded in 8-bit, 16-bit, or 32-bit code units. One, two, or 72all three of these libraries may be simultaneously installed. The 73<b>pcre2test</b> program can be used to test all the libraries. However, its own 74input and output are always in 8-bit format. When testing the 16-bit or 32-bit 75libraries, patterns and subject strings are converted to 16-bit or 32-bit 76format before being passed to the library functions. Results are converted back 77to 8-bit code units for output. 78</P> 79<P> 80In the rest of this document, the names of library functions and structures 81are given in generic form, for example, <b>pcre2_compile()</b>. The actual 82names used in the libraries have a suffix _8, _16, or _32, as appropriate. 83<a name="inputencoding"></a></P> 84<br><a name="SEC3" href="#TOC1">INPUT ENCODING</a><br> 85<P> 86Input to <b>pcre2test</b> is processed line by line, either by calling the C 87library's <b>fgets()</b> function, or via the <b>libreadline</b> or <b>libedit</b> 88library. In some Windows environments character 26 (hex 1A) causes an immediate 89end of file, and no further data is read, so this character should be avoided 90unless you really want that action. 91</P> 92<P> 93The input is processed using using C's string functions, so must not 94contain binary zeros, even though in Unix-like environments, <b>fgets()</b> 95treats any bytes other than newline as data characters. An error is generated 96if a binary zero is encountered. By default subject lines are processed for 97backslash escapes, which makes it possible to include any data value in strings 98that are passed to the library for matching. For patterns, there is a facility 99for specifying some or all of the 8-bit input characters as hexadecimal pairs, 100which makes it possible to include binary zeros. 101</P> 102<br><b> 103Input for the 16-bit and 32-bit libraries 104</b><br> 105<P> 106When testing the 16-bit or 32-bit libraries, there is a need to be able to 107generate character code points greater than 255 in the strings that are passed 108to the library. For subject lines, backslash escapes can be used. In addition, 109when the <b>utf</b> modifier (see 110<a href="#optionmodifiers">"Setting compilation options"</a> 111below) is set, the pattern and any following subject lines are interpreted as 112UTF-8 strings and translated to UTF-16 or UTF-32 as appropriate. 113</P> 114<P> 115For non-UTF testing of wide characters, the <b>utf8_input</b> modifier can be 116used. This is mutually exclusive with <b>utf</b>, and is allowed only in 16-bit 117or 32-bit mode. It causes the pattern and following subject lines to be treated 118as UTF-8 according to the original definition (RFC 2279), which allows for 119character values up to 0x7fffffff. Each character is placed in one 16-bit or 12032-bit code unit (in the 16-bit case, values greater than 0xffff cause an error 121to occur). 122</P> 123<P> 124UTF-8 (in its original definition) is not capable of encoding values greater 125than 0x7fffffff, but such values can be handled by the 32-bit library. When 126testing this library in non-UTF mode with <b>utf8_input</b> set, if any 127character is preceded by the byte 0xff (which is an invalid byte in UTF-8) 1280x80000000 is added to the character's value. This is the only way of passing 129such code points in a pattern string. For subject strings, using an escape 130sequence is preferable. 131</P> 132<br><a name="SEC4" href="#TOC1">COMMAND LINE OPTIONS</a><br> 133<P> 134<b>-8</b> 135If the 8-bit library has been built, this option causes it to be used (this is 136the default). If the 8-bit library has not been built, this option causes an 137error. 138</P> 139<P> 140<b>-16</b> 141If the 16-bit library has been built, this option causes it to be used. If only 142the 16-bit library has been built, this is the default. If the 16-bit library 143has not been built, this option causes an error. 144</P> 145<P> 146<b>-32</b> 147If the 32-bit library has been built, this option causes it to be used. If only 148the 32-bit library has been built, this is the default. If the 32-bit library 149has not been built, this option causes an error. 150</P> 151<P> 152<b>-ac</b> 153Behave as if each pattern has the <b>auto_callout</b> modifier, that is, insert 154automatic callouts into every pattern that is compiled. 155</P> 156<P> 157<b>-AC</b> 158As for <b>-ac</b>, but in addition behave as if each subject line has the 159<b>callout_extra</b> modifier, that is, show additional information from 160callouts. 161</P> 162<P> 163<b>-b</b> 164Behave as if each pattern has the <b>fullbincode</b> modifier; the full 165internal binary form of the pattern is output after compilation. 166</P> 167<P> 168<b>-C</b> 169Output the version number of the PCRE2 library, and all available information 170about the optional features that are included, and then exit with zero exit 171code. All other options are ignored. If both -C and -LM are present, whichever 172is first is recognized. 173</P> 174<P> 175<b>-C</b> <i>option</i> 176Output information about a specific build-time option, then exit. This 177functionality is intended for use in scripts such as <b>RunTest</b>. The 178following options output the value and set the exit code as indicated: 179<pre> 180 ebcdic-nl the code for LF (= NL) in an EBCDIC environment: 181 0x15 or 0x25 182 0 if used in an ASCII environment 183 exit code is always 0 184 linksize the configured internal link size (2, 3, or 4) 185 exit code is set to the link size 186 newline the default newline setting: 187 CR, LF, CRLF, ANYCRLF, ANY, or NUL 188 exit code is always 0 189 bsr the default setting for what \R matches: 190 ANYCRLF or ANY 191 exit code is always 0 192</pre> 193The following options output 1 for true or 0 for false, and set the exit code 194to the same value: 195<pre> 196 backslash-C \C is supported (not locked out) 197 ebcdic compiled for an EBCDIC environment 198 jit just-in-time support is available 199 pcre2-16 the 16-bit library was built 200 pcre2-32 the 32-bit library was built 201 pcre2-8 the 8-bit library was built 202 unicode Unicode support is available 203</pre> 204If an unknown option is given, an error message is output; the exit code is 0. 205</P> 206<P> 207<b>-d</b> 208Behave as if each pattern has the <b>debug</b> modifier; the internal 209form and information about the compiled pattern is output after compilation; 210<b>-d</b> is equivalent to <b>-b -i</b>. 211</P> 212<P> 213<b>-dfa</b> 214Behave as if each subject line has the <b>dfa</b> modifier; matching is done 215using the <b>pcre2_dfa_match()</b> function instead of the default 216<b>pcre2_match()</b>. 217</P> 218<P> 219<b>-error</b> <i>number[,number,...]</i> 220Call <b>pcre2_get_error_message()</b> for each of the error numbers in the 221comma-separated list, display the resulting messages on the standard output, 222then exit with zero exit code. The numbers may be positive or negative. This is 223a convenience facility for PCRE2 maintainers. 224</P> 225<P> 226<b>-help</b> 227Output a brief summary these options and then exit. 228</P> 229<P> 230<b>-i</b> 231Behave as if each pattern has the <b>info</b> modifier; information about the 232compiled pattern is given after compilation. 233</P> 234<P> 235<b>-jit</b> 236Behave as if each pattern line has the <b>jit</b> modifier; after successful 237compilation, each pattern is passed to the just-in-time compiler, if available. 238</P> 239<P> 240<b>-jitfast</b> 241Behave as if each pattern line has the <b>jitfast</b> modifier; after 242successful compilation, each pattern is passed to the just-in-time compiler, if 243available, and each subject line is passed directly to the JIT matcher via its 244"fast path". 245</P> 246<P> 247<b>-jitverify</b> 248Behave as if each pattern line has the <b>jitverify</b> modifier; after 249successful compilation, each pattern is passed to the just-in-time compiler, if 250available, and the use of JIT for matching is verified. 251</P> 252<P> 253<b>-LM</b> 254List modifiers: write a list of available pattern and subject modifiers to the 255standard output, then exit with zero exit code. All other options are ignored. 256If both -C and any -Lx options are present, whichever is first is recognized. 257</P> 258<P> 259<b>-LP</b> 260List properties: write a list of recognized Unicode properties to the standard 261output, then exit with zero exit code. All other options are ignored. If both 262-C and any -Lx options are present, whichever is first is recognized. 263</P> 264<P> 265<b>-LS</b> 266List scripts: write a list of recogized Unicode script names to the standard 267output, then exit with zero exit code. All other options are ignored. If both 268-C and any -Lx options are present, whichever is first is recognized. 269</P> 270<P> 271<b>-pattern</b> <i>modifier-list</i> 272Behave as if each pattern line contains the given modifiers. 273</P> 274<P> 275<b>-q</b> 276Do not output the version number of <b>pcre2test</b> at the start of execution. 277</P> 278<P> 279<b>-S</b> <i>size</i> 280On Unix-like systems, set the size of the run-time stack to <i>size</i> 281mebibytes (units of 1024*1024 bytes). 282</P> 283<P> 284<b>-subject</b> <i>modifier-list</i> 285Behave as if each subject line contains the given modifiers. 286</P> 287<P> 288<b>-t</b> 289Run each compile and match many times with a timer, and output the resulting 290times per compile or match. When JIT is used, separate times are given for the 291initial compile and the JIT compile. You can control the number of iterations 292that are used for timing by following <b>-t</b> with a number (as a separate 293item on the command line). For example, "-t 1000" iterates 1000 times. The 294default is to iterate 500,000 times. 295</P> 296<P> 297<b>-tm</b> 298This is like <b>-t</b> except that it times only the matching phase, not the 299compile phase. 300</P> 301<P> 302<b>-T</b> <b>-TM</b> 303These behave like <b>-t</b> and <b>-tm</b>, but in addition, at the end of a run, 304the total times for all compiles and matches are output. 305</P> 306<P> 307<b>-version</b> 308Output the PCRE2 version number and then exit. 309</P> 310<br><a name="SEC5" href="#TOC1">DESCRIPTION</a><br> 311<P> 312If <b>pcre2test</b> is given two filename arguments, it reads from the first and 313writes to the second. If the first name is "-", input is taken from the 314standard input. If <b>pcre2test</b> is given only one argument, it reads from 315that file and writes to stdout. Otherwise, it reads from stdin and writes to 316stdout. 317</P> 318<P> 319When <b>pcre2test</b> is built, a configuration option can specify that it 320should be linked with the <b>libreadline</b> or <b>libedit</b> library. When this 321is done, if the input is from a terminal, it is read using the <b>readline()</b> 322function. This provides line-editing and history facilities. The output from 323the <b>-help</b> option states whether or not <b>readline()</b> will be used. 324</P> 325<P> 326The program handles any number of tests, each of which consists of a set of 327input lines. Each set starts with a regular expression pattern, followed by any 328number of subject lines to be matched against that pattern. In between sets of 329test data, command lines that begin with # may appear. This file format, with 330some restrictions, can also be processed by the <b>perltest.sh</b> script that 331is distributed with PCRE2 as a means of checking that the behaviour of PCRE2 332and Perl is the same. For a specification of <b>perltest.sh</b>, see the 333comments near its beginning. See also the #perltest command below. 334</P> 335<P> 336When the input is a terminal, <b>pcre2test</b> prompts for each line of input, 337using "re>" to prompt for regular expression patterns, and "data>" to prompt 338for subject lines. Command lines starting with # can be entered only in 339response to the "re>" prompt. 340</P> 341<P> 342Each subject line is matched separately and independently. If you want to do 343multi-line matches, you have to use the \n escape sequence (or \r or \r\n, 344etc., depending on the newline setting) in a single line of input to encode the 345newline sequences. There is no limit on the length of subject lines; the input 346buffer is automatically extended if it is too small. There are replication 347features that makes it possible to generate long repetitive pattern or subject 348lines without having to supply them explicitly. 349</P> 350<P> 351An empty line or the end of the file signals the end of the subject lines for a 352test, at which point a new pattern or command line is expected if there is 353still input to be read. 354</P> 355<br><a name="SEC6" href="#TOC1">COMMAND LINES</a><br> 356<P> 357In between sets of test data, a line that begins with # is interpreted as a 358command line. If the first character is followed by white space or an 359exclamation mark, the line is treated as a comment, and ignored. Otherwise, the 360following commands are recognized: 361<pre> 362 #forbid_utf 363</pre> 364Subsequent patterns automatically have the PCRE2_NEVER_UTF and PCRE2_NEVER_UCP 365options set, which locks out the use of the PCRE2_UTF and PCRE2_UCP options and 366the use of (*UTF) and (*UCP) at the start of patterns. This command also forces 367an error if a subsequent pattern contains any occurrences of \P, \p, or \X, 368which are still supported when PCRE2_UTF is not set, but which require Unicode 369property support to be included in the library. 370</P> 371<P> 372This is a trigger guard that is used in test files to ensure that UTF or 373Unicode property tests are not accidentally added to files that are used when 374Unicode support is not included in the library. Setting PCRE2_NEVER_UTF and 375PCRE2_NEVER_UCP as a default can also be obtained by the use of <b>#pattern</b>; 376the difference is that <b>#forbid_utf</b> cannot be unset, and the automatic 377options are not displayed in pattern information, to avoid cluttering up test 378output. 379<pre> 380 #load <filename> 381</pre> 382This command is used to load a set of precompiled patterns from a file, as 383described in the section entitled "Saving and restoring compiled patterns" 384<a href="#saverestore">below.</a> 385<pre> 386 #loadtables <filename> 387</pre> 388This command is used to load a set of binary character tables that can be 389accessed by the tables=3 qualifier. Such tables can be created by the 390<b>pcre2_dftables</b> program with the -b option. 391<pre> 392 #newline_default [<newline-list>] 393</pre> 394When PCRE2 is built, a default newline convention can be specified. This 395determines which characters and/or character pairs are recognized as indicating 396a newline in a pattern or subject string. The default can be overridden when a 397pattern is compiled. The standard test files contain tests of various newline 398conventions, but the majority of the tests expect a single linefeed to be 399recognized as a newline by default. Without special action the tests would fail 400when PCRE2 is compiled with either CR or CRLF as the default newline. 401</P> 402<P> 403The #newline_default command specifies a list of newline types that are 404acceptable as the default. The types must be one of CR, LF, CRLF, ANYCRLF, 405ANY, or NUL (in upper or lower case), for example: 406<pre> 407 #newline_default LF Any anyCRLF 408</pre> 409If the default newline is in the list, this command has no effect. Otherwise, 410except when testing the POSIX API, a <b>newline</b> modifier that specifies the 411first newline convention in the list (LF in the above example) is added to any 412pattern that does not already have a <b>newline</b> modifier. If the newline 413list is empty, the feature is turned off. This command is present in a number 414of the standard test input files. 415</P> 416<P> 417When the POSIX API is being tested there is no way to override the default 418newline convention, though it is possible to set the newline convention from 419within the pattern. A warning is given if the <b>posix</b> or <b>posix_nosub</b> 420modifier is used when <b>#newline_default</b> would set a default for the 421non-POSIX API. 422<pre> 423 #pattern <modifier-list> 424</pre> 425This command sets a default modifier list that applies to all subsequent 426patterns. Modifiers on a pattern can change these settings. 427<pre> 428 #perltest 429</pre> 430This line is used in test files that can also be processed by <b>perltest.sh</b> 431to confirm that Perl gives the same results as PCRE2. Subsequent tests are 432checked for the use of <b>pcre2test</b> features that are incompatible with the 433<b>perltest.sh</b> script. 434</P> 435<P> 436Patterns must use '/' as their delimiter, and only certain modifiers are 437supported. Comment lines, #pattern commands, and #subject commands that set or 438unset "mark" are recognized and acted on. The #perltest, #forbid_utf, and 439#newline_default commands, which are needed in the relevant pcre2test files, 440are silently ignored. All other command lines are ignored, but give a warning 441message. The <b>#perltest</b> command helps detect tests that are accidentally 442put in the wrong file or use the wrong delimiter. For more details of the 443<b>perltest.sh</b> script see the comments it contains. 444<pre> 445 #pop [<modifiers>] 446 #popcopy [<modifiers>] 447</pre> 448These commands are used to manipulate the stack of compiled patterns, as 449described in the section entitled "Saving and restoring compiled patterns" 450<a href="#saverestore">below.</a> 451<pre> 452 #save <filename> 453</pre> 454This command is used to save a set of compiled patterns to a file, as described 455in the section entitled "Saving and restoring compiled patterns" 456<a href="#saverestore">below.</a> 457<pre> 458 #subject <modifier-list> 459</pre> 460This command sets a default modifier list that applies to all subsequent 461subject lines. Modifiers on a subject line can change these settings. 462</P> 463<br><a name="SEC7" href="#TOC1">MODIFIER SYNTAX</a><br> 464<P> 465Modifier lists are used with both pattern and subject lines. Items in a list 466are separated by commas followed by optional white space. Trailing whitespace 467in a modifier list is ignored. Some modifiers may be given for both patterns 468and subject lines, whereas others are valid only for one or the other. Each 469modifier has a long name, for example "anchored", and some of them must be 470followed by an equals sign and a value, for example, "offset=12". Values cannot 471contain comma characters, but may contain spaces. Modifiers that do not take 472values may be preceded by a minus sign to turn off a previous setting. 473</P> 474<P> 475A few of the more common modifiers can also be specified as single letters, for 476example "i" for "caseless". In documentation, following the Perl convention, 477these are written with a slash ("the /i modifier") for clarity. Abbreviated 478modifiers must all be concatenated in the first item of a modifier list. If the 479first item is not recognized as a long modifier name, it is interpreted as a 480sequence of these abbreviations. For example: 481<pre> 482 /abc/ig,newline=cr,jit=3 483</pre> 484This is a pattern line whose modifier list starts with two one-letter modifiers 485(/i and /g). The lower-case abbreviated modifiers are the same as used in Perl. 486</P> 487<br><a name="SEC8" href="#TOC1">PATTERN SYNTAX</a><br> 488<P> 489A pattern line must start with one of the following characters (common symbols, 490excluding pattern meta-characters): 491<pre> 492 / ! " ' ` - = _ : ; , % & @ ~ 493</pre> 494This is interpreted as the pattern's delimiter. A regular expression may be 495continued over several input lines, in which case the newline characters are 496included within it. It is possible to include the delimiter as a literal within 497the pattern by escaping it with a backslash, for example 498<pre> 499 /abc\/def/ 500</pre> 501If you do this, the escape and the delimiter form part of the pattern, but 502since the delimiters are all non-alphanumeric, the inclusion of the backslash 503does not affect the pattern's interpretation. Note, however, that this trick 504does not work within \Q...\E literal bracketing because the backslash will 505itself be interpreted as a literal. If the terminating delimiter is immediately 506followed by a backslash, for example, 507<pre> 508 /abc/\ 509</pre> 510then a backslash is added to the end of the pattern. This is done to provide a 511way of testing the error condition that arises if a pattern finishes with a 512backslash, because 513<pre> 514 /abc\/ 515</pre> 516is interpreted as the first line of a pattern that starts with "abc/", causing 517pcre2test to read the next line as a continuation of the regular expression. 518</P> 519<P> 520A pattern can be followed by a modifier list (details below). 521</P> 522<br><a name="SEC9" href="#TOC1">SUBJECT LINE SYNTAX</a><br> 523<P> 524Before each subject line is passed to <b>pcre2_match()</b>, 525<b>pcre2_dfa_match()</b>, or <b>pcre2_jit_match()</b>, leading and trailing white 526space is removed, and the line is scanned for backslash escapes, unless the 527<b>subject_literal</b> modifier was set for the pattern. The following provide a 528means of encoding non-printing characters in a visible way: 529<pre> 530 \a alarm (BEL, \x07) 531 \b backspace (\x08) 532 \e escape (\x27) 533 \f form feed (\x0c) 534 \n newline (\x0a) 535 \r carriage return (\x0d) 536 \t tab (\x09) 537 \v vertical tab (\x0b) 538 \nnn octal character (up to 3 octal digits); always 539 a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode 540 \o{dd...} octal character (any number of octal digits} 541 \xhh hexadecimal byte (up to 2 hex digits) 542 \x{hh...} hexadecimal character (any number of hex digits) 543</pre> 544The use of \x{hh...} is not dependent on the use of the <b>utf</b> modifier on 545the pattern. It is recognized always. There may be any number of hexadecimal 546digits inside the braces; invalid values provoke error messages. 547</P> 548<P> 549Note that \xhh specifies one byte rather than one character in UTF-8 mode; 550this makes it possible to construct invalid UTF-8 sequences for testing 551purposes. On the other hand, \x{hh} is interpreted as a UTF-8 character in 552UTF-8 mode, generating more than one byte if the value is greater than 127. 553When testing the 8-bit library not in UTF-8 mode, \x{hh} generates one byte 554for values less than 256, and causes an error for greater values. 555</P> 556<P> 557In UTF-16 mode, all 4-digit \x{hhhh} values are accepted. This makes it 558possible to construct invalid UTF-16 sequences for testing purposes. 559</P> 560<P> 561In UTF-32 mode, all 4- to 8-digit \x{...} values are accepted. This makes it 562possible to construct invalid UTF-32 sequences for testing purposes. 563</P> 564<P> 565There is a special backslash sequence that specifies replication of one or more 566characters: 567<pre> 568 \[<characters>]{<count>} 569</pre> 570This makes it possible to test long strings without having to provide them as 571part of the file. For example: 572<pre> 573 \[abc]{4} 574</pre> 575is converted to "abcabcabcabc". This feature does not support nesting. To 576include a closing square bracket in the characters, code it as \x5D. 577</P> 578<P> 579A backslash followed by an equals sign marks the end of the subject string and 580the start of a modifier list. For example: 581<pre> 582 abc\=notbol,notempty 583</pre> 584If the subject string is empty and \= is followed by whitespace, the line is 585treated as a comment line, and is not used for matching. For example: 586<pre> 587 \= This is a comment. 588 abc\= This is an invalid modifier list. 589</pre> 590A backslash followed by any other non-alphanumeric character just escapes that 591character. A backslash followed by anything else causes an error. However, if 592the very last character in the line is a backslash (and there is no modifier 593list), it is ignored. This gives a way of passing an empty line as data, since 594a real empty line terminates the data input. 595</P> 596<P> 597If the <b>subject_literal</b> modifier is set for a pattern, all subject lines 598that follow are treated as literals, with no special treatment of backslashes. 599No replication is possible, and any subject modifiers must be set as defaults 600by a <b>#subject</b> command. 601</P> 602<br><a name="SEC10" href="#TOC1">PATTERN MODIFIERS</a><br> 603<P> 604There are several types of modifier that can appear in pattern lines. Except 605where noted below, they may also be used in <b>#pattern</b> commands. A 606pattern's modifier list can add to or override default modifiers that were set 607by a previous <b>#pattern</b> command. 608<a name="optionmodifiers"></a></P> 609<br><b> 610Setting compilation options 611</b><br> 612<P> 613The following modifiers set options for <b>pcre2_compile()</b>. Most of them set 614bits in the options argument of that function, but those whose names start with 615PCRE2_EXTRA are additional options that are set in the compile context. For the 616main options, there are some single-letter abbreviations that are the same as 617Perl options. There is special handling for /x: if a second x is present, 618PCRE2_EXTENDED is converted into PCRE2_EXTENDED_MORE as in Perl. A third 619appearance adds PCRE2_EXTENDED as well, though this makes no difference to the 620way <b>pcre2_compile()</b> behaves. See 621<a href="pcre2api.html"><b>pcre2api</b></a> 622for a description of the effects of these options. 623<pre> 624 allow_empty_class set PCRE2_ALLOW_EMPTY_CLASS 625 allow_lookaround_bsk set PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK 626 allow_surrogate_escapes set PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES 627 alt_bsux set PCRE2_ALT_BSUX 628 alt_circumflex set PCRE2_ALT_CIRCUMFLEX 629 alt_verbnames set PCRE2_ALT_VERBNAMES 630 anchored set PCRE2_ANCHORED 631 auto_callout set PCRE2_AUTO_CALLOUT 632 bad_escape_is_literal set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL 633 /i caseless set PCRE2_CASELESS 634 dollar_endonly set PCRE2_DOLLAR_ENDONLY 635 /s dotall set PCRE2_DOTALL 636 dupnames set PCRE2_DUPNAMES 637 endanchored set PCRE2_ENDANCHORED 638 escaped_cr_is_lf set PCRE2_EXTRA_ESCAPED_CR_IS_LF 639 /x extended set PCRE2_EXTENDED 640 /xx extended_more set PCRE2_EXTENDED_MORE 641 extra_alt_bsux set PCRE2_EXTRA_ALT_BSUX 642 firstline set PCRE2_FIRSTLINE 643 literal set PCRE2_LITERAL 644 match_line set PCRE2_EXTRA_MATCH_LINE 645 match_invalid_utf set PCRE2_MATCH_INVALID_UTF 646 match_unset_backref set PCRE2_MATCH_UNSET_BACKREF 647 match_word set PCRE2_EXTRA_MATCH_WORD 648 /m multiline set PCRE2_MULTILINE 649 never_backslash_c set PCRE2_NEVER_BACKSLASH_C 650 never_ucp set PCRE2_NEVER_UCP 651 never_utf set PCRE2_NEVER_UTF 652 /n no_auto_capture set PCRE2_NO_AUTO_CAPTURE 653 no_auto_possess set PCRE2_NO_AUTO_POSSESS 654 no_dotstar_anchor set PCRE2_NO_DOTSTAR_ANCHOR 655 no_start_optimize set PCRE2_NO_START_OPTIMIZE 656 no_utf_check set PCRE2_NO_UTF_CHECK 657 ucp set PCRE2_UCP 658 ungreedy set PCRE2_UNGREEDY 659 use_offset_limit set PCRE2_USE_OFFSET_LIMIT 660 utf set PCRE2_UTF 661</pre> 662As well as turning on the PCRE2_UTF option, the <b>utf</b> modifier causes all 663non-printing characters in output strings to be printed using the \x{hh...} 664notation. Otherwise, those less than 0x100 are output in hex without the curly 665brackets. Setting <b>utf</b> in 16-bit or 32-bit mode also causes pattern and 666subject strings to be translated to UTF-16 or UTF-32, respectively, before 667being passed to library functions. 668<a name="controlmodifiers"></a></P> 669<br><b> 670Setting compilation controls 671</b><br> 672<P> 673The following modifiers affect the compilation process or request information 674about the pattern. There are single-letter abbreviations for some that are 675heavily used in the test files. 676<pre> 677 bsr=[anycrlf|unicode] specify \R handling 678 /B bincode show binary code without lengths 679 callout_info show callout information 680 convert=<options> request foreign pattern conversion 681 convert_glob_escape=c set glob escape character 682 convert_glob_separator=c set glob separator character 683 convert_length set convert buffer length 684 debug same as info,fullbincode 685 framesize show matching frame size 686 fullbincode show binary code with lengths 687 /I info show info about compiled pattern 688 hex unquoted characters are hexadecimal 689 jit[=<number>] use JIT 690 jitfast use JIT fast path 691 jitverify verify JIT use 692 locale=<name> use this locale 693 max_pattern_length=<n> set the maximum pattern length 694 memory show memory used 695 newline=<type> set newline type 696 null_context compile with a NULL context 697 parens_nest_limit=<n> set maximum parentheses depth 698 posix use the POSIX API 699 posix_nosub use the POSIX API with REG_NOSUB 700 push push compiled pattern onto the stack 701 pushcopy push a copy onto the stack 702 stackguard=<number> test the stackguard feature 703 subject_literal treat all subject lines as literal 704 tables=[0|1|2|3] select internal tables 705 use_length do not zero-terminate the pattern 706 utf8_input treat input as UTF-8 707</pre> 708The effects of these modifiers are described in the following sections. 709</P> 710<br><b> 711Newline and \R handling 712</b><br> 713<P> 714The <b>bsr</b> modifier specifies what \R in a pattern should match. If it is 715set to "anycrlf", \R matches CR, LF, or CRLF only. If it is set to "unicode", 716\R matches any Unicode newline sequence. The default can be specified when 717PCRE2 is built; if it is not, the default is set to Unicode. 718</P> 719<P> 720The <b>newline</b> modifier specifies which characters are to be interpreted as 721newlines, both in the pattern and in subject lines. The type must be one of CR, 722LF, CRLF, ANYCRLF, ANY, or NUL (in upper or lower case). 723</P> 724<br><b> 725Information about a pattern 726</b><br> 727<P> 728The <b>debug</b> modifier is a shorthand for <b>info,fullbincode</b>, requesting 729all available information. 730</P> 731<P> 732The <b>bincode</b> modifier causes a representation of the compiled code to be 733output after compilation. This information does not contain length and offset 734values, which ensures that the same output is generated for different internal 735link sizes and different code unit widths. By using <b>bincode</b>, the same 736regression tests can be used in different environments. 737</P> 738<P> 739The <b>fullbincode</b> modifier, by contrast, <i>does</i> include length and 740offset values. This is used in a few special tests that run only for specific 741code unit widths and link sizes, and is also useful for one-off tests. 742</P> 743<P> 744The <b>info</b> modifier requests information about the compiled pattern 745(whether it is anchored, has a fixed first character, and so on). The 746information is obtained from the <b>pcre2_pattern_info()</b> function. Here are 747some typical examples: 748<pre> 749 re> /(?i)(^a|^b)/m,info 750 Capture group count = 1 751 Compile options: multiline 752 Overall options: caseless multiline 753 First code unit at start or follows newline 754 Subject length lower bound = 1 755 756 re> /(?i)abc/info 757 Capture group count = 0 758 Compile options: <none> 759 Overall options: caseless 760 First code unit = 'a' (caseless) 761 Last code unit = 'c' (caseless) 762 Subject length lower bound = 3 763</pre> 764"Compile options" are those specified by modifiers; "overall options" have 765added options that are taken or deduced from the pattern. If both sets of 766options are the same, just a single "options" line is output; if there are no 767options, the line is omitted. "First code unit" is where any match must start; 768if there is more than one they are listed as "starting code units". "Last code 769unit" is the last literal code unit that must be present in any match. This is 770not necessarily the last character. These lines are omitted if no starting or 771ending code units are recorded. The subject length line is omitted when 772<b>no_start_optimize</b> is set because the minimum length is not calculated 773when it can never be used. 774</P> 775<P> 776The <b>framesize</b> modifier shows the size, in bytes, of the storage frames 777used by <b>pcre2_match()</b> for handling backtracking. The size depends on the 778number of capturing parentheses in the pattern. 779</P> 780<P> 781The <b>callout_info</b> modifier requests information about all the callouts in 782the pattern. A list of them is output at the end of any other information that 783is requested. For each callout, either its number or string is given, followed 784by the item that follows it in the pattern. 785</P> 786<br><b> 787Passing a NULL context 788</b><br> 789<P> 790Normally, <b>pcre2test</b> passes a context block to <b>pcre2_compile()</b>. If 791the <b>null_context</b> modifier is set, however, NULL is passed. This is for 792testing that <b>pcre2_compile()</b> behaves correctly in this case (it uses 793default values). 794</P> 795<br><b> 796Specifying pattern characters in hexadecimal 797</b><br> 798<P> 799The <b>hex</b> modifier specifies that the characters of the pattern, except for 800substrings enclosed in single or double quotes, are to be interpreted as pairs 801of hexadecimal digits. This feature is provided as a way of creating patterns 802that contain binary zeros and other non-printing characters. White space is 803permitted between pairs of digits. For example, this pattern contains three 804characters: 805<pre> 806 /ab 32 59/hex 807</pre> 808Parts of such a pattern are taken literally if quoted. This pattern contains 809nine characters, only two of which are specified in hexadecimal: 810<pre> 811 /ab "literal" 32/hex 812</pre> 813Either single or double quotes may be used. There is no way of including 814the delimiter within a substring. The <b>hex</b> and <b>expand</b> modifiers are 815mutually exclusive. 816</P> 817<br><b> 818Specifying the pattern's length 819</b><br> 820<P> 821By default, patterns are passed to the compiling functions as zero-terminated 822strings but can be passed by length instead of being zero-terminated. The 823<b>use_length</b> modifier causes this to happen. Using a length happens 824automatically (whether or not <b>use_length</b> is set) when <b>hex</b> is set, 825because patterns specified in hexadecimal may contain binary zeros. 826</P> 827<P> 828If <b>hex</b> or <b>use_length</b> is used with the POSIX wrapper API (see 829<a href="#posixwrapper">"Using the POSIX wrapper API"</a> 830below), the REG_PEND extension is used to pass the pattern's length. 831</P> 832<br><b> 833Specifying wide characters in 16-bit and 32-bit modes 834</b><br> 835<P> 836In 16-bit and 32-bit modes, all input is automatically treated as UTF-8 and 837translated to UTF-16 or UTF-32 when the <b>utf</b> modifier is set. For testing 838the 16-bit and 32-bit libraries in non-UTF mode, the <b>utf8_input</b> modifier 839can be used. It is mutually exclusive with <b>utf</b>. Input lines are 840interpreted as UTF-8 as a means of specifying wide characters. More details are 841given in 842<a href="#inputencoding">"Input encoding"</a> 843above. 844</P> 845<br><b> 846Generating long repetitive patterns 847</b><br> 848<P> 849Some tests use long patterns that are very repetitive. Instead of creating a 850very long input line for such a pattern, you can use a special repetition 851feature, similar to the one described for subject lines above. If the 852<b>expand</b> modifier is present on a pattern, parts of the pattern that have 853the form 854<pre> 855 \[<characters>]{<count>} 856</pre> 857are expanded before the pattern is passed to <b>pcre2_compile()</b>. For 858example, \[AB]{6000} is expanded to "ABAB..." 6000 times. This construction 859cannot be nested. An initial "\[" sequence is recognized only if "]{" followed 860by decimal digits and "}" is found later in the pattern. If not, the characters 861remain in the pattern unaltered. The <b>expand</b> and <b>hex</b> modifiers are 862mutually exclusive. 863</P> 864<P> 865If part of an expanded pattern looks like an expansion, but is really part of 866the actual pattern, unwanted expansion can be avoided by giving two values in 867the quantifier. For example, \[AB]{6000,6000} is not recognized as an 868expansion item. 869</P> 870<P> 871If the <b>info</b> modifier is set on an expanded pattern, the result of the 872expansion is included in the information that is output. 873</P> 874<br><b> 875JIT compilation 876</b><br> 877<P> 878Just-in-time (JIT) compiling is a heavyweight optimization that can greatly 879speed up pattern matching. See the 880<a href="pcre2jit.html"><b>pcre2jit</b></a> 881documentation for details. JIT compiling happens, optionally, after a pattern 882has been successfully compiled into an internal form. The JIT compiler converts 883this to optimized machine code. It needs to know whether the match-time options 884PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT are going to be used, because 885different code is generated for the different cases. See the <b>partial</b> 886modifier in "Subject Modifiers" 887<a href="#subjectmodifiers">below</a> 888for details of how these options are specified for each match attempt. 889</P> 890<P> 891JIT compilation is requested by the <b>jit</b> pattern modifier, which may 892optionally be followed by an equals sign and a number in the range 0 to 7. 893The three bits that make up the number specify which of the three JIT operating 894modes are to be compiled: 895<pre> 896 1 compile JIT code for non-partial matching 897 2 compile JIT code for soft partial matching 898 4 compile JIT code for hard partial matching 899</pre> 900The possible values for the <b>jit</b> modifier are therefore: 901<pre> 902 0 disable JIT 903 1 normal matching only 904 2 soft partial matching only 905 3 normal and soft partial matching 906 4 hard partial matching only 907 6 soft and hard partial matching only 908 7 all three modes 909</pre> 910If no number is given, 7 is assumed. The phrase "partial matching" means a call 911to <b>pcre2_match()</b> with either the PCRE2_PARTIAL_SOFT or the 912PCRE2_PARTIAL_HARD option set. Note that such a call may return a complete 913match; the options enable the possibility of a partial match, but do not 914require it. Note also that if you request JIT compilation only for partial 915matching (for example, jit=2) but do not set the <b>partial</b> modifier on a 916subject line, that match will not use JIT code because none was compiled for 917non-partial matching. 918</P> 919<P> 920If JIT compilation is successful, the compiled JIT code will automatically be 921used when an appropriate type of match is run, except when incompatible 922run-time options are specified. For more details, see the 923<a href="pcre2jit.html"><b>pcre2jit</b></a> 924documentation. See also the <b>jitstack</b> modifier below for a way of 925setting the size of the JIT stack. 926</P> 927<P> 928If the <b>jitfast</b> modifier is specified, matching is done using the JIT 929"fast path" interface, <b>pcre2_jit_match()</b>, which skips some of the sanity 930checks that are done by <b>pcre2_match()</b>, and of course does not work when 931JIT is not supported. If <b>jitfast</b> is specified without <b>jit</b>, jit=7 is 932assumed. 933</P> 934<P> 935If the <b>jitverify</b> modifier is specified, information about the compiled 936pattern shows whether JIT compilation was or was not successful. If 937<b>jitverify</b> is specified without <b>jit</b>, jit=7 is assumed. If JIT 938compilation is successful when <b>jitverify</b> is set, the text "(JIT)" is 939added to the first output line after a match or non match when JIT-compiled 940code was actually used in the match. 941</P> 942<br><b> 943Setting a locale 944</b><br> 945<P> 946The <b>locale</b> modifier must specify the name of a locale, for example: 947<pre> 948 /pattern/locale=fr_FR 949</pre> 950The given locale is set, <b>pcre2_maketables()</b> is called to build a set of 951character tables for the locale, and this is then passed to 952<b>pcre2_compile()</b> when compiling the regular expression. The same tables 953are used when matching the following subject lines. The <b>locale</b> modifier 954applies only to the pattern on which it appears, but can be given in a 955<b>#pattern</b> command if a default is needed. Setting a locale and alternate 956character tables are mutually exclusive. 957</P> 958<br><b> 959Showing pattern memory 960</b><br> 961<P> 962The <b>memory</b> modifier causes the size in bytes of the memory used to hold 963the compiled pattern to be output. This does not include the size of the 964<b>pcre2_code</b> block; it is just the actual compiled data. If the pattern is 965subsequently passed to the JIT compiler, the size of the JIT compiled code is 966also output. Here is an example: 967<pre> 968 re> /a(b)c/jit,memory 969 Memory allocation (code space): 21 970 Memory allocation (JIT code): 1910 971 972</PRE> 973</P> 974<br><b> 975Limiting nested parentheses 976</b><br> 977<P> 978The <b>parens_nest_limit</b> modifier sets a limit on the depth of nested 979parentheses in a pattern. Breaching the limit causes a compilation error. 980The default for the library is set when PCRE2 is built, but <b>pcre2test</b> 981sets its own default of 220, which is required for running the standard test 982suite. 983</P> 984<br><b> 985Limiting the pattern length 986</b><br> 987<P> 988The <b>max_pattern_length</b> modifier sets a limit, in code units, to the 989length of pattern that <b>pcre2_compile()</b> will accept. Breaching the limit 990causes a compilation error. The default is the largest number a PCRE2_SIZE 991variable can hold (essentially unlimited). 992<a name="posixwrapper"></a></P> 993<br><b> 994Using the POSIX wrapper API 995</b><br> 996<P> 997The <b>posix</b> and <b>posix_nosub</b> modifiers cause <b>pcre2test</b> to call 998PCRE2 via the POSIX wrapper API rather than its native API. When 999<b>posix_nosub</b> is used, the POSIX option REG_NOSUB is passed to 1000<b>regcomp()</b>. The POSIX wrapper supports only the 8-bit library. Note that 1001it does not imply POSIX matching semantics; for more detail see the 1002<a href="pcre2posix.html"><b>pcre2posix</b></a> 1003documentation. The following pattern modifiers set options for the 1004<b>regcomp()</b> function: 1005<pre> 1006 caseless REG_ICASE 1007 multiline REG_NEWLINE 1008 dotall REG_DOTALL ) 1009 ungreedy REG_UNGREEDY ) These options are not part of 1010 ucp REG_UCP ) the POSIX standard 1011 utf REG_UTF8 ) 1012</pre> 1013The <b>regerror_buffsize</b> modifier specifies a size for the error buffer that 1014is passed to <b>regerror()</b> in the event of a compilation error. For example: 1015<pre> 1016 /abc/posix,regerror_buffsize=20 1017</pre> 1018This provides a means of testing the behaviour of <b>regerror()</b> when the 1019buffer is too small for the error message. If this modifier has not been set, a 1020large buffer is used. 1021</P> 1022<P> 1023The <b>aftertext</b> and <b>allaftertext</b> subject modifiers work as described 1024below. All other modifiers are either ignored, with a warning message, or cause 1025an error. 1026</P> 1027<P> 1028The pattern is passed to <b>regcomp()</b> as a zero-terminated string by 1029default, but if the <b>use_length</b> or <b>hex</b> modifiers are set, the 1030REG_PEND extension is used to pass it by length. 1031</P> 1032<br><b> 1033Testing the stack guard feature 1034</b><br> 1035<P> 1036The <b>stackguard</b> modifier is used to test the use of 1037<b>pcre2_set_compile_recursion_guard()</b>, a function that is provided to 1038enable stack availability to be checked during compilation (see the 1039<a href="pcre2api.html"><b>pcre2api</b></a> 1040documentation for details). If the number specified by the modifier is greater 1041than zero, <b>pcre2_set_compile_recursion_guard()</b> is called to set up 1042callback from <b>pcre2_compile()</b> to a local function. The argument it 1043receives is the current nesting parenthesis depth; if this is greater than the 1044value given by the modifier, non-zero is returned, causing the compilation to 1045be aborted. 1046</P> 1047<br><b> 1048Using alternative character tables 1049</b><br> 1050<P> 1051The value specified for the <b>tables</b> modifier must be one of the digits 0, 10521, 2, or 3. It causes a specific set of built-in character tables to be passed 1053to <b>pcre2_compile()</b>. This is used in the PCRE2 tests to check behaviour 1054with different character tables. The digit specifies the tables as follows: 1055<pre> 1056 0 do not pass any special character tables 1057 1 the default ASCII tables, as distributed in 1058 pcre2_chartables.c.dist 1059 2 a set of tables defining ISO 8859 characters 1060 3 a set of tables loaded by the #loadtables command 1061</pre> 1062In tables 2, some characters whose codes are greater than 128 are identified as 1063letters, digits, spaces, etc. Tables 3 can be used only after a 1064<b>#loadtables</b> command has loaded them from a binary file. Setting alternate 1065character tables and a locale are mutually exclusive. 1066</P> 1067<br><b> 1068Setting certain match controls 1069</b><br> 1070<P> 1071The following modifiers are really subject modifiers, and are described under 1072"Subject Modifiers" below. However, they may be included in a pattern's 1073modifier list, in which case they are applied to every subject line that is 1074processed with that pattern. These modifiers do not affect the compilation 1075process. 1076<pre> 1077 aftertext show text after match 1078 allaftertext show text after captures 1079 allcaptures show all captures 1080 allvector show the entire ovector 1081 allusedtext show all consulted text 1082 altglobal alternative global matching 1083 /g global global matching 1084 jitstack=<n> set size of JIT stack 1085 mark show mark values 1086 replace=<string> specify a replacement string 1087 startchar show starting character when relevant 1088 substitute_callout use substitution callouts 1089 substitute_extended use PCRE2_SUBSTITUTE_EXTENDED 1090 substitute_literal use PCRE2_SUBSTITUTE_LITERAL 1091 substitute_matched use PCRE2_SUBSTITUTE_MATCHED 1092 substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH 1093 substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY 1094 substitute_skip=<n> skip substitution <n> 1095 substitute_stop=<n> skip substitution <n> and following 1096 substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET 1097 substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY 1098</pre> 1099These modifiers may not appear in a <b>#pattern</b> command. If you want them as 1100defaults, set them in a <b>#subject</b> command. 1101</P> 1102<br><b> 1103Specifying literal subject lines 1104</b><br> 1105<P> 1106If the <b>subject_literal</b> modifier is present on a pattern, all the subject 1107lines that it matches are taken as literal strings, with no interpretation of 1108backslashes. It is not possible to set subject modifiers on such lines, but any 1109that are set as defaults by a <b>#subject</b> command are recognized. 1110</P> 1111<br><b> 1112Saving a compiled pattern 1113</b><br> 1114<P> 1115When a pattern with the <b>push</b> modifier is successfully compiled, it is 1116pushed onto a stack of compiled patterns, and <b>pcre2test</b> expects the next 1117line to contain a new pattern (or a command) instead of a subject line. This 1118facility is used when saving compiled patterns to a file, as described in the 1119section entitled "Saving and restoring compiled patterns" 1120<a href="#saverestore">below.</a> 1121If <b>pushcopy</b> is used instead of <b>push</b>, a copy of the compiled 1122pattern is stacked, leaving the original as current, ready to match the 1123following input lines. This provides a way of testing the 1124<b>pcre2_code_copy()</b> function. 1125The <b>push</b> and <b>pushcopy </b> modifiers are incompatible with compilation 1126modifiers such as <b>global</b> that act at match time. Any that are specified 1127are ignored (for the stacked copy), with a warning message, except for 1128<b>replace</b>, which causes an error. Note that <b>jitverify</b>, which is 1129allowed, does not carry through to any subsequent matching that uses a stacked 1130pattern. 1131</P> 1132<br><b> 1133Testing foreign pattern conversion 1134</b><br> 1135<P> 1136The experimental foreign pattern conversion functions in PCRE2 can be tested by 1137setting the <b>convert</b> modifier. Its argument is a colon-separated list of 1138options, which set the equivalent option for the <b>pcre2_pattern_convert()</b> 1139function: 1140<pre> 1141 glob PCRE2_CONVERT_GLOB 1142 glob_no_starstar PCRE2_CONVERT_GLOB_NO_STARSTAR 1143 glob_no_wild_separator PCRE2_CONVERT_GLOB_NO_WILD_SEPARATOR 1144 posix_basic PCRE2_CONVERT_POSIX_BASIC 1145 posix_extended PCRE2_CONVERT_POSIX_EXTENDED 1146 unset Unset all options 1147</pre> 1148The "unset" value is useful for turning off a default that has been set by a 1149<b>#pattern</b> command. When one of these options is set, the input pattern is 1150passed to <b>pcre2_pattern_convert()</b>. If the conversion is successful, the 1151result is reflected in the output and then passed to <b>pcre2_compile()</b>. The 1152normal <b>utf</b> and <b>no_utf_check</b> options, if set, cause the 1153PCRE2_CONVERT_UTF and PCRE2_CONVERT_NO_UTF_CHECK options to be passed to 1154<b>pcre2_pattern_convert()</b>. 1155</P> 1156<P> 1157By default, the conversion function is allowed to allocate a buffer for its 1158output. However, if the <b>convert_length</b> modifier is set to a value greater 1159than zero, <b>pcre2test</b> passes a buffer of the given length. This makes it 1160possible to test the length check. 1161</P> 1162<P> 1163The <b>convert_glob_escape</b> and <b>convert_glob_separator</b> modifiers can be 1164used to specify the escape and separator characters for glob processing, 1165overriding the defaults, which are operating-system dependent. 1166<a name="subjectmodifiers"></a></P> 1167<br><a name="SEC11" href="#TOC1">SUBJECT MODIFIERS</a><br> 1168<P> 1169The modifiers that can appear in subject lines and the <b>#subject</b> 1170command are of two types. 1171</P> 1172<br><b> 1173Setting match options 1174</b><br> 1175<P> 1176The following modifiers set options for <b>pcre2_match()</b> or 1177<b>pcre2_dfa_match()</b>. See 1178<a href="pcreapi.html"><b>pcreapi</b></a> 1179for a description of their effects. 1180<pre> 1181 anchored set PCRE2_ANCHORED 1182 endanchored set PCRE2_ENDANCHORED 1183 dfa_restart set PCRE2_DFA_RESTART 1184 dfa_shortest set PCRE2_DFA_SHORTEST 1185 no_jit set PCRE2_NO_JIT 1186 no_utf_check set PCRE2_NO_UTF_CHECK 1187 notbol set PCRE2_NOTBOL 1188 notempty set PCRE2_NOTEMPTY 1189 notempty_atstart set PCRE2_NOTEMPTY_ATSTART 1190 noteol set PCRE2_NOTEOL 1191 partial_hard (or ph) set PCRE2_PARTIAL_HARD 1192 partial_soft (or ps) set PCRE2_PARTIAL_SOFT 1193</pre> 1194The partial matching modifiers are provided with abbreviations because they 1195appear frequently in tests. 1196</P> 1197<P> 1198If the <b>posix</b> or <b>posix_nosub</b> modifier was present on the pattern, 1199causing the POSIX wrapper API to be used, the only option-setting modifiers 1200that have any effect are <b>notbol</b>, <b>notempty</b>, and <b>noteol</b>, 1201causing REG_NOTBOL, REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to 1202<b>regexec()</b>. The other modifiers are ignored, with a warning message. 1203</P> 1204<P> 1205There is one additional modifier that can be used with the POSIX wrapper. It is 1206ignored (with a warning) if used for non-POSIX matching. 1207<pre> 1208 posix_startend=<n>[:<m>] 1209</pre> 1210This causes the subject string to be passed to <b>regexec()</b> using the 1211REG_STARTEND option, which uses offsets to specify which part of the string is 1212searched. If only one number is given, the end offset is passed as the end of 1213the subject string. For more detail of REG_STARTEND, see the 1214<a href="pcre2posix.html"><b>pcre2posix</b></a> 1215documentation. If the subject string contains binary zeros (coded as escapes 1216such as \x{00} because <b>pcre2test</b> does not support actual binary zeros in 1217its input), you must use <b>posix_startend</b> to specify its length. 1218</P> 1219<br><b> 1220Setting match controls 1221</b><br> 1222<P> 1223The following modifiers affect the matching process or request additional 1224information. Some of them may also be specified on a pattern line (see above), 1225in which case they apply to every subject line that is matched against that 1226pattern, but can be overridden by modifiers on the subject. 1227<pre> 1228 aftertext show text after match 1229 allaftertext show text after captures 1230 allcaptures show all captures 1231 allvector show the entire ovector 1232 allusedtext show all consulted text (non-JIT only) 1233 altglobal alternative global matching 1234 callout_capture show captures at callout time 1235 callout_data=<n> set a value to pass via callouts 1236 callout_error=<n>[:<m>] control callout error 1237 callout_extra show extra callout information 1238 callout_fail=<n>[:<m>] control callout failure 1239 callout_no_where do not show position of a callout 1240 callout_none do not supply a callout function 1241 copy=<number or name> copy captured substring 1242 depth_limit=<n> set a depth limit 1243 dfa use <b>pcre2_dfa_match()</b> 1244 find_limits find match and depth limits 1245 get=<number or name> extract captured substring 1246 getall extract all captured substrings 1247 /g global global matching 1248 heap_limit=<n> set a limit on heap memory (Kbytes) 1249 jitstack=<n> set size of JIT stack 1250 mark show mark values 1251 match_limit=<n> set a match limit 1252 memory show heap memory usage 1253 null_context match with a NULL context 1254 null_replacement substitute with NULL replacement 1255 null_subject match with NULL subject 1256 offset=<n> set starting offset 1257 offset_limit=<n> set offset limit 1258 ovector=<n> set size of output vector 1259 recursion_limit=<n> obsolete synonym for depth_limit 1260 replace=<string> specify a replacement string 1261 startchar show startchar when relevant 1262 startoffset=<n> same as offset=<n> 1263 substitute_callout use substitution callouts 1264 substitute_extedded use PCRE2_SUBSTITUTE_EXTENDED 1265 substitute_literal use PCRE2_SUBSTITUTE_LITERAL 1266 substitute_matched use PCRE2_SUBSTITUTE_MATCHED 1267 substitute_overflow_length use PCRE2_SUBSTITUTE_OVERFLOW_LENGTH 1268 substitute_replacement_only use PCRE2_SUBSTITUTE_REPLACEMENT_ONLY 1269 substitute_skip=<n> skip substitution number n 1270 substitute_stop=<n> skip substitution number n and greater 1271 substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET 1272 substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY 1273 zero_terminate pass the subject as zero-terminated 1274</pre> 1275The effects of these modifiers are described in the following sections. When 1276matching via the POSIX wrapper API, the <b>aftertext</b>, <b>allaftertext</b>, 1277and <b>ovector</b> subject modifiers work as described below. All other 1278modifiers are either ignored, with a warning message, or cause an error. 1279</P> 1280<br><b> 1281Showing more text 1282</b><br> 1283<P> 1284The <b>aftertext</b> modifier requests that as well as outputting the part of 1285the subject string that matched the entire pattern, <b>pcre2test</b> should in 1286addition output the remainder of the subject string. This is useful for tests 1287where the subject contains multiple copies of the same substring. The 1288<b>allaftertext</b> modifier requests the same action for captured substrings as 1289well as the main matched substring. In each case the remainder is output on the 1290following line with a plus character following the capture number. 1291</P> 1292<P> 1293The <b>allusedtext</b> modifier requests that all the text that was consulted 1294during a successful pattern match by the interpreter should be shown, for both 1295full and partial matches. This feature is not supported for JIT matching, and 1296if requested with JIT it is ignored (with a warning message). Setting this 1297modifier affects the output if there is a lookbehind at the start of a match, 1298or, for a complete match, a lookahead at the end, or if \K is used in the 1299pattern. Characters that precede or follow the start and end of the actual 1300match are indicated in the output by '<' or '>' characters underneath them. 1301Here is an example: 1302<pre> 1303 re> /(?<=pqr)abc(?=xyz)/ 1304 data> 123pqrabcxyz456\=allusedtext 1305 0: pqrabcxyz 1306 <<< >>> 1307 data> 123pqrabcxy\=ph,allusedtext 1308 Partial match: pqrabcxy 1309 <<< 1310</pre> 1311The first, complete match shows that the matched string is "abc", with the 1312preceding and following strings "pqr" and "xyz" having been consulted during 1313the match (when processing the assertions). The partial match can indicate only 1314the preceding string. 1315</P> 1316<P> 1317The <b>startchar</b> modifier requests that the starting character for the match 1318be indicated, if it is different to the start of the matched string. The only 1319time when this occurs is when \K has been processed as part of the match. In 1320this situation, the output for the matched string is displayed from the 1321starting character instead of from the match point, with circumflex characters 1322under the earlier characters. For example: 1323<pre> 1324 re> /abc\Kxyz/ 1325 data> abcxyz\=startchar 1326 0: abcxyz 1327 ^^^ 1328</pre> 1329Unlike <b>allusedtext</b>, the <b>startchar</b> modifier can be used with JIT. 1330However, these two modifiers are mutually exclusive. 1331</P> 1332<br><b> 1333Showing the value of all capture groups 1334</b><br> 1335<P> 1336The <b>allcaptures</b> modifier requests that the values of all potential 1337captured parentheses be output after a match. By default, only those up to the 1338highest one actually used in the match are output (corresponding to the return 1339code from <b>pcre2_match()</b>). Groups that did not take part in the match 1340are output as "<unset>". This modifier is not relevant for DFA matching (which 1341does no capturing) and does not apply when <b>replace</b> is specified; it is 1342ignored, with a warning message, if present. 1343</P> 1344<br><b> 1345Showing the entire ovector, for all outcomes 1346</b><br> 1347<P> 1348The <b>allvector</b> modifier requests that the entire ovector be shown, 1349whatever the outcome of the match. Compare <b>allcaptures</b>, which shows only 1350up to the maximum number of capture groups for the pattern, and then only for a 1351successful complete non-DFA match. This modifier, which acts after any match 1352result, and also for DFA matching, provides a means of checking that there are 1353no unexpected modifications to ovector fields. Before each match attempt, the 1354ovector is filled with a special value, and if this is found in both elements 1355of a capturing pair, "<unchanged>" is output. After a successful match, this 1356applies to all groups after the maximum capture group for the pattern. In other 1357cases it applies to the entire ovector. After a partial match, the first two 1358elements are the only ones that should be set. After a DFA match, the amount of 1359ovector that is used depends on the number of matches that were found. 1360</P> 1361<br><b> 1362Testing pattern callouts 1363</b><br> 1364<P> 1365A callout function is supplied when <b>pcre2test</b> calls the library matching 1366functions, unless <b>callout_none</b> is specified. Its behaviour can be 1367controlled by various modifiers listed above whose names begin with 1368<b>callout_</b>. Details are given in the section entitled "Callouts" 1369<a href="#callouts">below.</a> 1370Testing callouts from <b>pcre2_substitute()</b> is decribed separately in 1371"Testing the substitution function" 1372<a href="#substitution">below.</a> 1373</P> 1374<br><b> 1375Finding all matches in a string 1376</b><br> 1377<P> 1378Searching for all possible matches within a subject can be requested by the 1379<b>global</b> or <b>altglobal</b> modifier. After finding a match, the matching 1380function is called again to search the remainder of the subject. The difference 1381between <b>global</b> and <b>altglobal</b> is that the former uses the 1382<i>start_offset</i> argument to <b>pcre2_match()</b> or <b>pcre2_dfa_match()</b> 1383to start searching at a new point within the entire string (which is what Perl 1384does), whereas the latter passes over a shortened subject. This makes a 1385difference to the matching process if the pattern begins with a lookbehind 1386assertion (including \b or \B). 1387</P> 1388<P> 1389If an empty string is matched, the next match is done with the 1390PCRE2_NOTEMPTY_ATSTART and PCRE2_ANCHORED flags set, in order to search for 1391another, non-empty, match at the same point in the subject. If this match 1392fails, the start offset is advanced, and the normal match is retried. This 1393imitates the way Perl handles such cases when using the <b>/g</b> modifier or 1394the <b>split()</b> function. Normally, the start offset is advanced by one 1395character, but if the newline convention recognizes CRLF as a newline, and the 1396current character is CR followed by LF, an advance of two characters occurs. 1397</P> 1398<br><b> 1399Testing substring extraction functions 1400</b><br> 1401<P> 1402The <b>copy</b> and <b>get</b> modifiers can be used to test the 1403<b>pcre2_substring_copy_xxx()</b> and <b>pcre2_substring_get_xxx()</b> functions. 1404They can be given more than once, and each can specify a capture group name or 1405number, for example: 1406<pre> 1407 abcd\=copy=1,copy=3,get=G1 1408</pre> 1409If the <b>#subject</b> command is used to set default copy and/or get lists, 1410these can be unset by specifying a negative number to cancel all numbered 1411groups and an empty name to cancel all named groups. 1412</P> 1413<P> 1414The <b>getall</b> modifier tests <b>pcre2_substring_list_get()</b>, which 1415extracts all captured substrings. 1416</P> 1417<P> 1418If the subject line is successfully matched, the substrings extracted by the 1419convenience functions are output with C, G, or L after the string number 1420instead of a colon. This is in addition to the normal full list. The string 1421length (that is, the return from the extraction function) is given in 1422parentheses after each substring, followed by the name when the extraction was 1423by name. 1424<a name="substitution"></a></P> 1425<br><b> 1426Testing the substitution function 1427</b><br> 1428<P> 1429If the <b>replace</b> modifier is set, the <b>pcre2_substitute()</b> function is 1430called instead of one of the matching functions (or after one call of 1431<b>pcre2_match()</b> in the case of PCRE2_SUBSTITUTE_MATCHED). Note that 1432replacement strings cannot contain commas, because a comma signifies the end of 1433a modifier. This is not thought to be an issue in a test program. 1434</P> 1435<P> 1436Specifying a completely empty replacement string disables this modifier. 1437However, it is possible to specify an empty replacement by providing a buffer 1438length, as described below, for an otherwise empty replacement. 1439</P> 1440<P> 1441Unlike subject strings, <b>pcre2test</b> does not process replacement strings 1442for escape sequences. In UTF mode, a replacement string is checked to see if it 1443is a valid UTF-8 string. If so, it is correctly converted to a UTF string of 1444the appropriate code unit width. If it is not a valid UTF-8 string, the 1445individual code units are copied directly. This provides a means of passing an 1446invalid UTF-8 string for testing purposes. 1447</P> 1448<P> 1449The following modifiers set options (in additional to the normal match options) 1450for <b>pcre2_substitute()</b>: 1451<pre> 1452 global PCRE2_SUBSTITUTE_GLOBAL 1453 substitute_extended PCRE2_SUBSTITUTE_EXTENDED 1454 substitute_literal PCRE2_SUBSTITUTE_LITERAL 1455 substitute_matched PCRE2_SUBSTITUTE_MATCHED 1456 substitute_overflow_length PCRE2_SUBSTITUTE_OVERFLOW_LENGTH 1457 substitute_replacement_only PCRE2_SUBSTITUTE_REPLACEMENT_ONLY 1458 substitute_unknown_unset PCRE2_SUBSTITUTE_UNKNOWN_UNSET 1459 substitute_unset_empty PCRE2_SUBSTITUTE_UNSET_EMPTY 1460</pre> 1461See the 1462<a href="pcre2api.html"><b>pcre2api</b></a> 1463documentation for details of these options. 1464</P> 1465<P> 1466After a successful substitution, the modified string is output, preceded by the 1467number of replacements. This may be zero if there were no matches. Here is a 1468simple example of a substitution test: 1469<pre> 1470 /abc/replace=xxx 1471 =abc=abc= 1472 1: =xxx=abc= 1473 =abc=abc=\=global 1474 2: =xxx=xxx= 1475</pre> 1476Subject and replacement strings should be kept relatively short (fewer than 256 1477characters) for substitution tests, as fixed-size buffers are used. To make it 1478easy to test for buffer overflow, if the replacement string starts with a 1479number in square brackets, that number is passed to <b>pcre2_substitute()</b> as 1480the size of the output buffer, with the replacement string starting at the next 1481character. Here is an example that tests the edge case: 1482<pre> 1483 /abc/ 1484 123abc123\=replace=[10]XYZ 1485 1: 123XYZ123 1486 123abc123\=replace=[9]XYZ 1487 Failed: error -47: no more memory 1488</pre> 1489The default action of <b>pcre2_substitute()</b> is to return 1490PCRE2_ERROR_NOMEMORY when the output buffer is too small. However, if the 1491PCRE2_SUBSTITUTE_OVERFLOW_LENGTH option is set (by using the 1492<b>substitute_overflow_length</b> modifier), <b>pcre2_substitute()</b> continues 1493to go through the motions of matching and substituting (but not doing any 1494callouts), in order to compute the size of buffer that is required. When this 1495happens, <b>pcre2test</b> shows the required buffer length (which includes space 1496for the trailing zero) as part of the error message. For example: 1497<pre> 1498 /abc/substitute_overflow_length 1499 123abc123\=replace=[9]XYZ 1500 Failed: error -47: no more memory: 10 code units are needed 1501</pre> 1502A replacement string is ignored with POSIX and DFA matching. Specifying partial 1503matching provokes an error return ("bad option value") from 1504<b>pcre2_substitute()</b>. 1505</P> 1506<br><b> 1507Testing substitute callouts 1508</b><br> 1509<P> 1510If the <b>substitute_callout</b> modifier is set, a substitution callout 1511function is set up. The <b>null_context</b> modifier must not be set, because 1512the address of the callout function is passed in a match context. When the 1513callout function is called (after each substitution), details of the the input 1514and output strings are output. For example: 1515<pre> 1516 /abc/g,replace=<$0>,substitute_callout 1517 abcdefabcpqr 1518 1(1) Old 0 3 "abc" New 0 5 "<abc>" 1519 2(1) Old 6 9 "abc" New 8 13 "<abc>" 1520 2: <abc>def<abc>pqr 1521</pre> 1522The first number on each callout line is the count of matches. The 1523parenthesized number is the number of pairs that are set in the ovector (that 1524is, one more than the number of capturing groups that were set). Then are 1525listed the offsets of the old substring, its contents, and the same for the 1526replacement. 1527</P> 1528<P> 1529By default, the substitution callout function returns zero, which accepts the 1530replacement and causes matching to continue if /g was used. Two further 1531modifiers can be used to test other return values. If <b>substitute_skip</b> is 1532set to a value greater than zero the callout function returns +1 for the match 1533of that number, and similarly <b>substitute_stop</b> returns -1. These cause the 1534replacement to be rejected, and -1 causes no further matching to take place. If 1535either of them are set, <b>substitute_callout</b> is assumed. For example: 1536<pre> 1537 /abc/g,replace=<$0>,substitute_skip=1 1538 abcdefabcpqr 1539 1(1) Old 0 3 "abc" New 0 5 "<abc> SKIPPED" 1540 2(1) Old 6 9 "abc" New 6 11 "<abc>" 1541 2: abcdef<abc>pqr 1542 abcdefabcpqr\=substitute_stop=1 1543 1(1) Old 0 3 "abc" New 0 5 "<abc> STOPPED" 1544 1: abcdefabcpqr 1545</pre> 1546If both are set for the same number, stop takes precedence. Only a single skip 1547or stop is supported, which is sufficient for testing that the feature works. 1548</P> 1549<br><b> 1550Setting the JIT stack size 1551</b><br> 1552<P> 1553The <b>jitstack</b> modifier provides a way of setting the maximum stack size 1554that is used by the just-in-time optimization code. It is ignored if JIT 1555optimization is not being used. The value is a number of kibibytes (units of 15561024 bytes). Setting zero reverts to the default of 32KiB. Providing a stack 1557that is larger than the default is necessary only for very complicated 1558patterns. If <b>jitstack</b> is set non-zero on a subject line it overrides any 1559value that was set on the pattern. 1560</P> 1561<br><b> 1562Setting heap, match, and depth limits 1563</b><br> 1564<P> 1565The <b>heap_limit</b>, <b>match_limit</b>, and <b>depth_limit</b> modifiers set 1566the appropriate limits in the match context. These values are ignored when the 1567<b>find_limits</b> modifier is specified. 1568</P> 1569<br><b> 1570Finding minimum limits 1571</b><br> 1572<P> 1573If the <b>find_limits</b> modifier is present on a subject line, <b>pcre2test</b> 1574calls the relevant matching function several times, setting different values in 1575the match context via <b>pcre2_set_heap_limit()</b>, 1576<b>pcre2_set_match_limit()</b>, or <b>pcre2_set_depth_limit()</b> until it finds 1577the minimum values for each parameter that allows the match to complete without 1578error. If JIT is being used, only the match limit is relevant. 1579</P> 1580<P> 1581When using this modifier, the pattern should not contain any limit settings 1582such as (*LIMIT_MATCH=...) within it. If such a setting is present and is 1583lower than the minimum matching value, the minimum value cannot be found 1584because <b>pcre2_set_match_limit()</b> etc. are only able to reduce the value of 1585an in-pattern limit; they cannot increase it. 1586</P> 1587<P> 1588For non-DFA matching, the minimum <i>depth_limit</i> number is a measure of how 1589much nested backtracking happens (that is, how deeply the pattern's tree is 1590searched). In the case of DFA matching, <i>depth_limit</i> controls the depth of 1591recursive calls of the internal function that is used for handling pattern 1592recursion, lookaround assertions, and atomic groups. 1593</P> 1594<P> 1595For non-DFA matching, the <i>match_limit</i> number is a measure of the amount 1596of backtracking that takes place, and learning the minimum value can be 1597instructive. For most simple matches, the number is quite small, but for 1598patterns with very large numbers of matching possibilities, it can become large 1599very quickly with increasing length of subject string. In the case of DFA 1600matching, <i>match_limit</i> controls the total number of calls, both recursive 1601and non-recursive, to the internal matching function, thus controlling the 1602overall amount of computing resource that is used. 1603</P> 1604<P> 1605For both kinds of matching, the <i>heap_limit</i> number, which is in kibibytes 1606(units of 1024 bytes), limits the amount of heap memory used for matching. A 1607value of zero disables the use of any heap memory; many simple pattern matches 1608can be done without using the heap, so zero is not an unreasonable setting. 1609</P> 1610<br><b> 1611Showing MARK names 1612</b><br> 1613<P> 1614The <b>mark</b> modifier causes the names from backtracking control verbs that 1615are returned from calls to <b>pcre2_match()</b> to be displayed. If a mark is 1616returned for a match, non-match, or partial match, <b>pcre2test</b> shows it. 1617For a match, it is on a line by itself, tagged with "MK:". Otherwise, it 1618is added to the non-match message. 1619</P> 1620<br><b> 1621Showing memory usage 1622</b><br> 1623<P> 1624The <b>memory</b> modifier causes <b>pcre2test</b> to log the sizes of all heap 1625memory allocation and freeing calls that occur during a call to 1626<b>pcre2_match()</b> or <b>pcre2_dfa_match()</b>. These occur only when a match 1627requires a bigger vector than the default for remembering backtracking points 1628(<b>pcre2_match()</b>) or for internal workspace (<b>pcre2_dfa_match()</b>). In 1629many cases there will be no heap memory used and therefore no additional 1630output. No heap memory is allocated during matching with JIT, so in that case 1631the <b>memory</b> modifier never has any effect. For this modifier to work, the 1632<b>null_context</b> modifier must not be set on both the pattern and the 1633subject, though it can be set on one or the other. 1634</P> 1635<br><b> 1636Setting a starting offset 1637</b><br> 1638<P> 1639The <b>offset</b> modifier sets an offset in the subject string at which 1640matching starts. Its value is a number of code units, not characters. 1641</P> 1642<br><b> 1643Setting an offset limit 1644</b><br> 1645<P> 1646The <b>offset_limit</b> modifier sets a limit for unanchored matches. If a match 1647cannot be found starting at or before this offset in the subject, a "no match" 1648return is given. The data value is a number of code units, not characters. When 1649this modifier is used, the <b>use_offset_limit</b> modifier must have been set 1650for the pattern; if not, an error is generated. 1651</P> 1652<br><b> 1653Setting the size of the output vector 1654</b><br> 1655<P> 1656The <b>ovector</b> modifier applies only to the subject line in which it 1657appears, though of course it can also be used to set a default in a 1658<b>#subject</b> command. It specifies the number of pairs of offsets that are 1659available for storing matching information. The default is 15. 1660</P> 1661<P> 1662A value of zero is useful when testing the POSIX API because it causes 1663<b>regexec()</b> to be called with a NULL capture vector. When not testing the 1664POSIX API, a value of zero is used to cause 1665<b>pcre2_match_data_create_from_pattern()</b> to be called, in order to create a 1666match block of exactly the right size for the pattern. (It is not possible to 1667create a match block with a zero-length ovector; there is always at least one 1668pair of offsets.) 1669</P> 1670<br><b> 1671Passing the subject as zero-terminated 1672</b><br> 1673<P> 1674By default, the subject string is passed to a native API matching function with 1675its correct length. In order to test the facility for passing a zero-terminated 1676string, the <b>zero_terminate</b> modifier is provided. It causes the length to 1677be passed as PCRE2_ZERO_TERMINATED. When matching via the POSIX interface, 1678this modifier is ignored, with a warning. 1679</P> 1680<P> 1681When testing <b>pcre2_substitute()</b>, this modifier also has the effect of 1682passing the replacement string as zero-terminated. 1683</P> 1684<br><b> 1685Passing a NULL context, subject, or replacement 1686</b><br> 1687<P> 1688Normally, <b>pcre2test</b> passes a context block to <b>pcre2_match()</b>, 1689<b>pcre2_dfa_match()</b>, <b>pcre2_jit_match()</b> or <b>pcre2_substitute()</b>. 1690If the <b>null_context</b> modifier is set, however, NULL is passed. This is for 1691testing that the matching and substitution functions behave correctly in this 1692case (they use default values). This modifier cannot be used with the 1693<b>find_limits</b> or <b>substitute_callout</b> modifiers. 1694</P> 1695<P> 1696Similarly, for testing purposes, if the <b>null_subject</b> or 1697<b>null_replacement</b> modifier is set, the subject or replacement string 1698pointers are passed as NULL, respectively, to the relevant functions. 1699</P> 1700<br><a name="SEC12" href="#TOC1">THE ALTERNATIVE MATCHING FUNCTION</a><br> 1701<P> 1702By default, <b>pcre2test</b> uses the standard PCRE2 matching function, 1703<b>pcre2_match()</b> to match each subject line. PCRE2 also supports an 1704alternative matching function, <b>pcre2_dfa_match()</b>, which operates in a 1705different way, and has some restrictions. The differences between the two 1706functions are described in the 1707<a href="pcre2matching.html"><b>pcre2matching</b></a> 1708documentation. 1709</P> 1710<P> 1711If the <b>dfa</b> modifier is set, the alternative matching function is used. 1712This function finds all possible matches at a given point in the subject. If, 1713however, the <b>dfa_shortest</b> modifier is set, processing stops after the 1714first match is found. This is always the shortest possible match. 1715</P> 1716<br><a name="SEC13" href="#TOC1">DEFAULT OUTPUT FROM pcre2test</a><br> 1717<P> 1718This section describes the output when the normal matching function, 1719<b>pcre2_match()</b>, is being used. 1720</P> 1721<P> 1722When a match succeeds, <b>pcre2test</b> outputs the list of captured substrings, 1723starting with number 0 for the string that matched the whole pattern. 1724Otherwise, it outputs "No match" when the return is PCRE2_ERROR_NOMATCH, or 1725"Partial match:" followed by the partially matching substring when the 1726return is PCRE2_ERROR_PARTIAL. (Note that this is the 1727entire substring that was inspected during the partial match; it may include 1728characters before the actual match start if a lookbehind assertion, \K, \b, 1729or \B was involved.) 1730</P> 1731<P> 1732For any other return, <b>pcre2test</b> outputs the PCRE2 negative error number 1733and a short descriptive phrase. If the error is a failed UTF string check, the 1734code unit offset of the start of the failing character is also output. Here is 1735an example of an interactive <b>pcre2test</b> run. 1736<pre> 1737 $ pcre2test 1738 PCRE2 version 10.22 2016-07-29 1739 1740 re> /^abc(\d+)/ 1741 data> abc123 1742 0: abc123 1743 1: 123 1744 data> xyz 1745 No match 1746</pre> 1747Unset capturing substrings that are not followed by one that is set are not 1748shown by <b>pcre2test</b> unless the <b>allcaptures</b> modifier is specified. In 1749the following example, there are two capturing substrings, but when the first 1750data line is matched, the second, unset substring is not shown. An "internal" 1751unset substring is shown as "<unset>", as for the second data line. 1752<pre> 1753 re> /(a)|(b)/ 1754 data> a 1755 0: a 1756 1: a 1757 data> b 1758 0: b 1759 1: <unset> 1760 2: b 1761</pre> 1762If the strings contain any non-printing characters, they are output as \xhh 1763escapes if the value is less than 256 and UTF mode is not set. Otherwise they 1764are output as \x{hh...} escapes. See below for the definition of non-printing 1765characters. If the <b>aftertext</b> modifier is set, the output for substring 17660 is followed by the the rest of the subject string, identified by "0+" like 1767this: 1768<pre> 1769 re> /cat/aftertext 1770 data> cataract 1771 0: cat 1772 0+ aract 1773</pre> 1774If global matching is requested, the results of successive matching attempts 1775are output in sequence, like this: 1776<pre> 1777 re> /\Bi(\w\w)/g 1778 data> Mississippi 1779 0: iss 1780 1: ss 1781 0: iss 1782 1: ss 1783 0: ipp 1784 1: pp 1785</pre> 1786"No match" is output only if the first match attempt fails. Here is an example 1787of a failure message (the offset 4 that is specified by the <b>offset</b> 1788modifier is past the end of the subject string): 1789<pre> 1790 re> /xyz/ 1791 data> xyz\=offset=4 1792 Error -24 (bad offset value) 1793</PRE> 1794</P> 1795<P> 1796Note that whereas patterns can be continued over several lines (a plain ">" 1797prompt is used for continuations), subject lines may not. However newlines can 1798be included in a subject by means of the \n escape (or \r, \r\n, etc., 1799depending on the newline sequence setting). 1800</P> 1801<br><a name="SEC14" href="#TOC1">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a><br> 1802<P> 1803When the alternative matching function, <b>pcre2_dfa_match()</b>, is used, the 1804output consists of a list of all the matches that start at the first point in 1805the subject where there is at least one match. For example: 1806<pre> 1807 re> /(tang|tangerine|tan)/ 1808 data> yellow tangerine\=dfa 1809 0: tangerine 1810 1: tang 1811 2: tan 1812</pre> 1813Using the normal matching function on this data finds only "tang". The 1814longest matching string is always given first (and numbered zero). After a 1815PCRE2_ERROR_PARTIAL return, the output is "Partial match:", followed by the 1816partially matching substring. Note that this is the entire substring that was 1817inspected during the partial match; it may include characters before the actual 1818match start if a lookbehind assertion, \b, or \B was involved. (\K is not 1819supported for DFA matching.) 1820</P> 1821<P> 1822If global matching is requested, the search for further matches resumes 1823at the end of the longest match. For example: 1824<pre> 1825 re> /(tang|tangerine|tan)/g 1826 data> yellow tangerine and tangy sultana\=dfa 1827 0: tangerine 1828 1: tang 1829 2: tan 1830 0: tang 1831 1: tan 1832 0: tan 1833</pre> 1834The alternative matching function does not support substring capture, so the 1835modifiers that are concerned with captured substrings are not relevant. 1836</P> 1837<br><a name="SEC15" href="#TOC1">RESTARTING AFTER A PARTIAL MATCH</a><br> 1838<P> 1839When the alternative matching function has given the PCRE2_ERROR_PARTIAL 1840return, indicating that the subject partially matched the pattern, you can 1841restart the match with additional subject data by means of the 1842<b>dfa_restart</b> modifier. For example: 1843<pre> 1844 re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 1845 data> 23ja\=ps,dfa 1846 Partial match: 23ja 1847 data> n05\=dfa,dfa_restart 1848 0: n05 1849</pre> 1850For further information about partial matching, see the 1851<a href="pcre2partial.html"><b>pcre2partial</b></a> 1852documentation. 1853<a name="callouts"></a></P> 1854<br><a name="SEC16" href="#TOC1">CALLOUTS</a><br> 1855<P> 1856If the pattern contains any callout requests, <b>pcre2test</b>'s callout 1857function is called during matching unless <b>callout_none</b> is specified. This 1858works with both matching functions, and with JIT, though there are some 1859differences in behaviour. The output for callouts with numerical arguments and 1860those with string arguments is slightly different. 1861</P> 1862<br><b> 1863Callouts with numerical arguments 1864</b><br> 1865<P> 1866By default, the callout function displays the callout number, the start and 1867current positions in the subject text at the callout time, and the next pattern 1868item to be tested. For example: 1869<pre> 1870 --->pqrabcdef 1871 0 ^ ^ \d 1872</pre> 1873This output indicates that callout number 0 occurred for a match attempt 1874starting at the fourth character of the subject string, when the pointer was at 1875the seventh character, and when the next pattern item was \d. Just 1876one circumflex is output if the start and current positions are the same, or if 1877the current position precedes the start position, which can happen if the 1878callout is in a lookbehind assertion. 1879</P> 1880<P> 1881Callouts numbered 255 are assumed to be automatic callouts, inserted as a 1882result of the <b>auto_callout</b> pattern modifier. In this case, instead of 1883showing the callout number, the offset in the pattern, preceded by a plus, is 1884output. For example: 1885<pre> 1886 re> /\d?[A-E]\*/auto_callout 1887 data> E* 1888 --->E* 1889 +0 ^ \d? 1890 +3 ^ [A-E] 1891 +8 ^^ \* 1892 +10 ^ ^ 1893 0: E* 1894</pre> 1895If a pattern contains (*MARK) items, an additional line is output whenever 1896a change of latest mark is passed to the callout function. For example: 1897<pre> 1898 re> /a(*MARK:X)bc/auto_callout 1899 data> abc 1900 --->abc 1901 +0 ^ a 1902 +1 ^^ (*MARK:X) 1903 +10 ^^ b 1904 Latest Mark: X 1905 +11 ^ ^ c 1906 +12 ^ ^ 1907 0: abc 1908</pre> 1909The mark changes between matching "a" and "b", but stays the same for the rest 1910of the match, so nothing more is output. If, as a result of backtracking, the 1911mark reverts to being unset, the text "<unset>" is output. 1912</P> 1913<br><b> 1914Callouts with string arguments 1915</b><br> 1916<P> 1917The output for a callout with a string argument is similar, except that instead 1918of outputting a callout number before the position indicators, the callout 1919string and its offset in the pattern string are output before the reflection of 1920the subject string, and the subject string is reflected for each callout. For 1921example: 1922<pre> 1923 re> /^ab(?C'first')cd(?C"second")ef/ 1924 data> abcdefg 1925 Callout (7): 'first' 1926 --->abcdefg 1927 ^ ^ c 1928 Callout (20): "second" 1929 --->abcdefg 1930 ^ ^ e 1931 0: abcdef 1932 1933</PRE> 1934</P> 1935<br><b> 1936Callout modifiers 1937</b><br> 1938<P> 1939The callout function in <b>pcre2test</b> returns zero (carry on matching) by 1940default, but you can use a <b>callout_fail</b> modifier in a subject line to 1941change this and other parameters of the callout (see below). 1942</P> 1943<P> 1944If the <b>callout_capture</b> modifier is set, the current captured groups are 1945output when a callout occurs. This is useful only for non-DFA matching, as 1946<b>pcre2_dfa_match()</b> does not support capturing, so no captures are ever 1947shown. 1948</P> 1949<P> 1950The normal callout output, showing the callout number or pattern offset (as 1951described above) is suppressed if the <b>callout_no_where</b> modifier is set. 1952</P> 1953<P> 1954When using the interpretive matching function <b>pcre2_match()</b> without JIT, 1955setting the <b>callout_extra</b> modifier causes additional output from 1956<b>pcre2test</b>'s callout function to be generated. For the first callout in a 1957match attempt at a new starting position in the subject, "New match attempt" is 1958output. If there has been a backtrack since the last callout (or start of 1959matching if this is the first callout), "Backtrack" is output, followed by "No 1960other matching paths" if the backtrack ended the previous match attempt. For 1961example: 1962<pre> 1963 re> /(a+)b/auto_callout,no_start_optimize,no_auto_possess 1964 data> aac\=callout_extra 1965 New match attempt 1966 --->aac 1967 +0 ^ ( 1968 +1 ^ a+ 1969 +3 ^ ^ ) 1970 +4 ^ ^ b 1971 Backtrack 1972 --->aac 1973 +3 ^^ ) 1974 +4 ^^ b 1975 Backtrack 1976 No other matching paths 1977 New match attempt 1978 --->aac 1979 +0 ^ ( 1980 +1 ^ a+ 1981 +3 ^^ ) 1982 +4 ^^ b 1983 Backtrack 1984 No other matching paths 1985 New match attempt 1986 --->aac 1987 +0 ^ ( 1988 +1 ^ a+ 1989 Backtrack 1990 No other matching paths 1991 New match attempt 1992 --->aac 1993 +0 ^ ( 1994 +1 ^ a+ 1995 No match 1996</pre> 1997Notice that various optimizations must be turned off if you want all possible 1998matching paths to be scanned. If <b>no_start_optimize</b> is not used, there is 1999an immediate "no match", without any callouts, because the starting 2000optimization fails to find "b" in the subject, which it knows must be present 2001for any match. If <b>no_auto_possess</b> is not used, the "a+" item is turned 2002into "a++", which reduces the number of backtracks. 2003</P> 2004<P> 2005The <b>callout_extra</b> modifier has no effect if used with the DFA matching 2006function, or with JIT. 2007</P> 2008<br><b> 2009Return values from callouts 2010</b><br> 2011<P> 2012The default return from the callout function is zero, which allows matching to 2013continue. The <b>callout_fail</b> modifier can be given one or two numbers. If 2014there is only one number, 1 is returned instead of 0 (causing matching to 2015backtrack) when a callout of that number is reached. If two numbers (<n>:<m>) 2016are given, 1 is returned when callout <n> is reached and there have been at 2017least <m> callouts. The <b>callout_error</b> modifier is similar, except that 2018PCRE2_ERROR_CALLOUT is returned, causing the entire matching process to be 2019aborted. If both these modifiers are set for the same callout number, 2020<b>callout_error</b> takes precedence. Note that callouts with string arguments 2021are always given the number zero. 2022</P> 2023<P> 2024The <b>callout_data</b> modifier can be given an unsigned or a negative number. 2025This is set as the "user data" that is passed to the matching function, and 2026passed back when the callout function is invoked. Any value other than zero is 2027used as a return from <b>pcre2test</b>'s callout function. 2028</P> 2029<P> 2030Inserting callouts can be helpful when using <b>pcre2test</b> to check 2031complicated regular expressions. For further information about callouts, see 2032the 2033<a href="pcre2callout.html"><b>pcre2callout</b></a> 2034documentation. 2035</P> 2036<br><a name="SEC17" href="#TOC1">NON-PRINTING CHARACTERS</a><br> 2037<P> 2038When <b>pcre2test</b> is outputting text in the compiled version of a pattern, 2039bytes other than 32-126 are always treated as non-printing characters and are 2040therefore shown as hex escapes. 2041</P> 2042<P> 2043When <b>pcre2test</b> is outputting text that is a matched part of a subject 2044string, it behaves in the same way, unless a different locale has been set for 2045the pattern (using the <b>locale</b> modifier). In this case, the 2046<b>isprint()</b> function is used to distinguish printing and non-printing 2047characters. 2048<a name="saverestore"></a></P> 2049<br><a name="SEC18" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br> 2050<P> 2051It is possible to save compiled patterns on disc or elsewhere, and reload them 2052later, subject to a number of restrictions. JIT data cannot be saved. The host 2053on which the patterns are reloaded must be running the same version of PCRE2, 2054with the same code unit width, and must also have the same endianness, pointer 2055width and PCRE2_SIZE type. Before compiled patterns can be saved they must be 2056serialized, that is, converted to a stream of bytes. A single byte stream may 2057contain any number of compiled patterns, but they must all use the same 2058character tables. A single copy of the tables is included in the byte stream 2059(its size is 1088 bytes). 2060</P> 2061<P> 2062The functions whose names begin with <b>pcre2_serialize_</b> are used 2063for serializing and de-serializing. They are described in the 2064<a href="pcre2serialize.html"><b>pcre2serialize</b></a> 2065documentation. In this section we describe the features of <b>pcre2test</b> that 2066can be used to test these functions. 2067</P> 2068<P> 2069Note that "serialization" in PCRE2 does not convert compiled patterns to an 2070abstract format like Java or .NET. It just makes a reloadable byte code stream. 2071Hence the restrictions on reloading mentioned above. 2072</P> 2073<P> 2074In <b>pcre2test</b>, when a pattern with <b>push</b> modifier is successfully 2075compiled, it is pushed onto a stack of compiled patterns, and <b>pcre2test</b> 2076expects the next line to contain a new pattern (or command) instead of a 2077subject line. By contrast, the <b>pushcopy</b> modifier causes a copy of the 2078compiled pattern to be stacked, leaving the original available for immediate 2079matching. By using <b>push</b> and/or <b>pushcopy</b>, a number of patterns can 2080be compiled and retained. These modifiers are incompatible with <b>posix</b>, 2081and control modifiers that act at match time are ignored (with a message) for 2082the stacked patterns. The <b>jitverify</b> modifier applies only at compile 2083time. 2084</P> 2085<P> 2086The command 2087<pre> 2088 #save <filename> 2089</pre> 2090causes all the stacked patterns to be serialized and the result written to the 2091named file. Afterwards, all the stacked patterns are freed. The command 2092<pre> 2093 #load <filename> 2094</pre> 2095reads the data in the file, and then arranges for it to be de-serialized, with 2096the resulting compiled patterns added to the pattern stack. The pattern on the 2097top of the stack can be retrieved by the #pop command, which must be followed 2098by lines of subjects that are to be matched with the pattern, terminated as 2099usual by an empty line or end of file. This command may be followed by a 2100modifier list containing only 2101<a href="#controlmodifiers">control modifiers</a> 2102that act after a pattern has been compiled. In particular, <b>hex</b>, 2103<b>posix</b>, <b>posix_nosub</b>, <b>push</b>, and <b>pushcopy</b> are not allowed, 2104nor are any 2105<a href="#optionmodifiers">option-setting modifiers.</a> 2106The JIT modifiers are, however permitted. Here is an example that saves and 2107reloads two patterns. 2108<pre> 2109 /abc/push 2110 /xyz/push 2111 #save tempfile 2112 #load tempfile 2113 #pop info 2114 xyz 2115 2116 #pop jit,bincode 2117 abc 2118</pre> 2119If <b>jitverify</b> is used with #pop, it does not automatically imply 2120<b>jit</b>, which is different behaviour from when it is used on a pattern. 2121</P> 2122<P> 2123The #popcopy command is analagous to the <b>pushcopy</b> modifier in that it 2124makes current a copy of the topmost stack pattern, leaving the original still 2125on the stack. 2126</P> 2127<br><a name="SEC19" href="#TOC1">SEE ALSO</a><br> 2128<P> 2129<b>pcre2</b>(3), <b>pcre2api</b>(3), <b>pcre2callout</b>(3), 2130<b>pcre2jit</b>, <b>pcre2matching</b>(3), <b>pcre2partial</b>(d), 2131<b>pcre2pattern</b>(3), <b>pcre2serialize</b>(3). 2132</P> 2133<br><a name="SEC20" href="#TOC1">AUTHOR</a><br> 2134<P> 2135Philip Hazel 2136<br> 2137Retired from University Computing Service 2138<br> 2139Cambridge, England. 2140<br> 2141</P> 2142<br><a name="SEC21" href="#TOC1">REVISION</a><br> 2143<P> 2144Last updated: 12 January 2022 2145<br> 2146Copyright © 1997-2022 University of Cambridge. 2147<br> 2148<p> 2149Return to the <a href="index.html">PCRE2 index page</a>. 2150</p> 2151