diff options
| author | ph10 <ph10@2f5784b3-3f2a-0410-8824-cb99058d5e15> | 2013-03-22 16:13:13 +0000 |
|---|---|---|
| committer | ph10 <ph10@2f5784b3-3f2a-0410-8824-cb99058d5e15> | 2013-03-22 16:13:13 +0000 |
| commit | 712f2578028ec79534921d1b06f7b9d0fa1e643b (patch) | |
| tree | 210216515dd0488a595b09cbe3b12bc2353f5645 | |
| parent | 54b46b0215cca9f79390afab565a31db76372d74 (diff) | |
| download | pcre-712f2578028ec79534921d1b06f7b9d0fa1e643b.tar.gz | |
Fix COMMIT in recursion; document backtracking verbs in assertions and
subroutines.
git-svn-id: svn://vcs.exim.org/pcre/code/trunk@1298 2f5784b3-3f2a-0410-8824-cb99058d5e15
| -rw-r--r-- | ChangeLog | 4 | ||||
| -rw-r--r-- | doc/html/index.html | 17 | ||||
| -rw-r--r-- | doc/html/pcreapi.html | 23 | ||||
| -rw-r--r-- | doc/html/pcrecallout.html | 48 | ||||
| -rw-r--r-- | doc/html/pcrecompat.html | 53 | ||||
| -rw-r--r-- | doc/html/pcrejit.html | 16 | ||||
| -rw-r--r-- | doc/html/pcrepartial.html | 81 | ||||
| -rw-r--r-- | doc/html/pcrepattern.html | 269 | ||||
| -rw-r--r-- | doc/html/pcresyntax.html | 6 | ||||
| -rw-r--r-- | doc/html/pcretest.html | 24 | ||||
| -rw-r--r-- | doc/html/pcreunicode.html | 20 | ||||
| -rw-r--r-- | doc/pcre-config.txt | 2 | ||||
| -rw-r--r-- | doc/pcre.txt | 2163 | ||||
| -rw-r--r-- | doc/pcregrep.txt | 4 | ||||
| -rw-r--r-- | doc/pcrepattern.3 | 67 | ||||
| -rw-r--r-- | doc/pcretest.txt | 639 | ||||
| -rw-r--r-- | pcre_exec.c | 12 | ||||
| -rw-r--r-- | testdata/testoutput2 | 2 |
18 files changed, 1899 insertions, 1551 deletions
@@ -118,8 +118,8 @@ Version 8.33 xx-xxxx-201x rather obscure rules do not always do the same thing. (2) Previously, backtracking verbs were confined within assertions. This is - no longer the case. Again, this sometimes improves Perl compatibility, and - sometimes does not. + no longer the case, except for (*ACCEPT). Again, this sometimes improves + Perl compatibility, and sometimes does not. Version 8.32 30-November-2012 diff --git a/doc/html/index.html b/doc/html/index.html index 21e11ea..335703a 100644 --- a/doc/html/index.html +++ b/doc/html/index.html @@ -1,10 +1,10 @@ <html> -<!-- This is a manually maintained file that is the root of the HTML version of - the PCRE documentation. When the HTML documents are built from the man - page versions, the entire doc/html directory is emptied, this file is then - copied into doc/html/index.html, and the remaining files therein are +<!-- This is a manually maintained file that is the root of the HTML version of + the PCRE documentation. When the HTML documents are built from the man + page versions, the entire doc/html directory is emptied, this file is then + copied into doc/html/index.html, and the remaining files therein are created by the 132html script. ---> +--> <head> <title>PCRE specification</title> </head> @@ -94,7 +94,7 @@ in the library. There is a single page for each triple of 8-bit/16-bit/32-bit functions. </p> -<table> +<table> <tr><td><a href="pcre_assign_jit_stack.html">pcre_assign_jit_stack</a></td> <td> Assign stack for JIT matching</td></tr> @@ -146,9 +146,6 @@ functions. <tr><td><a href="pcre_get_substring_list.html">pcre_get_substring_list</a></td> <td> Extract all substrings into new memory</td></tr> -<tr><td><a href="pcre_info.html">pcre_info</a></td> - <td> Obsolete information extraction function</td></tr> - <tr><td><a href="pcre_jit_stack_alloc.html">pcre_jit_stack_alloc</a></td> <td> Create a stack for JIT matching</td></tr> @@ -157,7 +154,7 @@ functions. <tr><td><a href="pcre_maketables.html">pcre_maketables</a></td> <td> Build character tables in current locale</td></tr> - + <tr><td><a href="pcre_pattern_to_host_byte_order.html">pcre_pattern_to_host_byte_order</a></td> <td> Convert compiled pattern to host byte order if necessary</td></tr> diff --git a/doc/html/pcreapi.html b/doc/html/pcreapi.html index 59398df..656cfc1 100644 --- a/doc/html/pcreapi.html +++ b/doc/html/pcreapi.html @@ -1288,9 +1288,14 @@ be used. PCRE_INFO_MAXLOOKBEHIND </pre> Return the number of characters (NB not bytes) in the longest lookbehind -assertion in the pattern. Note that the simple assertions \b and \B require a -one-character lookbehind. This information is useful when doing multi-segment -matching using the partial matching facilities. +assertion in the pattern. This information is useful when doing multi-segment +matching using the partial matching facilities. Note that the simple assertions +\b and \B require a one-character lookbehind. \A also registers a +one-character lookbehind, though it does not actually inspect the previous +character. This is to ensure that at least one character from the old segment +is retained when a new segment is processed. Otherwise, if there are no +lookbehinds in the pattern, \A might match incorrectly at the start of a new +segment. <pre> PCRE_INFO_MINLENGTH </pre> @@ -2332,10 +2337,12 @@ character. The first byte of a character has the value 0xfe or 0xff. These values can never occur in a valid UTF-8 string. <pre> - PCRE_UTF8_ERR2 + PCRE_UTF8_ERR22 </pre> -Non-character. These are the last two characters in each plane (0xfffe, 0xffff, -0x1fffe, 0x1ffff .. 0x10fffe, 0x10ffff), and the characters 0xfdd0..0xfdef. +This error code was formerly used when the presence of a so-called +"non-character" caused an error. Unicode corrigendum #9 makes it clear that +such characters should not cause a string to be rejected, and so this code is +no longer in use and is never returned. </P> <br><a name="SEC18" href="#TOC1">EXTRACTING CAPTURED SUBSTRINGS BY NUMBER</a><br> <P> @@ -2777,9 +2784,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC26" href="#TOC1">REVISION</a><br> <P> -Last updated: 08 November 2012 +Last updated: 27 February 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcrecallout.html b/doc/html/pcrecallout.html index b28e347..da111d7 100644 --- a/doc/html/pcrecallout.html +++ b/doc/html/pcrecallout.html @@ -64,18 +64,23 @@ it is processed as if it were <br> <br> Notice that there is a callout before and after each parenthesis and -alternation bar. Automatic callouts can be used for tracking the progress of -pattern matching. The +alternation bar. If the pattern contains a conditional group whose condition is +an assertion, an automatic callout is inserted immediately before the +condition. Such a callout may also be inserted explicitly, for example: +<pre> + (?(?C9)(?=a)ab|de) +</pre> +This applies only to assertion conditions (because they are themselves +independent groups). +</P> +<P> +Automatic callouts can be used for tracking the progress of pattern matching. +The <a href="pcretest.html"><b>pcretest</b></a> command has an option that sets automatic callouts; when it is used, the output indicates how the pattern is matched. This is useful information when you are trying to optimize the performance of a particular pattern. </P> -<P> -The use of callouts in a pattern makes it ineligible for optimization by the -just-in-time compiler. Studying such a pattern with the PCRE_STUDY_JIT_COMPILE -option always fails. -</P> <br><a name="SEC3" href="#TOC1">MISSING CALLOUTS</a><br> <P> You should be aware that, because of optimizations in the way PCRE matches @@ -141,10 +146,10 @@ automatically generated callouts). <P> The <i>offset_vector</i> field is a pointer to the vector of offsets that was passed by the caller to the matching function. When <b>pcre_exec()</b> or -<b>pcre[16|32]_exec()</b> is used, the contents can be inspected, in order to extract -substrings that have been matched so far, in the same way as for extracting -substrings after a match has completed. For the DFA matching functions, this -field is not useful. +<b>pcre[16|32]_exec()</b> is used, the contents can be inspected, in order to +extract substrings that have been matched so far, in the same way as for +extracting substrings after a match has completed. For the DFA matching +functions, this field is not useful. </P> <P> The <i>subject</i> and <i>subject_length</i> fields contain copies of the values @@ -171,8 +176,10 @@ functions are used, because they do not support captured substrings. </P> <P> The <i>capture_last</i> field contains the number of the most recently captured -substring. If no substrings have been captured, its value is -1. This is always -the case for the DFA matching functions. +substring. However, when a recursion exits, the value reverts to what it was +outside the recursion, as do the values of all captured substrings. If no +substrings have been captured, the value of <i>capture_last</i> is -1. This is +always the case for the DFA matching functions. </P> <P> The <i>callout_data</i> field contains a value that is passed to a matching @@ -203,11 +210,12 @@ same callout number. However, they are set for all callouts. </P> <P> The <i>mark</i> field is present from version 2 of the callout structure. In -callouts from <b>pcre_exec()</b> or <b>pcre[16|32]_exec()</b> it contains a pointer to -the zero-terminated name of the most recently passed (*MARK), (*PRUNE), or -(*THEN) item in the match, or NULL if no such items have been passed. Instances -of (*PRUNE) or (*THEN) without a name do not obliterate a previous (*MARK). In -callouts from the DFA matching functions this field always contains NULL. +callouts from <b>pcre_exec()</b> or <b>pcre[16|32]_exec()</b> it contains a +pointer to the zero-terminated name of the most recently passed (*MARK), +(*PRUNE), or (*THEN) item in the match, or NULL if no such items have been +passed. Instances of (*PRUNE) or (*THEN) without a name do not obliterate a +previous (*MARK). In callouts from the DFA matching functions this field always +contains NULL. </P> <br><a name="SEC5" href="#TOC1">RETURN VALUES</a><br> <P> @@ -234,9 +242,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC7" href="#TOC1">REVISION</a><br> <P> -Last updated: 24 June 2012 +Last updated: 03 March 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcrecompat.html b/doc/html/pcrecompat.html index 0637781..cf18559 100644 --- a/doc/html/pcrecompat.html +++ b/doc/html/pcrecompat.html @@ -36,10 +36,8 @@ these do not seem to have any use. </P> <P> 3. Capturing subpatterns that occur inside negative lookahead assertions are -counted, but their entries in the offsets vector are never set. Perl sets its -numerical variables from any such patterns that are matched before the -assertion fails to match something (thereby succeeding), but only if the -negative lookahead assertion contains just one branch. +counted, but their entries in the offsets vector are never set. Perl sometimes +(but not always) sets its numerical variables from inside negative assertions. </P> <P> 4. Though binary zero characters are supported in the subject string, they are @@ -102,24 +100,32 @@ in the page. </P> <P> -10. If any of the backtracking control verbs are used in an assertion or in a -subpattern that is called as a subroutine (whether or not recursively), their -effect is confined to that subpattern; it does not extend to the surrounding -pattern. This is not always the case in Perl. In particular, if (*THEN) is -present in a group that is called as a subroutine, its action is limited to -that group, even if the group does not contain any | characters. There is one -exception to this: the name from a *(MARK), (*PRUNE), or (*THEN) that is -encountered in a successful positive assertion <i>is</i> passed back when a -match succeeds (compare capturing parentheses in assertions). Note that such -subpatterns are processed as anchored at the point where they are tested. +10. If any of the backtracking control verbs are used in a subpattern that is +called as a subroutine (whether or not recursively), their effect is confined +to that subpattern; it does not extend to the surrounding pattern. This is not +always the case in Perl. In particular, if (*THEN) is present in a group that +is called as a subroutine, its action is limited to that group, even if the +group does not contain any | characters. Note that such subpatterns are +processed as anchored at the point where they are tested. </P> <P> -11. There are some differences that are concerned with the settings of captured +11. If a pattern contains more than one backtracking control verb, the first +one that is backtracked onto acts. For example, in the pattern +A(*COMMIT)B(*PRUNE)C a failure in B triggers (*COMMIT), but a failure in C +triggers (*PRUNE). Perl's behaviour is more complex; in many cases it is the +same as PCRE, but there are examples where it differs. +</P> +<P> +12. Most backtracking verbs in assertions have their normal actions. They are +not confined to the assertion. +</P> +<P> +13. There are some differences that are concerned with the settings of captured strings when part of a pattern is repeated. For example, matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2 unset, but in PCRE it is set to "b". </P> <P> -12. PCRE's handling of duplicate subpattern numbers and duplicate subpattern +14. PCRE's handling of duplicate subpattern numbers and duplicate subpattern names is not as general as Perl's. This is a consequence of the fact the PCRE works internally just with numbers, using an external table to translate between numbers and names. In particular, a pattern such as (?|(?<a>A)|(?<b)B), @@ -130,13 +136,20 @@ names map to capturing subpattern number 1. To avoid this confusing situation, an error is given at compile time. </P> <P> -13. Perl recognizes comments in some places that PCRE does not, for example, +15. Perl recognizes comments in some places that PCRE does not, for example, between the ( and ? at the start of a subpattern. If the /x modifier is set, Perl allows white space between ( and ? but PCRE never does, even if the PCRE_EXTENDED option is set. </P> <P> -14. PCRE provides some extensions to the Perl regular expression facilities. +16. In PCRE, the upper/lower case character properties Lu and Ll are not +affected when case-independent matching is specified. For example, \p{Lu} +always matches an upper case letter. I think Perl has changed in this respect; +in the release at the time of writing (5.16), \p{Lu} and \p{Ll} match all +letters, regardless of case, when case independence is specified. +</P> +<P> +17. PCRE provides some extensions to the Perl regular expression facilities. Perl 5.10 includes new features that are not in earlier versions of Perl, some of which (such as named parentheses) have been in PCRE for some time. This list is with respect to Perl 5.10: @@ -207,9 +220,9 @@ Cambridge CB2 3QH, England. REVISION </b><br> <P> -Last updated: 25 August 2012 +Last updated: 19 March 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcrejit.html b/doc/html/pcrejit.html index 6286fcc..3202506 100644 --- a/doc/html/pcrejit.html +++ b/doc/html/pcrejit.html @@ -172,15 +172,9 @@ PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, PCRE_PARTIAL_HARD, and PCRE_PARTIAL_SOFT. </P> <P> -The unsupported pattern items are: -<pre> - \C match a single byte; not supported in UTF-8 mode - (?Cn) callouts - (*PRUNE) ) - (*SKIP) ) backtracking control verbs - (*THEN) ) -</pre> -Support for some of these may be added in future. +The only unsupported pattern items are \C (match a single data unit) when +running in a UTF mode, and a callout immediately before an assertion condition +in a conditional group. </P> <br><a name="SEC6" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br> <P> @@ -449,9 +443,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC14" href="#TOC1">REVISION</a><br> <P> -Last updated: 31 October 2012 +Last updated: 17 March 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcrepartial.html b/doc/html/pcrepartial.html index 298f92e..1ae06bb 100644 --- a/doc/html/pcrepartial.html +++ b/doc/html/pcrepartial.html @@ -81,33 +81,36 @@ strings. This optimization is also disabled for partial matching. <br><a name="SEC2" href="#TOC1">PARTIAL MATCHING USING pcre_exec() OR pcre[16|32]_exec()</a><br> <P> A partial match occurs during a call to <b>pcre_exec()</b> or -<b>pcre[16|32]_exec()</b> when the end of the subject string is reached successfully, -but matching cannot continue because more characters are needed. However, at -least one character in the subject must have been inspected. This character -need not form part of the final matched string; lookbehind assertions and the -\K escape sequence provide ways of inspecting characters before the start of a -matched substring. The requirement for inspecting at least one character exists -because an empty string can always be matched; without such a restriction there -would always be a partial match of an empty string at the end of the subject. +<b>pcre[16|32]_exec()</b> when the end of the subject string is reached +successfully, but matching cannot continue because more characters are needed. +However, at least one character in the subject must have been inspected. This +character need not form part of the final matched string; lookbehind assertions +and the \K escape sequence provide ways of inspecting characters before the +start of a matched substring. The requirement for inspecting at least one +character exists because an empty string can always be matched; without such a +restriction there would always be a partial match of an empty string at the end +of the subject. </P> <P> If there are at least two slots in the offsets vector when a partial match is returned, the first slot is set to the offset of the earliest character that was inspected. For convenience, the second offset points to the end of the -subject so that a substring can easily be identified. +subject so that a substring can easily be identified. If there are at least +three slots in the offsets vector, the third slot is set to the offset of the +character where matching started. </P> <P> -For the majority of patterns, the first offset identifies the start of the -partially matched string. However, for patterns that contain lookbehind -assertions, or \K, or begin with \b or \B, earlier characters have been -inspected while carrying out the match. For example: +For the majority of patterns, the contents of the first and third slots will be +the same. However, for patterns that contain lookbehind assertions, or begin +with \b or \B, characters before the one where matching started may have been +inspected while carrying out the match. For example, consider this pattern: <pre> /(?<=abc)123/ </pre> This pattern matches "123", but only if it is preceded by "abc". If the subject -string is "xyzabc12", the offsets after a partial match are for the substring -"abc12", because all these characters are needed if another match is tried -with extra characters added to the subject. +string is "xyzabc12", the first two offsets after a partial match are for the +substring "abc12", because all these characters were inspected. However, the +third offset is set to 6, because that is the offset where matching began. </P> <P> What happens when a partial match is identified depends on which of the two @@ -334,10 +337,9 @@ processing time is needed. <P> <b>Note:</b> If the pattern contains lookbehind assertions, or \K, or starts with \b or \B, the string that is returned for a partial match includes -characters that precede the partially matched string itself, because these must -be retained when adding on more characters for a subsequent matching attempt. -However, in some cases you may need to retain even earlier characters, as -discussed in the next section. +characters that precede the start of what would be returned for a complete +match, because it contains all the characters that were inspected during the +partial match. </P> <br><a name="SEC9" href="#TOC1">ISSUES WITH MULTI-SEGMENT MATCHING</a><br> <P> @@ -356,12 +358,35 @@ includes the effect of PCRE_NOTEOL. offsets that are returned for a partial match. However a lookbehind assertion later in the pattern could require even earlier characters to be inspected. You can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the -<b>pcre_fullinfo()</b> or <b>pcre[16|32]_fullinfo()</b> functions to obtain the length -of the largest lookbehind in the pattern. This length is given in characters, -not bytes. If you always retain at least that many characters before the -partially matched string, all should be well. (Of course, near the start of the -subject, fewer characters may be present; in that case all characters should be -retained.) +<b>pcre_fullinfo()</b> or <b>pcre[16|32]_fullinfo()</b> functions to obtain the +length of the longest lookbehind in the pattern. This length is given in +characters, not bytes. If you always retain at least that many characters +before the partially matched string, all should be well. (Of course, near the +start of the subject, fewer characters may be present; in that case all +characters should be retained.) +</P> +<P> +From release 8.33, there is a more accurate way of deciding which characters to +retain. Instead of subtracting the length of the longest lookbehind from the +earliest inspected character (<i>offsets[0]</i>), the match start position +(<i>offsets[2]</i>) should be used, and the next match attempt started at the +<i>offsets[2]</i> character by setting the <i>startoffset</i> argument of +<b>pcre_exec()</b> or <b>pcre_dfa_exec()</b>. +</P> +<P> +For example, if the pattern "(?<=123)abc" is partially +matched against the string "xx123a", the three offset values returned are 2, 6, +and 5. This indicates that the matching process that gave a partial match +started at offset 5, but the characters "123a" were all inspected. The maximum +lookbehind for that pattern is 3, so taking that away from 5 shows that we need +only keep "123a", and the next match attempt can be started at offset 3 (that +is, at "a") when further characters have been added. When the match start is +not the earliest inspected character, <b>pcretest</b> shows it explicitly: +<pre> + re> "(?<=123)abc" + data> xx123a\P\P + Partial match at offset 5: 123a +</PRE> </P> <P> 3. Because a partial match must always contain at least one character, what @@ -465,9 +490,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC11" href="#TOC1">REVISION</a><br> <P> -Last updated: 24 June 2012 +Last updated: 20 February 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcrepattern.html b/doc/html/pcrepattern.html index ee55d06..c9eb7e5 100644 --- a/doc/html/pcrepattern.html +++ b/doc/html/pcrepattern.html @@ -806,7 +806,8 @@ Unicode table. </P> <P> Specifying caseless matching does not affect these escape sequences. For -example, \p{Lu} always matches only upper case letters. +example, \p{Lu} always matches only upper case letters. This is different from +the behaviour of current versions of Perl. </P> <P> Matching characters by Unicode property is not fast, because PCRE has to do a @@ -871,7 +872,8 @@ PCRE's additional properties As well as the standard Unicode properties described above, PCRE supports four more that make it possible to convert traditional escape sequences such as \w and \s and POSIX character classes to use Unicode properties. PCRE uses these -non-standard, non-Perl properties internally when PCRE_UCP is set. They are: +non-standard, non-Perl properties internally when PCRE_UCP is set. However, +they may also be used explicitly. These properties are: <pre> Xan Any alphanumeric character Xps Any POSIX space character @@ -883,6 +885,16 @@ property. Xps matches the characters tab, linefeed, vertical tab, form feed, or carriage return, and any other character that has the Z (separator) property. Xsp is the same as Xps, except that vertical tab is excluded. Xwd matches the same characters as Xan, plus underscore. +</P> +<P> +There is another non-standard property, Xuc, which matches any character that +can be represented by a Universal Character Name in C++ and other programming +languages. These are the characters $, @, ` (grave accent), and all characters +with Unicode code points greater than or equal to U+00A0, except for the +surrogates U+D800 to U+DFFF. Note that most base (ASCII) characters are +excluded. (Universal Character Names are of the form \uHHHH or \UHHHHHHHH +where H is a hexadecimal digit. Note that the Xuc property does not match these +sequences but the characters that they represent.) <a name="resetmatchstart"></a></P> <br><b> Resetting the match start @@ -1950,8 +1962,8 @@ except that it does not cause the current matching position to be changed. Assertion subpatterns are not capturing subpatterns. If such an assertion contains capturing subpatterns within it, these are counted for the purposes of numbering the capturing subpatterns in the whole pattern. However, substring -capturing is carried out only for positive assertions, because it does not make -sense for negative assertions. +capturing is carried out only for positive assertions. (Perl sometimes, but not +always, does do capturing in negative assertions.) </P> <P> For compatibility with Perl, assertion subpatterns may be repeated; though @@ -2605,7 +2617,14 @@ For example, this pattern has two callout points: </pre> If the PCRE_AUTO_CALLOUT flag is passed to a compiling function, callouts are automatically installed before each item in the pattern. They are all numbered -255. +255. If there is a conditional group in the pattern whose condition is an +assertion, an additional callout is inserted just before the condition. An +explicit callout may also be set at this position, as in this example: +<pre> + (?(?C9)(?=a)abc|def) +</pre> +Note that this applies only to assertion conditions, not to other types of +condition. </P> <P> During matching, when PCRE reaches a callout point, the external function is @@ -2620,38 +2639,37 @@ documentation. <br><a name="SEC26" href="#TOC1">BACKTRACKING CONTROL</a><br> <P> Perl 5.10 introduced a number of "Special Backtracking Control Verbs", which -are described in the Perl documentation as "experimental and subject to change -or removal in a future version of Perl". It goes on to say: "Their usage in -production code should be noted to avoid problems during upgrades." The same +are still described in the Perl documentation as "experimental and subject to +change or removal in a future version of Perl". It goes on to say: "Their usage +in production code should be noted to avoid problems during upgrades." The same remarks apply to the PCRE features described in this section. </P> <P> -Since these verbs are specifically related to backtracking, most of them can be -used only when the pattern is to be matched using one of the traditional -matching functions, which use a backtracking algorithm. With the exception of -(*FAIL), which behaves like a failing negative assertion, they cause an error -if encountered by a DFA matching function. -</P> -<P> -If any of these verbs are used in an assertion or in a subpattern that is -called as a subroutine (whether or not recursively), their effect is confined -to that subpattern; it does not extend to the surrounding pattern, with one -exception: the name from a *(MARK), (*PRUNE), or (*THEN) that is encountered in -a successful positive assertion <i>is</i> passed back when a match succeeds -(compare capturing parentheses in assertions). Note that such subpatterns are -processed as anchored at the point where they are tested. Note also that Perl's -treatment of subroutines and assertions is different in some cases. -</P> -<P> The new verbs make use of what was previously invalid syntax: an opening parenthesis followed by an asterisk. They are generally of the form (*VERB) or (*VERB:NAME). Some may take either form, with differing behaviour, -depending on whether or not an argument is present. A name is any sequence of +depending on whether or not a name is present. A name is any sequence of characters that does not include a closing parenthesis. The maximum length of -name is 255 in the 8-bit library and 65535 in the 16-bit and 32-bit library. +name is 255 in the 8-bit library and 65535 in the 16-bit and 32-bit libraries. If the name is empty, that is, if the closing parenthesis immediately follows the colon, the effect is as if the colon were not there. Any number of these verbs may occur in a pattern. +</P> +<P> +Since these verbs are specifically related to backtracking, most of them can be +used only when the pattern is to be matched using one of the traditional +matching functions, because these use a backtracking algorithm. With the +exception of (*FAIL), which behaves like a failing negative assertion, the +backtracking control verbs cause an error if encountered by a DFA matching +function. +</P> +<P> +The behaviour of these verbs in +<a href="#btrepeat">repeated groups,</a> +<a href="#btassert">assertions,</a> +and in +<a href="#btsub">subpatterns called as subroutines</a> +(whether or not recursively) is documented below. <a name="nooptimize"></a></P> <br><b> Optimizations that affect backtracking verbs @@ -2660,7 +2678,7 @@ Optimizations that affect backtracking verbs PCRE contains some optimizations that are used to speed up matching by running some checks at the start of each match attempt. For example, it may know the minimum length of matching subject, or that a particular character must be -present. When one of these optimizations suppresses the running of a match, any +present. When one of these optimizations bypasses the running of a match, any included backtracking verbs will not, of course, be processed. You can suppress the start-of-match optimizations by setting the PCRE_NO_START_OPTIMIZE option when calling <b>pcre_compile()</b> or <b>pcre_exec()</b>, or by starting the @@ -2687,8 +2705,12 @@ followed by a name. This verb causes the match to end successfully, skipping the remainder of the pattern. However, when it is inside a subpattern that is called as a subroutine, only that subpattern is ended successfully. Matching then continues -at the outer level. If (*ACCEPT) is inside capturing parentheses, the data so -far is captured. For example: +at the outer level. If (*ACCEPT) in triggered in a positive assertion, the +assertion succeeds; in a negative assertion, the assertion fails. +</P> +<P> +If (*ACCEPT) is inside capturing parentheses, the data so far is captured. For +example: <pre> A((?:A|B(*ACCEPT)|C)D) </pre> @@ -2722,8 +2744,9 @@ A name is always required with this verb. There may be as many instances of (*MARK) as you like in a pattern, and their names do not have to be unique. </P> <P> -When a match succeeds, the name of the last-encountered (*MARK) on the matching -path is passed back to the caller as described in the section entitled +When a match succeeds, the name of the last-encountered (*MARK:NAME), +(*PRUNE:NAME), or (*THEN:NAME) on the matching path is passed back to the +caller as described in the section entitled <a href="pcreapi.html#extradata">"Extra data for <b>pcre_exec()</b>"</a> in the <a href="pcreapi.html"><b>pcreapi</b></a> @@ -2744,13 +2767,13 @@ of obtaining this information than putting each alternative in its own capturing parentheses. </P> <P> -If (*MARK) is encountered in a positive assertion, its name is recorded and -passed back if it is the last-encountered. This does not happen for negative -assertions. +If a verb with a name is encountered in a positive assertion, its name is +recorded and passed back if it is the last-encountered. This does not happen +for negative assertions. </P> <P> -After a partial match or a failed match, the name of the last encountered -(*MARK) in the entire match process is returned. For example: +After a partial match or a failed match, the last encountered name in the +entire match process is returned. For example: <pre> re> /X(*MARK:A)Y|X(*MARK:B)Z/K data> XP @@ -2774,11 +2797,12 @@ Verbs that act after backtracking The following verbs do nothing when they are encountered. Matching continues with what follows, but if there is no subsequent match, causing a backtrack to the verb, a failure is forced. That is, backtracking cannot pass to the left of -the verb. However, when one of these verbs appears inside an atomic group, its -effect is confined to that group, because once the group has been matched, -there is never any backtracking into it. In this situation, backtracking can -"jump back" to the left of the entire atomic group. (Remember also, as stated -above, that this localization also applies in subroutine calls and assertions.) +the verb. However, when one of these verbs appears inside an atomic group or an +assertion, its effect is confined to that group, because once the group has +been matched, there is never any backtracking into it. In this situation, +backtracking can "jump back" to the left of the entire atomic group or +assertion. (Remember also, as stated above, that this localization also applies +in subroutine calls.) </P> <P> These verbs differ in exactly what kind of failure occurs when backtracking @@ -2787,10 +2811,12 @@ reaches them. (*COMMIT) </pre> This verb, which may not be followed by a name, causes the whole match to fail -outright if the rest of the pattern does not match. Even if the pattern is -unanchored, no further attempts to find a match by advancing the starting point -take place. Once (*COMMIT) has been passed, <b>pcre_exec()</b> is committed to -finding a match at the current starting point, or not at all. For example: +outright if there is a later matching failure that causes backtracking to reach +it. Even if the pattern is unanchored, no further attempts to find a match by +advancing the starting point take place. If (*COMMIT) is the only backtracking +verb that is encountered, once it has been passed <b>pcre_exec()</b> is +committed to finding a match at the current starting point, or not at all. For +example: <pre> a+(*COMMIT)b </pre> @@ -2800,6 +2826,11 @@ recently passed (*MARK) in the path is passed back when (*COMMIT) forces a match failure. </P> <P> +If there is more than one backtracking verb in a pattern, a different one that +follows (*COMMIT) may be triggered first, so merely passing (*COMMIT) during a +match does not always guarantee that a match must be at this starting point. +</P> +<P> Note that (*COMMIT) at the start of a pattern is not the same as an anchor, unless PCRE's start-of-match optimizations are turned off, as shown in this <b>pcretest</b> example: @@ -2819,15 +2850,20 @@ starting points. (*PRUNE) or (*PRUNE:NAME) </pre> This verb causes the match to fail at the current starting position in the -subject if the rest of the pattern does not match. If the pattern is -unanchored, the normal "bumpalong" advance to the next starting character then -happens. Backtracking can occur as usual to the left of (*PRUNE), before it is -reached, or when matching to the right of (*PRUNE), but if there is no match to -the right, backtracking cannot cross (*PRUNE). In simple cases, the use of -(*PRUNE) is just an alternative to an atomic group or possessive quantifier, -but there are some uses of (*PRUNE) that cannot be expressed in any other way. -The behaviour of (*PRUNE:NAME) is the same as (*MARK:NAME)(*PRUNE). In an -anchored pattern (*PRUNE) has the same effect as (*COMMIT). +subject if there is a later matching failure that causes backtracking to reach +it. If the pattern is unanchored, the normal "bumpalong" advance to the next +starting character then happens. Backtracking can occur as usual to the left of +(*PRUNE), before it is reached, or when matching to the right of (*PRUNE), but +if there is no match to the right, backtracking cannot cross (*PRUNE). In +simple cases, the use of (*PRUNE) is just an alternative to an atomic group or +possessive quantifier, but there are some uses of (*PRUNE) that cannot be +expressed in any other way. In an anchored pattern (*PRUNE) has the same effect +as (*COMMIT). +</P> +<P> +The behaviour of (*PRUNE:NAME) is the not the same as (*MARK:NAME)(*PRUNE). +It is like (*MARK:NAME) in that the name is remembered for passing back to the +caller. However, (*SKIP:NAME) searches only for names set with (*MARK). <pre> (*SKIP) </pre> @@ -2848,31 +2884,39 @@ instead of skipping on to "c". <pre> (*SKIP:NAME) </pre> -When (*SKIP) has an associated name, its behaviour is modified. If the -following pattern fails to match, the previous path through the pattern is -searched for the most recent (*MARK) that has the same name. If one is found, -the "bumpalong" advance is to the subject position that corresponds to that -(*MARK) instead of to where (*SKIP) was encountered. If no (*MARK) with a -matching name is found, the (*SKIP) is ignored. +When (*SKIP) has an associated name, its behaviour is modified. When it is +triggered, the previous path through the pattern is searched for the most +recent (*MARK) that has the same name. If one is found, the "bumpalong" advance +is to the subject position that corresponds to that (*MARK) instead of to where +(*SKIP) was encountered. If no (*MARK) with a matching name is found, the +(*SKIP) is ignored. +</P> +<P> +Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It ignores +names that are set by (*PRUNE:NAME) or (*THEN:NAME). <pre> (*THEN) or (*THEN:NAME) </pre> -This verb causes a skip to the next innermost alternative if the rest of the -pattern does not match. That is, it cancels pending backtracking, but only -within the current alternative. Its name comes from the observation that it can -be used for a pattern-based if-then-else block: +This verb causes a skip to the next innermost alternative when backtracking +reaches it. That is, it cancels any further backtracking within the current +alternative. Its name comes from the observation that it can be used for a +pattern-based if-then-else block: <pre> ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ... </pre> If the COND1 pattern matches, FOO is tried (and possibly further items after the end of the group if FOO succeeds); on failure, the matcher skips to the -second alternative and tries COND2, without backtracking into COND1. The -behaviour of (*THEN:NAME) is exactly the same as (*MARK:NAME)(*THEN). +second alternative and tries COND2, without backtracking into COND1. If (*THEN) is not inside an alternation, it acts like (*PRUNE). </P> <P> -Note that a subpattern that does not contain a | character is just a part of -the enclosing alternative; it is not a nested alternation with only one +The behaviour of (*THEN:NAME) is the not the same as (*MARK:NAME)(*THEN). +It is like (*MARK:NAME) in that the name is remembered for passing back to the +caller. However, (*SKIP:NAME) searches only for names set with (*MARK). +</P> +<P> +A subpattern that does not contain a | character is just a part of the +enclosing alternative; it is not a nested alternation with only one alternative. The effect of (*THEN) extends beyond such a subpattern to the enclosing alternative. Consider this pattern, where A, B, etc. are complex pattern fragments that do not contain any | characters at this level: @@ -2892,7 +2936,7 @@ because there are no more alternatives to try. In this case, matching does now backtrack into A. </P> <P> -Note also that a conditional subpattern is not considered as having two +Note that a conditional subpattern is not considered as having two alternatives, because only one is ever used. In other words, the | character in a conditional subpattern has a different meaning. Ignoring white space, consider: @@ -2916,17 +2960,82 @@ unanchored pattern). (*SKIP) is similar, except that the advance may be more than one character. (*COMMIT) is the strongest, causing the entire match to fail. </P> +<br><b> +More than one backtracking verb +</b><br> +<P> +If more than one backtracking verb is present in a pattern, the one that is +backtracked onto first acts. For example, consider this pattern, where A, B, +etc. are complex pattern fragments: +<pre> + (A(*COMMIT)B(*THEN)C|ABD) +</pre> +If A matches but B fails, the backtrack to (*COMMIT) causes the entire match to +fail. However, if A and B match, but C fails, the backtrack to (*THEN) causes +the next alternative (ABD) to be tried. This behaviour is consistent, but is +not always the same as Perl's. It means that if two or more backtracking verbs +appear in succession, all the the last of them has no effect. Consider this +example: +<pre> + ...(*COMMIT)(*PRUNE)... +</pre> +If there is a matching failure to the right, backtracking onto (*PRUNE) cases +it to be triggered, and its action is taken. There can never be a backtrack +onto (*COMMIT). +<a name="btrepeat"></a></P> +<br><b> +Backtracking verbs in repeated groups +</b><br> <P> -If more than one such verb is present in a pattern, the "strongest" one wins. -For example, consider this pattern, where A, B, etc. are complex pattern -fragments: +PCRE differs from Perl in its handling of backtracking verbs in repeated +groups. For example, consider: <pre> - (A(*COMMIT)B(*THEN)C|D) + /(a(*COMMIT)b)+ac/ </pre> -Once A has matched, PCRE is committed to this match, at the current starting -position. If subsequently B matches, but C does not, the normal (*THEN) action -of trying the next alternative (that is, D) does not happen because (*COMMIT) -overrides. +If the subject is "abac", Perl matches, but PCRE fails because the (*COMMIT) in +the second repeat of the group acts. +<a name="btassert"></a></P> +<br><b> +Backtracking verbs in assertions +</b><br> +<P> +(*FAIL) in an assertion has its normal effect: it forces an immediate backtrack. +</P> +<P> +(*ACCEPT) in a positive assertion causes the assertion to succeed without any +further processing. In a negative assertion, (*ACCEPT) causes the assertion to +fail without any further processing. +</P> +<P> +The other backtracking verbs are not treated specially if they appear in an +assertion. In particular, (*THEN) skips to the next alternative in the +innermost enclosing group that has alternations, whether or not this is within +the assertion. +<a name="btsub"></a></P> +<br><b> +Backtracking verbs in subroutines +</b><br> +<P> +These behaviours occur whether or not the subpattern is called recursively. +Perl's treatment of subroutines is different in some cases. +</P> +<P> +(*FAIL) in a subpattern called as a subroutine has its normal effect: it forces +an immediate backtrack. +</P> +<P> +(*ACCEPT) in a subpattern called as a subroutine causes the subroutine match to +succeed without any further processing. Matching then continues after the +subroutine call. +</P> +<P> +(*COMMIT), (*SKIP), and (*PRUNE) in a subpattern called as a subroutine cause +the subroutine match to fail. +</P> +<P> +(*THEN) skips to the next alternative in the innermost enclosing group within +the subpattern that has alternatives. If there is no such group within the +subpattern, (*THEN) causes the subroutine match to fail. </P> <br><a name="SEC27" href="#TOC1">SEE ALSO</a><br> <P> @@ -2944,9 +3053,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC29" href="#TOC1">REVISION</a><br> <P> -Last updated: 11 November 2012 +Last updated: 22 March 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcresyntax.html b/doc/html/pcresyntax.html index 22a253e..b35cfa9 100644 --- a/doc/html/pcresyntax.html +++ b/doc/html/pcresyntax.html @@ -151,6 +151,8 @@ PCRE_UCP option. Xan Alphanumeric: union of properties L and N Xps POSIX space: property Z or tab, NL, VT, FF, CR Xsp Perl space: property Z or tab, NL, FF, CR + Xuc Univerally-named character: one that can be + represented by a Universal Character Name Xwd Perl word: property Xan or underscore </PRE> </P> @@ -512,9 +514,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC27" href="#TOC1">REVISION</a><br> <P> -Last updated: 11 November 2012 +Last updated: 27 February 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcretest.html b/doc/html/pcretest.html index 8423ba5..e27ef3f 100644 --- a/doc/html/pcretest.html +++ b/doc/html/pcretest.html @@ -110,23 +110,27 @@ internal form is output after compilation. <P> <b>-C</b> Output the version number of the PCRE library, and all available information -about the optional features that are included, and then exit. All other options -are ignored. +about the optional features that are included, and then exit with zero exit +code. All other options are ignored. </P> <P> <b>-C</b> <i>option</i> Output information about a specific build-time option, then exit. This functionality is intended for use in scripts such as <b>RunTest</b>. The -following options output the value indicated: +following options output the value and set the exit code as indicated: <pre> ebcdic-nl the code for LF (= NL) in an EBCDIC environment: 0x15 or 0x25 0 if used in an ASCII environment - linksize the internal link size (2, 3, or 4) + exit code is always 0 + linksize the configured internal link size (2, 3, or 4) + exit code is set to the link size newline the default newline setting: CR, LF, CRLF, ANYCRLF, or ANY + exit code is always 0 </pre> -The following options output 1 for true or zero for false: +The following options output 1 for true or 0 for false, and set the exit code +to the same value: <pre> ebcdic compiled for an EBCDIC environment jit just-in-time support is available @@ -134,8 +138,10 @@ The following options output 1 for true or zero for false: pcre32 the 32-bit library was built pcre8 the 8-bit library was built ucp Unicode property support is available - utf UTF-8 and/or UTF-16 and/or UTF-32 support is available -</PRE> + utf UTF-8 and/or UTF-16 and/or UTF-32 support + is available +</pre> +If an unknown option is given, an error message is output; the exit code is 0. </P> <P> <b>-d</b> @@ -1073,9 +1079,9 @@ Cambridge CB2 3QH, England. </P> <br><a name="SEC16" href="#TOC1">REVISION</a><br> <P> -Last updated: 10 September 2012 +Last updated: 22 February 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/html/pcreunicode.html b/doc/html/pcreunicode.html index c3c51ea..1b05b19 100644 --- a/doc/html/pcreunicode.html +++ b/doc/html/pcreunicode.html @@ -85,7 +85,9 @@ place. From release 7.3 of PCRE, the check is according the rules of RFC 3629, which are themselves derived from the Unicode specification. Earlier releases of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0 -to U+10FFFF, excluding the surrogate area and the non-characters. +to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called +"non-character" code points are no longer excluded because Unicode corrigendum +#9 makes it clear that they should not be.) </P> <P> Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16, @@ -96,10 +98,6 @@ surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and UTF-32.) </P> <P> -Also excluded are the "Non-Character" code points, which are U+FDD0 to U+FDEF -and the last two code points in each plane, U+??FFFE and U+??FFFF. -</P> -<P> If an invalid UTF-8 string is passed to PCRE, an error return is given. At compile time, the only additional information is the offset to the first byte of the failing character. The run-time functions <b>pcre_exec()</b> and @@ -135,10 +133,6 @@ U+D800 to U+DFFF are independent code points. Values in the surrogate range must be used in pairs in the correct manner. </P> <P> -Excluded are the "Non-Character" code points, which are U+FDD0 to U+FDEF -and the last two code points in each plane, U+??FFFE and U+??FFFF. -</P> -<P> If an invalid UTF-16 string is passed to PCRE, an error return is given. At compile time, the only additional information is the offset to the first data unit of the failing character. The run-time functions <b>pcre16_exec()</b> and @@ -160,9 +154,7 @@ Validity of UTF-32 strings When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are passed as patterns and subjects are (by default) checked for validity on entry to the relevant functions. This check allows only values in the range U+0 -to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF, and the -"Non-Character" code points, which are U+FDD0 to U+FDEF and the last two -characters in each plane, U+??FFFE and U+??FFFF. +to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF. </P> <P> If an invalid UTF-32 string is passed to PCRE, an error return is given. At @@ -261,9 +253,9 @@ Cambridge CB2 3QH, England. REVISION </b><br> <P> -Last updated: 11 November 2012 +Last updated: 27 February 2013 <br> -Copyright © 1997-2012 University of Cambridge. +Copyright © 1997-2013 University of Cambridge. <br> <p> Return to the <a href="index.html">PCRE index page</a>. diff --git a/doc/pcre-config.txt b/doc/pcre-config.txt index 3ee9777..97a6abe 100644 --- a/doc/pcre-config.txt +++ b/doc/pcre-config.txt @@ -1,4 +1,4 @@ -PCRE-CONFIG(1) PCRE-CONFIG(1) +PCRE-CONFIG(1) General Commands Manual PCRE-CONFIG(1) diff --git a/doc/pcre.txt b/doc/pcre.txt index 2a2a82c..caba20c 100644 --- a/doc/pcre.txt +++ b/doc/pcre.txt @@ -8,13 +8,13 @@ pcretest commands. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) +PCRE(3) Library Functions Manual PCRE(3) + NAME PCRE - Perl-compatible regular expressions - INTRODUCTION The PCRE library is a set of functions that implement regular expres- @@ -177,10 +177,11 @@ REVISION Last updated: 11 November 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRE(3) Library Functions Manual PCRE(3) -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions @@ -506,10 +507,11 @@ REVISION Last updated: 08 November 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRE(3) Library Functions Manual PCRE(3) -PCRE(3) PCRE(3) - NAME PCRE - Perl-compatible regular expressions @@ -832,15 +834,15 @@ REVISION Last updated: 08 November 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREBUILD(3) Library Functions Manual PCREBUILD(3) -PCREBUILD(3) PCREBUILD(3) - NAME PCRE - Perl-compatible regular expressions - PCRE BUILD-TIME OPTIONS This document describes the optional features of PCRE that can be @@ -1322,15 +1324,15 @@ REVISION Last updated: 30 October 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREMATCHING(3) Library Functions Manual PCREMATCHING(3) -PCREMATCHING(3) PCREMATCHING(3) - NAME PCRE - Perl-compatible regular expressions - PCRE MATCHING ALGORITHMS This document describes the two different algorithms that are available @@ -1531,10 +1533,11 @@ REVISION Last updated: 08 January 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREAPI(3) Library Functions Manual PCREAPI(3) -PCREAPI(3) PCREAPI(3) - NAME PCRE - Perl-compatible regular expressions @@ -2707,67 +2710,72 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_MAXLOOKBEHIND Return the number of characters (NB not bytes) in the longest lookbe- - hind assertion in the pattern. Note that the simple assertions \b and - \B require a one-character lookbehind. This information is useful when - doing multi-segment matching using the partial matching facilities. + hind assertion in the pattern. This information is useful when doing + multi-segment matching using the partial matching facilities. Note that + the simple assertions \b and \B require a one-character lookbehind. \A + also registers a one-character lookbehind, though it does not actually + inspect the previous character. This is to ensure that at least one + character from the old segment is retained when a new segment is pro- + cessed. Otherwise, if there are no lookbehinds in the pattern, \A might + match incorrectly at the start of a new segment. PCRE_INFO_MINLENGTH - If the pattern was studied and a minimum length for matching subject - strings was computed, its value is returned. Otherwise the returned - value is -1. The value is a number of characters, which in UTF-8 mode - may be different from the number of bytes. The fourth argument should - point to an int variable. A non-negative value is a lower bound to the - length of any matching string. There may not be any strings of that - length that do actually match, but every string that does match is at + If the pattern was studied and a minimum length for matching subject + strings was computed, its value is returned. Otherwise the returned + value is -1. The value is a number of characters, which in UTF-8 mode + may be different from the number of bytes. The fourth argument should + point to an int variable. A non-negative value is a lower bound to the + length of any matching string. There may not be any strings of that + length that do actually match, but every string that does match is at least that long. PCRE_INFO_NAMECOUNT PCRE_INFO_NAMEENTRYSIZE PCRE_INFO_NAMETABLE - PCRE supports the use of named as well as numbered capturing parenthe- - ses. The names are just an additional way of identifying the parenthe- + PCRE supports the use of named as well as numbered capturing parenthe- + ses. The names are just an additional way of identifying the parenthe- ses, which still acquire numbers. Several convenience functions such as - pcre_get_named_substring() are provided for extracting captured sub- - strings by name. It is also possible to extract the data directly, by - first converting the name to a number in order to access the correct + pcre_get_named_substring() are provided for extracting captured sub- + strings by name. It is also possible to extract the data directly, by + first converting the name to a number in order to access the correct pointers in the output vector (described with pcre_exec() below). To do - the conversion, you need to use the name-to-number map, which is + the conversion, you need to use the name-to-number map, which is described by these three values. The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size - of each entry; both of these return an int value. The entry size - depends on the length of the longest name. PCRE_INFO_NAMETABLE returns + of each entry; both of these return an int value. The entry size + depends on the length of the longest name. PCRE_INFO_NAMETABLE returns a pointer to the first entry of the table. This is a pointer to char in the 8-bit library, where the first two bytes of each entry are the num- - ber of the capturing parenthesis, most significant byte first. In the - 16-bit library, the pointer points to 16-bit data units, the first of - which contains the parenthesis number. In the 32-bit library, the - pointer points to 32-bit data units, the first of which contains the - parenthesis number. The rest of the entry is the corresponding name, + ber of the capturing parenthesis, most significant byte first. In the + 16-bit library, the pointer points to 16-bit data units, the first of + which contains the parenthesis number. In the 32-bit library, the + pointer points to 32-bit data units, the first of which contains the + parenthesis number. The rest of the entry is the corresponding name, zero terminated. - The names are in alphabetical order. Duplicate names may appear if (?| + The names are in alphabetical order. Duplicate names may appear if (?| is used to create multiple groups with the same number, as described in - the section on duplicate subpattern numbers in the pcrepattern page. - Duplicate names for subpatterns with different numbers are permitted - only if PCRE_DUPNAMES is set. In all cases of duplicate names, they - appear in the table in the order in which they were found in the pat- - tern. In the absence of (?| this is the order of increasing number; + the section on duplicate subpattern numbers in the pcrepattern page. + Duplicate names for subpatterns with different numbers are permitted + only if PCRE_DUPNAMES is set. In all cases of duplicate names, they + appear in the table in the order in which they were found in the pat- + tern. In the absence of (?| this is the order of increasing number; when (?| is used this is not necessarily the case because later subpat- terns may have lower numbers. - As a simple example of the name/number table, consider the following + As a simple example of the name/number table, consider the following pattern after compilation by the 8-bit library (assume PCRE_EXTENDED is set, so white space - including newlines - is ignored): (?<date> (?<year>(\d\d)?\d\d) - (?<month>\d\d) - (?<day>\d\d) ) - There are four named subpatterns, so the table has four entries, and - each entry in the table is eight bytes long. The table is as follows, + There are four named subpatterns, so the table has four entries, and + each entry in the table is eight bytes long. The table is as follows, with non-printing bytes shows in hexadecimal, and undefined bytes shown as ??: @@ -2776,31 +2784,31 @@ INFORMATION ABOUT A PATTERN 00 04 m o n t h 00 00 02 y e a r 00 ?? - When writing code to extract data from named subpatterns using the - name-to-number map, remember that the length of the entries is likely + When writing code to extract data from named subpatterns using the + name-to-number map, remember that the length of the entries is likely to be different for each compiled pattern. PCRE_INFO_OKPARTIAL - Return 1 if the pattern can be used for partial matching with - pcre_exec(), otherwise 0. The fourth argument should point to an int - variable. From release 8.00, this always returns 1, because the - restrictions that previously applied to partial matching have been - lifted. The pcrepartial documentation gives details of partial match- + Return 1 if the pattern can be used for partial matching with + pcre_exec(), otherwise 0. The fourth argument should point to an int + variable. From release 8.00, this always returns 1, because the + restrictions that previously applied to partial matching have been + lifted. The pcrepartial documentation gives details of partial match- ing. PCRE_INFO_OPTIONS - Return a copy of the options with which the pattern was compiled. The - fourth argument should point to an unsigned long int variable. These + Return a copy of the options with which the pattern was compiled. The + fourth argument should point to an unsigned long int variable. These option bits are those specified in the call to pcre_compile(), modified by any top-level option settings at the start of the pattern itself. In - other words, they are the options that will be in force when matching - starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with - the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE, + other words, they are the options that will be in force when matching + starts. For example, if the pattern /(?im)abc(?-i)d/ is compiled with + the PCRE_EXTENDED option, the result is PCRE_CASELESS, PCRE_MULTILINE, and PCRE_EXTENDED. - A pattern is automatically anchored by PCRE if all of its top-level + A pattern is automatically anchored by PCRE if all of its top-level alternatives begin with one of the following: ^ unless PCRE_MULTILINE is set @@ -2814,40 +2822,40 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE - Return the size of the compiled pattern in bytes (for both libraries). - The fourth argument should point to a size_t variable. This value does - not include the size of the pcre structure that is returned by - pcre_compile(). The value that is passed as the argument to pcre_mal- - loc() when pcre_compile() is getting memory in which to place the com- - piled data is the value returned by this option plus the size of the - pcre structure. Studying a compiled pattern, with or without JIT, does + Return the size of the compiled pattern in bytes (for both libraries). + The fourth argument should point to a size_t variable. This value does + not include the size of the pcre structure that is returned by + pcre_compile(). The value that is passed as the argument to pcre_mal- + loc() when pcre_compile() is getting memory in which to place the com- + piled data is the value returned by this option plus the size of the + pcre structure. Studying a compiled pattern, with or without JIT, does not alter the value returned by this option. PCRE_INFO_STUDYSIZE Return the size in bytes of the data block pointed to by the study_data - field in a pcre_extra block. If pcre_extra is NULL, or there is no - study data, zero is returned. The fourth argument should point to a - size_t variable. The study_data field is set by pcre_study() to record - information that will speed up matching (see the section entitled + field in a pcre_extra block. If pcre_extra is NULL, or there is no + study data, zero is returned. The fourth argument should point to a + size_t variable. The study_data field is set by pcre_study() to record + information that will speed up matching (see the section entitled "Studying a pattern" above). The format of the study_data block is pri- - vate, but its length is made available via this option so that it can + vate, but its length is made available via this option so that it can be saved and restored (see the pcreprecompile documentation for details). PCRE_INFO_FIRSTCHARACTERFLAGS Return information about the first data unit of any matched string, for - a non-anchored pattern. The fourth argument should point to an int + a non-anchored pattern. The fourth argument should point to an int variable. - If there is a fixed first value, for example, the letter "c" from a - pattern such as (cat|cow|coyote), 1 is returned, and the character + If there is a fixed first value, for example, the letter "c" from a + pattern such as (cat|cow|coyote), 1 is returned, and the character value can be retrieved using PCRE_INFO_FIRSTCHARACTER. If there is no fixed first value, and if either - (a) the pattern was compiled with the PCRE_MULTILINE option, and every + (a) the pattern was compiled with the PCRE_MULTILINE option, and every branch starts with "^", or (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not @@ -2859,45 +2867,45 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_FIRSTCHARACTER - Return the fixed first character value, if PCRE_INFO_FIRSTCHARACTER- + Return the fixed first character value, if PCRE_INFO_FIRSTCHARACTER- FLAGS returned 1; otherwise returns 0. The fourth argument should point to an uint_t variable. - In the 8-bit library, the value is always less than 256. In the 16-bit - library the value can be up to 0xffff. In the 32-bit library in UTF-32 - mode the value can be up to 0x10ffff, and up to 0xffffffff when not + In the 8-bit library, the value is always less than 256. In the 16-bit + library the value can be up to 0xffff. In the 32-bit library in UTF-32 + mode the value can be up to 0x10ffff, and up to 0xffffffff when not using UTF-32 mode. If there is no fixed first value, and if either - (a) the pattern was compiled with the PCRE_MULTILINE option, and every + (a) the pattern was compiled with the PCRE_MULTILINE option, and every branch starts with "^", or (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not set (if it were set, the pattern would be anchored), - -1 is returned, indicating that the pattern matches only at the start - of a subject string or after any newline within the string. Otherwise + -1 is returned, indicating that the pattern matches only at the start + of a subject string or after any newline within the string. Otherwise -2 is returned. For anchored patterns, -2 is returned. PCRE_INFO_REQUIREDCHARFLAGS - Returns 1 if there is a rightmost literal data unit that must exist in + Returns 1 if there is a rightmost literal data unit that must exist in any matched string, other than at its start. The fourth argument should - point to an int variable. If there is no such value, 0 is returned. If + point to an int variable. If there is no such value, 0 is returned. If returning 1, the character value itself can be retrieved using PCRE_INFO_REQUIREDCHAR. For anchored patterns, a last literal value is recorded only if it fol- - lows something of variable length. For example, for the pattern - /^a\d+z\d+/ the returned value 1 (with "z" returned from + lows something of variable length. For example, for the pattern + /^a\d+z\d+/ the returned value 1 (with "z" returned from PCRE_INFO_REQUIREDCHAR), but for /^a\dz\d/ the returned value is 0. PCRE_INFO_REQUIREDCHAR - Return the value of the rightmost literal data unit that must exist in - any matched string, other than at its start, if such a value has been - recorded. The fourth argument should point to an uint32_t variable. If + Return the value of the rightmost literal data unit that must exist in + any matched string, other than at its start, if such a value has been + recorded. The fourth argument should point to an uint32_t variable. If there is no such value, 0 is returned. @@ -2905,21 +2913,21 @@ REFERENCE COUNTS int pcre_refcount(pcre *code, int adjust); - The pcre_refcount() function is used to maintain a reference count in + The pcre_refcount() function is used to maintain a reference count in the data block that contains a compiled pattern. It is provided for the - benefit of applications that operate in an object-oriented manner, + benefit of applications that operate in an object-oriented manner, where different parts of the application may be using the same compiled pattern, but you want to free the block when they are all done. When a pattern is compiled, the reference count field is initialized to - zero. It is changed only by calling this function, whose action is to - add the adjust value (which may be positive or negative) to it. The + zero. It is changed only by calling this function, whose action is to + add the adjust value (which may be positive or negative) to it. The yield of the function is the new value. However, the value of the count - is constrained to lie between 0 and 65535, inclusive. If the new value + is constrained to lie between 0 and 65535, inclusive. If the new value is outside these limits, it is forced to the appropriate limit value. - Except when it is zero, the reference count is not correctly preserved - if a pattern is compiled on one host and then transferred to a host + Except when it is zero, the reference count is not correctly preserved + if a pattern is compiled on one host and then transferred to a host whose byte-order is different. (This seems a highly unlikely scenario.) @@ -2929,22 +2937,22 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION const char *subject, int length, int startoffset, int options, int *ovector, int ovecsize); - The function pcre_exec() is called to match a subject string against a - compiled pattern, which is passed in the code argument. If the pattern - was studied, the result of the study should be passed in the extra - argument. You can call pcre_exec() with the same code and extra argu- - ments as many times as you like, in order to match different subject + The function pcre_exec() is called to match a subject string against a + compiled pattern, which is passed in the code argument. If the pattern + was studied, the result of the study should be passed in the extra + argument. You can call pcre_exec() with the same code and extra argu- + ments as many times as you like, in order to match different subject strings with the same pattern. - This function is the main matching facility of the library, and it - operates in a Perl-like manner. For specialist use there is also an - alternative matching function, which is described below in the section + This function is the main matching facility of the library, and it + operates in a Perl-like manner. For specialist use there is also an + alternative matching function, which is described below in the section about the pcre_dfa_exec() function. - In most applications, the pattern will have been compiled (and option- - ally studied) in the same process that calls pcre_exec(). However, it + In most applications, the pattern will have been compiled (and option- + ally studied) in the same process that calls pcre_exec(). However, it is possible to save compiled patterns and study data, and then use them - later in different processes, possibly even on different hosts. For a + later in different processes, possibly even on different hosts. For a discussion about this, see the pcreprecompile documentation. Here is an example of a simple call to pcre_exec(): @@ -2963,10 +2971,10 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION Extra data for pcre_exec() - If the extra argument is not NULL, it must point to a pcre_extra data - block. The pcre_study() function returns such a block (when it doesn't - return NULL), but you can also create one for yourself, and pass addi- - tional information in it. The pcre_extra block contains the following + If the extra argument is not NULL, it must point to a pcre_extra data + block. The pcre_study() function returns such a block (when it doesn't + return NULL), but you can also create one for yourself, and pass addi- + tional information in it. The pcre_extra block contains the following fields (not necessarily in this order): unsigned long int flags; @@ -2978,13 +2986,13 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION const unsigned char *tables; unsigned char **mark; - In the 16-bit version of this structure, the mark field has type + In the 16-bit version of this structure, the mark field has type "PCRE_UCHAR16 **". - In the 32-bit version of this structure, the mark field has type + In the 32-bit version of this structure, the mark field has type "PCRE_UCHAR32 **". - The flags field is used to specify which of the other fields are set. + The flags field is used to specify which of the other fields are set. The flag bits are: PCRE_EXTRA_CALLOUT_DATA @@ -2995,112 +3003,112 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_EXTRA_STUDY_DATA PCRE_EXTRA_TABLES - Other flag bits should be set to zero. The study_data field and some- - times the executable_jit field are set in the pcre_extra block that is - returned by pcre_study(), together with the appropriate flag bits. You - should not set these yourself, but you may add to the block by setting + Other flag bits should be set to zero. The study_data field and some- + times the executable_jit field are set in the pcre_extra block that is + returned by pcre_study(), together with the appropriate flag bits. You + should not set these yourself, but you may add to the block by setting other fields and their corresponding flag bits. The match_limit field provides a means of preventing PCRE from using up - a vast amount of resources when running patterns that are not going to - match, but which have a very large number of possibilities in their - search trees. The classic example is a pattern that uses nested unlim- + a vast amount of resources when running patterns that are not going to + match, but which have a very large number of possibilities in their + search trees. The classic example is a pattern that uses nested unlim- ited repeats. - Internally, pcre_exec() uses a function called match(), which it calls - repeatedly (sometimes recursively). The limit set by match_limit is - imposed on the number of times this function is called during a match, - which has the effect of limiting the amount of backtracking that can + Internally, pcre_exec() uses a function called match(), which it calls + repeatedly (sometimes recursively). The limit set by match_limit is + imposed on the number of times this function is called during a match, + which has the effect of limiting the amount of backtracking that can take place. For patterns that are not anchored, the count restarts from zero for each position in the subject string. When pcre_exec() is called with a pattern that was successfully studied - with a JIT option, the way that the matching is executed is entirely + with a JIT option, the way that the matching is executed is entirely different. However, there is still the possibility of runaway matching that goes on for a very long time, and so the match_limit value is also used in this case (but in a different way) to limit how long the match- ing can continue. - The default value for the limit can be set when PCRE is built; the - default default is 10 million, which handles all but the most extreme - cases. You can override the default by suppling pcre_exec() with a - pcre_extra block in which match_limit is set, and - PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is + The default value for the limit can be set when PCRE is built; the + default default is 10 million, which handles all but the most extreme + cases. You can override the default by suppling pcre_exec() with a + pcre_extra block in which match_limit is set, and + PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT. - The match_limit_recursion field is similar to match_limit, but instead + The match_limit_recursion field is similar to match_limit, but instead of limiting the total number of times that match() is called, it limits - the depth of recursion. The recursion depth is a smaller number than - the total number of calls, because not all calls to match() are recur- + the depth of recursion. The recursion depth is a smaller number than + the total number of calls, because not all calls to match() are recur- sive. This limit is of use only if it is set smaller than match_limit. - Limiting the recursion depth limits the amount of machine stack that - can be used, or, when PCRE has been compiled to use memory on the heap - instead of the stack, the amount of heap memory that can be used. This - limit is not relevant, and is ignored, when matching is done using JIT + Limiting the recursion depth limits the amount of machine stack that + can be used, or, when PCRE has been compiled to use memory on the heap + instead of the stack, the amount of heap memory that can be used. This + limit is not relevant, and is ignored, when matching is done using JIT compiled code. - The default value for match_limit_recursion can be set when PCRE is - built; the default default is the same value as the default for - match_limit. You can override the default by suppling pcre_exec() with - a pcre_extra block in which match_limit_recursion is set, and - PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the + The default value for match_limit_recursion can be set when PCRE is + built; the default default is the same value as the default for + match_limit. You can override the default by suppling pcre_exec() with + a pcre_extra block in which match_limit_recursion is set, and + PCRE_EXTRA_MATCH_LIMIT_RECURSION is set in the flags field. If the limit is exceeded, pcre_exec() returns PCRE_ERROR_RECURSIONLIMIT. - The callout_data field is used in conjunction with the "callout" fea- + The callout_data field is used in conjunction with the "callout" fea- ture, and is described in the pcrecallout documentation. - The tables field is used to pass a character tables pointer to - pcre_exec(); this overrides the value that is stored with the compiled - pattern. A non-NULL value is stored with the compiled pattern only if - custom tables were supplied to pcre_compile() via its tableptr argu- + The tables field is used to pass a character tables pointer to + pcre_exec(); this overrides the value that is stored with the compiled + pattern. A non-NULL value is stored with the compiled pattern only if + custom tables were supplied to pcre_compile() via its tableptr argu- ment. If NULL is passed to pcre_exec() using this mechanism, it forces - PCRE's internal tables to be used. This facility is helpful when re- - using patterns that have been saved after compiling with an external - set of tables, because the external tables might be at a different - address when pcre_exec() is called. See the pcreprecompile documenta- + PCRE's internal tables to be used. This facility is helpful when re- + using patterns that have been saved after compiling with an external + set of tables, because the external tables might be at a different + address when pcre_exec() is called. See the pcreprecompile documenta- tion for a discussion of saving compiled patterns for later use. - If PCRE_EXTRA_MARK is set in the flags field, the mark field must be - set to point to a suitable variable. If the pattern contains any back- - tracking control verbs such as (*MARK:NAME), and the execution ends up - with a name to pass back, a pointer to the name string (zero termi- - nated) is placed in the variable pointed to by the mark field. The - names are within the compiled pattern; if you wish to retain such a - name you must copy it before freeing the memory of a compiled pattern. - If there is no name to pass back, the variable pointed to by the mark - field is set to NULL. For details of the backtracking control verbs, + If PCRE_EXTRA_MARK is set in the flags field, the mark field must be + set to point to a suitable variable. If the pattern contains any back- + tracking control verbs such as (*MARK:NAME), and the execution ends up + with a name to pass back, a pointer to the name string (zero termi- + nated) is placed in the variable pointed to by the mark field. The + names are within the compiled pattern; if you wish to retain such a + name you must copy it before freeing the memory of a compiled pattern. + If there is no name to pass back, the variable pointed to by the mark + field is set to NULL. For details of the backtracking control verbs, see the section entitled "Backtracking control" in the pcrepattern doc- umentation. Option bits for pcre_exec() - The unused bits of the options argument for pcre_exec() must be zero. - The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx, - PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, - PCRE_NO_START_OPTIMIZE, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL_HARD, and + The unused bits of the options argument for pcre_exec() must be zero. + The only bits that may be set are PCRE_ANCHORED, PCRE_NEWLINE_xxx, + PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, + PCRE_NO_START_OPTIMIZE, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL_HARD, and PCRE_PARTIAL_SOFT. - If the pattern was successfully studied with one of the just-in-time + If the pattern was successfully studied with one of the just-in-time (JIT) compile options, the only supported options for JIT execution are - PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, - PCRE_NOTEMPTY_ATSTART, PCRE_PARTIAL_HARD, and PCRE_PARTIAL_SOFT. If an - unsupported option is used, JIT execution is disabled and the normal + PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, + PCRE_NOTEMPTY_ATSTART, PCRE_PARTIAL_HARD, and PCRE_PARTIAL_SOFT. If an + unsupported option is used, JIT execution is disabled and the normal interpretive code in pcre_exec() is run. PCRE_ANCHORED - The PCRE_ANCHORED option limits pcre_exec() to matching at the first - matching position. If a pattern was compiled with PCRE_ANCHORED, or - turned out to be anchored by virtue of its contents, it cannot be made + The PCRE_ANCHORED option limits pcre_exec() to matching at the first + matching position. If a pattern was compiled with PCRE_ANCHORED, or + turned out to be anchored by virtue of its contents, it cannot be made unachored at matching time. PCRE_BSR_ANYCRLF PCRE_BSR_UNICODE These options (which are mutually exclusive) control what the \R escape - sequence matches. The choice is either to match only CR, LF, or CRLF, - or to match any Unicode newline sequence. These options override the + sequence matches. The choice is either to match only CR, LF, or CRLF, + or to match any Unicode newline sequence. These options override the choice that was made or defaulted when the pattern was compiled. PCRE_NEWLINE_CR @@ -3109,337 +3117,337 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_NEWLINE_ANYCRLF PCRE_NEWLINE_ANY - These options override the newline definition that was chosen or - defaulted when the pattern was compiled. For details, see the descrip- - tion of pcre_compile() above. During matching, the newline choice - affects the behaviour of the dot, circumflex, and dollar metacharac- - ters. It may also alter the way the match position is advanced after a + These options override the newline definition that was chosen or + defaulted when the pattern was compiled. For details, see the descrip- + tion of pcre_compile() above. During matching, the newline choice + affects the behaviour of the dot, circumflex, and dollar metacharac- + ters. It may also alter the way the match position is advanced after a match failure for an unanchored pattern. - When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is - set, and a match attempt for an unanchored pattern fails when the cur- - rent position is at a CRLF sequence, and the pattern contains no - explicit matches for CR or LF characters, the match position is + When PCRE_NEWLINE_CRLF, PCRE_NEWLINE_ANYCRLF, or PCRE_NEWLINE_ANY is + set, and a match attempt for an unanchored pattern fails when the cur- + rent position is at a CRLF sequence, and the pattern contains no + explicit matches for CR or LF characters, the match position is advanced by two characters instead of one, in other words, to after the CRLF. The above rule is a compromise that makes the most common cases work as - expected. For example, if the pattern is .+A (and the PCRE_DOTALL + expected. For example, if the pattern is .+A (and the PCRE_DOTALL option is not set), it does not match the string "\r\nA" because, after - failing at the start, it skips both the CR and the LF before retrying. - However, the pattern [\r\n]A does match that string, because it con- + failing at the start, it skips both the CR and the LF before retrying. + However, the pattern [\r\n]A does match that string, because it con- tains an explicit CR or LF reference, and so advances only by one char- acter after the first failure. An explicit match for CR of LF is either a literal appearance of one of - those characters, or one of the \r or \n escape sequences. Implicit - matches such as [^X] do not count, nor does \s (which includes CR and + those characters, or one of the \r or \n escape sequences. Implicit + matches such as [^X] do not count, nor does \s (which includes CR and LF in the characters that it matches). - Notwithstanding the above, anomalous effects may still occur when CRLF + Notwithstanding the above, anomalous effects may still occur when CRLF is a valid newline sequence and explicit \r or \n escapes appear in the pattern. PCRE_NOTBOL This option specifies that first character of the subject string is not - the beginning of a line, so the circumflex metacharacter should not - match before it. Setting this without PCRE_MULTILINE (at compile time) - causes circumflex never to match. This option affects only the behav- + the beginning of a line, so the circumflex metacharacter should not + match before it. Setting this without PCRE_MULTILINE (at compile time) + causes circumflex never to match. This option affects only the behav- iour of the circumflex metacharacter. It does not affect \A. PCRE_NOTEOL This option specifies that the end of the subject string is not the end - of a line, so the dollar metacharacter should not match it nor (except - in multiline mode) a newline immediately before it. Setting this with- + of a line, so the dollar metacharacter should not match it nor (except + in multiline mode) a newline immediately before it. Setting this with- out PCRE_MULTILINE (at compile time) causes dollar never to match. This - option affects only the behaviour of the dollar metacharacter. It does + option affects only the behaviour of the dollar metacharacter. It does not affect \Z or \z. PCRE_NOTEMPTY An empty string is not considered to be a valid match if this option is - set. If there are alternatives in the pattern, they are tried. If all - the alternatives match the empty string, the entire match fails. For + set. If there are alternatives in the pattern, they are tried. If all + the alternatives match the empty string, the entire match fails. For example, if the pattern a?b? - is applied to a string not beginning with "a" or "b", it matches an - empty string at the start of the subject. With PCRE_NOTEMPTY set, this + is applied to a string not beginning with "a" or "b", it matches an + empty string at the start of the subject. With PCRE_NOTEMPTY set, this match is not valid, so PCRE searches further into the string for occur- rences of "a" or "b". PCRE_NOTEMPTY_ATSTART - This is like PCRE_NOTEMPTY, except that an empty string match that is - not at the start of the subject is permitted. If the pattern is + This is like PCRE_NOTEMPTY, except that an empty string match that is + not at the start of the subject is permitted. If the pattern is anchored, such a match can occur only if the pattern contains \K. - Perl has no direct equivalent of PCRE_NOTEMPTY or - PCRE_NOTEMPTY_ATSTART, but it does make a special case of a pattern - match of the empty string within its split() function, and when using - the /g modifier. It is possible to emulate Perl's behaviour after + Perl has no direct equivalent of PCRE_NOTEMPTY or + PCRE_NOTEMPTY_ATSTART, but it does make a special case of a pattern + match of the empty string within its split() function, and when using + the /g modifier. It is possible to emulate Perl's behaviour after matching a null string by first trying the match again at the same off- - set with PCRE_NOTEMPTY_ATSTART and PCRE_ANCHORED, and then if that + set with PCRE_NOTEMPTY_ATSTART and PCRE_ANCHORED, and then if that fails, by advancing the starting offset (see below) and trying an ordi- - nary match again. There is some code that demonstrates how to do this - in the pcredemo sample program. In the most general case, you have to - check to see if the newline convention recognizes CRLF as a newline, - and if so, and the current character is CR followed by LF, advance the + nary match again. There is some code that demonstrates how to do this + in the pcredemo sample program. In the most general case, you have to + check to see if the newline convention recognizes CRLF as a newline, + and if so, and the current character is CR followed by LF, advance the starting offset by two characters instead of one. PCRE_NO_START_OPTIMIZE - There are a number of optimizations that pcre_exec() uses at the start - of a match, in order to speed up the process. For example, if it is + There are a number of optimizations that pcre_exec() uses at the start + of a match, in order to speed up the process. For example, if it is known that an unanchored match must start with a specific character, it - searches the subject for that character, and fails immediately if it - cannot find it, without actually running the main matching function. + searches the subject for that character, and fails immediately if it + cannot find it, without actually running the main matching function. This means that a special item such as (*COMMIT) at the start of a pat- - tern is not considered until after a suitable starting point for the - match has been found. When callouts or (*MARK) items are in use, these + tern is not considered until after a suitable starting point for the + match has been found. When callouts or (*MARK) items are in use, these "start-up" optimizations can cause them to be skipped if the pattern is - never actually used. The start-up optimizations are in effect a pre- + never actually used. The start-up optimizations are in effect a pre- scan of the subject that takes place before the pattern is run. - The PCRE_NO_START_OPTIMIZE option disables the start-up optimizations, - possibly causing performance to suffer, but ensuring that in cases - where the result is "no match", the callouts do occur, and that items + The PCRE_NO_START_OPTIMIZE option disables the start-up optimizations, + possibly causing performance to suffer, but ensuring that in cases + where the result is "no match", the callouts do occur, and that items such as (*COMMIT) and (*MARK) are considered at every possible starting - position in the subject string. If PCRE_NO_START_OPTIMIZE is set at - compile time, it cannot be unset at matching time. The use of + position in the subject string. If PCRE_NO_START_OPTIMIZE is set at + compile time, it cannot be unset at matching time. The use of PCRE_NO_START_OPTIMIZE disables JIT execution; when it is set, matching is always done using interpretively. - Setting PCRE_NO_START_OPTIMIZE can change the outcome of a matching + Setting PCRE_NO_START_OPTIMIZE can change the outcome of a matching operation. Consider the pattern (*COMMIT)ABC - When this is compiled, PCRE records the fact that a match must start - with the character "A". Suppose the subject string is "DEFABC". The - start-up optimization scans along the subject, finds "A" and runs the - first match attempt from there. The (*COMMIT) item means that the pat- - tern must match the current starting position, which in this case, it - does. However, if the same match is run with PCRE_NO_START_OPTIMIZE - set, the initial scan along the subject string does not happen. The - first match attempt is run starting from "D" and when this fails, - (*COMMIT) prevents any further matches being tried, so the overall - result is "no match". If the pattern is studied, more start-up opti- - mizations may be used. For example, a minimum length for the subject + When this is compiled, PCRE records the fact that a match must start + with the character "A". Suppose the subject string is "DEFABC". The + start-up optimization scans along the subject, finds "A" and runs the + first match attempt from there. The (*COMMIT) item means that the pat- + tern must match the current starting position, which in this case, it + does. However, if the same match is run with PCRE_NO_START_OPTIMIZE + set, the initial scan along the subject string does not happen. The + first match attempt is run starting from "D" and when this fails, + (*COMMIT) prevents any further matches being tried, so the overall + result is "no match". If the pattern is studied, more start-up opti- + mizations may be used. For example, a minimum length for the subject may be recorded. Consider the pattern (*MARK:A)(X|Y) - The minimum length for a match is one character. If the subject is - "ABC", there will be attempts to match "ABC", "BC", "C", and then - finally an empty string. If the pattern is studied, the final attempt - does not take place, because PCRE knows that the subject is too short, - and so the (*MARK) is never encountered. In this case, studying the - pattern does not affect the overall match result, which is still "no + The minimum length for a match is one character. If the subject is + "ABC", there will be attempts to match "ABC", "BC", "C", and then + finally an empty string. If the pattern is studied, the final attempt + does not take place, because PCRE knows that the subject is too short, + and so the (*MARK) is never encountered. In this case, studying the + pattern does not affect the overall match result, which is still "no match", but it does affect the auxiliary information that is returned. PCRE_NO_UTF8_CHECK When PCRE_UTF8 is set at compile time, the validity of the subject as a - UTF-8 string is automatically checked when pcre_exec() is subsequently + UTF-8 string is automatically checked when pcre_exec() is subsequently called. The entire string is checked before any other processing takes - place. The value of startoffset is also checked to ensure that it - points to the start of a UTF-8 character. There is a discussion about - the validity of UTF-8 strings in the pcreunicode page. If an invalid - sequence of bytes is found, pcre_exec() returns the error + place. The value of startoffset is also checked to ensure that it + points to the start of a UTF-8 character. There is a discussion about + the validity of UTF-8 strings in the pcreunicode page. If an invalid + sequence of bytes is found, pcre_exec() returns the error PCRE_ERROR_BADUTF8 or, if PCRE_PARTIAL_HARD is set and the problem is a truncated character at the end of the subject, PCRE_ERROR_SHORTUTF8. In - both cases, information about the precise nature of the error may also - be returned (see the descriptions of these errors in the section enti- - tled Error return values from pcre_exec() below). If startoffset con- + both cases, information about the precise nature of the error may also + be returned (see the descriptions of these errors in the section enti- + tled Error return values from pcre_exec() below). If startoffset con- tains a value that does not point to the start of a UTF-8 character (or to the end of the subject), PCRE_ERROR_BADUTF8_OFFSET is returned. - If you already know that your subject is valid, and you want to skip - these checks for performance reasons, you can set the - PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to - do this for the second and subsequent calls to pcre_exec() if you are - making repeated calls to find all the matches in a single subject - string. However, you should be sure that the value of startoffset - points to the start of a character (or the end of the subject). When + If you already know that your subject is valid, and you want to skip + these checks for performance reasons, you can set the + PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to + do this for the second and subsequent calls to pcre_exec() if you are + making repeated calls to find all the matches in a single subject + string. However, you should be sure that the value of startoffset + points to the start of a character (or the end of the subject). When PCRE_NO_UTF8_CHECK is set, the effect of passing an invalid string as a - subject or an invalid value of startoffset is undefined. Your program + subject or an invalid value of startoffset is undefined. Your program may crash. PCRE_PARTIAL_HARD PCRE_PARTIAL_SOFT - These options turn on the partial matching feature. For backwards com- - patibility, PCRE_PARTIAL is a synonym for PCRE_PARTIAL_SOFT. A partial - match occurs if the end of the subject string is reached successfully, - but there are not enough subject characters to complete the match. If + These options turn on the partial matching feature. For backwards com- + patibility, PCRE_PARTIAL is a synonym for PCRE_PARTIAL_SOFT. A partial + match occurs if the end of the subject string is reached successfully, + but there are not enough subject characters to complete the match. If this happens when PCRE_PARTIAL_SOFT (but not PCRE_PARTIAL_HARD) is set, - matching continues by testing any remaining alternatives. Only if no - complete match can be found is PCRE_ERROR_PARTIAL returned instead of - PCRE_ERROR_NOMATCH. In other words, PCRE_PARTIAL_SOFT says that the - caller is prepared to handle a partial match, but only if no complete + matching continues by testing any remaining alternatives. Only if no + complete match can be found is PCRE_ERROR_PARTIAL returned instead of + PCRE_ERROR_NOMATCH. In other words, PCRE_PARTIAL_SOFT says that the + caller is prepared to handle a partial match, but only if no complete match can be found. - If PCRE_PARTIAL_HARD is set, it overrides PCRE_PARTIAL_SOFT. In this - case, if a partial match is found, pcre_exec() immediately returns - PCRE_ERROR_PARTIAL, without considering any other alternatives. In - other words, when PCRE_PARTIAL_HARD is set, a partial match is consid- + If PCRE_PARTIAL_HARD is set, it overrides PCRE_PARTIAL_SOFT. In this + case, if a partial match is found, pcre_exec() immediately returns + PCRE_ERROR_PARTIAL, without considering any other alternatives. In + other words, when PCRE_PARTIAL_HARD is set, a partial match is consid- ered to be more important that an alternative complete match. - In both cases, the portion of the string that was inspected when the + In both cases, the portion of the string that was inspected when the partial match was found is set as the first matching string. There is a - more detailed discussion of partial and multi-segment matching, with + more detailed discussion of partial and multi-segment matching, with examples, in the pcrepartial documentation. The string to be matched by pcre_exec() - The subject string is passed to pcre_exec() as a pointer in subject, a - length in bytes in length, and a starting byte offset in startoffset. - If this is negative or greater than the length of the subject, - pcre_exec() returns PCRE_ERROR_BADOFFSET. When the starting offset is - zero, the search for a match starts at the beginning of the subject, + The subject string is passed to pcre_exec() as a pointer in subject, a + length in bytes in length, and a starting byte offset in startoffset. + If this is negative or greater than the length of the subject, + pcre_exec() returns PCRE_ERROR_BADOFFSET. When the starting offset is + zero, the search for a match starts at the beginning of the subject, and this is by far the most common case. In UTF-8 mode, the byte offset - must point to the start of a UTF-8 character (or the end of the sub- - ject). Unlike the pattern string, the subject may contain binary zero + must point to the start of a UTF-8 character (or the end of the sub- + ject). Unlike the pattern string, the subject may contain binary zero bytes. - A non-zero starting offset is useful when searching for another match - in the same subject by calling pcre_exec() again after a previous suc- - cess. Setting startoffset differs from just passing over a shortened - string and setting PCRE_NOTBOL in the case of a pattern that begins + A non-zero starting offset is useful when searching for another match + in the same subject by calling pcre_exec() again after a previous suc- + cess. Setting startoffset differs from just passing over a shortened + string and setting PCRE_NOTBOL in the case of a pattern that begins with any kind of lookbehind. For example, consider the pattern \Biss\B - which finds occurrences of "iss" in the middle of words. (\B matches - only if the current position in the subject is not a word boundary.) - When applied to the string "Mississipi" the first call to pcre_exec() - finds the first occurrence. If pcre_exec() is called again with just - the remainder of the subject, namely "issipi", it does not match, + which finds occurrences of "iss" in the middle of words. (\B matches + only if the current position in the subject is not a word boundary.) + When applied to the string "Mississipi" the first call to pcre_exec() + finds the first occurrence. If pcre_exec() is called again with just + the remainder of the subject, namely "issipi", it does not match, because \B is always false at the start of the subject, which is deemed - to be a word boundary. However, if pcre_exec() is passed the entire + to be a word boundary. However, if pcre_exec() is passed the entire string again, but with startoffset set to 4, it finds the second occur- - rence of "iss" because it is able to look behind the starting point to + rence of "iss" because it is able to look behind the starting point to discover that it is preceded by a letter. - Finding all the matches in a subject is tricky when the pattern can + Finding all the matches in a subject is tricky when the pattern can match an empty string. It is possible to emulate Perl's /g behaviour by - first trying the match again at the same offset, with the - PCRE_NOTEMPTY_ATSTART and PCRE_ANCHORED options, and then if that - fails, advancing the starting offset and trying an ordinary match + first trying the match again at the same offset, with the + PCRE_NOTEMPTY_ATSTART and PCRE_ANCHORED options, and then if that + fails, advancing the starting offset and trying an ordinary match again. There is some code that demonstrates how to do this in the pcre- demo sample program. In the most general case, you have to check to see - if the newline convention recognizes CRLF as a newline, and if so, and + if the newline convention recognizes CRLF as a newline, and if so, and the current character is CR followed by LF, advance the starting offset by two characters instead of one. - If a non-zero starting offset is passed when the pattern is anchored, + If a non-zero starting offset is passed when the pattern is anchored, one attempt to match at the given offset is made. This can only succeed - if the pattern does not require the match to be at the start of the + if the pattern does not require the match to be at the start of the subject. How pcre_exec() returns captured substrings - In general, a pattern matches a certain portion of the subject, and in - addition, further substrings from the subject may be picked out by - parts of the pattern. Following the usage in Jeffrey Friedl's book, - this is called "capturing" in what follows, and the phrase "capturing - subpattern" is used for a fragment of a pattern that picks out a sub- - string. PCRE supports several other kinds of parenthesized subpattern + In general, a pattern matches a certain portion of the subject, and in + addition, further substrings from the subject may be picked out by + parts of the pattern. Following the usage in Jeffrey Friedl's book, + this is called "capturing" in what follows, and the phrase "capturing + subpattern" is used for a fragment of a pattern that picks out a sub- + string. PCRE supports several other kinds of parenthesized subpattern that do not cause substrings to be captured. Captured substrings are returned to the caller via a vector of integers - whose address is passed in ovector. The number of elements in the vec- - tor is passed in ovecsize, which must be a non-negative number. Note: + whose address is passed in ovector. The number of elements in the vec- + tor is passed in ovecsize, which must be a non-negative number. Note: this argument is NOT the size of ovector in bytes. - The first two-thirds of the vector is used to pass back captured sub- - strings, each substring using a pair of integers. The remaining third - of the vector is used as workspace by pcre_exec() while matching cap- - turing subpatterns, and is not available for passing back information. - The number passed in ovecsize should always be a multiple of three. If + The first two-thirds of the vector is used to pass back captured sub- + strings, each substring using a pair of integers. The remaining third + of the vector is used as workspace by pcre_exec() while matching cap- + turing subpatterns, and is not available for passing back information. + The number passed in ovecsize should always be a multiple of three. If it is not, it is rounded down. - When a match is successful, information about captured substrings is - returned in pairs of integers, starting at the beginning of ovector, - and continuing up to two-thirds of its length at the most. The first - element of each pair is set to the byte offset of the first character - in a substring, and the second is set to the byte offset of the first - character after the end of a substring. Note: these values are always + When a match is successful, information about captured substrings is + returned in pairs of integers, starting at the beginning of ovector, + and continuing up to two-thirds of its length at the most. The first + element of each pair is set to the byte offset of the first character + in a substring, and the second is set to the byte offset of the first + character after the end of a substring. Note: these values are always byte offsets, even in UTF-8 mode. They are not character counts. - The first pair of integers, ovector[0] and ovector[1], identify the - portion of the subject string matched by the entire pattern. The next - pair is used for the first capturing subpattern, and so on. The value + The first pair of integers, ovector[0] and ovector[1], identify the + portion of the subject string matched by the entire pattern. The next + pair is used for the first capturing subpattern, and so on. The value returned by pcre_exec() is one more than the highest numbered pair that - has been set. For example, if two substrings have been captured, the - returned value is 3. If there are no capturing subpatterns, the return + has been set. For example, if two substrings have been captured, the + returned value is 3. If there are no capturing subpatterns, the return value from a successful match is 1, indicating that just the first pair of offsets has been set. If a capturing subpattern is matched repeatedly, it is the last portion of the string that it matched that is returned. - If the vector is too small to hold all the captured substring offsets, + If the vector is too small to hold all the captured substring offsets, it is used as far as possible (up to two-thirds of its length), and the - function returns a value of zero. If neither the actual string matched - nor any captured substrings are of interest, pcre_exec() may be called - with ovector passed as NULL and ovecsize as zero. However, if the pat- - tern contains back references and the ovector is not big enough to - remember the related substrings, PCRE has to get additional memory for - use during matching. Thus it is usually advisable to supply an ovector + function returns a value of zero. If neither the actual string matched + nor any captured substrings are of interest, pcre_exec() may be called + with ovector passed as NULL and ovecsize as zero. However, if the pat- + tern contains back references and the ovector is not big enough to + remember the related substrings, PCRE has to get additional memory for + use during matching. Thus it is usually advisable to supply an ovector of reasonable size. - There are some cases where zero is returned (indicating vector over- - flow) when in fact the vector is exactly the right size for the final + There are some cases where zero is returned (indicating vector over- + flow) when in fact the vector is exactly the right size for the final match. For example, consider the pattern (a)(?:(b)c|bd) - If a vector of 6 elements (allowing for only 1 captured substring) is + If a vector of 6 elements (allowing for only 1 captured substring) is given with subject string "abd", pcre_exec() will try to set the second captured string, thereby recording a vector overflow, before failing to - match "c" and backing up to try the second alternative. The zero - return, however, does correctly indicate that the maximum number of + match "c" and backing up to try the second alternative. The zero + return, however, does correctly indicate that the maximum number of slots (namely 2) have been filled. In similar cases where there is tem- - porary overflow, but the final number of used slots is actually less + porary overflow, but the final number of used slots is actually less than the maximum, a non-zero value is returned. The pcre_fullinfo() function can be used to find out how many capturing - subpatterns there are in a compiled pattern. The smallest size for - ovector that will allow for n captured substrings, in addition to the + subpatterns there are in a compiled pattern. The smallest size for + ovector that will allow for n captured substrings, in addition to the offsets of the substring matched by the whole pattern, is (n+1)*3. - It is possible for capturing subpattern number n+1 to match some part + It is possible for capturing subpattern number n+1 to match some part of the subject when subpattern n has not been used at all. For example, - if the string "abc" is matched against the pattern (a|(z))(bc) the + if the string "abc" is matched against the pattern (a|(z))(bc) the return from the function is 4, and subpatterns 1 and 3 are matched, but - 2 is not. When this happens, both values in the offset pairs corre- + 2 is not. When this happens, both values in the offset pairs corre- sponding to unused subpatterns are set to -1. - Offset values that correspond to unused subpatterns at the end of the - expression are also set to -1. For example, if the string "abc" is - matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not - matched. The return from the function is 2, because the highest used - capturing subpattern number is 1, and the offsets for for the second - and third capturing subpatterns (assuming the vector is large enough, + Offset values that correspond to unused subpatterns at the end of the + expression are also set to -1. For example, if the string "abc" is + matched against the pattern (abc)(x(yz)?)? subpatterns 2 and 3 are not + matched. The return from the function is 2, because the highest used + capturing subpattern number is 1, and the offsets for for the second + and third capturing subpatterns (assuming the vector is large enough, of course) are set to -1. - Note: Elements in the first two-thirds of ovector that do not corre- - spond to capturing parentheses in the pattern are never changed. That - is, if a pattern contains n capturing parentheses, no more than ovec- - tor[0] to ovector[2n+1] are set by pcre_exec(). The other elements (in + Note: Elements in the first two-thirds of ovector that do not corre- + spond to capturing parentheses in the pattern are never changed. That + is, if a pattern contains n capturing parentheses, no more than ovec- + tor[0] to ovector[2n+1] are set by pcre_exec(). The other elements (in the first two-thirds) retain whatever values they previously had. - Some convenience functions are provided for extracting the captured + Some convenience functions are provided for extracting the captured substrings as separate strings. These are described below. Error return values from pcre_exec() - If pcre_exec() fails, it returns a negative number. The following are + If pcre_exec() fails, it returns a negative number. The following are defined in the header file: PCRE_ERROR_NOMATCH (-1) @@ -3448,7 +3456,7 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_NULL (-2) - Either code or subject was passed as NULL, or ovector was NULL and + Either code or subject was passed as NULL, or ovector was NULL and ovecsize was not zero. PCRE_ERROR_BADOPTION (-3) @@ -3457,82 +3465,82 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_BADMAGIC (-4) - PCRE stores a 4-byte "magic number" at the start of the compiled code, + PCRE stores a 4-byte "magic number" at the start of the compiled code, to catch the case when it is passed a junk pointer and to detect when a pattern that was compiled in an environment of one endianness is run in - an environment with the other endianness. This is the error that PCRE + an environment with the other endianness. This is the error that PCRE gives when the magic number is not present. PCRE_ERROR_UNKNOWN_OPCODE (-5) While running the pattern match, an unknown item was encountered in the - compiled pattern. This error could be caused by a bug in PCRE or by + compiled pattern. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. PCRE_ERROR_NOMEMORY (-6) - If a pattern contains back references, but the ovector that is passed + If a pattern contains back references, but the ovector that is passed to pcre_exec() is not big enough to remember the referenced substrings, - PCRE gets a block of memory at the start of matching to use for this - purpose. If the call via pcre_malloc() fails, this error is given. The + PCRE gets a block of memory at the start of matching to use for this + purpose. If the call via pcre_malloc() fails, this error is given. The memory is automatically freed at the end of matching. - This error is also given if pcre_stack_malloc() fails in pcre_exec(). - This can happen only when PCRE has been compiled with --disable-stack- + This error is also given if pcre_stack_malloc() fails in pcre_exec(). + This can happen only when PCRE has been compiled with --disable-stack- for-recursion. PCRE_ERROR_NOSUBSTRING (-7) - This error is used by the pcre_copy_substring(), pcre_get_substring(), + This error is used by the pcre_copy_substring(), pcre_get_substring(), and pcre_get_substring_list() functions (see below). It is never returned by pcre_exec(). PCRE_ERROR_MATCHLIMIT (-8) - The backtracking limit, as specified by the match_limit field in a - pcre_extra structure (or defaulted) was reached. See the description + The backtracking limit, as specified by the match_limit field in a + pcre_extra structure (or defaulted) was reached. See the description above. PCRE_ERROR_CALLOUT (-9) This error is never generated by pcre_exec() itself. It is provided for - use by callout functions that want to yield a distinctive error code. + use by callout functions that want to yield a distinctive error code. See the pcrecallout documentation for details. PCRE_ERROR_BADUTF8 (-10) - A string that contains an invalid UTF-8 byte sequence was passed as a - subject, and the PCRE_NO_UTF8_CHECK option was not set. If the size of - the output vector (ovecsize) is at least 2, the byte offset to the - start of the the invalid UTF-8 character is placed in the first ele- - ment, and a reason code is placed in the second element. The reason + A string that contains an invalid UTF-8 byte sequence was passed as a + subject, and the PCRE_NO_UTF8_CHECK option was not set. If the size of + the output vector (ovecsize) is at least 2, the byte offset to the + start of the the invalid UTF-8 character is placed in the first ele- + ment, and a reason code is placed in the second element. The reason codes are listed in the following section. For backward compatibility, - if PCRE_PARTIAL_HARD is set and the problem is a truncated UTF-8 char- - acter at the end of the subject (reason codes 1 to 5), + if PCRE_PARTIAL_HARD is set and the problem is a truncated UTF-8 char- + acter at the end of the subject (reason codes 1 to 5), PCRE_ERROR_SHORTUTF8 is returned instead of PCRE_ERROR_BADUTF8. PCRE_ERROR_BADUTF8_OFFSET (-11) - The UTF-8 byte sequence that was passed as a subject was checked and - found to be valid (the PCRE_NO_UTF8_CHECK option was not set), but the - value of startoffset did not point to the beginning of a UTF-8 charac- + The UTF-8 byte sequence that was passed as a subject was checked and + found to be valid (the PCRE_NO_UTF8_CHECK option was not set), but the + value of startoffset did not point to the beginning of a UTF-8 charac- ter or the end of the subject. PCRE_ERROR_PARTIAL (-12) - The subject string did not match, but it did match partially. See the + The subject string did not match, but it did match partially. See the pcrepartial documentation for details of partial matching. PCRE_ERROR_BADPARTIAL (-13) - This code is no longer in use. It was formerly returned when the - PCRE_PARTIAL option was used with a compiled pattern containing items - that were not supported for partial matching. From release 8.00 + This code is no longer in use. It was formerly returned when the + PCRE_PARTIAL option was used with a compiled pattern containing items + that were not supported for partial matching. From release 8.00 onwards, there are no restrictions on partial matching. PCRE_ERROR_INTERNAL (-14) - An unexpected internal error has occurred. This error could be caused + An unexpected internal error has occurred. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. PCRE_ERROR_BADCOUNT (-15) @@ -3542,7 +3550,7 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_RECURSIONLIMIT (-21) The internal recursion limit, as specified by the match_limit_recursion - field in a pcre_extra structure (or defaulted) was reached. See the + field in a pcre_extra structure (or defaulted) was reached. See the description above. PCRE_ERROR_BADNEWLINE (-23) @@ -3556,29 +3564,29 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_SHORTUTF8 (-25) - This error is returned instead of PCRE_ERROR_BADUTF8 when the subject - string ends with a truncated UTF-8 character and the PCRE_PARTIAL_HARD - option is set. Information about the failure is returned as for - PCRE_ERROR_BADUTF8. It is in fact sufficient to detect this case, but - this special error code for PCRE_PARTIAL_HARD precedes the implementa- - tion of returned information; it is retained for backwards compatibil- + This error is returned instead of PCRE_ERROR_BADUTF8 when the subject + string ends with a truncated UTF-8 character and the PCRE_PARTIAL_HARD + option is set. Information about the failure is returned as for + PCRE_ERROR_BADUTF8. It is in fact sufficient to detect this case, but + this special error code for PCRE_PARTIAL_HARD precedes the implementa- + tion of returned information; it is retained for backwards compatibil- ity. PCRE_ERROR_RECURSELOOP (-26) This error is returned when pcre_exec() detects a recursion loop within - the pattern. Specifically, it means that either the whole pattern or a - subpattern has been called recursively for the second time at the same + the pattern. Specifically, it means that either the whole pattern or a + subpattern has been called recursively for the second time at the same position in the subject string. Some simple patterns that might do this - are detected and faulted at compile time, but more complicated cases, + are detected and faulted at compile time, but more complicated cases, in particular mutual recursions between two different subpatterns, can- not be detected until run time. PCRE_ERROR_JIT_STACKLIMIT (-27) - This error is returned when a pattern that was successfully studied - using a JIT compile option is being matched, but the memory available - for the just-in-time processing stack is not large enough. See the + This error is returned when a pattern that was successfully studied + using a JIT compile option is being matched, but the memory available + for the just-in-time processing stack is not large enough. See the pcrejit documentation for more details. PCRE_ERROR_BADMODE (-28) @@ -3588,38 +3596,38 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_ERROR_BADENDIANNESS (-29) - This error is given if a pattern that was compiled and saved is - reloaded on a host with different endianness. The utility function + This error is given if a pattern that was compiled and saved is + reloaded on a host with different endianness. The utility function pcre_pattern_to_host_byte_order() can be used to convert such a pattern so that it runs on the new host. PCRE_ERROR_JIT_BADOPTION - This error is returned when a pattern that was successfully studied - using a JIT compile option is being matched, but the matching mode - (partial or complete match) does not correspond to any JIT compilation - mode. When the JIT fast path function is used, this error may be also - given for invalid options. See the pcrejit documentation for more + This error is returned when a pattern that was successfully studied + using a JIT compile option is being matched, but the matching mode + (partial or complete match) does not correspond to any JIT compilation + mode. When the JIT fast path function is used, this error may be also + given for invalid options. See the pcrejit documentation for more details. PCRE_ERROR_BADLENGTH (-32) - This error is given if pcre_exec() is called with a negative value for + This error is given if pcre_exec() is called with a negative value for the length argument. Error numbers -16 to -20, -22, and 30 are not used by pcre_exec(). Reason codes for invalid UTF-8 strings - This section applies only to the 8-bit library. The corresponding - information for the 16-bit and 32-bit libraries is given in the pcre16 + This section applies only to the 8-bit library. The corresponding + information for the 16-bit and 32-bit libraries is given in the pcre16 and pcre32 pages. When pcre_exec() returns either PCRE_ERROR_BADUTF8 or PCRE_ERROR_SHORT- - UTF8, and the size of the output vector (ovecsize) is at least 2, the - offset of the start of the invalid UTF-8 character is placed in the + UTF8, and the size of the output vector (ovecsize) is at least 2, the + offset of the start of the invalid UTF-8 character is placed in the first output vector element (ovector[0]) and a reason code is placed in - the second element (ovector[1]). The reason codes are given names in + the second element (ovector[1]). The reason codes are given names in the pcre.h header file: PCRE_UTF8_ERR1 @@ -3628,10 +3636,10 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_UTF8_ERR4 PCRE_UTF8_ERR5 - The string ends with a truncated UTF-8 character; the code specifies - how many bytes are missing (1 to 5). Although RFC 3629 restricts UTF-8 - characters to be no longer than 4 bytes, the encoding scheme (origi- - nally defined by RFC 2279) allows for up to 6 bytes, and this is + The string ends with a truncated UTF-8 character; the code specifies + how many bytes are missing (1 to 5). Although RFC 3629 restricts UTF-8 + characters to be no longer than 4 bytes, the encoding scheme (origi- + nally defined by RFC 2279) allows for up to 6 bytes, and this is checked first; hence the possibility of 4 or 5 missing bytes. PCRE_UTF8_ERR6 @@ -3641,24 +3649,24 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_UTF8_ERR10 The two most significant bits of the 2nd, 3rd, 4th, 5th, or 6th byte of - the character do not have the binary value 0b10 (that is, either the + the character do not have the binary value 0b10 (that is, either the most significant bit is 0, or the next bit is 1). PCRE_UTF8_ERR11 PCRE_UTF8_ERR12 - A character that is valid by the RFC 2279 rules is either 5 or 6 bytes + A character that is valid by the RFC 2279 rules is either 5 or 6 bytes long; these code points are excluded by RFC 3629. PCRE_UTF8_ERR13 - A 4-byte character has a value greater than 0x10fff; these code points + A 4-byte character has a value greater than 0x10fff; these code points are excluded by RFC 3629. PCRE_UTF8_ERR14 - A 3-byte character has a value in the range 0xd800 to 0xdfff; this - range of code points are reserved by RFC 3629 for use with UTF-16, and + A 3-byte character has a value in the range 0xd800 to 0xdfff; this + range of code points are reserved by RFC 3629 for use with UTF-16, and so are excluded from UTF-8. PCRE_UTF8_ERR15 @@ -3667,28 +3675,29 @@ MATCHING A PATTERN: THE TRADITIONAL FUNCTION PCRE_UTF8_ERR18 PCRE_UTF8_ERR19 - A 2-, 3-, 4-, 5-, or 6-byte character is "overlong", that is, it codes - for a value that can be represented by fewer bytes, which is invalid. - For example, the two bytes 0xc0, 0xae give the value 0x2e, whose cor- + A 2-, 3-, 4-, 5-, or 6-byte character is "overlong", that is, it codes + for a value that can be represented by fewer bytes, which is invalid. + For example, the two bytes 0xc0, 0xae give the value 0x2e, whose cor- rect coding uses just one byte. PCRE_UTF8_ERR20 The two most significant bits of the first byte of a character have the - binary value 0b10 (that is, the most significant bit is 1 and the sec- - ond is 0). Such a byte can only validly occur as the second or subse- + binary value 0b10 (that is, the most significant bit is 1 and the sec- + ond is 0). Such a byte can only validly occur as the second or subse- quent byte of a multi-byte character. PCRE_UTF8_ERR21 - The first byte of a character has the value 0xfe or 0xff. These values + The first byte of a character has the value 0xfe or 0xff. These values can never occur in a valid UTF-8 string. - PCRE_UTF8_ERR2 + PCRE_UTF8_ERR22 - Non-character. These are the last two characters in each plane (0xfffe, - 0xffff, 0x1fffe, 0x1ffff .. 0x10fffe, 0x10ffff), and the characters - 0xfdd0..0xfdef. + This error code was formerly used when the presence of a so-called + "non-character" caused an error. Unicode corrigendum #9 makes it clear + that such characters should not cause a string to be rejected, and so + this code is no longer in use and is never returned. EXTRACTING CAPTURED SUBSTRINGS BY NUMBER @@ -4101,18 +4110,18 @@ AUTHOR REVISION - Last updated: 08 November 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 27 February 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRECALLOUT(3) Library Functions Manual PCRECALLOUT(3) -PCRECALLOUT(3) PCRECALLOUT(3) - NAME PCRE - Perl-compatible regular expressions - SYNOPSIS #include <pcre.h> @@ -4153,15 +4162,21 @@ DESCRIPTION (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255) Notice that there is a callout before and after each parenthesis and - alternation bar. Automatic callouts can be used for tracking the - progress of pattern matching. The pcretest command has an option that - sets automatic callouts; when it is used, the output indicates how the - pattern is matched. This is useful information when you are trying to - optimize the performance of a particular pattern. + alternation bar. If the pattern contains a conditional group whose con- + dition is an assertion, an automatic callout is inserted immediately + before the condition. Such a callout may also be inserted explicitly, + for example: + + (?(?C9)(?=a)ab|de) + + This applies only to assertion conditions (because they are themselves + independent groups). - The use of callouts in a pattern makes it ineligible for optimization - by the just-in-time compiler. Studying such a pattern with the - PCRE_STUDY_JIT_COMPILE option always fails. + Automatic callouts can be used for tracking the progress of pattern + matching. The pcretest command has an option that sets automatic call- + outs; when it is used, the output indicates how the pattern is matched. + This is useful information when you are trying to optimize the perfor- + mance of a particular pattern. MISSING CALLOUTS @@ -4251,53 +4266,56 @@ THE CALLOUT INTERFACE are used, because they do not support captured substrings. The capture_last field contains the number of the most recently cap- - tured substring. If no substrings have been captured, its value is -1. - This is always the case for the DFA matching functions. - - The callout_data field contains a value that is passed to a matching - function specifically so that it can be passed back in callouts. It is - passed in the callout_data field of a pcre_extra or pcre[16|32]_extra - data structure. If no such data was passed, the value of callout_data - in a callout block is NULL. There is a description of the pcre_extra + tured substring. However, when a recursion exits, the value reverts to + what it was outside the recursion, as do the values of all captured + substrings. If no substrings have been captured, the value of cap- + ture_last is -1. This is always the case for the DFA matching func- + tions. + + The callout_data field contains a value that is passed to a matching + function specifically so that it can be passed back in callouts. It is + passed in the callout_data field of a pcre_extra or pcre[16|32]_extra + data structure. If no such data was passed, the value of callout_data + in a callout block is NULL. There is a description of the pcre_extra structure in the pcreapi documentation. - The pattern_position field is present from version 1 of the callout + The pattern_position field is present from version 1 of the callout structure. It contains the offset to the next item to be matched in the pattern string. - The next_item_length field is present from version 1 of the callout + The next_item_length field is present from version 1 of the callout structure. It contains the length of the next item to be matched in the - pattern string. When the callout immediately precedes an alternation - bar, a closing parenthesis, or the end of the pattern, the length is - zero. When the callout precedes an opening parenthesis, the length is + pattern string. When the callout immediately precedes an alternation + bar, a closing parenthesis, or the end of the pattern, the length is + zero. When the callout precedes an opening parenthesis, the length is that of the entire subpattern. - The pattern_position and next_item_length fields are intended to help - in distinguishing between different automatic callouts, which all have + The pattern_position and next_item_length fields are intended to help + in distinguishing between different automatic callouts, which all have the same callout number. However, they are set for all callouts. - The mark field is present from version 2 of the callout structure. In - callouts from pcre_exec() or pcre[16|32]_exec() it contains a pointer - to the zero-terminated name of the most recently passed (*MARK), - (*PRUNE), or (*THEN) item in the match, or NULL if no such items have - been passed. Instances of (*PRUNE) or (*THEN) without a name do not - obliterate a previous (*MARK). In callouts from the DFA matching func- + The mark field is present from version 2 of the callout structure. In + callouts from pcre_exec() or pcre[16|32]_exec() it contains a pointer + to the zero-terminated name of the most recently passed (*MARK), + (*PRUNE), or (*THEN) item in the match, or NULL if no such items have + been passed. Instances of (*PRUNE) or (*THEN) without a name do not + obliterate a previous (*MARK). In callouts from the DFA matching func- tions this field always contains NULL. RETURN VALUES - The external callout function returns an integer to PCRE. If the value - is zero, matching proceeds as normal. If the value is greater than - zero, matching fails at the current point, but the testing of other + The external callout function returns an integer to PCRE. If the value + is zero, matching proceeds as normal. If the value is greater than + zero, matching fails at the current point, but the testing of other matching possibilities goes ahead, just as if a lookahead assertion had - failed. If the value is less than zero, the match is abandoned, the + failed. If the value is less than zero, the match is abandoned, the matching function returns the negative value. - Negative values should normally be chosen from the set of + Negative values should normally be chosen from the set of PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan- - dard "no match" failure. The error number PCRE_ERROR_CALLOUT is - reserved for use by callout functions; it will never be used by PCRE + dard "no match" failure. The error number PCRE_ERROR_CALLOUT is + reserved for use by callout functions; it will never be used by PCRE itself. @@ -4310,18 +4328,18 @@ AUTHOR REVISION - Last updated: 24 June 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 03 March 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRECOMPAT(3) Library Functions Manual PCRECOMPAT(3) -PCRECOMPAT(3) PCRECOMPAT(3) - NAME PCRE - Perl-compatible regular expressions - DIFFERENCES BETWEEN PCRE AND PERL This document describes the differences in the ways that PCRE and Perl @@ -4340,10 +4358,8 @@ DIFFERENCES BETWEEN PCRE AND PERL 3. Capturing subpatterns that occur inside negative lookahead asser- tions are counted, but their entries in the offsets vector are never - set. Perl sets its numerical variables from any such patterns that are - matched before the assertion fails to match something (thereby succeed- - ing), but only if the negative lookahead assertion contains just one - branch. + set. Perl sometimes (but not always) sets its numerical variables from + inside negative assertions. 4. Though binary zero characters are supported in the subject string, they are not allowed in a pattern string because it is passed as a nor- @@ -4398,24 +4414,30 @@ DIFFERENCES BETWEEN PCRE AND PERL There is a discussion that explains these differences in more detail in the section on recursion differences from Perl in the pcrepattern page. - 10. If any of the backtracking control verbs are used in an assertion - or in a subpattern that is called as a subroutine (whether or not - recursively), their effect is confined to that subpattern; it does not - extend to the surrounding pattern. This is not always the case in Perl. - In particular, if (*THEN) is present in a group that is called as a - subroutine, its action is limited to that group, even if the group does - not contain any | characters. There is one exception to this: the name - from a *(MARK), (*PRUNE), or (*THEN) that is encountered in a success- - ful positive assertion is passed back when a match succeeds (compare - capturing parentheses in assertions). Note that such subpatterns are - processed as anchored at the point where they are tested. - - 11. There are some differences that are concerned with the settings of + 10. If any of the backtracking control verbs are used in a subpattern + that is called as a subroutine (whether or not recursively), their + effect is confined to that subpattern; it does not extend to the sur- + rounding pattern. This is not always the case in Perl. In particular, + if (*THEN) is present in a group that is called as a subroutine, its + action is limited to that group, even if the group does not contain any + | characters. Note that such subpatterns are processed as anchored at + the point where they are tested. + + 11. If a pattern contains more than one backtracking control verb, the + first one that is backtracked onto acts. For example, in the pattern + A(*COMMIT)B(*PRUNE)C a failure in B triggers (*COMMIT), but a failure + in C triggers (*PRUNE). Perl's behaviour is more complex; in many cases + it is the same as PCRE, but there are examples where it differs. + + 12. Most backtracking verbs in assertions have their normal actions. + They are not confined to the assertion. + + 13. There are some differences that are concerned with the settings of captured strings when part of a pattern is repeated. For example, matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2 unset, but in PCRE it is set to "b". - 12. PCRE's handling of duplicate subpattern numbers and duplicate sub- + 14. PCRE's handling of duplicate subpattern numbers and duplicate sub- pattern names is not as general as Perl's. This is a consequence of the fact the PCRE works internally just with numbers, using an external ta- ble to translate between numbers and names. In particular, a pattern @@ -4426,29 +4448,36 @@ DIFFERENCES BETWEEN PCRE AND PERL turing subpattern number 1. To avoid this confusing situation, an error is given at compile time. - 13. Perl recognizes comments in some places that PCRE does not, for + 15. Perl recognizes comments in some places that PCRE does not, for example, between the ( and ? at the start of a subpattern. If the /x modifier is set, Perl allows white space between ( and ? but PCRE never does, even if the PCRE_EXTENDED option is set. - 14. PCRE provides some extensions to the Perl regular expression facil- - ities. Perl 5.10 includes new features that are not in earlier ver- - sions of Perl, some of which (such as named parentheses) have been in + 16. In PCRE, the upper/lower case character properties Lu and Ll are + not affected when case-independent matching is specified. For example, + \p{Lu} always matches an upper case letter. I think Perl has changed in + this respect; in the release at the time of writing (5.16), \p{Lu} and + \p{Ll} match all letters, regardless of case, when case independence is + specified. + + 17. PCRE provides some extensions to the Perl regular expression facil- + ities. Perl 5.10 includes new features that are not in earlier ver- + sions of Perl, some of which (such as named parentheses) have been in PCRE for some time. This list is with respect to Perl 5.10: - (a) Although lookbehind assertions in PCRE must match fixed length - strings, each alternative branch of a lookbehind assertion can match a - different length of string. Perl requires them all to have the same + (a) Although lookbehind assertions in PCRE must match fixed length + strings, each alternative branch of a lookbehind assertion can match a + different length of string. Perl requires them all to have the same length. - (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $ + (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $ meta-character matches only at the very end of the string. (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe- cial meaning is faulted. Otherwise, like Perl, the backslash is quietly ignored. (Perl can be made to issue a warning.) - (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti- + (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti- fiers is inverted, that is, by default they are not greedy, but if fol- lowed by a question mark they are. @@ -4456,10 +4485,10 @@ DIFFERENCES BETWEEN PCRE AND PERL tried only at the first matching position in the subject string. (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, - and PCRE_NO_AUTO_CAPTURE options for pcre_exec() have no Perl equiva- + and PCRE_NO_AUTO_CAPTURE options for pcre_exec() have no Perl equiva- lents. - (g) The \R escape sequence can be restricted to match only CR, LF, or + (g) The \R escape sequence can be restricted to match only CR, LF, or CRLF by the PCRE_BSR_ANYCRLF option. (h) The callout facility is PCRE-specific. @@ -4467,14 +4496,14 @@ DIFFERENCES BETWEEN PCRE AND PERL (i) The partial matching facility is PCRE-specific. (j) Patterns compiled by PCRE can be saved and re-used at a later time, - even on different hosts that have the other endianness. However, this + even on different hosts that have the other endianness. However, this does not apply to optimized data created by the just-in-time compiler. (k) The alternative matching functions (pcre_dfa_exec(), - pcre16_dfa_exec() and pcre32_dfa_exec(),) match in a different way and + pcre16_dfa_exec() and pcre32_dfa_exec(),) match in a different way and are not Perl-compatible. - (l) PCRE recognizes some special sequences such as (*CR) at the start + (l) PCRE recognizes some special sequences such as (*CR) at the start of a pattern that set overall options that cannot be changed within the pattern. @@ -4488,18 +4517,18 @@ AUTHOR REVISION - Last updated: 25 August 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 19 March 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREPATTERN(3) Library Functions Manual PCREPATTERN(3) -PCREPATTERN(3) PCREPATTERN(3) - NAME PCRE - Perl-compatible regular expressions - PCRE REGULAR EXPRESSION DETAILS The syntax and semantics of the regular expressions that are supported @@ -5136,52 +5165,53 @@ BACKSLASH in the Unicode table. Specifying caseless matching does not affect these escape sequences. - For example, \p{Lu} always matches only upper case letters. + For example, \p{Lu} always matches only upper case letters. This is + different from the behaviour of current versions of Perl. - Matching characters by Unicode property is not fast, because PCRE has - to do a multistage table lookup in order to find a character's prop- + Matching characters by Unicode property is not fast, because PCRE has + to do a multistage table lookup in order to find a character's prop- erty. That is why the traditional escape sequences such as \d and \w do not use Unicode properties in PCRE by default, though you can make them - do so by setting the PCRE_UCP option or by starting the pattern with + do so by setting the PCRE_UCP option or by starting the pattern with (*UCP). Extended grapheme clusters - The \X escape matches any number of Unicode characters that form an + The \X escape matches any number of Unicode characters that form an "extended grapheme cluster", and treats the sequence as an atomic group - (see below). Up to and including release 8.31, PCRE matched an ear- + (see below). Up to and including release 8.31, PCRE matched an ear- lier, simpler definition that was equivalent to (?>\PM\pM*) - That is, it matched a character without the "mark" property, followed - by zero or more characters with the "mark" property. Characters with - the "mark" property are typically non-spacing accents that affect the + That is, it matched a character without the "mark" property, followed + by zero or more characters with the "mark" property. Characters with + the "mark" property are typically non-spacing accents that affect the preceding character. - This simple definition was extended in Unicode to include more compli- - cated kinds of composite character by giving each character a grapheme - breaking property, and creating rules that use these properties to - define the boundaries of extended grapheme clusters. In releases of + This simple definition was extended in Unicode to include more compli- + cated kinds of composite character by giving each character a grapheme + breaking property, and creating rules that use these properties to + define the boundaries of extended grapheme clusters. In releases of PCRE later than 8.31, \X matches one of these clusters. - \X always matches at least one character. Then it decides whether to + \X always matches at least one character. Then it decides whether to add additional characters according to the following rules for ending a cluster: 1. End at the end of the subject string. - 2. Do not end between CR and LF; otherwise end after any control char- + 2. Do not end between CR and LF; otherwise end after any control char- acter. - 3. Do not break Hangul (a Korean script) syllable sequences. Hangul - characters are of five types: L, V, T, LV, and LVT. An L character may - be followed by an L, V, LV, or LVT character; an LV or V character may + 3. Do not break Hangul (a Korean script) syllable sequences. Hangul + characters are of five types: L, V, T, LV, and LVT. An L character may + be followed by an L, V, LV, or LVT character; an LV or V character may be followed by a V or T character; an LVT or T character may be follwed only by a T character. - 4. Do not end before extending characters or spacing marks. Characters - with the "mark" property always have the "extend" grapheme breaking + 4. Do not end before extending characters or spacing marks. Characters + with the "mark" property always have the "extend" grapheme breaking property. 5. Do not end after prepend characters. @@ -5190,11 +5220,12 @@ BACKSLASH PCRE's additional properties - As well as the standard Unicode properties described above, PCRE sup- - ports four more that make it possible to convert traditional escape - sequences such as \w and \s and POSIX character classes to use Unicode - properties. PCRE uses these non-standard, non-Perl properties inter- - nally when PCRE_UCP is set. They are: + As well as the standard Unicode properties described above, PCRE sup- + ports four more that make it possible to convert traditional escape + sequences such as \w and \s and POSIX character classes to use Unicode + properties. PCRE uses these non-standard, non-Perl properties inter- + nally when PCRE_UCP is set. However, they may also be used explicitly. + These properties are: Xan Any alphanumeric character Xps Any POSIX space character @@ -5207,6 +5238,16 @@ BACKSLASH (separator) property. Xsp is the same as Xps, except that vertical tab is excluded. Xwd matches the same characters as Xan, plus underscore. + There is another non-standard property, Xuc, which matches any charac- + ter that can be represented by a Universal Character Name in C++ and + other programming languages. These are the characters $, @, ` (grave + accent), and all characters with Unicode code points greater than or + equal to U+00A0, except for the surrogates U+D800 to U+DFFF. Note that + most base (ASCII) characters are excluded. (Universal Character Names + are of the form \uHHHH or \UHHHHHHHH where H is a hexadecimal digit. + Note that the Xuc property does not match these sequences but the char- + acters that they represent.) + Resetting the match start The escape sequence \K causes any previously matched characters not to @@ -6227,24 +6268,25 @@ ASSERTIONS tion contains capturing subpatterns within it, these are counted for the purposes of numbering the capturing subpatterns in the whole pat- tern. However, substring capturing is carried out only for positive - assertions, because it does not make sense for negative assertions. + assertions. (Perl sometimes, but not always, does do capturing in nega- + tive assertions.) - For compatibility with Perl, assertion subpatterns may be repeated; - though it makes no sense to assert the same thing several times, the - side effect of capturing parentheses may occasionally be useful. In + For compatibility with Perl, assertion subpatterns may be repeated; + though it makes no sense to assert the same thing several times, the + side effect of capturing parentheses may occasionally be useful. In practice, there only three cases: - (1) If the quantifier is {0}, the assertion is never obeyed during - matching. However, it may contain internal capturing parenthesized + (1) If the quantifier is {0}, the assertion is never obeyed during + matching. However, it may contain internal capturing parenthesized groups that are called from elsewhere via the subroutine mechanism. - (2) If quantifier is {0,n} where n is greater than zero, it is treated - as if it were {0,1}. At run time, the rest of the pattern match is + (2) If quantifier is {0,n} where n is greater than zero, it is treated + as if it were {0,1}. At run time, the rest of the pattern match is tried with and without the assertion, the order depending on the greed- iness of the quantifier. - (3) If the minimum repetition is greater than zero, the quantifier is - ignored. The assertion is obeyed just once when encountered during + (3) If the minimum repetition is greater than zero, the quantifier is + ignored. The assertion is obeyed just once when encountered during matching. Lookahead assertions @@ -6254,38 +6296,38 @@ ASSERTIONS \w+(?=;) - matches a word followed by a semicolon, but does not include the semi- + matches a word followed by a semicolon, but does not include the semi- colon in the match, and foo(?!bar) - matches any occurrence of "foo" that is not followed by "bar". Note + matches any occurrence of "foo" that is not followed by "bar". Note that the apparently similar pattern (?!foo)bar - does not find an occurrence of "bar" that is preceded by something - other than "foo"; it finds any occurrence of "bar" whatsoever, because + does not find an occurrence of "bar" that is preceded by something + other than "foo"; it finds any occurrence of "bar" whatsoever, because the assertion (?!foo) is always true when the next three characters are "bar". A lookbehind assertion is needed to achieve the other effect. If you want to force a matching failure at some point in a pattern, the - most convenient way to do it is with (?!) because an empty string - always matches, so an assertion that requires there not to be an empty + most convenient way to do it is with (?!) because an empty string + always matches, so an assertion that requires there not to be an empty string must always fail. The backtracking control verb (*FAIL) or (*F) is a synonym for (?!). Lookbehind assertions - Lookbehind assertions start with (?<= for positive assertions and (?<! + Lookbehind assertions start with (?<= for positive assertions and (?<! for negative assertions. For example, (?<!foo)bar - does find an occurrence of "bar" that is not preceded by "foo". The - contents of a lookbehind assertion are restricted such that all the + does find an occurrence of "bar" that is not preceded by "foo". The + contents of a lookbehind assertion are restricted such that all the strings it matches must have a fixed length. However, if there are sev- - eral top-level alternatives, they do not all have to have the same + eral top-level alternatives, they do not all have to have the same fixed length. Thus (?<=bullock|donkey) @@ -6294,62 +6336,62 @@ ASSERTIONS (?<!dogs?|cats?) - causes an error at compile time. Branches that match different length - strings are permitted only at the top level of a lookbehind assertion. + causes an error at compile time. Branches that match different length + strings are permitted only at the top level of a lookbehind assertion. This is an extension compared with Perl, which requires all branches to match the same length of string. An assertion such as (?<=ab(c|de)) - is not permitted, because its single top-level branch can match two + is not permitted, because its single top-level branch can match two different lengths, but it is acceptable to PCRE if rewritten to use two top-level branches: (?<=abc|abde) - In some cases, the escape sequence \K (see above) can be used instead + In some cases, the escape sequence \K (see above) can be used instead of a lookbehind assertion to get round the fixed-length restriction. - The implementation of lookbehind assertions is, for each alternative, - to temporarily move the current position back by the fixed length and + The implementation of lookbehind assertions is, for each alternative, + to temporarily move the current position back by the fixed length and then try to match. If there are insufficient characters before the cur- rent position, the assertion fails. - In a UTF mode, PCRE does not allow the \C escape (which matches a sin- - gle data unit even in a UTF mode) to appear in lookbehind assertions, - because it makes it impossible to calculate the length of the lookbe- - hind. The \X and \R escapes, which can match different numbers of data + In a UTF mode, PCRE does not allow the \C escape (which matches a sin- + gle data unit even in a UTF mode) to appear in lookbehind assertions, + because it makes it impossible to calculate the length of the lookbe- + hind. The \X and \R escapes, which can match different numbers of data units, are also not permitted. - "Subroutine" calls (see below) such as (?2) or (?&X) are permitted in - lookbehinds, as long as the subpattern matches a fixed-length string. + "Subroutine" calls (see below) such as (?2) or (?&X) are permitted in + lookbehinds, as long as the subpattern matches a fixed-length string. Recursion, however, is not supported. - Possessive quantifiers can be used in conjunction with lookbehind + Possessive quantifiers can be used in conjunction with lookbehind assertions to specify efficient matching of fixed-length strings at the end of subject strings. Consider a simple pattern such as abcd$ - when applied to a long string that does not match. Because matching + when applied to a long string that does not match. Because matching proceeds from left to right, PCRE will look for each "a" in the subject - and then see if what follows matches the rest of the pattern. If the + and then see if what follows matches the rest of the pattern. If the pattern is specified as ^.*abcd$ - the initial .* matches the entire string at first, but when this fails + the initial .* matches the entire string at first, but when this fails (because there is no following "a"), it backtracks to match all but the - last character, then all but the last two characters, and so on. Once - again the search for "a" covers the entire string, from right to left, + last character, then all but the last two characters, and so on. Once + again the search for "a" covers the entire string, from right to left, so we are no better off. However, if the pattern is written as ^.*+(?<=abcd) - there can be no backtracking for the .*+ item; it can match only the - entire string. The subsequent lookbehind assertion does a single test - on the last four characters. If it fails, the match fails immediately. - For long strings, this approach makes a significant difference to the + there can be no backtracking for the .*+ item; it can match only the + entire string. The subsequent lookbehind assertion does a single test + on the last four characters. If it fails, the match fails immediately. + For long strings, this approach makes a significant difference to the processing time. Using multiple assertions @@ -6358,18 +6400,18 @@ ASSERTIONS (?<=\d{3})(?<!999)foo - matches "foo" preceded by three digits that are not "999". Notice that - each of the assertions is applied independently at the same point in - the subject string. First there is a check that the previous three - characters are all digits, and then there is a check that the same + matches "foo" preceded by three digits that are not "999". Notice that + each of the assertions is applied independently at the same point in + the subject string. First there is a check that the previous three + characters are all digits, and then there is a check that the same three characters are not "999". This pattern does not match "foo" pre- - ceded by six characters, the first of which are digits and the last - three of which are not "999". For example, it doesn't match "123abc- + ceded by six characters, the first of which are digits and the last + three of which are not "999". For example, it doesn't match "123abc- foo". A pattern to do that is (?<=\d{3}...)(?<!999)foo - This time the first assertion looks at the preceding six characters, + This time the first assertion looks at the preceding six characters, checking that the first three are digits, and then the second assertion checks that the preceding three characters are not "999". @@ -6377,29 +6419,29 @@ ASSERTIONS (?<=(?<!foo)bar)baz - matches an occurrence of "baz" that is preceded by "bar" which in turn + matches an occurrence of "baz" that is preceded by "bar" which in turn is not preceded by "foo", while (?<=\d{3}(?!999)...)foo - is another pattern that matches "foo" preceded by three digits and any + is another pattern that matches "foo" preceded by three digits and any three characters that are not "999". CONDITIONAL SUBPATTERNS - It is possible to cause the matching process to obey a subpattern con- - ditionally or to choose between two alternative subpatterns, depending - on the result of an assertion, or whether a specific capturing subpat- - tern has already been matched. The two possible forms of conditional + It is possible to cause the matching process to obey a subpattern con- + ditionally or to choose between two alternative subpatterns, depending + on the result of an assertion, or whether a specific capturing subpat- + tern has already been matched. The two possible forms of conditional subpattern are: (?(condition)yes-pattern) (?(condition)yes-pattern|no-pattern) - If the condition is satisfied, the yes-pattern is used; otherwise the - no-pattern (if present) is used. If there are more than two alterna- - tives in the subpattern, a compile-time error occurs. Each of the two + If the condition is satisfied, the yes-pattern is used; otherwise the + no-pattern (if present) is used. If there are more than two alterna- + tives in the subpattern, a compile-time error occurs. Each of the two alternatives may itself contain nested subpatterns of any form, includ- ing conditional subpatterns; the restriction to two alternatives applies only at the level of the condition. This pattern fragment is an @@ -6408,73 +6450,73 @@ CONDITIONAL SUBPATTERNS (?(1) (A|B|C) | (D | (?(2)E|F) | E) ) - There are four kinds of condition: references to subpatterns, refer- + There are four kinds of condition: references to subpatterns, refer- ences to recursion, a pseudo-condition called DEFINE, and assertions. Checking for a used subpattern by number - If the text between the parentheses consists of a sequence of digits, + If the text between the parentheses consists of a sequence of digits, the condition is true if a capturing subpattern of that number has pre- - viously matched. If there is more than one capturing subpattern with - the same number (see the earlier section about duplicate subpattern - numbers), the condition is true if any of them have matched. An alter- - native notation is to precede the digits with a plus or minus sign. In - this case, the subpattern number is relative rather than absolute. The - most recently opened parentheses can be referenced by (?(-1), the next - most recent by (?(-2), and so on. Inside loops it can also make sense + viously matched. If there is more than one capturing subpattern with + the same number (see the earlier section about duplicate subpattern + numbers), the condition is true if any of them have matched. An alter- + native notation is to precede the digits with a plus or minus sign. In + this case, the subpattern number is relative rather than absolute. The + most recently opened parentheses can be referenced by (?(-1), the next + most recent by (?(-2), and so on. Inside loops it can also make sense to refer to subsequent groups. The next parentheses to be opened can be - referenced as (?(+1), and so on. (The value zero in any of these forms + referenced as (?(+1), and so on. (The value zero in any of these forms is not used; it provokes a compile-time error.) - Consider the following pattern, which contains non-significant white + Consider the following pattern, which contains non-significant white space to make it more readable (assume the PCRE_EXTENDED option) and to divide it into three parts for ease of discussion: ( \( )? [^()]+ (?(1) \) ) - The first part matches an optional opening parenthesis, and if that + The first part matches an optional opening parenthesis, and if that character is present, sets it as the first captured substring. The sec- - ond part matches one or more characters that are not parentheses. The - third part is a conditional subpattern that tests whether or not the - first set of parentheses matched. If they did, that is, if subject - started with an opening parenthesis, the condition is true, and so the - yes-pattern is executed and a closing parenthesis is required. Other- - wise, since no-pattern is not present, the subpattern matches nothing. - In other words, this pattern matches a sequence of non-parentheses, + ond part matches one or more characters that are not parentheses. The + third part is a conditional subpattern that tests whether or not the + first set of parentheses matched. If they did, that is, if subject + started with an opening parenthesis, the condition is true, and so the + yes-pattern is executed and a closing parenthesis is required. Other- + wise, since no-pattern is not present, the subpattern matches nothing. + In other words, this pattern matches a sequence of non-parentheses, optionally enclosed in parentheses. - If you were embedding this pattern in a larger one, you could use a + If you were embedding this pattern in a larger one, you could use a relative reference: ...other stuff... ( \( )? [^()]+ (?(-1) \) ) ... - This makes the fragment independent of the parentheses in the larger + This makes the fragment independent of the parentheses in the larger pattern. Checking for a used subpattern by name - Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a - used subpattern by name. For compatibility with earlier versions of - PCRE, which had this facility before Perl, the syntax (?(name)...) is - also recognized. However, there is a possible ambiguity with this syn- - tax, because subpattern names may consist entirely of digits. PCRE - looks first for a named subpattern; if it cannot find one and the name - consists entirely of digits, PCRE looks for a subpattern of that num- - ber, which must be greater than zero. Using subpattern names that con- + Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a + used subpattern by name. For compatibility with earlier versions of + PCRE, which had this facility before Perl, the syntax (?(name)...) is + also recognized. However, there is a possible ambiguity with this syn- + tax, because subpattern names may consist entirely of digits. PCRE + looks first for a named subpattern; if it cannot find one and the name + consists entirely of digits, PCRE looks for a subpattern of that num- + ber, which must be greater than zero. Using subpattern names that con- sist entirely of digits is not recommended. Rewriting the above example to use a named subpattern gives this: (?<OPEN> \( )? [^()]+ (?(<OPEN>) \) ) - If the name used in a condition of this kind is a duplicate, the test - is applied to all subpatterns of the same name, and is true if any one + If the name used in a condition of this kind is a duplicate, the test + is applied to all subpatterns of the same name, and is true if any one of them has matched. Checking for pattern recursion If the condition is the string (R), and there is no subpattern with the - name R, the condition is true if a recursive call to the whole pattern + name R, the condition is true if a recursive call to the whole pattern or any subpattern has been made. If digits or a name preceded by amper- sand follow the letter R, for example: @@ -6482,51 +6524,51 @@ CONDITIONAL SUBPATTERNS the condition is true if the most recent recursion is into a subpattern whose number or name is given. This condition does not check the entire - recursion stack. If the name used in a condition of this kind is a + recursion stack. If the name used in a condition of this kind is a duplicate, the test is applied to all subpatterns of the same name, and is true if any one of them is the most recent recursion. - At "top level", all these recursion test conditions are false. The + At "top level", all these recursion test conditions are false. The syntax for recursive patterns is described below. Defining subpatterns for use by reference only - If the condition is the string (DEFINE), and there is no subpattern - with the name DEFINE, the condition is always false. In this case, - there may be only one alternative in the subpattern. It is always - skipped if control reaches this point in the pattern; the idea of - DEFINE is that it can be used to define subroutines that can be refer- - enced from elsewhere. (The use of subroutines is described below.) For - example, a pattern to match an IPv4 address such as "192.168.23.245" + If the condition is the string (DEFINE), and there is no subpattern + with the name DEFINE, the condition is always false. In this case, + there may be only one alternative in the subpattern. It is always + skipped if control reaches this point in the pattern; the idea of + DEFINE is that it can be used to define subroutines that can be refer- + enced from elsewhere. (The use of subroutines is described below.) For + example, a pattern to match an IPv4 address such as "192.168.23.245" could be written like this (ignore white space and line breaks): (?(DEFINE) (?<byte> 2[0-4]\d | 25[0-5] | 1\d\d | [1-9]?\d) ) \b (?&byte) (\.(?&byte)){3} \b - The first part of the pattern is a DEFINE group inside which a another - group named "byte" is defined. This matches an individual component of - an IPv4 address (a number less than 256). When matching takes place, - this part of the pattern is skipped because DEFINE acts like a false - condition. The rest of the pattern uses references to the named group - to match the four dot-separated components of an IPv4 address, insist- + The first part of the pattern is a DEFINE group inside which a another + group named "byte" is defined. This matches an individual component of + an IPv4 address (a number less than 256). When matching takes place, + this part of the pattern is skipped because DEFINE acts like a false + condition. The rest of the pattern uses references to the named group + to match the four dot-separated components of an IPv4 address, insist- ing on a word boundary at each end. Assertion conditions - If the condition is not in any of the above formats, it must be an - assertion. This may be a positive or negative lookahead or lookbehind - assertion. Consider this pattern, again containing non-significant + If the condition is not in any of the above formats, it must be an + assertion. This may be a positive or negative lookahead or lookbehind + assertion. Consider this pattern, again containing non-significant white space, and with the two alternatives on the second line: (?(?=[^a-z]*[a-z]) \d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} ) - The condition is a positive lookahead assertion that matches an - optional sequence of non-letters followed by a letter. In other words, - it tests for the presence of at least one letter in the subject. If a - letter is found, the subject is matched against the first alternative; - otherwise it is matched against the second. This pattern matches - strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are + The condition is a positive lookahead assertion that matches an + optional sequence of non-letters followed by a letter. In other words, + it tests for the presence of at least one letter in the subject. If a + letter is found, the subject is matched against the first alternative; + otherwise it is matched against the second. This pattern matches + strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are letters and dd are digits. @@ -6535,41 +6577,41 @@ COMMENTS There are two ways of including comments in patterns that are processed by PCRE. In both cases, the start of the comment must not be in a char- acter class, nor in the middle of any other sequence of related charac- - ters such as (?: or a subpattern name or number. The characters that + ters such as (?: or a subpattern name or number. The characters that make up a comment play no part in the pattern matching. - The sequence (?# marks the start of a comment that continues up to the - next closing parenthesis. Nested parentheses are not permitted. If the + The sequence (?# marks the start of a comment that continues up to the + next closing parenthesis. Nested parentheses are not permitted. If the PCRE_EXTENDED option is set, an unescaped # character also introduces a - comment, which in this case continues to immediately after the next - newline character or character sequence in the pattern. Which charac- + comment, which in this case continues to immediately after the next + newline character or character sequence in the pattern. Which charac- ters are interpreted as newlines is controlled by the options passed to - a compiling function or by a special sequence at the start of the pat- + a compiling function or by a special sequence at the start of the pat- tern, as described in the section entitled "Newline conventions" above. Note that the end of this type of comment is a literal newline sequence - in the pattern; escape sequences that happen to represent a newline do - not count. For example, consider this pattern when PCRE_EXTENDED is + in the pattern; escape sequences that happen to represent a newline do + not count. For example, consider this pattern when PCRE_EXTENDED is set, and the default newline convention is in force: abc #comment \n still comment - On encountering the # character, pcre_compile() skips along, looking - for a newline in the pattern. The sequence \n is still literal at this - stage, so it does not terminate the comment. Only an actual character + On encountering the # character, pcre_compile() skips along, looking + for a newline in the pattern. The sequence \n is still literal at this + stage, so it does not terminate the comment. Only an actual character with the code value 0x0a (the default newline) does so. RECURSIVE PATTERNS - Consider the problem of matching a string in parentheses, allowing for - unlimited nested parentheses. Without the use of recursion, the best - that can be done is to use a pattern that matches up to some fixed - depth of nesting. It is not possible to handle an arbitrary nesting + Consider the problem of matching a string in parentheses, allowing for + unlimited nested parentheses. Without the use of recursion, the best + that can be done is to use a pattern that matches up to some fixed + depth of nesting. It is not possible to handle an arbitrary nesting depth. For some time, Perl has provided a facility that allows regular expres- - sions to recurse (amongst other things). It does this by interpolating - Perl code in the expression at run time, and the code can refer to the + sions to recurse (amongst other things). It does this by interpolating + Perl code in the expression at run time, and the code can refer to the expression itself. A Perl pattern using code interpolation to solve the parentheses problem can be created like this: @@ -6579,201 +6621,201 @@ RECURSIVE PATTERNS refers recursively to the pattern in which it appears. Obviously, PCRE cannot support the interpolation of Perl code. Instead, - it supports special syntax for recursion of the entire pattern, and - also for individual subpattern recursion. After its introduction in - PCRE and Python, this kind of recursion was subsequently introduced + it supports special syntax for recursion of the entire pattern, and + also for individual subpattern recursion. After its introduction in + PCRE and Python, this kind of recursion was subsequently introduced into Perl at release 5.10. - A special item that consists of (? followed by a number greater than - zero and a closing parenthesis is a recursive subroutine call of the - subpattern of the given number, provided that it occurs inside that - subpattern. (If not, it is a non-recursive subroutine call, which is - described in the next section.) The special item (?R) or (?0) is a + A special item that consists of (? followed by a number greater than + zero and a closing parenthesis is a recursive subroutine call of the + subpattern of the given number, provided that it occurs inside that + subpattern. (If not, it is a non-recursive subroutine call, which is + described in the next section.) The special item (?R) or (?0) is a recursive call of the entire regular expression. - This PCRE pattern solves the nested parentheses problem (assume the + This PCRE pattern solves the nested parentheses problem (assume the PCRE_EXTENDED option is set so that white space is ignored): \( ( [^()]++ | (?R) )* \) - First it matches an opening parenthesis. Then it matches any number of - substrings which can either be a sequence of non-parentheses, or a - recursive match of the pattern itself (that is, a correctly parenthe- + First it matches an opening parenthesis. Then it matches any number of + substrings which can either be a sequence of non-parentheses, or a + recursive match of the pattern itself (that is, a correctly parenthe- sized substring). Finally there is a closing parenthesis. Note the use of a possessive quantifier to avoid backtracking into sequences of non- parentheses. - If this were part of a larger pattern, you would not want to recurse + If this were part of a larger pattern, you would not want to recurse the entire pattern, so instead you could use this: ( \( ( [^()]++ | (?1) )* \) ) - We have put the pattern into parentheses, and caused the recursion to + We have put the pattern into parentheses, and caused the recursion to refer to them instead of the whole pattern. - In a larger pattern, keeping track of parenthesis numbers can be - tricky. This is made easier by the use of relative references. Instead + In a larger pattern, keeping track of parenthesis numbers can be + tricky. This is made easier by the use of relative references. Instead of (?1) in the pattern above you can write (?-2) to refer to the second - most recently opened parentheses preceding the recursion. In other - words, a negative number counts capturing parentheses leftwards from + most recently opened parentheses preceding the recursion. In other + words, a negative number counts capturing parentheses leftwards from the point at which it is encountered. - It is also possible to refer to subsequently opened parentheses, by - writing references such as (?+2). However, these cannot be recursive - because the reference is not inside the parentheses that are refer- - enced. They are always non-recursive subroutine calls, as described in + It is also possible to refer to subsequently opened parentheses, by + writing references such as (?+2). However, these cannot be recursive + because the reference is not inside the parentheses that are refer- + enced. They are always non-recursive subroutine calls, as described in the next section. - An alternative approach is to use named parentheses instead. The Perl - syntax for this is (?&name); PCRE's earlier syntax (?P>name) is also + An alternative approach is to use named parentheses instead. The Perl + syntax for this is (?&name); PCRE's earlier syntax (?P>name) is also supported. We could rewrite the above example as follows: (?<pn> \( ( [^()]++ | (?&pn) )* \) ) - If there is more than one subpattern with the same name, the earliest + If there is more than one subpattern with the same name, the earliest one is used. - This particular example pattern that we have been looking at contains + This particular example pattern that we have been looking at contains nested unlimited repeats, and so the use of a possessive quantifier for matching strings of non-parentheses is important when applying the pat- - tern to strings that do not match. For example, when this pattern is + tern to strings that do not match. For example, when this pattern is applied to (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa() - it yields "no match" quickly. However, if a possessive quantifier is - not used, the match runs for a very long time indeed because there are - so many different ways the + and * repeats can carve up the subject, + it yields "no match" quickly. However, if a possessive quantifier is + not used, the match runs for a very long time indeed because there are + so many different ways the + and * repeats can carve up the subject, and all have to be tested before failure can be reported. - At the end of a match, the values of capturing parentheses are those - from the outermost level. If you want to obtain intermediate values, a - callout function can be used (see below and the pcrecallout documenta- + At the end of a match, the values of capturing parentheses are those + from the outermost level. If you want to obtain intermediate values, a + callout function can be used (see below and the pcrecallout documenta- tion). If the pattern above is matched against (ab(cd)ef) - the value for the inner capturing parentheses (numbered 2) is "ef", - which is the last value taken on at the top level. If a capturing sub- - pattern is not matched at the top level, its final captured value is - unset, even if it was (temporarily) set at a deeper level during the + the value for the inner capturing parentheses (numbered 2) is "ef", + which is the last value taken on at the top level. If a capturing sub- + pattern is not matched at the top level, its final captured value is + unset, even if it was (temporarily) set at a deeper level during the matching process. - If there are more than 15 capturing parentheses in a pattern, PCRE has - to obtain extra memory to store data during a recursion, which it does + If there are more than 15 capturing parentheses in a pattern, PCRE has + to obtain extra memory to store data during a recursion, which it does by using pcre_malloc, freeing it via pcre_free afterwards. If no memory can be obtained, the match fails with the PCRE_ERROR_NOMEMORY error. - Do not confuse the (?R) item with the condition (R), which tests for - recursion. Consider this pattern, which matches text in angle brack- - ets, allowing for arbitrary nesting. Only digits are allowed in nested - brackets (that is, when recursing), whereas any characters are permit- + Do not confuse the (?R) item with the condition (R), which tests for + recursion. Consider this pattern, which matches text in angle brack- + ets, allowing for arbitrary nesting. Only digits are allowed in nested + brackets (that is, when recursing), whereas any characters are permit- ted at the outer level. < (?: (?(R) \d++ | [^<>]*+) | (?R)) * > - In this pattern, (?(R) is the start of a conditional subpattern, with - two different alternatives for the recursive and non-recursive cases. + In this pattern, (?(R) is the start of a conditional subpattern, with + two different alternatives for the recursive and non-recursive cases. The (?R) item is the actual recursive call. Differences in recursion processing between PCRE and Perl - Recursion processing in PCRE differs from Perl in two important ways. - In PCRE (like Python, but unlike Perl), a recursive subpattern call is + Recursion processing in PCRE differs from Perl in two important ways. + In PCRE (like Python, but unlike Perl), a recursive subpattern call is always treated as an atomic group. That is, once it has matched some of the subject string, it is never re-entered, even if it contains untried - alternatives and there is a subsequent matching failure. This can be - illustrated by the following pattern, which purports to match a palin- - dromic string that contains an odd number of characters (for example, + alternatives and there is a subsequent matching failure. This can be + illustrated by the following pattern, which purports to match a palin- + dromic string that contains an odd number of characters (for example, "a", "aba", "abcba", "abcdcba"): ^(.|(.)(?1)\2)$ The idea is that it either matches a single character, or two identical - characters surrounding a sub-palindrome. In Perl, this pattern works; - in PCRE it does not if the pattern is longer than three characters. + characters surrounding a sub-palindrome. In Perl, this pattern works; + in PCRE it does not if the pattern is longer than three characters. Consider the subject string "abcba": - At the top level, the first character is matched, but as it is not at + At the top level, the first character is matched, but as it is not at the end of the string, the first alternative fails; the second alterna- tive is taken and the recursion kicks in. The recursive call to subpat- - tern 1 successfully matches the next character ("b"). (Note that the + tern 1 successfully matches the next character ("b"). (Note that the beginning and end of line tests are not part of the recursion). - Back at the top level, the next character ("c") is compared with what - subpattern 2 matched, which was "a". This fails. Because the recursion - is treated as an atomic group, there are now no backtracking points, - and so the entire match fails. (Perl is able, at this point, to re- - enter the recursion and try the second alternative.) However, if the + Back at the top level, the next character ("c") is compared with what + subpattern 2 matched, which was "a". This fails. Because the recursion + is treated as an atomic group, there are now no backtracking points, + and so the entire match fails. (Perl is able, at this point, to re- + enter the recursion and try the second alternative.) However, if the pattern is written with the alternatives in the other order, things are different: ^((.)(?1)\2|.)$ - This time, the recursing alternative is tried first, and continues to - recurse until it runs out of characters, at which point the recursion - fails. But this time we do have another alternative to try at the - higher level. That is the big difference: in the previous case the + This time, the recursing alternative is tried first, and continues to + recurse until it runs out of characters, at which point the recursion + fails. But this time we do have another alternative to try at the + higher level. That is the big difference: in the previous case the remaining alternative is at a deeper recursion level, which PCRE cannot use. - To change the pattern so that it matches all palindromic strings, not - just those with an odd number of characters, it is tempting to change + To change the pattern so that it matches all palindromic strings, not + just those with an odd number of characters, it is tempting to change the pattern to this: ^((.)(?1)\2|.?)$ - Again, this works in Perl, but not in PCRE, and for the same reason. - When a deeper recursion has matched a single character, it cannot be - entered again in order to match an empty string. The solution is to - separate the two cases, and write out the odd and even cases as alter- + Again, this works in Perl, but not in PCRE, and for the same reason. + When a deeper recursion has matched a single character, it cannot be + entered again in order to match an empty string. The solution is to + separate the two cases, and write out the odd and even cases as alter- natives at the higher level: ^(?:((.)(?1)\2|)|((.)(?3)\4|.)) - If you want to match typical palindromic phrases, the pattern has to + If you want to match typical palindromic phrases, the pattern has to ignore all non-word characters, which can be done like this: ^\W*+(?:((.)\W*+(?1)\W*+\2|)|((.)\W*+(?3)\W*+\4|\W*+.\W*+))\W*+$ If run with the PCRE_CASELESS option, this pattern matches phrases such as "A man, a plan, a canal: Panama!" and it works well in both PCRE and - Perl. Note the use of the possessive quantifier *+ to avoid backtrack- - ing into sequences of non-word characters. Without this, PCRE takes a - great deal longer (ten times or more) to match typical phrases, and + Perl. Note the use of the possessive quantifier *+ to avoid backtrack- + ing into sequences of non-word characters. Without this, PCRE takes a + great deal longer (ten times or more) to match typical phrases, and Perl takes so long that you think it has gone into a loop. - WARNING: The palindrome-matching patterns above work only if the sub- - ject string does not start with a palindrome that is shorter than the - entire string. For example, although "abcba" is correctly matched, if - the subject is "ababa", PCRE finds the palindrome "aba" at the start, - then fails at top level because the end of the string does not follow. - Once again, it cannot jump back into the recursion to try other alter- + WARNING: The palindrome-matching patterns above work only if the sub- + ject string does not start with a palindrome that is shorter than the + entire string. For example, although "abcba" is correctly matched, if + the subject is "ababa", PCRE finds the palindrome "aba" at the start, + then fails at top level because the end of the string does not follow. + Once again, it cannot jump back into the recursion to try other alter- natives, so the entire match fails. - The second way in which PCRE and Perl differ in their recursion pro- - cessing is in the handling of captured values. In Perl, when a subpat- - tern is called recursively or as a subpattern (see the next section), - it has no access to any values that were captured outside the recur- - sion, whereas in PCRE these values can be referenced. Consider this + The second way in which PCRE and Perl differ in their recursion pro- + cessing is in the handling of captured values. In Perl, when a subpat- + tern is called recursively or as a subpattern (see the next section), + it has no access to any values that were captured outside the recur- + sion, whereas in PCRE these values can be referenced. Consider this pattern: ^(.)(\1|a(?2)) - In PCRE, this pattern matches "bab". The first capturing parentheses - match "b", then in the second group, when the back reference \1 fails - to match "b", the second alternative matches "a" and then recurses. In - the recursion, \1 does now match "b" and so the whole match succeeds. - In Perl, the pattern fails to match because inside the recursive call + In PCRE, this pattern matches "bab". The first capturing parentheses + match "b", then in the second group, when the back reference \1 fails + to match "b", the second alternative matches "a" and then recurses. In + the recursion, \1 does now match "b" and so the whole match succeeds. + In Perl, the pattern fails to match because inside the recursive call \1 cannot access the externally set value. SUBPATTERNS AS SUBROUTINES - If the syntax for a recursive subpattern call (either by number or by - name) is used outside the parentheses to which it refers, it operates - like a subroutine in a programming language. The called subpattern may - be defined before or after the reference. A numbered reference can be + If the syntax for a recursive subpattern call (either by number or by + name) is used outside the parentheses to which it refers, it operates + like a subroutine in a programming language. The called subpattern may + be defined before or after the reference. A numbered reference can be absolute or relative, as in these examples: (...(absolute)...)...(?2)... @@ -6784,132 +6826,135 @@ SUBPATTERNS AS SUBROUTINES (sens|respons)e and \1ibility - matches "sense and sensibility" and "response and responsibility", but + matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If instead the pattern (sens|respons)e and (?1)ibility - is used, it does match "sense and responsibility" as well as the other - two strings. Another example is given in the discussion of DEFINE + is used, it does match "sense and responsibility" as well as the other + two strings. Another example is given in the discussion of DEFINE above. - All subroutine calls, whether recursive or not, are always treated as - atomic groups. That is, once a subroutine has matched some of the sub- + All subroutine calls, whether recursive or not, are always treated as + atomic groups. That is, once a subroutine has matched some of the sub- ject string, it is never re-entered, even if it contains untried alter- - natives and there is a subsequent matching failure. Any capturing - parentheses that are set during the subroutine call revert to their + natives and there is a subsequent matching failure. Any capturing + parentheses that are set during the subroutine call revert to their previous values afterwards. - Processing options such as case-independence are fixed when a subpat- - tern is defined, so if it is used as a subroutine, such options cannot + Processing options such as case-independence are fixed when a subpat- + tern is defined, so if it is used as a subroutine, such options cannot be changed for different calls. For example, consider this pattern: (abc)(?i:(?-1)) - It matches "abcabc". It does not match "abcABC" because the change of + It matches "abcabc". It does not match "abcABC" because the change of processing option does not affect the called subpattern. ONIGURUMA SUBROUTINE SYNTAX - For compatibility with Oniguruma, the non-Perl syntax \g followed by a + For compatibility with Oniguruma, the non-Perl syntax \g followed by a name or a number enclosed either in angle brackets or single quotes, is - an alternative syntax for referencing a subpattern as a subroutine, - possibly recursively. Here are two of the examples used above, rewrit- + an alternative syntax for referencing a subpattern as a subroutine, + possibly recursively. Here are two of the examples used above, rewrit- ten using this syntax: (?<pn> \( ( (?>[^()]+) | \g<pn> )* \) ) (sens|respons)e and \g'1'ibility - PCRE supports an extension to Oniguruma: if a number is preceded by a + PCRE supports an extension to Oniguruma: if a number is preceded by a plus or a minus sign it is taken as a relative reference. For example: (abc)(?i:\g<-1>) - Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not - synonymous. The former is a back reference; the latter is a subroutine + Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not + synonymous. The former is a back reference; the latter is a subroutine call. CALLOUTS Perl has a feature whereby using the sequence (?{...}) causes arbitrary - Perl code to be obeyed in the middle of matching a regular expression. + Perl code to be obeyed in the middle of matching a regular expression. This makes it possible, amongst other things, to extract different sub- strings that match the same pair of parentheses when there is a repeti- tion. PCRE provides a similar feature, but of course it cannot obey arbitrary Perl code. The feature is called "callout". The caller of PCRE provides - an external function by putting its entry point in the global variable - pcre_callout (8-bit library) or pcre[16|32]_callout (16-bit or 32-bit - library). By default, this variable contains NULL, which disables all + an external function by putting its entry point in the global variable + pcre_callout (8-bit library) or pcre[16|32]_callout (16-bit or 32-bit + library). By default, this variable contains NULL, which disables all calling out. - Within a regular expression, (?C) indicates the points at which the - external function is to be called. If you want to identify different - callout points, you can put a number less than 256 after the letter C. - The default value is zero. For example, this pattern has two callout + Within a regular expression, (?C) indicates the points at which the + external function is to be called. If you want to identify different + callout points, you can put a number less than 256 after the letter C. + The default value is zero. For example, this pattern has two callout points: (?C1)abc(?C2)def - If the PCRE_AUTO_CALLOUT flag is passed to a compiling function, call- - outs are automatically installed before each item in the pattern. They - are all numbered 255. + If the PCRE_AUTO_CALLOUT flag is passed to a compiling function, call- + outs are automatically installed before each item in the pattern. They + are all numbered 255. If there is a conditional group in the pattern + whose condition is an assertion, an additional callout is inserted just + before the condition. An explicit callout may also be set at this posi- + tion, as in this example: - During matching, when PCRE reaches a callout point, the external func- - tion is called. It is provided with the number of the callout, the - position in the pattern, and, optionally, one item of data originally - supplied by the caller of the matching function. The callout function - may cause matching to proceed, to backtrack, or to fail altogether. A - complete description of the interface to the callout function is given - in the pcrecallout documentation. + (?(?C9)(?=a)abc|def) + Note that this applies only to assertion conditions, not to other types + of condition. -BACKTRACKING CONTROL + During matching, when PCRE reaches a callout point, the external func- + tion is called. It is provided with the number of the callout, the + position in the pattern, and, optionally, one item of data originally + supplied by the caller of the matching function. The callout function + may cause matching to proceed, to backtrack, or to fail altogether. A + complete description of the interface to the callout function is given + in the pcrecallout documentation. - Perl 5.10 introduced a number of "Special Backtracking Control Verbs", - which are described in the Perl documentation as "experimental and sub- - ject to change or removal in a future version of Perl". It goes on to - say: "Their usage in production code should be noted to avoid problems - during upgrades." The same remarks apply to the PCRE features described - in this section. - Since these verbs are specifically related to backtracking, most of - them can be used only when the pattern is to be matched using one of - the traditional matching functions, which use a backtracking algorithm. - With the exception of (*FAIL), which behaves like a failing negative - assertion, they cause an error if encountered by a DFA matching func- - tion. +BACKTRACKING CONTROL - If any of these verbs are used in an assertion or in a subpattern that - is called as a subroutine (whether or not recursively), their effect is - confined to that subpattern; it does not extend to the surrounding pat- - tern, with one exception: the name from a *(MARK), (*PRUNE), or (*THEN) - that is encountered in a successful positive assertion is passed back - when a match succeeds (compare capturing parentheses in assertions). - Note that such subpatterns are processed as anchored at the point where - they are tested. Note also that Perl's treatment of subroutines and - assertions is different in some cases. + Perl 5.10 introduced a number of "Special Backtracking Control Verbs", + which are still described in the Perl documentation as "experimental + and subject to change or removal in a future version of Perl". It goes + on to say: "Their usage in production code should be noted to avoid + problems during upgrades." The same remarks apply to the PCRE features + described in this section. The new verbs make use of what was previously invalid syntax: an open- ing parenthesis followed by an asterisk. They are generally of the form (*VERB) or (*VERB:NAME). Some may take either form, with differing be- - haviour, depending on whether or not an argument is present. A name is - any sequence of characters that does not include a closing parenthesis. - The maximum length of name is 255 in the 8-bit library and 65535 in the - 16-bit and 32-bit library. If the name is empty, that is, if the clos- - ing parenthesis immediately follows the colon, the effect is as if the - colon were not there. Any number of these verbs may occur in a pattern. + haviour, depending on whether or not a name is present. A name is any + sequence of characters that does not include a closing parenthesis. The + maximum length of name is 255 in the 8-bit library and 65535 in the + 16-bit and 32-bit libraries. If the name is empty, that is, if the + closing parenthesis immediately follows the colon, the effect is as if + the colon were not there. Any number of these verbs may occur in a pat- + tern. + + Since these verbs are specifically related to backtracking, most of + them can be used only when the pattern is to be matched using one of + the traditional matching functions, because these use a backtracking + algorithm. With the exception of (*FAIL), which behaves like a failing + negative assertion, the backtracking control verbs cause an error if + encountered by a DFA matching function. + + The behaviour of these verbs in repeated groups, assertions, and in + subpatterns called as subroutines (whether or not recursively) is docu- + mented below. Optimizations that affect backtracking verbs PCRE contains some optimizations that are used to speed up matching by running some checks at the start of each match attempt. For example, it may know the minimum length of matching subject, or that a particular - character must be present. When one of these optimizations suppresses - the running of a match, any included backtracking verbs will not, of + character must be present. When one of these optimizations bypasses the + running of a match, any included backtracking verbs will not, of course, be processed. You can suppress the start-of-match optimizations by setting the PCRE_NO_START_OPTIMIZE option when calling pcre_com- pile() or pcre_exec(), or by starting the pattern with (*NO_START_OPT). @@ -6929,8 +6974,12 @@ BACKTRACKING CONTROL This verb causes the match to end successfully, skipping the remainder of the pattern. However, when it is inside a subpattern that is called as a subroutine, only that subpattern is ended successfully. Matching - then continues at the outer level. If (*ACCEPT) is inside capturing - parentheses, the data so far is captured. For example: + then continues at the outer level. If (*ACCEPT) in triggered in a posi- + tive assertion, the assertion succeeds; in a negative assertion, the + assertion fails. + + If (*ACCEPT) is inside capturing parentheses, the data so far is cap- + tured. For example: A((?:A|B(*ACCEPT)|C)D) @@ -6963,11 +7012,12 @@ BACKTRACKING CONTROL instances of (*MARK) as you like in a pattern, and their names do not have to be unique. - When a match succeeds, the name of the last-encountered (*MARK) on the - matching path is passed back to the caller as described in the section - entitled "Extra data for pcre_exec()" in the pcreapi documentation. - Here is an example of pcretest output, where the /K modifier requests - the retrieval and outputting of (*MARK) data: + When a match succeeds, the name of the last-encountered (*MARK:NAME), + (*PRUNE:NAME), or (*THEN:NAME) on the matching path is passed back to + the caller as described in the section entitled "Extra data for + pcre_exec()" in the pcreapi documentation. Here is an example of + pcretest output, where the /K modifier requests the retrieval and out- + putting of (*MARK) data: re> /X(*MARK:A)Y|X(*MARK:B)Z/K data> XY @@ -6978,41 +7028,42 @@ BACKTRACKING CONTROL MK: B The (*MARK) name is tagged with "MK:" in this output, and in this exam- - ple it indicates which of the two alternatives matched. This is a more - efficient way of obtaining this information than putting each alterna- + ple it indicates which of the two alternatives matched. This is a more + efficient way of obtaining this information than putting each alterna- tive in its own capturing parentheses. - If (*MARK) is encountered in a positive assertion, its name is recorded - and passed back if it is the last-encountered. This does not happen for - negative assertions. + If a verb with a name is encountered in a positive assertion, its name + is recorded and passed back if it is the last-encountered. This does + not happen for negative assertions. - After a partial match or a failed match, the name of the last encoun- - tered (*MARK) in the entire match process is returned. For example: + After a partial match or a failed match, the last encountered name in + the entire match process is returned. For example: re> /X(*MARK:A)Y|X(*MARK:B)Z/K data> XP No match, mark = B - Note that in this unanchored example the mark is retained from the + Note that in this unanchored example the mark is retained from the match attempt that started at the letter "X" in the subject. Subsequent match attempts starting at "P" and then with an empty string do not get as far as the (*MARK) item, but nevertheless do not reset it. - If you are interested in (*MARK) values after failed matches, you - should probably set the PCRE_NO_START_OPTIMIZE option (see above) to + If you are interested in (*MARK) values after failed matches, you + should probably set the PCRE_NO_START_OPTIMIZE option (see above) to ensure that the match is always attempted. Verbs that act after backtracking The following verbs do nothing when they are encountered. Matching con- - tinues with what follows, but if there is no subsequent match, causing - a backtrack to the verb, a failure is forced. That is, backtracking - cannot pass to the left of the verb. However, when one of these verbs - appears inside an atomic group, its effect is confined to that group, - because once the group has been matched, there is never any backtrack- - ing into it. In this situation, backtracking can "jump back" to the - left of the entire atomic group. (Remember also, as stated above, that - this localization also applies in subroutine calls and assertions.) + tinues with what follows, but if there is no subsequent match, causing + a backtrack to the verb, a failure is forced. That is, backtracking + cannot pass to the left of the verb. However, when one of these verbs + appears inside an atomic group or an assertion, its effect is confined + to that group, because once the group has been matched, there is never + any backtracking into it. In this situation, backtracking can "jump + back" to the left of the entire atomic group or assertion. (Remember + also, as stated above, that this localization also applies in subrou- + tine calls.) These verbs differ in exactly what kind of failure occurs when back- tracking reaches them. @@ -7020,19 +7071,25 @@ BACKTRACKING CONTROL (*COMMIT) This verb, which may not be followed by a name, causes the whole match - to fail outright if the rest of the pattern does not match. Even if the - pattern is unanchored, no further attempts to find a match by advancing - the starting point take place. Once (*COMMIT) has been passed, - pcre_exec() is committed to finding a match at the current starting - point, or not at all. For example: + to fail outright if there is a later matching failure that causes back- + tracking to reach it. Even if the pattern is unanchored, no further + attempts to find a match by advancing the starting point take place. If + (*COMMIT) is the only backtracking verb that is encountered, once it + has been passed pcre_exec() is committed to finding a match at the cur- + rent starting point, or not at all. For example: a+(*COMMIT)b - This matches "xxaab" but not "aacaab". It can be thought of as a kind + This matches "xxaab" but not "aacaab". It can be thought of as a kind of dynamic anchor, or "I've started, so I must finish." The name of the - most recently passed (*MARK) in the path is passed back when (*COMMIT) + most recently passed (*MARK) in the path is passed back when (*COMMIT) forces a match failure. + If there is more than one backtracking verb in a pattern, a different + one that follows (*COMMIT) may be triggered first, so merely passing + (*COMMIT) during a match does not always guarantee that a match must be + at this starting point. + Note that (*COMMIT) at the start of a pattern is not the same as an anchor, unless PCRE's start-of-match optimizations are turned off, as shown in this pcretest example: @@ -7052,66 +7109,78 @@ BACKTRACKING CONTROL (*PRUNE) or (*PRUNE:NAME) This verb causes the match to fail at the current starting position in - the subject if the rest of the pattern does not match. If the pattern - is unanchored, the normal "bumpalong" advance to the next starting - character then happens. Backtracking can occur as usual to the left of - (*PRUNE), before it is reached, or when matching to the right of - (*PRUNE), but if there is no match to the right, backtracking cannot - cross (*PRUNE). In simple cases, the use of (*PRUNE) is just an alter- - native to an atomic group or possessive quantifier, but there are some - uses of (*PRUNE) that cannot be expressed in any other way. The behav- - iour of (*PRUNE:NAME) is the same as (*MARK:NAME)(*PRUNE). In an - anchored pattern (*PRUNE) has the same effect as (*COMMIT). + the subject if there is a later matching failure that causes backtrack- + ing to reach it. If the pattern is unanchored, the normal "bumpalong" + advance to the next starting character then happens. Backtracking can + occur as usual to the left of (*PRUNE), before it is reached, or when + matching to the right of (*PRUNE), but if there is no match to the + right, backtracking cannot cross (*PRUNE). In simple cases, the use of + (*PRUNE) is just an alternative to an atomic group or possessive quan- + tifier, but there are some uses of (*PRUNE) that cannot be expressed in + any other way. In an anchored pattern (*PRUNE) has the same effect as + (*COMMIT). + + The behaviour of (*PRUNE:NAME) is the not the same as + (*MARK:NAME)(*PRUNE). It is like (*MARK:NAME) in that the name is + remembered for passing back to the caller. However, (*SKIP:NAME) + searches only for names set with (*MARK). (*SKIP) - This verb, when given without a name, is like (*PRUNE), except that if - the pattern is unanchored, the "bumpalong" advance is not to the next + This verb, when given without a name, is like (*PRUNE), except that if + the pattern is unanchored, the "bumpalong" advance is not to the next character, but to the position in the subject where (*SKIP) was encoun- - tered. (*SKIP) signifies that whatever text was matched leading up to + tered. (*SKIP) signifies that whatever text was matched leading up to it cannot be part of a successful match. Consider: a+(*SKIP)b - If the subject is "aaaac...", after the first match attempt fails - (starting at the first character in the string), the starting point + If the subject is "aaaac...", after the first match attempt fails + (starting at the first character in the string), the starting point skips on to start the next attempt at "c". Note that a possessive quan- - tifer does not have the same effect as this example; although it would - suppress backtracking during the first match attempt, the second - attempt would start at the second character instead of skipping on to + tifer does not have the same effect as this example; although it would + suppress backtracking during the first match attempt, the second + attempt would start at the second character instead of skipping on to "c". (*SKIP:NAME) - When (*SKIP) has an associated name, its behaviour is modified. If the - following pattern fails to match, the previous path through the pattern - is searched for the most recent (*MARK) that has the same name. If one - is found, the "bumpalong" advance is to the subject position that cor- - responds to that (*MARK) instead of to where (*SKIP) was encountered. - If no (*MARK) with a matching name is found, the (*SKIP) is ignored. + When (*SKIP) has an associated name, its behaviour is modified. When it + is triggered, the previous path through the pattern is searched for the + most recent (*MARK) that has the same name. If one is found, the + "bumpalong" advance is to the subject position that corresponds to that + (*MARK) instead of to where (*SKIP) was encountered. If no (*MARK) with + a matching name is found, the (*SKIP) is ignored. + + Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It + ignores names that are set by (*PRUNE:NAME) or (*THEN:NAME). (*THEN) or (*THEN:NAME) - This verb causes a skip to the next innermost alternative if the rest - of the pattern does not match. That is, it cancels pending backtrack- - ing, but only within the current alternative. Its name comes from the - observation that it can be used for a pattern-based if-then-else block: + This verb causes a skip to the next innermost alternative when back- + tracking reaches it. That is, it cancels any further backtracking + within the current alternative. Its name comes from the observation + that it can be used for a pattern-based if-then-else block: ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ... If the COND1 pattern matches, FOO is tried (and possibly further items after the end of the group if FOO succeeds); on failure, the matcher skips to the second alternative and tries COND2, without backtracking - into COND1. The behaviour of (*THEN:NAME) is exactly the same as - (*MARK:NAME)(*THEN). If (*THEN) is not inside an alternation, it acts - like (*PRUNE). + into COND1. If (*THEN) is not inside an alternation, it acts like + (*PRUNE). - Note that a subpattern that does not contain a | character is just a - part of the enclosing alternative; it is not a nested alternation with - only one alternative. The effect of (*THEN) extends beyond such a sub- - pattern to the enclosing alternative. Consider this pattern, where A, - B, etc. are complex pattern fragments that do not contain any | charac- - ters at this level: + The behaviour of (*THEN:NAME) is the not the same as + (*MARK:NAME)(*THEN). It is like (*MARK:NAME) in that the name is + remembered for passing back to the caller. However, (*SKIP:NAME) + searches only for names set with (*MARK). + + A subpattern that does not contain a | character is just a part of the + enclosing alternative; it is not a nested alternation with only one + alternative. The effect of (*THEN) extends beyond such a subpattern to + the enclosing alternative. Consider this pattern, where A, B, etc. are + complex pattern fragments that do not contain any | characters at this + level: A (B(*THEN)C) | D @@ -7127,7 +7196,7 @@ BACKTRACKING CONTROL tern to fail because there are no more alternatives to try. In this case, matching does now backtrack into A. - Note also that a conditional subpattern is not considered as having two + Note that a conditional subpattern is not considered as having two alternatives, because only one is ever used. In other words, the | character in a conditional subpattern has a different meaning. Ignoring white space, consider: @@ -7151,21 +7220,74 @@ BACKTRACKING CONTROL the advance may be more than one character. (*COMMIT) is the strongest, causing the entire match to fail. - If more than one such verb is present in a pattern, the "strongest" one - wins. For example, consider this pattern, where A, B, etc. are complex - pattern fragments: + More than one backtracking verb + + If more than one backtracking verb is present in a pattern, the one + that is backtracked onto first acts. For example, consider this pat- + tern, where A, B, etc. are complex pattern fragments: + + (A(*COMMIT)B(*THEN)C|ABD) + + If A matches but B fails, the backtrack to (*COMMIT) causes the entire + match to fail. However, if A and B match, but C fails, the backtrack to + (*THEN) causes the next alternative (ABD) to be tried. This behaviour + is consistent, but is not always the same as Perl's. It means that if + two or more backtracking verbs appear in succession, all the the last + of them has no effect. Consider this example: + + ...(*COMMIT)(*PRUNE)... + + If there is a matching failure to the right, backtracking onto (*PRUNE) + cases it to be triggered, and its action is taken. There can never be a + backtrack onto (*COMMIT). + + Backtracking verbs in repeated groups + + PCRE differs from Perl in its handling of backtracking verbs in + repeated groups. For example, consider: + + /(a(*COMMIT)b)+ac/ + + If the subject is "abac", Perl matches, but PCRE fails because the + (*COMMIT) in the second repeat of the group acts. + + Backtracking verbs in assertions + + (*FAIL) in an assertion has its normal effect: it forces an immediate + backtrack. + + (*ACCEPT) in a positive assertion causes the assertion to succeed with- + out any further processing. In a negative assertion, (*ACCEPT) causes + the assertion to fail without any further processing. + + The other backtracking verbs are not treated specially if they appear + in an assertion. In particular, (*THEN) skips to the next alternative + in the innermost enclosing group that has alternations, whether or not + this is within the assertion. + + Backtracking verbs in subroutines + + These behaviours occur whether or not the subpattern is called recur- + sively. Perl's treatment of subroutines is different in some cases. - (A(*COMMIT)B(*THEN)C|D) + (*FAIL) in a subpattern called as a subroutine has its normal effect: + it forces an immediate backtrack. - Once A has matched, PCRE is committed to this match, at the current - starting position. If subsequently B matches, but C does not, the nor- - mal (*THEN) action of trying the next alternative (that is, D) does not - happen because (*COMMIT) overrides. + (*ACCEPT) in a subpattern called as a subroutine causes the subroutine + match to succeed without any further processing. Matching then contin- + ues after the subroutine call. + + (*COMMIT), (*SKIP), and (*PRUNE) in a subpattern called as a subroutine + cause the subroutine match to fail. + + (*THEN) skips to the next alternative in the innermost enclosing group + within the subpattern that has alternatives. If there is no such group + within the subpattern, (*THEN) causes the subroutine match to fail. SEE ALSO - pcreapi(3), pcrecallout(3), pcrematching(3), pcresyntax(3), pcre(3), + pcreapi(3), pcrecallout(3), pcrematching(3), pcresyntax(3), pcre(3), pcre16(3), pcre32(3). @@ -7178,18 +7300,18 @@ AUTHOR REVISION - Last updated: 11 November 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 22 March 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRESYNTAX(3) Library Functions Manual PCRESYNTAX(3) -PCRESYNTAX(3) PCRESYNTAX(3) - NAME PCRE - Perl-compatible regular expressions - PCRE REGULAR EXPRESSION SYNTAX SUMMARY The full syntax and semantics of the regular expressions that are sup- @@ -7296,6 +7418,8 @@ PCRE SPECIAL CATEGORY PROPERTIES FOR \p and \P Xan Alphanumeric: union of properties L and N Xps POSIX space: property Z or tab, NL, VT, FF, CR Xsp Perl space: property Z or tab, NL, FF, CR + Xuc Univerally-named character: one that can be + represented by a Universal Character Name Xwd Perl word: property Xan or underscore @@ -7558,18 +7682,18 @@ AUTHOR REVISION - Last updated: 11 November 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 27 February 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREUNICODE(3) Library Functions Manual PCREUNICODE(3) -PCREUNICODE(3) PCREUNICODE(3) - NAME PCRE - Perl-compatible regular expressions - UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) @@ -7632,7 +7756,9 @@ UNICODE PROPERTY SUPPORT fication. Earlier releases of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0 to U+10FFFF, exclud- - ing the surrogate area and the non-characters. + ing the surrogate area. (From release 8.33 the so-called "non-charac- + ter" code points are no longer excluded because Unicode corrigendum #9 + makes it clear that they should not be.) Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16, where they are used in pairs to encode codepoints with values @@ -7641,10 +7767,6 @@ UNICODE PROPERTY SUPPORT other words, the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and UTF-32.) - Also excluded are the "Non-Character" code points, which are U+FDD0 to - U+FDEF and the last two code points in each plane, U+??FFFE and - U+??FFFF. - If an invalid UTF-8 string is passed to PCRE, an error return is given. At compile time, the only additional information is the offset to the first byte of the failing character. The run-time functions pcre_exec() @@ -7676,10 +7798,6 @@ UNICODE PROPERTY SUPPORT surrogate range U+D800 to U+DFFF are independent code points. Values in the surrogate range must be used in pairs in the correct manner. - Excluded are the "Non-Character" code points, which are U+FDD0 to - U+FDEF and the last two code points in each plane, U+??FFFE and - U+??FFFF. - If an invalid UTF-16 string is passed to PCRE, an error return is given. At compile time, the only additional information is the offset to the first data unit of the failing character. The run-time functions @@ -7701,77 +7819,76 @@ UNICODE PROPERTY SUPPORT are passed as patterns and subjects are (by default) checked for valid- ity on entry to the relevant functions. This check allows only values in the range U+0 to U+10FFFF, excluding the surrogate area U+D800 to - U+DFFF, and the "Non-Character" code points, which are U+FDD0 to U+FDEF - and the last two characters in each plane, U+??FFFE and U+??FFFF. + U+DFFF. - If an invalid UTF-32 string is passed to PCRE, an error return is - given. At compile time, the only additional information is the offset + If an invalid UTF-32 string is passed to PCRE, an error return is + given. At compile time, the only additional information is the offset to the first data unit of the failing character. The run-time functions pcre32_exec() and pcre32_dfa_exec() also pass back this information, as - well as a more detailed reason code if the caller has provided memory + well as a more detailed reason code if the caller has provided memory in which to do this. - In some situations, you may already know that your strings are valid, - and therefore want to skip these checks in order to improve perfor- - mance. If you set the PCRE_NO_UTF32_CHECK flag at compile time or at + In some situations, you may already know that your strings are valid, + and therefore want to skip these checks in order to improve perfor- + mance. If you set the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that the pattern or subject it is given (respec- tively) contains only valid UTF-32 sequences. In this case, it does not - diagnose an invalid UTF-32 string. However, if an invalid string is + diagnose an invalid UTF-32 string. However, if an invalid string is passed, the result is undefined. General comments about UTF modes - 1. Codepoints less than 256 can be specified in patterns by either + 1. Codepoints less than 256 can be specified in patterns by either braced or unbraced hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger values have to use braced sequences. - 2. Octal numbers up to \777 are recognized, and in UTF-8 mode they + 2. Octal numbers up to \777 are recognized, and in UTF-8 mode they match two-byte characters for values greater than \177. 3. Repeat quantifiers apply to complete UTF characters, not to individ- ual data units, for example: \x{100}{3}. - 4. The dot metacharacter matches one UTF character instead of a single + 4. The dot metacharacter matches one UTF character instead of a single data unit. - 5. The escape sequence \C can be used to match a single byte in UTF-8 - mode, or a single 16-bit data unit in UTF-16 mode, or a single 32-bit - data unit in UTF-32 mode, but its use can lead to some strange effects - because it breaks up multi-unit characters (see the description of \C - in the pcrepattern documentation). The use of \C is not supported in - the alternative matching function pcre[16|32]_dfa_exec(), nor is it + 5. The escape sequence \C can be used to match a single byte in UTF-8 + mode, or a single 16-bit data unit in UTF-16 mode, or a single 32-bit + data unit in UTF-32 mode, but its use can lead to some strange effects + because it breaks up multi-unit characters (see the description of \C + in the pcrepattern documentation). The use of \C is not supported in + the alternative matching function pcre[16|32]_dfa_exec(), nor is it supported in UTF mode by the JIT optimization of pcre[16|32]_exec(). If - JIT optimization is requested for a UTF pattern that contains \C, it + JIT optimization is requested for a UTF pattern that contains \C, it will not succeed, and so the matching will be carried out by the normal interpretive function. - 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly + 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly test characters of any code value, but, by default, the characters that - PCRE recognizes as digits, spaces, or word characters remain the same - set as in non-UTF mode, all with values less than 256. This remains - true even when PCRE is built to include Unicode property support, + PCRE recognizes as digits, spaces, or word characters remain the same + set as in non-UTF mode, all with values less than 256. This remains + true even when PCRE is built to include Unicode property support, because to do otherwise would slow down PCRE in many common cases. Note - in particular that this applies to \b and \B, because they are defined + in particular that this applies to \b and \B, because they are defined in terms of \w and \W. If you really want to test for a wider sense of, - say, "digit", you can use explicit Unicode property tests such as + say, "digit", you can use explicit Unicode property tests such as \p{Nd}. Alternatively, if you set the PCRE_UCP option, the way that the - character escapes work is changed so that Unicode properties are used + character escapes work is changed so that Unicode properties are used to determine which characters match. There are more details in the sec- tion on generic character types in the pcrepattern documentation. - 7. Similarly, characters that match the POSIX named character classes + 7. Similarly, characters that match the POSIX named character classes are all low-valued characters, unless the PCRE_UCP option is set. - 8. However, the horizontal and vertical white space matching escapes - (\h, \H, \v, and \V) do match all the appropriate Unicode characters, + 8. However, the horizontal and vertical white space matching escapes + (\h, \H, \v, and \V) do match all the appropriate Unicode characters, whether or not PCRE_UCP is set. - 9. Case-insensitive matching applies only to characters whose values - are less than 128, unless PCRE is built with Unicode property support. - A few Unicode characters such as Greek sigma have more than two code- + 9. Case-insensitive matching applies only to characters whose values + are less than 128, unless PCRE is built with Unicode property support. + A few Unicode characters such as Greek sigma have more than two code- points that are case-equivalent. Up to and including PCRE release 8.31, - only one-to-one case mappings were supported, but later releases (with - Unicode property support) do treat as case-equivalent all versions of + only one-to-one case mappings were supported, but later releases (with + Unicode property support) do treat as case-equivalent all versions of characters such as Greek sigma. @@ -7784,18 +7901,18 @@ AUTHOR REVISION - Last updated: 11 November 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 27 February 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREJIT(3) Library Functions Manual PCREJIT(3) -PCREJIT(3) PCREJIT(3) - NAME PCRE - Perl-compatible regular expressions - PCRE JUST-IN-TIME COMPILER SUPPORT Just-in-time compiling is a heavyweight optimization that can greatly @@ -7941,15 +8058,9 @@ UNSUPPORTED OPTIONS AND PATTERN ITEMS BOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, PCRE_PAR- TIAL_HARD, and PCRE_PARTIAL_SOFT. - The unsupported pattern items are: - - \C match a single byte; not supported in UTF-8 mode - (?Cn) callouts - (*PRUNE) ) - (*SKIP) ) backtracking control verbs - (*THEN) ) - - Support for some of these may be added in future. + The only unsupported pattern items are \C (match a single data unit) + when running in a UTF mode, and a callout immediately before an asser- + tion condition in a conditional group. RETURN VALUES FROM JIT EXECUTION @@ -8203,18 +8314,18 @@ AUTHOR REVISION - Last updated: 31 October 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 17 March 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREPARTIAL(3) Library Functions Manual PCREPARTIAL(3) -PCREPARTIAL(3) PCREPARTIAL(3) - NAME PCRE - Perl-compatible regular expressions - PARTIAL MATCHING IN PCRE In normal use of PCRE, if the subject string that is passed to a match- @@ -8273,7 +8384,7 @@ PARTIAL MATCHING USING pcre_exec() OR pcre[16|32]_exec() A partial match occurs during a call to pcre_exec() or pcre[16|32]_exec() when the end of the subject string is reached suc- cessfully, but matching cannot continue because more characters are - needed. However, at least one character in the subject must have been + needed. However, at least one character in the subject must have been inspected. This character need not form part of the final matched string; lookbehind assertions and the \K escape sequence provide ways of inspecting characters before the start of a matched substring. The @@ -8286,18 +8397,22 @@ PARTIAL MATCHING USING pcre_exec() OR pcre[16|32]_exec() match is returned, the first slot is set to the offset of the earliest character that was inspected. For convenience, the second offset points to the end of the subject so that a substring can easily be identified. + If there are at least three slots in the offsets vector, the third slot + is set to the offset of the character where matching started. - For the majority of patterns, the first offset identifies the start of - the partially matched string. However, for patterns that contain look- - behind assertions, or \K, or begin with \b or \B, earlier characters - have been inspected while carrying out the match. For example: + For the majority of patterns, the contents of the first and third slots + will be the same. However, for patterns that contain lookbehind asser- + tions, or begin with \b or \B, characters before the one where matching + started may have been inspected while carrying out the match. For exam- + ple, consider this pattern: /(?<=abc)123/ This pattern matches "123", but only if it is preceded by "abc". If the - subject string is "xyzabc12", the offsets after a partial match are for - the substring "abc12", because all these characters are needed if - another match is tried with extra characters added to the subject. + subject string is "xyzabc12", the first two offsets after a partial + match are for the substring "abc12", because all these characters were + inspected. However, the third offset is set to 6, because that is the + offset where matching began. What happens when a partial match is identified depends on which of the two partial matching options are set. @@ -8523,10 +8638,9 @@ MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre[16|32]_exec() Note: If the pattern contains lookbehind assertions, or \K, or starts with \b or \B, the string that is returned for a partial match includes - characters that precede the partially matched string itself, because - these must be retained when adding on more characters for a subsequent - matching attempt. However, in some cases you may need to retain even - earlier characters, as discussed in the next section. + characters that precede the start of what would be returned for a com- + plete match, because it contains all the characters that were inspected + during the partial match. ISSUES WITH MULTI-SEGMENT MATCHING @@ -8535,23 +8649,44 @@ ISSUES WITH MULTI-SEGMENT MATCHING whichever matching function is used. 1. If the pattern contains a test for the beginning of a line, you need - to pass the PCRE_NOTBOL option when the subject string for any call - does start at the beginning of a line. There is also a PCRE_NOTEOL + to pass the PCRE_NOTBOL option when the subject string for any call + does start at the beginning of a line. There is also a PCRE_NOTEOL option, but in practice when doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which includes the effect of PCRE_NOTEOL. - 2. Lookbehind assertions that have already been obeyed are catered for + 2. Lookbehind assertions that have already been obeyed are catered for in the offsets that are returned for a partial match. However a lookbe- - hind assertion later in the pattern could require even earlier charac- - ters to be inspected. You can handle this case by using the + hind assertion later in the pattern could require even earlier charac- + ters to be inspected. You can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the pcre_fullinfo() or - pcre[16|32]_fullinfo() functions to obtain the length of the largest - lookbehind in the pattern. This length is given in characters, not - bytes. If you always retain at least that many characters before the - partially matched string, all should be well. (Of course, near the + pcre[16|32]_fullinfo() functions to obtain the length of the longest + lookbehind in the pattern. This length is given in characters, not + bytes. If you always retain at least that many characters before the + partially matched string, all should be well. (Of course, near the start of the subject, fewer characters may be present; in that case all characters should be retained.) + From release 8.33, there is a more accurate way of deciding which char- + acters to retain. Instead of subtracting the length of the longest + lookbehind from the earliest inspected character (offsets[0]), the + match start position (offsets[2]) should be used, and the next match + attempt started at the offsets[2] character by setting the startoffset + argument of pcre_exec() or pcre_dfa_exec(). + + For example, if the pattern "(?<=123)abc" is partially matched against + the string "xx123a", the three offset values returned are 2, 6, and 5. + This indicates that the matching process that gave a partial match + started at offset 5, but the characters "123a" were all inspected. The + maximum lookbehind for that pattern is 3, so taking that away from 5 + shows that we need only keep "123a", and the next match attempt can be + started at offset 3 (that is, at "a") when further characters have been + added. When the match start is not the earliest inspected character, + pcretest shows it explicitly: + + re> "(?<=123)abc" + data> xx123a\P\P + Partial match at offset 5: 123a + 3. Because a partial match must always contain at least one character, what might be considered a partial match of an empty string actually gives a "no match" result. For example: @@ -8654,18 +8789,18 @@ AUTHOR REVISION - Last updated: 24 June 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 20 February 2013 + Copyright (c) 1997-2013 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREPRECOMPILE(3) Library Functions Manual PCREPRECOMPILE(3) -PCREPRECOMPILE(3) PCREPRECOMPILE(3) - NAME PCRE - Perl-compatible regular expressions - SAVING AND RE-USING PRECOMPILED PCRE PATTERNS If you are running an application that uses a large number of regular @@ -8792,15 +8927,15 @@ REVISION Last updated: 24 June 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREPERFORM(3) Library Functions Manual PCREPERFORM(3) -PCREPERFORM(3) PCREPERFORM(3) - NAME PCRE - Perl-compatible regular expressions - PCRE PERFORMANCE Two aspects of performance are discussed below: memory usage and pro- @@ -8962,15 +9097,15 @@ REVISION Last updated: 25 August 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCREPOSIX(3) Library Functions Manual PCREPOSIX(3) -PCREPOSIX(3) PCREPOSIX(3) - NAME PCRE - Perl-compatible regular expressions. - SYNOPSIS OF POSIX API #include <pcreposix.h> @@ -9227,15 +9362,15 @@ REVISION Last updated: 09 January 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRECPP(3) Library Functions Manual PCRECPP(3) -PCRECPP(3) PCRECPP(3) - NAME PCRE - Perl-compatible regular expressions. - SYNOPSIS OF C++ WRAPPER #include <pcrecpp.h> @@ -9570,15 +9705,15 @@ REVISION Last updated: 08 January 2012 ------------------------------------------------------------------------------ + + +PCRESAMPLE(3) Library Functions Manual PCRESAMPLE(3) -PCRESAMPLE(3) PCRESAMPLE(3) - NAME PCRE - Perl-compatible regular expressions - PCRE SAMPLE PROGRAM A simple, complete demonstration program, to get you started with using @@ -9658,13 +9793,13 @@ REVISION Last updated: 10 January 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ -PCRELIMITS(3) PCRELIMITS(3) +PCRELIMITS(3) Library Functions Manual PCRELIMITS(3) + NAME PCRE - Perl-compatible regular expressions - SIZE AND OTHER LIMITATIONS There are some size limitations in PCRE but it is hoped that they will @@ -9719,15 +9854,15 @@ REVISION Last updated: 04 May 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ + + +PCRESTACK(3) Library Functions Manual PCRESTACK(3) -PCRESTACK(3) PCRESTACK(3) - NAME PCRE - Perl-compatible regular expressions - PCRE DISCUSSION OF STACK USAGE When you call pcre[16|32]_exec(), it makes use of an internal function @@ -9905,5 +10040,5 @@ REVISION Last updated: 24 June 2012 Copyright (c) 1997-2012 University of Cambridge. ------------------------------------------------------------------------------ - - + + diff --git a/doc/pcregrep.txt b/doc/pcregrep.txt index 0d5a12f..b31eb77 100644 --- a/doc/pcregrep.txt +++ b/doc/pcregrep.txt @@ -1,10 +1,10 @@ -PCREGREP(1) PCREGREP(1) +PCREGREP(1) General Commands Manual PCREGREP(1) + NAME pcregrep - a grep with Perl-compatible regular expressions. - SYNOPSIS pcregrep [options] [long options] [pattern] [path1 path2 ...] diff --git a/doc/pcrepattern.3 b/doc/pcrepattern.3 index c8258e3..caf8724 100644 --- a/doc/pcrepattern.3 +++ b/doc/pcrepattern.3 @@ -1,4 +1,4 @@ -.TH PCREPATTERN 3 "19 March 2013" "PCRE 8.33" +.TH PCREPATTERN 3 "22 March 2013" "PCRE 8.33" .SH NAME PCRE - Perl-compatible regular expressions .SH "PCRE REGULAR EXPRESSION DETAILS" @@ -2688,15 +2688,21 @@ exception of (*FAIL), which behaves like a failing negative assertion, the backtracking control verbs cause an error if encountered by a DFA matching function. .P -If any of these verbs are used in a subpattern that is called as a subroutine -(whether or not recursively), their effect is confined to that subpattern; it -does not extend to the surrounding pattern. (This is the same as for capturing -parentheses in subroutines.) Note that such subpatterns are processed as -anchored at the point where they are tested. Note also that Perl's treatment of -subroutines is different in some cases. -.P -By contrast, if any of these verbs are used in an assertion, they have their -normal action. Again, Perl's treatment differs in some cases. +The behaviour of these verbs in +.\" HTML <a href="#btrepeat"> +.\" </a> +repeated groups, +.\" +.\" HTML <a href="#btassert"> +.\" </a> +assertions, +.\" +and in +.\" HTML <a href="#btsub"> +.\" </a> +subpatterns called as subroutines +.\" +(whether or not recursively) is documented below. . . .\" HTML <a name="nooptimize"></a> @@ -3010,6 +3016,7 @@ it to be triggered, and its action is taken. There can never be a backtrack onto (*COMMIT). . . +.\" HTML <a name="btrepeat"></a> .SS "Backtracking verbs in repeated groups" .rs .sp @@ -3022,6 +3029,44 @@ If the subject is "abac", Perl matches, but PCRE fails because the (*COMMIT) in the second repeat of the group acts. . . +.\" HTML <a name="btassert"></a> +.SS "Backtracking verbs in assertions" +.rs +.sp +(*FAIL) in an assertion has its normal effect: it forces an immediate backtrack. +.P +(*ACCEPT) in a positive assertion causes the assertion to succeed without any +further processing. In a negative assertion, (*ACCEPT) causes the assertion to +fail without any further processing. +.P +The other backtracking verbs are not treated specially if they appear in an +assertion. In particular, (*THEN) skips to the next alternative in the +innermost enclosing group that has alternations, whether or not this is within +the assertion. +. +. +.\" HTML <a name="btsub"></a> +.SS "Backtracking verbs in subroutines" +.rs +.sp +These behaviours occur whether or not the subpattern is called recursively. +Perl's treatment of subroutines is different in some cases. +.P +(*FAIL) in a subpattern called as a subroutine has its normal effect: it forces +an immediate backtrack. +.P +(*ACCEPT) in a subpattern called as a subroutine causes the subroutine match to +succeed without any further processing. Matching then continues after the +subroutine call. +.P +(*COMMIT), (*SKIP), and (*PRUNE) in a subpattern called as a subroutine cause +the subroutine match to fail. +.P +(*THEN) skips to the next alternative in the innermost enclosing group within +the subpattern that has alternatives. If there is no such group within the +subpattern, (*THEN) causes the subroutine match to fail. +. +. .SH "SEE ALSO" .rs .sp @@ -3043,6 +3088,6 @@ Cambridge CB2 3QH, England. .rs .sp .nf -Last updated: 19 March 2013 +Last updated: 22 March 2013 Copyright (c) 1997-2013 University of Cambridge. .fi diff --git a/doc/pcretest.txt b/doc/pcretest.txt index fddc888..82a5df7 100644 --- a/doc/pcretest.txt +++ b/doc/pcretest.txt @@ -1,10 +1,10 @@ -PCRETEST(1) PCRETEST(1) +PCRETEST(1) General Commands Manual PCRETEST(1) + NAME pcretest - a program for testing Perl-compatible regular expressions. - SYNOPSIS pcretest [options] [input file [output file]] @@ -71,20 +71,26 @@ COMMAND LINE OPTIONS -C Output the version number of the PCRE library, and all avail- able information about the optional features that are - included, and then exit. All other options are ignored. + included, and then exit with zero exit code. All other + options are ignored. - -C option Output information about a specific build-time option, then - exit. This functionality is intended for use in scripts such - as RunTest. The following options output the value indicated: + -C option Output information about a specific build-time option, then + exit. This functionality is intended for use in scripts such + as RunTest. The following options output the value and set + the exit code as indicated: ebcdic-nl the code for LF (= NL) in an EBCDIC environment: 0x15 or 0x25 0 if used in an ASCII environment - linksize the internal link size (2, 3, or 4) + exit code is always 0 + linksize the configured internal link size (2, 3, or 4) + exit code is set to the link size newline the default newline setting: CR, LF, CRLF, ANYCRLF, or ANY + exit code is always 0 - The following options output 1 for true or zero for false: + The following options output 1 for true or 0 for false, and + set the exit code to the same value: ebcdic compiled for an EBCDIC environment jit just-in-time support is available @@ -92,58 +98,61 @@ COMMAND LINE OPTIONS pcre32 the 32-bit library was built pcre8 the 8-bit library was built ucp Unicode property support is available - utf UTF-8 and/or UTF-16 and/or UTF-32 support is - available + utf UTF-8 and/or UTF-16 and/or UTF-32 support + is available + + If an unknown option is given, an error message is output; + the exit code is 0. - -d Behave as if each pattern has the /D (debug) modifier; the - internal form and information about the compiled pattern is + -d Behave as if each pattern has the /D (debug) modifier; the + internal form and information about the compiled pattern is output after compilation; -d is equivalent to -b -i. - -dfa Behave as if each data line contains the \D escape sequence; + -dfa Behave as if each data line contains the \D escape sequence; this causes the alternative matching function, - pcre[16|32]_dfa_exec(), to be used instead of the standard + pcre[16|32]_dfa_exec(), to be used instead of the standard pcre[16|32]_exec() function (more detail is given below). -help Output a brief summary these options and then exit. - -i Behave as if each pattern has the /I modifier; information + -i Behave as if each pattern has the /I modifier; information about the compiled pattern is given after compilation. - -M Behave as if each data line contains the \M escape sequence; - this causes PCRE to discover the minimum MATCH_LIMIT and - MATCH_LIMIT_RECURSION settings by calling pcre[16|32]_exec() + -M Behave as if each data line contains the \M escape sequence; + this causes PCRE to discover the minimum MATCH_LIMIT and + MATCH_LIMIT_RECURSION settings by calling pcre[16|32]_exec() repeatedly with different limits. - -m Output the size of each compiled pattern after it has been - compiled. This is equivalent to adding /M to each regular + -m Output the size of each compiled pattern after it has been + compiled. This is equivalent to adding /M to each regular expression. The size is given in bytes for both libraries. - -o osize Set the number of elements in the output vector that is used - when calling pcre[16|32]_exec() or pcre[16|32]_dfa_exec() to - be osize. The default value is 45, which is enough for 14 + -o osize Set the number of elements in the output vector that is used + when calling pcre[16|32]_exec() or pcre[16|32]_dfa_exec() to + be osize. The default value is 45, which is enough for 14 capturing subexpressions for pcre[16|32]_exec() or 22 differ- - ent matches for pcre[16|32]_dfa_exec(). The vector size can - be changed for individual matching calls by including \O in + ent matches for pcre[16|32]_dfa_exec(). The vector size can + be changed for individual matching calls by including \O in the data line (see below). - -p Behave as if each pattern has the /P modifier; the POSIX - wrapper API is used to call PCRE. None of the other options - has any effect when -p is set. This option can be used only + -p Behave as if each pattern has the /P modifier; the POSIX + wrapper API is used to call PCRE. None of the other options + has any effect when -p is set. This option can be used only with the 8-bit library. - -q Do not output the version number of pcretest at the start of + -q Do not output the version number of pcretest at the start of execution. - -S size On Unix-like systems, set the size of the run-time stack to + -S size On Unix-like systems, set the size of the run-time stack to size megabytes. - -s or -s+ Behave as if each pattern has the /S modifier; in other - words, force each pattern to be studied. If -s+ is used, all - the JIT compile options are passed to pcre[16|32]_study(), - causing just-in-time optimization to be set up if it is - available, for both full and partial matching. Specific JIT + -s or -s+ Behave as if each pattern has the /S modifier; in other + words, force each pattern to be studied. If -s+ is used, all + the JIT compile options are passed to pcre[16|32]_study(), + causing just-in-time optimization to be set up if it is + available, for both full and partial matching. Specific JIT compile options can be selected by following -s+ with a digit - in the range 1 to 7, which selects the JIT compile modes as + in the range 1 to 7, which selects the JIT compile modes as follows: 1 normal match only @@ -153,41 +162,41 @@ COMMAND LINE OPTIONS 6 soft and hard partial match 7 all three modes (default) - If -s++ is used instead of -s+ (with or without a following - digit), the text "(JIT)" is added to the first output line + If -s++ is used instead of -s+ (with or without a following + digit), the text "(JIT)" is added to the first output line after a match or no match when JIT-compiled code was actually used. - Note that there are pattern options that can override -s, + Note that there are pattern options that can override -s, either specifying no studying at all, or suppressing JIT com- pilation. - If the /I or /D option is present on a pattern (requesting - output about the compiled pattern), information about the - result of studying is not included when studying is caused - only by -s and neither -i nor -d is present on the command - line. This behaviour means that the output from tests that - are run with and without -s should be identical, except when + If the /I or /D option is present on a pattern (requesting + output about the compiled pattern), information about the + result of studying is not included when studying is caused + only by -s and neither -i nor -d is present on the command + line. This behaviour means that the output from tests that + are run with and without -s should be identical, except when options that output information about the actual running of a match are set. - The -M, -t, and -tm options, which give information about - resources used, are likely to produce different output with - and without -s. Output may also differ if the /C option is + The -M, -t, and -tm options, which give information about + resources used, are likely to produce different output with + and without -s. Output may also differ if the /C option is present on an individual pattern. This uses callouts to trace - the the matching process, and this may be different between - studied and non-studied patterns. If the pattern contains - (*MARK) items there may also be differences, for the same + the the matching process, and this may be different between + studied and non-studied patterns. If the pattern contains + (*MARK) items there may also be differences, for the same reason. The -s command line option can be overridden for spe- - cific patterns that should never be studied (see the /S pat- + cific patterns that should never be studied (see the /S pat- tern modifier below). - -t Run each compile, study, and match many times with a timer, - and output resulting time per compile or match (in millisec- - onds). Do not set -m with -t, because you will then get the - size output a zillion times, and the timing will be dis- - torted. You can control the number of iterations that are - used for timing by following -t with a number (as a separate + -t Run each compile, study, and match many times with a timer, + and output resulting time per compile or match (in millisec- + onds). Do not set -m with -t, because you will then get the + size output a zillion times, and the timing will be dis- + torted. You can control the number of iterations that are + used for timing by following -t with a number (as a separate item on the command line). For example, "-t 1000" would iter- ate 1000 times. The default is to iterate 500000 times. @@ -197,71 +206,71 @@ COMMAND LINE OPTIONS DESCRIPTION - If pcretest is given two filename arguments, it reads from the first + If pcretest is given two filename arguments, it reads from the first and writes to the second. If it is given only one filename argument, it - reads from that file and writes to stdout. Otherwise, it reads from - stdin and writes to stdout, and prompts for each line of input, using + reads from that file and writes to stdout. Otherwise, it reads from + stdin and writes to stdout, and prompts for each line of input, using "re>" to prompt for regular expressions, and "data>" to prompt for data lines. - When pcretest is built, a configuration option can specify that it - should be linked with the libreadline library. When this is done, if + When pcretest is built, a configuration option can specify that it + should be linked with the libreadline library. When this is done, if the input is from a terminal, it is read using the readline() function. - This provides line-editing and history facilities. The output from the + This provides line-editing and history facilities. The output from the -help option states whether or not readline() will be used. The program handles any number of sets of input on a single input file. - Each set starts with a regular expression, and continues with any num- + Each set starts with a regular expression, and continues with any num- ber of data lines to be matched against the pattern. - Each data line is matched separately and independently. If you want to + Each data line is matched separately and independently. If you want to do multi-line matches, you have to use the \n escape sequence (or \r or \r\n, etc., depending on the newline setting) in a single line of input - to encode the newline sequences. There is no limit on the length of - data lines; the input buffer is automatically extended if it is too + to encode the newline sequences. There is no limit on the length of + data lines; the input buffer is automatically extended if it is too small. - An empty line signals the end of the data lines, at which point a new - regular expression is read. The regular expressions are given enclosed + An empty line signals the end of the data lines, at which point a new + regular expression is read. The regular expressions are given enclosed in any non-alphanumeric delimiters other than backslash, for example: /(a|bc)x+yz/ - White space before the initial delimiter is ignored. A regular expres- - sion may be continued over several input lines, in which case the new- - line characters are included within it. It is possible to include the + White space before the initial delimiter is ignored. A regular expres- + sion may be continued over several input lines, in which case the new- + line characters are included within it. It is possible to include the delimiter within the pattern by escaping it, for example /abc\/def/ - If you do so, the escape and the delimiter form part of the pattern, - but since delimiters are always non-alphanumeric, this does not affect - its interpretation. If the terminating delimiter is immediately fol- + If you do so, the escape and the delimiter form part of the pattern, + but since delimiters are always non-alphanumeric, this does not affect + its interpretation. If the terminating delimiter is immediately fol- lowed by a backslash, for example, /abc/\ - then a backslash is added to the end of the pattern. This is done to - provide a way of testing the error condition that arises if a pattern + then a backslash is added to the end of the pattern. This is done to + provide a way of testing the error condition that arises if a pattern finishes with a backslash, because /abc\/ - is interpreted as the first line of a pattern that starts with "abc/", + is interpreted as the first line of a pattern that starts with "abc/", causing pcretest to read the next line as a continuation of the regular expression. PATTERN MODIFIERS - A pattern may be followed by any number of modifiers, which are mostly - single characters, though some of these can be qualified by further - characters. Following Perl usage, these are referred to below as, for - example, "the /i modifier", even though the delimiter of the pattern - need not always be a slash, and no slash is used when writing modi- - fiers. White space may appear between the final pattern delimiter and - the first modifier, and between the modifiers themselves. For refer- - ence, here is a complete list of modifiers. They fall into several + A pattern may be followed by any number of modifiers, which are mostly + single characters, though some of these can be qualified by further + characters. Following Perl usage, these are referred to below as, for + example, "the /i modifier", even though the delimiter of the pattern + need not always be a slash, and no slash is used when writing modi- + fiers. White space may appear between the final pattern delimiter and + the first modifier, and between the modifiers themselves. For refer- + ence, here is a complete list of modifiers. They fall into several groups that are described in detail in the following sections. /8 set UTF mode @@ -310,8 +319,8 @@ PATTERN MODIFIERS Perl-compatible modifiers The /i, /m, /s, and /x modifiers set the PCRE_CASELESS, PCRE_MULTILINE, - PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when - pcre[16|32]_compile() is called. These four modifier letters have the + PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when + pcre[16|32]_compile() is called. These four modifier letters have the same effect as they do in Perl. For example: /caseless/i @@ -319,7 +328,7 @@ PATTERN MODIFIERS Modifiers for other PCRE options - The following table shows additional modifiers for setting PCRE com- + The following table shows additional modifiers for setting PCRE com- pile-time options that do not correspond to anything in Perl: /8 PCRE_UTF8 ) when using the 8-bit @@ -350,138 +359,138 @@ PATTERN MODIFIERS /<bsr_unicode> PCRE_BSR_UNICODE /<JS> PCRE_JAVASCRIPT_COMPAT - The modifiers that are enclosed in angle brackets are literal strings - as shown, including the angle brackets, but the letters within can be - in either case. This example sets multiline matching with CRLF as the + The modifiers that are enclosed in angle brackets are literal strings + as shown, including the angle brackets, but the letters within can be + in either case. This example sets multiline matching with CRLF as the line ending sequence: /^abc/m<CRLF> - As well as turning on the PCRE_UTF8/16/32 option, the /8 modifier - causes all non-printing characters in output strings to be printed + As well as turning on the PCRE_UTF8/16/32 option, the /8 modifier + causes all non-printing characters in output strings to be printed using the \x{hh...} notation. Otherwise, those less than 0x100 are out- put in hex without the curly brackets. - Full details of the PCRE options are given in the pcreapi documenta- + Full details of the PCRE options are given in the pcreapi documenta- tion. Finding all matches in a string - Searching for all possible matches within each subject string can be - requested by the /g or /G modifier. After finding a match, PCRE is + Searching for all possible matches within each subject string can be + requested by the /g or /G modifier. After finding a match, PCRE is called again to search the remainder of the subject string. The differ- ence between /g and /G is that the former uses the startoffset argument - to pcre[16|32]_exec() to start searching at a new point within the - entire string (which is in effect what Perl does), whereas the latter - passes over a shortened substring. This makes a difference to the - matching process if the pattern begins with a lookbehind assertion + to pcre[16|32]_exec() to start searching at a new point within the + entire string (which is in effect what Perl does), whereas the latter + passes over a shortened substring. This makes a difference to the + matching process if the pattern begins with a lookbehind assertion (including \b or \B). - If any call to pcre[16|32]_exec() in a /g or /G sequence matches an - empty string, the next call is done with the PCRE_NOTEMPTY_ATSTART and - PCRE_ANCHORED flags set in order to search for another, non-empty, - match at the same point. If this second match fails, the start offset - is advanced, and the normal match is retried. This imitates the way + If any call to pcre[16|32]_exec() in a /g or /G sequence matches an + empty string, the next call is done with the PCRE_NOTEMPTY_ATSTART and + PCRE_ANCHORED flags set in order to search for another, non-empty, + match at the same point. If this second match fails, the start offset + is advanced, and the normal match is retried. This imitates the way Perl handles such cases when using the /g modifier or the split() func- - tion. Normally, the start offset is advanced by one character, but if - the newline convention recognizes CRLF as a newline, and the current + tion. Normally, the start offset is advanced by one character, but if + the newline convention recognizes CRLF as a newline, and the current character is CR followed by LF, an advance of two is used. Other modifiers There are yet more modifiers for controlling the way pcretest operates. - The /+ modifier requests that as well as outputting the substring that - matched the entire pattern, pcretest should in addition output the - remainder of the subject string. This is useful for tests where the - subject contains multiple copies of the same substring. If the + modi- - fier appears twice, the same action is taken for captured substrings. - In each case the remainder is output on the following line with a plus - character following the capture number. Note that this modifier must - not immediately follow the /S modifier because /S+ and /S++ have other + The /+ modifier requests that as well as outputting the substring that + matched the entire pattern, pcretest should in addition output the + remainder of the subject string. This is useful for tests where the + subject contains multiple copies of the same substring. If the + modi- + fier appears twice, the same action is taken for captured substrings. + In each case the remainder is output on the following line with a plus + character following the capture number. Note that this modifier must + not immediately follow the /S modifier because /S+ and /S++ have other meanings. - The /= modifier requests that the values of all potential captured - parentheses be output after a match. By default, only those up to the + The /= modifier requests that the values of all potential captured + parentheses be output after a match. By default, only those up to the highest one actually used in the match are output (corresponding to the return code from pcre[16|32]_exec()). Values in the offsets vector cor- - responding to higher numbers should be set to -1, and these are output - as "<unset>". This modifier gives a way of checking that this is hap- + responding to higher numbers should be set to -1, and these are output + as "<unset>". This modifier gives a way of checking that this is hap- pening. - The /B modifier is a debugging feature. It requests that pcretest out- - put a representation of the compiled code after compilation. Normally - this information contains length and offset values; however, if /Z is - also present, this data is replaced by spaces. This is a special fea- - ture for use in the automatic test scripts; it ensures that the same + The /B modifier is a debugging feature. It requests that pcretest out- + put a representation of the compiled code after compilation. Normally + this information contains length and offset values; however, if /Z is + also present, this data is replaced by spaces. This is a special fea- + ture for use in the automatic test scripts; it ensures that the same output is generated for different internal link sizes. - The /D modifier is a PCRE debugging feature, and is equivalent to /BI, + The /D modifier is a PCRE debugging feature, and is equivalent to /BI, that is, both the /B and the /I modifiers. - The /F modifier causes pcretest to flip the byte order of the 2-byte + The /F modifier causes pcretest to flip the byte order of the 2-byte and 4-byte fields in the compiled pattern. This facility is for testing - the feature in PCRE that allows it to execute patterns that were com- + the feature in PCRE that allows it to execute patterns that were com- piled on a host with a different endianness. This feature is not avail- - able when the POSIX interface to PCRE is being used, that is, when the + able when the POSIX interface to PCRE is being used, that is, when the /P pattern modifier is specified. See also the section about saving and reloading compiled patterns below. - The /I modifier requests that pcretest output information about the - compiled pattern (whether it is anchored, has a fixed first character, - and so on). It does this by calling pcre[16|32]_fullinfo() after com- - piling a pattern. If the pattern is studied, the results of that are + The /I modifier requests that pcretest output information about the + compiled pattern (whether it is anchored, has a fixed first character, + and so on). It does this by calling pcre[16|32]_fullinfo() after com- + piling a pattern. If the pattern is studied, the results of that are also output. - The /K modifier requests pcretest to show names from backtracking con- - trol verbs that are returned from calls to pcre[16|32]_exec(). It - causes pcretest to create a pcre[16|32]_extra block if one has not - already been created by a call to pcre[16|32]_study(), and to set the - PCRE_EXTRA_MARK flag and the mark field within it, every time that - pcre[16|32]_exec() is called. If the variable that the mark field - points to is non-NULL for a match, non-match, or partial match, - pcretest prints the string to which it points. For a match, this is - shown on a line by itself, tagged with "MK:". For a non-match it is + The /K modifier requests pcretest to show names from backtracking con- + trol verbs that are returned from calls to pcre[16|32]_exec(). It + causes pcretest to create a pcre[16|32]_extra block if one has not + already been created by a call to pcre[16|32]_study(), and to set the + PCRE_EXTRA_MARK flag and the mark field within it, every time that + pcre[16|32]_exec() is called. If the variable that the mark field + points to is non-NULL for a match, non-match, or partial match, + pcretest prints the string to which it points. For a match, this is + shown on a line by itself, tagged with "MK:". For a non-match it is added to the message. - The /L modifier must be followed directly by the name of a locale, for + The /L modifier must be followed directly by the name of a locale, for example, /pattern/Lfr_FR For this reason, it must be the last modifier. The given locale is set, - pcre[16|32]_maketables() is called to build a set of character tables - for the locale, and this is then passed to pcre[16|32]_compile() when - compiling the regular expression. Without an /L (or /T) modifier, NULL - is passed as the tables pointer; that is, /L applies only to the + pcre[16|32]_maketables() is called to build a set of character tables + for the locale, and this is then passed to pcre[16|32]_compile() when + compiling the regular expression. Without an /L (or /T) modifier, NULL + is passed as the tables pointer; that is, /L applies only to the expression on which it appears. - The /M modifier causes the size in bytes of the memory block used to - hold the compiled pattern to be output. This does not include the size - of the pcre[16|32] block; it is just the actual compiled data. If the + The /M modifier causes the size in bytes of the memory block used to + hold the compiled pattern to be output. This does not include the size + of the pcre[16|32] block; it is just the actual compiled data. If the pattern is successfully studied with the PCRE_STUDY_JIT_COMPILE option, the size of the JIT compiled code is also output. - The /S modifier causes pcre[16|32]_study() to be called after the - expression has been compiled, and the results used when the expression + The /S modifier causes pcre[16|32]_study() to be called after the + expression has been compiled, and the results used when the expression is matched. There are a number of qualifying characters that may follow /S. They may appear in any order. - If S is followed by an exclamation mark, pcre[16|32]_study() is called - with the PCRE_STUDY_EXTRA_NEEDED option, causing it always to return a + If S is followed by an exclamation mark, pcre[16|32]_study() is called + with the PCRE_STUDY_EXTRA_NEEDED option, causing it always to return a pcre_extra block, even when studying discovers no useful information. If /S is followed by a second S character, it suppresses studying, even - if it was requested externally by the -s command line option. This - makes it possible to specify that certain patterns are always studied, + if it was requested externally by the -s command line option. This + makes it possible to specify that certain patterns are always studied, and others are never studied, independently of -s. This feature is used in the test files in a few cases where the output is different when the pattern is studied. - If the /S modifier is followed by a + character, the call to - pcre[16|32]_study() is made with all the JIT study options, requesting - just-in-time optimization support if it is available, for both normal - and partial matching. If you want to restrict the JIT compiling modes, + If the /S modifier is followed by a + character, the call to + pcre[16|32]_study() is made with all the JIT study options, requesting + just-in-time optimization support if it is available, for both normal + and partial matching. If you want to restrict the JIT compiling modes, you can follow /S+ with a digit in the range 1 to 7: 1 normal match only @@ -492,40 +501,40 @@ PATTERN MODIFIERS 7 all three modes (default) If /S++ is used instead of /S+ (with or without a following digit), the - text "(JIT)" is added to the first output line after a match or no + text "(JIT)" is added to the first output line after a match or no match when JIT-compiled code was actually used. - Note that there is also an independent /+ modifier; it must not be + Note that there is also an independent /+ modifier; it must not be given immediately after /S or /S+ because this will be misinterpreted. If JIT studying is successful, the compiled JIT code will automatically - be used when pcre[16|32]_exec() is run, except when incompatible run- - time options are specified. For more details, see the pcrejit documen- - tation. See also the \J escape sequence below for a way of setting the + be used when pcre[16|32]_exec() is run, except when incompatible run- + time options are specified. For more details, see the pcrejit documen- + tation. See also the \J escape sequence below for a way of setting the size of the JIT stack. - Finally, if /S is followed by a minus character, JIT compilation is - suppressed, even if it was requested externally by the -s command line - option. This makes it possible to specify that JIT is never to be used + Finally, if /S is followed by a minus character, JIT compilation is + suppressed, even if it was requested externally by the -s command line + option. This makes it possible to specify that JIT is never to be used for certain patterns. - The /T modifier must be followed by a single digit. It causes a spe- + The /T modifier must be followed by a single digit. It causes a spe- cific set of built-in character tables to be passed to pcre[16|32]_com- - pile(). It is used in the standard PCRE tests to check behaviour with + pile(). It is used in the standard PCRE tests to check behaviour with different character tables. The digit specifies the tables as follows: 0 the default ASCII tables, as distributed in pcre_chartables.c.dist 1 a set of tables defining ISO 8859 characters - In table 1, some characters whose codes are greater than 128 are iden- + In table 1, some characters whose codes are greater than 128 are iden- tified as letters, digits, spaces, etc. Using the POSIX wrapper API - The /P modifier causes pcretest to call PCRE via the POSIX wrapper API - rather than its native API. This supports only the 8-bit library. When - /P is set, the following modifiers set options for the regcomp() func- + The /P modifier causes pcretest to call PCRE via the POSIX wrapper API + rather than its native API. This supports only the 8-bit library. When + /P is set, the following modifiers set options for the regcomp() func- tion: /i REG_ICASE @@ -536,17 +545,17 @@ PATTERN MODIFIERS /W REG_UCP ) the POSIX standard /8 REG_UTF8 ) - The /+ modifier works as described above. All other modifiers are + The /+ modifier works as described above. All other modifiers are ignored. DATA LINES - Before each data line is passed to pcre[16|32]_exec(), leading and - trailing white space is removed, and it is then scanned for \ escapes. - Some of these are pretty esoteric features, intended for checking out - some of the more complicated features of PCRE. If you are just testing - "ordinary" regular expressions, you probably don't need any of these. + Before each data line is passed to pcre[16|32]_exec(), leading and + trailing white space is removed, and it is then scanned for \ escapes. + Some of these are pretty esoteric features, intended for checking out + some of the more complicated features of PCRE. If you are just testing + "ordinary" regular expressions, you probably don't need any of these. The following escapes are recognized: \a alarm (BEL, \x07) @@ -606,7 +615,7 @@ DATA LINES (any number of digits) \R pass the PCRE_DFA_RESTART option to pcre[16|32]_dfa_exec() \S output details of memory get/free calls during matching - \Y pass the PCRE_NO_START_OPTIMIZE option to + \Y pass the PCRE_NO_START_OPTIMIZE option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() \Z pass the PCRE_NOTEOL option to pcre[16|32]_exec() @@ -615,7 +624,7 @@ DATA LINES pcre[16|32]_exec() or pcre[16|32]_dfa_exec() \>dd start the match at offset dd (optional "-"; then any number of digits); this sets the startoffset - argument for pcre[16|32]_exec() or + argument for pcre[16|32]_exec() or pcre[16|32]_dfa_exec() \<cr> pass the PCRE_NEWLINE_CR option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() @@ -628,102 +637,102 @@ DATA LINES \<any> pass the PCRE_NEWLINE_ANY option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() - The use of \x{hh...} is not dependent on the use of the /8 modifier on - the pattern. It is recognized always. There may be any number of hexa- - decimal digits inside the braces; invalid values provoke error mes- + The use of \x{hh...} is not dependent on the use of the /8 modifier on + the pattern. It is recognized always. There may be any number of hexa- + decimal digits inside the braces; invalid values provoke error mes- sages. - Note that \xhh specifies one byte rather than one character in UTF-8 - mode; this makes it possible to construct invalid UTF-8 sequences for - testing purposes. On the other hand, \x{hh} is interpreted as a UTF-8 - character in UTF-8 mode, generating more than one byte if the value is - greater than 127. When testing the 8-bit library not in UTF-8 mode, + Note that \xhh specifies one byte rather than one character in UTF-8 + mode; this makes it possible to construct invalid UTF-8 sequences for + testing purposes. On the other hand, \x{hh} is interpreted as a UTF-8 + character in UTF-8 mode, generating more than one byte if the value is + greater than 127. When testing the 8-bit library not in UTF-8 mode, \x{hh} generates one byte for values less than 256, and causes an error for greater values. In UTF-16 mode, all 4-digit \x{hhhh} values are accepted. This makes it possible to construct invalid UTF-16 sequences for testing purposes. - In UTF-32 mode, all 4- to 8-digit \x{...} values are accepted. This - makes it possible to construct invalid UTF-32 sequences for testing + In UTF-32 mode, all 4- to 8-digit \x{...} values are accepted. This + makes it possible to construct invalid UTF-32 sequences for testing purposes. - The escapes that specify line ending sequences are literal strings, + The escapes that specify line ending sequences are literal strings, exactly as shown. No more than one newline setting should be present in any data line. - A backslash followed by anything else just escapes the anything else. - If the very last character is a backslash, it is ignored. This gives a - way of passing an empty line as data, since a real empty line termi- + A backslash followed by anything else just escapes the anything else. + If the very last character is a backslash, it is ignored. This gives a + way of passing an empty line as data, since a real empty line termi- nates the data input. - The \J escape provides a way of setting the maximum stack size that is - used by the just-in-time optimization code. It is ignored if JIT opti- - mization is not being used. Providing a stack that is larger than the + The \J escape provides a way of setting the maximum stack size that is + used by the just-in-time optimization code. It is ignored if JIT opti- + mization is not being used. Providing a stack that is larger than the default 32K is necessary only for very complicated patterns. If \M is present, pcretest calls pcre[16|32]_exec() several times, with different values in the match_limit and match_limit_recursion fields of - the pcre[16|32]_extra data structure, until it finds the minimum num- + the pcre[16|32]_extra data structure, until it finds the minimum num- bers for each parameter that allow pcre[16|32]_exec() to complete with- - out error. Because this is testing a specific feature of the normal + out error. Because this is testing a specific feature of the normal interpretive pcre[16|32]_exec() execution, the use of any JIT optimiza- - tion that might have been set up by the /S+ qualifier of -s+ option is + tion that might have been set up by the /S+ qualifier of -s+ option is disabled. - The match_limit number is a measure of the amount of backtracking that - takes place, and checking it out can be instructive. For most simple - matches, the number is quite small, but for patterns with very large - numbers of matching possibilities, it can become large very quickly - with increasing length of subject string. The match_limit_recursion - number is a measure of how much stack (or, if PCRE is compiled with - NO_RECURSE, how much heap) memory is needed to complete the match + The match_limit number is a measure of the amount of backtracking that + takes place, and checking it out can be instructive. For most simple + matches, the number is quite small, but for patterns with very large + numbers of matching possibilities, it can become large very quickly + with increasing length of subject string. The match_limit_recursion + number is a measure of how much stack (or, if PCRE is compiled with + NO_RECURSE, how much heap) memory is needed to complete the match attempt. - When \O is used, the value specified may be higher or lower than the + When \O is used, the value specified may be higher or lower than the size set by the -O command line option (or defaulted to 45); \O applies - only to the call of pcre[16|32]_exec() for the line in which it + only to the call of pcre[16|32]_exec() for the line in which it appears. - If the /P modifier was present on the pattern, causing the POSIX wrap- - per API to be used, the only option-setting sequences that have any - effect are \B, \N, and \Z, causing REG_NOTBOL, REG_NOTEMPTY, and + If the /P modifier was present on the pattern, causing the POSIX wrap- + per API to be used, the only option-setting sequences that have any + effect are \B, \N, and \Z, causing REG_NOTBOL, REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to regexec(). THE ALTERNATIVE MATCHING FUNCTION - By default, pcretest uses the standard PCRE matching function, - pcre[16|32]_exec() to match each data line. PCRE also supports an - alternative matching function, pcre[16|32]_dfa_test(), which operates - in a different way, and has some restrictions. The differences between + By default, pcretest uses the standard PCRE matching function, + pcre[16|32]_exec() to match each data line. PCRE also supports an + alternative matching function, pcre[16|32]_dfa_test(), which operates + in a different way, and has some restrictions. The differences between the two functions are described in the pcrematching documentation. - If a data line contains the \D escape sequence, or if the command line - contains the -dfa option, the alternative matching function is used. + If a data line contains the \D escape sequence, or if the command line + contains the -dfa option, the alternative matching function is used. This function finds all possible matches at a given point. If, however, - the \F escape sequence is present in the data line, it stops after the + the \F escape sequence is present in the data line, it stops after the first match is found. This is always the shortest possible match. DEFAULT OUTPUT FROM PCRETEST - This section describes the output when the normal matching function, + This section describes the output when the normal matching function, pcre[16|32]_exec(), is being used. When a match succeeds, pcretest outputs the list of captured substrings - that pcre[16|32]_exec() returns, starting with number 0 for the string - that matched the whole pattern. Otherwise, it outputs "No match" when - the return is PCRE_ERROR_NOMATCH, and "Partial match:" followed by the - partially matching substring when pcre[16|32]_exec() returns - PCRE_ERROR_PARTIAL. (Note that this is the entire substring that was - inspected during the partial match; it may include characters before - the actual match start if a lookbehind assertion, \K, \b, or \B was - involved.) For any other return, pcretest outputs the PCRE negative - error number and a short descriptive phrase. If the error is a failed - UTF string check, the offset of the start of the failing character and - the reason code are also output, provided that the size of the output - vector is at least two. Here is an example of an interactive pcretest + that pcre[16|32]_exec() returns, starting with number 0 for the string + that matched the whole pattern. Otherwise, it outputs "No match" when + the return is PCRE_ERROR_NOMATCH, and "Partial match:" followed by the + partially matching substring when pcre[16|32]_exec() returns + PCRE_ERROR_PARTIAL. (Note that this is the entire substring that was + inspected during the partial match; it may include characters before + the actual match start if a lookbehind assertion, \K, \b, or \B was + involved.) For any other return, pcretest outputs the PCRE negative + error number and a short descriptive phrase. If the error is a failed + UTF string check, the offset of the start of the failing character and + the reason code are also output, provided that the size of the output + vector is at least two. Here is an example of an interactive pcretest run. $ pcretest @@ -737,10 +746,10 @@ DEFAULT OUTPUT FROM PCRETEST No match Unset capturing substrings that are not followed by one that is set are - not returned by pcre[16|32]_exec(), and are not shown by pcretest. In + not returned by pcre[16|32]_exec(), and are not shown by pcretest. In the following example, there are two capturing substrings, but when the - first data line is matched, the second, unset substring is not shown. - An "internal" unset substring is shown as "<unset>", as for the second + first data line is matched, the second, unset substring is not shown. + An "internal" unset substring is shown as "<unset>", as for the second data line. re> /(a)|(b)/ @@ -752,11 +761,11 @@ DEFAULT OUTPUT FROM PCRETEST 1: <unset> 2: b - If the strings contain any non-printing characters, they are output as - \xhh escapes if the value is less than 256 and UTF mode is not set. + If the strings contain any non-printing characters, they are output as + \xhh escapes if the value is less than 256 and UTF mode is not set. Otherwise they are output as \x{hh...} escapes. See below for the defi- - nition of non-printing characters. If the pattern has the /+ modifier, - the output for substring 0 is followed by the the rest of the subject + nition of non-printing characters. If the pattern has the /+ modifier, + the output for substring 0 is followed by the the rest of the subject string, identified by "0+" like this: re> /cat/+ @@ -764,7 +773,7 @@ DEFAULT OUTPUT FROM PCRETEST 0: cat 0+ aract - If the pattern has the /g or /G modifier, the results of successive + If the pattern has the /g or /G modifier, the results of successive matching attempts are output in sequence, like this: re> /\Bi(\w\w)/g @@ -776,32 +785,32 @@ DEFAULT OUTPUT FROM PCRETEST 0: ipp 1: pp - "No match" is output only if the first match attempt fails. Here is an - example of a failure message (the offset 4 that is specified by \>4 is + "No match" is output only if the first match attempt fails. Here is an + example of a failure message (the offset 4 that is specified by \>4 is past the end of the subject string): re> /xyz/ data> xyz\>4 Error -24 (bad offset value) - If any of the sequences \C, \G, or \L are present in a data line that - is successfully matched, the substrings extracted by the convenience + If any of the sequences \C, \G, or \L are present in a data line that + is successfully matched, the substrings extracted by the convenience functions are output with C, G, or L after the string number instead of a colon. This is in addition to the normal full list. The string length - (that is, the return from the extraction function) is given in paren- + (that is, the return from the extraction function) is given in paren- theses after each string for \C and \G. Note that whereas patterns can be continued over several lines (a plain ">" prompt is used for continuations), data lines may not. However new- - lines can be included in data by means of the \n escape (or \r, \r\n, + lines can be included in data by means of the \n escape (or \r, \r\n, etc., depending on the newline sequence setting). OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION When the alternative matching function, pcre[16|32]_dfa_exec(), is used - (by means of the \D escape sequence or the -dfa command line option), - the output consists of a list of all the matches that start at the + (by means of the \D escape sequence or the -dfa command line option), + the output consists of a list of all the matches that start at the first point in the subject where there is at least one match. For exam- ple: @@ -811,11 +820,11 @@ OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION 1: tang 2: tan - (Using the normal matching function on this data finds only "tang".) - The longest matching string is always given first (and numbered zero). + (Using the normal matching function on this data finds only "tang".) + The longest matching string is always given first (and numbered zero). After a PCRE_ERROR_PARTIAL return, the output is "Partial match:", fol- - lowed by the partially matching substring. (Note that this is the - entire substring that was inspected during the partial match; it may + lowed by the partially matching substring. (Note that this is the + entire substring that was inspected during the partial match; it may include characters before the actual match start if a lookbehind asser- tion, \K, \b, or \B was involved.) @@ -831,16 +840,16 @@ OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION 1: tan 0: tan - Since the matching function does not support substring capture, the - escape sequences that are concerned with captured substrings are not + Since the matching function does not support substring capture, the + escape sequences that are concerned with captured substrings are not relevant. RESTARTING AFTER A PARTIAL MATCH When the alternative matching function has given the PCRE_ERROR_PARTIAL - return, indicating that the subject partially matched the pattern, you - can restart the match with additional subject data by means of the \R + return, indicating that the subject partially matched the pattern, you + can restart the match with additional subject data by means of the \R escape sequence. For example: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ @@ -849,30 +858,30 @@ RESTARTING AFTER A PARTIAL MATCH data> n05\R\D 0: n05 - For further information about partial matching, see the pcrepartial + For further information about partial matching, see the pcrepartial documentation. CALLOUTS - If the pattern contains any callout requests, pcretest's callout func- - tion is called during matching. This works with both matching func- + If the pattern contains any callout requests, pcretest's callout func- + tion is called during matching. This works with both matching func- tions. By default, the called function displays the callout number, the - start and current positions in the text at the callout time, and the + start and current positions in the text at the callout time, and the next pattern item to be tested. For example: --->pqrabcdef 0 ^ ^ \d - This output indicates that callout number 0 occurred for a match - attempt starting at the fourth character of the subject string, when + This output indicates that callout number 0 occurred for a match + attempt starting at the fourth character of the subject string, when the pointer was at the seventh character of the data, and when the next - pattern item was \d. Just one circumflex is output if the start and + pattern item was \d. Just one circumflex is output if the start and current positions are the same. Callouts numbered 255 are assumed to be automatic callouts, inserted as - a result of the /C pattern modifier. In this case, instead of showing - the callout number, the offset in the pattern, preceded by a plus, is + a result of the /C pattern modifier. In this case, instead of showing + the callout number, the offset in the pattern, preceded by a plus, is output. For example: re> /\d?[A-E]\*/C @@ -885,7 +894,7 @@ CALLOUTS 0: E* If a pattern contains (*MARK) items, an additional line is output when- - ever a change of latest mark is passed to the callout function. For + ever a change of latest mark is passed to the callout function. For example: re> /a(*MARK:X)bc/C @@ -899,59 +908,59 @@ CALLOUTS +12 ^ ^ 0: abc - The mark changes between matching "a" and "b", but stays the same for - the rest of the match, so nothing more is output. If, as a result of - backtracking, the mark reverts to being unset, the text "<unset>" is + The mark changes between matching "a" and "b", but stays the same for + the rest of the match, so nothing more is output. If, as a result of + backtracking, the mark reverts to being unset, the text "<unset>" is output. - The callout function in pcretest returns zero (carry on matching) by - default, but you can use a \C item in a data line (as described above) + The callout function in pcretest returns zero (carry on matching) by + default, but you can use a \C item in a data line (as described above) to change this and other parameters of the callout. - Inserting callouts can be helpful when using pcretest to check compli- - cated regular expressions. For further information about callouts, see + Inserting callouts can be helpful when using pcretest to check compli- + cated regular expressions. For further information about callouts, see the pcrecallout documentation. NON-PRINTING CHARACTERS - When pcretest is outputting text in the compiled version of a pattern, - bytes other than 32-126 are always treated as non-printing characters + When pcretest is outputting text in the compiled version of a pattern, + bytes other than 32-126 are always treated as non-printing characters are are therefore shown as hex escapes. - When pcretest is outputting text that is a matched part of a subject - string, it behaves in the same way, unless a different locale has been - set for the pattern (using the /L modifier). In this case, the + When pcretest is outputting text that is a matched part of a subject + string, it behaves in the same way, unless a different locale has been + set for the pattern (using the /L modifier). In this case, the isprint() function to distinguish printing and non-printing characters. SAVING AND RELOADING COMPILED PATTERNS - The facilities described in this section are not available when the - POSIX interface to PCRE is being used, that is, when the /P pattern + The facilities described in this section are not available when the + POSIX interface to PCRE is being used, that is, when the /P pattern modifier is specified. When the POSIX interface is not in use, you can cause pcretest to write - a compiled pattern to a file, by following the modifiers with > and a + a compiled pattern to a file, by following the modifiers with > and a file name. For example: /pattern/im >/some/file - See the pcreprecompile documentation for a discussion about saving and - re-using compiled patterns. Note that if the pattern was successfully + See the pcreprecompile documentation for a discussion about saving and + re-using compiled patterns. Note that if the pattern was successfully studied with JIT optimization, the JIT data cannot be saved. - The data that is written is binary. The first eight bytes are the - length of the compiled pattern data followed by the length of the - optional study data, each written as four bytes in big-endian order - (most significant byte first). If there is no study data (either the + The data that is written is binary. The first eight bytes are the + length of the compiled pattern data followed by the length of the + optional study data, each written as four bytes in big-endian order + (most significant byte first). If there is no study data (either the pattern was not studied, or studying did not return any data), the sec- - ond length is zero. The lengths are followed by an exact copy of the - compiled pattern. If there is additional study data, this (excluding - any JIT data) follows immediately after the compiled pattern. After + ond length is zero. The lengths are followed by an exact copy of the + compiled pattern. If there is additional study data, this (excluding + any JIT data) follows immediately after the compiled pattern. After writing the file, pcretest expects to read a new pattern. - A saved pattern can be reloaded into pcretest by specifying < and a + A saved pattern can be reloaded into pcretest by specifying < and a file name instead of a pattern. The name of the file must not contain a < character, as otherwise pcretest will interpret the line as a pattern delimited by < characters. For example: @@ -960,42 +969,42 @@ SAVING AND RELOADING COMPILED PATTERNS Compiled pattern loaded from /some/file No study data - If the pattern was previously studied with the JIT optimization, the - JIT information cannot be saved and restored, and so is lost. When the - pattern has been loaded, pcretest proceeds to read data lines in the + If the pattern was previously studied with the JIT optimization, the + JIT information cannot be saved and restored, and so is lost. When the + pattern has been loaded, pcretest proceeds to read data lines in the usual way. - You can copy a file written by pcretest to a different host and reload - it there, even if the new host has opposite endianness to the one on - which the pattern was compiled. For example, you can compile on an i86 - machine and run on a SPARC machine. When a pattern is reloaded on a + You can copy a file written by pcretest to a different host and reload + it there, even if the new host has opposite endianness to the one on + which the pattern was compiled. For example, you can compile on an i86 + machine and run on a SPARC machine. When a pattern is reloaded on a host with different endianness, the confirmation message is changed to: Compiled pattern (byte-inverted) loaded from /some/file The test suite contains some saved pre-compiled patterns with different - endianness. These are reloaded using "<!" instead of just "<". This + endianness. These are reloaded using "<!" instead of just "<". This suppresses the "(byte-inverted)" text so that the output is the same on - all hosts. It also forces debugging output once the pattern has been + all hosts. It also forces debugging output once the pattern has been reloaded. - File names for saving and reloading can be absolute or relative, but - note that the shell facility of expanding a file name that starts with + File names for saving and reloading can be absolute or relative, but + note that the shell facility of expanding a file name that starts with a tilde (~) is not available. - The ability to save and reload files in pcretest is intended for test- - ing and experimentation. It is not intended for production use because - only a single pattern can be written to a file. Furthermore, there is - no facility for supplying custom character tables for use with a - reloaded pattern. If the original pattern was compiled with custom - tables, an attempt to match a subject string using a reloaded pattern - is likely to cause pcretest to crash. Finally, if you attempt to load + The ability to save and reload files in pcretest is intended for test- + ing and experimentation. It is not intended for production use because + only a single pattern can be written to a file. Furthermore, there is + no facility for supplying custom character tables for use with a + reloaded pattern. If the original pattern was compiled with custom + tables, an attempt to match a subject string using a reloaded pattern + is likely to cause pcretest to crash. Finally, if you attempt to load a file that is not in the correct format, the result is undefined. SEE ALSO - pcre(3), pcre16(3), pcre32(3), pcreapi(3), pcrecallout(3), pcrejit, + pcre(3), pcre16(3), pcre32(3), pcreapi(3), pcrecallout(3), pcrejit, pcrematching(3), pcrepartial(d), pcrepattern(3), pcreprecompile(3). @@ -1008,5 +1017,5 @@ AUTHOR REVISION - Last updated: 10 September 2012 - Copyright (c) 1997-2012 University of Cambridge. + Last updated: 22 February 2013 + Copyright (c) 1997-2013 University of Cambridge. diff --git a/pcre_exec.c b/pcre_exec.c index bcc2c63..79c14db 100644 --- a/pcre_exec.c +++ b/pcre_exec.c @@ -1863,11 +1863,15 @@ for (;;) } /* PCRE does not allow THEN, SKIP, PRUNE or COMMIT to escape beyond a - recursion; they are treated as NOMATCH. These codes are defined in a - range that can be tested for. Any other return code is an error. */ + recursion; they cause a NOMATCH for the entire recursion. These codes + are defined in a range that can be tested for. */ + + if (rrc >= MATCH_BACKTRACK_MIN && rrc <= MATCH_BACKTRACK_MAX) + RRETURN(MATCH_NOMATCH); + + /* Any return code other than NOMATCH is an error. */ - else if (rrc != MATCH_NOMATCH && - (rrc < MATCH_BACKTRACK_MIN || rrc > MATCH_BACKTRACK_MAX)) + if (rrc != MATCH_NOMATCH) { DPRINTF(("Recursion gave error %d\n", rrc)); if (new_recursive.offset_save != stacksave) diff --git a/testdata/testoutput2 b/testdata/testoutput2 index 7ef3f96..1718b0e 100644 --- a/testdata/testoutput2 +++ b/testdata/testoutput2 @@ -12362,6 +12362,8 @@ assertion, and therefore fails the entire subroutine call. --/ /((?=a(*COMMIT)b)ab|ac){0}(?:(?1)|a(c))/ ac 0: ac + 1: <unset> + 2: c /-- These are all run as real matches in test 1; here we are just checking the settings of the anchored and startline bits. --/ |