diff options
Diffstat (limited to 'ext/pcre/pcrelib/doc/pcre.txt')
| -rw-r--r-- | ext/pcre/pcrelib/doc/pcre.txt | 2515 |
1 files changed, 1728 insertions, 787 deletions
diff --git a/ext/pcre/pcrelib/doc/pcre.txt b/ext/pcre/pcrelib/doc/pcre.txt index fdf0d6ff67..801bde279b 100644 --- a/ext/pcre/pcrelib/doc/pcre.txt +++ b/ext/pcre/pcrelib/doc/pcre.txt @@ -6,26 +6,35 @@ synopses of each function in the library have not been included. There are separate text files for the pcregrep and pcretest commands. ----------------------------------------------------------------------------- -PCRE(3) PCRE(3) +PCRE(3) PCRE(3) NAME PCRE - Perl-compatible regular expressions + INTRODUCTION The PCRE library is a set of functions that implement regular expres- sion pattern matching using the same syntax and semantics as Perl, with just a few differences. The current implementation of PCRE (release - 5.x) corresponds approximately with Perl 5.8, including support for + 6.x) corresponds approximately with Perl 5.8, including support for UTF-8 encoded strings and Unicode general category properties. However, this support has to be explicitly enabled; it is not the default. + In addition to the Perl-compatible matching function, PCRE also con- + tains an alternative matching function that matches the same compiled + patterns in a different way. In certain circumstances, the alternative + function has some advantages. For a discussion of the two matching + algorithms, see the pcrematching page. + PCRE is written in C and released as a C library. A number of people - have written wrappers and interfaces of various kinds. A C++ class is - included in these contributions, which can be found in the Contrib - directory at the primary FTP site, which is: + have written wrappers and interfaces of various kinds. In particular, + Google Inc. have provided a comprehensive C++ wrapper. This is now + included as part of the PCRE distribution. The pcrecpp page has details + of this interface. Other people's contributions can be found in the + Contrib directory at the primary FTP site, which is: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre @@ -40,6 +49,12 @@ INTRODUCTION ing PCRE for various operating systems can be found in the README file in the source distribution. + The library contains a number of undocumented internal functions and + data tables that are used by more than one of the exported external + functions, but which are not intended for use by external callers. + Their names all begin with "_pcre_", which hopefully will not provoke + any name clashes. + USER DOCUMENTATION @@ -50,22 +65,24 @@ USER DOCUMENTATION of searching. The sections are as follows: pcre this document - pcreapi details of PCRE's native API + pcreapi details of PCRE's native C API pcrebuild options for building PCRE pcrecallout details of the callout feature pcrecompat discussion of Perl compatibility + pcrecpp details of the C++ wrapper pcregrep description of the pcregrep command + pcrematching discussion of the two matching algorithms pcrepartial details of the partial matching facility pcrepattern syntax and semantics of supported regular expressions pcreperform discussion of performance issues - pcreposix the POSIX-compatible API + pcreposix the POSIX-compatible C API pcreprecompile details of saving and re-using precompiled patterns pcresample discussion of the sample program pcretest description of the pcretest testing command In addition, in the "man" and HTML formats, there is a short page for - each library function, listing its arguments and results. + each C library function, listing its arguments and results. LIMITATIONS @@ -90,70 +107,71 @@ LIMITATIONS tern, is 200. The maximum length of a subject string is the largest positive number - that an integer variable can hold. However, PCRE uses recursion to han- - dle subpatterns and indefinite repetition. This means that the avail- - able stack space may limit the size of a subject string that can be - processed by certain patterns. + that an integer variable can hold. However, when using the traditional + matching function, PCRE uses recursion to handle subpatterns and indef- + inite repetition. This means that the available stack space may limit + the size of a subject string that can be processed by certain patterns. UTF-8 AND UNICODE PROPERTY SUPPORT - From release 3.3, PCRE has had some support for character strings - encoded in the UTF-8 format. For release 4.0 this was greatly extended - to cover most common requirements, and in release 5.0 additional sup- + From release 3.3, PCRE has had some support for character strings + encoded in the UTF-8 format. For release 4.0 this was greatly extended + to cover most common requirements, and in release 5.0 additional sup- port for Unicode general category properties was added. - In order process UTF-8 strings, you must build PCRE to include UTF-8 - support in the code, and, in addition, you must call pcre_compile() - with the PCRE_UTF8 option flag. When you do this, both the pattern and - any subject strings that are matched against it are treated as UTF-8 + In order process UTF-8 strings, you must build PCRE to include UTF-8 + support in the code, and, in addition, you must call pcre_compile() + with the PCRE_UTF8 option flag. When you do this, both the pattern and + any subject strings that are matched against it are treated as UTF-8 strings instead of just strings of bytes. - If you compile PCRE with UTF-8 support, but do not use it at run time, - the library will be a bit bigger, but the additional run time overhead - is limited to testing the PCRE_UTF8 flag in several places, so should + If you compile PCRE with UTF-8 support, but do not use it at run time, + the library will be a bit bigger, but the additional run time overhead + is limited to testing the PCRE_UTF8 flag in several places, so should not be very large. If PCRE is built with Unicode character property support (which implies - UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup- + UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup- ported. The available properties that can be tested are limited to the - general category properties such as Lu for an upper case letter or Nd - for a decimal number. A full list is given in the pcrepattern documen- + general category properties such as Lu for an upper case letter or Nd + for a decimal number. A full list is given in the pcrepattern documen- tation. The PCRE library is increased in size by about 90K when Unicode property support is included. The following comments apply when PCRE is running in UTF-8 mode: - 1. When you set the PCRE_UTF8 flag, the strings passed as patterns and - subjects are checked for validity on entry to the relevant functions. + 1. When you set the PCRE_UTF8 flag, the strings passed as patterns and + subjects are checked for validity on entry to the relevant functions. If an invalid UTF-8 string is passed, an error return is given. In some - situations, you may already know that your strings are valid, and + situations, you may already know that your strings are valid, and therefore want to skip these checks in order to improve performance. If - you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, - PCRE assumes that the pattern or subject it is given (respectively) - contains only valid UTF-8 codes. In this case, it does not diagnose an - invalid UTF-8 string. If you pass an invalid UTF-8 string to PCRE when - PCRE_NO_UTF8_CHECK is set, the results are undefined. Your program may + you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, + PCRE assumes that the pattern or subject it is given (respectively) + contains only valid UTF-8 codes. In this case, it does not diagnose an + invalid UTF-8 string. If you pass an invalid UTF-8 string to PCRE when + PCRE_NO_UTF8_CHECK is set, the results are undefined. Your program may crash. 2. In a pattern, the escape sequence \x{...}, where the contents of the - braces is a string of hexadecimal digits, is interpreted as a UTF-8 - character whose code number is the given hexadecimal number, for exam- - ple: \x{1234}. If a non-hexadecimal digit appears between the braces, + braces is a string of hexadecimal digits, is interpreted as a UTF-8 + character whose code number is the given hexadecimal number, for exam- + ple: \x{1234}. If a non-hexadecimal digit appears between the braces, the item is not recognized. This escape sequence can be used either as a literal, or within a character class. - 3. The original hexadecimal escape sequence, \xhh, matches a two-byte + 3. The original hexadecimal escape sequence, \xhh, matches a two-byte UTF-8 character if the value is greater than 127. - 4. Repeat quantifiers apply to complete UTF-8 characters, not to indi- + 4. Repeat quantifiers apply to complete UTF-8 characters, not to indi- vidual bytes, for example: \x{100}{3}. - 5. The dot metacharacter matches one UTF-8 character instead of a sin- + 5. The dot metacharacter matches one UTF-8 character instead of a sin- gle byte. - 6. The escape sequence \C can be used to match a single byte in UTF-8 - mode, but its use can lead to some strange effects. + 6. The escape sequence \C can be used to match a single byte in UTF-8 + mode, but its use can lead to some strange effects. This facility is + not available in the alternative matching function, pcre_dfa_exec(). 7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly test characters of any code value, but the characters that PCRE recog- @@ -177,22 +195,26 @@ UTF-8 AND UNICODE PROPERTY SUPPORT AUTHOR - Philip Hazel <ph10@cam.ac.uk> + Philip Hazel University Computing Service, Cambridge CB2 3QG, England. - Phone: +44 1223 334714 -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ + Putting an actual email address here seems to have been a spam magnet, + so I've taken it away. If you want to email me, use my initial and sur- + name, separated by a dot, at the domain ucs.cam.ac.uk. + +Last updated: 07 March 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCREBUILD(3) PCREBUILD(3) NAME PCRE - Perl-compatible regular expressions + PCRE BUILD-TIME OPTIONS This document describes the optional features of PCRE that can be @@ -287,16 +309,18 @@ POSIX MALLOC USAGE LIMITING PCRE RESOURCE USAGE Internally, PCRE has a function called match(), which it calls repeat- - edly (possibly recursively) when matching a pattern. By controlling the - maximum number of times this function may be called during a single - matching operation, a limit can be placed on the resources used by a - single call to pcre_exec(). The limit can be changed at run time, as - described in the pcreapi documentation. The default is 10 million, but - this can be changed by adding a setting such as + edly (possibly recursively) when matching a pattern with the + pcre_exec() function. By controlling the maximum number of times this + function may be called during a single matching operation, a limit can + be placed on the resources used by a single call to pcre_exec(). The + limit can be changed at run time, as described in the pcreapi documen- + tation. The default is 10 million, but this can be changed by adding a + setting such as --with-match-limit=500000 - to the configure command. + to the configure command. This setting has no effect on the + pcre_dfa_exec() matching function. HANDLING VERY LARGE PATTERNS @@ -324,14 +348,14 @@ HANDLING VERY LARGE PATTERNS AVOIDING EXCESSIVE STACK USAGE - PCRE implements backtracking while matching by making recursive calls - to an internal function called match(). In environments where the size - of the stack is limited, this can severely limit PCRE's operation. (The - Unix environment does not usually suffer from this problem.) An alter- - native approach that uses memory from the heap to remember data, - instead of using recursive function calls, has been implemented to work - round this problem. If you want to build a version of PCRE that works - this way, add + When matching with the pcre_exec() function, PCRE implements backtrack- + ing by making recursive calls to an internal function called match(). + In environments where the size of the stack is limited, this can se- + verely limit PCRE's operation. (The Unix environment does not usually + suffer from this problem.) An alternative approach that uses memory + from the heap to remember data, instead of using recursive function + calls, has been implemented to work round this problem. If you want to + build a version of PCRE that works this way, add --disable-stack-for-recursion @@ -342,31 +366,201 @@ AVOIDING EXCESSIVE STACK USAGE the blocks are always freed in reverse order. A calling program might be able to implement optimized functions that perform better than the standard malloc() and free() functions. PCRE runs noticeably more - slowly when built in this way. + slowly when built in this way. This option affects only the pcre_exec() + function; it is not relevant for the the pcre_dfa_exec() function. USING EBCDIC CODE - PCRE assumes by default that it will run in an environment where the - character code is ASCII (or Unicode, which is a superset of ASCII). - PCRE can, however, be compiled to run in an EBCDIC environment by + PCRE assumes by default that it will run in an environment where the + character code is ASCII (or Unicode, which is a superset of ASCII). + PCRE can, however, be compiled to run in an EBCDIC environment by adding --enable-ebcdic to the configure command. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCREMATCHING(3) PCREMATCHING(3) NAME PCRE - Perl-compatible regular expressions + +PCRE MATCHING ALGORITHMS + + This document describes the two different algorithms that are available + in PCRE for matching a compiled regular expression against a given sub- + ject string. The "standard" algorithm is the one provided by the + pcre_exec() function. This works in the same was as Perl's matching + function, and provides a Perl-compatible matching operation. + + An alternative algorithm is provided by the pcre_dfa_exec() function; + this operates in a different way, and is not Perl-compatible. It has + advantages and disadvantages compared with the standard algorithm, and + these are described below. + + When there is only one possible way in which a given subject string can + match a pattern, the two algorithms give the same answer. A difference + arises, however, when there are multiple possibilities. For example, if + the pattern + + ^<.*> + + is matched against the string + + <something> <something else> <something further> + + there are three possible answers. The standard algorithm finds only one + of them, whereas the DFA algorithm finds all three. + + +REGULAR EXPRESSIONS AS TREES + + The set of strings that are matched by a regular expression can be rep- + resented as a tree structure. An unlimited repetition in the pattern + makes the tree of infinite size, but it is still a tree. Matching the + pattern to a given subject string (from a given starting point) can be + thought of as a search of the tree. There are two standard ways to + search a tree: depth-first and breadth-first, and these correspond to + the two matching algorithms provided by PCRE. + + +THE STANDARD MATCHING ALGORITHM + + In the terminology of Jeffrey Friedl's book Mastering Regular Expres- + sions, the standard algorithm is an "NFA algorithm". It conducts a + depth-first search of the pattern tree. That is, it proceeds along a + single path through the tree, checking that the subject matches what is + required. When there is a mismatch, the algorithm tries any alterna- + tives at the current point, and if they all fail, it backs up to the + previous branch point in the tree, and tries the next alternative + branch at that level. This often involves backing up (moving to the + left) in the subject string as well. The order in which repetition + branches are tried is controlled by the greedy or ungreedy nature of + the quantifier. + + If a leaf node is reached, a matching string has been found, and at + that point the algorithm stops. Thus, if there is more than one possi- + ble match, this algorithm returns the first one that it finds. Whether + this is the shortest, the longest, or some intermediate length depends + on the way the greedy and ungreedy repetition quantifiers are specified + in the pattern. + + Because it ends up with a single path through the tree, it is rela- + tively straightforward for this algorithm to keep track of the sub- + strings that are matched by portions of the pattern in parentheses. + This provides support for capturing parentheses and back references. + + +THE DFA MATCHING ALGORITHM + + DFA stands for "deterministic finite automaton", but you do not need to + understand the origins of that name. This algorithm conducts a breadth- + first search of the tree. Starting from the first matching point in the + subject, it scans the subject string from left to right, once, charac- + ter by character, and as it does this, it remembers all the paths + through the tree that represent valid matches. + + The scan continues until either the end of the subject is reached, or + there are no more unterminated paths. At this point, terminated paths + represent the different matching possibilities (if there are none, the + match has failed). Thus, if there is more than one possible match, + this algorithm finds all of them, and in particular, it finds the long- + est. In PCRE, there is an option to stop the algorithm after the first + match (which is necessarily the shortest) has been found. + + Note that all the matches that are found start at the same point in the + subject. If the pattern + + cat(er(pillar)?) + + is matched against the string "the caterpillar catchment", the result + will be the three strings "cat", "cater", and "caterpillar" that start + at the fourth character of the subject. The algorithm does not automat- + ically move on to find matches that start at later positions. + + There are a number of features of PCRE regular expressions that are not + supported by the DFA matching algorithm. They are as follows: + + 1. Because the algorithm finds all possible matches, the greedy or + ungreedy nature of repetition quantifiers is not relevant. Greedy and + ungreedy quantifiers are treated in exactly the same way. + + 2. When dealing with multiple paths through the tree simultaneously, it + is not straightforward to keep track of captured substrings for the + different matching possibilities, and PCRE's implementation of this + algorithm does not attempt to do this. This means that no captured sub- + strings are available. + + 3. Because no substrings are captured, back references within the pat- + tern are not supported, and cause errors if encountered. + + 4. For the same reason, conditional expressions that use a backrefer- + ence as the condition are not supported. + + 5. Callouts are supported, but the value of the capture_top field is + always 1, and the value of the capture_last field is always -1. + + 6. The \C escape sequence, which (in the standard algorithm) matches a + single byte, even in UTF-8 mode, is not supported because the DFA algo- + rithm moves through the subject string one character at a time, for all + active paths through the tree. + + +ADVANTAGES OF THE DFA ALGORITHM + + Using the DFA matching algorithm provides the following advantages: + + 1. All possible matches (at a single point in the subject) are automat- + ically found, and in particular, the longest match is found. To find + more than one match using the standard algorithm, you have to do kludgy + things with callouts. + + 2. There is much better support for partial matching. The restrictions + on the content of the pattern that apply when using the standard algo- + rithm for partial matching do not apply to the DFA algorithm. For non- + anchored patterns, the starting position of a partial match is avail- + able. + + 3. Because the DFA algorithm scans the subject string just once, and + never needs to backtrack, it is possible to pass very long subject + strings to the matching function in several pieces, checking for par- + tial matching each time. + + +DISADVANTAGES OF THE DFA ALGORITHM + + The DFA algorithm suffers from a number of disadvantages: + + 1. It is substantially slower than the standard algorithm. This is + partly because it has to search for all possible matches, but is also + because it is less susceptible to optimization. + + 2. Capturing parentheses and back references are not supported. + + 3. The "atomic group" feature of PCRE regular expressions is supported, + but does not provide the advantage that it does for the standard algo- + rithm. + +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ + + +PCREAPI(3) PCREAPI(3) + + +NAME + PCRE - Perl-compatible regular expressions + + PCRE NATIVE API #include <pcre.h> @@ -375,6 +569,11 @@ PCRE NATIVE API const char **errptr, int *erroffset, const unsigned char *tableptr); + pcre *pcre_compile2(const char *pattern, int options, + int *errorcodeptr, + const char **errptr, int *erroffset, + const unsigned char *tableptr); + pcre_extra *pcre_study(const pcre *code, int options, const char **errptr); @@ -382,6 +581,11 @@ PCRE NATIVE API const char *subject, int length, int startoffset, int options, int *ovector, int ovecsize); + int pcre_dfa_exec(const pcre *code, const pcre_extra *extra, + const char *subject, int length, int startoffset, + int options, int *ovector, int ovecsize, + int *workspace, int wscount); + int pcre_copy_named_substring(const pcre *code, const char *subject, int *ovector, int stringcount, const char *stringname, @@ -417,6 +621,8 @@ PCRE NATIVE API int pcre_info(const pcre *code, int *optptr, int *firstcharptr); + int pcre_refcount(pcre *code, int adjust); + int pcre_config(int what, void *where); char *pcre_version(void); @@ -436,25 +642,36 @@ PCRE API OVERVIEW PCRE has its own native API, which is described in this document. There is also a set of wrapper functions that correspond to the POSIX regular - expression API. These are described in the pcreposix documentation. + expression API. These are described in the pcreposix documentation. + Both of these APIs define a set of C function calls. A C++ wrapper is + distributed with PCRE. It is documented in the pcrecpp page. - The native API function prototypes are defined in the header file - pcre.h, and on Unix systems the library itself is called libpcre. It + The native API C function prototypes are defined in the header file + pcre.h, and on Unix systems the library itself is called libpcre. It can normally be accessed by adding -lpcre to the command for linking an application that uses PCRE. The header file defines the macros PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num- bers for the library. Applications can use these to include support for different releases of PCRE. - The functions pcre_compile(), pcre_study(), and pcre_exec() are used - for compiling and matching regular expressions. A sample program that - demonstrates the simplest way of using them is provided in the file - called pcredemo.c in the source distribution. The pcresample documenta- - tion describes how to run it. - - In addition to the main compiling and matching functions, there are - convenience functions for extracting captured substrings from a matched - subject string. They are: + The functions pcre_compile(), pcre_compile2(), pcre_study(), and + pcre_exec() are used for compiling and matching regular expressions in + a Perl-compatible manner. A sample program that demonstrates the sim- + plest way of using them is provided in the file called pcredemo.c in + the source distribution. The pcresample documentation describes how to + run it. + + A second matching function, pcre_dfa_exec(), which is not Perl-compati- + ble, is also provided. This uses a different algorithm for the match- + ing. This allows it to find all possible matches (at a given point in + the subject), not just one. However, this algorithm does not return + captured substrings. A description of the two matching algorithms and + their advantages and disadvantages is given in the pcrematching docu- + mentation. + + In addition to the main compiling and matching functions, there are + convenience functions for extracting captured substrings from a subject + string that is matched by pcre_exec(). They are: pcre_copy_substring() pcre_copy_named_substring() @@ -466,11 +683,12 @@ PCRE API OVERVIEW pcre_free_substring() and pcre_free_substring_list() are also provided, to free the memory used for extracted strings. - The function pcre_maketables() is used to build a set of character - tables in the current locale for passing to pcre_compile() or - pcre_exec(). This is an optional facility that is provided for spe- - cialist use. Most commonly, no special tables are passed, in which case - internal tables that are generated when PCRE is built are used. + The function pcre_maketables() is used to build a set of character + tables in the current locale for passing to pcre_compile(), + pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is + provided for specialist use. Most commonly, no special tables are + passed, in which case internal tables that are generated when PCRE is + built are used. The function pcre_fullinfo() is used to find out information about a compiled pattern; pcre_info() is an obsolete version that returns only @@ -478,6 +696,10 @@ PCRE API OVERVIEW patibility. The function pcre_version() returns a pointer to a string containing the version of PCRE and its date of release. + The function pcre_refcount() maintains a reference count in a data + block containing a compiled pattern. This is provided for the benefit + of object-oriented applications. + The global variables pcre_malloc and pcre_free initially contain the entry points of the standard malloc() and free() functions, respec- tively. PCRE calls the memory management functions via these variables, @@ -487,28 +709,28 @@ PCRE API OVERVIEW The global variables pcre_stack_malloc and pcre_stack_free are also indirections to memory management functions. These special functions are used only when PCRE is compiled to use the heap for remembering - data, instead of recursive function calls. This is a non-standard way - of building PCRE, for use in environments that have limited stacks. - Because of the greater use of memory management, it runs more slowly. - Separate functions are provided so that special-purpose external code - can be used for this case. When used, these functions are always called - in a stack-like manner (last obtained, first freed), and always for - memory blocks of the same size. + data, instead of recursive function calls, when running the pcre_exec() + function. This is a non-standard way of building PCRE, for use in envi- + ronments that have limited stacks. Because of the greater use of memory + management, it runs more slowly. Separate functions are provided so + that special-purpose external code can be used for this case. When + used, these functions are always called in a stack-like manner (last + obtained, first freed), and always for memory blocks of the same size. The global variable pcre_callout initially contains NULL. It can be set - by the caller to a "callout" function, which PCRE will then call at - specified points during a matching operation. Details are given in the + by the caller to a "callout" function, which PCRE will then call at + specified points during a matching operation. Details are given in the pcrecallout documentation. MULTITHREADING - The PCRE functions can be used in multi-threading applications, with + The PCRE functions can be used in multi-threading applications, with the proviso that the memory management functions pointed to by pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the callout function pointed to by pcre_callout, are shared by all threads. - The compiled form of a regular expression is not altered during match- + The compiled form of a regular expression is not altered during match- ing, so the same compiled pattern can safely be used by several threads at once. @@ -516,8 +738,8 @@ MULTITHREADING SAVING PRECOMPILED PATTERNS FOR LATER USE The compiled form of a regular expression can be saved and re-used at a - later time, possibly by a different program, and even on a host other - than the one on which it was compiled. Details are given in the + later time, possibly by a different program, and even on a host other + than the one on which it was compiled. Details are given in the pcreprecompile documentation. @@ -525,63 +747,63 @@ CHECKING BUILD-TIME OPTIONS int pcre_config(int what, void *where); - The function pcre_config() makes it possible for a PCRE client to dis- + The function pcre_config() makes it possible for a PCRE client to dis- cover which optional features have been compiled into the PCRE library. - The pcrebuild documentation has more details about these optional fea- + The pcrebuild documentation has more details about these optional fea- tures. - The first argument for pcre_config() is an integer, specifying which + The first argument for pcre_config() is an integer, specifying which information is required; the second argument is a pointer to a variable - into which the information is placed. The following information is + into which the information is placed. The following information is available: PCRE_CONFIG_UTF8 - The output is an integer that is set to one if UTF-8 support is avail- + The output is an integer that is set to one if UTF-8 support is avail- able; otherwise it is set to zero. PCRE_CONFIG_UNICODE_PROPERTIES - The output is an integer that is set to one if support for Unicode + The output is an integer that is set to one if support for Unicode character properties is available; otherwise it is set to zero. PCRE_CONFIG_NEWLINE - The output is an integer that is set to the value of the code that is - used for the newline character. It is either linefeed (10) or carriage - return (13), and should normally be the standard character for your + The output is an integer that is set to the value of the code that is + used for the newline character. It is either linefeed (10) or carriage + return (13), and should normally be the standard character for your operating system. PCRE_CONFIG_LINK_SIZE - The output is an integer that contains the number of bytes used for + The output is an integer that contains the number of bytes used for internal linkage in compiled regular expressions. The value is 2, 3, or - 4. Larger values allow larger regular expressions to be compiled, at - the expense of slower matching. The default value of 2 is sufficient - for all but the most massive patterns, since it allows the compiled + 4. Larger values allow larger regular expressions to be compiled, at + the expense of slower matching. The default value of 2 is sufficient + for all but the most massive patterns, since it allows the compiled pattern to be up to 64K in size. PCRE_CONFIG_POSIX_MALLOC_THRESHOLD - The output is an integer that contains the threshold above which the - POSIX interface uses malloc() for output vectors. Further details are + The output is an integer that contains the threshold above which the + POSIX interface uses malloc() for output vectors. Further details are given in the pcreposix documentation. PCRE_CONFIG_MATCH_LIMIT The output is an integer that gives the default limit for the number of - internal matching function calls in a pcre_exec() execution. Further + internal matching function calls in a pcre_exec() execution. Further details are given with pcre_exec() below. PCRE_CONFIG_STACKRECURSE - The output is an integer that is set to one if internal recursion is - implemented by recursive function calls that use the stack to remember - their state. This is the usual way that PCRE is compiled. The output is - zero if PCRE was compiled to use blocks of data on the heap instead of - recursive function calls. In this case, pcre_stack_malloc and - pcre_stack_free are called to manage memory blocks on the heap, thus - avoiding the use of the stack. + The output is an integer that is set to one if internal recursion when + running pcre_exec() is implemented by recursive function calls that use + the stack to remember their state. This is the usual way that PCRE is + compiled. The output is zero if PCRE was compiled to use blocks of data + on the heap instead of recursive function calls. In this case, + pcre_stack_malloc and pcre_stack_free are called to manage memory + blocks on the heap, thus avoiding the use of the stack. COMPILING A PATTERN @@ -590,38 +812,52 @@ COMPILING A PATTERN const char **errptr, int *erroffset, const unsigned char *tableptr); - The function pcre_compile() is called to compile a pattern into an - internal form. The pattern is a C string terminated by a binary zero, - and is passed in the pattern argument. A pointer to a single block of - memory that is obtained via pcre_malloc is returned. This contains the - compiled code and related data. The pcre type is defined for the - returned block; this is a typedef for a structure whose contents are - not externally defined. It is up to the caller to free the memory when - it is no longer required. + pcre *pcre_compile2(const char *pattern, int options, + int *errorcodeptr, + const char **errptr, int *erroffset, + const unsigned char *tableptr); - Although the compiled code of a PCRE regex is relocatable, that is, it + Either of the functions pcre_compile() or pcre_compile2() can be called + to compile a pattern into an internal form. The only difference between + the two interfaces is that pcre_compile2() has an additional argument, + errorcodeptr, via which a numerical error code can be returned. + + The pattern is a C string terminated by a binary zero, and is passed in + the pattern argument. A pointer to a single block of memory that is + obtained via pcre_malloc is returned. This contains the compiled code + and related data. The pcre type is defined for the returned block; this + is a typedef for a structure whose contents are not externally defined. + It is up to the caller to free the memory when it is no longer + required. + + Although the compiled code of a PCRE regex is relocatable, that is, it does not depend on memory location, the complete pcre data block is not - fully relocatable, because it may contain a copy of the tableptr argu- + fully relocatable, because it may contain a copy of the tableptr argu- ment, which is an address (see below). The options argument contains independent bits that affect the compila- - tion. It should be zero if no options are required. The available - options are described below. Some of them, in particular, those that - are compatible with Perl, can also be set and unset from within the - pattern (see the detailed description in the pcrepattern documenta- - tion). For these options, the contents of the options argument speci- - fies their initial settings at the start of compilation and execution. - The PCRE_ANCHORED option can be set at the time of matching as well as + tion. It should be zero if no options are required. The available + options are described below. Some of them, in particular, those that + are compatible with Perl, can also be set and unset from within the + pattern (see the detailed description in the pcrepattern documenta- + tion). For these options, the contents of the options argument speci- + fies their initial settings at the start of compilation and execution. + The PCRE_ANCHORED option can be set at the time of matching as well as at compile time. If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise, - if compilation of a pattern fails, pcre_compile() returns NULL, and + if compilation of a pattern fails, pcre_compile() returns NULL, and sets the variable pointed to by errptr to point to a textual error mes- - sage. The offset from the start of the pattern to the character where - the error was discovered is placed in the variable pointed to by - erroffset, which must not be NULL. If it is, an immediate error is + sage. The offset from the start of the pattern to the character where + the error was discovered is placed in the variable pointed to by + erroffset, which must not be NULL. If it is, an immediate error is given. + If pcre_compile2() is used instead of pcre_compile(), and the error- + codeptr argument is not NULL, a non-zero error code number is returned + via this argument in the event of an error. This is in addition to the + textual error message. Error codes and messages are listed below. + If the final argument, tableptr, is NULL, PCRE uses a default set of character tables that are built when PCRE is compiled, using the default C locale. Otherwise, tableptr must be an address that is the @@ -664,139 +900,206 @@ COMPILING A PATTERN If this bit is set, letters in the pattern match both upper and lower case letters. It is equivalent to Perl's /i option, and it can be - changed within a pattern by a (?i) option setting. When running in - UTF-8 mode, case support for high-valued characters is available only - when PCRE is built with Unicode character property support. + changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE + always understands the concept of case for characters whose values are + less than 128, so caseless matching is always possible. For characters + with higher values, the concept of case is supported if PCRE is com- + piled with Unicode property support, but not otherwise. If you want to + use caseless matching for characters 128 and above, you must ensure + that PCRE is compiled with Unicode property support as well as with + UTF-8 support. PCRE_DOLLAR_ENDONLY - If this bit is set, a dollar metacharacter in the pattern matches only - at the end of the subject string. Without this option, a dollar also - matches immediately before the final character if it is a newline (but - not before any other newlines). The PCRE_DOLLAR_ENDONLY option is + If this bit is set, a dollar metacharacter in the pattern matches only + at the end of the subject string. Without this option, a dollar also + matches immediately before the final character if it is a newline (but + not before any other newlines). The PCRE_DOLLAR_ENDONLY option is ignored if PCRE_MULTILINE is set. There is no equivalent to this option in Perl, and no way to set it within a pattern. PCRE_DOTALL If this bit is set, a dot metacharater in the pattern matches all char- - acters, including newlines. Without it, newlines are excluded. This - option is equivalent to Perl's /s option, and it can be changed within - a pattern by a (?s) option setting. A negative class such as [^a] - always matches a newline character, independent of the setting of this + acters, including newlines. Without it, newlines are excluded. This + option is equivalent to Perl's /s option, and it can be changed within + a pattern by a (?s) option setting. A negative class such as [^a] + always matches a newline character, independent of the setting of this option. PCRE_EXTENDED - If this bit is set, whitespace data characters in the pattern are - totally ignored except when escaped or inside a character class. - Whitespace does not include the VT character (code 11). In addition, - characters between an unescaped # outside a character class and the - next newline character, inclusive, are also ignored. This is equivalent - to Perl's /x option, and it can be changed within a pattern by a (?x) + If this bit is set, whitespace data characters in the pattern are + totally ignored except when escaped or inside a character class. White- + space does not include the VT character (code 11). In addition, charac- + ters between an unescaped # outside a character class and the next new- + line character, inclusive, are also ignored. This is equivalent to + Perl's /x option, and it can be changed within a pattern by a (?x) option setting. - This option makes it possible to include comments inside complicated - patterns. Note, however, that this applies only to data characters. - Whitespace characters may never appear within special character - sequences in a pattern, for example within the sequence (?( which + This option makes it possible to include comments inside complicated + patterns. Note, however, that this applies only to data characters. + Whitespace characters may never appear within special character + sequences in a pattern, for example within the sequence (?( which introduces a conditional subpattern. PCRE_EXTRA - This option was invented in order to turn on additional functionality - of PCRE that is incompatible with Perl, but it is currently of very - little use. When set, any backslash in a pattern that is followed by a - letter that has no special meaning causes an error, thus reserving - these combinations for future expansion. By default, as in Perl, a - backslash followed by a letter with no special meaning is treated as a - literal. There are at present no other features controlled by this + This option was invented in order to turn on additional functionality + of PCRE that is incompatible with Perl, but it is currently of very + little use. When set, any backslash in a pattern that is followed by a + letter that has no special meaning causes an error, thus reserving + these combinations for future expansion. By default, as in Perl, a + backslash followed by a letter with no special meaning is treated as a + literal. There are at present no other features controlled by this option. It can also be set by a (?X) option setting within a pattern. + PCRE_FIRSTLINE + + If this option is set, an unanchored pattern is required to match + before or at the first newline character in the subject string, though + the matched text may continue over the newline. + PCRE_MULTILINE - By default, PCRE treats the subject string as consisting of a single - line of characters (even if it actually contains newlines). The "start - of line" metacharacter (^) matches only at the start of the string, - while the "end of line" metacharacter ($) matches only at the end of + By default, PCRE treats the subject string as consisting of a single + line of characters (even if it actually contains newlines). The "start + of line" metacharacter (^) matches only at the start of the string, + while the "end of line" metacharacter ($) matches only at the end of the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY is set). This is the same as Perl. - When PCRE_MULTILINE it is set, the "start of line" and "end of line" - constructs match immediately following or immediately before any new- - line in the subject string, respectively, as well as at the very start - and end. This is equivalent to Perl's /m option, and it can be changed + When PCRE_MULTILINE it is set, the "start of line" and "end of line" + constructs match immediately following or immediately before any new- + line in the subject string, respectively, as well as at the very start + and end. This is equivalent to Perl's /m option, and it can be changed within a pattern by a (?m) option setting. If there are no "\n" charac- - ters in a subject string, or no occurrences of ^ or $ in a pattern, + ters in a subject string, or no occurrences of ^ or $ in a pattern, setting PCRE_MULTILINE has no effect. PCRE_NO_AUTO_CAPTURE If this option is set, it disables the use of numbered capturing paren- - theses in the pattern. Any opening parenthesis that is not followed by - ? behaves as if it were followed by ?: but named parentheses can still - be used for capturing (and they acquire numbers in the usual way). + theses in the pattern. Any opening parenthesis that is not followed by + ? behaves as if it were followed by ?: but named parentheses can still + be used for capturing (and they acquire numbers in the usual way). There is no equivalent of this option in Perl. PCRE_UNGREEDY - This option inverts the "greediness" of the quantifiers so that they - are not greedy by default, but become greedy if followed by "?". It is - not compatible with Perl. It can also be set by a (?U) option setting + This option inverts the "greediness" of the quantifiers so that they + are not greedy by default, but become greedy if followed by "?". It is + not compatible with Perl. It can also be set by a (?U) option setting within the pattern. PCRE_UTF8 - This option causes PCRE to regard both the pattern and the subject as - strings of UTF-8 characters instead of single-byte character strings. - However, it is available only when PCRE is built to include UTF-8 sup- - port. If not, the use of this option provokes an error. Details of how - this option changes the behaviour of PCRE are given in the section on + This option causes PCRE to regard both the pattern and the subject as + strings of UTF-8 characters instead of single-byte character strings. + However, it is available only when PCRE is built to include UTF-8 sup- + port. If not, the use of this option provokes an error. Details of how + this option changes the behaviour of PCRE are given in the section on UTF-8 support in the main pcre page. PCRE_NO_UTF8_CHECK When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is - automatically checked. If an invalid UTF-8 sequence of bytes is found, - pcre_compile() returns an error. If you already know that your pattern - is valid, and you want to skip this check for performance reasons, you - can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of + automatically checked. If an invalid UTF-8 sequence of bytes is found, + pcre_compile() returns an error. If you already know that your pattern + is valid, and you want to skip this check for performance reasons, you + can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of passing an invalid UTF-8 string as a pattern is undefined. It may cause - your program to crash. Note that this option can also be passed to - pcre_exec(), to suppress the UTF-8 validity checking of subject - strings. + your program to crash. Note that this option can also be passed to + pcre_exec() and pcre_dfa_exec(), to suppress the UTF-8 validity check- + ing of subject strings. + + +COMPILATION ERROR CODES + + The following table lists the error codes than may be returned by + pcre_compile2(), along with the error messages that may be returned by + both compiling functions. + + 0 no error + 1 \ at end of pattern + 2 \c at end of pattern + 3 unrecognized character follows \ + 4 numbers out of order in {} quantifier + 5 number too big in {} quantifier + 6 missing terminating ] for character class + 7 invalid escape sequence in character class + 8 range out of order in character class + 9 nothing to repeat + 10 operand of unlimited repeat could match the empty string + 11 internal error: unexpected repeat + 12 unrecognized character after (? + 13 POSIX named classes are supported only within a class + 14 missing ) + 15 reference to non-existent subpattern + 16 erroffset passed as NULL + 17 unknown option bit(s) set + 18 missing ) after comment + 19 parentheses nested too deeply + 20 regular expression too large + 21 failed to get memory + 22 unmatched parentheses + 23 internal error: code overflow + 24 unrecognized character after (?< + 25 lookbehind assertion is not fixed length + 26 malformed number after (?( + 27 conditional group contains more than two branches + 28 assertion expected after (?( + 29 (?R or (?digits must be followed by ) + 30 unknown POSIX class name + 31 POSIX collating elements are not supported + 32 this version of PCRE is not compiled with PCRE_UTF8 support + 33 spare error + 34 character value in \x{...} sequence is too large + 35 invalid condition (?(0) + 36 \C not allowed in lookbehind assertion + 37 PCRE does not support \L, \l, \N, \U, or \u + 38 number after (?C is > 255 + 39 closing ) for (?C expected + 40 recursive call could loop indefinitely + 41 unrecognized character after (?P + 42 syntax error after (?P + 43 two named groups have the same name + 44 invalid UTF-8 string + 45 support for \P, \p, and \X has not been compiled + 46 malformed \P or \p sequence + 47 unknown property name after \P or \p STUDYING A PATTERN - pcre_extra *pcre_study(const pcre *code, int options, + pcre_extra *pcre_study(const pcre *code, int options const char **errptr); - If a compiled pattern is going to be used several times, it is worth + If a compiled pattern is going to be used several times, it is worth spending more time analyzing it in order to speed up the time taken for - matching. The function pcre_study() takes a pointer to a compiled pat- + matching. The function pcre_study() takes a pointer to a compiled pat- tern as its first argument. If studying the pattern produces additional - information that will help speed up matching, pcre_study() returns a - pointer to a pcre_extra block, in which the study_data field points to + information that will help speed up matching, pcre_study() returns a + pointer to a pcre_extra block, in which the study_data field points to the results of the study. The returned value from pcre_study() can be passed directly to - pcre_exec(). However, a pcre_extra block also contains other fields - that can be set by the caller before the block is passed; these are + pcre_exec(). However, a pcre_extra block also contains other fields + that can be set by the caller before the block is passed; these are described below in the section on matching a pattern. - If studying the pattern does not produce any additional information, + If studying the pattern does not produce any additional information pcre_study() returns NULL. In that circumstance, if the calling program - wants to pass any of the other fields to pcre_exec(), it must set up + wants to pass any of the other fields to pcre_exec(), it must set up its own pcre_extra block. - The second argument of pcre_study() contains option bits. At present, + The second argument of pcre_study() contains option bits. At present, no options are defined, and this argument should always be zero. - The third argument for pcre_study() is a pointer for an error message. - If studying succeeds (even if no data is returned), the variable it - points to is set to NULL. Otherwise it points to a textual error mes- - sage. You should therefore test the error pointer for NULL after call- + The third argument for pcre_study() is a pointer for an error message. + If studying succeeds (even if no data is returned), the variable it + points to is set to NULL. Otherwise it points to a textual error mes- + sage. You should therefore test the error pointer for NULL after call- ing pcre_study(), to be sure that it has run successfully. This is a typical call to pcre_study(): @@ -808,51 +1111,51 @@ STUDYING A PATTERN &error); /* set to NULL or points to a message */ At present, studying a pattern is useful only for non-anchored patterns - that do not have a single fixed starting character. A bitmap of possi- + that do not have a single fixed starting character. A bitmap of possi- ble starting bytes is created. LOCALE SUPPORT - PCRE handles caseless matching, and determines whether characters are - letters, digits, or whatever, by reference to a set of tables, indexed - by character value. (When running in UTF-8 mode, this applies only to - characters with codes less than 128. Higher-valued codes never match - escapes such as \w or \d, but can be tested with \p if PCRE is built - with Unicode character property support.) - - An internal set of tables is created in the default C locale when PCRE - is built. This is used when the final argument of pcre_compile() is - NULL, and is sufficient for many applications. An alternative set of - tables can, however, be supplied. These may be created in a different - locale from the default. As more and more applications change to using + PCRE handles caseless matching, and determines whether characters are + letters digits, or whatever, by reference to a set of tables, indexed + by character value. When running in UTF-8 mode, this applies only to + characters with codes less than 128. Higher-valued codes never match + escapes such as \w or \d, but can be tested with \p if PCRE is built + with Unicode character property support. + + An internal set of tables is created in the default C locale when PCRE + is built. This is used when the final argument of pcre_compile() is + NULL, and is sufficient for many applications. An alternative set of + tables can, however, be supplied. These may be created in a different + locale from the default. As more and more applications change to using Unicode, the need for this locale support is expected to die away. - External tables are built by calling the pcre_maketables() function, - which has no arguments, in the relevant locale. The result can then be - passed to pcre_compile() or pcre_exec() as often as necessary. For - example, to build and use tables that are appropriate for the French - locale (where accented characters with values greater than 128 are + External tables are built by calling the pcre_maketables() function, + which has no arguments, in the relevant locale. The result can then be + passed to pcre_compile() or pcre_exec() as often as necessary. For + example, to build and use tables that are appropriate for the French + locale (where accented characters with values greater than 128 are treated as letters), the following code could be used: setlocale(LC_CTYPE, "fr_FR"); tables = pcre_maketables(); re = pcre_compile(..., tables); - When pcre_maketables() runs, the tables are built in memory that is - obtained via pcre_malloc. It is the caller's responsibility to ensure - that the memory containing the tables remains available for as long as + When pcre_maketables() runs, the tables are built in memory that is + obtained via pcre_malloc. It is the caller's responsibility to ensure + that the memory containing the tables remains available for as long as it is needed. The pointer that is passed to pcre_compile() is saved with the compiled - pattern, and the same tables are used via this pointer by pcre_study() + pattern, and the same tables are used via this pointer by pcre_study() and normally also by pcre_exec(). Thus, by default, for any single pat- tern, compilation, studying and matching all happen in the same locale, but different patterns can be compiled in different locales. - It is possible to pass a table pointer or NULL (indicating the use of - the internal tables) to pcre_exec(). Although not intended for this - purpose, this facility could be used to match a pattern in a different + It is possible to pass a table pointer or NULL (indicating the use of + the internal tables) to pcre_exec(). Although not intended for this + purpose, this facility could be used to match a pattern in a different locale from the one in which it was compiled. Passing table pointers at run time is discussed below in the section on matching a pattern. @@ -862,15 +1165,15 @@ INFORMATION ABOUT A PATTERN int pcre_fullinfo(const pcre *code, const pcre_extra *extra, int what, void *where); - The pcre_fullinfo() function returns information about a compiled pat- + The pcre_fullinfo() function returns information about a compiled pat- tern. It replaces the obsolete pcre_info() function, which is neverthe- less retained for backwards compability (and is documented below). - The first argument for pcre_fullinfo() is a pointer to the compiled - pattern. The second argument is the result of pcre_study(), or NULL if - the pattern was not studied. The third argument specifies which piece - of information is required, and the fourth argument is a pointer to a - variable to receive the data. The yield of the function is zero for + The first argument for pcre_fullinfo() is a pointer to the compiled + pattern. The second argument is the result of pcre_study(), or NULL if + the pattern was not studied. The third argument specifies which piece + of information is required, and the fourth argument is a pointer to a + variable to receive the data. The yield of the function is zero for success, or one of the following negative numbers: PCRE_ERROR_NULL the argument code was NULL @@ -878,9 +1181,9 @@ INFORMATION ABOUT A PATTERN PCRE_ERROR_BADMAGIC the "magic number" was not found PCRE_ERROR_BADOPTION the value of what was invalid - The "magic number" is placed at the start of each compiled pattern as - an simple check against passing an arbitrary memory pointer. Here is a - typical call of pcre_fullinfo(), to obtain the length of the compiled + The "magic number" is placed at the start of each compiled pattern as + an simple check against passing an arbitrary memory pointer. Here is a + typical call of pcre_fullinfo(), to obtain the length of the compiled pattern: int rc; @@ -891,64 +1194,64 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE, /* what is required */ &length); /* where to put the data */ - The possible values for the third argument are defined in pcre.h, and + The possible values for the third argument are defined in pcre.h, and are as follows: PCRE_INFO_BACKREFMAX - Return the number of the highest back reference in the pattern. The - fourth argument should point to an int variable. Zero is returned if + Return the number of the highest back reference in the pattern. The + fourth argument should point to an int variable. Zero is returned if there are no back references. PCRE_INFO_CAPTURECOUNT - Return the number of capturing subpatterns in the pattern. The fourth + Return the number of capturing subpatterns in the pattern. The fourth argument should point to an int variable. - PCRE_INFO_DEFAULTTABLES + PCRE_INFO_DEFAULT_TABLES - Return a pointer to the internal default character tables within PCRE. - The fourth argument should point to an unsigned char * variable. This + Return a pointer to the internal default character tables within PCRE. + The fourth argument should point to an unsigned char * variable. This information call is provided for internal use by the pcre_study() func- - tion. External callers can cause PCRE to use its internal tables by + tion. External callers can cause PCRE to use its internal tables by passing a NULL table pointer. PCRE_INFO_FIRSTBYTE - Return information about the first byte of any matched string, for a - non-anchored pattern. (This option used to be called - PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards + Return information about the first byte of any matched string, for a + non-anchored pattern. (This option used to be called + PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards compatibility.) - If there is a fixed first byte, for example, from a pattern such as - (cat|cow|coyote), it is returned in the integer pointed to by where. + If there is a fixed first byte, for example, from a pattern such as + (cat|cow|coyote), it is returned in the integer pointed to by where. Otherwise, if either - (a) the pattern was compiled with the PCRE_MULTILINE option, and every + (a) the pattern was compiled with the PCRE_MULTILINE option, and every branch starts with "^", or (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not set (if it were set, the pattern would be anchored), - -1 is returned, indicating that the pattern matches only at the start - of a subject string or after any newline within the string. Otherwise + -1 is returned, indicating that the pattern matches only at the start + of a subject string or after any newline within the string. Otherwise -2 is returned. For anchored patterns, -2 is returned. PCRE_INFO_FIRSTTABLE - If the pattern was studied, and this resulted in the construction of a + If the pattern was studied, and this resulted in the construction of a 256-bit table indicating a fixed set of bytes for the first byte in any - matching string, a pointer to the table is returned. Otherwise NULL is - returned. The fourth argument should point to an unsigned char * vari- + matching string, a pointer to the table is returned. Otherwise NULL is + returned. The fourth argument should point to an unsigned char * vari- able. PCRE_INFO_LASTLITERAL - Return the value of the rightmost literal byte that must exist in any - matched string, other than at its start, if such a byte has been + Return the value of the rightmost literal byte that must exist in any + matched string, other than at its start, if such a byte has been recorded. The fourth argument should point to an int variable. If there - is no such byte, -1 is returned. For anchored patterns, a last literal - byte is recorded only if it follows something of variable length. For + is no such byte, -1 is returned. For anchored patterns, a last literal + byte is recorded only if it follows something of variable length. For example, for the pattern /^a\d+z\d+/ the returned value is "z", but for /^a\dz\d/ the returned value is -1. @@ -956,32 +1259,32 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_NAMEENTRYSIZE PCRE_INFO_NAMETABLE - PCRE supports the use of named as well as numbered capturing parenthe- - ses. The names are just an additional way of identifying the parenthe- + PCRE supports the use of named as well as numbered capturing parenthe- + ses. The names are just an additional way of identifying the parenthe- ses, which still acquire numbers. A convenience function called - pcre_get_named_substring() is provided for extracting an individual - captured substring by name. It is also possible to extract the data - directly, by first converting the name to a number in order to access - the correct pointers in the output vector (described with pcre_exec() - below). To do the conversion, you need to use the name-to-number map, + pcre_get_named_substring() is provided for extracting an individual + captured substring by name. It is also possible to extract the data + directly, by first converting the name to a number in order to access + the correct pointers in the output vector (described with pcre_exec() + below). To do the conversion, you need to use the name-to-number map, which is described by these three values. The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size - of each entry; both of these return an int value. The entry size - depends on the length of the longest name. PCRE_INFO_NAMETABLE returns - a pointer to the first entry of the table (a pointer to char). The + of each entry; both of these return an int value. The entry size + depends on the length of the longest name. PCRE_INFO_NAMETABLE returns + a pointer to the first entry of the table (a pointer to char). The first two bytes of each entry are the number of the capturing parenthe- - sis, most significant byte first. The rest of the entry is the corre- - sponding name, zero terminated. The names are in alphabetical order. - For example, consider the following pattern (assume PCRE_EXTENDED is + sis, most significant byte first. The rest of the entry is the corre- + sponding name, zero terminated. The names are in alphabetical order. + For example, consider the following pattern (assume PCRE_EXTENDED is set, so white space - including newlines - is ignored): (?P<date> (?P<year>(\d\d)?\d\d) - (?P<month>\d\d) - (?P<day>\d\d) ) - There are four named subpatterns, so the table has four entries, and - each entry in the table is eight bytes long. The table is as follows, + There are four named subpatterns, so the table has four entries, and + each entry in the table is eight bytes long. The table is as follows, with non-printing bytes shows in hexadecimal, and undefined bytes shown as ??: @@ -990,18 +1293,18 @@ INFORMATION ABOUT A PATTERN 00 04 m o n t h 00 00 02 y e a r 00 ?? - When writing code to extract data from named subpatterns using the + When writing code to extract data from named subpatterns using the name-to-number map, remember that the length of each entry is likely to be different for each compiled pattern. PCRE_INFO_OPTIONS - Return a copy of the options with which the pattern was compiled. The - fourth argument should point to an unsigned long int variable. These + Return a copy of the options with which the pattern was compiled. The + fourth argument should point to an unsigned long int variable. These option bits are those specified in the call to pcre_compile(), modified by any top-level option settings within the pattern itself. - A pattern is automatically anchored by PCRE if all of its top-level + A pattern is automatically anchored by PCRE if all of its top-level alternatives begin with one of the following: ^ unless PCRE_MULTILINE is set @@ -1015,7 +1318,7 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_SIZE - Return the size of the compiled pattern, that is, the value that was + Return the size of the compiled pattern, that is, the value that was passed as the argument to pcre_malloc() when PCRE was getting memory in which to place the compiled data. The fourth argument should point to a size_t variable. @@ -1023,9 +1326,9 @@ INFORMATION ABOUT A PATTERN PCRE_INFO_STUDYSIZE Return the size of the data block pointed to by the study_data field in - a pcre_extra block. That is, it is the value that was passed to + a pcre_extra block. That is, it is the value that was passed to pcre_malloc() when PCRE was getting memory into which to place the data - created by pcre_study(). The fourth argument should point to a size_t + created by pcre_study(). The fourth argument should point to a size_t variable. @@ -1033,34 +1336,59 @@ OBSOLETE INFO FUNCTION int pcre_info(const pcre *code, int *optptr, int *firstcharptr); - The pcre_info() function is now obsolete because its interface is too - restrictive to return all the available data about a compiled pattern. - New programs should use pcre_fullinfo() instead. The yield of - pcre_info() is the number of capturing subpatterns, or one of the fol- + The pcre_info() function is now obsolete because its interface is too + restrictive to return all the available data about a compiled pattern. + New programs should use pcre_fullinfo() instead. The yield of + pcre_info() is the number of capturing subpatterns, or one of the fol- lowing negative numbers: PCRE_ERROR_NULL the argument code was NULL PCRE_ERROR_BADMAGIC the "magic number" was not found - If the optptr argument is not NULL, a copy of the options with which - the pattern was compiled is placed in the integer it points to (see + If the optptr argument is not NULL, a copy of the options with which + the pattern was compiled is placed in the integer it points to (see PCRE_INFO_OPTIONS above). - If the pattern is not anchored and the firstcharptr argument is not - NULL, it is used to pass back information about the first character of + If the pattern is not anchored and the firstcharptr argument is not + NULL, it is used to pass back information about the first character of any matched string (see PCRE_INFO_FIRSTBYTE above). -MATCHING A PATTERN +REFERENCE COUNTS + + int pcre_refcount(pcre *code, int adjust); + + The pcre_refcount() function is used to maintain a reference count in + the data block that contains a compiled pattern. It is provided for the + benefit of applications that operate in an object-oriented manner, + where different parts of the application may be using the same compiled + pattern, but you want to free the block when they are all done. + + When a pattern is compiled, the reference count field is initialized to + zero. It is changed only by calling this function, whose action is to + add the adjust value (which may be positive or negative) to it. The + yield of the function is the new value. However, the value of the count + is constrained to lie between 0 and 65535, inclusive. If the new value + is outside these limits, it is forced to the appropriate limit value. + + Except when it is zero, the reference count is not correctly preserved + if a pattern is compiled on one host and then transferred to a host + whose byte-order is different. (This seems a highly unlikely scenario.) + + +MATCHING A PATTERN: THE TRADITIONAL FUNCTION int pcre_exec(const pcre *code, const pcre_extra *extra, const char *subject, int length, int startoffset, int options, int *ovector, int ovecsize); - The function pcre_exec() is called to match a subject string against a - compiled pattern, which is passed in the code argument. If the pattern + The function pcre_exec() is called to match a subject string against a + compiled pattern, which is passed in the code argument. If the pattern has been studied, the result of the study should be passed in the extra - argument. + argument. This function is the main matching facility of the library, + and it operates in a Perl-like manner. For specialist use there is also + an alternative matching function, which is described below in the sec- + tion about the pcre_dfa_exec() function. In most applications, the pattern will have been compiled (and option- ally studied) in the same process that calls pcre_exec(). However, it @@ -1080,15 +1408,14 @@ MATCHING A PATTERN 0, /* start at offset 0 in the subject */ 0, /* default options */ ovector, /* vector of integers for substring information */ - 30); /* number of elements in the vector (NOT size in - bytes) */ + 30); /* number of elements (NOT size in bytes) */ Extra data for pcre_exec() - If the extra argument is not NULL, it must point to a pcre_extra data - block. The pcre_study() function returns such a block (when it doesn't - return NULL), but you can also create one for yourself, and pass addi- - tional information in it. The fields in a pcre_extra block are as fol- + If the extra argument is not NULL, it must point to a pcre_extra data + block. The pcre_study() function returns such a block (when it doesn't + return NULL), but you can also create one for yourself, and pass addi- + tional information in it. The fields in a pcre_extra block are as fol- lows: unsigned long int flags; @@ -1097,7 +1424,7 @@ MATCHING A PATTERN void *callout_data; const unsigned char *tables; - The flags field is a bitmap that specifies which of the other fields + The flags field is a bitmap that specifies which of the other fields are set. The flag bits are: PCRE_EXTRA_STUDY_DATA @@ -1105,229 +1432,229 @@ MATCHING A PATTERN PCRE_EXTRA_CALLOUT_DATA PCRE_EXTRA_TABLES - Other flag bits should be set to zero. The study_data field is set in - the pcre_extra block that is returned by pcre_study(), together with + Other flag bits should be set to zero. The study_data field is set in + the pcre_extra block that is returned by pcre_study(), together with the appropriate flag bit. You should not set this yourself, but you may - add to the block by setting the other fields and their corresponding + add to the block by setting the other fields and their corresponding flag bits. The match_limit field provides a means of preventing PCRE from using up - a vast amount of resources when running patterns that are not going to - match, but which have a very large number of possibilities in their - search trees. The classic example is the use of nested unlimited + a vast amount of resources when running patterns that are not going to + match, but which have a very large number of possibilities in their + search trees. The classic example is the use of nested unlimited repeats. - Internally, PCRE uses a function called match() which it calls repeat- - edly (sometimes recursively). The limit is imposed on the number of - times this function is called during a match, which has the effect of - limiting the amount of recursion and backtracking that can take place. + Internally, PCRE uses a function called match() which it calls repeat- + edly (sometimes recursively). The limit is imposed on the number of + times this function is called during a match, which has the effect of + limiting the amount of recursion and backtracking that can take place. For patterns that are not anchored, the count starts from zero for each position in the subject string. - The default limit for the library can be set when PCRE is built; the - default default is 10 million, which handles all but the most extreme - cases. You can reduce the default by suppling pcre_exec() with a - pcre_extra block in which match_limit is set to a smaller value, and - PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is + The default limit for the library can be set when PCRE is built; the + default default is 10 million, which handles all but the most extreme + cases. You can reduce the default by suppling pcre_exec() with a + pcre_extra block in which match_limit is set to a smaller value, and + PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT. - The pcre_callout field is used in conjunction with the "callout" fea- + The pcre_callout field is used in conjunction with the "callout" fea- ture, which is described in the pcrecallout documentation. - The tables field is used to pass a character tables pointer to - pcre_exec(); this overrides the value that is stored with the compiled - pattern. A non-NULL value is stored with the compiled pattern only if - custom tables were supplied to pcre_compile() via its tableptr argu- + The tables field is used to pass a character tables pointer to + pcre_exec(); this overrides the value that is stored with the compiled + pattern. A non-NULL value is stored with the compiled pattern only if + custom tables were supplied to pcre_compile() via its tableptr argu- ment. If NULL is passed to pcre_exec() using this mechanism, it forces - PCRE's internal tables to be used. This facility is helpful when re- - using patterns that have been saved after compiling with an external - set of tables, because the external tables might be at a different - address when pcre_exec() is called. See the pcreprecompile documenta- + PCRE's internal tables to be used. This facility is helpful when re- + using patterns that have been saved after compiling with an external + set of tables, because the external tables might be at a different + address when pcre_exec() is called. See the pcreprecompile documenta- tion for a discussion of saving compiled patterns for later use. Option bits for pcre_exec() - The unused bits of the options argument for pcre_exec() must be zero. - The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, + The unused bits of the options argument for pcre_exec() must be zero. + The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and PCRE_PARTIAL. PCRE_ANCHORED - The PCRE_ANCHORED option limits pcre_exec() to matching at the first - matching position. If a pattern was compiled with PCRE_ANCHORED, or - turned out to be anchored by virtue of its contents, it cannot be made + The PCRE_ANCHORED option limits pcre_exec() to matching at the first + matching position. If a pattern was compiled with PCRE_ANCHORED, or + turned out to be anchored by virtue of its contents, it cannot be made unachored at matching time. PCRE_NOTBOL This option specifies that first character of the subject string is not - the beginning of a line, so the circumflex metacharacter should not - match before it. Setting this without PCRE_MULTILINE (at compile time) - causes circumflex never to match. This option affects only the - behaviour of the circumflex metacharacter. It does not affect \A. + the beginning of a line, so the circumflex metacharacter should not + match before it. Setting this without PCRE_MULTILINE (at compile time) + causes circumflex never to match. This option affects only the behav- + iour of the circumflex metacharacter. It does not affect \A. PCRE_NOTEOL This option specifies that the end of the subject string is not the end - of a line, so the dollar metacharacter should not match it nor (except - in multiline mode) a newline immediately before it. Setting this with- + of a line, so the dollar metacharacter should not match it nor (except + in multiline mode) a newline immediately before it. Setting this with- out PCRE_MULTILINE (at compile time) causes dollar never to match. This - option affects only the behaviour of the dollar metacharacter. It does + option affects only the behaviour of the dollar metacharacter. It does not affect \Z or \z. PCRE_NOTEMPTY An empty string is not considered to be a valid match if this option is - set. If there are alternatives in the pattern, they are tried. If all - the alternatives match the empty string, the entire match fails. For + set. If there are alternatives in the pattern, they are tried. If all + the alternatives match the empty string, the entire match fails. For example, if the pattern a?b? - is applied to a string not beginning with "a" or "b", it matches the - empty string at the start of the subject. With PCRE_NOTEMPTY set, this + is applied to a string not beginning with "a" or "b", it matches the + empty string at the start of the subject. With PCRE_NOTEMPTY set, this match is not valid, so PCRE searches further into the string for occur- rences of "a" or "b". Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe- - cial case of a pattern match of the empty string within its split() - function, and when using the /g modifier. It is possible to emulate + cial case of a pattern match of the empty string within its split() + function, and when using the /g modifier. It is possible to emulate Perl's behaviour after matching a null string by first trying the match again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then - if that fails by advancing the starting offset (see below) and trying + if that fails by advancing the starting offset (see below) and trying an ordinary match again. There is some code that demonstrates how to do this in the pcredemo.c sample program. PCRE_NO_UTF8_CHECK When PCRE_UTF8 is set at compile time, the validity of the subject as a - UTF-8 string is automatically checked when pcre_exec() is subsequently - called. The value of startoffset is also checked to ensure that it - points to the start of a UTF-8 character. If an invalid UTF-8 sequence + UTF-8 string is automatically checked when pcre_exec() is subsequently + called. The value of startoffset is also checked to ensure that it + points to the start of a UTF-8 character. If an invalid UTF-8 sequence of bytes is found, pcre_exec() returns the error PCRE_ERROR_BADUTF8. If - startoffset contains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is + startoffset contains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is returned. - If you already know that your subject is valid, and you want to skip - these checks for performance reasons, you can set the - PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to - do this for the second and subsequent calls to pcre_exec() if you are - making repeated calls to find all the matches in a single subject - string. However, you should be sure that the value of startoffset - points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is - set, the effect of passing an invalid UTF-8 string as a subject, or a - value of startoffset that does not point to the start of a UTF-8 char- + If you already know that your subject is valid, and you want to skip + these checks for performance reasons, you can set the + PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to + do this for the second and subsequent calls to pcre_exec() if you are + making repeated calls to find all the matches in a single subject + string. However, you should be sure that the value of startoffset + points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is + set, the effect of passing an invalid UTF-8 string as a subject, or a + value of startoffset that does not point to the start of a UTF-8 char- acter, is undefined. Your program may crash. PCRE_PARTIAL - This option turns on the partial matching feature. If the subject - string fails to match the pattern, but at some point during the match- - ing process the end of the subject was reached (that is, the subject - partially matches the pattern and the failure to match occurred only - because there were not enough subject characters), pcre_exec() returns - PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is - used, there are restrictions on what may appear in the pattern. These + This option turns on the partial matching feature. If the subject + string fails to match the pattern, but at some point during the match- + ing process the end of the subject was reached (that is, the subject + partially matches the pattern and the failure to match occurred only + because there were not enough subject characters), pcre_exec() returns + PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is + used, there are restrictions on what may appear in the pattern. These are discussed in the pcrepartial documentation. The string to be matched by pcre_exec() - The subject string is passed to pcre_exec() as a pointer in subject, a - length in length, and a starting byte offset in startoffset. In UTF-8 - mode, the byte offset must point to the start of a UTF-8 character. - Unlike the pattern string, the subject may contain binary zero bytes. - When the starting offset is zero, the search for a match starts at the + The subject string is passed to pcre_exec() as a pointer in subject, a + length in length, and a starting byte offset in startoffset. In UTF-8 + mode, the byte offset must point to the start of a UTF-8 character. + Unlike the pattern string, the subject may contain binary zero bytes. + When the starting offset is zero, the search for a match starts at the beginning of the subject, and this is by far the most common case. - A non-zero starting offset is useful when searching for another match - in the same subject by calling pcre_exec() again after a previous suc- - cess. Setting startoffset differs from just passing over a shortened - string and setting PCRE_NOTBOL in the case of a pattern that begins + A non-zero starting offset is useful when searching for another match + in the same subject by calling pcre_exec() again after a previous suc- + cess. Setting startoffset differs from just passing over a shortened + string and setting PCRE_NOTBOL in the case of a pattern that begins with any kind of lookbehind. For example, consider the pattern \Biss\B - which finds occurrences of "iss" in the middle of words. (\B matches - only if the current position in the subject is not a word boundary.) - When applied to the string "Mississipi" the first call to pcre_exec() - finds the first occurrence. If pcre_exec() is called again with just - the remainder of the subject, namely "issipi", it does not match, + which finds occurrences of "iss" in the middle of words. (\B matches + only if the current position in the subject is not a word boundary.) + When applied to the string "Mississipi" the first call to pcre_exec() + finds the first occurrence. If pcre_exec() is called again with just + the remainder of the subject, namely "issipi", it does not match, because \B is always false at the start of the subject, which is deemed - to be a word boundary. However, if pcre_exec() is passed the entire + to be a word boundary. However, if pcre_exec() is passed the entire string again, but with startoffset set to 4, it finds the second occur- - rence of "iss" because it is able to look behind the starting point to + rence of "iss" because it is able to look behind the starting point to discover that it is preceded by a letter. - If a non-zero starting offset is passed when the pattern is anchored, + If a non-zero starting offset is passed when the pattern is anchored, one attempt to match at the given offset is made. This can only succeed - if the pattern does not require the match to be at the start of the + if the pattern does not require the match to be at the start of the subject. How pcre_exec() returns captured substrings - In general, a pattern matches a certain portion of the subject, and in - addition, further substrings from the subject may be picked out by - parts of the pattern. Following the usage in Jeffrey Friedl's book, - this is called "capturing" in what follows, and the phrase "capturing - subpattern" is used for a fragment of a pattern that picks out a sub- - string. PCRE supports several other kinds of parenthesized subpattern + In general, a pattern matches a certain portion of the subject, and in + addition, further substrings from the subject may be picked out by + parts of the pattern. Following the usage in Jeffrey Friedl's book, + this is called "capturing" in what follows, and the phrase "capturing + subpattern" is used for a fragment of a pattern that picks out a sub- + string. PCRE supports several other kinds of parenthesized subpattern that do not cause substrings to be captured. - Captured substrings are returned to the caller via a vector of integer - offsets whose address is passed in ovector. The number of elements in - the vector is passed in ovecsize, which must be a non-negative number. + Captured substrings are returned to the caller via a vector of integer + offsets whose address is passed in ovector. The number of elements in + the vector is passed in ovecsize, which must be a non-negative number. Note: this argument is NOT the size of ovector in bytes. - The first two-thirds of the vector is used to pass back captured sub- - strings, each substring using a pair of integers. The remaining third - of the vector is used as workspace by pcre_exec() while matching cap- - turing subpatterns, and is not available for passing back information. - The length passed in ovecsize should always be a multiple of three. If + The first two-thirds of the vector is used to pass back captured sub- + strings, each substring using a pair of integers. The remaining third + of the vector is used as workspace by pcre_exec() while matching cap- + turing subpatterns, and is not available for passing back information. + The length passed in ovecsize should always be a multiple of three. If it is not, it is rounded down. - When a match is successful, information about captured substrings is - returned in pairs of integers, starting at the beginning of ovector, - and continuing up to two-thirds of its length at the most. The first + When a match is successful, information about captured substrings is + returned in pairs of integers, starting at the beginning of ovector, + and continuing up to two-thirds of its length at the most. The first element of a pair is set to the offset of the first character in a sub- - string, and the second is set to the offset of the first character - after the end of a substring. The first pair, ovector[0] and ovec- - tor[1], identify the portion of the subject string matched by the - entire pattern. The next pair is used for the first capturing subpat- - tern, and so on. The value returned by pcre_exec() is the number of - pairs that have been set. If there are no capturing subpatterns, the - return value from a successful match is 1, indicating that just the + string, and the second is set to the offset of the first character + after the end of a substring. The first pair, ovector[0] and ovec- + tor[1], identify the portion of the subject string matched by the + entire pattern. The next pair is used for the first capturing subpat- + tern, and so on. The value returned by pcre_exec() is the number of + pairs that have been set. If there are no capturing subpatterns, the + return value from a successful match is 1, indicating that just the first pair of offsets has been set. - Some convenience functions are provided for extracting the captured - substrings as separate strings. These are described in the following + Some convenience functions are provided for extracting the captured + substrings as separate strings. These are described in the following section. - It is possible for an capturing subpattern number n+1 to match some - part of the subject when subpattern n has not been used at all. For + It is possible for an capturing subpattern number n+1 to match some + part of the subject when subpattern n has not been used at all. For example, if the string "abc" is matched against the pattern (a|(z))(bc) - subpatterns 1 and 3 are matched, but 2 is not. When this happens, both + subpatterns 1 and 3 are matched, but 2 is not. When this happens, both offset values corresponding to the unused subpattern are set to -1. If a capturing subpattern is matched repeatedly, it is the last portion of the string that it matched that is returned. - If the vector is too small to hold all the captured substring offsets, + If the vector is too small to hold all the captured substring offsets, it is used as far as possible (up to two-thirds of its length), and the - function returns a value of zero. In particular, if the substring off- + function returns a value of zero. In particular, if the substring off- sets are not of interest, pcre_exec() may be called with ovector passed - as NULL and ovecsize as zero. However, if the pattern contains back - references and the ovector is not big enough to remember the related - substrings, PCRE has to get additional memory for use during matching. + as NULL and ovecsize as zero. However, if the pattern contains back + references and the ovector is not big enough to remember the related + substrings, PCRE has to get additional memory for use during matching. Thus it is usually advisable to supply an ovector. - Note that pcre_info() can be used to find out how many capturing sub- + Note that pcre_info() can be used to find out how many capturing sub- patterns there are in a compiled pattern. The smallest size for ovector - that will allow for n captured substrings, in addition to the offsets + that will allow for n captured substrings, in addition to the offsets of the substring matched by the whole pattern, is (n+1)*3. Return values from pcre_exec() - If pcre_exec() fails, it returns a negative number. The following are + If pcre_exec() fails, it returns a negative number. The following are defined in the header file: PCRE_ERROR_NOMATCH (-1) @@ -1336,7 +1663,7 @@ MATCHING A PATTERN PCRE_ERROR_NULL (-2) - Either code or subject was passed as NULL, or ovector was NULL and + Either code or subject was passed as NULL, or ovector was NULL and ovecsize was not zero. PCRE_ERROR_BADOPTION (-3) @@ -1345,74 +1672,74 @@ MATCHING A PATTERN PCRE_ERROR_BADMAGIC (-4) - PCRE stores a 4-byte "magic number" at the start of the compiled code, + PCRE stores a 4-byte "magic number" at the start of the compiled code, to catch the case when it is passed a junk pointer and to detect when a pattern that was compiled in an environment of one endianness is run in - an environment with the other endianness. This is the error that PCRE + an environment with the other endianness. This is the error that PCRE gives when the magic number is not present. PCRE_ERROR_UNKNOWN_NODE (-5) While running the pattern match, an unknown item was encountered in the - compiled pattern. This error could be caused by a bug in PCRE or by + compiled pattern. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. PCRE_ERROR_NOMEMORY (-6) - If a pattern contains back references, but the ovector that is passed + If a pattern contains back references, but the ovector that is passed to pcre_exec() is not big enough to remember the referenced substrings, - PCRE gets a block of memory at the start of matching to use for this - purpose. If the call via pcre_malloc() fails, this error is given. The + PCRE gets a block of memory at the start of matching to use for this + purpose. If the call via pcre_malloc() fails, this error is given. The memory is automatically freed at the end of matching. PCRE_ERROR_NOSUBSTRING (-7) - This error is used by the pcre_copy_substring(), pcre_get_substring(), + This error is used by the pcre_copy_substring(), pcre_get_substring(), and pcre_get_substring_list() functions (see below). It is never returned by pcre_exec(). PCRE_ERROR_MATCHLIMIT (-8) - The recursion and backtracking limit, as specified by the match_limit - field in a pcre_extra structure (or defaulted) was reached. See the + The recursion and backtracking limit, as specified by the match_limit + field in a pcre_extra structure (or defaulted) was reached. See the description above. PCRE_ERROR_CALLOUT (-9) This error is never generated by pcre_exec() itself. It is provided for - use by callout functions that want to yield a distinctive error code. + use by callout functions that want to yield a distinctive error code. See the pcrecallout documentation for details. PCRE_ERROR_BADUTF8 (-10) - A string that contains an invalid UTF-8 byte sequence was passed as a + A string that contains an invalid UTF-8 byte sequence was passed as a subject. PCRE_ERROR_BADUTF8_OFFSET (-11) The UTF-8 byte sequence that was passed as a subject was valid, but the - value of startoffset did not point to the beginning of a UTF-8 charac- + value of startoffset did not point to the beginning of a UTF-8 charac- ter. - PCRE_ERROR_PARTIAL (-12) + PCRE_ERROR_PARTIAL (-12) - The subject string did not match, but it did match partially. See the + The subject string did not match, but it did match partially. See the pcrepartial documentation for details of partial matching. - PCRE_ERROR_BAD_PARTIAL (-13) + PCRE_ERROR_BADPARTIAL (-13) - The PCRE_PARTIAL option was used with a compiled pattern containing - items that are not supported for partial matching. See the pcrepartial + The PCRE_PARTIAL option was used with a compiled pattern containing + items that are not supported for partial matching. See the pcrepartial documentation for details of partial matching. - PCRE_ERROR_INTERNAL (-14) + PCRE_ERROR_INTERNAL (-14) - An unexpected internal error has occurred. This error could be caused + An unexpected internal error has occurred. This error could be caused by a bug in PCRE or by overwriting of the compiled pattern. - PCRE_ERROR_BADCOUNT (-15) + PCRE_ERROR_BADCOUNT (-15) - This error is given if the value of the ovecsize argument is negative. + This error is given if the value of the ovecsize argument is negative. EXTRACTING CAPTURED SUBSTRINGS BY NUMBER @@ -1428,72 +1755,72 @@ EXTRACTING CAPTURED SUBSTRINGS BY NUMBER int pcre_get_substring_list(const char *subject, int *ovector, int stringcount, const char ***listptr); - Captured substrings can be accessed directly by using the offsets - returned by pcre_exec() in ovector. For convenience, the functions + Captured substrings can be accessed directly by using the offsets + returned by pcre_exec() in ovector. For convenience, the functions pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub- - string_list() are provided for extracting captured substrings as new, - separate, zero-terminated strings. These functions identify substrings - by number. The next section describes functions for extracting named - substrings. A substring that contains a binary zero is correctly - extracted and has a further zero added on the end, but the result is + string_list() are provided for extracting captured substrings as new, + separate, zero-terminated strings. These functions identify substrings + by number. The next section describes functions for extracting named + substrings. A substring that contains a binary zero is correctly + extracted and has a further zero added on the end, but the result is not, of course, a C string. - The first three arguments are the same for all three of these func- - tions: subject is the subject string that has just been successfully + The first three arguments are the same for all three of these func- + tions: subject is the subject string that has just been successfully matched, ovector is a pointer to the vector of integer offsets that was passed to pcre_exec(), and stringcount is the number of substrings that - were captured by the match, including the substring that matched the + were captured by the match, including the substring that matched the entire regular expression. This is the value returned by pcre_exec() if - it is greater than zero. If pcre_exec() returned zero, indicating that - it ran out of space in ovector, the value passed as stringcount should + it is greater than zero. If pcre_exec() returned zero, indicating that + it ran out of space in ovector, the value passed as stringcount should be the number of elements in the vector divided by three. - The functions pcre_copy_substring() and pcre_get_substring() extract a - single substring, whose number is given as stringnumber. A value of - zero extracts the substring that matched the entire pattern, whereas - higher values extract the captured substrings. For pcre_copy_sub- - string(), the string is placed in buffer, whose length is given by - buffersize, while for pcre_get_substring() a new block of memory is - obtained via pcre_malloc, and its address is returned via stringptr. - The yield of the function is the length of the string, not including + The functions pcre_copy_substring() and pcre_get_substring() extract a + single substring, whose number is given as stringnumber. A value of + zero extracts the substring that matched the entire pattern, whereas + higher values extract the captured substrings. For pcre_copy_sub- + string(), the string is placed in buffer, whose length is given by + buffersize, while for pcre_get_substring() a new block of memory is + obtained via pcre_malloc, and its address is returned via stringptr. + The yield of the function is the length of the string, not including the terminating zero, or one of PCRE_ERROR_NOMEMORY (-6) - The buffer was too small for pcre_copy_substring(), or the attempt to + The buffer was too small for pcre_copy_substring(), or the attempt to get memory failed for pcre_get_substring(). PCRE_ERROR_NOSUBSTRING (-7) There is no substring whose number is stringnumber. - The pcre_get_substring_list() function extracts all available sub- - strings and builds a list of pointers to them. All this is done in a + The pcre_get_substring_list() function extracts all available sub- + strings and builds a list of pointers to them. All this is done in a single block of memory that is obtained via pcre_malloc. The address of - the memory block is returned via listptr, which is also the start of - the list of string pointers. The end of the list is marked by a NULL + the memory block is returned via listptr, which is also the start of + the list of string pointers. The end of the list is marked by a NULL pointer. The yield of the function is zero if all went well, or PCRE_ERROR_NOMEMORY (-6) if the attempt to get the memory block failed. - When any of these functions encounter a substring that is unset, which - can happen when capturing subpattern number n+1 matches some part of - the subject, but subpattern n has not been used at all, they return an + When any of these functions encounter a substring that is unset, which + can happen when capturing subpattern number n+1 matches some part of + the subject, but subpattern n has not been used at all, they return an empty string. This can be distinguished from a genuine zero-length sub- - string by inspecting the appropriate offset in ovector, which is nega- + string by inspecting the appropriate offset in ovector, which is nega- tive for unset substrings. - The two convenience functions pcre_free_substring() and pcre_free_sub- - string_list() can be used to free the memory returned by a previous + The two convenience functions pcre_free_substring() and pcre_free_sub- + string_list() can be used to free the memory returned by a previous call of pcre_get_substring() or pcre_get_substring_list(), respec- - tively. They do nothing more than call the function pointed to by - pcre_free, which of course could be called directly from a C program. - However, PCRE is used in some situations where it is linked via a spe- + tively. They do nothing more than call the function pointed to by + pcre_free, which of course could be called directly from a C program. + However, PCRE is used in some situations where it is linked via a spe- cial interface to another programming language which cannot use - pcre_free directly; it is for these cases that the functions are - provided. + pcre_free directly; it is for these cases that the functions are pro- + vided. EXTRACTING CAPTURED SUBSTRINGS BY NAME @@ -1511,47 +1838,219 @@ EXTRACTING CAPTURED SUBSTRINGS BY NAME int stringcount, const char *stringname, const char **stringptr); - To extract a substring by name, you first have to find associated num- + To extract a substring by name, you first have to find associated num- ber. For example, for this pattern - (a+)b(?<xxx>\d+)... + (a+)b(?P<xxx>\d+)... the number of the subpattern called "xxx" is 2. You can find the number from the name by calling pcre_get_stringnumber(). The first argument is - the compiled pattern, and the second is the name. The yield of the - function is the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if + the compiled pattern, and the second is the name. The yield of the + function is the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if there is no subpattern of that name. Given the number, you can extract the substring directly, or use one of the functions described in the previous section. For convenience, there are also two functions that do the whole job. - Most of the arguments of pcre_copy_named_substring() and - pcre_get_named_substring() are the same as those for the similarly - named functions that extract by number. As these are described in the - previous section, they are not re-described here. There are just two + Most of the arguments of pcre_copy_named_substring() and + pcre_get_named_substring() are the same as those for the similarly + named functions that extract by number. As these are described in the + previous section, they are not re-described here. There are just two differences: - First, instead of a substring number, a substring name is given. Sec- + First, instead of a substring number, a substring name is given. Sec- ond, there is an extra argument, given at the start, which is a pointer - to the compiled pattern. This is needed in order to gain access to the + to the compiled pattern. This is needed in order to gain access to the name-to-number translation table. - These functions call pcre_get_stringnumber(), and if it succeeds, they - then call pcre_copy_substring() or pcre_get_substring(), as appropri- + These functions call pcre_get_stringnumber(), and if it succeeds, they + then call pcre_copy_substring() or pcre_get_substring(), as appropri- ate. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +FINDING ALL POSSIBLE MATCHES + + The traditional matching function uses a similar algorithm to Perl, + which stops when it finds the first match, starting at a given point in + the subject. If you want to find all possible matches, or the longest + possible match, consider using the alternative matching function (see + below) instead. If you cannot use the alternative function, but still + need to find all possible matches, you can kludge it up by making use + of the callout facility, which is described in the pcrecallout documen- + tation. + + What you have to do is to insert a callout right at the end of the pat- + tern. When your callout function is called, extract and save the cur- + rent matched substring. Then return 1, which forces pcre_exec() to + backtrack and try other alternatives. Ultimately, when it runs out of + matches, pcre_exec() will yield PCRE_ERROR_NOMATCH. + + +MATCHING A PATTERN: THE ALTERNATIVE FUNCTION + + int pcre_dfa_exec(const pcre *code, const pcre_extra *extra, + const char *subject, int length, int startoffset, + int options, int *ovector, int ovecsize, + int *workspace, int wscount); + + The function pcre_dfa_exec() is called to match a subject string + against a compiled pattern, using a "DFA" matching algorithm. This has + different characteristics to the normal algorithm, and is not compati- + ble with Perl. Some of the features of PCRE patterns are not supported. + Nevertheless, there are times when this kind of matching can be useful. + For a discussion of the two matching algorithms, see the pcrematching + documentation. + + The arguments for the pcre_dfa_exec() function are the same as for + pcre_exec(), plus two extras. The ovector argument is used in a differ- + ent way, and this is described below. The other common arguments are + used in the same way as for pcre_exec(), so their description is not + repeated here. + + The two additional arguments provide workspace for the function. The + workspace vector should contain at least 20 elements. It is used for + keeping track of multiple paths through the pattern tree. More + workspace will be needed for patterns and subjects where there are a + lot of possible matches. + + Here is an example of a simple call to pcre_exec(): + + int rc; + int ovector[10]; + int wspace[20]; + rc = pcre_exec( + re, /* result of pcre_compile() */ + NULL, /* we didn't study the pattern */ + "some string", /* the subject string */ + 11, /* the length of the subject string */ + 0, /* start at offset 0 in the subject */ + 0, /* default options */ + ovector, /* vector of integers for substring information */ + 10, /* number of elements (NOT size in bytes) */ + wspace, /* working space vector */ + 20); /* number of elements (NOT size in bytes) */ + + Option bits for pcre_dfa_exec() + + The unused bits of the options argument for pcre_dfa_exec() must be + zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL, + PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL, + PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last three of + these are the same as for pcre_exec(), so their description is not + repeated here. + + PCRE_PARTIAL + + This has the same general effect as it does for pcre_exec(), but the + details are slightly different. When PCRE_PARTIAL is set for + pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into + PCRE_ERROR_PARTIAL if the end of the subject is reached, there have + been no complete matches, but there is still at least one matching pos- + sibility. The portion of the string that provided the partial match is + set as the first matching string. + + PCRE_DFA_SHORTEST + + Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to + stop as soon as it has found one match. Because of the way the DFA + algorithm works, this is necessarily the shortest possible match at the + first possible matching point in the subject string. + + PCRE_DFA_RESTART + + When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and + returns a partial match, it is possible to call it again, with addi- + tional subject characters, and have it continue with the same match. + The PCRE_DFA_RESTART option requests this action; when it is set, the + workspace and wscount options must reference the same vector as before + because data about the match so far is left in them after a partial + match. There is more discussion of this facility in the pcrepartial + documentation. + + Successful returns from pcre_dfa_exec() + + When pcre_dfa_exec() succeeds, it may have matched more than one sub- + string in the subject. Note, however, that all the matches from one run + of the function start at the same point in the subject. The shorter + matches are all initial substrings of the longer matches. For example, + if the pattern + + <.*> + + is matched against the string + + This is <something> <something else> <something further> no more + + the three matched strings are + + <something> + <something> <something else> + <something> <something else> <something further> + + On success, the yield of the function is a number greater than zero, + which is the number of matched substrings. The substrings themselves + are returned in ovector. Each string uses two elements; the first is + the offset to the start, and the second is the offset to the end. All + the strings have the same start offset. (Space could have been saved by + giving this only once, but it was decided to retain some compatibility + with the way pcre_exec() returns data, even though the meaning of the + strings is different.) + + The strings are returned in reverse order of length; that is, the long- + est matching string is given first. If there were too many matches to + fit into ovector, the yield of the function is zero, and the vector is + filled with the longest matches. + + Error returns from pcre_dfa_exec() + + The pcre_dfa_exec() function returns a negative number when it fails. + Many of the errors are the same as for pcre_exec(), and these are + described above. There are in addition the following errors that are + specific to pcre_dfa_exec(): + + PCRE_ERROR_DFA_UITEM (-16) + + This return is given if pcre_dfa_exec() encounters an item in the pat- + tern that it does not support, for instance, the use of \C or a back + reference. + + PCRE_ERROR_DFA_UCOND (-17) + + This return is given if pcre_dfa_exec() encounters a condition item in + a pattern that uses a back reference for the condition. This is not + supported. + + PCRE_ERROR_DFA_UMLIMIT (-18) + + This return is given if pcre_dfa_exec() is called with an extra block + that contains a setting of the match_limit field. This is not supported + (it is meaningless). + + PCRE_ERROR_DFA_WSSIZE (-19) + + This return is given if pcre_dfa_exec() runs out of space in the + workspace vector. + + PCRE_ERROR_DFA_RECURSE (-20) + + When a recursive subpattern is processed, the matching function calls + itself recursively, using private vectors for ovector and workspace. + This error is given if the output vector is not large enough. This + should be extremely rare, as a vector of size 1000 is used. + +Last updated: 16 May 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ + +PCRECALLOUT(3) PCRECALLOUT(3) NAME PCRE - Perl-compatible regular expressions + PCRE CALLOUTS int (*pcre_callout)(pcre_callout_block *); @@ -1606,9 +2105,10 @@ MISSING CALLOUTS THE CALLOUT INTERFACE During matching, when PCRE reaches a callout point, the external func- - tion defined by pcre_callout is called (if it is set). The only argu- - ment is a pointer to a pcre_callout block. This structure contains the - following fields: + tion defined by pcre_callout is called (if it is set). This applies to + both the pcre_exec() and the pcre_dfa_exec() matching functions. The + only argument to the callout function is a pointer to a pcre_callout + block. This structure contains the following fields: int version; int callout_number; @@ -1623,87 +2123,93 @@ THE CALLOUT INTERFACE int pattern_position; int next_item_length; - The version field is an integer containing the version number of the - block format. The initial version was 0; the current version is 1. The - version number will change again in future if additional fields are + The version field is an integer containing the version number of the + block format. The initial version was 0; the current version is 1. The + version number will change again in future if additional fields are added, but the intention is never to remove any of the existing fields. - The callout_number field contains the number of the callout, as com- - piled into the pattern (that is, the number after ?C for manual call- + The callout_number field contains the number of the callout, as com- + piled into the pattern (that is, the number after ?C for manual call- outs, and 255 for automatically generated callouts). - The offset_vector field is a pointer to the vector of offsets that was - passed by the caller to pcre_exec(). The contents can be inspected in - order to extract substrings that have been matched so far, in the same - way as for extracting substrings after a match has completed. + The offset_vector field is a pointer to the vector of offsets that was + passed by the caller to pcre_exec() or pcre_dfa_exec(). When + pcre_exec() is used, the contents can be inspected in order to extract + substrings that have been matched so far, in the same way as for + extracting substrings after a match has completed. For pcre_dfa_exec() + this field is not useful. The subject and subject_length fields contain copies of the values that were passed to pcre_exec(). - The start_match field contains the offset within the subject at which - the current match attempt started. If the pattern is not anchored, the + The start_match field contains the offset within the subject at which + the current match attempt started. If the pattern is not anchored, the callout function may be called several times from the same point in the pattern for different starting points in the subject. - The current_position field contains the offset within the subject of + The current_position field contains the offset within the subject of the current match pointer. - The capture_top field contains one more than the number of the highest - numbered captured substring so far. If no substrings have been cap- - tured, the value of capture_top is one. - - The capture_last field contains the number of the most recently cap- - tured substring. If no substrings have been captured, its value is -1. - - The callout_data field contains a value that is passed to pcre_exec() - by the caller specifically so that it can be passed back in callouts. - It is passed in the pcre_callout field of the pcre_extra data struc- - ture. If no such data was passed, the value of callout_data in a - pcre_callout block is NULL. There is a description of the pcre_extra + When the pcre_exec() function is used, the capture_top field contains + one more than the number of the highest numbered captured substring so + far. If no substrings have been captured, the value of capture_top is + one. This is always the case when pcre_dfa_exec() is used, because it + does not support captured substrings. + + The capture_last field contains the number of the most recently cap- + tured substring. If no substrings have been captured, its value is -1. + This is always the case when pcre_dfa_exec() is used. + + The callout_data field contains a value that is passed to pcre_exec() + or pcre_dfa_exec() specifically so that it can be passed back in call- + outs. It is passed in the pcre_callout field of the pcre_extra data + structure. If no such data was passed, the value of callout_data in a + pcre_callout block is NULL. There is a description of the pcre_extra structure in the pcreapi documentation. - The pattern_position field is present from version 1 of the pcre_call- + The pattern_position field is present from version 1 of the pcre_call- out structure. It contains the offset to the next item to be matched in the pattern string. - The next_item_length field is present from version 1 of the pcre_call- + The next_item_length field is present from version 1 of the pcre_call- out structure. It contains the length of the next item to be matched in - the pattern string. When the callout immediately precedes an alterna- - tion bar, a closing parenthesis, or the end of the pattern, the length - is zero. When the callout precedes an opening parenthesis, the length + the pattern string. When the callout immediately precedes an alterna- + tion bar, a closing parenthesis, or the end of the pattern, the length + is zero. When the callout precedes an opening parenthesis, the length is that of the entire subpattern. - The pattern_position and next_item_length fields are intended to help - in distinguishing between different automatic callouts, which all have + The pattern_position and next_item_length fields are intended to help + in distinguishing between different automatic callouts, which all have the same callout number. However, they are set for all callouts. RETURN VALUES - The external callout function returns an integer to PCRE. If the value - is zero, matching proceeds as normal. If the value is greater than - zero, matching fails at the current point, but backtracking to test - other matching possibilities goes ahead, just as if a lookahead asser- - tion had failed. If the value is less than zero, the match is aban- - doned, and pcre_exec() returns the negative value. + The external callout function returns an integer to PCRE. If the value + is zero, matching proceeds as normal. If the value is greater than + zero, matching fails at the current point, but the testing of other + matching possibilities goes ahead, just as if a lookahead assertion had + failed. If the value is less than zero, the match is abandoned, and + pcre_exec() (or pcre_dfa_exec()) returns the negative value. - Negative values should normally be chosen from the set of + Negative values should normally be chosen from the set of PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan- - dard "no match" failure. The error number PCRE_ERROR_CALLOUT is - reserved for use by callout functions; it will never be used by PCRE + dard "no match" failure. The error number PCRE_ERROR_CALLOUT is + reserved for use by callout functions; it will never be used by PCRE itself. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCRECOMPAT(3) PCRECOMPAT(3) NAME PCRE - Perl-compatible regular expressions + DIFFERENCES BETWEEN PCRE AND PERL This document describes the differences in the ways that PCRE and Perl @@ -1808,17 +2314,21 @@ DIFFERENCES BETWEEN PCRE AND PERL (m) Patterns compiled by PCRE can be saved and re-used at a later time, even on different hosts that have the other endianness. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ + (n) The alternative matching function (pcre_dfa_exec()) matches in a + different way and is not Perl-compatible. + +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCREPATTERN(3) PCREPATTERN(3) NAME PCRE - Perl-compatible regular expressions + PCRE REGULAR EXPRESSION DETAILS The syntax and semantics of the regular expressions supported by PCRE @@ -1836,6 +2346,14 @@ PCRE REGULAR EXPRESSION DETAILS of UTF-8 features in the section on UTF-8 support in the main pcre page. + The remainder of this document discusses the patterns that are sup- + ported by PCRE when its main matching function, pcre_exec(), is used. + From release 6.0, PCRE offers a second matching function, + pcre_dfa_exec(), which matches using a different algorithm that is not + Perl-compatible. The advantages and disadvantages of the alternative + function, and how it differs from the normal function, are discussed in + the pcrematching page. + A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a @@ -1843,15 +2361,24 @@ PCRE REGULAR EXPRESSION DETAILS The quick brown fox - matches a portion of a subject string that is identical to itself. The - power of regular expressions comes from the ability to include alterna- - tives and repetitions in the pattern. These are encoded in the pattern - by the use of metacharacters, which do not stand for themselves but - instead are interpreted in some special way. - - There are two different sets of metacharacters: those that are recog- - nized anywhere in the pattern except within square brackets, and those - that are recognized in square brackets. Outside square brackets, the + matches a portion of a subject string that is identical to itself. When + caseless matching is specified (the PCRE_CASELESS option), letters are + matched independently of case. In UTF-8 mode, PCRE always understands + the concept of case for characters whose values are less than 128, so + caseless matching is always possible. For characters with higher val- + ues, the concept of case is supported if PCRE is compiled with Unicode + property support, but not otherwise. If you want to use caseless + matching for characters 128 and above, you must ensure that PCRE is + compiled with Unicode property support as well as with UTF-8 support. + + The power of regular expressions comes from the ability to include + alternatives and repetitions in the pattern. These are encoded in the + pattern by the use of metacharacters, which do not stand for themselves + but instead are interpreted in some special way. + + There are two different sets of metacharacters: those that are recog- + nized anywhere in the pattern except within square brackets, and those + that are recognized in square brackets. Outside square brackets, the metacharacters are as follows: \ general escape character with several uses @@ -1870,7 +2397,7 @@ PCRE REGULAR EXPRESSION DETAILS also "possessive quantifier" { start min/max quantifier - Part of a pattern that is in square brackets is called a "character + Part of a pattern that is in square brackets is called a "character class". In a character class the only metacharacters are: \ general escape character @@ -1880,33 +2407,33 @@ PCRE REGULAR EXPRESSION DETAILS syntax) ] terminates the character class - The following sections describe the use of each of the metacharacters. + The following sections describe the use of each of the metacharacters. BACKSLASH The backslash character has several uses. Firstly, if it is followed by - a non-alphanumeric character, it takes away any special meaning that - character may have. This use of backslash as an escape character + a non-alphanumeric character, it takes away any special meaning that + character may have. This use of backslash as an escape character applies both inside and outside character classes. - For example, if you want to match a * character, you write \* in the - pattern. This escaping action applies whether or not the following - character would otherwise be interpreted as a metacharacter, so it is - always safe to precede a non-alphanumeric with backslash to specify - that it stands for itself. In particular, if you want to match a back- + For example, if you want to match a * character, you write \* in the + pattern. This escaping action applies whether or not the following + character would otherwise be interpreted as a metacharacter, so it is + always safe to precede a non-alphanumeric with backslash to specify + that it stands for itself. In particular, if you want to match a back- slash, you write \\. - If a pattern is compiled with the PCRE_EXTENDED option, whitespace in - the pattern (other than in a character class) and characters between a + If a pattern is compiled with the PCRE_EXTENDED option, whitespace in + the pattern (other than in a character class) and characters between a # outside a character class and the next newline character are ignored. - An escaping backslash can be used to include a whitespace or # charac- + An escaping backslash can be used to include a whitespace or # charac- ter as part of the pattern. - If you want to remove the special meaning from a sequence of charac- - ters, you can do so by putting them between \Q and \E. This is differ- - ent from Perl in that $ and @ are handled as literals in \Q...\E - sequences in PCRE, whereas in Perl, $ and @ cause variable interpola- + If you want to remove the special meaning from a sequence of charac- + ters, you can do so by putting them between \Q and \E. This is differ- + ent from Perl in that $ and @ are handled as literals in \Q...\E + sequences in PCRE, whereas in Perl, $ and @ cause variable interpola- tion. Note the following examples: Pattern PCRE matches Perl matches @@ -1916,16 +2443,16 @@ BACKSLASH \Qabc\$xyz\E abc\$xyz abc\$xyz \Qabc\E\$\Qxyz\E abc$xyz abc$xyz - The \Q...\E sequence is recognized both inside and outside character + The \Q...\E sequence is recognized both inside and outside character classes. Non-printing characters A second use of backslash provides a way of encoding non-printing char- - acters in patterns in a visible manner. There is no restriction on the - appearance of non-printing characters, apart from the binary zero that - terminates a pattern, but when a pattern is being prepared by text - editing, it is usually easier to use one of the following escape + acters in patterns in a visible manner. There is no restriction on the + appearance of non-printing characters, apart from the binary zero that + terminates a pattern, but when a pattern is being prepared by text + editing, it is usually easier to use one of the following escape sequences than the binary character it represents: \a alarm, that is, the BEL character (hex 07) @@ -1939,44 +2466,44 @@ BACKSLASH \xhh character with hex code hh \x{hhh..} character with hex code hhh... (UTF-8 mode only) - The precise effect of \cx is as follows: if x is a lower case letter, - it is converted to upper case. Then bit 6 of the character (hex 40) is - inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; + The precise effect of \cx is as follows: if x is a lower case letter, + it is converted to upper case. Then bit 6 of the character (hex 40) is + inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; becomes hex 7B. - After \x, from zero to two hexadecimal digits are read (letters can be - in upper or lower case). In UTF-8 mode, any number of hexadecimal dig- - its may appear between \x{ and }, but the value of the character code - must be less than 2**31 (that is, the maximum hexadecimal value is - 7FFFFFFF). If characters other than hexadecimal digits appear between - \x{ and }, or if there is no terminating }, this form of escape is not - recognized. Instead, the initial \x will be interpreted as a basic hex- - adecimal escape, with no following digits, giving a character whose + After \x, from zero to two hexadecimal digits are read (letters can be + in upper or lower case). In UTF-8 mode, any number of hexadecimal dig- + its may appear between \x{ and }, but the value of the character code + must be less than 2**31 (that is, the maximum hexadecimal value is + 7FFFFFFF). If characters other than hexadecimal digits appear between + \x{ and }, or if there is no terminating }, this form of escape is not + recognized. Instead, the initial \x will be interpreted as a basic + hexadecimal escape, with no following digits, giving a character whose value is zero. Characters whose value is less than 256 can be defined by either of the - two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference - in the way they are handled. For example, \xdc is exactly the same as + two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference + in the way they are handled. For example, \xdc is exactly the same as \x{dc}. - After \0 up to two further octal digits are read. In both cases, if - there are fewer than two digits, just those that are present are used. - Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL - character (code value 7). Make sure you supply two digits after the - initial zero if the pattern character that follows is itself an octal + After \0 up to two further octal digits are read. In both cases, if + there are fewer than two digits, just those that are present are used. + Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL + character (code value 7). Make sure you supply two digits after the + initial zero if the pattern character that follows is itself an octal digit. The handling of a backslash followed by a digit other than 0 is compli- cated. Outside a character class, PCRE reads it and any following dig- - its as a decimal number. If the number is less than 10, or if there + its as a decimal number. If the number is less than 10, or if there have been at least that many previous capturing left parentheses in the - expression, the entire sequence is taken as a back reference. A - description of how this works is given later, following the discussion + expression, the entire sequence is taken as a back reference. A + description of how this works is given later, following the discussion of parenthesized subpatterns. - Inside a character class, or if the decimal number is greater than 9 - and there have not been that many capturing subpatterns, PCRE re-reads - up to three octal digits following the backslash, and generates a sin- + Inside a character class, or if the decimal number is greater than 9 + and there have not been that many capturing subpatterns, PCRE re-reads + up to three octal digits following the backslash, and generates a sin- gle byte from the least significant 8 bits of the value. Any subsequent digits stand for themselves. For example: @@ -1995,19 +2522,19 @@ BACKSLASH \81 is either a back reference, or a binary zero followed by the two characters "8" and "1" - Note that octal values of 100 or greater must not be introduced by a + Note that octal values of 100 or greater must not be introduced by a leading zero, because no more than three octal digits are ever read. - All the sequences that define a single byte value or a single UTF-8 + All the sequences that define a single byte value or a single UTF-8 character (in UTF-8 mode) can be used both inside and outside character - classes. In addition, inside a character class, the sequence \b is + classes. In addition, inside a character class, the sequence \b is interpreted as the backspace character (hex 08), and the sequence \X is - interpreted as the character "X". Outside a character class, these + interpreted as the character "X". Outside a character class, these sequences have different meanings (see below). Generic character types - The third use of backslash is for specifying generic character types. + The third use of backslash is for specifying generic character types. The following are always recognized: \d any decimal digit @@ -2018,48 +2545,48 @@ BACKSLASH \W any "non-word" character Each pair of escape sequences partitions the complete set of characters - into two disjoint sets. Any given character matches one, and only one, + into two disjoint sets. Any given character matches one, and only one, of each pair. These character type sequences can appear both inside and outside char- - acter classes. They each match one character of the appropriate type. - If the current matching point is at the end of the subject string, all + acter classes. They each match one character of the appropriate type. + If the current matching point is at the end of the subject string, all of them fail, since there is no character to match. - For compatibility with Perl, \s does not match the VT character (code - 11). This makes it different from the the POSIX "space" class. The \s + For compatibility with Perl, \s does not match the VT character (code + 11). This makes it different from the the POSIX "space" class. The \s characters are HT (9), LF (10), FF (12), CR (13), and space (32). A "word" character is an underscore or any character less than 256 that - is a letter or digit. The definition of letters and digits is con- - trolled by PCRE's low-valued character tables, and may vary if locale- - specific matching is taking place (see "Locale support" in the pcreapi - page). For example, in the "fr_FR" (French) locale, some character - codes greater than 128 are used for accented letters, and these are + is a letter or digit. The definition of letters and digits is con- + trolled by PCRE's low-valued character tables, and may vary if locale- + specific matching is taking place (see "Locale support" in the pcreapi + page). For example, in the "fr_FR" (French) locale, some character + codes greater than 128 are used for accented letters, and these are matched by \w. - In UTF-8 mode, characters with values greater than 128 never match \d, + In UTF-8 mode, characters with values greater than 128 never match \d, \s, or \w, and always match \D, \S, and \W. This is true even when Uni- code character property support is available. Unicode character properties When PCRE is built with Unicode character property support, three addi- - tional escape sequences to match generic character types are available + tional escape sequences to match generic character types are available when UTF-8 mode is selected. They are: \p{xx} a character with the xx property \P{xx} a character without the xx property \X an extended Unicode sequence - The property names represented by xx above are limited to the Unicode - general category properties. Each character has exactly one such prop- - erty, specified by a two-letter abbreviation. For compatibility with - Perl, negation can be specified by including a circumflex between the - opening brace and the property name. For example, \p{^Lu} is the same + The property names represented by xx above are limited to the Unicode + general category properties. Each character has exactly one such prop- + erty, specified by a two-letter abbreviation. For compatibility with + Perl, negation can be specified by including a circumflex between the + opening brace and the property name. For example, \p{^Lu} is the same as \P{Lu}. - If only one letter is specified with \p or \P, it includes all the + If only one letter is specified with \p or \P, it includes all the properties that start with that letter. In this case, in the absence of negation, the curly brackets in the escape sequence are optional; these two examples have the same effect: @@ -2113,33 +2640,33 @@ BACKSLASH Zp Paragraph separator Zs Space separator - Extended properties such as "Greek" or "InMusicalSymbols" are not sup- + Extended properties such as "Greek" or "InMusicalSymbols" are not sup- ported by PCRE. - Specifying caseless matching does not affect these escape sequences. + Specifying caseless matching does not affect these escape sequences. For example, \p{Lu} always matches only upper case letters. - The \X escape matches any number of Unicode characters that form an + The \X escape matches any number of Unicode characters that form an extended Unicode sequence. \X is equivalent to (?>\PM\pM*) - That is, it matches a character without the "mark" property, followed - by zero or more characters with the "mark" property, and treats the - sequence as an atomic group (see below). Characters with the "mark" + That is, it matches a character without the "mark" property, followed + by zero or more characters with the "mark" property, and treats the + sequence as an atomic group (see below). Characters with the "mark" property are typically accents that affect the preceding character. - Matching characters by Unicode property is not fast, because PCRE has - to search a structure that contains data for over fifteen thousand + Matching characters by Unicode property is not fast, because PCRE has + to search a structure that contains data for over fifteen thousand characters. That is why the traditional escape sequences such as \d and \w do not use Unicode properties in PCRE. Simple assertions The fourth use of backslash is for certain simple assertions. An asser- - tion specifies a condition that has to be met at a particular point in - a match, without consuming any characters from the subject string. The - use of subpatterns for more complicated assertions is described below. + tion specifies a condition that has to be met at a particular point in + a match, without consuming any characters from the subject string. The + use of subpatterns for more complicated assertions is described below. The backslashed assertions are: \b matches at a word boundary @@ -2149,42 +2676,42 @@ BACKSLASH \z matches at end of subject \G matches at first matching position in subject - These assertions may not appear in character classes (but note that \b + These assertions may not appear in character classes (but note that \b has a different meaning, namely the backspace character, inside a char- acter class). - A word boundary is a position in the subject string where the current - character and the previous character do not both match \w or \W (i.e. - one matches \w and the other matches \W), or the start or end of the + A word boundary is a position in the subject string where the current + character and the previous character do not both match \w or \W (i.e. + one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively. - The \A, \Z, and \z assertions differ from the traditional circumflex + The \A, \Z, and \z assertions differ from the traditional circumflex and dollar (described in the next section) in that they only ever match - at the very start and end of the subject string, whatever options are - set. Thus, they are independent of multiline mode. These three asser- + at the very start and end of the subject string, whatever options are + set. Thus, they are independent of multiline mode. These three asser- tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which - affect only the behaviour of the circumflex and dollar metacharacters. - However, if the startoffset argument of pcre_exec() is non-zero, indi- + affect only the behaviour of the circumflex and dollar metacharacters. + However, if the startoffset argument of pcre_exec() is non-zero, indi- cating that matching is to start at a point other than the beginning of - the subject, \A can never match. The difference between \Z and \z is - that \Z matches before a newline that is the last character of the - string as well as at the end of the string, whereas \z matches only at + the subject, \A can never match. The difference between \Z and \z is + that \Z matches before a newline that is the last character of the + string as well as at the end of the string, whereas \z matches only at the end. - The \G assertion is true only when the current matching position is at - the start point of the match, as specified by the startoffset argument - of pcre_exec(). It differs from \A when the value of startoffset is - non-zero. By calling pcre_exec() multiple times with appropriate argu- + The \G assertion is true only when the current matching position is at + the start point of the match, as specified by the startoffset argument + of pcre_exec(). It differs from \A when the value of startoffset is + non-zero. By calling pcre_exec() multiple times with appropriate argu- ments, you can mimic Perl's /g option, and it is in this kind of imple- mentation where \G can be useful. - Note, however, that PCRE's interpretation of \G, as the start of the + Note, however, that PCRE's interpretation of \G, as the start of the current match, is subtly different from Perl's, which defines it as the - end of the previous match. In Perl, these can be different when the - previously matched string was empty. Because PCRE does just one match + end of the previous match. In Perl, these can be different when the + previously matched string was empty. Because PCRE does just one match at a time, it cannot reproduce this behaviour. - If all the alternatives of a pattern begin with \G, the expression is + If all the alternatives of a pattern begin with \G, the expression is anchored to the starting match position, and the "anchored" flag is set in the compiled regular expression. @@ -2192,73 +2719,73 @@ BACKSLASH CIRCUMFLEX AND DOLLAR Outside a character class, in the default matching mode, the circumflex - character is an assertion that is true only if the current matching - point is at the start of the subject string. If the startoffset argu- - ment of pcre_exec() is non-zero, circumflex can never match if the - PCRE_MULTILINE option is unset. Inside a character class, circumflex + character is an assertion that is true only if the current matching + point is at the start of the subject string. If the startoffset argu- + ment of pcre_exec() is non-zero, circumflex can never match if the + PCRE_MULTILINE option is unset. Inside a character class, circumflex has an entirely different meaning (see below). - Circumflex need not be the first character of the pattern if a number - of alternatives are involved, but it should be the first thing in each - alternative in which it appears if the pattern is ever to match that - branch. If all possible alternatives start with a circumflex, that is, - if the pattern is constrained to match only at the start of the sub- - ject, it is said to be an "anchored" pattern. (There are also other + Circumflex need not be the first character of the pattern if a number + of alternatives are involved, but it should be the first thing in each + alternative in which it appears if the pattern is ever to match that + branch. If all possible alternatives start with a circumflex, that is, + if the pattern is constrained to match only at the start of the sub- + ject, it is said to be an "anchored" pattern. (There are also other constructs that can cause a pattern to be anchored.) - A dollar character is an assertion that is true only if the current - matching point is at the end of the subject string, or immediately + A dollar character is an assertion that is true only if the current + matching point is at the end of the subject string, or immediately before a newline character that is the last character in the string (by - default). Dollar need not be the last character of the pattern if a - number of alternatives are involved, but it should be the last item in - any branch in which it appears. Dollar has no special meaning in a + default). Dollar need not be the last character of the pattern if a + number of alternatives are involved, but it should be the last item in + any branch in which it appears. Dollar has no special meaning in a character class. - The meaning of dollar can be changed so that it matches only at the - very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at + The meaning of dollar can be changed so that it matches only at the + very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at compile time. This does not affect the \Z assertion. The meanings of the circumflex and dollar characters are changed if the PCRE_MULTILINE option is set. When this is the case, they match immedi- - ately after and immediately before an internal newline character, - respectively, in addition to matching at the start and end of the sub- - ject string. For example, the pattern /^abc$/ matches the subject - string "def\nabc" (where \n represents a newline character) in multi- + ately after and immediately before an internal newline character, + respectively, in addition to matching at the start and end of the sub- + ject string. For example, the pattern /^abc$/ matches the subject + string "def\nabc" (where \n represents a newline character) in multi- line mode, but not otherwise. Consequently, patterns that are anchored - in single line mode because all branches start with ^ are not anchored - in multiline mode, and a match for circumflex is possible when the - startoffset argument of pcre_exec() is non-zero. The PCRE_DOL- + in single line mode because all branches start with ^ are not anchored + in multiline mode, and a match for circumflex is possible when the + startoffset argument of pcre_exec() is non-zero. The PCRE_DOL- LAR_ENDONLY option is ignored if PCRE_MULTILINE is set. - Note that the sequences \A, \Z, and \z can be used to match the start - and end of the subject in both modes, and if all branches of a pattern - start with \A it is always anchored, whether PCRE_MULTILINE is set or + Note that the sequences \A, \Z, and \z can be used to match the start + and end of the subject in both modes, and if all branches of a pattern + start with \A it is always anchored, whether PCRE_MULTILINE is set or not. FULL STOP (PERIOD, DOT) Outside a character class, a dot in the pattern matches any one charac- - ter in the subject, including a non-printing character, but not (by - default) newline. In UTF-8 mode, a dot matches any UTF-8 character, + ter in the subject, including a non-printing character, but not (by + default) newline. In UTF-8 mode, a dot matches any UTF-8 character, which might be more than one byte long, except (by default) newline. If - the PCRE_DOTALL option is set, dots match newlines as well. The han- - dling of dot is entirely independent of the handling of circumflex and - dollar, the only relationship being that they both involve newline + the PCRE_DOTALL option is set, dots match newlines as well. The han- + dling of dot is entirely independent of the handling of circumflex and + dollar, the only relationship being that they both involve newline characters. Dot has no special meaning in a character class. MATCHING A SINGLE BYTE Outside a character class, the escape sequence \C matches any one byte, - both in and out of UTF-8 mode. Unlike a dot, it can match a newline. - The feature is provided in Perl in order to match individual bytes in - UTF-8 mode. Because it breaks up UTF-8 characters into individual - bytes, what remains in the string may be a malformed UTF-8 string. For + both in and out of UTF-8 mode. Unlike a dot, it can match a newline. + The feature is provided in Perl in order to match individual bytes in + UTF-8 mode. Because it breaks up UTF-8 characters into individual + bytes, what remains in the string may be a malformed UTF-8 string. For this reason, the \C escape sequence is best avoided. - PCRE does not allow \C to appear in lookbehind assertions (described - below), because in UTF-8 mode this would make it impossible to calcu- + PCRE does not allow \C to appear in lookbehind assertions (described + below), because in UTF-8 mode this would make it impossible to calcu- late the length of the lookbehind. @@ -2267,35 +2794,40 @@ SQUARE BRACKETS AND CHARACTER CLASSES An opening square bracket introduces a character class, terminated by a closing square bracket. A closing square bracket on its own is not spe- cial. If a closing square bracket is required as a member of the class, - it should be the first data character in the class (after an initial + it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash. - A character class matches a single character in the subject. In UTF-8 - mode, the character may occupy more than one byte. A matched character + A character class matches a single character in the subject. In UTF-8 + mode, the character may occupy more than one byte. A matched character must be in the set of characters defined by the class, unless the first - character in the class definition is a circumflex, in which case the - subject character must not be in the set defined by the class. If a - circumflex is actually required as a member of the class, ensure it is + character in the class definition is a circumflex, in which case the + subject character must not be in the set defined by the class. If a + circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash. - For example, the character class [aeiou] matches any lower case vowel, - while [^aeiou] matches any character that is not a lower case vowel. + For example, the character class [aeiou] matches any lower case vowel, + while [^aeiou] matches any character that is not a lower case vowel. Note that a circumflex is just a convenient notation for specifying the - characters that are in the class by enumerating those that are not. A - class that starts with a circumflex is not an assertion: it still con- - sumes a character from the subject string, and therefore it fails if + characters that are in the class by enumerating those that are not. A + class that starts with a circumflex is not an assertion: it still con- + sumes a character from the subject string, and therefore it fails if the current pointer is at the end of the string. - In UTF-8 mode, characters with values greater than 255 can be included - in a class as a literal string of bytes, or by using the \x{ escaping + In UTF-8 mode, characters with values greater than 255 can be included + in a class as a literal string of bytes, or by using the \x{ escaping mechanism. - When caseless matching is set, any letters in a class represent both - their upper case and lower case versions, so for example, a caseless - [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not - match "A", whereas a caseful version would. When running in UTF-8 mode, - PCRE supports the concept of case for characters with values greater - than 128 only when it is compiled with Unicode property support. + When caseless matching is set, any letters in a class represent both + their upper case and lower case versions, so for example, a caseless + [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not + match "A", whereas a caseful version would. In UTF-8 mode, PCRE always + understands the concept of case for characters whose values are less + than 128, so caseless matching is always possible. For characters with + higher values, the concept of case is supported if PCRE is compiled + with Unicode property support, but not otherwise. If you want to use + caseless matching for characters 128 and above, you must ensure that + PCRE is compiled with Unicode property support as well as with UTF-8 + support. The newline character is never treated in any special way in character classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE @@ -3215,24 +3747,25 @@ CALLOUTS gether. A complete description of the interface to the callout function is given in the pcrecallout documentation. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCREPARTIAL(3) PCREPARTIAL(3) NAME PCRE - Perl-compatible regular expressions + PARTIAL MATCHING IN PCRE In normal use of PCRE, if the subject string that is passed to - pcre_exec() matches as far as it goes, but is too short to match the - entire pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances - where it might be helpful to distinguish this case from other cases in - which there is no match. + pcre_exec() or pcre_dfa_exec() matches as far as it goes, but is too + short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. + There are circumstances where it might be helpful to distinguish this + case from other cases in which there is no match. Consider, for example, an application where a human is required to type in data for a field with specific formatting requirements. An example @@ -3248,11 +3781,19 @@ PARTIAL MATCHING IN PCRE until the entire string has been entered. PCRE supports the concept of partial matching by means of the PCRE_PAR- - TIAL option, which can be set when calling pcre_exec(). When this is - done, the return code PCRE_ERROR_NOMATCH is converted into - PCRE_ERROR_PARTIAL if at any time during the matching process the - entire subject string matched part of the pattern. No captured data is - set when this occurs. + TIAL option, which can be set when calling pcre_exec() or + pcre_dfa_exec(). When this flag is set for pcre_exec(), the return code + PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time + during the matching process the last part of the subject string matched + part of the pattern. Unfortunately, for non-anchored matching, it is + not possible to obtain the position of the start of the partial match. + No captured data is set when PCRE_ERROR_PARTIAL is returned. + + When PCRE_PARTIAL is set for pcre_dfa_exec(), the return code + PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of + the subject is reached, there have been no complete matches, but there + is still at least one matching possibility. The portion of the string + that provided the partial match is set as the first matching string. Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers the last literal byte in a pattern, and abandons matching immediately @@ -3263,8 +3804,9 @@ PARTIAL MATCHING IN PCRE RESTRICTED PATTERNS FOR PCRE_PARTIAL Because of the way certain internal optimizations are implemented in - PCRE, the PCRE_PARTIAL option cannot be used with all patterns. - Repeated single characters such as + the pcre_exec() function, the PCRE_PARTIAL option cannot be used with + all patterns. These restrictions do not apply when pcre_dfa_exec() is + used. For pcre_exec(), repeated single characters such as a{2,4} @@ -3272,26 +3814,26 @@ RESTRICTED PATTERNS FOR PCRE_PARTIAL \d+ - are not permitted if the maximum number of occurrences is greater than + are not permitted if the maximum number of occurrences is greater than one. Optional items such as \d? (where the maximum is one) are permit- - ted. Quantifiers with any values are permitted after parentheses, so + ted. Quantifiers with any values are permitted after parentheses, so the invalid examples above can be coded thus: (a){2,4} (\d)+ - These constructions run more slowly, but for the kinds of application - that are envisaged for this facility, this is not felt to be a major + These constructions run more slowly, but for the kinds of application + that are envisaged for this facility, this is not felt to be a major restriction. - If PCRE_PARTIAL is set for a pattern that does not conform to the - restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL + If PCRE_PARTIAL is set for a pattern that does not conform to the + restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL (-13). EXAMPLE OF PARTIAL MATCHING USING PCRETEST - If the escape sequence \P is present in a pcretest data line, the + If the escape sequence \P is present in a pcretest data line, the PCRE_PARTIAL flag is used for the match. Here is a run of pcretest that uses the date example quoted above: @@ -3308,21 +3850,105 @@ EXAMPLE OF PARTIAL MATCHING USING PCRETEST data> jP No match - The first data string is matched completely, so pcretest shows the - matched substrings. The remaining four strings do not match the com- - plete pattern, but the first two are partial matches. + The first data string is matched completely, so pcretest shows the + matched substrings. The remaining four strings do not match the com- + plete pattern, but the first two are partial matches. The same test, + using DFA matching (by means of the \D escape sequence), produces the + following output: -Last updated: 08 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ + re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/ + data> 25jun04\P\D + 0: 25jun04 + data> 23dec3\P\D + Partial match: 23dec3 + data> 3ju\P\D + Partial match: 3ju + data> 3juj\P\D + No match + data> j\P\D + No match -PCRE(3) PCRE(3) + Notice that in this case the portion of the string that was matched is + made available. + + +MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() + + When a partial match has been found using pcre_dfa_exec(), it is possi- + ble to continue the match by providing additional subject data and + calling pcre_dfa_exec() again with the PCRE_DFA_RESTART option and the + same working space (where details of the previous partial match are + stored). Here is an example using pcretest, where the \R escape + sequence sets the PCRE_DFA_RESTART option and the \D escape sequence + requests the use of pcre_dfa_exec(): + + re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/ + data> 23ja\P\D + Partial match: 23ja + data> n05\R\D + 0: n05 + + The first call has "23ja" as the subject, and requests partial match- + ing; the second call has "n05" as the subject for the continued + (restarted) match. Notice that when the match is complete, only the + last part is shown; PCRE does not retain the previously partially- + matched string. It is up to the calling program to do that if it needs + to. + + This facility can be used to pass very long subject strings to + pcre_dfa_exec(). However, some care is needed for certain types of pat- + tern. + + 1. If the pattern contains tests for the beginning or end of a line, + you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri- + ate, when the subject string for any call does not contain the begin- + ning or end of a line. + + 2. If the pattern contains backward assertions (including \b or \B), + you need to arrange for some overlap in the subject strings to allow + for this. For example, you could pass the subject in chunks that were + 500 bytes long, but in a buffer of 700 bytes, with the starting offset + set to 200 and the previous 200 bytes at the start of the buffer. + + 3. Matching a subject string that is split into multiple segments does + not always produce exactly the same result as matching over one single + long string. The difference arises when there are multiple matching + possibilities, because a partial match result is given only when there + are no completed matches in a call to fBpcre_dfa_exec(). This means + that as soon as the shortest match has been found, continuation to a + new subject segment is no longer possible. Consider this pcretest + example: + + re> /dog(sbody)?/ + data> do\P\D + Partial match: do + data> gsb\R\P\D + 0: g + data> dogsbody\D + 0: dogsbody + 1: dog + + The pattern matches the words "dog" or "dogsbody". When the subject is + presented in several parts ("do" and "gsb" being the first two) the + match stops when "dog" has been found, and it is not possible to con- + tinue. On the other hand, if "dogsbody" is presented as a single + string, both matches are found. + + Because of this phenomenon, it does not usually make sense to end a + pattern that is going to be matched in this way with a variable repeat. + +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ +PCREPRECOMPILE(3) PCREPRECOMPILE(3) + NAME PCRE - Perl-compatible regular expressions + SAVING AND RE-USING PRECOMPILED PCRE PATTERNS If you are running an application that uses a large number of regular @@ -3391,16 +4017,18 @@ SAVING A COMPILED PATTERN RE-USING A PRECOMPILED PATTERN Re-using a precompiled pattern is straightforward. Having reloaded it - into main memory, you pass its pointer to pcre_exec() in the usual way. - This should work even on another host, and even if that host has the - opposite endianness to the one where the pattern was compiled. - - However, if you passed a pointer to custom character tables when the - pattern was compiled (the tableptr argument of pcre_compile()), you - must now pass a similar pointer to pcre_exec(), because the value saved - with the compiled pattern will obviously be nonsense. A field in a - pcre_extra() block is used to pass this data, as described in the sec- - tion on matching a pattern in the pcreapi documentation. + into main memory, you pass its pointer to pcre_exec() or + pcre_dfa_exec() in the usual way. This should work even on another + host, and even if that host has the opposite endianness to the one + where the pattern was compiled. + + However, if you passed a pointer to custom character tables when the + pattern was compiled (the tableptr argument of pcre_compile()), you + must now pass a similar pointer to pcre_exec() or pcre_dfa_exec(), + because the value saved with the compiled pattern will obviously be + nonsense. A field in a pcre_extra() block is used to pass this data, as + described in the section on matching a pattern in the pcreapi documen- + tation. If you did not provide custom character tables when the pattern was compiled, the pointer in the compiled pattern is NULL, which causes @@ -3411,29 +4039,31 @@ RE-USING A PRECOMPILED PATTERN your own pcre_extra data block and set the study_data field to point to the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA bit in the flags field to indicate that study data is present. Then - pass the pcre_extra block to pcre_exec() in the usual way. + pass the pcre_extra block to pcre_exec() or pcre_dfa_exec() in the + usual way. COMPATIBILITY WITH DIFFERENT PCRE RELEASES - The layout of the control block that is at the start of the data that - makes up a compiled pattern was changed for release 5.0. If you have - any saved patterns that were compiled with previous releases (not a - facility that was previously advertised), you will have to recompile - them for release 5.0. However, from now on, it should be possible to - make changes in a compabible manner. + The layout of the control block that is at the start of the data that + makes up a compiled pattern was changed for release 5.0. If you have + any saved patterns that were compiled with previous releases (not a + facility that was previously advertised), you will have to recompile + them for release 5.0. However, from now on, it should be possible to + make changes in a compatible manner. -Last updated: 10 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCREPERFORM(3) PCREPERFORM(3) NAME PCRE - Perl-compatible regular expressions + PCRE PERFORMANCE Certain items that may appear in regular expression patterns are more @@ -3469,9 +4099,9 @@ PCRE PERFORMANCE If you are using such a pattern with subject strings that do not con- tain newlines, the best performance is obtained by setting PCRE_DOTALL, - or starting the pattern with ^.* to indicate explicit anchoring. That - saves PCRE from having to scan along the subject looking for a newline - to restart at. + or starting the pattern with ^.* or ^.*? to indicate explicit anchor- + ing. That saves PCRE from having to scan along the subject looking for + a newline to restart at. Beware of patterns that contain nested indefinite repeats. These can take a long time to run when applied to a string that does not match. @@ -3492,9 +4122,9 @@ PCRE PERFORMANCE (a+)*b where a literal character follows. Before embarking on the standard - matching procedure, PCRE checks that there is a "b" later in the - subject string, and if there is not, it fails the match immediately. - However, when there is no following literal this optimization cannot be + matching procedure, PCRE checks that there is a "b" later in the sub- + ject string, and if there is not, it fails the match immediately. How- + ever, when there is no following literal this optimization cannot be used. You can see the difference by comparing the behaviour of (a+)*\d @@ -3506,17 +4136,18 @@ PCRE PERFORMANCE In many cases, the solution to this kind of performance issue is to use an atomic group or a possessive quantifier. -Last updated: 09 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCREPOSIX(3) PCREPOSIX(3) NAME PCRE - Perl-compatible regular expressions. + SYNOPSIS OF POSIX API #include <pcreposix.h> @@ -3537,7 +4168,7 @@ DESCRIPTION This set of functions provides a POSIX-style API to the PCRE regular expression package. See the pcreapi documentation for a description of - PCRE's native API, which contains additional functionality. + PCRE's native API, which contains much additional functionality. The functions described here are just wrapper functions that ultimately call the PCRE native API. Their prototypes are defined in the @@ -3581,6 +4212,12 @@ COMPILING A PATTERN The argument cflags is either zero, or contains one or more of the bits defined by the following macros: + REG_DOTALL + + The PCRE_DOTALL option is set when the expression is passed for compi- + lation to the native function. Note that REG_DOTALL is not part of the + POSIX standard. + REG_ICASE The PCRE_CASELESS option is set when the expression is passed for com- @@ -3692,21 +4329,326 @@ MEMORY USAGE AUTHOR - Philip Hazel <ph10@cam.ac.uk> + Philip Hazel University Computing Service, Cambridge CB2 3QG, England. -Last updated: 07 September 2004 -Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ +Last updated: 28 February 2005 +Copyright (c) 1997-2005 University of Cambridge. +------------------------------------------------------------------------------ -PCRE(3) PCRE(3) +PCRECPP(3) PCRECPP(3) + + +NAME + PCRE - Perl-compatible regular expressions. + + +SYNOPSIS OF C++ WRAPPER + + #include <pcrecpp.h> + + +DESCRIPTION + + The C++ wrapper for PCRE was provided by Google Inc. Some additional + functionality was added by Giuseppe Maxia. This brief man page was con- + structed from the notes in the pcrecpp.h file, which should be con- + sulted for further details. + + +MATCHING INTERFACE + + The "FullMatch" operation checks that supplied text matches a supplied + pattern exactly. If pointer arguments are supplied, it copies matched + sub-strings that match sub-patterns into them. + + Example: successful match + pcrecpp::RE re("h.*o"); + re.FullMatch("hello"); + + Example: unsuccessful match (requires full match): + pcrecpp::RE re("e"); + !re.FullMatch("hello"); + + Example: creating a temporary RE object: + pcrecpp::RE("h.*o").FullMatch("hello"); + + You can pass in a "const char*" or a "string" for "text". The examples + below tend to use a const char*. You can, as in the different examples + above, store the RE object explicitly in a variable or use a temporary + RE object. The examples below use one mode or the other arbitrarily. + Either could correctly be used for any of these examples. + + You must supply extra pointer arguments to extract matched subpieces. + + Example: extracts "ruby" into "s" and 1234 into "i" + int i; + string s; + pcrecpp::RE re("(\\w+):(\\d+)"); + re.FullMatch("ruby:1234", &s, &i); + + Example: does not try to extract any extra sub-patterns + re.FullMatch("ruby:1234", &s); + + Example: does not try to extract into NULL + re.FullMatch("ruby:1234", NULL, &i); + + Example: integer overflow causes failure + !re.FullMatch("ruby:1234567891234", NULL, &i); + + Example: fails because there aren't enough sub-patterns: + !pcrecpp::RE("\\w+:\\d+").FullMatch("ruby:1234", &s); + + Example: fails because string cannot be stored in integer + !pcrecpp::RE("(.*)").FullMatch("ruby", &i); + + The provided pointer arguments can be pointers to any scalar numeric + type, or one of: + + string (matched piece is copied to string) + StringPiece (StringPiece is mutated to point to matched piece) + T (where "bool T::ParseFrom(const char*, int)" exists) + NULL (the corresponding matched sub-pattern is not copied) + + The function returns true iff all of the following conditions are sat- + isfied: + + a. "text" matches "pattern" exactly; + + b. The number of matched sub-patterns is >= number of supplied + pointers; + + c. The "i"th argument has a suitable type for holding the + string captured as the "i"th sub-pattern. If you pass in + NULL for the "i"th argument, or pass fewer arguments than + number of sub-patterns, "i"th captured sub-pattern is + ignored. + + The matching interface supports at most 16 arguments per call. If you + need more, consider using the more general interface + pcrecpp::RE::DoMatch. See pcrecpp.h for the signature for DoMatch. + + +PARTIAL MATCHES + + You can use the "PartialMatch" operation when you want the pattern to + match any substring of the text. + + Example: simple search for a string: + pcrecpp::RE("ell").PartialMatch("hello"); + + Example: find first number in a string: + int number; + pcrecpp::RE re("(\\d+)"); + re.PartialMatch("x*100 + 20", &number); + assert(number == 100); + + +UTF-8 AND THE MATCHING INTERFACE + + By default, pattern and text are plain text, one byte per character. + The UTF8 flag, passed to the constructor, causes both pattern and + string to be treated as UTF-8 text, still a byte stream but potentially + multiple bytes per character. In practice, the text is likelier to be + UTF-8 than the pattern, but the match returned may depend on the UTF8 + flag, so always use it when matching UTF8 text. For example, "." will + match one byte normally but with UTF8 set may match up to three bytes + of a multi-byte character. + + Example: + pcrecpp::RE_Options options; + options.set_utf8(); + pcrecpp::RE re(utf8_pattern, options); + re.FullMatch(utf8_string); + + Example: using the convenience function UTF8(): + pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8()); + re.FullMatch(utf8_string); + + NOTE: The UTF8 flag is ignored if pcre was not configured with the + --enable-utf8 flag. + + +PASSING MODIFIERS TO THE REGULAR EXPRESSION ENGINE + + PCRE defines some modifiers to change the behavior of the regular + expression engine. The C++ wrapper defines an auxiliary class, + RE_Options, as a vehicle to pass such modifiers to a RE class. Cur- + rently, the following modifiers are supported: + + modifier description Perl corresponding + + PCRE_CASELESS case insensitive match /i + PCRE_MULTILINE multiple lines match /m + PCRE_DOTALL dot matches newlines /s + PCRE_DOLLAR_ENDONLY $ matches only at end N/A + PCRE_EXTRA strict escape parsing N/A + PCRE_EXTENDED ignore whitespaces /x + PCRE_UTF8 handles UTF8 chars built-in + PCRE_UNGREEDY reverses * and *? N/A + PCRE_NO_AUTO_CAPTURE disables capturing parens N/A (*) + + (*) Both Perl and PCRE allow non capturing parentheses by means of the + "?:" modifier within the pattern itself. e.g. (?:ab|cd) does not cap- + ture, while (ab|cd) does. + + For a full account on how each modifier works, please check the PCRE + API reference page. + + For each modifier, there are two member functions whose name is made + out of the modifier in lowercase, without the "PCRE_" prefix. For + instance, PCRE_CASELESS is handled by + + bool caseless() + + which returns true if the modifier is set, and + + RE_Options & set_caseless(bool) + + which sets or unsets the modifier. Moreover, PCRE_CONFIG_MATCH_LIMIT + can be accessed through the set_match_limit() and match_limit() member + functions. Setting match_limit to a non-zero value will limit the exe- + cution of pcre to keep it from doing bad things like blowing the stack + or taking an eternity to return a result. A value of 5000 is good + enough to stop stack blowup in a 2MB thread stack. Setting match_limit + to zero disables match limiting. + + Normally, to pass one or more modifiers to a RE class, you declare a + RE_Options object, set the appropriate options, and pass this object to + a RE constructor. Example: + + RE_options opt; + opt.set_caseless(true); + if (RE("HELLO", opt).PartialMatch("hello world")) ... + + RE_options has two constructors. The default constructor takes no argu- + ments and creates a set of flags that are off by default. The optional + parameter option_flags is to facilitate transfer of legacy code from C + programs. This lets you do + + RE(pattern, + RE_Options(PCRE_CASELESS|PCRE_MULTILINE)).PartialMatch(str); + + However, new code is better off doing + + RE(pattern, + RE_Options().set_caseless(true).set_multiline(true)) + .PartialMatch(str); + + If you are going to pass one of the most used modifiers, there are some + convenience functions that return a RE_Options class with the appropri- + ate modifier already set: CASELESS(), UTF8(), MULTILINE(), DOTALL(), + and EXTENDED(). + + If you need to set several options at once, and you don't want to go + through the pains of declaring a RE_Options object and setting several + options, there is a parallel method that give you such ability on the + fly. You can concatenate several set_xxxxx() member functions, since + each of them returns a reference to its class object. For example, to + pass PCRE_CASELESS, PCRE_EXTENDED, and PCRE_MULTILINE to a RE with one + statement, you may write: + + RE(" ^ xyz \\s+ .* blah$", + RE_Options() + .set_caseless(true) + .set_extended(true) + .set_multiline(true)).PartialMatch(sometext); + + +SCANNING TEXT INCREMENTALLY + + The "Consume" operation may be useful if you want to repeatedly match + regular expressions at the front of a string and skip over them as they + match. This requires use of the "StringPiece" type, which represents a + sub-range of a real string. Like RE, StringPiece is defined in the + pcrecpp namespace. + + Example: read lines of the form "var = value" from a string. + string contents = ...; // Fill string somehow + pcrecpp::StringPiece input(contents); // Wrap in a StringPiece + + string var; + int value; + pcrecpp::RE re("(\\w+) = (\\d+)\n"); + while (re.Consume(&input, &var, &value)) { + ...; + } + + Each successful call to "Consume" will set "var/value", and also + advance "input" so it points past the matched text. + + The "FindAndConsume" operation is similar to "Consume" but does not + anchor your match at the beginning of the string. For example, you + could extract all words from a string by repeatedly calling + + pcrecpp::RE("(\\w+)").FindAndConsume(&input, &word) + + +PARSING HEX/OCTAL/C-RADIX NUMBERS + + By default, if you pass a pointer to a numeric value, the corresponding + text is interpreted as a base-10 number. You can instead wrap the + pointer with a call to one of the operators Hex(), Octal(), or CRadix() + to interpret the text in another base. The CRadix operator interprets + C-style "0" (base-8) and "0x" (base-16) prefixes, but defaults to + base-10. + + Example: + int a, b, c, d; + pcrecpp::RE re("(.*) (.*) (.*) (.*)"); + re.FullMatch("100 40 0100 0x40", + pcrecpp::Octal(&a), pcrecpp::Hex(&b), + pcrecpp::CRadix(&c), pcrecpp::CRadix(&d)); + + will leave 64 in a, b, c, and d. + + +REPLACING PARTS OF STRINGS + + You can replace the first match of "pattern" in "str" with "rewrite". + Within "rewrite", backslash-escaped digits (\1 to \9) can be used to + insert text matching corresponding parenthesized group from the pat- + tern. \0 in "rewrite" refers to the entire matching text. For example: + + string s = "yabba dabba doo"; + pcrecpp::RE("b+").Replace("d", &s); + + will leave "s" containing "yada dabba doo". The result is true if the + pattern matches and a replacement occurs, false otherwise. + + GlobalReplace is like Replace except that it replaces all occurrences + of the pattern in the string with the rewrite. Replacements are not + subject to re-matching. For example: + + string s = "yabba dabba doo"; + pcrecpp::RE("b+").GlobalReplace("d", &s); + + will leave "s" containing "yada dada doo". It returns the number of + replacements made. + + Extract is like Replace, except that if the pattern matches, "rewrite" + is copied into "out" (an additional argument) with substitutions. The + non-matching portions of "text" are ignored. Returns true iff a match + occurred and the extraction happened successfully; if no match occurs, + the string is left unaffected. + + +AUTHOR + + The C++ wrapper was contributed by Google Inc. + Copyright (c) 2005 Google Inc. +------------------------------------------------------------------------------ + + +PCRESAMPLE(3) PCRESAMPLE(3) NAME PCRE - Perl-compatible regular expressions + PCRE SAMPLE PROGRAM A simple, complete demonstration program, to get you started with using @@ -3765,5 +4707,4 @@ PCRE SAMPLE PROGRAM Last updated: 09 September 2004 Copyright (c) 1997-2004 University of Cambridge. ------------------------------------------------------------------------------ - +------------------------------------------------------------------------------ |