--- code/trunk/doc/html/pcrepattern.html 2013/11/19 10:44:59 1403 +++ code/trunk/doc/html/pcrepattern.html 2013/11/19 15:36:57 1404 @@ -116,21 +116,33 @@ Unicode property support

-Another special sequence that may appear at the start of a pattern is -

-  (*UCP)
-
+Another special sequence that may appear at the start of a pattern is (*UCP). This has the same effect as setting the PCRE_UCP option: it causes sequences such as \d and \w to use Unicode properties to determine character types, instead of recognizing only characters with codes less than 128 via a lookup table.


+Disabling auto-possessification +
+

+If a pattern starts with (*NO_AUTO_POSSESS), it has the same effect as setting +the PCRE_NO_AUTO_POSSESS option at compile time. This stops PCRE from making +quantifiers possessive when what follows cannot match the repeated item. For +example, by default a+b is treated as a++b. For more details, see the +pcreapi +documentation. +

+
Disabling start-up optimizations

If a pattern starts with (*NO_START_OPT), it has the same effect as setting the -PCRE_NO_START_OPTIMIZE option either at compile or matching time. +PCRE_NO_START_OPTIMIZE option either at compile or matching time. This disables +several optimizations for quickly reaching "no match" results. For more +details, see the +pcreapi +documentation.


Newline conventions @@ -193,10 +205,10 @@ (*LIMIT_RECURSION=d) where d is any number of decimal digits. However, the value of the setting must -be less than the value set by the caller of pcre_exec() for it to have -any effect. In other words, the pattern writer can lower the limit set by the -programmer, but not raise it. If there is more than one setting of one of these -limits, the lower value is used. +be less than the value set (or defaulted) by the caller of pcre_exec() +for it to have any effect. In other words, the pattern writer can lower the +limits set by the programmer, but not raise them. If there is more than one +setting of one of these limits, the lower value is used.


EBCDIC CHARACTER CODES

@@ -283,10 +295,11 @@ greater than 127) are treated as literals.

-If a pattern is compiled with the PCRE_EXTENDED option, white space in the -pattern (other than in a character class) and characters between a # outside -a character class and the next newline are ignored. An escaping backslash can -be used to include a white space or # character as part of the pattern. +If a pattern is compiled with the PCRE_EXTENDED option, most white space in the +pattern (other than in a character class), and characters between a # outside a +character class and the next newline, inclusive, are ignored. An escaping +backslash can be used to include a white space or # character as part of the +pattern.

If you want to remove the special meaning from a sequence of characters, you @@ -324,7 +337,9 @@ \n linefeed (hex 0A) \r carriage return (hex 0D) \t tab (hex 09) + \0dd character with octal code 0dd \ddd character with octal code ddd, or back reference + \o{ddd..} character with octal code ddd.. \xhh character with hex code hh \x{hhh..} character with hex code hhh.. (non-JavaScript mode) \uhhhh character with hex code hhhh (JavaScript mode only) @@ -347,42 +362,6 @@ characters also generate different values.

-By default, after \x, from zero to two hexadecimal digits are read (letters -can be in upper or lower case). Any number of hexadecimal digits may appear -between \x{ and }, but the character code is constrained as follows: -

-  8-bit non-UTF mode    less than 0x100
-  8-bit UTF-8 mode      less than 0x10ffff and a valid codepoint
-  16-bit non-UTF mode   less than 0x10000
-  16-bit UTF-16 mode    less than 0x10ffff and a valid codepoint
-  32-bit non-UTF mode   less than 0x80000000
-  32-bit UTF-32 mode    less than 0x10ffff and a valid codepoint
-
-Invalid Unicode codepoints are the range 0xd800 to 0xdfff (the so-called -"surrogate" codepoints), and 0xffef. -

-

-If characters other than hexadecimal digits appear between \x{ and }, or if -there is no terminating }, this form of escape is not recognized. Instead, the -initial \x will be interpreted as a basic hexadecimal escape, with no -following digits, giving a character whose value is zero. -

-

-If the PCRE_JAVASCRIPT_COMPAT option is set, the interpretation of \x is -as just described only when it is followed by two hexadecimal digits. -Otherwise, it matches a literal "x" character. In JavaScript mode, support for -code points greater than 256 is provided by \u, which must be followed by -four hexadecimal digits; otherwise it matches a literal "u" character. -Character codes specified by \u in JavaScript mode are constrained in the same -was as those specified by \x in non-JavaScript mode. -

-

-Characters whose value is less than 256 can be defined by either of the two -syntaxes for \x (or by \u in JavaScript mode). There is no difference in the -way they are handled. For example, \xdc is exactly the same as \x{dc} (or -\u00dc in JavaScript mode). -

-

After \0 up to two further octal digits are read. If there are fewer than two digits, just those that are present are used. Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL character (code value 7). Make @@ -390,9 +369,23 @@ follows is itself an octal digit.

-The handling of a backslash followed by a digit other than 0 is complicated. -Outside a character class, PCRE reads it and any following digits as a decimal -number. If the number is less than 10, or if there have been at least that many +The escape \o must be followed by a sequence of octal digits, enclosed in +braces. An error occurs if this is not the case. This escape is a recent +addition to Perl; it provides way of specifying character code points as octal +numbers greater than 0777, and it also allows octal numbers and back references +to be unambiguously specified. +

+

+For greater clarity and unambiguity, it is best to avoid following \ by a +digit greater than zero. Instead, use \o{} or \x{} to specify character +numbers, and \g{} to specify back references. The following paragraphs +describe the old, ambiguous syntax. +

+

+The handling of a backslash followed by a digit other than 0 is complicated, +and Perl has changed in recent releases, causing PCRE also to change. Outside a +character class, PCRE reads the digit and any following digits as a decimal +number. If the number is less than 8, or if there have been at least that many previous capturing left parentheses in the expression, the entire sequence is taken as a back reference. A description of how this works is given later, @@ -400,12 +393,11 @@ parenthesized subpatterns.

-Inside a character class, or if the decimal number is greater than 9 and there -have not been that many capturing subpatterns, PCRE re-reads up to three octal -digits following the backslash, and uses them to generate a data character. Any -subsequent digits stand for themselves. The value of the character is -constrained in the same way as characters specified in hexadecimal. -For example: +Inside a character class, or if the decimal number following \ is greater than +7 and there have not been that many capturing subpatterns, PCRE handles \8 and +\9 as the literal characters "8" and "9", and otherwise re-reads up to three +octal digits following the backslash, using them to generate a data character. +Any subsequent digits stand for themselves. For example:

   \040   is another way of writing an ASCII space
   \40    is the same, provided there are fewer than 40 previous capturing subpatterns
@@ -415,11 +407,52 @@
   \0113  is a tab followed by the character "3"
   \113   might be a back reference, otherwise the character with octal code 113
   \377   might be a back reference, otherwise the value 255 (decimal)
-  \81    is either a back reference, or a binary zero followed by the two characters "8" and "1"
+  \81    is either a back reference, or the two characters "8" and "1"
+
+Note that octal values of 100 or greater that are specified using this syntax +must not be introduced by a leading zero, because no more than three octal +digits are ever read. +

+

+By default, after \x that is not followed by {, from zero to two hexadecimal +digits are read (letters can be in upper or lower case). Any number of +hexadecimal digits may appear between \x{ and }. If a character other than +a hexadecimal digit appears between \x{ and }, or if there is no terminating +}, an error occurs. +

+

+If the PCRE_JAVASCRIPT_COMPAT option is set, the interpretation of \x is +as just described only when it is followed by two hexadecimal digits. +Otherwise, it matches a literal "x" character. In JavaScript mode, support for +code points greater than 256 is provided by \u, which must be followed by +four hexadecimal digits; otherwise it matches a literal "u" character. +

+

+Characters whose value is less than 256 can be defined by either of the two +syntaxes for \x (or by \u in JavaScript mode). There is no difference in the +way they are handled. For example, \xdc is exactly the same as \x{dc} (or +\u00dc in JavaScript mode). +

+
+Constraints on character values +
+

+Characters that are specified using octal or hexadecimal numbers are +limited to certain values, as follows: +

+  8-bit non-UTF mode    less than 0x100
+  8-bit UTF-8 mode      less than 0x10ffff and a valid codepoint
+  16-bit non-UTF mode   less than 0x10000
+  16-bit UTF-16 mode    less than 0x10ffff and a valid codepoint
+  32-bit non-UTF mode   less than 0x100000000
+  32-bit UTF-32 mode    less than 0x10ffff and a valid codepoint
 
-Note that octal values of 100 or greater must not be introduced by a leading -zero, because no more than three octal digits are ever read. +Invalid Unicode codepoints are the range 0xd800 to 0xdfff (the so-called +"surrogate" codepoints), and 0xffef.

+
+Escape sequences in character classes +

All the sequences that define a single character value can be used both inside and outside character classes. In addition, inside a character class, \b is @@ -498,11 +531,13 @@ there is no character to match.

-For compatibility with Perl, \s does not match the VT character (code 11). -This makes it different from the the POSIX "space" class. The \s characters -are HT (9), LF (10), FF (12), CR (13), and space (32). If "use locale;" is -included in a Perl script, \s may match the VT character. In PCRE, it never -does. +For compatibility with Perl, \s did not used to match the VT character (code +11), which made it different from the the POSIX "space" class. However, Perl +added VT at release 5.18, and PCRE followed suit at release 8.34. The default +\s characters are now HT (9), LF (10), VT (11), FF (12), CR (13), and space +(32), which are defined as white space in the "C" locale. This list may vary if +locale-specific matching is taking place; in particular, in some locales the +"non-breaking space" character (\xA0) is recognized as white space.

A "word" character is an underscore or any character that is a letter or digit. @@ -513,21 +548,23 @@ in the pcreapi page). For example, in a French locale such as "fr_FR" in Unix-like systems, -or "french" in Windows, some character codes greater than 128 are used for +or "french" in Windows, some character codes greater than 127 are used for accented letters, and these are then matched by \w. The use of locales with Unicode is discouraged.

-By default, in a UTF mode, characters with values greater than 128 never match -\d, \s, or \w, and always match \D, \S, and \W. These sequences retain -their original meanings from before UTF support was available, mainly for -efficiency reasons. However, if PCRE is compiled with Unicode property support, -and the PCRE_UCP option is set, the behaviour is changed so that Unicode -properties are used to determine character types, as follows: -

-  \d  any character that \p{Nd} matches (decimal digit)
-  \s  any character that \p{Z} matches, plus HT, LF, FF, CR
-  \w  any character that \p{L} or \p{N} matches, plus underscore
+By default, characters whose code points are greater than 127 never match \d,
+\s, or \w, and always match \D, \S, and \W, although this may vary for
+characters in the range 128-255 when locale-specific matching is happening.
+These escape sequences retain their original meanings from before Unicode
+support was available, mainly for efficiency reasons. If PCRE is compiled with
+Unicode property support, and the PCRE_UCP option is set, the behaviour is
+changed so that Unicode properties are used to determine character types, as
+follows:
+
+  \d  any character that matches \p{Nd} (decimal digit)
+  \s  any character that matches \p{Z} or \h or \v
+  \w  any character that matches \p{L} or \p{N}, plus underscore
 
The upper case escapes match the inverse sets of characters. Note that \d matches only decimal digits, whereas \w matches any Unicode digit, as well as @@ -538,7 +575,7 @@

The sequences \h, \H, \v, and \V are features that were added to Perl at release 5.10. In contrast to the other sequences, which match only ASCII -characters by default, these always match certain high-valued codepoints, +characters by default, these always match certain high-valued code points, whether or not PCRE_UCP is set. The horizontal space characters are:

   U+0009     Horizontal tab (HT)
@@ -913,9 +950,9 @@
 

As well as the standard Unicode properties described above, PCRE supports four more that make it possible to convert traditional escape sequences such as \w -and \s and POSIX character classes to use Unicode properties. PCRE uses these -non-standard, non-Perl properties internally when PCRE_UCP is set. However, -they may also be used explicitly. These properties are: +and \s to use Unicode properties. PCRE uses these non-standard, non-Perl +properties internally when PCRE_UCP is set. However, they may also be used +explicitly. These properties are:

   Xan   Any alphanumeric character
   Xps   Any POSIX space character
@@ -925,8 +962,9 @@
 Xan matches characters that have either the L (letter) or the N (number)
 property. Xps matches the characters tab, linefeed, vertical tab, form feed, or
 carriage return, and any other character that has the Z (separator) property.
-Xsp is the same as Xps, except that vertical tab is excluded. Xwd matches the
-same characters as Xan, plus underscore.
+Xsp is the same as Xps; it used to exclude vertical tab, for Perl
+compatibility, but Perl changed, and so PCRE followed at release 8.34. Xwd
+matches the same characters as Xan, plus underscore.
 

There is another non-standard property, Xuc, which matches any character that @@ -1218,7 +1256,9 @@ character class. For example, [d-m] matches any letter between d and m, inclusive. If a minus character is required in a class, it must be escaped with a backslash or appear in a position where it cannot be interpreted as -indicating a range, typically as the first or last character in the class. +indicating a range, typically as the first or last character in the class, or +immediately after a range. For example, [b-d-z] matches letters in the range b +to d, a hyphen character, or z.

It is not possible to have the literal character "]" as the end character of a @@ -1230,6 +1270,12 @@ "]" can also be used to end a range.

+An error is generated if a POSIX character class (see below) or an escape +sequence other than one that defines a single character appears at a point +where a range ending character is expected. For example, [z-\xff] is valid, +but [A-\d] and [A-[:digit:]] are not. +

+

Ranges operate in the collating sequence of character values. They can also be used for characters specified numerically, for example [\000-\037]. Ranges can include any characters that are valid for the current mode. @@ -1294,15 +1340,17 @@ lower lower case letters print printing characters, including space punct printing characters, excluding letters and digits and space - space white space (not quite the same as \s) + space white space (the same as \s from PCRE 8.34) upper upper case letters word "word" characters (same as \w) xdigit hexadecimal digits

-The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), and -space (32). Notice that this list includes the VT character (code 11). This -makes "space" different to \s, which does not include VT (for Perl -compatibility). +The default "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), +and space (32). If locale-specific matching is taking place, there may be +additional space characters. "Space" used to be different to \s, which did not +include VT, for Perl compatibility. However, Perl changed at release 5.18, and +PCRE followed at release 8.34. "Space" and \s now match the same set of +characters.

The name "word" is a Perl extension, and "blank" is a GNU extension from Perl @@ -1316,11 +1364,11 @@ supported, and an error is given if they are encountered.

-By default, in UTF modes, characters with values greater than 128 do not match -any of the POSIX character classes. However, if the PCRE_UCP option is passed -to pcre_compile(), some of the classes are changed so that Unicode -character properties are used. This is achieved by replacing the POSIX classes -by other sequences, as follows: +By default, characters with values greater than 128 do not match any of the +POSIX character classes. However, if the PCRE_UCP option is passed to +pcre_compile(), some of the classes are changed so that Unicode character +properties are used. This is achieved by replacing certain POSIX classes by +other sequences, as follows:

   [:alnum:]  becomes  \p{Xan}
   [:alpha:]  becomes  \p{L}
@@ -1331,9 +1379,35 @@
   [:upper:]  becomes  \p{Lu}
   [:word:]   becomes  \p{Xwd}
 
-Negated versions, such as [:^alpha:] use \P instead of \p. The other POSIX -classes are unchanged, and match only characters with code points less than -128. +Negated versions, such as [:^alpha:] use \P instead of \p. Three other POSIX +classes are handled specially in UCP mode: +

+

+[:graph:] +This matches characters that have glyphs that mark the page when printed. In +Unicode property terms, it matches all characters with the L, M, N, P, S, or Cf +properties, except for: +

+  U+061C           Arabic Letter Mark
+  U+180E           Mongolian Vowel Separator
+  U+2066 - U+2069  Various "isolate"s
+
+
+

+

+[:print:] +This matches the same characters as [:graph:] plus space characters that are +not controls, that is, characters with the Zs property. +

+

+[:punct:] +This matches all characters that have the Unicode P (punctuation) property, +plus those characters whose code points are less than 128 that have the S +(Symbol) property. +

+

+The other POSIX classes are unchanged, and match only characters with code +points less than 128.


VERTICAL BAR

@@ -1535,11 +1609,12 @@ can be made by name as well as by number.

-Names consist of up to 32 alphanumeric characters and underscores. Named -capturing parentheses are still allocated numbers as well as names, exactly as -if the names were not present. The PCRE API provides function calls for -extracting the name-to-number translation table from a compiled pattern. There -is also a convenience function for extracting a captured substring by name. +Names consist of up to 32 alphanumeric characters and underscores, but must +start with a non-digit. Named capturing parentheses are still allocated numbers +as well as names, exactly as if the names were not present. The PCRE API +provides function calls for extracting the name-to-number translation table +from a compiled pattern. There is also a convenience function for extracting a +captured substring by name.

By default, a name must be unique within a pattern, but it is possible to relax @@ -1568,9 +1643,23 @@

If you make a back reference to a non-unique named subpattern from elsewhere in -the pattern, the one that corresponds to the first occurrence of the name is -used. In the absence of duplicate numbers (see the previous section) this is -the one with the lowest number. If you use a named reference in a condition +the pattern, the subpatterns to which the name refers are checked in the order +in which they appear in the overall pattern. The first one that is set is used +for the reference. For example, this pattern matches both "foofoo" and +"barbar" but not "foobar" or "barfoo": +

+  (?:(?<n>foo)|(?<n>bar))\k<n>
+
+
+

+

+If you make a subroutine call to a non-unique named subpattern, the one that +corresponds to the first occurrence of the name is used. In the absence of +duplicate numbers (see the previous section) this is the one with the lowest +number. +

+

+If you use a named reference in a condition test (see the section about conditions below), either to check whether a subpattern has matched, or to check for @@ -1585,8 +1674,9 @@ Warning: You cannot use different names to distinguish between two subpatterns with the same number because PCRE uses only the numbers when matching. For this reason, an error is given at compile time if different names -are given to subpatterns with the same number. However, you can give the same -name to subpatterns with the same number, even when PCRE_DUPNAMES is not set. +are given to subpatterns with the same number. However, you can always give the +same name to subpatterns with the same number, even when PCRE_DUPNAMES is not +set.


REPETITION

@@ -2252,12 +2342,7 @@

Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a used subpattern by name. For compatibility with earlier versions of PCRE, which had -this facility before Perl, the syntax (?(name)...) is also recognized. However, -there is a possible ambiguity with this syntax, because subpattern names may -consist entirely of digits. PCRE looks first for a named subpattern; if it -cannot find one and the name consists entirely of digits, PCRE looks for a -subpattern of that number, which must be greater than zero. Using subpattern -names that consist entirely of digits is not recommended. +this facility before Perl, the syntax (?(name)...) is also recognized.

Rewriting the above example to use a named subpattern gives this: @@ -2674,8 +2759,14 @@ called. It is provided with the number of the callout, the position in the pattern, and, optionally, one item of data originally supplied by the caller of the matching function. The callout function may cause matching to proceed, to -backtrack, or to fail altogether. A complete description of the interface to -the callout function is given in the +backtrack, or to fail altogether. +

+

+By default, PCRE implements a number of optimizations at compile time and +matching time, and one side-effect is that sometimes callouts are skipped. If +you need all possible callouts to happen, you need to set options that disable +the relevant optimizations. More details, and a complete description of the +interface to the callout function, are given in the pcrecallout documentation.

@@ -3026,7 +3117,7 @@
   ...(*COMMIT)(*PRUNE)...
 
-If there is a matching failure to the right, backtracking onto (*PRUNE) cases +If there is a matching failure to the right, backtracking onto (*PRUNE) causes it to be triggered, and its action is taken. There can never be a backtrack onto (*COMMIT).

@@ -3109,7 +3200,7 @@


REVISION

-Last updated: 26 April 2013 +Last updated: 12 November 2013
Copyright © 1997-2013 University of Cambridge.