/[pcre]/code/trunk/doc/pcrepattern.3
ViewVC logotype

Diff of /code/trunk/doc/pcrepattern.3

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 139 by ph10, Fri Mar 30 13:41:47 2007 UTC revision 208 by ph10, Mon Aug 6 15:23:29 2007 UTC
# Line 4  PCRE - Perl-compatible regular expressio Line 4  PCRE - Perl-compatible regular expressio
4  .SH "PCRE REGULAR EXPRESSION DETAILS"  .SH "PCRE REGULAR EXPRESSION DETAILS"
5  .rs  .rs
6  .sp  .sp
7  The syntax and semantics of the regular expressions supported by PCRE are  The syntax and semantics of the regular expressions that are supported by PCRE
8  described below. Regular expressions are also described in the Perl  are described in detail below. There is a quick-reference syntax summary in the
9  documentation and in a number of books, some of which have copious examples.  .\" HREF
10  Jeffrey Friedl's "Mastering Regular Expressions", published by O'Reilly, covers  \fBpcresyntax\fP
11  regular expressions in great detail. This description of PCRE's regular  .\"
12  expressions is intended as reference material.  page. Perl's regular expressions are described in its own documentation, and
13    regular expressions in general are covered in a number of books, some of which
14    have copious examples. Jeffrey Friedl's "Mastering Regular Expressions",
15    published by O'Reilly, covers regular expressions in great detail. This
16    description of PCRE's regular expressions is intended as reference material.
17  .P  .P
18  The original operation of PCRE was on strings of one-byte characters. However,  The original operation of PCRE was on strings of one-byte characters. However,
19  there is now also support for UTF-8 character strings. To use this, you must  there is now also support for UTF-8 character strings. To use this, you must
# Line 30  The remainder of this document discusses Line 34  The remainder of this document discusses
34  PCRE when its main matching function, \fBpcre_exec()\fP, is used.  PCRE when its main matching function, \fBpcre_exec()\fP, is used.
35  From release 6.0, PCRE offers a second matching function,  From release 6.0, PCRE offers a second matching function,
36  \fBpcre_dfa_exec()\fP, which matches using a different algorithm that is not  \fBpcre_dfa_exec()\fP, which matches using a different algorithm that is not
37  Perl-compatible. The advantages and disadvantages of the alternative function,  Perl-compatible. Some of the features discussed below are not available when
38  and how it differs from the normal function, are discussed in the  \fBpcre_dfa_exec()\fP is used. The advantages and disadvantages of the
39    alternative function, and how it differs from the normal function, are
40    discussed in the
41  .\" HREF  .\" HREF
42  \fBpcrematching\fP  \fBpcrematching\fP
43  .\"  .\"
# Line 238  meanings Line 244  meanings
244  .SS "Absolute and relative back references"  .SS "Absolute and relative back references"
245  .rs  .rs
246  .sp  .sp
247  The sequence \eg followed by a positive or negative number, optionally enclosed  The sequence \eg followed by an unsigned or a negative number, optionally
248  in braces, is an absolute or relative back reference. Back references are  enclosed in braces, is an absolute or relative back reference. A named back
249  discussed  reference can be coded as \eg{name}. Back references are discussed
250  .\" HTML <a href="#backreferences">  .\" HTML <a href="#backreferences">
251  .\" </a>  .\" </a>
252  later,  later,
# Line 260  following are always recognized: Line 266  following are always recognized:
266  .sp  .sp
267    \ed     any decimal digit    \ed     any decimal digit
268    \eD     any character that is not a decimal digit    \eD     any character that is not a decimal digit
269      \eh     any horizontal whitespace character
270      \eH     any character that is not a horizontal whitespace character
271    \es     any whitespace character    \es     any whitespace character
272    \eS     any character that is not a whitespace character    \eS     any character that is not a whitespace character
273      \ev     any vertical whitespace character
274      \eV     any character that is not a vertical whitespace character
275    \ew     any "word" character    \ew     any "word" character
276    \eW     any "non-word" character    \eW     any "non-word" character
277  .sp  .sp
# Line 275  there is no character to match. Line 285  there is no character to match.
285  .P  .P
286  For compatibility with Perl, \es does not match the VT character (code 11).  For compatibility with Perl, \es does not match the VT character (code 11).
287  This makes it different from the the POSIX "space" class. The \es characters  This makes it different from the the POSIX "space" class. The \es characters
288  are HT (9), LF (10), FF (12), CR (13), and space (32). (If "use locale;" is  are HT (9), LF (10), FF (12), CR (13), and space (32). If "use locale;" is
289  included in a Perl script, \es may match the VT character. In PCRE, it never  included in a Perl script, \es may match the VT character. In PCRE, it never
290  does.)  does.
291    .P
292    In UTF-8 mode, characters with values greater than 128 never match \ed, \es, or
293    \ew, and always match \eD, \eS, and \eW. This is true even when Unicode
294    character property support is available. These sequences retain their original
295    meanings from before UTF-8 support was available, mainly for efficiency
296    reasons.
297    .P
298    The sequences \eh, \eH, \ev, and \eV are Perl 5.10 features. In contrast to the
299    other sequences, these do match certain high-valued codepoints in UTF-8 mode.
300    The horizontal space characters are:
301    .sp
302      U+0009     Horizontal tab
303      U+0020     Space
304      U+00A0     Non-break space
305      U+1680     Ogham space mark
306      U+180E     Mongolian vowel separator
307      U+2000     En quad
308      U+2001     Em quad
309      U+2002     En space
310      U+2003     Em space
311      U+2004     Three-per-em space
312      U+2005     Four-per-em space
313      U+2006     Six-per-em space
314      U+2007     Figure space
315      U+2008     Punctuation space
316      U+2009     Thin space
317      U+200A     Hair space
318      U+202F     Narrow no-break space
319      U+205F     Medium mathematical space
320      U+3000     Ideographic space
321    .sp
322    The vertical space characters are:
323    .sp
324      U+000A     Linefeed
325      U+000B     Vertical tab
326      U+000C     Formfeed
327      U+000D     Carriage return
328      U+0085     Next line
329      U+2028     Line separator
330      U+2029     Paragraph separator
331  .P  .P
332  A "word" character is an underscore or any character less than 256 that is a  A "word" character is an underscore or any character less than 256 that is a
333  letter or digit. The definition of letters and digits is controlled by PCRE's  letter or digit. The definition of letters and digits is controlled by PCRE's
# Line 293  in the Line 343  in the
343  .\"  .\"
344  page). For example, in a French locale such as "fr_FR" in Unix-like systems,  page). For example, in a French locale such as "fr_FR" in Unix-like systems,
345  or "french" in Windows, some character codes greater than 128 are used for  or "french" in Windows, some character codes greater than 128 are used for
346  accented letters, and these are matched by \ew.  accented letters, and these are matched by \ew. The use of locales with Unicode
347  .P  is discouraged.
 In UTF-8 mode, characters with values greater than 128 never match \ed, \es, or  
 \ew, and always match \eD, \eS, and \eW. This is true even when Unicode  
 character property support is available. The use of locales with Unicode is  
 discouraged.  
348  .  .
349  .  .
350  .SS "Newline sequences"  .SS "Newline sequences"
351  .rs  .rs
352  .sp  .sp
353  Outside a character class, the escape sequence \eR matches any Unicode newline  Outside a character class, the escape sequence \eR matches any Unicode newline
354  sequence. This is an extension to Perl. In non-UTF-8 mode \eR is equivalent to  sequence. This is a Perl 5.10 feature. In non-UTF-8 mode \eR is equivalent to
355  the following:  the following:
356  .sp  .sp
357    (?>\er\en|\en|\ex0b|\ef|\er|\ex85)    (?>\er\en|\en|\ex0b|\ef|\er|\ex85)
# Line 334  Inside a character class, \eR matches th Line 380  Inside a character class, \eR matches th
380  .rs  .rs
381  .sp  .sp
382  When PCRE is built with Unicode character property support, three additional  When PCRE is built with Unicode character property support, three additional
383  escape sequences to match character properties are available when UTF-8 mode  escape sequences that match characters with specific properties are available.
384  is selected. They are:  When not in UTF-8 mode, these sequences are of course limited to testing
385    characters whose codepoints are less than 256, but they do work in this mode.
386    The extra escape sequences are:
387  .sp  .sp
388    \ep{\fIxx\fP}   a character with the \fIxx\fP property    \ep{\fIxx\fP}   a character with the \fIxx\fP property
389    \eP{\fIxx\fP}   a character without the \fIxx\fP property    \eP{\fIxx\fP}   a character without the \fIxx\fP property
# Line 511  atomic group Line 559  atomic group
559  (see below).  (see below).
560  .\"  .\"
561  Characters with the "mark" property are typically accents that affect the  Characters with the "mark" property are typically accents that affect the
562  preceding character.  preceding character. None of them have codepoints less than 256, so in
563    non-UTF-8 mode \eX matches any one character.
564  .P  .P
565  Matching characters by Unicode property is not fast, because PCRE has to search  Matching characters by Unicode property is not fast, because PCRE has to search
566  a structure that contains data for over fifteen thousand characters. That is  a structure that contains data for over fifteen thousand characters. That is
# Line 519  why the traditional escape sequences suc Line 568  why the traditional escape sequences suc
568  properties in PCRE.  properties in PCRE.
569  .  .
570  .  .
571    .\" HTML <a name="resetmatchstart"></a>
572    .SS "Resetting the match start"
573    .rs
574    .sp
575    The escape sequence \eK, which is a Perl 5.10 feature, causes any previously
576    matched characters not to be included in the final matched sequence. For
577    example, the pattern:
578    .sp
579      foo\eKbar
580    .sp
581    matches "foobar", but reports that it has matched "bar". This feature is
582    similar to a lookbehind assertion
583    .\" HTML <a href="#lookbehind">
584    .\" </a>
585    (described below).
586    .\"
587    However, in this case, the part of the subject before the real match does not
588    have to be of fixed length, as lookbehind assertions do. The use of \eK does
589    not interfere with the setting of
590    .\" HTML <a href="#subpattern">
591    .\" </a>
592    captured substrings.
593    .\"
594    For example, when the pattern
595    .sp
596      (foo)\eKbar
597    .sp
598    matches "foobar", the first substring is still set to "foo".
599    .
600    .
601  .\" HTML <a name="smallassertions"></a>  .\" HTML <a name="smallassertions"></a>
602  .SS "Simple assertions"  .SS "Simple assertions"
603  .rs  .rs
# Line 926  is reached, an option setting in one bra Line 1005  is reached, an option setting in one bra
1005  the above patterns match "SUNDAY" as well as "Saturday".  the above patterns match "SUNDAY" as well as "Saturday".
1006  .  .
1007  .  .
1008    .SH "DUPLICATE SUBPATTERN NUMBERS"
1009    .rs
1010    .sp
1011    Perl 5.10 introduced a feature whereby each alternative in a subpattern uses
1012    the same numbers for its capturing parentheses. Such a subpattern starts with
1013    (?| and is itself a non-capturing subpattern. For example, consider this
1014    pattern:
1015    .sp
1016      (?|(Sat)ur|(Sun))day
1017    .sp
1018    Because the two alternatives are inside a (?| group, both sets of capturing
1019    parentheses are numbered one. Thus, when the pattern matches, you can look
1020    at captured substring number one, whichever alternative matched. This construct
1021    is useful when you want to capture part, but not all, of one of a number of
1022    alternatives. Inside a (?| group, parentheses are numbered as usual, but the
1023    number is reset at the start of each branch. The numbers of any capturing
1024    buffers that follow the subpattern start after the highest number used in any
1025    branch. The following example is taken from the Perl documentation.
1026    The numbers underneath show in which buffer the captured content will be
1027    stored.
1028    .sp
1029      # before  ---------------branch-reset----------- after
1030      / ( a )  (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
1031      # 1            2         2  3        2     3     4
1032    .sp
1033    A backreference or a recursive call to a numbered subpattern always refers to
1034    the first one in the pattern with the given number.
1035    .P
1036    An alternative approach to using this "branch reset" feature is to use
1037    duplicate named subpatterns, as described in the next section.
1038    .
1039    .
1040  .SH "NAMED SUBPATTERNS"  .SH "NAMED SUBPATTERNS"
1041  .rs  .rs
1042  .sp  .sp
# Line 975  abbreviation. This pattern (ignoring the Line 1086  abbreviation. This pattern (ignoring the
1086    (?<DN>Sat)(?:urday)?    (?<DN>Sat)(?:urday)?
1087  .sp  .sp
1088  There are five capturing substrings, but only one is ever set after a match.  There are five capturing substrings, but only one is ever set after a match.
1089    (An alternative way of solving this problem is to use a "branch reset"
1090    subpattern, as described in the previous section.)
1091    .P
1092  The convenience function for extracting the data by name returns the substring  The convenience function for extracting the data by name returns the substring
1093  for the first (and in this example, the only) subpattern of that name that  for the first (and in this example, the only) subpattern of that name that
1094  matched. This saves searching to find which numbered subpattern it was. If you  matched. This saves searching to find which numbered subpattern it was. If you
# Line 1180  previous example can be rewritten as Line 1294  previous example can be rewritten as
1294  .sp  .sp
1295    \ed++foo    \ed++foo
1296  .sp  .sp
1297    Note that a possessive quantifier can be used with an entire group, for
1298    example:
1299    .sp
1300      (abc|xyz){2,3}+
1301    .sp
1302  Possessive quantifiers are always greedy; the setting of the PCRE_UNGREEDY  Possessive quantifiers are always greedy; the setting of the PCRE_UNGREEDY
1303  option is ignored. They are a convenient notation for the simpler forms of  option is ignored. They are a convenient notation for the simpler forms of
1304  atomic group. However, there is no difference in the meaning of a possessive  atomic group. However, there is no difference in the meaning of a possessive
# Line 1254  subpattern is possible using named paren Line 1373  subpattern is possible using named paren
1373  .P  .P
1374  Another way of avoiding the ambiguity inherent in the use of digits following a  Another way of avoiding the ambiguity inherent in the use of digits following a
1375  backslash is to use the \eg escape sequence, which is a feature introduced in  backslash is to use the \eg escape sequence, which is a feature introduced in
1376  Perl 5.10. This escape must be followed by a positive or a negative number,  Perl 5.10. This escape must be followed by an unsigned number or a negative
1377  optionally enclosed in braces. These examples are all identical:  number, optionally enclosed in braces. These examples are all identical:
1378  .sp  .sp
1379    (ring), \e1    (ring), \e1
1380    (ring), \eg1    (ring), \eg1
1381    (ring), \eg{1}    (ring), \eg{1}
1382  .sp  .sp
1383  A positive number specifies an absolute reference without the ambiguity that is  An unsigned number specifies an absolute reference without the ambiguity that
1384  present in the older syntax. It is also useful when literal digits follow the  is present in the older syntax. It is also useful when literal digits follow
1385  reference. A negative number is a relative reference. Consider this example:  the reference. A negative number is a relative reference. Consider this
1386    example:
1387  .sp  .sp
1388    (abc(def)ghi)\eg{-1}    (abc(def)ghi)\eg{-1}
1389  .sp  .sp
# Line 1293  back reference, the case of letters is r Line 1413  back reference, the case of letters is r
1413  matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original  matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original
1414  capturing subpattern is matched caselessly.  capturing subpattern is matched caselessly.
1415  .P  .P
1416  Back references to named subpatterns use the Perl syntax \ek<name> or \ek'name'  There are several different ways of writing back references to named
1417  or the Python syntax (?P=name). We could rewrite the above example in either of  subpatterns. The .NET syntax \ek{name} and the Perl syntax \ek<name> or
1418    \ek'name' are supported, as is the Python syntax (?P=name). Perl 5.10's unified
1419    back reference syntax, in which \eg can be used for both numeric and named
1420    references, is also supported. We could rewrite the above example in any of
1421  the following ways:  the following ways:
1422  .sp  .sp
1423    (?<p1>(?i)rah)\es+\ek<p1>    (?<p1>(?i)rah)\es+\ek<p1>
1424      (?'p1'(?i)rah)\es+\ek{p1}
1425    (?P<p1>(?i)rah)\es+(?P=p1)    (?P<p1>(?i)rah)\es+(?P=p1)
1426      (?<p1>(?i)rah)\es+\eg{p1}
1427  .sp  .sp
1428  A subpattern that is referenced by name may appear in the pattern before or  A subpattern that is referenced by name may appear in the pattern before or
1429  after the reference.  after the reference.
# Line 1421  lengths, but it is acceptable if rewritt Line 1546  lengths, but it is acceptable if rewritt
1546  .sp  .sp
1547    (?<=abc|abde)    (?<=abc|abde)
1548  .sp  .sp
1549    In some cases, the Perl 5.10 escape sequence \eK
1550    .\" HTML <a href="#resetmatchstart">
1551    .\" </a>
1552    (see above)
1553    .\"
1554    can be used instead of a lookbehind assertion; this is not restricted to a
1555    fixed-length.
1556    .P
1557  The implementation of lookbehind assertions is, for each alternative, to  The implementation of lookbehind assertions is, for each alternative, to
1558  temporarily move the current position back by the fixed length and then try to  temporarily move the current position back by the fixed length and then try to
1559  match. If there are insufficient characters before the current position, the  match. If there are insufficient characters before the current position, the
# Line 1515  recursion, a pseudo-condition called DEF Line 1648  recursion, a pseudo-condition called DEF
1648  .sp  .sp
1649  If the text between the parentheses consists of a sequence of digits, the  If the text between the parentheses consists of a sequence of digits, the
1650  condition is true if the capturing subpattern of that number has previously  condition is true if the capturing subpattern of that number has previously
1651  matched.  matched. An alternative notation is to precede the digits with a plus or minus
1652    sign. In this case, the subpattern number is relative rather than absolute.
1653    The most recently opened parentheses can be referenced by (?(-1), the next most
1654    recent by (?(-2), and so on. In looping constructs it can also make sense to
1655    refer to subsequent groups with constructs such as (?(+2).
1656  .P  .P
1657  Consider the following pattern, which contains non-significant white space to  Consider the following pattern, which contains non-significant white space to
1658  make it more readable (assume the PCRE_EXTENDED option) and to divide it into  make it more readable (assume the PCRE_EXTENDED option) and to divide it into
# Line 1532  the condition is true, and so the yes-pa Line 1669  the condition is true, and so the yes-pa
1669  parenthesis is required. Otherwise, since no-pattern is not present, the  parenthesis is required. Otherwise, since no-pattern is not present, the
1670  subpattern matches nothing. In other words, this pattern matches a sequence of  subpattern matches nothing. In other words, this pattern matches a sequence of
1671  non-parentheses, optionally enclosed in parentheses.  non-parentheses, optionally enclosed in parentheses.
1672    .P
1673    If you were embedding this pattern in a larger one, you could use a relative
1674    reference:
1675    .sp
1676      ...other stuff... ( \e( )?    [^()]+    (?(-1) \e) ) ...
1677    .sp
1678    This makes the fragment independent of the parentheses in the larger pattern.
1679  .  .
1680  .SS "Checking for a used subpattern by name"  .SS "Checking for a used subpattern by name"
1681  .rs  .rs
# Line 1674  pattern, so instead you could use this: Line 1818  pattern, so instead you could use this:
1818    ( \e( ( (?>[^()]+) | (?1) )* \e) )    ( \e( ( (?>[^()]+) | (?1) )* \e) )
1819  .sp  .sp
1820  We have put the pattern into parentheses, and caused the recursion to refer to  We have put the pattern into parentheses, and caused the recursion to refer to
1821  them instead of the whole pattern. In a larger pattern, keeping track of  them instead of the whole pattern.
1822  parenthesis numbers can be tricky. It may be more convenient to use named  .P
1823  parentheses instead. The Perl syntax for this is (?&name); PCRE's earlier  In a larger pattern, keeping track of parenthesis numbers can be tricky. This
1824  syntax (?P>name) is also supported. We could rewrite the above example as  is made easier by the use of relative references. (A Perl 5.10 feature.)
1825  follows:  Instead of (?1) in the pattern above you can write (?-2) to refer to the second
1826    most recently opened parentheses preceding the recursion. In other words, a
1827    negative number counts capturing parentheses leftwards from the point at which
1828    it is encountered.
1829    .P
1830    It is also possible to refer to subsequently opened parentheses, by writing
1831    references such as (?+2). However, these cannot be recursive because the
1832    reference is not inside the parentheses that are referenced. They are always
1833    "subroutine" calls, as described in the next section.
1834    .P
1835    An alternative approach is to use named parentheses instead. The Perl syntax
1836    for this is (?&name); PCRE's earlier syntax (?P>name) is also supported. We
1837    could rewrite the above example as follows:
1838  .sp  .sp
1839    (?<pn> \e( ( (?>[^()]+) | (?&pn) )* \e) )    (?<pn> \e( ( (?>[^()]+) | (?&pn) )* \e) )
1840  .sp  .sp
1841  If there is more than one subpattern with the same name, the earliest one is  If there is more than one subpattern with the same name, the earliest one is
1842  used. This particular example pattern contains nested unlimited repeats, and so  used.
1843  the use of atomic grouping for matching strings of non-parentheses is important  .P
1844  when applying the pattern to strings that do not match. For example, when this  This particular example pattern that we have been looking at contains nested
1845  pattern is applied to  unlimited repeats, and so the use of atomic grouping for matching strings of
1846    non-parentheses is important when applying the pattern to strings that do not
1847    match. For example, when this pattern is applied to
1848  .sp  .sp
1849    (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()    (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
1850  .sp  .sp
# Line 1738  is the actual recursive call. Line 1896  is the actual recursive call.
1896  If the syntax for a recursive subpattern reference (either by number or by  If the syntax for a recursive subpattern reference (either by number or by
1897  name) is used outside the parentheses to which it refers, it operates like a  name) is used outside the parentheses to which it refers, it operates like a
1898  subroutine in a programming language. The "called" subpattern may be defined  subroutine in a programming language. The "called" subpattern may be defined
1899  before or after the reference. An earlier example pointed out that the pattern  before or after the reference. A numbered reference can be absolute or
1900    relative, as in these examples:
1901    .sp
1902      (...(absolute)...)...(?2)...
1903      (...(relative)...)...(?-1)...
1904      (...(?+1)...(relative)...
1905    .sp
1906    An earlier example pointed out that the pattern
1907  .sp  .sp
1908    (sens|respons)e and \e1ibility    (sens|respons)e and \e1ibility
1909  .sp  .sp
# Line 1759  When a subpattern is used as a subroutin Line 1924  When a subpattern is used as a subroutin
1924  case-independence are fixed when the subpattern is defined. They cannot be  case-independence are fixed when the subpattern is defined. They cannot be
1925  changed for different calls. For example, consider this pattern:  changed for different calls. For example, consider this pattern:
1926  .sp  .sp
1927    (abc)(?i:(?1))    (abc)(?i:(?-1))
1928  .sp  .sp
1929  It matches "abcabc". It does not match "abcABC" because the change of  It matches "abcabc". It does not match "abcABC" because the change of
1930  processing option does not affect the called subpattern.  processing option does not affect the called subpattern.
# Line 1783  function is to be called. If you want to Line 1948  function is to be called. If you want to
1948  can put a number less than 256 after the letter C. The default value is zero.  can put a number less than 256 after the letter C. The default value is zero.
1949  For example, this pattern has two callout points:  For example, this pattern has two callout points:
1950  .sp  .sp
1951    (?C1)\dabc(?C2)def    (?C1)abc(?C2)def
1952  .sp  .sp
1953  If the PCRE_AUTO_CALLOUT flag is passed to \fBpcre_compile()\fP, callouts are  If the PCRE_AUTO_CALLOUT flag is passed to \fBpcre_compile()\fP, callouts are
1954  automatically installed before each item in the pattern. They are all numbered  automatically installed before each item in the pattern. They are all numbered
# Line 1821  Cambridge CB2 3QH, England. Line 1986  Cambridge CB2 3QH, England.
1986  .rs  .rs
1987  .sp  .sp
1988  .nf  .nf
1989  Last updated: 06 March 2007  Last updated: 06 August 2007
1990  Copyright (c) 1997-2007 University of Cambridge.  Copyright (c) 1997-2007 University of Cambridge.
1991  .fi  .fi

Legend:
Removed from v.139  
changed lines
  Added in v.208

  ViewVC Help
Powered by ViewVC 1.1.5