/[pcre]/code/trunk/doc/pcre.txt
ViewVC logotype

Contents of /code/trunk/doc/pcre.txt

Parent Directory Parent Directory | Revision Log Revision Log


Revision 81 - (show annotations)
Sat Feb 24 21:40:59 2007 UTC (12 years, 7 months ago) by nigel
File MIME type: text/plain
File size: 223795 byte(s)
Load pcre-6.2 into code/trunk.
1 -----------------------------------------------------------------------------
2 This file contains a concatenation of the PCRE man pages, converted to plain
3 text format for ease of searching with a text editor, or for use on systems
4 that do not have a man page processor. The small individual files that give
5 synopses of each function in the library have not been included. There are
6 separate text files for the pcregrep and pcretest commands.
7 -----------------------------------------------------------------------------
8
9
10 PCRE(3) PCRE(3)
11
12
13 NAME
14 PCRE - Perl-compatible regular expressions
15
16
17 INTRODUCTION
18
19 The PCRE library is a set of functions that implement regular expres-
20 sion pattern matching using the same syntax and semantics as Perl, with
21 just a few differences. The current implementation of PCRE (release
22 6.x) corresponds approximately with Perl 5.8, including support for
23 UTF-8 encoded strings and Unicode general category properties. However,
24 this support has to be explicitly enabled; it is not the default.
25
26 In addition to the Perl-compatible matching function, PCRE also con-
27 tains an alternative matching function that matches the same compiled
28 patterns in a different way. In certain circumstances, the alternative
29 function has some advantages. For a discussion of the two matching
30 algorithms, see the pcrematching page.
31
32 PCRE is written in C and released as a C library. A number of people
33 have written wrappers and interfaces of various kinds. In particular,
34 Google Inc. have provided a comprehensive C++ wrapper. This is now
35 included as part of the PCRE distribution. The pcrecpp page has details
36 of this interface. Other people's contributions can be found in the
37 Contrib directory at the primary FTP site, which is:
38
39 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre
40
41 Details of exactly which Perl regular expression features are and are
42 not supported by PCRE are given in separate documents. See the pcrepat-
43 tern and pcrecompat pages.
44
45 Some features of PCRE can be included, excluded, or changed when the
46 library is built. The pcre_config() function makes it possible for a
47 client to discover which features are available. The features them-
48 selves are described in the pcrebuild page. Documentation about build-
49 ing PCRE for various operating systems can be found in the README file
50 in the source distribution.
51
52 The library contains a number of undocumented internal functions and
53 data tables that are used by more than one of the exported external
54 functions, but which are not intended for use by external callers.
55 Their names all begin with "_pcre_", which hopefully will not provoke
56 any name clashes.
57
58
59 USER DOCUMENTATION
60
61 The user documentation for PCRE comprises a number of different sec-
62 tions. In the "man" format, each of these is a separate "man page". In
63 the HTML format, each is a separate page, linked from the index page.
64 In the plain text format, all the sections are concatenated, for ease
65 of searching. The sections are as follows:
66
67 pcre this document
68 pcreapi details of PCRE's native C API
69 pcrebuild options for building PCRE
70 pcrecallout details of the callout feature
71 pcrecompat discussion of Perl compatibility
72 pcrecpp details of the C++ wrapper
73 pcregrep description of the pcregrep command
74 pcrematching discussion of the two matching algorithms
75 pcrepartial details of the partial matching facility
76 pcrepattern syntax and semantics of supported
77 regular expressions
78 pcreperform discussion of performance issues
79 pcreposix the POSIX-compatible C API
80 pcreprecompile details of saving and re-using precompiled patterns
81 pcresample discussion of the sample program
82 pcretest description of the pcretest testing command
83
84 In addition, in the "man" and HTML formats, there is a short page for
85 each C library function, listing its arguments and results.
86
87
88 LIMITATIONS
89
90 There are some size limitations in PCRE but it is hoped that they will
91 never in practice be relevant.
92
93 The maximum length of a compiled pattern is 65539 (sic) bytes if PCRE
94 is compiled with the default internal linkage size of 2. If you want to
95 process regular expressions that are truly enormous, you can compile
96 PCRE with an internal linkage size of 3 or 4 (see the README file in
97 the source distribution and the pcrebuild documentation for details).
98 In these cases the limit is substantially larger. However, the speed
99 of execution will be slower.
100
101 All values in repeating quantifiers must be less than 65536. The maxi-
102 mum number of capturing subpatterns is 65535.
103
104 There is no limit to the number of non-capturing subpatterns, but the
105 maximum depth of nesting of all kinds of parenthesized subpattern,
106 including capturing subpatterns, assertions, and other types of subpat-
107 tern, is 200.
108
109 The maximum length of a subject string is the largest positive number
110 that an integer variable can hold. However, when using the traditional
111 matching function, PCRE uses recursion to handle subpatterns and indef-
112 inite repetition. This means that the available stack space may limit
113 the size of a subject string that can be processed by certain patterns.
114
115
116 UTF-8 AND UNICODE PROPERTY SUPPORT
117
118 From release 3.3, PCRE has had some support for character strings
119 encoded in the UTF-8 format. For release 4.0 this was greatly extended
120 to cover most common requirements, and in release 5.0 additional sup-
121 port for Unicode general category properties was added.
122
123 In order process UTF-8 strings, you must build PCRE to include UTF-8
124 support in the code, and, in addition, you must call pcre_compile()
125 with the PCRE_UTF8 option flag. When you do this, both the pattern and
126 any subject strings that are matched against it are treated as UTF-8
127 strings instead of just strings of bytes.
128
129 If you compile PCRE with UTF-8 support, but do not use it at run time,
130 the library will be a bit bigger, but the additional run time overhead
131 is limited to testing the PCRE_UTF8 flag in several places, so should
132 not be very large.
133
134 If PCRE is built with Unicode character property support (which implies
135 UTF-8 support), the escape sequences \p{..}, \P{..}, and \X are sup-
136 ported. The available properties that can be tested are limited to the
137 general category properties such as Lu for an upper case letter or Nd
138 for a decimal number. A full list is given in the pcrepattern documen-
139 tation. The PCRE library is increased in size by about 90K when Unicode
140 property support is included.
141
142 The following comments apply when PCRE is running in UTF-8 mode:
143
144 1. When you set the PCRE_UTF8 flag, the strings passed as patterns and
145 subjects are checked for validity on entry to the relevant functions.
146 If an invalid UTF-8 string is passed, an error return is given. In some
147 situations, you may already know that your strings are valid, and
148 therefore want to skip these checks in order to improve performance. If
149 you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time,
150 PCRE assumes that the pattern or subject it is given (respectively)
151 contains only valid UTF-8 codes. In this case, it does not diagnose an
152 invalid UTF-8 string. If you pass an invalid UTF-8 string to PCRE when
153 PCRE_NO_UTF8_CHECK is set, the results are undefined. Your program may
154 crash.
155
156 2. In a pattern, the escape sequence \x{...}, where the contents of the
157 braces is a string of hexadecimal digits, is interpreted as a UTF-8
158 character whose code number is the given hexadecimal number, for exam-
159 ple: \x{1234}. If a non-hexadecimal digit appears between the braces,
160 the item is not recognized. This escape sequence can be used either as
161 a literal, or within a character class.
162
163 3. The original hexadecimal escape sequence, \xhh, matches a two-byte
164 UTF-8 character if the value is greater than 127.
165
166 4. Repeat quantifiers apply to complete UTF-8 characters, not to indi-
167 vidual bytes, for example: \x{100}{3}.
168
169 5. The dot metacharacter matches one UTF-8 character instead of a sin-
170 gle byte.
171
172 6. The escape sequence \C can be used to match a single byte in UTF-8
173 mode, but its use can lead to some strange effects. This facility is
174 not available in the alternative matching function, pcre_dfa_exec().
175
176 7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
177 test characters of any code value, but the characters that PCRE recog-
178 nizes as digits, spaces, or word characters remain the same set as
179 before, all with values less than 256. This remains true even when PCRE
180 includes Unicode property support, because to do otherwise would slow
181 down PCRE in many common cases. If you really want to test for a wider
182 sense of, say, "digit", you must use Unicode property tests such as
183 \p{Nd}.
184
185 8. Similarly, characters that match the POSIX named character classes
186 are all low-valued characters.
187
188 9. Case-insensitive matching applies only to characters whose values
189 are less than 128, unless PCRE is built with Unicode property support.
190 Even when Unicode property support is available, PCRE still uses its
191 own character tables when checking the case of low-valued characters,
192 so as not to degrade performance. The Unicode property information is
193 used only for characters with higher values.
194
195
196 AUTHOR
197
198 Philip Hazel
199 University Computing Service,
200 Cambridge CB2 3QG, England.
201
202 Putting an actual email address here seems to have been a spam magnet,
203 so I've taken it away. If you want to email me, use my initial and sur-
204 name, separated by a dot, at the domain ucs.cam.ac.uk.
205
206 Last updated: 07 March 2005
207 Copyright (c) 1997-2005 University of Cambridge.
208 ------------------------------------------------------------------------------
209
210
211 PCREBUILD(3) PCREBUILD(3)
212
213
214 NAME
215 PCRE - Perl-compatible regular expressions
216
217
218 PCRE BUILD-TIME OPTIONS
219
220 This document describes the optional features of PCRE that can be
221 selected when the library is compiled. They are all selected, or dese-
222 lected, by providing options to the configure script that is run before
223 the make command. The complete list of options for configure (which
224 includes the standard ones such as the selection of the installation
225 directory) can be obtained by running
226
227 ./configure --help
228
229 The following sections describe certain options whose names begin with
230 --enable or --disable. These settings specify changes to the defaults
231 for the configure command. Because of the way that configure works,
232 --enable and --disable always come in pairs, so the complementary
233 option always exists as well, but as it specifies the default, it is
234 not described.
235
236
237 UTF-8 SUPPORT
238
239 To build PCRE with support for UTF-8 character strings, add
240
241 --enable-utf8
242
243 to the configure command. Of itself, this does not make PCRE treat
244 strings as UTF-8. As well as compiling PCRE with this option, you also
245 have have to set the PCRE_UTF8 option when you call the pcre_compile()
246 function.
247
248
249 UNICODE CHARACTER PROPERTY SUPPORT
250
251 UTF-8 support allows PCRE to process character values greater than 255
252 in the strings that it handles. On its own, however, it does not pro-
253 vide any facilities for accessing the properties of such characters. If
254 you want to be able to use the pattern escapes \P, \p, and \X, which
255 refer to Unicode character properties, you must add
256
257 --enable-unicode-properties
258
259 to the configure command. This implies UTF-8 support, even if you have
260 not explicitly requested it.
261
262 Including Unicode property support adds around 90K of tables to the
263 PCRE library, approximately doubling its size. Only the general cate-
264 gory properties such as Lu and Nd are supported. Details are given in
265 the pcrepattern documentation.
266
267
268 CODE VALUE OF NEWLINE
269
270 By default, PCRE treats character 10 (linefeed) as the newline charac-
271 ter. This is the normal newline character on Unix-like systems. You can
272 compile PCRE to use character 13 (carriage return) instead by adding
273
274 --enable-newline-is-cr
275
276 to the configure command. For completeness there is also a --enable-
277 newline-is-lf option, which explicitly specifies linefeed as the new-
278 line character.
279
280
281 BUILDING SHARED AND STATIC LIBRARIES
282
283 The PCRE building process uses libtool to build both shared and static
284 Unix libraries by default. You can suppress one of these by adding one
285 of
286
287 --disable-shared
288 --disable-static
289
290 to the configure command, as required.
291
292
293 POSIX MALLOC USAGE
294
295 When PCRE is called through the POSIX interface (see the pcreposix doc-
296 umentation), additional working storage is required for holding the
297 pointers to capturing substrings, because PCRE requires three integers
298 per substring, whereas the POSIX interface provides only two. If the
299 number of expected substrings is small, the wrapper function uses space
300 on the stack, because this is faster than using malloc() for each call.
301 The default threshold above which the stack is no longer used is 10; it
302 can be changed by adding a setting such as
303
304 --with-posix-malloc-threshold=20
305
306 to the configure command.
307
308
309 LIMITING PCRE RESOURCE USAGE
310
311 Internally, PCRE has a function called match(), which it calls repeat-
312 edly (possibly recursively) when matching a pattern with the
313 pcre_exec() function. By controlling the maximum number of times this
314 function may be called during a single matching operation, a limit can
315 be placed on the resources used by a single call to pcre_exec(). The
316 limit can be changed at run time, as described in the pcreapi documen-
317 tation. The default is 10 million, but this can be changed by adding a
318 setting such as
319
320 --with-match-limit=500000
321
322 to the configure command. This setting has no effect on the
323 pcre_dfa_exec() matching function.
324
325
326 HANDLING VERY LARGE PATTERNS
327
328 Within a compiled pattern, offset values are used to point from one
329 part to another (for example, from an opening parenthesis to an alter-
330 nation metacharacter). By default, two-byte values are used for these
331 offsets, leading to a maximum size for a compiled pattern of around
332 64K. This is sufficient to handle all but the most gigantic patterns.
333 Nevertheless, some people do want to process enormous patterns, so it
334 is possible to compile PCRE to use three-byte or four-byte offsets by
335 adding a setting such as
336
337 --with-link-size=3
338
339 to the configure command. The value given must be 2, 3, or 4. Using
340 longer offsets slows down the operation of PCRE because it has to load
341 additional bytes when handling them.
342
343 If you build PCRE with an increased link size, test 2 (and test 5 if
344 you are using UTF-8) will fail. Part of the output of these tests is a
345 representation of the compiled pattern, and this changes with the link
346 size.
347
348
349 AVOIDING EXCESSIVE STACK USAGE
350
351 When matching with the pcre_exec() function, PCRE implements backtrack-
352 ing by making recursive calls to an internal function called match().
353 In environments where the size of the stack is limited, this can se-
354 verely limit PCRE's operation. (The Unix environment does not usually
355 suffer from this problem.) An alternative approach that uses memory
356 from the heap to remember data, instead of using recursive function
357 calls, has been implemented to work round this problem. If you want to
358 build a version of PCRE that works this way, add
359
360 --disable-stack-for-recursion
361
362 to the configure command. With this configuration, PCRE will use the
363 pcre_stack_malloc and pcre_stack_free variables to call memory manage-
364 ment functions. Separate functions are provided because the usage is
365 very predictable: the block sizes requested are always the same, and
366 the blocks are always freed in reverse order. A calling program might
367 be able to implement optimized functions that perform better than the
368 standard malloc() and free() functions. PCRE runs noticeably more
369 slowly when built in this way. This option affects only the pcre_exec()
370 function; it is not relevant for the the pcre_dfa_exec() function.
371
372
373 USING EBCDIC CODE
374
375 PCRE assumes by default that it will run in an environment where the
376 character code is ASCII (or Unicode, which is a superset of ASCII).
377 PCRE can, however, be compiled to run in an EBCDIC environment by
378 adding
379
380 --enable-ebcdic
381
382 to the configure command.
383
384 Last updated: 28 February 2005
385 Copyright (c) 1997-2005 University of Cambridge.
386 ------------------------------------------------------------------------------
387
388
389 PCREMATCHING(3) PCREMATCHING(3)
390
391
392 NAME
393 PCRE - Perl-compatible regular expressions
394
395
396 PCRE MATCHING ALGORITHMS
397
398 This document describes the two different algorithms that are available
399 in PCRE for matching a compiled regular expression against a given sub-
400 ject string. The "standard" algorithm is the one provided by the
401 pcre_exec() function. This works in the same was as Perl's matching
402 function, and provides a Perl-compatible matching operation.
403
404 An alternative algorithm is provided by the pcre_dfa_exec() function;
405 this operates in a different way, and is not Perl-compatible. It has
406 advantages and disadvantages compared with the standard algorithm, and
407 these are described below.
408
409 When there is only one possible way in which a given subject string can
410 match a pattern, the two algorithms give the same answer. A difference
411 arises, however, when there are multiple possibilities. For example, if
412 the pattern
413
414 ^<.*>
415
416 is matched against the string
417
418 <something> <something else> <something further>
419
420 there are three possible answers. The standard algorithm finds only one
421 of them, whereas the DFA algorithm finds all three.
422
423
424 REGULAR EXPRESSIONS AS TREES
425
426 The set of strings that are matched by a regular expression can be rep-
427 resented as a tree structure. An unlimited repetition in the pattern
428 makes the tree of infinite size, but it is still a tree. Matching the
429 pattern to a given subject string (from a given starting point) can be
430 thought of as a search of the tree. There are two standard ways to
431 search a tree: depth-first and breadth-first, and these correspond to
432 the two matching algorithms provided by PCRE.
433
434
435 THE STANDARD MATCHING ALGORITHM
436
437 In the terminology of Jeffrey Friedl's book Mastering Regular Expres-
438 sions, the standard algorithm is an "NFA algorithm". It conducts a
439 depth-first search of the pattern tree. That is, it proceeds along a
440 single path through the tree, checking that the subject matches what is
441 required. When there is a mismatch, the algorithm tries any alterna-
442 tives at the current point, and if they all fail, it backs up to the
443 previous branch point in the tree, and tries the next alternative
444 branch at that level. This often involves backing up (moving to the
445 left) in the subject string as well. The order in which repetition
446 branches are tried is controlled by the greedy or ungreedy nature of
447 the quantifier.
448
449 If a leaf node is reached, a matching string has been found, and at
450 that point the algorithm stops. Thus, if there is more than one possi-
451 ble match, this algorithm returns the first one that it finds. Whether
452 this is the shortest, the longest, or some intermediate length depends
453 on the way the greedy and ungreedy repetition quantifiers are specified
454 in the pattern.
455
456 Because it ends up with a single path through the tree, it is rela-
457 tively straightforward for this algorithm to keep track of the sub-
458 strings that are matched by portions of the pattern in parentheses.
459 This provides support for capturing parentheses and back references.
460
461
462 THE DFA MATCHING ALGORITHM
463
464 DFA stands for "deterministic finite automaton", but you do not need to
465 understand the origins of that name. This algorithm conducts a breadth-
466 first search of the tree. Starting from the first matching point in the
467 subject, it scans the subject string from left to right, once, charac-
468 ter by character, and as it does this, it remembers all the paths
469 through the tree that represent valid matches.
470
471 The scan continues until either the end of the subject is reached, or
472 there are no more unterminated paths. At this point, terminated paths
473 represent the different matching possibilities (if there are none, the
474 match has failed). Thus, if there is more than one possible match,
475 this algorithm finds all of them, and in particular, it finds the long-
476 est. In PCRE, there is an option to stop the algorithm after the first
477 match (which is necessarily the shortest) has been found.
478
479 Note that all the matches that are found start at the same point in the
480 subject. If the pattern
481
482 cat(er(pillar)?)
483
484 is matched against the string "the caterpillar catchment", the result
485 will be the three strings "cat", "cater", and "caterpillar" that start
486 at the fourth character of the subject. The algorithm does not automat-
487 ically move on to find matches that start at later positions.
488
489 There are a number of features of PCRE regular expressions that are not
490 supported by the DFA matching algorithm. They are as follows:
491
492 1. Because the algorithm finds all possible matches, the greedy or
493 ungreedy nature of repetition quantifiers is not relevant. Greedy and
494 ungreedy quantifiers are treated in exactly the same way.
495
496 2. When dealing with multiple paths through the tree simultaneously, it
497 is not straightforward to keep track of captured substrings for the
498 different matching possibilities, and PCRE's implementation of this
499 algorithm does not attempt to do this. This means that no captured sub-
500 strings are available.
501
502 3. Because no substrings are captured, back references within the pat-
503 tern are not supported, and cause errors if encountered.
504
505 4. For the same reason, conditional expressions that use a backrefer-
506 ence as the condition are not supported.
507
508 5. Callouts are supported, but the value of the capture_top field is
509 always 1, and the value of the capture_last field is always -1.
510
511 6. The \C escape sequence, which (in the standard algorithm) matches a
512 single byte, even in UTF-8 mode, is not supported because the DFA algo-
513 rithm moves through the subject string one character at a time, for all
514 active paths through the tree.
515
516
517 ADVANTAGES OF THE DFA ALGORITHM
518
519 Using the DFA matching algorithm provides the following advantages:
520
521 1. All possible matches (at a single point in the subject) are automat-
522 ically found, and in particular, the longest match is found. To find
523 more than one match using the standard algorithm, you have to do kludgy
524 things with callouts.
525
526 2. There is much better support for partial matching. The restrictions
527 on the content of the pattern that apply when using the standard algo-
528 rithm for partial matching do not apply to the DFA algorithm. For non-
529 anchored patterns, the starting position of a partial match is avail-
530 able.
531
532 3. Because the DFA algorithm scans the subject string just once, and
533 never needs to backtrack, it is possible to pass very long subject
534 strings to the matching function in several pieces, checking for par-
535 tial matching each time.
536
537
538 DISADVANTAGES OF THE DFA ALGORITHM
539
540 The DFA algorithm suffers from a number of disadvantages:
541
542 1. It is substantially slower than the standard algorithm. This is
543 partly because it has to search for all possible matches, but is also
544 because it is less susceptible to optimization.
545
546 2. Capturing parentheses and back references are not supported.
547
548 3. The "atomic group" feature of PCRE regular expressions is supported,
549 but does not provide the advantage that it does for the standard algo-
550 rithm.
551
552 Last updated: 28 February 2005
553 Copyright (c) 1997-2005 University of Cambridge.
554 ------------------------------------------------------------------------------
555
556
557 PCREAPI(3) PCREAPI(3)
558
559
560 NAME
561 PCRE - Perl-compatible regular expressions
562
563
564 PCRE NATIVE API
565
566 #include <pcre.h>
567
568 pcre *pcre_compile(const char *pattern, int options,
569 const char **errptr, int *erroffset,
570 const unsigned char *tableptr);
571
572 pcre *pcre_compile2(const char *pattern, int options,
573 int *errorcodeptr,
574 const char **errptr, int *erroffset,
575 const unsigned char *tableptr);
576
577 pcre_extra *pcre_study(const pcre *code, int options,
578 const char **errptr);
579
580 int pcre_exec(const pcre *code, const pcre_extra *extra,
581 const char *subject, int length, int startoffset,
582 int options, int *ovector, int ovecsize);
583
584 int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
585 const char *subject, int length, int startoffset,
586 int options, int *ovector, int ovecsize,
587 int *workspace, int wscount);
588
589 int pcre_copy_named_substring(const pcre *code,
590 const char *subject, int *ovector,
591 int stringcount, const char *stringname,
592 char *buffer, int buffersize);
593
594 int pcre_copy_substring(const char *subject, int *ovector,
595 int stringcount, int stringnumber, char *buffer,
596 int buffersize);
597
598 int pcre_get_named_substring(const pcre *code,
599 const char *subject, int *ovector,
600 int stringcount, const char *stringname,
601 const char **stringptr);
602
603 int pcre_get_stringnumber(const pcre *code,
604 const char *name);
605
606 int pcre_get_substring(const char *subject, int *ovector,
607 int stringcount, int stringnumber,
608 const char **stringptr);
609
610 int pcre_get_substring_list(const char *subject,
611 int *ovector, int stringcount, const char ***listptr);
612
613 void pcre_free_substring(const char *stringptr);
614
615 void pcre_free_substring_list(const char **stringptr);
616
617 const unsigned char *pcre_maketables(void);
618
619 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
620 int what, void *where);
621
622 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
623
624 int pcre_refcount(pcre *code, int adjust);
625
626 int pcre_config(int what, void *where);
627
628 char *pcre_version(void);
629
630 void *(*pcre_malloc)(size_t);
631
632 void (*pcre_free)(void *);
633
634 void *(*pcre_stack_malloc)(size_t);
635
636 void (*pcre_stack_free)(void *);
637
638 int (*pcre_callout)(pcre_callout_block *);
639
640
641 PCRE API OVERVIEW
642
643 PCRE has its own native API, which is described in this document. There
644 is also a set of wrapper functions that correspond to the POSIX regular
645 expression API. These are described in the pcreposix documentation.
646 Both of these APIs define a set of C function calls. A C++ wrapper is
647 distributed with PCRE. It is documented in the pcrecpp page.
648
649 The native API C function prototypes are defined in the header file
650 pcre.h, and on Unix systems the library itself is called libpcre. It
651 can normally be accessed by adding -lpcre to the command for linking an
652 application that uses PCRE. The header file defines the macros
653 PCRE_MAJOR and PCRE_MINOR to contain the major and minor release num-
654 bers for the library. Applications can use these to include support
655 for different releases of PCRE.
656
657 The functions pcre_compile(), pcre_compile2(), pcre_study(), and
658 pcre_exec() are used for compiling and matching regular expressions in
659 a Perl-compatible manner. A sample program that demonstrates the sim-
660 plest way of using them is provided in the file called pcredemo.c in
661 the source distribution. The pcresample documentation describes how to
662 run it.
663
664 A second matching function, pcre_dfa_exec(), which is not Perl-compati-
665 ble, is also provided. This uses a different algorithm for the match-
666 ing. This allows it to find all possible matches (at a given point in
667 the subject), not just one. However, this algorithm does not return
668 captured substrings. A description of the two matching algorithms and
669 their advantages and disadvantages is given in the pcrematching docu-
670 mentation.
671
672 In addition to the main compiling and matching functions, there are
673 convenience functions for extracting captured substrings from a subject
674 string that is matched by pcre_exec(). They are:
675
676 pcre_copy_substring()
677 pcre_copy_named_substring()
678 pcre_get_substring()
679 pcre_get_named_substring()
680 pcre_get_substring_list()
681 pcre_get_stringnumber()
682
683 pcre_free_substring() and pcre_free_substring_list() are also provided,
684 to free the memory used for extracted strings.
685
686 The function pcre_maketables() is used to build a set of character
687 tables in the current locale for passing to pcre_compile(),
688 pcre_exec(), or pcre_dfa_exec(). This is an optional facility that is
689 provided for specialist use. Most commonly, no special tables are
690 passed, in which case internal tables that are generated when PCRE is
691 built are used.
692
693 The function pcre_fullinfo() is used to find out information about a
694 compiled pattern; pcre_info() is an obsolete version that returns only
695 some of the available information, but is retained for backwards com-
696 patibility. The function pcre_version() returns a pointer to a string
697 containing the version of PCRE and its date of release.
698
699 The function pcre_refcount() maintains a reference count in a data
700 block containing a compiled pattern. This is provided for the benefit
701 of object-oriented applications.
702
703 The global variables pcre_malloc and pcre_free initially contain the
704 entry points of the standard malloc() and free() functions, respec-
705 tively. PCRE calls the memory management functions via these variables,
706 so a calling program can replace them if it wishes to intercept the
707 calls. This should be done before calling any PCRE functions.
708
709 The global variables pcre_stack_malloc and pcre_stack_free are also
710 indirections to memory management functions. These special functions
711 are used only when PCRE is compiled to use the heap for remembering
712 data, instead of recursive function calls, when running the pcre_exec()
713 function. This is a non-standard way of building PCRE, for use in envi-
714 ronments that have limited stacks. Because of the greater use of memory
715 management, it runs more slowly. Separate functions are provided so
716 that special-purpose external code can be used for this case. When
717 used, these functions are always called in a stack-like manner (last
718 obtained, first freed), and always for memory blocks of the same size.
719
720 The global variable pcre_callout initially contains NULL. It can be set
721 by the caller to a "callout" function, which PCRE will then call at
722 specified points during a matching operation. Details are given in the
723 pcrecallout documentation.
724
725
726 MULTITHREADING
727
728 The PCRE functions can be used in multi-threading applications, with
729 the proviso that the memory management functions pointed to by
730 pcre_malloc, pcre_free, pcre_stack_malloc, and pcre_stack_free, and the
731 callout function pointed to by pcre_callout, are shared by all threads.
732
733 The compiled form of a regular expression is not altered during match-
734 ing, so the same compiled pattern can safely be used by several threads
735 at once.
736
737
738 SAVING PRECOMPILED PATTERNS FOR LATER USE
739
740 The compiled form of a regular expression can be saved and re-used at a
741 later time, possibly by a different program, and even on a host other
742 than the one on which it was compiled. Details are given in the
743 pcreprecompile documentation.
744
745
746 CHECKING BUILD-TIME OPTIONS
747
748 int pcre_config(int what, void *where);
749
750 The function pcre_config() makes it possible for a PCRE client to dis-
751 cover which optional features have been compiled into the PCRE library.
752 The pcrebuild documentation has more details about these optional fea-
753 tures.
754
755 The first argument for pcre_config() is an integer, specifying which
756 information is required; the second argument is a pointer to a variable
757 into which the information is placed. The following information is
758 available:
759
760 PCRE_CONFIG_UTF8
761
762 The output is an integer that is set to one if UTF-8 support is avail-
763 able; otherwise it is set to zero.
764
765 PCRE_CONFIG_UNICODE_PROPERTIES
766
767 The output is an integer that is set to one if support for Unicode
768 character properties is available; otherwise it is set to zero.
769
770 PCRE_CONFIG_NEWLINE
771
772 The output is an integer that is set to the value of the code that is
773 used for the newline character. It is either linefeed (10) or carriage
774 return (13), and should normally be the standard character for your
775 operating system.
776
777 PCRE_CONFIG_LINK_SIZE
778
779 The output is an integer that contains the number of bytes used for
780 internal linkage in compiled regular expressions. The value is 2, 3, or
781 4. Larger values allow larger regular expressions to be compiled, at
782 the expense of slower matching. The default value of 2 is sufficient
783 for all but the most massive patterns, since it allows the compiled
784 pattern to be up to 64K in size.
785
786 PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
787
788 The output is an integer that contains the threshold above which the
789 POSIX interface uses malloc() for output vectors. Further details are
790 given in the pcreposix documentation.
791
792 PCRE_CONFIG_MATCH_LIMIT
793
794 The output is an integer that gives the default limit for the number of
795 internal matching function calls in a pcre_exec() execution. Further
796 details are given with pcre_exec() below.
797
798 PCRE_CONFIG_STACKRECURSE
799
800 The output is an integer that is set to one if internal recursion when
801 running pcre_exec() is implemented by recursive function calls that use
802 the stack to remember their state. This is the usual way that PCRE is
803 compiled. The output is zero if PCRE was compiled to use blocks of data
804 on the heap instead of recursive function calls. In this case,
805 pcre_stack_malloc and pcre_stack_free are called to manage memory
806 blocks on the heap, thus avoiding the use of the stack.
807
808
809 COMPILING A PATTERN
810
811 pcre *pcre_compile(const char *pattern, int options,
812 const char **errptr, int *erroffset,
813 const unsigned char *tableptr);
814
815 pcre *pcre_compile2(const char *pattern, int options,
816 int *errorcodeptr,
817 const char **errptr, int *erroffset,
818 const unsigned char *tableptr);
819
820 Either of the functions pcre_compile() or pcre_compile2() can be called
821 to compile a pattern into an internal form. The only difference between
822 the two interfaces is that pcre_compile2() has an additional argument,
823 errorcodeptr, via which a numerical error code can be returned.
824
825 The pattern is a C string terminated by a binary zero, and is passed in
826 the pattern argument. A pointer to a single block of memory that is
827 obtained via pcre_malloc is returned. This contains the compiled code
828 and related data. The pcre type is defined for the returned block; this
829 is a typedef for a structure whose contents are not externally defined.
830 It is up to the caller to free the memory when it is no longer
831 required.
832
833 Although the compiled code of a PCRE regex is relocatable, that is, it
834 does not depend on memory location, the complete pcre data block is not
835 fully relocatable, because it may contain a copy of the tableptr argu-
836 ment, which is an address (see below).
837
838 The options argument contains independent bits that affect the compila-
839 tion. It should be zero if no options are required. The available
840 options are described below. Some of them, in particular, those that
841 are compatible with Perl, can also be set and unset from within the
842 pattern (see the detailed description in the pcrepattern documenta-
843 tion). For these options, the contents of the options argument speci-
844 fies their initial settings at the start of compilation and execution.
845 The PCRE_ANCHORED option can be set at the time of matching as well as
846 at compile time.
847
848 If errptr is NULL, pcre_compile() returns NULL immediately. Otherwise,
849 if compilation of a pattern fails, pcre_compile() returns NULL, and
850 sets the variable pointed to by errptr to point to a textual error mes-
851 sage. The offset from the start of the pattern to the character where
852 the error was discovered is placed in the variable pointed to by
853 erroffset, which must not be NULL. If it is, an immediate error is
854 given.
855
856 If pcre_compile2() is used instead of pcre_compile(), and the error-
857 codeptr argument is not NULL, a non-zero error code number is returned
858 via this argument in the event of an error. This is in addition to the
859 textual error message. Error codes and messages are listed below.
860
861 If the final argument, tableptr, is NULL, PCRE uses a default set of
862 character tables that are built when PCRE is compiled, using the
863 default C locale. Otherwise, tableptr must be an address that is the
864 result of a call to pcre_maketables(). This value is stored with the
865 compiled pattern, and used again by pcre_exec(), unless another table
866 pointer is passed to it. For more discussion, see the section on locale
867 support below.
868
869 This code fragment shows a typical straightforward call to pcre_com-
870 pile():
871
872 pcre *re;
873 const char *error;
874 int erroffset;
875 re = pcre_compile(
876 "^A.*Z", /* the pattern */
877 0, /* default options */
878 &error, /* for error message */
879 &erroffset, /* for error offset */
880 NULL); /* use default character tables */
881
882 The following names for option bits are defined in the pcre.h header
883 file:
884
885 PCRE_ANCHORED
886
887 If this bit is set, the pattern is forced to be "anchored", that is, it
888 is constrained to match only at the first matching point in the string
889 that is being searched (the "subject string"). This effect can also be
890 achieved by appropriate constructs in the pattern itself, which is the
891 only way to do it in Perl.
892
893 PCRE_AUTO_CALLOUT
894
895 If this bit is set, pcre_compile() automatically inserts callout items,
896 all with number 255, before each pattern item. For discussion of the
897 callout facility, see the pcrecallout documentation.
898
899 PCRE_CASELESS
900
901 If this bit is set, letters in the pattern match both upper and lower
902 case letters. It is equivalent to Perl's /i option, and it can be
903 changed within a pattern by a (?i) option setting. In UTF-8 mode, PCRE
904 always understands the concept of case for characters whose values are
905 less than 128, so caseless matching is always possible. For characters
906 with higher values, the concept of case is supported if PCRE is com-
907 piled with Unicode property support, but not otherwise. If you want to
908 use caseless matching for characters 128 and above, you must ensure
909 that PCRE is compiled with Unicode property support as well as with
910 UTF-8 support.
911
912 PCRE_DOLLAR_ENDONLY
913
914 If this bit is set, a dollar metacharacter in the pattern matches only
915 at the end of the subject string. Without this option, a dollar also
916 matches immediately before the final character if it is a newline (but
917 not before any other newlines). The PCRE_DOLLAR_ENDONLY option is
918 ignored if PCRE_MULTILINE is set. There is no equivalent to this option
919 in Perl, and no way to set it within a pattern.
920
921 PCRE_DOTALL
922
923 If this bit is set, a dot metacharater in the pattern matches all char-
924 acters, including newlines. Without it, newlines are excluded. This
925 option is equivalent to Perl's /s option, and it can be changed within
926 a pattern by a (?s) option setting. A negative class such as [^a]
927 always matches a newline character, independent of the setting of this
928 option.
929
930 PCRE_EXTENDED
931
932 If this bit is set, whitespace data characters in the pattern are
933 totally ignored except when escaped or inside a character class. White-
934 space does not include the VT character (code 11). In addition, charac-
935 ters between an unescaped # outside a character class and the next new-
936 line character, inclusive, are also ignored. This is equivalent to
937 Perl's /x option, and it can be changed within a pattern by a (?x)
938 option setting.
939
940 This option makes it possible to include comments inside complicated
941 patterns. Note, however, that this applies only to data characters.
942 Whitespace characters may never appear within special character
943 sequences in a pattern, for example within the sequence (?( which
944 introduces a conditional subpattern.
945
946 PCRE_EXTRA
947
948 This option was invented in order to turn on additional functionality
949 of PCRE that is incompatible with Perl, but it is currently of very
950 little use. When set, any backslash in a pattern that is followed by a
951 letter that has no special meaning causes an error, thus reserving
952 these combinations for future expansion. By default, as in Perl, a
953 backslash followed by a letter with no special meaning is treated as a
954 literal. There are at present no other features controlled by this
955 option. It can also be set by a (?X) option setting within a pattern.
956
957 PCRE_FIRSTLINE
958
959 If this option is set, an unanchored pattern is required to match
960 before or at the first newline character in the subject string, though
961 the matched text may continue over the newline.
962
963 PCRE_MULTILINE
964
965 By default, PCRE treats the subject string as consisting of a single
966 line of characters (even if it actually contains newlines). The "start
967 of line" metacharacter (^) matches only at the start of the string,
968 while the "end of line" metacharacter ($) matches only at the end of
969 the string, or before a terminating newline (unless PCRE_DOLLAR_ENDONLY
970 is set). This is the same as Perl.
971
972 When PCRE_MULTILINE it is set, the "start of line" and "end of line"
973 constructs match immediately following or immediately before any new-
974 line in the subject string, respectively, as well as at the very start
975 and end. This is equivalent to Perl's /m option, and it can be changed
976 within a pattern by a (?m) option setting. If there are no "\n" charac-
977 ters in a subject string, or no occurrences of ^ or $ in a pattern,
978 setting PCRE_MULTILINE has no effect.
979
980 PCRE_NO_AUTO_CAPTURE
981
982 If this option is set, it disables the use of numbered capturing paren-
983 theses in the pattern. Any opening parenthesis that is not followed by
984 ? behaves as if it were followed by ?: but named parentheses can still
985 be used for capturing (and they acquire numbers in the usual way).
986 There is no equivalent of this option in Perl.
987
988 PCRE_UNGREEDY
989
990 This option inverts the "greediness" of the quantifiers so that they
991 are not greedy by default, but become greedy if followed by "?". It is
992 not compatible with Perl. It can also be set by a (?U) option setting
993 within the pattern.
994
995 PCRE_UTF8
996
997 This option causes PCRE to regard both the pattern and the subject as
998 strings of UTF-8 characters instead of single-byte character strings.
999 However, it is available only when PCRE is built to include UTF-8 sup-
1000 port. If not, the use of this option provokes an error. Details of how
1001 this option changes the behaviour of PCRE are given in the section on
1002 UTF-8 support in the main pcre page.
1003
1004 PCRE_NO_UTF8_CHECK
1005
1006 When PCRE_UTF8 is set, the validity of the pattern as a UTF-8 string is
1007 automatically checked. If an invalid UTF-8 sequence of bytes is found,
1008 pcre_compile() returns an error. If you already know that your pattern
1009 is valid, and you want to skip this check for performance reasons, you
1010 can set the PCRE_NO_UTF8_CHECK option. When it is set, the effect of
1011 passing an invalid UTF-8 string as a pattern is undefined. It may cause
1012 your program to crash. Note that this option can also be passed to
1013 pcre_exec() and pcre_dfa_exec(), to suppress the UTF-8 validity check-
1014 ing of subject strings.
1015
1016
1017 COMPILATION ERROR CODES
1018
1019 The following table lists the error codes than may be returned by
1020 pcre_compile2(), along with the error messages that may be returned by
1021 both compiling functions.
1022
1023 0 no error
1024 1 \ at end of pattern
1025 2 \c at end of pattern
1026 3 unrecognized character follows \
1027 4 numbers out of order in {} quantifier
1028 5 number too big in {} quantifier
1029 6 missing terminating ] for character class
1030 7 invalid escape sequence in character class
1031 8 range out of order in character class
1032 9 nothing to repeat
1033 10 operand of unlimited repeat could match the empty string
1034 11 internal error: unexpected repeat
1035 12 unrecognized character after (?
1036 13 POSIX named classes are supported only within a class
1037 14 missing )
1038 15 reference to non-existent subpattern
1039 16 erroffset passed as NULL
1040 17 unknown option bit(s) set
1041 18 missing ) after comment
1042 19 parentheses nested too deeply
1043 20 regular expression too large
1044 21 failed to get memory
1045 22 unmatched parentheses
1046 23 internal error: code overflow
1047 24 unrecognized character after (?<
1048 25 lookbehind assertion is not fixed length
1049 26 malformed number after (?(
1050 27 conditional group contains more than two branches
1051 28 assertion expected after (?(
1052 29 (?R or (?digits must be followed by )
1053 30 unknown POSIX class name
1054 31 POSIX collating elements are not supported
1055 32 this version of PCRE is not compiled with PCRE_UTF8 support
1056 33 spare error
1057 34 character value in \x{...} sequence is too large
1058 35 invalid condition (?(0)
1059 36 \C not allowed in lookbehind assertion
1060 37 PCRE does not support \L, \l, \N, \U, or \u
1061 38 number after (?C is > 255
1062 39 closing ) for (?C expected
1063 40 recursive call could loop indefinitely
1064 41 unrecognized character after (?P
1065 42 syntax error after (?P
1066 43 two named groups have the same name
1067 44 invalid UTF-8 string
1068 45 support for \P, \p, and \X has not been compiled
1069 46 malformed \P or \p sequence
1070 47 unknown property name after \P or \p
1071
1072
1073 STUDYING A PATTERN
1074
1075 pcre_extra *pcre_study(const pcre *code, int options
1076 const char **errptr);
1077
1078 If a compiled pattern is going to be used several times, it is worth
1079 spending more time analyzing it in order to speed up the time taken for
1080 matching. The function pcre_study() takes a pointer to a compiled pat-
1081 tern as its first argument. If studying the pattern produces additional
1082 information that will help speed up matching, pcre_study() returns a
1083 pointer to a pcre_extra block, in which the study_data field points to
1084 the results of the study.
1085
1086 The returned value from pcre_study() can be passed directly to
1087 pcre_exec(). However, a pcre_extra block also contains other fields
1088 that can be set by the caller before the block is passed; these are
1089 described below in the section on matching a pattern.
1090
1091 If studying the pattern does not produce any additional information
1092 pcre_study() returns NULL. In that circumstance, if the calling program
1093 wants to pass any of the other fields to pcre_exec(), it must set up
1094 its own pcre_extra block.
1095
1096 The second argument of pcre_study() contains option bits. At present,
1097 no options are defined, and this argument should always be zero.
1098
1099 The third argument for pcre_study() is a pointer for an error message.
1100 If studying succeeds (even if no data is returned), the variable it
1101 points to is set to NULL. Otherwise it points to a textual error mes-
1102 sage. You should therefore test the error pointer for NULL after call-
1103 ing pcre_study(), to be sure that it has run successfully.
1104
1105 This is a typical call to pcre_study():
1106
1107 pcre_extra *pe;
1108 pe = pcre_study(
1109 re, /* result of pcre_compile() */
1110 0, /* no options exist */
1111 &error); /* set to NULL or points to a message */
1112
1113 At present, studying a pattern is useful only for non-anchored patterns
1114 that do not have a single fixed starting character. A bitmap of possi-
1115 ble starting bytes is created.
1116
1117
1118 LOCALE SUPPORT
1119
1120 PCRE handles caseless matching, and determines whether characters are
1121 letters digits, or whatever, by reference to a set of tables, indexed
1122 by character value. When running in UTF-8 mode, this applies only to
1123 characters with codes less than 128. Higher-valued codes never match
1124 escapes such as \w or \d, but can be tested with \p if PCRE is built
1125 with Unicode character property support.
1126
1127 An internal set of tables is created in the default C locale when PCRE
1128 is built. This is used when the final argument of pcre_compile() is
1129 NULL, and is sufficient for many applications. An alternative set of
1130 tables can, however, be supplied. These may be created in a different
1131 locale from the default. As more and more applications change to using
1132 Unicode, the need for this locale support is expected to die away.
1133
1134 External tables are built by calling the pcre_maketables() function,
1135 which has no arguments, in the relevant locale. The result can then be
1136 passed to pcre_compile() or pcre_exec() as often as necessary. For
1137 example, to build and use tables that are appropriate for the French
1138 locale (where accented characters with values greater than 128 are
1139 treated as letters), the following code could be used:
1140
1141 setlocale(LC_CTYPE, "fr_FR");
1142 tables = pcre_maketables();
1143 re = pcre_compile(..., tables);
1144
1145 When pcre_maketables() runs, the tables are built in memory that is
1146 obtained via pcre_malloc. It is the caller's responsibility to ensure
1147 that the memory containing the tables remains available for as long as
1148 it is needed.
1149
1150 The pointer that is passed to pcre_compile() is saved with the compiled
1151 pattern, and the same tables are used via this pointer by pcre_study()
1152 and normally also by pcre_exec(). Thus, by default, for any single pat-
1153 tern, compilation, studying and matching all happen in the same locale,
1154 but different patterns can be compiled in different locales.
1155
1156 It is possible to pass a table pointer or NULL (indicating the use of
1157 the internal tables) to pcre_exec(). Although not intended for this
1158 purpose, this facility could be used to match a pattern in a different
1159 locale from the one in which it was compiled. Passing table pointers at
1160 run time is discussed below in the section on matching a pattern.
1161
1162
1163 INFORMATION ABOUT A PATTERN
1164
1165 int pcre_fullinfo(const pcre *code, const pcre_extra *extra,
1166 int what, void *where);
1167
1168 The pcre_fullinfo() function returns information about a compiled pat-
1169 tern. It replaces the obsolete pcre_info() function, which is neverthe-
1170 less retained for backwards compability (and is documented below).
1171
1172 The first argument for pcre_fullinfo() is a pointer to the compiled
1173 pattern. The second argument is the result of pcre_study(), or NULL if
1174 the pattern was not studied. The third argument specifies which piece
1175 of information is required, and the fourth argument is a pointer to a
1176 variable to receive the data. The yield of the function is zero for
1177 success, or one of the following negative numbers:
1178
1179 PCRE_ERROR_NULL the argument code was NULL
1180 the argument where was NULL
1181 PCRE_ERROR_BADMAGIC the "magic number" was not found
1182 PCRE_ERROR_BADOPTION the value of what was invalid
1183
1184 The "magic number" is placed at the start of each compiled pattern as
1185 an simple check against passing an arbitrary memory pointer. Here is a
1186 typical call of pcre_fullinfo(), to obtain the length of the compiled
1187 pattern:
1188
1189 int rc;
1190 unsigned long int length;
1191 rc = pcre_fullinfo(
1192 re, /* result of pcre_compile() */
1193 pe, /* result of pcre_study(), or NULL */
1194 PCRE_INFO_SIZE, /* what is required */
1195 &length); /* where to put the data */
1196
1197 The possible values for the third argument are defined in pcre.h, and
1198 are as follows:
1199
1200 PCRE_INFO_BACKREFMAX
1201
1202 Return the number of the highest back reference in the pattern. The
1203 fourth argument should point to an int variable. Zero is returned if
1204 there are no back references.
1205
1206 PCRE_INFO_CAPTURECOUNT
1207
1208 Return the number of capturing subpatterns in the pattern. The fourth
1209 argument should point to an int variable.
1210
1211 PCRE_INFO_DEFAULT_TABLES
1212
1213 Return a pointer to the internal default character tables within PCRE.
1214 The fourth argument should point to an unsigned char * variable. This
1215 information call is provided for internal use by the pcre_study() func-
1216 tion. External callers can cause PCRE to use its internal tables by
1217 passing a NULL table pointer.
1218
1219 PCRE_INFO_FIRSTBYTE
1220
1221 Return information about the first byte of any matched string, for a
1222 non-anchored pattern. (This option used to be called
1223 PCRE_INFO_FIRSTCHAR; the old name is still recognized for backwards
1224 compatibility.)
1225
1226 If there is a fixed first byte, for example, from a pattern such as
1227 (cat|cow|coyote), it is returned in the integer pointed to by where.
1228 Otherwise, if either
1229
1230 (a) the pattern was compiled with the PCRE_MULTILINE option, and every
1231 branch starts with "^", or
1232
1233 (b) every branch of the pattern starts with ".*" and PCRE_DOTALL is not
1234 set (if it were set, the pattern would be anchored),
1235
1236 -1 is returned, indicating that the pattern matches only at the start
1237 of a subject string or after any newline within the string. Otherwise
1238 -2 is returned. For anchored patterns, -2 is returned.
1239
1240 PCRE_INFO_FIRSTTABLE
1241
1242 If the pattern was studied, and this resulted in the construction of a
1243 256-bit table indicating a fixed set of bytes for the first byte in any
1244 matching string, a pointer to the table is returned. Otherwise NULL is
1245 returned. The fourth argument should point to an unsigned char * vari-
1246 able.
1247
1248 PCRE_INFO_LASTLITERAL
1249
1250 Return the value of the rightmost literal byte that must exist in any
1251 matched string, other than at its start, if such a byte has been
1252 recorded. The fourth argument should point to an int variable. If there
1253 is no such byte, -1 is returned. For anchored patterns, a last literal
1254 byte is recorded only if it follows something of variable length. For
1255 example, for the pattern /^a\d+z\d+/ the returned value is "z", but for
1256 /^a\dz\d/ the returned value is -1.
1257
1258 PCRE_INFO_NAMECOUNT
1259 PCRE_INFO_NAMEENTRYSIZE
1260 PCRE_INFO_NAMETABLE
1261
1262 PCRE supports the use of named as well as numbered capturing parenthe-
1263 ses. The names are just an additional way of identifying the parenthe-
1264 ses, which still acquire numbers. A convenience function called
1265 pcre_get_named_substring() is provided for extracting an individual
1266 captured substring by name. It is also possible to extract the data
1267 directly, by first converting the name to a number in order to access
1268 the correct pointers in the output vector (described with pcre_exec()
1269 below). To do the conversion, you need to use the name-to-number map,
1270 which is described by these three values.
1271
1272 The map consists of a number of fixed-size entries. PCRE_INFO_NAMECOUNT
1273 gives the number of entries, and PCRE_INFO_NAMEENTRYSIZE gives the size
1274 of each entry; both of these return an int value. The entry size
1275 depends on the length of the longest name. PCRE_INFO_NAMETABLE returns
1276 a pointer to the first entry of the table (a pointer to char). The
1277 first two bytes of each entry are the number of the capturing parenthe-
1278 sis, most significant byte first. The rest of the entry is the corre-
1279 sponding name, zero terminated. The names are in alphabetical order.
1280 For example, consider the following pattern (assume PCRE_EXTENDED is
1281 set, so white space - including newlines - is ignored):
1282
1283 (?P<date> (?P<year>(\d\d)?\d\d) -
1284 (?P<month>\d\d) - (?P<day>\d\d) )
1285
1286 There are four named subpatterns, so the table has four entries, and
1287 each entry in the table is eight bytes long. The table is as follows,
1288 with non-printing bytes shows in hexadecimal, and undefined bytes shown
1289 as ??:
1290
1291 00 01 d a t e 00 ??
1292 00 05 d a y 00 ?? ??
1293 00 04 m o n t h 00
1294 00 02 y e a r 00 ??
1295
1296 When writing code to extract data from named subpatterns using the
1297 name-to-number map, remember that the length of each entry is likely to
1298 be different for each compiled pattern.
1299
1300 PCRE_INFO_OPTIONS
1301
1302 Return a copy of the options with which the pattern was compiled. The
1303 fourth argument should point to an unsigned long int variable. These
1304 option bits are those specified in the call to pcre_compile(), modified
1305 by any top-level option settings within the pattern itself.
1306
1307 A pattern is automatically anchored by PCRE if all of its top-level
1308 alternatives begin with one of the following:
1309
1310 ^ unless PCRE_MULTILINE is set
1311 \A always
1312 \G always
1313 .* if PCRE_DOTALL is set and there are no back
1314 references to the subpattern in which .* appears
1315
1316 For such patterns, the PCRE_ANCHORED bit is set in the options returned
1317 by pcre_fullinfo().
1318
1319 PCRE_INFO_SIZE
1320
1321 Return the size of the compiled pattern, that is, the value that was
1322 passed as the argument to pcre_malloc() when PCRE was getting memory in
1323 which to place the compiled data. The fourth argument should point to a
1324 size_t variable.
1325
1326 PCRE_INFO_STUDYSIZE
1327
1328 Return the size of the data block pointed to by the study_data field in
1329 a pcre_extra block. That is, it is the value that was passed to
1330 pcre_malloc() when PCRE was getting memory into which to place the data
1331 created by pcre_study(). The fourth argument should point to a size_t
1332 variable.
1333
1334
1335 OBSOLETE INFO FUNCTION
1336
1337 int pcre_info(const pcre *code, int *optptr, int *firstcharptr);
1338
1339 The pcre_info() function is now obsolete because its interface is too
1340 restrictive to return all the available data about a compiled pattern.
1341 New programs should use pcre_fullinfo() instead. The yield of
1342 pcre_info() is the number of capturing subpatterns, or one of the fol-
1343 lowing negative numbers:
1344
1345 PCRE_ERROR_NULL the argument code was NULL
1346 PCRE_ERROR_BADMAGIC the "magic number" was not found
1347
1348 If the optptr argument is not NULL, a copy of the options with which
1349 the pattern was compiled is placed in the integer it points to (see
1350 PCRE_INFO_OPTIONS above).
1351
1352 If the pattern is not anchored and the firstcharptr argument is not
1353 NULL, it is used to pass back information about the first character of
1354 any matched string (see PCRE_INFO_FIRSTBYTE above).
1355
1356
1357 REFERENCE COUNTS
1358
1359 int pcre_refcount(pcre *code, int adjust);
1360
1361 The pcre_refcount() function is used to maintain a reference count in
1362 the data block that contains a compiled pattern. It is provided for the
1363 benefit of applications that operate in an object-oriented manner,
1364 where different parts of the application may be using the same compiled
1365 pattern, but you want to free the block when they are all done.
1366
1367 When a pattern is compiled, the reference count field is initialized to
1368 zero. It is changed only by calling this function, whose action is to
1369 add the adjust value (which may be positive or negative) to it. The
1370 yield of the function is the new value. However, the value of the count
1371 is constrained to lie between 0 and 65535, inclusive. If the new value
1372 is outside these limits, it is forced to the appropriate limit value.
1373
1374 Except when it is zero, the reference count is not correctly preserved
1375 if a pattern is compiled on one host and then transferred to a host
1376 whose byte-order is different. (This seems a highly unlikely scenario.)
1377
1378
1379 MATCHING A PATTERN: THE TRADITIONAL FUNCTION
1380
1381 int pcre_exec(const pcre *code, const pcre_extra *extra,
1382 const char *subject, int length, int startoffset,
1383 int options, int *ovector, int ovecsize);
1384
1385 The function pcre_exec() is called to match a subject string against a
1386 compiled pattern, which is passed in the code argument. If the pattern
1387 has been studied, the result of the study should be passed in the extra
1388 argument. This function is the main matching facility of the library,
1389 and it operates in a Perl-like manner. For specialist use there is also
1390 an alternative matching function, which is described below in the sec-
1391 tion about the pcre_dfa_exec() function.
1392
1393 In most applications, the pattern will have been compiled (and option-
1394 ally studied) in the same process that calls pcre_exec(). However, it
1395 is possible to save compiled patterns and study data, and then use them
1396 later in different processes, possibly even on different hosts. For a
1397 discussion about this, see the pcreprecompile documentation.
1398
1399 Here is an example of a simple call to pcre_exec():
1400
1401 int rc;
1402 int ovector[30];
1403 rc = pcre_exec(
1404 re, /* result of pcre_compile() */
1405 NULL, /* we didn't study the pattern */
1406 "some string", /* the subject string */
1407 11, /* the length of the subject string */
1408 0, /* start at offset 0 in the subject */
1409 0, /* default options */
1410 ovector, /* vector of integers for substring information */
1411 30); /* number of elements (NOT size in bytes) */
1412
1413 Extra data for pcre_exec()
1414
1415 If the extra argument is not NULL, it must point to a pcre_extra data
1416 block. The pcre_study() function returns such a block (when it doesn't
1417 return NULL), but you can also create one for yourself, and pass addi-
1418 tional information in it. The fields in a pcre_extra block are as fol-
1419 lows:
1420
1421 unsigned long int flags;
1422 void *study_data;
1423 unsigned long int match_limit;
1424 void *callout_data;
1425 const unsigned char *tables;
1426
1427 The flags field is a bitmap that specifies which of the other fields
1428 are set. The flag bits are:
1429
1430 PCRE_EXTRA_STUDY_DATA
1431 PCRE_EXTRA_MATCH_LIMIT
1432 PCRE_EXTRA_CALLOUT_DATA
1433 PCRE_EXTRA_TABLES
1434
1435 Other flag bits should be set to zero. The study_data field is set in
1436 the pcre_extra block that is returned by pcre_study(), together with
1437 the appropriate flag bit. You should not set this yourself, but you may
1438 add to the block by setting the other fields and their corresponding
1439 flag bits.
1440
1441 The match_limit field provides a means of preventing PCRE from using up
1442 a vast amount of resources when running patterns that are not going to
1443 match, but which have a very large number of possibilities in their
1444 search trees. The classic example is the use of nested unlimited
1445 repeats.
1446
1447 Internally, PCRE uses a function called match() which it calls repeat-
1448 edly (sometimes recursively). The limit is imposed on the number of
1449 times this function is called during a match, which has the effect of
1450 limiting the amount of recursion and backtracking that can take place.
1451 For patterns that are not anchored, the count starts from zero for each
1452 position in the subject string.
1453
1454 The default limit for the library can be set when PCRE is built; the
1455 default default is 10 million, which handles all but the most extreme
1456 cases. You can reduce the default by suppling pcre_exec() with a
1457 pcre_extra block in which match_limit is set to a smaller value, and
1458 PCRE_EXTRA_MATCH_LIMIT is set in the flags field. If the limit is
1459 exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
1460
1461 The pcre_callout field is used in conjunction with the "callout" fea-
1462 ture, which is described in the pcrecallout documentation.
1463
1464 The tables field is used to pass a character tables pointer to
1465 pcre_exec(); this overrides the value that is stored with the compiled
1466 pattern. A non-NULL value is stored with the compiled pattern only if
1467 custom tables were supplied to pcre_compile() via its tableptr argu-
1468 ment. If NULL is passed to pcre_exec() using this mechanism, it forces
1469 PCRE's internal tables to be used. This facility is helpful when re-
1470 using patterns that have been saved after compiling with an external
1471 set of tables, because the external tables might be at a different
1472 address when pcre_exec() is called. See the pcreprecompile documenta-
1473 tion for a discussion of saving compiled patterns for later use.
1474
1475 Option bits for pcre_exec()
1476
1477 The unused bits of the options argument for pcre_exec() must be zero.
1478 The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL,
1479 PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK and PCRE_PARTIAL.
1480
1481 PCRE_ANCHORED
1482
1483 The PCRE_ANCHORED option limits pcre_exec() to matching at the first
1484 matching position. If a pattern was compiled with PCRE_ANCHORED, or
1485 turned out to be anchored by virtue of its contents, it cannot be made
1486 unachored at matching time.
1487
1488 PCRE_NOTBOL
1489
1490 This option specifies that first character of the subject string is not
1491 the beginning of a line, so the circumflex metacharacter should not
1492 match before it. Setting this without PCRE_MULTILINE (at compile time)
1493 causes circumflex never to match. This option affects only the behav-
1494 iour of the circumflex metacharacter. It does not affect \A.
1495
1496 PCRE_NOTEOL
1497
1498 This option specifies that the end of the subject string is not the end
1499 of a line, so the dollar metacharacter should not match it nor (except
1500 in multiline mode) a newline immediately before it. Setting this with-
1501 out PCRE_MULTILINE (at compile time) causes dollar never to match. This
1502 option affects only the behaviour of the dollar metacharacter. It does
1503 not affect \Z or \z.
1504
1505 PCRE_NOTEMPTY
1506
1507 An empty string is not considered to be a valid match if this option is
1508 set. If there are alternatives in the pattern, they are tried. If all
1509 the alternatives match the empty string, the entire match fails. For
1510 example, if the pattern
1511
1512 a?b?
1513
1514 is applied to a string not beginning with "a" or "b", it matches the
1515 empty string at the start of the subject. With PCRE_NOTEMPTY set, this
1516 match is not valid, so PCRE searches further into the string for occur-
1517 rences of "a" or "b".
1518
1519 Perl has no direct equivalent of PCRE_NOTEMPTY, but it does make a spe-
1520 cial case of a pattern match of the empty string within its split()
1521 function, and when using the /g modifier. It is possible to emulate
1522 Perl's behaviour after matching a null string by first trying the match
1523 again at the same offset with PCRE_NOTEMPTY and PCRE_ANCHORED, and then
1524 if that fails by advancing the starting offset (see below) and trying
1525 an ordinary match again. There is some code that demonstrates how to do
1526 this in the pcredemo.c sample program.
1527
1528 PCRE_NO_UTF8_CHECK
1529
1530 When PCRE_UTF8 is set at compile time, the validity of the subject as a
1531 UTF-8 string is automatically checked when pcre_exec() is subsequently
1532 called. The value of startoffset is also checked to ensure that it
1533 points to the start of a UTF-8 character. If an invalid UTF-8 sequence
1534 of bytes is found, pcre_exec() returns the error PCRE_ERROR_BADUTF8. If
1535 startoffset contains an invalid value, PCRE_ERROR_BADUTF8_OFFSET is
1536 returned.
1537
1538 If you already know that your subject is valid, and you want to skip
1539 these checks for performance reasons, you can set the
1540 PCRE_NO_UTF8_CHECK option when calling pcre_exec(). You might want to
1541 do this for the second and subsequent calls to pcre_exec() if you are
1542 making repeated calls to find all the matches in a single subject
1543 string. However, you should be sure that the value of startoffset
1544 points to the start of a UTF-8 character. When PCRE_NO_UTF8_CHECK is
1545 set, the effect of passing an invalid UTF-8 string as a subject, or a
1546 value of startoffset that does not point to the start of a UTF-8 char-
1547 acter, is undefined. Your program may crash.
1548
1549 PCRE_PARTIAL
1550
1551 This option turns on the partial matching feature. If the subject
1552 string fails to match the pattern, but at some point during the match-
1553 ing process the end of the subject was reached (that is, the subject
1554 partially matches the pattern and the failure to match occurred only
1555 because there were not enough subject characters), pcre_exec() returns
1556 PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. When PCRE_PARTIAL is
1557 used, there are restrictions on what may appear in the pattern. These
1558 are discussed in the pcrepartial documentation.
1559
1560 The string to be matched by pcre_exec()
1561
1562 The subject string is passed to pcre_exec() as a pointer in subject, a
1563 length in length, and a starting byte offset in startoffset. In UTF-8
1564 mode, the byte offset must point to the start of a UTF-8 character.
1565 Unlike the pattern string, the subject may contain binary zero bytes.
1566 When the starting offset is zero, the search for a match starts at the
1567 beginning of the subject, and this is by far the most common case.
1568
1569 A non-zero starting offset is useful when searching for another match
1570 in the same subject by calling pcre_exec() again after a previous suc-
1571 cess. Setting startoffset differs from just passing over a shortened
1572 string and setting PCRE_NOTBOL in the case of a pattern that begins
1573 with any kind of lookbehind. For example, consider the pattern
1574
1575 \Biss\B
1576
1577 which finds occurrences of "iss" in the middle of words. (\B matches
1578 only if the current position in the subject is not a word boundary.)
1579 When applied to the string "Mississipi" the first call to pcre_exec()
1580 finds the first occurrence. If pcre_exec() is called again with just
1581 the remainder of the subject, namely "issipi", it does not match,
1582 because \B is always false at the start of the subject, which is deemed
1583 to be a word boundary. However, if pcre_exec() is passed the entire
1584 string again, but with startoffset set to 4, it finds the second occur-
1585 rence of "iss" because it is able to look behind the starting point to
1586 discover that it is preceded by a letter.
1587
1588 If a non-zero starting offset is passed when the pattern is anchored,
1589 one attempt to match at the given offset is made. This can only succeed
1590 if the pattern does not require the match to be at the start of the
1591 subject.
1592
1593 How pcre_exec() returns captured substrings
1594
1595 In general, a pattern matches a certain portion of the subject, and in
1596 addition, further substrings from the subject may be picked out by
1597 parts of the pattern. Following the usage in Jeffrey Friedl's book,
1598 this is called "capturing" in what follows, and the phrase "capturing
1599 subpattern" is used for a fragment of a pattern that picks out a sub-
1600 string. PCRE supports several other kinds of parenthesized subpattern
1601 that do not cause substrings to be captured.
1602
1603 Captured substrings are returned to the caller via a vector of integer
1604 offsets whose address is passed in ovector. The number of elements in
1605 the vector is passed in ovecsize, which must be a non-negative number.
1606 Note: this argument is NOT the size of ovector in bytes.
1607
1608 The first two-thirds of the vector is used to pass back captured sub-
1609 strings, each substring using a pair of integers. The remaining third
1610 of the vector is used as workspace by pcre_exec() while matching cap-
1611 turing subpatterns, and is not available for passing back information.
1612 The length passed in ovecsize should always be a multiple of three. If
1613 it is not, it is rounded down.
1614
1615 When a match is successful, information about captured substrings is
1616 returned in pairs of integers, starting at the beginning of ovector,
1617 and continuing up to two-thirds of its length at the most. The first
1618 element of a pair is set to the offset of the first character in a sub-
1619 string, and the second is set to the offset of the first character
1620 after the end of a substring. The first pair, ovector[0] and ovec-
1621 tor[1], identify the portion of the subject string matched by the
1622 entire pattern. The next pair is used for the first capturing subpat-
1623 tern, and so on. The value returned by pcre_exec() is the number of
1624 pairs that have been set. If there are no capturing subpatterns, the
1625 return value from a successful match is 1, indicating that just the
1626 first pair of offsets has been set.
1627
1628 Some convenience functions are provided for extracting the captured
1629 substrings as separate strings. These are described in the following
1630 section.
1631
1632 It is possible for an capturing subpattern number n+1 to match some
1633 part of the subject when subpattern n has not been used at all. For
1634 example, if the string "abc" is matched against the pattern (a|(z))(bc)
1635 subpatterns 1 and 3 are matched, but 2 is not. When this happens, both
1636 offset values corresponding to the unused subpattern are set to -1.
1637
1638 If a capturing subpattern is matched repeatedly, it is the last portion
1639 of the string that it matched that is returned.
1640
1641 If the vector is too small to hold all the captured substring offsets,
1642 it is used as far as possible (up to two-thirds of its length), and the
1643 function returns a value of zero. In particular, if the substring off-
1644 sets are not of interest, pcre_exec() may be called with ovector passed
1645 as NULL and ovecsize as zero. However, if the pattern contains back
1646 references and the ovector is not big enough to remember the related
1647 substrings, PCRE has to get additional memory for use during matching.
1648 Thus it is usually advisable to supply an ovector.
1649
1650 Note that pcre_info() can be used to find out how many capturing sub-
1651 patterns there are in a compiled pattern. The smallest size for ovector
1652 that will allow for n captured substrings, in addition to the offsets
1653 of the substring matched by the whole pattern, is (n+1)*3.
1654
1655 Return values from pcre_exec()
1656
1657 If pcre_exec() fails, it returns a negative number. The following are
1658 defined in the header file:
1659
1660 PCRE_ERROR_NOMATCH (-1)
1661
1662 The subject string did not match the pattern.
1663
1664 PCRE_ERROR_NULL (-2)
1665
1666 Either code or subject was passed as NULL, or ovector was NULL and
1667 ovecsize was not zero.
1668
1669 PCRE_ERROR_BADOPTION (-3)
1670
1671 An unrecognized bit was set in the options argument.
1672
1673 PCRE_ERROR_BADMAGIC (-4)
1674
1675 PCRE stores a 4-byte "magic number" at the start of the compiled code,
1676 to catch the case when it is passed a junk pointer and to detect when a
1677 pattern that was compiled in an environment of one endianness is run in
1678 an environment with the other endianness. This is the error that PCRE
1679 gives when the magic number is not present.
1680
1681 PCRE_ERROR_UNKNOWN_NODE (-5)
1682
1683 While running the pattern match, an unknown item was encountered in the
1684 compiled pattern. This error could be caused by a bug in PCRE or by
1685 overwriting of the compiled pattern.
1686
1687 PCRE_ERROR_NOMEMORY (-6)
1688
1689 If a pattern contains back references, but the ovector that is passed
1690 to pcre_exec() is not big enough to remember the referenced substrings,
1691 PCRE gets a block of memory at the start of matching to use for this
1692 purpose. If the call via pcre_malloc() fails, this error is given. The
1693 memory is automatically freed at the end of matching.
1694
1695 PCRE_ERROR_NOSUBSTRING (-7)
1696
1697 This error is used by the pcre_copy_substring(), pcre_get_substring(),
1698 and pcre_get_substring_list() functions (see below). It is never
1699 returned by pcre_exec().
1700
1701 PCRE_ERROR_MATCHLIMIT (-8)
1702
1703 The recursion and backtracking limit, as specified by the match_limit
1704 field in a pcre_extra structure (or defaulted) was reached. See the
1705 description above.
1706
1707 PCRE_ERROR_CALLOUT (-9)
1708
1709 This error is never generated by pcre_exec() itself. It is provided for
1710 use by callout functions that want to yield a distinctive error code.
1711 See the pcrecallout documentation for details.
1712
1713 PCRE_ERROR_BADUTF8 (-10)
1714
1715 A string that contains an invalid UTF-8 byte sequence was passed as a
1716 subject.
1717
1718 PCRE_ERROR_BADUTF8_OFFSET (-11)
1719
1720 The UTF-8 byte sequence that was passed as a subject was valid, but the
1721 value of startoffset did not point to the beginning of a UTF-8 charac-
1722 ter.
1723
1724 PCRE_ERROR_PARTIAL (-12)
1725
1726 The subject string did not match, but it did match partially. See the
1727 pcrepartial documentation for details of partial matching.
1728
1729 PCRE_ERROR_BADPARTIAL (-13)
1730
1731 The PCRE_PARTIAL option was used with a compiled pattern containing
1732 items that are not supported for partial matching. See the pcrepartial
1733 documentation for details of partial matching.
1734
1735 PCRE_ERROR_INTERNAL (-14)
1736
1737 An unexpected internal error has occurred. This error could be caused
1738 by a bug in PCRE or by overwriting of the compiled pattern.
1739
1740 PCRE_ERROR_BADCOUNT (-15)
1741
1742 This error is given if the value of the ovecsize argument is negative.
1743
1744
1745 EXTRACTING CAPTURED SUBSTRINGS BY NUMBER
1746
1747 int pcre_copy_substring(const char *subject, int *ovector,
1748 int stringcount, int stringnumber, char *buffer,
1749 int buffersize);
1750
1751 int pcre_get_substring(const char *subject, int *ovector,
1752 int stringcount, int stringnumber,
1753 const char **stringptr);
1754
1755 int pcre_get_substring_list(const char *subject,
1756 int *ovector, int stringcount, const char ***listptr);
1757
1758 Captured substrings can be accessed directly by using the offsets
1759 returned by pcre_exec() in ovector. For convenience, the functions
1760 pcre_copy_substring(), pcre_get_substring(), and pcre_get_sub-
1761 string_list() are provided for extracting captured substrings as new,
1762 separate, zero-terminated strings. These functions identify substrings
1763 by number. The next section describes functions for extracting named
1764 substrings. A substring that contains a binary zero is correctly
1765 extracted and has a further zero added on the end, but the result is
1766 not, of course, a C string.
1767
1768 The first three arguments are the same for all three of these func-
1769 tions: subject is the subject string that has just been successfully
1770 matched, ovector is a pointer to the vector of integer offsets that was
1771 passed to pcre_exec(), and stringcount is the number of substrings that
1772 were captured by the match, including the substring that matched the
1773 entire regular expression. This is the value returned by pcre_exec() if
1774 it is greater than zero. If pcre_exec() returned zero, indicating that
1775 it ran out of space in ovector, the value passed as stringcount should
1776 be the number of elements in the vector divided by three.
1777
1778 The functions pcre_copy_substring() and pcre_get_substring() extract a
1779 single substring, whose number is given as stringnumber. A value of
1780 zero extracts the substring that matched the entire pattern, whereas
1781 higher values extract the captured substrings. For pcre_copy_sub-
1782 string(), the string is placed in buffer, whose length is given by
1783 buffersize, while for pcre_get_substring() a new block of memory is
1784 obtained via pcre_malloc, and its address is returned via stringptr.
1785 The yield of the function is the length of the string, not including
1786 the terminating zero, or one of
1787
1788 PCRE_ERROR_NOMEMORY (-6)
1789
1790 The buffer was too small for pcre_copy_substring(), or the attempt to
1791 get memory failed for pcre_get_substring().
1792
1793 PCRE_ERROR_NOSUBSTRING (-7)
1794
1795 There is no substring whose number is stringnumber.
1796
1797 The pcre_get_substring_list() function extracts all available sub-
1798 strings and builds a list of pointers to them. All this is done in a
1799 single block of memory that is obtained via pcre_malloc. The address of
1800 the memory block is returned via listptr, which is also the start of
1801 the list of string pointers. The end of the list is marked by a NULL
1802 pointer. The yield of the function is zero if all went well, or
1803
1804 PCRE_ERROR_NOMEMORY (-6)
1805
1806 if the attempt to get the memory block failed.
1807
1808 When any of these functions encounter a substring that is unset, which
1809 can happen when capturing subpattern number n+1 matches some part of
1810 the subject, but subpattern n has not been used at all, they return an
1811 empty string. This can be distinguished from a genuine zero-length sub-
1812 string by inspecting the appropriate offset in ovector, which is nega-
1813 tive for unset substrings.
1814
1815 The two convenience functions pcre_free_substring() and pcre_free_sub-
1816 string_list() can be used to free the memory returned by a previous
1817 call of pcre_get_substring() or pcre_get_substring_list(), respec-
1818 tively. They do nothing more than call the function pointed to by
1819 pcre_free, which of course could be called directly from a C program.
1820 However, PCRE is used in some situations where it is linked via a spe-
1821 cial interface to another programming language which cannot use
1822 pcre_free directly; it is for these cases that the functions are pro-
1823 vided.
1824
1825
1826 EXTRACTING CAPTURED SUBSTRINGS BY NAME
1827
1828 int pcre_get_stringnumber(const pcre *code,
1829 const char *name);
1830
1831 int pcre_copy_named_substring(const pcre *code,
1832 const char *subject, int *ovector,
1833 int stringcount, const char *stringname,
1834 char *buffer, int buffersize);
1835
1836 int pcre_get_named_substring(const pcre *code,
1837 const char *subject, int *ovector,
1838 int stringcount, const char *stringname,
1839 const char **stringptr);
1840
1841 To extract a substring by name, you first have to find associated num-
1842 ber. For example, for this pattern
1843
1844 (a+)b(?P<xxx>\d+)...
1845
1846 the number of the subpattern called "xxx" is 2. You can find the number
1847 from the name by calling pcre_get_stringnumber(). The first argument is
1848 the compiled pattern, and the second is the name. The yield of the
1849 function is the subpattern number, or PCRE_ERROR_NOSUBSTRING (-7) if
1850 there is no subpattern of that name.
1851
1852 Given the number, you can extract the substring directly, or use one of
1853 the functions described in the previous section. For convenience, there
1854 are also two functions that do the whole job.
1855
1856 Most of the arguments of pcre_copy_named_substring() and
1857 pcre_get_named_substring() are the same as those for the similarly
1858 named functions that extract by number. As these are described in the
1859 previous section, they are not re-described here. There are just two
1860 differences:
1861
1862 First, instead of a substring number, a substring name is given. Sec-
1863 ond, there is an extra argument, given at the start, which is a pointer
1864 to the compiled pattern. This is needed in order to gain access to the
1865 name-to-number translation table.
1866
1867 These functions call pcre_get_stringnumber(), and if it succeeds, they
1868 then call pcre_copy_substring() or pcre_get_substring(), as appropri-
1869 ate.
1870
1871
1872 FINDING ALL POSSIBLE MATCHES
1873
1874 The traditional matching function uses a similar algorithm to Perl,
1875 which stops when it finds the first match, starting at a given point in
1876 the subject. If you want to find all possible matches, or the longest
1877 possible match, consider using the alternative matching function (see
1878 below) instead. If you cannot use the alternative function, but still
1879 need to find all possible matches, you can kludge it up by making use
1880 of the callout facility, which is described in the pcrecallout documen-
1881 tation.
1882
1883 What you have to do is to insert a callout right at the end of the pat-
1884 tern. When your callout function is called, extract and save the cur-
1885 rent matched substring. Then return 1, which forces pcre_exec() to
1886 backtrack and try other alternatives. Ultimately, when it runs out of
1887 matches, pcre_exec() will yield PCRE_ERROR_NOMATCH.
1888
1889
1890 MATCHING A PATTERN: THE ALTERNATIVE FUNCTION
1891
1892 int pcre_dfa_exec(const pcre *code, const pcre_extra *extra,
1893 const char *subject, int length, int startoffset,
1894 int options, int *ovector, int ovecsize,
1895 int *workspace, int wscount);
1896
1897 The function pcre_dfa_exec() is called to match a subject string
1898 against a compiled pattern, using a "DFA" matching algorithm. This has
1899 different characteristics to the normal algorithm, and is not compati-
1900 ble with Perl. Some of the features of PCRE patterns are not supported.
1901 Nevertheless, there are times when this kind of matching can be useful.
1902 For a discussion of the two matching algorithms, see the pcrematching
1903 documentation.
1904
1905 The arguments for the pcre_dfa_exec() function are the same as for
1906 pcre_exec(), plus two extras. The ovector argument is used in a differ-
1907 ent way, and this is described below. The other common arguments are
1908 used in the same way as for pcre_exec(), so their description is not
1909 repeated here.
1910
1911 The two additional arguments provide workspace for the function. The
1912 workspace vector should contain at least 20 elements. It is used for
1913 keeping track of multiple paths through the pattern tree. More
1914 workspace will be needed for patterns and subjects where there are a
1915 lot of possible matches.
1916
1917 Here is an example of a simple call to pcre_exec():
1918
1919 int rc;
1920 int ovector[10];
1921 int wspace[20];
1922 rc = pcre_exec(
1923 re, /* result of pcre_compile() */
1924 NULL, /* we didn't study the pattern */
1925 "some string", /* the subject string */
1926 11, /* the length of the subject string */
1927 0, /* start at offset 0 in the subject */
1928 0, /* default options */
1929 ovector, /* vector of integers for substring information */
1930 10, /* number of elements (NOT size in bytes) */
1931 wspace, /* working space vector */
1932 20); /* number of elements (NOT size in bytes) */
1933
1934 Option bits for pcre_dfa_exec()
1935
1936 The unused bits of the options argument for pcre_dfa_exec() must be
1937 zero. The only bits that may be set are PCRE_ANCHORED, PCRE_NOTBOL,
1938 PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NO_UTF8_CHECK, PCRE_PARTIAL,
1939 PCRE_DFA_SHORTEST, and PCRE_DFA_RESTART. All but the last three of
1940 these are the same as for pcre_exec(), so their description is not
1941 repeated here.
1942
1943 PCRE_PARTIAL
1944
1945 This has the same general effect as it does for pcre_exec(), but the
1946 details are slightly different. When PCRE_PARTIAL is set for
1947 pcre_dfa_exec(), the return code PCRE_ERROR_NOMATCH is converted into
1948 PCRE_ERROR_PARTIAL if the end of the subject is reached, there have
1949 been no complete matches, but there is still at least one matching pos-
1950 sibility. The portion of the string that provided the partial match is
1951 set as the first matching string.
1952
1953 PCRE_DFA_SHORTEST
1954
1955 Setting the PCRE_DFA_SHORTEST option causes the matching algorithm to
1956 stop as soon as it has found one match. Because of the way the DFA
1957 algorithm works, this is necessarily the shortest possible match at the
1958 first possible matching point in the subject string.
1959
1960 PCRE_DFA_RESTART
1961
1962 When pcre_dfa_exec() is called with the PCRE_PARTIAL option, and
1963 returns a partial match, it is possible to call it again, with addi-
1964 tional subject characters, and have it continue with the same match.
1965 The PCRE_DFA_RESTART option requests this action; when it is set, the
1966 workspace and wscount options must reference the same vector as before
1967 because data about the match so far is left in them after a partial
1968 match. There is more discussion of this facility in the pcrepartial
1969 documentation.
1970
1971 Successful returns from pcre_dfa_exec()
1972
1973 When pcre_dfa_exec() succeeds, it may have matched more than one sub-
1974 string in the subject. Note, however, that all the matches from one run
1975 of the function start at the same point in the subject. The shorter
1976 matches are all initial substrings of the longer matches. For example,
1977 if the pattern
1978
1979 <.*>
1980
1981 is matched against the string
1982
1983 This is <something> <something else> <something further> no more
1984
1985 the three matched strings are
1986
1987 <something>
1988 <something> <something else>
1989 <something> <something else> <something further>
1990
1991 On success, the yield of the function is a number greater than zero,
1992 which is the number of matched substrings. The substrings themselves
1993 are returned in ovector. Each string uses two elements; the first is
1994 the offset to the start, and the second is the offset to the end. All
1995 the strings have the same start offset. (Space could have been saved by
1996 giving this only once, but it was decided to retain some compatibility
1997 with the way pcre_exec() returns data, even though the meaning of the
1998 strings is different.)
1999
2000 The strings are returned in reverse order of length; that is, the long-
2001 est matching string is given first. If there were too many matches to
2002 fit into ovector, the yield of the function is zero, and the vector is
2003 filled with the longest matches.
2004
2005 Error returns from pcre_dfa_exec()
2006
2007 The pcre_dfa_exec() function returns a negative number when it fails.
2008 Many of the errors are the same as for pcre_exec(), and these are
2009 described above. There are in addition the following errors that are
2010 specific to pcre_dfa_exec():
2011
2012 PCRE_ERROR_DFA_UITEM (-16)
2013
2014 This return is given if pcre_dfa_exec() encounters an item in the pat-
2015 tern that it does not support, for instance, the use of \C or a back
2016 reference.
2017
2018 PCRE_ERROR_DFA_UCOND (-17)
2019
2020 This return is given if pcre_dfa_exec() encounters a condition item in
2021 a pattern that uses a back reference for the condition. This is not
2022 supported.
2023
2024 PCRE_ERROR_DFA_UMLIMIT (-18)
2025
2026 This return is given if pcre_dfa_exec() is called with an extra block
2027 that contains a setting of the match_limit field. This is not supported
2028 (it is meaningless).
2029
2030 PCRE_ERROR_DFA_WSSIZE (-19)
2031
2032 This return is given if pcre_dfa_exec() runs out of space in the
2033 workspace vector.
2034
2035 PCRE_ERROR_DFA_RECURSE (-20)
2036
2037 When a recursive subpattern is processed, the matching function calls
2038 itself recursively, using private vectors for ovector and workspace.
2039 This error is given if the output vector is not large enough. This
2040 should be extremely rare, as a vector of size 1000 is used.
2041
2042 Last updated: 16 May 2005
2043 Copyright (c) 1997-2005 University of Cambridge.
2044 ------------------------------------------------------------------------------
2045
2046
2047 PCRECALLOUT(3) PCRECALLOUT(3)
2048
2049
2050 NAME
2051 PCRE - Perl-compatible regular expressions
2052
2053
2054 PCRE CALLOUTS
2055
2056 int (*pcre_callout)(pcre_callout_block *);
2057
2058 PCRE provides a feature called "callout", which is a means of temporar-
2059 ily passing control to the caller of PCRE in the middle of pattern
2060 matching. The caller of PCRE provides an external function by putting
2061 its entry point in the global variable pcre_callout. By default, this
2062 variable contains NULL, which disables all calling out.
2063
2064 Within a regular expression, (?C) indicates the points at which the
2065 external function is to be called. Different callout points can be
2066 identified by putting a number less than 256 after the letter C. The
2067 default value is zero. For example, this pattern has two callout
2068 points:
2069
2070 (?C1)eabc(?C2)def
2071
2072 If the PCRE_AUTO_CALLOUT option bit is set when pcre_compile() is
2073 called, PCRE automatically inserts callouts, all with number 255,
2074 before each item in the pattern. For example, if PCRE_AUTO_CALLOUT is
2075 used with the pattern
2076
2077 A(\d{2}|--)
2078
2079 it is processed as if it were
2080
2081 (?C255)A(?C255)((?C255)\d{2}(?C255)|(?C255)-(?C255)-(?C255))(?C255)
2082
2083 Notice that there is a callout before and after each parenthesis and
2084 alternation bar. Automatic callouts can be used for tracking the
2085 progress of pattern matching. The pcretest command has an option that
2086 sets automatic callouts; when it is used, the output indicates how the
2087 pattern is matched. This is useful information when you are trying to
2088 optimize the performance of a particular pattern.
2089
2090
2091 MISSING CALLOUTS
2092
2093 You should be aware that, because of optimizations in the way PCRE
2094 matches patterns, callouts sometimes do not happen. For example, if the
2095 pattern is
2096
2097 ab(?C4)cd
2098
2099 PCRE knows that any matching string must contain the letter "d". If the
2100 subject string is "abyz", the lack of "d" means that matching doesn't
2101 ever start, and the callout is never reached. However, with "abyd",
2102 though the result is still no match, the callout is obeyed.
2103
2104
2105 THE CALLOUT INTERFACE
2106
2107 During matching, when PCRE reaches a callout point, the external func-
2108 tion defined by pcre_callout is called (if it is set). This applies to
2109 both the pcre_exec() and the pcre_dfa_exec() matching functions. The
2110 only argument to the callout function is a pointer to a pcre_callout
2111 block. This structure contains the following fields:
2112
2113 int version;
2114 int callout_number;
2115 int *offset_vector;
2116 const char *subject;
2117 int subject_length;
2118 int start_match;
2119 int current_position;
2120 int capture_top;
2121 int capture_last;
2122 void *callout_data;
2123 int pattern_position;
2124 int next_item_length;
2125
2126 The version field is an integer containing the version number of the
2127 block format. The initial version was 0; the current version is 1. The
2128 version number will change again in future if additional fields are
2129 added, but the intention is never to remove any of the existing fields.
2130
2131 The callout_number field contains the number of the callout, as com-
2132 piled into the pattern (that is, the number after ?C for manual call-
2133 outs, and 255 for automatically generated callouts).
2134
2135 The offset_vector field is a pointer to the vector of offsets that was
2136 passed by the caller to pcre_exec() or pcre_dfa_exec(). When
2137 pcre_exec() is used, the contents can be inspected in order to extract
2138 substrings that have been matched so far, in the same way as for
2139 extracting substrings after a match has completed. For pcre_dfa_exec()
2140 this field is not useful.
2141
2142 The subject and subject_length fields contain copies of the values that
2143 were passed to pcre_exec().
2144
2145 The start_match field contains the offset within the subject at which
2146 the current match attempt started. If the pattern is not anchored, the
2147 callout function may be called several times from the same point in the
2148 pattern for different starting points in the subject.
2149
2150 The current_position field contains the offset within the subject of
2151 the current match pointer.
2152
2153 When the pcre_exec() function is used, the capture_top field contains
2154 one more than the number of the highest numbered captured substring so
2155 far. If no substrings have been captured, the value of capture_top is
2156 one. This is always the case when pcre_dfa_exec() is used, because it
2157 does not support captured substrings.
2158
2159 The capture_last field contains the number of the most recently cap-
2160 tured substring. If no substrings have been captured, its value is -1.
2161 This is always the case when pcre_dfa_exec() is used.
2162
2163 The callout_data field contains a value that is passed to pcre_exec()
2164 or pcre_dfa_exec() specifically so that it can be passed back in call-
2165 outs. It is passed in the pcre_callout field of the pcre_extra data
2166 structure. If no such data was passed, the value of callout_data in a
2167 pcre_callout block is NULL. There is a description of the pcre_extra
2168 structure in the pcreapi documentation.
2169
2170 The pattern_position field is present from version 1 of the pcre_call-
2171 out structure. It contains the offset to the next item to be matched in
2172 the pattern string.
2173
2174 The next_item_length field is present from version 1 of the pcre_call-
2175 out structure. It contains the length of the next item to be matched in
2176 the pattern string. When the callout immediately precedes an alterna-
2177 tion bar, a closing parenthesis, or the end of the pattern, the length
2178 is zero. When the callout precedes an opening parenthesis, the length
2179 is that of the entire subpattern.
2180
2181 The pattern_position and next_item_length fields are intended to help
2182 in distinguishing between different automatic callouts, which all have
2183 the same callout number. However, they are set for all callouts.
2184
2185
2186 RETURN VALUES
2187
2188 The external callout function returns an integer to PCRE. If the value
2189 is zero, matching proceeds as normal. If the value is greater than
2190 zero, matching fails at the current point, but the testing of other
2191 matching possibilities goes ahead, just as if a lookahead assertion had
2192 failed. If the value is less than zero, the match is abandoned, and
2193 pcre_exec() (or pcre_dfa_exec()) returns the negative value.
2194
2195 Negative values should normally be chosen from the set of
2196 PCRE_ERROR_xxx values. In particular, PCRE_ERROR_NOMATCH forces a stan-
2197 dard "no match" failure. The error number PCRE_ERROR_CALLOUT is
2198 reserved for use by callout functions; it will never be used by PCRE
2199 itself.
2200
2201 Last updated: 28 February 2005
2202 Copyright (c) 1997-2005 University of Cambridge.
2203 ------------------------------------------------------------------------------
2204
2205
2206 PCRECOMPAT(3) PCRECOMPAT(3)
2207
2208
2209 NAME
2210 PCRE - Perl-compatible regular expressions
2211
2212
2213 DIFFERENCES BETWEEN PCRE AND PERL
2214
2215 This document describes the differences in the ways that PCRE and Perl
2216 handle regular expressions. The differences described here are with
2217 respect to Perl 5.8.
2218
2219 1. PCRE does not have full UTF-8 support. Details of what it does have
2220 are given in the section on UTF-8 support in the main pcre page.
2221
2222 2. PCRE does not allow repeat quantifiers on lookahead assertions. Perl
2223 permits them, but they do not mean what you might think. For example,
2224 (?!a){3} does not assert that the next three characters are not "a". It
2225 just asserts that the next character is not "a" three times.
2226
2227 3. Capturing subpatterns that occur inside negative lookahead asser-
2228 tions are counted, but their entries in the offsets vector are never
2229 set. Perl sets its numerical variables from any such patterns that are
2230 matched before the assertion fails to match something (thereby succeed-
2231 ing), but only if the negative lookahead assertion contains just one
2232 branch.
2233
2234 4. Though binary zero characters are supported in the subject string,
2235 they are not allowed in a pattern string because it is passed as a nor-
2236 mal C string, terminated by zero. The escape sequence \0 can be used in
2237 the pattern to represent a binary zero.
2238
2239 5. The following Perl escape sequences are not supported: \l, \u, \L,
2240 \U, and \N. In fact these are implemented by Perl's general string-han-
2241 dling and are not part of its pattern matching engine. If any of these
2242 are encountered by PCRE, an error is generated.
2243
2244 6. The Perl escape sequences \p, \P, and \X are supported only if PCRE
2245 is built with Unicode character property support. The properties that
2246 can be tested with \p and \P are limited to the general category prop-
2247 erties such as Lu and Nd.
2248
2249 7. PCRE does support the \Q...\E escape for quoting substrings. Charac-
2250 ters in between are treated as literals. This is slightly different
2251 from Perl in that $ and @ are also handled as literals inside the
2252 quotes. In Perl, they cause variable interpolation (but of course PCRE
2253 does not have variables). Note the following examples:
2254
2255 Pattern PCRE matches Perl matches
2256
2257 \Qabc$xyz\E abc$xyz abc followed by the
2258 contents of $xyz
2259 \Qabc\$xyz\E abc\$xyz abc\$xyz
2260 \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2261
2262 The \Q...\E sequence is recognized both inside and outside character
2263 classes.
2264
2265 8. Fairly obviously, PCRE does not support the (?{code}) and (?p{code})
2266 constructions. However, there is support for recursive patterns using
2267 the non-Perl items (?R), (?number), and (?P>name). Also, the PCRE
2268 "callout" feature allows an external function to be called during pat-
2269 tern matching. See the pcrecallout documentation for details.
2270
2271 9. There are some differences that are concerned with the settings of
2272 captured strings when part of a pattern is repeated. For example,
2273 matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2
2274 unset, but in PCRE it is set to "b".
2275
2276 10. PCRE provides some extensions to the Perl regular expression facil-
2277 ities:
2278
2279 (a) Although lookbehind assertions must match fixed length strings,
2280 each alternative branch of a lookbehind assertion can match a different
2281 length of string. Perl requires them all to have the same length.
2282
2283 (b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $
2284 meta-character matches only at the very end of the string.
2285
2286 (c) If PCRE_EXTRA is set, a backslash followed by a letter with no spe-
2287 cial meaning is faulted.
2288
2289 (d) If PCRE_UNGREEDY is set, the greediness of the repetition quanti-
2290 fiers is inverted, that is, by default they are not greedy, but if fol-
2291 lowed by a question mark they are.
2292
2293 (e) PCRE_ANCHORED can be used at matching time to force a pattern to be
2294 tried only at the first matching position in the subject string.
2295
2296 (f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAP-
2297 TURE options for pcre_exec() have no Perl equivalents.
2298
2299 (g) The (?R), (?number), and (?P>name) constructs allows for recursive
2300 pattern matching (Perl can do this using the (?p{code}) construct,
2301 which PCRE cannot support.)
2302
2303 (h) PCRE supports named capturing substrings, using the Python syntax.
2304
2305 (i) PCRE supports the possessive quantifier "++" syntax, taken from
2306 Sun's Java package.
2307
2308 (j) The (R) condition, for testing recursion, is a PCRE extension.
2309
2310 (k) The callout facility is PCRE-specific.
2311
2312 (l) The partial matching facility is PCRE-specific.
2313
2314 (m) Patterns compiled by PCRE can be saved and re-used at a later time,
2315 even on different hosts that have the other endianness.
2316
2317 (n) The alternative matching function (pcre_dfa_exec()) matches in a
2318 different way and is not Perl-compatible.
2319
2320 Last updated: 28 February 2005
2321 Copyright (c) 1997-2005 University of Cambridge.
2322 ------------------------------------------------------------------------------
2323
2324
2325 PCREPATTERN(3) PCREPATTERN(3)
2326
2327
2328 NAME
2329 PCRE - Perl-compatible regular expressions
2330
2331
2332 PCRE REGULAR EXPRESSION DETAILS
2333
2334 The syntax and semantics of the regular expressions supported by PCRE
2335 are described below. Regular expressions are also described in the Perl
2336 documentation and in a number of books, some of which have copious
2337 examples. Jeffrey Friedl's "Mastering Regular Expressions", published
2338 by O'Reilly, covers regular expressions in great detail. This descrip-
2339 tion of PCRE's regular expressions is intended as reference material.
2340
2341 The original operation of PCRE was on strings of one-byte characters.
2342 However, there is now also support for UTF-8 character strings. To use
2343 this, you must build PCRE to include UTF-8 support, and then call
2344 pcre_compile() with the PCRE_UTF8 option. How this affects pattern
2345 matching is mentioned in several places below. There is also a summary
2346 of UTF-8 features in the section on UTF-8 support in the main pcre
2347 page.
2348
2349 The remainder of this document discusses the patterns that are sup-
2350 ported by PCRE when its main matching function, pcre_exec(), is used.
2351 From release 6.0, PCRE offers a second matching function,
2352 pcre_dfa_exec(), which matches using a different algorithm that is not
2353 Perl-compatible. The advantages and disadvantages of the alternative
2354 function, and how it differs from the normal function, are discussed in
2355 the pcrematching page.
2356
2357 A regular expression is a pattern that is matched against a subject
2358 string from left to right. Most characters stand for themselves in a
2359 pattern, and match the corresponding characters in the subject. As a
2360 trivial example, the pattern
2361
2362 The quick brown fox
2363
2364 matches a portion of a subject string that is identical to itself. When
2365 caseless matching is specified (the PCRE_CASELESS option), letters are
2366 matched independently of case. In UTF-8 mode, PCRE always understands
2367 the concept of case for characters whose values are less than 128, so
2368 caseless matching is always possible. For characters with higher val-
2369 ues, the concept of case is supported if PCRE is compiled with Unicode
2370 property support, but not otherwise. If you want to use caseless
2371 matching for characters 128 and above, you must ensure that PCRE is
2372 compiled with Unicode property support as well as with UTF-8 support.
2373
2374 The power of regular expressions comes from the ability to include
2375 alternatives and repetitions in the pattern. These are encoded in the
2376 pattern by the use of metacharacters, which do not stand for themselves
2377 but instead are interpreted in some special way.
2378
2379 There are two different sets of metacharacters: those that are recog-
2380 nized anywhere in the pattern except within square brackets, and those
2381 that are recognized in square brackets. Outside square brackets, the
2382 metacharacters are as follows:
2383
2384 \ general escape character with several uses
2385 ^ assert start of string (or line, in multiline mode)
2386 $ assert end of string (or line, in multiline mode)
2387 . match any character except newline (by default)
2388 [ start character class definition
2389 | start of alternative branch
2390 ( start subpattern
2391 ) end subpattern
2392 ? extends the meaning of (
2393 also 0 or 1 quantifier
2394 also quantifier minimizer
2395 * 0 or more quantifier
2396 + 1 or more quantifier
2397 also "possessive quantifier"
2398 { start min/max quantifier
2399
2400 Part of a pattern that is in square brackets is called a "character
2401 class". In a character class the only metacharacters are:
2402
2403 \ general escape character
2404 ^ negate the class, but only if the first character
2405 - indicates character range
2406 [ POSIX character class (only if followed by POSIX
2407 syntax)
2408 ] terminates the character class
2409
2410 The following sections describe the use of each of the metacharacters.
2411
2412
2413 BACKSLASH
2414
2415 The backslash character has several uses. Firstly, if it is followed by
2416 a non-alphanumeric character, it takes away any special meaning that
2417 character may have. This use of backslash as an escape character
2418 applies both inside and outside character classes.
2419
2420 For example, if you want to match a * character, you write \* in the
2421 pattern. This escaping action applies whether or not the following
2422 character would otherwise be interpreted as a metacharacter, so it is
2423 always safe to precede a non-alphanumeric with backslash to specify
2424 that it stands for itself. In particular, if you want to match a back-
2425 slash, you write \\.
2426
2427 If a pattern is compiled with the PCRE_EXTENDED option, whitespace in
2428 the pattern (other than in a character class) and characters between a
2429 # outside a character class and the next newline character are ignored.
2430 An escaping backslash can be used to include a whitespace or # charac-
2431 ter as part of the pattern.
2432
2433 If you want to remove the special meaning from a sequence of charac-
2434 ters, you can do so by putting them between \Q and \E. This is differ-
2435 ent from Perl in that $ and @ are handled as literals in \Q...\E
2436 sequences in PCRE, whereas in Perl, $ and @ cause variable interpola-
2437 tion. Note the following examples:
2438
2439 Pattern PCRE matches Perl matches
2440
2441 \Qabc$xyz\E abc$xyz abc followed by the
2442 contents of $xyz
2443 \Qabc\$xyz\E abc\$xyz abc\$xyz
2444 \Qabc\E\$\Qxyz\E abc$xyz abc$xyz
2445
2446 The \Q...\E sequence is recognized both inside and outside character
2447 classes.
2448
2449 Non-printing characters
2450
2451 A second use of backslash provides a way of encoding non-printing char-
2452 acters in patterns in a visible manner. There is no restriction on the
2453 appearance of non-printing characters, apart from the binary zero that
2454 terminates a pattern, but when a pattern is being prepared by text
2455 editing, it is usually easier to use one of the following escape
2456 sequences than the binary character it represents:
2457
2458 \a alarm, that is, the BEL character (hex 07)
2459 \cx "control-x", where x is any character
2460 \e escape (hex 1B)
2461 \f formfeed (hex 0C)
2462 \n newline (hex 0A)
2463 \r carriage return (hex 0D)
2464 \t tab (hex 09)
2465 \ddd character with octal code ddd, or backreference
2466 \xhh character with hex code hh
2467 \x{hhh..} character with hex code hhh... (UTF-8 mode only)
2468
2469 The precise effect of \cx is as follows: if x is a lower case letter,
2470 it is converted to upper case. Then bit 6 of the character (hex 40) is
2471 inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c;
2472 becomes hex 7B.
2473
2474 After \x, from zero to two hexadecimal digits are read (letters can be
2475 in upper or lower case). In UTF-8 mode, any number of hexadecimal dig-
2476 its may appear between \x{ and }, but the value of the character code
2477 must be less than 2**31 (that is, the maximum hexadecimal value is
2478 7FFFFFFF). If characters other than hexadecimal digits appear between
2479 \x{ and }, or if there is no terminating }, this form of escape is not
2480 recognized. Instead, the initial \x will be interpreted as a basic
2481 hexadecimal escape, with no following digits, giving a character whose
2482 value is zero.
2483
2484 Characters whose value is less than 256 can be defined by either of the
2485 two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference
2486 in the way they are handled. For example, \xdc is exactly the same as
2487 \x{dc}.
2488
2489 After \0 up to two further octal digits are read. In both cases, if
2490 there are fewer than two digits, just those that are present are used.
2491 Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL
2492 character (code value 7). Make sure you supply two digits after the
2493 initial zero if the pattern character that follows is itself an octal
2494 digit.
2495
2496 The handling of a backslash followed by a digit other than 0 is compli-
2497 cated. Outside a character class, PCRE reads it and any following dig-
2498 its as a decimal number. If the number is less than 10, or if there
2499 have been at least that many previous capturing left parentheses in the
2500 expression, the entire sequence is taken as a back reference. A
2501 description of how this works is given later, following the discussion
2502 of parenthesized subpatterns.
2503
2504 Inside a character class, or if the decimal number is greater than 9
2505 and there have not been that many capturing subpatterns, PCRE re-reads
2506 up to three octal digits following the backslash, and generates a sin-
2507 gle byte from the least significant 8 bits of the value. Any subsequent
2508 digits stand for themselves. For example:
2509
2510 \040 is another way of writing a space
2511 \40 is the same, provided there are fewer than 40
2512 previous capturing subpatterns
2513 \7 is always a back reference
2514 \11 might be a back reference, or another way of
2515 writing a tab
2516 \011 is always a tab
2517 \0113 is a tab followed by the character "3"
2518 \113 might be a back reference, otherwise the
2519 character with octal code 113
2520 \377 might be a back reference, otherwise
2521 the byte consisting entirely of 1 bits
2522 \81 is either a back reference, or a binary zero
2523 followed by the two characters "8" and "1"
2524
2525 Note that octal values of 100 or greater must not be introduced by a
2526 leading zero, because no more than three octal digits are ever read.
2527
2528 All the sequences that define a single byte value or a single UTF-8
2529 character (in UTF-8 mode) can be used both inside and outside character
2530 classes. In addition, inside a character class, the sequence \b is
2531 interpreted as the backspace character (hex 08), and the sequence \X is
2532 interpreted as the character "X". Outside a character class, these
2533 sequences have different meanings (see below).
2534
2535 Generic character types
2536
2537 The third use of backslash is for specifying generic character types.
2538 The following are always recognized:
2539
2540 \d any decimal digit
2541 \D any character that is not a decimal digit
2542 \s any whitespace character
2543 \S any character that is not a whitespace character
2544 \w any "word" character
2545 \W any "non-word" character
2546
2547 Each pair of escape sequences partitions the complete set of characters
2548 into two disjoint sets. Any given character matches one, and only one,
2549 of each pair.
2550
2551 These character type sequences can appear both inside and outside char-
2552 acter classes. They each match one character of the appropriate type.
2553 If the current matching point is at the end of the subject string, all
2554 of them fail, since there is no character to match.
2555
2556 For compatibility with Perl, \s does not match the VT character (code
2557 11). This makes it different from the the POSIX "space" class. The \s
2558 characters are HT (9), LF (10), FF (12), CR (13), and space (32).
2559
2560 A "word" character is an underscore or any character less than 256 that
2561 is a letter or digit. The definition of letters and digits is con-
2562 trolled by PCRE's low-valued character tables, and may vary if locale-
2563 specific matching is taking place (see "Locale support" in the pcreapi
2564 page). For example, in the "fr_FR" (French) locale, some character
2565 codes greater than 128 are used for accented letters, and these are
2566 matched by \w.
2567
2568 In UTF-8 mode, characters with values greater than 128 never match \d,
2569 \s, or \w, and always match \D, \S, and \W. This is true even when Uni-
2570 code character property support is available.
2571
2572 Unicode character properties
2573
2574 When PCRE is built with Unicode character property support, three addi-
2575 tional escape sequences to match generic character types are available
2576 when UTF-8 mode is selected. They are:
2577
2578 \p{xx} a character with the xx property
2579 \P{xx} a character without the xx property
2580 \X an extended Unicode sequence
2581
2582 The property names represented by xx above are limited to the Unicode
2583 general category properties. Each character has exactly one such prop-
2584 erty, specified by a two-letter abbreviation. For compatibility with
2585 Perl, negation can be specified by including a circumflex between the
2586 opening brace and the property name. For example, \p{^Lu} is the same
2587 as \P{Lu}.
2588
2589 If only one letter is specified with \p or \P, it includes all the
2590 properties that start with that letter. In this case, in the absence of
2591 negation, the curly brackets in the escape sequence are optional; these
2592 two examples have the same effect:
2593
2594 \p{L}
2595 \pL
2596
2597 The following property codes are supported:
2598
2599 C Other
2600 Cc Control
2601 Cf Format
2602 Cn Unassigned
2603 Co Private use
2604 Cs Surrogate
2605
2606 L Letter
2607 Ll Lower case letter
2608 Lm Modifier letter
2609 Lo Other letter
2610 Lt Title case letter
2611 Lu Upper case letter
2612
2613 M Mark
2614 Mc Spacing mark
2615 Me Enclosing mark
2616 Mn Non-spacing mark
2617
2618 N Number
2619 Nd Decimal number
2620 Nl Letter number
2621 No Other number
2622
2623 P Punctuation
2624 Pc Connector punctuation
2625 Pd Dash punctuation
2626 Pe Close punctuation
2627 Pf Final punctuation
2628 Pi Initial punctuation
2629 Po Other punctuation
2630 Ps Open punctuation
2631
2632 S Symbol
2633 Sc Currency symbol
2634 Sk Modifier symbol
2635 Sm Mathematical symbol
2636 So Other symbol
2637
2638 Z Separator
2639 Zl Line separator
2640 Zp Paragraph separator
2641 Zs Space separator
2642
2643 Extended properties such as "Greek" or "InMusicalSymbols" are not sup-
2644 ported by PCRE.
2645
2646 Specifying caseless matching does not affect these escape sequences.
2647 For example, \p{Lu} always matches only upper case letters.
2648
2649 The \X escape matches any number of Unicode characters that form an
2650 extended Unicode sequence. \X is equivalent to
2651
2652 (?>\PM\pM*)
2653
2654 That is, it matches a character without the "mark" property, followed
2655 by zero or more characters with the "mark" property, and treats the
2656 sequence as an atomic group (see below). Characters with the "mark"
2657 property are typically accents that affect the preceding character.
2658
2659 Matching characters by Unicode property is not fast, because PCRE has
2660 to search a structure that contains data for over fifteen thousand
2661 characters. That is why the traditional escape sequences such as \d and
2662 \w do not use Unicode properties in PCRE.
2663
2664 Simple assertions
2665
2666 The fourth use of backslash is for certain simple assertions. An asser-
2667 tion specifies a condition that has to be met at a particular point in
2668 a match, without consuming any characters from the subject string. The
2669 use of subpatterns for more complicated assertions is described below.
2670 The backslashed assertions are:
2671
2672 \b matches at a word boundary
2673 \B matches when not at a word boundary
2674 \A matches at start of subject
2675 \Z matches at end of subject or before newline at end
2676 \z matches at end of subject
2677 \G matches at first matching position in subject
2678
2679 These assertions may not appear in character classes (but note that \b
2680 has a different meaning, namely the backspace character, inside a char-
2681 acter class).
2682
2683 A word boundary is a position in the subject string where the current
2684 character and the previous character do not both match \w or \W (i.e.
2685 one matches \w and the other matches \W), or the start or end of the
2686 string if the first or last character matches \w, respectively.
2687
2688 The \A, \Z, and \z assertions differ from the traditional circumflex
2689 and dollar (described in the next section) in that they only ever match
2690 at the very start and end of the subject string, whatever options are
2691 set. Thus, they are independent of multiline mode. These three asser-
2692 tions are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options, which
2693 affect only the behaviour of the circumflex and dollar metacharacters.
2694 However, if the startoffset argument of pcre_exec() is non-zero, indi-
2695 cating that matching is to start at a point other than the beginning of
2696 the subject, \A can never match. The difference between \Z and \z is
2697 that \Z matches before a newline that is the last character of the
2698 string as well as at the end of the string, whereas \z matches only at
2699 the end.
2700
2701 The \G assertion is true only when the current matching position is at
2702 the start point of the match, as specified by the startoffset argument
2703 of pcre_exec(). It differs from \A when the value of startoffset is
2704 non-zero. By calling pcre_exec() multiple times with appropriate argu-
2705 ments, you can mimic Perl's /g option, and it is in this kind of imple-
2706 mentation where \G can be useful.
2707
2708 Note, however, that PCRE's interpretation of \G, as the start of the
2709 current match, is subtly different from Perl's, which defines it as the
2710 end of the previous match. In Perl, these can be different when the
2711 previously matched string was empty. Because PCRE does just one match
2712 at a time, it cannot reproduce this behaviour.
2713
2714 If all the alternatives of a pattern begin with \G, the expression is
2715 anchored to the starting match position, and the "anchored" flag is set
2716 in the compiled regular expression.
2717
2718
2719 CIRCUMFLEX AND DOLLAR
2720
2721 Outside a character class, in the default matching mode, the circumflex
2722 character is an assertion that is true only if the current matching
2723 point is at the start of the subject string. If the startoffset argu-
2724 ment of pcre_exec() is non-zero, circumflex can never match if the
2725 PCRE_MULTILINE option is unset. Inside a character class, circumflex
2726 has an entirely different meaning (see below).
2727
2728 Circumflex need not be the first character of the pattern if a number
2729 of alternatives are involved, but it should be the first thing in each
2730 alternative in which it appears if the pattern is ever to match that
2731 branch. If all possible alternatives start with a circumflex, that is,
2732 if the pattern is constrained to match only at the start of the sub-
2733 ject, it is said to be an "anchored" pattern. (There are also other
2734 constructs that can cause a pattern to be anchored.)
2735
2736 A dollar character is an assertion that is true only if the current
2737 matching point is at the end of the subject string, or immediately
2738 before a newline character that is the last character in the string (by
2739 default). Dollar need not be the last character of the pattern if a
2740 number of alternatives are involved, but it should be the last item in
2741 any branch in which it appears. Dollar has no special meaning in a
2742 character class.
2743
2744 The meaning of dollar can be changed so that it matches only at the
2745 very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at
2746 compile time. This does not affect the \Z assertion.
2747
2748 The meanings of the circumflex and dollar characters are changed if the
2749 PCRE_MULTILINE option is set. When this is the case, they match immedi-
2750 ately after and immediately before an internal newline character,
2751 respectively, in addition to matching at the start and end of the sub-
2752 ject string. For example, the pattern /^abc$/ matches the subject
2753 string "def\nabc" (where \n represents a newline character) in multi-
2754 line mode, but not otherwise. Consequently, patterns that are anchored
2755 in single line mode because all branches start with ^ are not anchored
2756 in multiline mode, and a match for circumflex is possible when the
2757 startoffset argument of pcre_exec() is non-zero. The PCRE_DOL-
2758 LAR_ENDONLY option is ignored if PCRE_MULTILINE is set.
2759
2760 Note that the sequences \A, \Z, and \z can be used to match the start
2761 and end of the subject in both modes, and if all branches of a pattern
2762 start with \A it is always anchored, whether PCRE_MULTILINE is set or
2763 not.
2764
2765
2766 FULL STOP (PERIOD, DOT)
2767
2768 Outside a character class, a dot in the pattern matches any one charac-
2769 ter in the subject, including a non-printing character, but not (by
2770 default) newline. In UTF-8 mode, a dot matches any UTF-8 character,
2771 which might be more than one byte long, except (by default) newline. If
2772 the PCRE_DOTALL option is set, dots match newlines as well. The han-
2773 dling of dot is entirely independent of the handling of circumflex and
2774 dollar, the only relationship being that they both involve newline
2775 characters. Dot has no special meaning in a character class.
2776
2777
2778 MATCHING A SINGLE BYTE
2779
2780 Outside a character class, the escape sequence \C matches any one byte,
2781 both in and out of UTF-8 mode. Unlike a dot, it can match a newline.
2782 The feature is provided in Perl in order to match individual bytes in
2783 UTF-8 mode. Because it breaks up UTF-8 characters into individual
2784 bytes, what remains in the string may be a malformed UTF-8 string. For
2785 this reason, the \C escape sequence is best avoided.
2786
2787 PCRE does not allow \C to appear in lookbehind assertions (described
2788 below), because in UTF-8 mode this would make it impossible to calcu-
2789 late the length of the lookbehind.
2790
2791
2792 SQUARE BRACKETS AND CHARACTER CLASSES
2793
2794 An opening square bracket introduces a character class, terminated by a
2795 closing square bracket. A closing square bracket on its own is not spe-
2796 cial. If a closing square bracket is required as a member of the class,
2797 it should be the first data character in the class (after an initial
2798 circumflex, if present) or escaped with a backslash.
2799
2800 A character class matches a single character in the subject. In UTF-8
2801 mode, the character may occupy more than one byte. A matched character
2802 must be in the set of characters defined by the class, unless the first
2803 character in the class definition is a circumflex, in which case the
2804 subject character must not be in the set defined by the class. If a
2805 circumflex is actually required as a member of the class, ensure it is
2806 not the first character, or escape it with a backslash.
2807
2808 For example, the character class [aeiou] matches any lower case vowel,
2809 while [^aeiou] matches any character that is not a lower case vowel.
2810 Note that a circumflex is just a convenient notation for specifying the
2811 characters that are in the class by enumerating those that are not. A
2812 class that starts with a circumflex is not an assertion: it still con-
2813 sumes a character from the subject string, and therefore it fails if
2814 the current pointer is at the end of the string.
2815
2816 In UTF-8 mode, characters with values greater than 255 can be included
2817 in a class as a literal string of bytes, or by using the \x{ escaping
2818 mechanism.
2819
2820 When caseless matching is set, any letters in a class represent both
2821 their upper case and lower case versions, so for example, a caseless
2822 [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not
2823 match "A", whereas a caseful version would. In UTF-8 mode, PCRE always
2824 understands the concept of case for characters whose values are less
2825 than 128, so caseless matching is always possible. For characters with
2826 higher values, the concept of case is supported if PCRE is compiled
2827 with Unicode property support, but not otherwise. If you want to use
2828 caseless matching for characters 128 and above, you must ensure that
2829 PCRE is compiled with Unicode property support as well as with UTF-8
2830 support.
2831
2832 The newline character is never treated in any special way in character
2833 classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE
2834 options is. A class such as [^a] will always match a newline.
2835
2836 The minus (hyphen) character can be used to specify a range of charac-
2837 ters in a character class. For example, [d-m] matches any letter
2838 between d and m, inclusive. If a minus character is required in a
2839 class, it must be escaped with a backslash or appear in a position
2840 where it cannot be interpreted as indicating a range, typically as the
2841 first or last character in the class.
2842
2843 It is not possible to have the literal character "]" as the end charac-
2844 ter of a range. A pattern such as [W-]46] is interpreted as a class of
2845 two characters ("W" and "-") followed by a literal string "46]", so it
2846 would match "W46]" or "-46]". However, if the "]" is escaped with a
2847 backslash it is interpreted as the end of range, so [W-\]46] is inter-
2848 preted as a class containing a range followed by two other characters.
2849 The octal or hexadecimal representation of "]" can also be used to end
2850 a range.
2851
2852 Ranges operate in the collating sequence of character values. They can
2853 also be used for characters specified numerically, for example
2854 [\000-\037]. In UTF-8 mode, ranges can include characters whose values
2855 are greater than 255, for example [\x{100}-\x{2ff}].
2856
2857 If a range that includes letters is used when caseless matching is set,
2858 it matches the letters in either case. For example, [W-c] is equivalent
2859 to [][\\^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if
2860 character tables for the "fr_FR" locale are in use, [\xc8-\xcb] matches
2861 accented E characters in both cases. In UTF-8 mode, PCRE supports the
2862 concept of case for characters with values greater than 128 only when
2863 it is compiled with Unicode property support.
2864
2865 The character types \d, \D, \p, \P, \s, \S, \w, and \W may also appear
2866 in a character class, and add the characters that they match to the
2867 class. For example, [\dABCDEF] matches any hexadecimal digit. A circum-
2868 flex can conveniently be used with the upper case character types to
2869 specify a more restricted set of characters than the matching lower
2870 case type. For example, the class [^\W_] matches any letter or digit,
2871 but not underscore.
2872
2873 The only metacharacters that are recognized in character classes are
2874 backslash, hyphen (only where it can be interpreted as specifying a
2875 range), circumflex (only at the start), opening square bracket (only
2876 when it can be interpreted as introducing a POSIX class name - see the
2877 next section), and the terminating closing square bracket. However,
2878 escaping other non-alphanumeric characters does no harm.
2879
2880
2881 POSIX CHARACTER CLASSES
2882
2883 Perl supports the POSIX notation for character classes. This uses names
2884 enclosed by [: and :] within the enclosing square brackets. PCRE also
2885 supports this notation. For example,
2886
2887 [01[:alpha:]%]
2888
2889 matches "0", "1", any alphabetic character, or "%". The supported class
2890 names are
2891
2892 alnum letters and digits
2893 alpha letters
2894 ascii character codes 0 - 127
2895 blank space or tab only
2896 cntrl control characters
2897 digit decimal digits (same as \d)
2898 graph printing characters, excluding space
2899 lower lower case letters
2900 print printing characters, including space
2901 punct printing characters, excluding letters and digits
2902 space white space (not quite the same as \s)
2903 upper upper case letters
2904 word "word" characters (same as \w)
2905 xdigit hexadecimal digits
2906
2907 The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13),
2908 and space (32). Notice that this list includes the VT character (code
2909 11). This makes "space" different to \s, which does not include VT (for
2910 Perl compatibility).
2911
2912 The name "word" is a Perl extension, and "blank" is a GNU extension
2913 from Perl 5.8. Another Perl extension is negation, which is indicated
2914 by a ^ character after the colon. For example,
2915
2916 [12[:^digit:]]
2917
2918 matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the
2919 POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but
2920 these are not supported, and an error is given if they are encountered.
2921
2922 In UTF-8 mode, characters with values greater than 128 do not match any
2923 of the POSIX character classes.
2924
2925
2926 VERTICAL BAR
2927
2928 Vertical bar characters are used to separate alternative patterns. For
2929 example, the pattern
2930
2931 gilbert|sullivan
2932
2933 matches either "gilbert" or "sullivan". Any number of alternatives may
2934 appear, and an empty alternative is permitted (matching the empty
2935 string). The matching process tries each alternative in turn, from
2936 left to right, and the first one that succeeds is used. If the alterna-
2937 tives are within a subpattern (defined below), "succeeds" means match-
2938 ing the rest of the main pattern as well as the alternative in the sub-
2939 pattern.
2940
2941
2942 INTERNAL OPTION SETTING
2943
2944 The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and
2945 PCRE_EXTENDED options can be changed from within the pattern by a
2946 sequence of Perl option letters enclosed between "(?" and ")". The
2947 option letters are
2948
2949 i for PCRE_CASELESS
2950 m for PCRE_MULTILINE
2951 s for PCRE_DOTALL
2952 x for PCRE_EXTENDED
2953
2954 For example, (?im) sets caseless, multiline matching. It is also possi-
2955 ble to unset these options by preceding the letter with a hyphen, and a
2956 combined setting and unsetting such as (?im-sx), which sets PCRE_CASE-
2957 LESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED,
2958 is also permitted. If a letter appears both before and after the
2959 hyphen, the option is unset.
2960
2961 When an option change occurs at top level (that is, not inside subpat-
2962 tern parentheses), the change applies to the remainder of the pattern
2963 that follows. If the change is placed right at the start of a pattern,
2964 PCRE extracts it into the global options (and it will therefore show up
2965 in data extracted by the pcre_fullinfo() function).
2966
2967 An option change within a subpattern affects only that part of the cur-
2968 rent pattern that follows it, so
2969
2970 (a(?i)b)c
2971
2972 matches abc and aBc and no other strings (assuming PCRE_CASELESS is not
2973 used). By this means, options can be made to have different settings
2974 in different parts of the pattern. Any changes made in one alternative
2975 do carry on into subsequent branches within the same subpattern. For
2976 example,
2977
2978 (a(?i)b|c)
2979
2980 matches "ab", "aB", "c", and "C", even though when matching "C" the
2981 first branch is abandoned before the option setting. This is because
2982 the effects of option settings happen at compile time. There would be
2983 some very weird behaviour otherwise.
2984
2985 The PCRE-specific options PCRE_UNGREEDY and PCRE_EXTRA can be changed
2986 in the same way as the Perl-compatible options by using the characters
2987 U and X respectively. The (?X) flag setting is special in that it must
2988 always occur earlier in the pattern than any of the additional features
2989 it turns on, even when it is at top level. It is best to put it at the
2990 start.
2991
2992
2993 SUBPATTERNS
2994
2995 Subpatterns are delimited by parentheses (round brackets), which can be
2996 nested. Turning part of a pattern into a subpattern does two things:
2997
2998 1. It localizes a set of alternatives. For example, the pattern
2999
3000 cat(aract|erpillar|)
3001
3002 matches one of the words "cat", "cataract", or "caterpillar". Without
3003 the parentheses, it would match "cataract", "erpillar" or the empty
3004 string.
3005
3006 2. It sets up the subpattern as a capturing subpattern. This means
3007 that, when the whole pattern matches, that portion of the subject
3008 string that matched the subpattern is passed back to the caller via the
3009 ovector argument of pcre_exec(). Opening parentheses are counted from
3010 left to right (starting from 1) to obtain numbers for the capturing
3011 subpatterns.
3012
3013 For example, if the string "the red king" is matched against the pat-
3014 tern
3015
3016 the ((red|white) (king|queen))
3017
3018 the captured substrings are "red king", "red", and "king", and are num-
3019 bered 1, 2, and 3, respectively.
3020
3021 The fact that plain parentheses fulfil two functions is not always
3022 helpful. There are often times when a grouping subpattern is required
3023 without a capturing requirement. If an opening parenthesis is followed
3024 by a question mark and a colon, the subpattern does not do any captur-
3025 ing, and is not counted when computing the number of any subsequent
3026 capturing subpatterns. For example, if the string "the white queen" is
3027 matched against the pattern
3028
3029 the ((?:red|white) (king|queen))
3030
3031 the captured substrings are "white queen" and "queen", and are numbered
3032 1 and 2. The maximum number of capturing subpatterns is 65535, and the
3033 maximum depth of nesting of all subpatterns, both capturing and non-
3034 capturing, is 200.
3035
3036 As a convenient shorthand, if any option settings are required at the
3037 start of a non-capturing subpattern, the option letters may appear
3038 between the "?" and the ":". Thus the two patterns
3039
3040 (?i:saturday|sunday)
3041 (?:(?i)saturday|sunday)
3042
3043 match exactly the same set of strings. Because alternative branches are
3044 tried from left to right, and options are not reset until the end of
3045 the subpattern is reached, an option setting in one branch does affect
3046 subsequent branches, so the above patterns match "SUNDAY" as well as
3047 "Saturday".
3048
3049
3050 NAMED SUBPATTERNS
3051
3052 Identifying capturing parentheses by number is simple, but it can be
3053 very hard to keep track of the numbers in complicated regular expres-
3054 sions. Furthermore, if an expression is modified, the numbers may
3055 change. To help with this difficulty, PCRE supports the naming of sub-
3056 patterns, something that Perl does not provide. The Python syntax
3057 (?P<name>...) is used. Names consist of alphanumeric characters and
3058 underscores, and must be unique within a pattern.
3059
3060 Named capturing parentheses are still allocated numbers as well as
3061 names. The PCRE API provides function calls for extracting the name-to-
3062 number translation table from a compiled pattern. There is also a con-
3063 venience function for extracting a captured substring by name. For fur-
3064 ther details see the pcreapi documentation.
3065
3066
3067 REPETITION
3068
3069 Repetition is specified by quantifiers, which can follow any of the
3070 following items:
3071
3072 a literal data character
3073 the . metacharacter
3074 the \C escape sequence
3075 the \X escape sequence (in UTF-8 mode with Unicode properties)
3076 an escape such as \d that matches a single character
3077 a character class
3078 a back reference (see next section)
3079 a parenthesized subpattern (unless it is an assertion)
3080
3081 The general repetition quantifier specifies a minimum and maximum num-
3082 ber of permitted matches, by giving the two numbers in curly brackets
3083 (braces), separated by a comma. The numbers must be less than 65536,
3084 and the first must be less than or equal to the second. For example:
3085
3086 z{2,4}
3087
3088 matches "zz", "zzz", or "zzzz". A closing brace on its own is not a
3089 special character. If the second number is omitted, but the comma is
3090 present, there is no upper limit; if the second number and the comma
3091 are both omitted, the quantifier specifies an exact number of required
3092 matches. Thus
3093
3094 [aeiou]{3,}
3095
3096 matches at least 3 successive vowels, but may match many more, while
3097
3098 \d{8}
3099
3100 matches exactly 8 digits. An opening curly bracket that appears in a
3101 position where a quantifier is not allowed, or one that does not match
3102 the syntax of a quantifier, is taken as a literal character. For exam-
3103 ple, {,6} is not a quantifier, but a literal string of four characters.
3104
3105 In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to
3106 individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 char-
3107 acters, each of which is represented by a two-byte sequence. Similarly,
3108 when Unicode property support is available, \X{3} matches three Unicode
3109 extended sequences, each of which may be several bytes long (and they
3110 may be of different lengths).
3111
3112 The quantifier {0} is permitted, causing the expression to behave as if
3113 the previous item and the quantifier were not present.
3114
3115 For convenience (and historical compatibility) the three most common
3116 quantifiers have single-character abbreviations:
3117
3118 * is equivalent to {0,}
3119 + is equivalent to {1,}
3120 ? is equivalent to {0,1}
3121
3122 It is possible to construct infinite loops by following a subpattern
3123 that can match no characters with a quantifier that has no upper limit,
3124 for example:
3125
3126 (a?)*
3127
3128 Earlier versions of Perl and PCRE used to give an error at compile time
3129 for such patterns. However, because there are cases where this can be
3130 useful, such patterns are now accepted, but if any repetition of the
3131 subpattern does in fact match no characters, the loop is forcibly bro-
3132 ken.
3133
3134 By default, the quantifiers are "greedy", that is, they match as much
3135 as possible (up to the maximum number of permitted times), without
3136 causing the rest of the pattern to fail. The classic example of where
3137 this gives problems is in trying to match comments in C programs. These
3138 appear between /* and */ and within the comment, individual * and /
3139 characters may appear. An attempt to match C comments by applying the
3140 pattern
3141
3142 /\*.*\*/
3143
3144 to the string
3145
3146 /* first comment */ not comment /* second comment */
3147
3148 fails, because it matches the entire string owing to the greediness of
3149 the .* item.
3150
3151 However, if a quantifier is followed by a question mark, it ceases to
3152 be greedy, and instead matches the minimum number of times possible, so
3153 the pattern
3154
3155 /\*.*?\*/
3156
3157 does the right thing with the C comments. The meaning of the various
3158 quantifiers is not otherwise changed, just the preferred number of
3159 matches. Do not confuse this use of question mark with its use as a
3160 quantifier in its own right. Because it has two uses, it can sometimes
3161 appear doubled, as in
3162
3163 \d??\d
3164
3165 which matches one digit by preference, but can match two if that is the
3166 only way the rest of the pattern matches.
3167
3168 If the PCRE_UNGREEDY option is set (an option which is not available in
3169 Perl), the quantifiers are not greedy by default, but individual ones
3170 can be made greedy by following them with a question mark. In other
3171 words, it inverts the default behaviour.
3172
3173 When a parenthesized subpattern is quantified with a minimum repeat
3174 count that is greater than 1 or with a limited maximum, more memory is
3175 required for the compiled pattern, in proportion to the size of the
3176 minimum or maximum.
3177
3178 If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equiv-
3179 alent to Perl's /s) is set, thus allowing the . to match newlines, the
3180 pattern is implicitly anchored, because whatever follows will be tried
3181 against every character position in the subject string, so there is no
3182 point in retrying the overall match at any position after the first.
3183 PCRE normally treats such a pattern as though it were preceded by \A.
3184
3185 In cases where it is known that the subject string contains no new-
3186 lines, it is worth setting PCRE_DOTALL in order to obtain this opti-
3187 mization, or alternatively using ^ to indicate anchoring explicitly.
3188
3189 However, there is one situation where the optimization cannot be used.
3190 When .* is inside capturing parentheses that are the subject of a
3191 backreference elsewhere in the pattern, a match at the start may fail,
3192 and a later one succeed. Consider, for example:
3193
3194 (.*)abc\1
3195
3196 If the subject is "xyz123abc123" the match point is the fourth charac-
3197 ter. For this reason, such a pattern is not implicitly anchored.
3198
3199 When a capturing subpattern is repeated, the value captured is the sub-
3200 string that matched the final iteration. For example, after
3201
3202 (tweedle[dume]{3}\s*)+
3203
3204 has matched "tweedledum tweedledee" the value of the captured substring
3205 is "tweedledee". However, if there are nested capturing subpatterns,
3206 the corresponding captured values may have been set in previous itera-
3207 tions. For example, after
3208
3209 /(a|(b))+/
3210
3211 matches "aba" the value of the second captured substring is "b".
3212
3213
3214 ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS
3215
3216 With both maximizing and minimizing repetition, failure of what follows
3217 normally causes the repeated item to be re-evaluated to see if a dif-
3218 ferent number of repeats allows the rest of the pattern to match. Some-
3219 times it is useful to prevent this, either to change the nature of the
3220 match, or to cause it fail earlier than it otherwise might, when the
3221 author of the pattern knows there is no point in carrying on.
3222
3223 Consider, for example, the pattern \d+foo when applied to the subject
3224 line
3225
3226 123456bar
3227
3228 After matching all 6 digits and then failing to match "foo", the normal
3229 action of the matcher is to try again with only 5 digits matching the
3230 \d+ item, and then with 4, and so on, before ultimately failing.
3231 "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides
3232 the means for specifying that once a subpattern has matched, it is not
3233 to be re-evaluated in this way.
3234
3235 If we use atomic grouping for the previous example, the matcher would
3236 give up immediately on failing to match "foo" the first time. The nota-
3237 tion is a kind of special parenthesis, starting with (?> as in this
3238 example:
3239
3240 (?>\d+)foo
3241
3242 This kind of parenthesis "locks up" the part of the pattern it con-
3243 tains once it has matched, and a failure further into the pattern is
3244 prevented from backtracking into it. Backtracking past it to previous
3245 items, however, works as normal.
3246
3247 An alternative description is that a subpattern of this type matches
3248 the string of characters that an identical standalone pattern would
3249 match, if anchored at the current point in the subject string.
3250
3251 Atomic grouping subpatterns are not capturing subpatterns. Simple cases
3252 such as the above example can be thought of as a maximizing repeat that
3253 must swallow everything it can. So, while both \d+ and \d+? are pre-
3254 pared to adjust the number of digits they match in order to make the
3255 rest of the pattern match, (?>\d+) can only match an entire sequence of
3256 digits.
3257
3258 Atomic groups in general can of course contain arbitrarily complicated
3259 subpatterns, and can be nested. However, when the subpattern for an
3260 atomic group is just a single repeated item, as in the example above, a
3261 simpler notation, called a "possessive quantifier" can be used. This
3262 consists of an additional + character following a quantifier. Using
3263 this notation, the previous example can be rewritten as
3264
3265 \d++foo
3266
3267 Possessive quantifiers are always greedy; the setting of the
3268 PCRE_UNGREEDY option is ignored. They are a convenient notation for the
3269 simpler forms of atomic group. However, there is no difference in the
3270 meaning or processing of a possessive quantifier and the equivalent
3271 atomic group.
3272
3273 The possessive quantifier syntax is an extension to the Perl syntax. It
3274 originates in Sun's Java package.
3275
3276 When a pattern contains an unlimited repeat inside a subpattern that
3277 can itself be repeated an unlimited number of times, the use of an
3278 atomic group is the only way to avoid some failing matches taking a
3279 very long time indeed. The pattern
3280
3281 (\D+|<\d+>)*[!?]
3282
3283 matches an unlimited number of substrings that either consist of non-
3284 digits, or digits enclosed in <>, followed by either ! or ?. When it
3285 matches, it runs quickly. However, if it is applied to
3286
3287 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
3288
3289 it takes a long time before reporting failure. This is because the
3290 string can be divided between the internal \D+ repeat and the external
3291 * repeat in a large number of ways, and all have to be tried. (The
3292 example uses [!?] rather than a single character at the end, because
3293 both PCRE and Perl have an optimization that allows for fast failure
3294 when a single character is used. They remember the last single charac-
3295 ter that is required for a match, and fail early if it is not present
3296 in the string.) If the pattern is changed so that it uses an atomic
3297 group, like this:
3298
3299 ((?>\D+)|<\d+>)*[!?]
3300
3301 sequences of non-digits cannot be broken, and failure happens quickly.
3302
3303
3304 BACK REFERENCES
3305
3306 Outside a character class, a backslash followed by a digit greater than
3307 0 (and possibly further digits) is a back reference to a capturing sub-
3308 pattern earlier (that is, to its left) in the pattern, provided there
3309 have been that many previous capturing left parentheses.
3310
3311 However, if the decimal number following the backslash is less than 10,
3312 it is always taken as a back reference, and causes an error only if
3313 there are not that many capturing left parentheses in the entire pat-
3314 tern. In other words, the parentheses that are referenced need not be
3315 to the left of the reference for numbers less than 10. See the subsec-
3316 tion entitled "Non-printing characters" above for further details of
3317 the handling of digits following a backslash.
3318
3319 A back reference matches whatever actually matched the capturing sub-
3320 pattern in the current subject string, rather than anything matching
3321 the subpattern itself (see "Subpatterns as subroutines" below for a way
3322 of doing that). So the pattern
3323
3324 (sens|respons)e and \1ibility
3325
3326 matches "sense and sensibility" and "response and responsibility", but
3327 not "sense and responsibility". If caseful matching is in force at the
3328 time of the back reference, the case of letters is relevant. For exam-
3329 ple,
3330
3331 ((?i)rah)\s+\1
3332
3333 matches "rah rah" and "RAH RAH", but not "RAH rah", even though the
3334 original capturing subpattern is matched caselessly.
3335
3336 Back references to named subpatterns use the Python syntax (?P=name).
3337 We could rewrite the above example as follows:
3338
3339 (?<p1>(?i)rah)\s+(?P=p1)
3340
3341 There may be more than one back reference to the same subpattern. If a
3342 subpattern has not actually been used in a particular match, any back
3343 references to it always fail. For example, the pattern
3344
3345 (a|(bc))\2
3346
3347 always fails if it starts to match "a" rather than "bc". Because there
3348 may be many capturing parentheses in a pattern, all digits following
3349 the backslash are taken as part of a potential back reference number.
3350 If the pattern continues with a digit character, some delimiter must be
3351 used to terminate the back reference. If the PCRE_EXTENDED option is
3352 set, this can be whitespace. Otherwise an empty comment (see "Com-
3353 ments" below) can be used.
3354
3355 A back reference that occurs inside the parentheses to which it refers
3356 fails when the subpattern is first used, so, for example, (a\1) never
3357 matches. However, such references can be useful inside repeated sub-
3358 patterns. For example, the pattern
3359
3360 (a|b\1)+
3361
3362 matches any number of "a"s and also "aba", "ababbaa" etc. At each iter-
3363 ation of the subpattern, the back reference matches the character
3364 string corresponding to the previous iteration. In order for this to
3365 work, the pattern must be such that the first iteration does not need
3366 to match the back reference. This can be done using alternation, as in
3367 the example above, or by a quantifier with a minimum of zero.
3368
3369
3370 ASSERTIONS
3371
3372 An assertion is a test on the characters following or preceding the
3373 current matching point that does not actually consume any characters.
3374 The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are
3375 described above.
3376
3377 More complicated assertions are coded as subpatterns. There are two
3378 kinds: those that look ahead of the current position in the subject
3379 string, and those that look behind it. An assertion subpattern is
3380 matched in the normal way, except that it does not cause the current
3381 matching position to be changed.
3382
3383 Assertion subpatterns are not capturing subpatterns, and may not be
3384 repeated, because it makes no sense to assert the same thing several
3385 times. If any kind of assertion contains capturing subpatterns within
3386 it, these are counted for the purposes of numbering the capturing sub-
3387 patterns in the whole pattern. However, substring capturing is carried
3388 out only for positive assertions, because it does not make sense for
3389 negative assertions.
3390
3391 Lookahead assertions
3392
3393 Lookahead assertions start with (?= for positive assertions and (?! for
3394 negative assertions. For example,
3395
3396 \w+(?=;)
3397
3398 matches a word followed by a semicolon, but does not include the semi-
3399 colon in the match, and
3400
3401 foo(?!bar)
3402
3403 matches any occurrence of "foo" that is not followed by "bar". Note
3404 that the apparently similar pattern
3405
3406 (?!foo)bar
3407
3408 does not find an occurrence of "bar" that is preceded by something
3409 other than "foo"; it finds any occurrence of "bar" whatsoever, because
3410 the assertion (?!foo) is always true when the next three characters are
3411 "bar". A lookbehind assertion is needed to achieve the other effect.
3412
3413 If you want to force a matching failure at some point in a pattern, the
3414 most convenient way to do it is with (?!) because an empty string
3415 always matches, so an assertion that requires there not to be an empty
3416 string must always fail.
3417
3418 Lookbehind assertions
3419
3420 Lookbehind assertions start with (?<= for positive assertions and (?<!
3421 for negative assertions. For example,
3422
3423 (?<!foo)bar
3424
3425 does find an occurrence of "bar" that is not preceded by "foo". The
3426 contents of a lookbehind assertion are restricted such that all the
3427 strings it matches must have a fixed length. However, if there are sev-
3428 eral alternatives, they do not all have to have the same fixed length.
3429 Thus
3430
3431 (?<=bullock|donkey)
3432
3433 is permitted, but
3434
3435 (?<!dogs?|cats?)
3436
3437 causes an error at compile time. Branches that match different length
3438 strings are permitted only at the top level of a lookbehind assertion.
3439 This is an extension compared with Perl (at least for 5.8), which
3440 requires all branches to match the same length of string. An assertion
3441 such as
3442
3443 (?<=ab(c|de))
3444
3445 is not permitted, because its single top-level branch can match two
3446 different lengths, but it is acceptable if rewritten to use two top-
3447 level branches:
3448
3449 (?<=abc|abde)
3450
3451 The implementation of lookbehind assertions is, for each alternative,
3452 to temporarily move the current position back by the fixed width and
3453 then try to match. If there are insufficient characters before the cur-
3454 rent position, the match is deemed to fail.
3455
3456 PCRE does not allow the \C escape (which matches a single byte in UTF-8
3457 mode) to appear in lookbehind assertions, because it makes it impossi-
3458 ble to calculate the length of the lookbehind. The \X escape, which can
3459 match different numbers of bytes, is also not permitted.
3460
3461 Atomic groups can be used in conjunction with lookbehind assertions to
3462 specify efficient matching at the end of the subject string. Consider a
3463 simple pattern such as
3464
3465 abcd$
3466
3467 when applied to a long string that does not match. Because matching
3468 proceeds from left to right, PCRE will look for each "a" in the subject
3469 and then see if what follows matches the rest of the pattern. If the
3470 pattern is specified as
3471
3472 ^.*abcd$
3473
3474 the initial .* matches the entire string at first, but when this fails
3475 (because there is no following "a"), it backtracks to match all but the
3476 last character, then all but the last two characters, and so on. Once
3477 again the search for "a" covers the entire string, from right to left,
3478 so we are no better off. However, if the pattern is written as
3479
3480 ^(?>.*)(?<=abcd)
3481
3482 or, equivalently, using the possessive quantifier syntax,
3483
3484 ^.*+(?<=abcd)
3485
3486 there can be no backtracking for the .* item; it can match only the
3487 entire string. The subsequent lookbehind assertion does a single test
3488 on the last four characters. If it fails, the match fails immediately.
3489 For long strings, this approach makes a significant difference to the
3490 processing time.
3491
3492 Using multiple assertions
3493
3494 Several assertions (of any sort) may occur in succession. For example,
3495
3496 (?<=\d{3})(?<!999)foo
3497
3498 matches "foo" preceded by three digits that are not "999". Notice that
3499 each of the assertions is applied independently at the same point in
3500 the subject string. First there is a check that the previous three
3501 characters are all digits, and then there is a check that the same
3502 three characters are not "999". This pattern does not match "foo" pre-
3503 ceded by six characters, the first of which are digits and the last
3504 three of which are not "999". For example, it doesn't match "123abc-
3505 foo". A pattern to do that is
3506
3507 (?<=\d{3}...)(?<!999)foo
3508
3509 This time the first assertion looks at the preceding six characters,
3510 checking that the first three are digits, and then the second assertion
3511 checks that the preceding three characters are not "999".
3512
3513 Assertions can be nested in any combination. For example,
3514
3515 (?<=(?<!foo)bar)baz
3516
3517 matches an occurrence of "baz" that is preceded by "bar" which in turn
3518 is not preceded by "foo", while
3519
3520 (?<=\d{3}(?!999)...)foo
3521
3522 is another pattern that matches "foo" preceded by three digits and any
3523 three characters that are not "999".
3524
3525
3526 CONDITIONAL SUBPATTERNS
3527
3528 It is possible to cause the matching process to obey a subpattern con-
3529 ditionally or to choose between two alternative subpatterns, depending
3530 on the result of an assertion, or whether a previous capturing subpat-
3531 tern matched or not. The two possible forms of conditional subpattern
3532 are
3533
3534 (?(condition)yes-pattern)
3535 (?(condition)yes-pattern|no-pattern)
3536
3537 If the condition is satisfied, the yes-pattern is used; otherwise the
3538 no-pattern (if present) is used. If there are more than two alterna-
3539 tives in the subpattern, a compile-time error occurs.
3540
3541 There are three kinds of condition. If the text between the parentheses
3542 consists of a sequence of digits, the condition is satisfied if the
3543 capturing subpattern of that number has previously matched. The number
3544 must be greater than zero. Consider the following pattern, which con-
3545 tains non-significant white space to make it more readable (assume the
3546 PCRE_EXTENDED option) and to divide it into three parts for ease of
3547 discussion:
3548
3549 ( \( )? [^()]+ (?(1) \) )
3550
3551 The first part matches an optional opening parenthesis, and if that
3552 character is present, sets it as the first captured substring. The sec-
3553 ond part matches one or more characters that are not parentheses. The
3554 third part is a conditional subpattern that tests whether the first set
3555 of parentheses matched or not. If they did, that is, if subject started
3556 with an opening parenthesis, the condition is true, and so the yes-pat-
3557 tern is executed and a closing parenthesis is required. Otherwise,
3558 since no-pattern is not present, the subpattern matches nothing. In
3559 other words, this pattern matches a sequence of non-parentheses,
3560 optionally enclosed in parentheses.
3561
3562 If the condition is the string (R), it is satisfied if a recursive call
3563 to the pattern or subpattern has been made. At "top level", the condi-
3564 tion is false. This is a PCRE extension. Recursive patterns are
3565 described in the next section.
3566
3567 If the condition is not a sequence of digits or (R), it must be an
3568 assertion. This may be a positive or negative lookahead or lookbehind
3569 assertion. Consider this pattern, again containing non-significant
3570 white space, and with the two alternatives on the second line:
3571
3572 (?(?=[^a-z]*[a-z])
3573 \d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} )
3574
3575 The condition is a positive lookahead assertion that matches an
3576 optional sequence of non-letters followed by a letter. In other words,
3577 it tests for the presence of at least one letter in the subject. If a
3578 letter is found, the subject is matched against the first alternative;
3579 otherwise it is matched against the second. This pattern matches
3580 strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are
3581 letters and dd are digits.
3582
3583
3584 COMMENTS
3585
3586 The sequence (?# marks the start of a comment that continues up to the
3587 next closing parenthesis. Nested parentheses are not permitted. The
3588 characters that make up a comment play no part in the pattern matching
3589 at all.
3590
3591 If the PCRE_EXTENDED option is set, an unescaped # character outside a
3592 character class introduces a comment that continues up to the next new-
3593 line character in the pattern.
3594
3595
3596 RECURSIVE PATTERNS
3597
3598 Consider the problem of matching a string in parentheses, allowing for
3599 unlimited nested parentheses. Without the use of recursion, the best
3600 that can be done is to use a pattern that matches up to some fixed
3601 depth of nesting. It is not possible to handle an arbitrary nesting
3602 depth. Perl provides a facility that allows regular expressions to
3603 recurse (amongst other things). It does this by interpolating Perl code
3604 in the expression at run time, and the code can refer to the expression
3605 itself. A Perl pattern to solve the parentheses problem can be created
3606 like this:
3607
3608 $re = qr{\( (?: (?>[^()]+) | (?p{$re}) )* \)}x;
3609
3610 The (?p{...}) item interpolates Perl code at run time, and in this case
3611 refers recursively to the pattern in which it appears. Obviously, PCRE
3612 cannot support the interpolation of Perl code. Instead, it supports
3613 some special syntax for recursion of the entire pattern, and also for
3614 individual subpattern recursion.
3615
3616 The special item that consists of (? followed by a number greater than
3617 zero and a closing parenthesis is a recursive call of the subpattern of
3618 the given number, provided that it occurs inside that subpattern. (If
3619 not, it is a "subroutine" call, which is described in the next sec-
3620 tion.) The special item (?R) is a recursive call of the entire regular
3621 expression.
3622
3623 For example, this PCRE pattern solves the nested parentheses problem
3624 (assume the PCRE_EXTENDED option is set so that white space is
3625 ignored):
3626
3627 \( ( (?>[^()]+) | (?R) )* \)
3628
3629 First it matches an opening parenthesis. Then it matches any number of
3630 substrings which can either be a sequence of non-parentheses, or a
3631 recursive match of the pattern itself (that is a correctly parenthe-
3632 sized substring). Finally there is a closing parenthesis.
3633
3634 If this were part of a larger pattern, you would not want to recurse
3635 the entire pattern, so instead you could use this:
3636
3637 ( \( ( (?>[^()]+) | (?1) )* \) )
3638
3639 We have put the pattern into parentheses, and caused the recursion to
3640 refer to them instead of the whole pattern. In a larger pattern, keep-
3641 ing track of parenthesis numbers can be tricky. It may be more conve-
3642 nient to use named parentheses instead. For this, PCRE uses (?P>name),
3643 which is an extension to the Python syntax that PCRE uses for named
3644 parentheses (Perl does not provide named parentheses). We could rewrite
3645 the above example as follows:
3646
3647 (?P<pn> \( ( (?>[^()]+) | (?P>pn) )* \) )
3648
3649 This particular example pattern contains nested unlimited repeats, and
3650 so the use of atomic grouping for matching strings of non-parentheses
3651 is important when applying the pattern to strings that do not match.
3652 For example, when this pattern is applied to
3653
3654 (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
3655
3656 it yields "no match" quickly. However, if atomic grouping is not used,
3657 the match runs for a very long time indeed because there are so many
3658 different ways the + and * repeats can carve up the subject, and all
3659 have to be tested before failure can be reported.
3660
3661 At the end of a match, the values set for any capturing subpatterns are
3662 those from the outermost level of the recursion at which the subpattern
3663 value is set. If you want to obtain intermediate values, a callout
3664 function can be used (see the next section and the pcrecallout documen-
3665 tation). If the pattern above is matched against
3666
3667 (ab(cd)ef)
3668
3669 the value for the capturing parentheses is "ef", which is the last
3670 value taken on at the top level. If additional parentheses are added,
3671 giving
3672
3673 \( ( ( (?>[^()]+) | (?R) )* ) \)
3674 ^ ^
3675 ^ ^
3676
3677 the string they capture is "ab(cd)ef", the contents of the top level
3678 parentheses. If there are more than 15 capturing parentheses in a pat-
3679 tern, PCRE has to obtain extra memory to store data during a recursion,
3680 which it does by using pcre_malloc, freeing it via pcre_free after-
3681 wards. If no memory can be obtained, the match fails with the
3682 PCRE_ERROR_NOMEMORY error.
3683
3684 Do not confuse the (?R) item with the condition (R), which tests for
3685 recursion. Consider this pattern, which matches text in angle brack-
3686 ets, allowing for arbitrary nesting. Only digits are allowed in nested
3687 brackets (that is, when recursing), whereas any characters are permit-
3688 ted at the outer level.
3689
3690 < (?: (?(R) \d++ | [^<>]*+) | (?R)) * >
3691
3692 In this pattern, (?(R) is the start of a conditional subpattern, with
3693 two different alternatives for the recursive and non-recursive cases.
3694 The (?R) item is the actual recursive call.
3695
3696
3697 SUBPATTERNS AS SUBROUTINES
3698
3699 If the syntax for a recursive subpattern reference (either by number or
3700 by name) is used outside the parentheses to which it refers, it oper-
3701 ates like a subroutine in a programming language. An earlier example
3702 pointed out that the pattern
3703
3704 (sens|respons)e and \1ibility
3705
3706 matches "sense and sensibility" and "response and responsibility", but
3707 not "sense and responsibility". If instead the pattern
3708
3709 (sens|respons)e and (?1)ibility
3710
3711 is used, it does match "sense and responsibility" as well as the other
3712 two strings. Such references must, however, follow the subpattern to
3713 which they refer.
3714
3715
3716 CALLOUTS
3717
3718 Perl has a feature whereby using the sequence (?{...}) causes arbitrary
3719 Perl code to be obeyed in the middle of matching a regular expression.
3720 This makes it possible, amongst other things, to extract different sub-
3721 strings that match the same pair of parentheses when there is a repeti-
3722 tion.
3723
3724 PCRE provides a similar feature, but of course it cannot obey arbitrary
3725 Perl code. The feature is called "callout". The caller of PCRE provides
3726 an external function by putting its entry point in the global variable
3727 pcre_callout. By default, this variable contains NULL, which disables
3728 all calling out.
3729
3730 Within a regular expression, (?C) indicates the points at which the
3731 external function is to be called. If you want to identify different
3732 callout points, you can put a number less than 256 after the letter C.
3733 The default value is zero. For example, this pattern has two callout
3734 points:
3735
3736 (?C1)abc(?C2)def
3737
3738 If the PCRE_AUTO_CALLOUT flag is passed to pcre_compile(), callouts are
3739 automatically installed before each item in the pattern. They are all
3740 numbered 255.
3741
3742 During matching, when PCRE reaches a callout point (and pcre_callout is
3743 set), the external function is called. It is provided with the number
3744 of the callout, the position in the pattern, and, optionally, one item
3745 of data originally supplied by the caller of pcre_exec(). The callout
3746 function may cause matching to proceed, to backtrack, or to fail alto-
3747 gether. A complete description of the interface to the callout function
3748 is given in the pcrecallout documentation.
3749
3750 Last updated: 28 February 2005
3751 Copyright (c) 1997-2005 University of Cambridge.
3752 ------------------------------------------------------------------------------
3753
3754
3755 PCREPARTIAL(3) PCREPARTIAL(3)
3756
3757
3758 NAME
3759 PCRE - Perl-compatible regular expressions
3760
3761
3762 PARTIAL MATCHING IN PCRE
3763
3764 In normal use of PCRE, if the subject string that is passed to
3765 pcre_exec() or pcre_dfa_exec() matches as far as it goes, but is too
3766 short to match the entire pattern, PCRE_ERROR_NOMATCH is returned.
3767 There are circumstances where it might be helpful to distinguish this
3768 case from other cases in which there is no match.
3769
3770 Consider, for example, an application where a human is required to type
3771 in data for a field with specific formatting requirements. An example
3772 might be a date in the form ddmmmyy, defined by this pattern:
3773
3774 ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$
3775
3776 If the application sees the user's keystrokes one by one, and can check
3777 that what has been typed so far is potentially valid, it is able to
3778 raise an error as soon as a mistake is made, possibly beeping and not
3779 reflecting the character that has been typed. This immediate feedback
3780 is likely to be a better user interface than a check that is delayed
3781 until the entire string has been entered.
3782
3783 PCRE supports the concept of partial matching by means of the PCRE_PAR-
3784 TIAL option, which can be set when calling pcre_exec() or
3785 pcre_dfa_exec(). When this flag is set for pcre_exec(), the return code
3786 PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time
3787 during the matching process the last part of the subject string matched
3788 part of the pattern. Unfortunately, for non-anchored matching, it is
3789 not possible to obtain the position of the start of the partial match.
3790 No captured data is set when PCRE_ERROR_PARTIAL is returned.
3791
3792 When PCRE_PARTIAL is set for pcre_dfa_exec(), the return code
3793 PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of
3794 the subject is reached, there have been no complete matches, but there
3795 is still at least one matching possibility. The portion of the string
3796 that provided the partial match is set as the first matching string.
3797
3798 Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers
3799 the last literal byte in a pattern, and abandons matching immediately
3800 if such a byte is not present in the subject string. This optimization
3801 cannot be used for a subject string that might match only partially.
3802
3803
3804 RESTRICTED PATTERNS FOR PCRE_PARTIAL
3805
3806 Because of the way certain internal optimizations are implemented in
3807 the pcre_exec() function, the PCRE_PARTIAL option cannot be used with
3808 all patterns. These restrictions do not apply when pcre_dfa_exec() is
3809 used. For pcre_exec(), repeated single characters such as
3810
3811 a{2,4}
3812
3813 and repeated single metasequences such as
3814
3815 \d+
3816
3817 are not permitted if the maximum number of occurrences is greater than
3818 one. Optional items such as \d? (where the maximum is one) are permit-
3819 ted. Quantifiers with any values are permitted after parentheses, so
3820 the invalid examples above can be coded thus:
3821
3822 (a){2,4}
3823 (\d)+
3824
3825 These constructions run more slowly, but for the kinds of application
3826 that are envisaged for this facility, this is not felt to be a major
3827 restriction.
3828
3829 If PCRE_PARTIAL is set for a pattern that does not conform to the
3830 restrictions, pcre_exec() returns the error code PCRE_ERROR_BADPARTIAL
3831 (-13).
3832
3833
3834 EXAMPLE OF PARTIAL MATCHING USING PCRETEST
3835
3836 If the escape sequence \P is present in a pcretest data line, the
3837 PCRE_PARTIAL flag is used for the match. Here is a run of pcretest that
3838 uses the date example quoted above:
3839
3840 re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
3841 data> 25jun04P
3842 0: 25jun04
3843 1: jun
3844 data> 25dec3P
3845 Partial match
3846 data> 3juP
3847 Partial match
3848 data> 3jujP
3849 No match
3850 data> jP
3851 No match
3852
3853 The first data string is matched completely, so pcretest shows the
3854 matched substrings. The remaining four strings do not match the com-
3855 plete pattern, but the first two are partial matches. The same test,
3856 using DFA matching (by means of the \D escape sequence), produces the
3857 following output:
3858
3859 re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/
3860 data> 25jun04\P\D
3861 0: 25jun04
3862 data> 23dec3\P\D
3863 Partial match: 23dec3
3864 data> 3ju\P\D
3865 Partial match: 3ju
3866 data> 3juj\P\D
3867 No match
3868 data> j\P\D
3869 No match
3870
3871 Notice that in this case the portion of the string that was matched is
3872 made available.
3873
3874
3875 MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()
3876
3877 When a partial match has been found using pcre_dfa_exec(), it is possi-
3878 ble to continue the match by providing additional subject data and
3879 calling pcre_dfa_exec() again with the PCRE_DFA_RESTART option and the
3880 same working space (where details of the previous partial match are
3881 stored). Here is an example using pcretest, where the \R escape
3882 sequence sets the PCRE_DFA_RESTART option and the \D escape sequence
3883 requests the use of pcre_dfa_exec():
3884
3885 re> /^?(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)$/
3886 data> 23ja\P\D
3887 Partial match: 23ja
3888 data> n05\R\D
3889 0: n05
3890
3891 The first call has "23ja" as the subject, and requests partial match-
3892 ing; the second call has "n05" as the subject for the continued
3893 (restarted) match. Notice that when the match is complete, only the
3894 last part is shown; PCRE does not retain the previously partially-
3895 matched string. It is up to the calling program to do that if it needs
3896 to.
3897
3898 This facility can be used to pass very long subject strings to
3899 pcre_dfa_exec(). However, some care is needed for certain types of pat-
3900 tern.
3901
3902 1. If the pattern contains tests for the beginning or end of a line,
3903 you need to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropri-
3904 ate, when the subject string for any call does not contain the begin-
3905 ning or end of a line.
3906
3907 2. If the pattern contains backward assertions (including \b or \B),
3908 you need to arrange for some overlap in the subject strings to allow
3909 for this. For example, you could pass the subject in chunks that were
3910 500 bytes long, but in a buffer of 700 bytes, with the starting offset
3911 set to 200 and the previous 200 bytes at the start of the buffer.
3912
3913 3. Matching a subject string that is split into multiple segments does
3914 not always produce exactly the same result as matching over one single
3915 long string. The difference arises when there are multiple matching
3916 possibilities, because a partial match result is given only when there
3917 are no completed matches in a call to fBpcre_dfa_exec(). This means
3918 that as soon as the shortest match has been found, continuation to a
3919 new subject segment is no longer possible. Consider this pcretest
3920 example:
3921
3922 re> /dog(sbody)?/
3923 data> do\P\D
3924 Partial match: do
3925 data> gsb\R\P\D
3926 0: g
3927 data> dogsbody\D
3928 0: dogsbody
3929 1: dog
3930
3931 The pattern matches the words "dog" or "dogsbody". When the subject is
3932 presented in several parts ("do" and "gsb" being the first two) the
3933 match stops when "dog" has been found, and it is not possible to con-
3934 tinue. On the other hand, if "dogsbody" is presented as a single
3935 string, both matches are found.
3936
3937 Because of this phenomenon, it does not usually make sense to end a
3938 pattern that is going to be matched in this way with a variable repeat.
3939
3940 Last updated: 28 February 2005
3941 Copyright (c) 1997-2005 University of Cambridge.
3942 ------------------------------------------------------------------------------
3943
3944
3945 PCREPRECOMPILE(3) PCREPRECOMPILE(3)
3946
3947
3948 NAME
3949 PCRE - Perl-compatible regular expressions
3950
3951
3952 SAVING AND RE-USING PRECOMPILED PCRE PATTERNS
3953
3954 If you are running an application that uses a large number of regular
3955 expression patterns, it may be useful to store them in a precompiled
3956 form instead of having to compile them every time the application is
3957 run. If you are not using any private character tables (see the
3958 pcre_maketables() documentation), this is relatively straightforward.
3959 If you are using private tables, it is a little bit more complicated.
3960
3961 If you save compiled patterns to a file, you can copy them to a differ-
3962 ent host and run them there. This works even if the new host has the
3963 opposite endianness to the one on which the patterns were compiled.
3964 There may be a small performance penalty, but it should be insignifi-
3965 cant.
3966
3967
3968 SAVING A COMPILED PATTERN
3969 The value returned by pcre_compile() points to a single block of memory
3970 that holds the compiled pattern and associated data. You can find the
3971 length of this block in bytes by calling pcre_fullinfo() with an argu-
3972 ment of PCRE_INFO_SIZE. You can then save the data in any appropriate
3973 manner. Here is sample code that compiles a pattern and writes it to a
3974 file. It assumes that the variable fd refers to a file that is open for
3975 output:
3976
3977 int erroroffset, rc, size;
3978 char *error;
3979 pcre *re;
3980
3981 re = pcre_compile("my pattern", 0, &error, &erroroffset, NULL);
3982 if (re == NULL) { ... handle errors ... }
3983 rc = pcre_fullinfo(re, NULL, PCRE_INFO_SIZE, &size);
3984 if (rc < 0) { ... handle errors ... }
3985 rc = fwrite(re, 1, size, fd);
3986 if (rc != size) { ... handle errors ... }
3987
3988 In this example, the bytes that comprise the compiled pattern are
3989 copied exactly. Note that this is binary data that may contain any of
3990 the 256 possible byte values. On systems that make a distinction
3991 between binary and non-binary data, be sure that the file is opened for
3992 binary output.
3993
3994 If you want to write more than one pattern to a file, you will have to
3995 devise a way of separating them. For binary data, preceding each pat-
3996 tern with its length is probably the most straightforward approach.
3997 Another possibility is to write out the data in hexadecimal instead of
3998 binary, one pattern to a line.
3999
4000 Saving compiled patterns in a file is only one possible way of storing
4001 them for later use. They could equally well be saved in a database, or
4002 in the memory of some daemon process that passes them via sockets to
4003 the processes that want them.
4004
4005 If the pattern has been studied, it is also possible to save the study
4006 data in a similar way to the compiled pattern itself. When studying
4007 generates additional information, pcre_study() returns a pointer to a
4008 pcre_extra data block. Its format is defined in the section on matching
4009 a pattern in the pcreapi documentation. The study_data field points to
4010 the binary study data, and this is what you must save (not the
4011 pcre_extra block itself). The length of the study data can be obtained
4012 by calling pcre_fullinfo() with an argument of PCRE_INFO_STUDYSIZE.
4013 Remember to check that pcre_study() did return a non-NULL value before
4014 trying to save the study data.
4015
4016
4017 RE-USING A PRECOMPILED PATTERN
4018
4019 Re-using a precompiled pattern is straightforward. Having reloaded it
4020 into main memory, you pass its pointer to pcre_exec() or
4021 pcre_dfa_exec() in the usual way. This should work even on another
4022 host, and even if that host has the opposite endianness to the one
4023 where the pattern was compiled.
4024
4025 However, if you passed a pointer to custom character tables when the
4026 pattern was compiled (the tableptr argument of pcre_compile()), you
4027 must now pass a similar pointer to pcre_exec() or pcre_dfa_exec(),
4028 because the value saved with the compiled pattern will obviously be
4029 nonsense. A field in a pcre_extra() block is used to pass this data, as
4030 described in the section on matching a pattern in the pcreapi documen-
4031 tation.
4032
4033 If you did not provide custom character tables when the pattern was
4034 compiled, the pointer in the compiled pattern is NULL, which causes
4035 pcre_exec() to use PCRE's internal tables. Thus, you do not need to
4036 take any special action at run time in this case.
4037
4038 If you saved study data with the compiled pattern, you need to create
4039 your own pcre_extra data block and set the study_data field to point to
4040 the reloaded study data. You must also set the PCRE_EXTRA_STUDY_DATA
4041 bit in the flags field to indicate that study data is present. Then
4042 pass the pcre_extra block to pcre_exec() or pcre_dfa_exec() in the
4043 usual way.
4044
4045
4046 COMPATIBILITY WITH DIFFERENT PCRE RELEASES
4047
4048 The layout of the control block that is at the start of the data that
4049 makes up a compiled pattern was changed for release 5.0. If you have
4050 any saved patterns that were compiled with previous releases (not a
4051 facility that was previously advertised), you will have to recompile
4052 them for release 5.0. However, from now on, it should be possible to
4053 make changes in a compatible manner.
4054
4055 Last updated: 28 February 2005
4056 Copyright (c) 1997-2005 University of Cambridge.
4057 ------------------------------------------------------------------------------
4058
4059
4060 PCREPERFORM(3) PCREPERFORM(3)
4061
4062
4063 NAME
4064 PCRE - Perl-compatible regular expressions
4065
4066
4067 PCRE PERFORMANCE
4068
4069 Certain items that may appear in regular expression patterns are more
4070 efficient than others. It is more efficient to use a character class
4071 like [aeiou] than a set of alternatives such as (a|e|i|o|u). In gen-
4072 eral, the simplest construction that provides the required behaviour is
4073 usually the most efficient. Jeffrey Friedl's book contains a lot of
4074 useful general discussion about optimizing regular expressions for
4075 efficient performance. This document contains a few observations about
4076 PCRE.
4077
4078 Using Unicode character properties (the \p, \P, and \X escapes) is
4079 slow, because PCRE has to scan a structure that contains data for over
4080 fifteen thousand characters whenever it needs a character's property.
4081 If you can find an alternative pattern that does not use character
4082 properties, it will probably be faster.
4083
4084 When a pattern begins with .* not in parentheses, or in parentheses
4085 that are not the subject of a backreference, and the PCRE_DOTALL option
4086 is set, the pattern is implicitly anchored by PCRE, since it can match
4087 only at the start of a subject string. However, if PCRE_DOTALL is not
4088 set, PCRE cannot make this optimization, because the . metacharacter
4089 does not then match a newline, and if the subject string contains new-
4090 lines, the pattern may match from the character immediately following
4091 one of them instead of from the very start. For example, the pattern
4092
4093 .*second
4094
4095 matches the subject "first\nand second" (where \n stands for a newline
4096 character), with the match starting at the seventh character. In order
4097 to do this, PCRE has to retry the match starting after every newline in
4098 the subject.
4099
4100 If you are using such a pattern with subject strings that do not con-
4101 tain newlines, the best performance is obtained by setting PCRE_DOTALL,
4102 or starting the pattern with ^.* or ^.*? to indicate explicit anchor-
4103 ing. That saves PCRE from having to scan along the subject looking for
4104 a newline to restart at.
4105
4106 Beware of patterns that contain nested indefinite repeats. These can
4107 take a long time to run when applied to a string that does not match.
4108 Consider the pattern fragment
4109
4110 (a+)*
4111
4112 This can match "aaaa" in 33 different ways, and this number increases
4113 very rapidly as the string gets longer. (The * repeat can match 0, 1,
4114 2, 3, or 4 times, and for each of those cases other than 0, the +
4115 repeats can match different numbers of times.) When the remainder of
4116 the pattern is such that the entire match is going to fail, PCRE has in
4117 principle to try every possible variation, and this can take an
4118 extremely long time.
4119
4120 An optimization catches some of the more simple cases such as
4121
4122 (a+)*b
4123
4124 where a literal character follows. Before embarking on the standard
4125 matching procedure, PCRE checks that there is a "b" later in the sub-
4126 ject string, and if there is not, it fails the match immediately. How-
4127 ever, when there is no following literal this optimization cannot be
4128 used. You can see the difference by comparing the behaviour of
4129
4130 (a+)*\d
4131
4132 with the pattern above. The former gives a failure almost instantly
4133 when applied to a whole line of "a" characters, whereas the latter
4134 takes an appreciable time with strings longer than about 20 characters.
4135
4136 In many cases, the solution to this kind of performance issue is to use
4137 an atomic group or a possessive quantifier.
4138
4139 Last updated: 28 February 2005
4140 Copyright (c) 1997-2005 University of Cambridge.
4141 ------------------------------------------------------------------------------
4142
4143
4144 PCREPOSIX(3) PCREPOSIX(3)
4145
4146
4147 NAME
4148 PCRE - Perl-compatible regular expressions.
4149
4150
4151 SYNOPSIS OF POSIX API
4152
4153 #include <pcreposix.h>
4154
4155 int regcomp(regex_t *preg, const char *pattern,
4156 int cflags);
4157
4158 int regexec(regex_t *preg, const char *string,
4159 size_t nmatch, regmatch_t pmatch[], int eflags);
4160
4161 size_t regerror(int errcode, const regex_t *preg,
4162 char *errbuf, size_t errbuf_size);
4163
4164 void regfree(regex_t *preg);
4165
4166
4167 DESCRIPTION
4168
4169 This set of functions provides a POSIX-style API to the PCRE regular
4170 expression package. See the pcreapi documentation for a description of
4171 PCRE's native API, which contains much additional functionality.
4172
4173 The functions described here are just wrapper functions that ultimately
4174 call the PCRE native API. Their prototypes are defined in the
4175 pcreposix.h header file, and on Unix systems the library itself is
4176 called pcreposix.a, so can be accessed by adding -lpcreposix to the
4177 command for linking an application that uses them. Because the POSIX
4178 functions call the native ones, it is also necessary to add -lpcre.
4179
4180 I have implemented only those option bits that can be reasonably mapped
4181 to PCRE native options. In addition, the options REG_EXTENDED and
4182 REG_NOSUB are defined with the value zero. They have no effect, but
4183 since programs that are written to the POSIX interface often use them,
4184 this makes it easier to slot in PCRE as a replacement library. Other
4185 POSIX options are not even defined.
4186
4187 When PCRE is called via these functions, it is only the API that is
4188 POSIX-like in style. The syntax and semantics of the regular expres-
4189 sions themselves are still those of Perl, subject to the setting of
4190 various PCRE options, as described below. "POSIX-like in style" means
4191 that the API approximates to the POSIX definition; it is not fully
4192 POSIX-compatible, and in multi-byte encoding domains it is probably
4193 even less compatible.
4194
4195 The header for these functions is supplied as pcreposix.h to avoid any
4196 potential clash with other POSIX libraries. It can, of course, be
4197 renamed or aliased as regex.h, which is the "correct" name. It provides
4198 two structure types, regex_t for compiled internal forms, and reg-
4199 match_t for returning captured substrings. It also defines some con-
4200 stants whose names start with "REG_"; these are used for setting
4201 options and identifying error codes.
4202
4203
4204 COMPILING A PATTERN
4205
4206 The function regcomp() is called to compile a pattern into an internal
4207 form. The pattern is a C string terminated by a binary zero, and is
4208 passed in the argument pattern. The preg argument is a pointer to a
4209 regex_t structure that is used as a base for storing information about
4210 the compiled expression.
4211
4212 The argument cflags is either zero, or contains one or more of the bits
4213 defined by the following macros:
4214
4215 REG_DOTALL
4216
4217 The PCRE_DOTALL option is set when the expression is passed for compi-
4218 lation to the native function. Note that REG_DOTALL is not part of the
4219 POSIX standard.
4220
4221 REG_ICASE
4222
4223 The PCRE_CASELESS option is set when the expression is passed for com-
4224 pilation to the native function.
4225
4226 REG_NEWLINE
4227
4228 The PCRE_MULTILINE option is set when the expression is passed for com-
4229 pilation to the native function. Note that this does not mimic the
4230 defined POSIX behaviour for REG_NEWLINE (see the following section).
4231
4232 In the absence of these flags, no options are passed to the native
4233 function. This means the the regex is compiled with PCRE default
4234 semantics. In particular, the way it handles newline characters in the
4235 subject string is the Perl way, not the POSIX way. Note that setting
4236 PCRE_MULTILINE has only some of the effects specified for REG_NEWLINE.
4237 It does not affect the way newlines are matched by . (they aren't) or
4238 by a negative class such as [^a] (they are).
4239
4240 The yield of regcomp() is zero on success, and non-zero otherwise. The
4241 preg structure is filled in on success, and one member of the structure
4242 is public: re_nsub contains the number of capturing subpatterns in the
4243 regular expression. Various error codes are defined in the header file.
4244
4245
4246 MATCHING NEWLINE CHARACTERS
4247
4248 This area is not simple, because POSIX and Perl take different views of
4249 things. It is not possible to get PCRE to obey POSIX semantics, but
4250 then PCRE was never intended to be a POSIX engine. The following table
4251 lists the different possibilities for matching newline characters in
4252 PCRE:
4253
4254 Default Change with
4255
4256 . matches newline no PCRE_DOTALL
4257 newline matches [^a] yes not changeable
4258 $ matches \n at end yes PCRE_DOLLARENDONLY
4259 $ matches \n in middle no PCRE_MULTILINE
4260 ^ matches \n in middle no PCRE_MULTILINE
4261
4262 This is the equivalent table for POSIX:
4263
4264 Default Change with
4265
4266 . matches newline yes REG_NEWLINE
4267 newline matches [^a] yes REG_NEWLINE
4268 $ matches \n at end no REG_NEWLINE
4269 $ matches \n in middle no REG_NEWLINE
4270 ^ matches \n in middle no REG_NEWLINE
4271
4272 PCRE's behaviour is the same as Perl's, except that there is no equiva-
4273 lent for PCRE_DOLLAR_ENDONLY in Perl. In both PCRE and Perl, there is
4274 no way to stop newline from matching [^a].
4275
4276 The default POSIX newline handling can be obtained by setting
4277 PCRE_DOTALL and PCRE_DOLLAR_ENDONLY, but there is no way to make PCRE
4278 behave exactly as for the REG_NEWLINE action.
4279
4280
4281 MATCHING A PATTERN
4282
4283 The function regexec() is called to match a compiled pattern preg
4284 against a given string, which is terminated by a zero byte, subject to
4285 the options in eflags. These can be:
4286
4287 REG_NOTBOL
4288
4289 The PCRE_NOTBOL option is set when calling the underlying PCRE matching
4290 function.
4291
4292 REG_NOTEOL
4293
4294 The PCRE_NOTEOL option is set when calling the underlying PCRE matching
4295 function.
4296
4297 The portion of the string that was matched, and also any captured sub-
4298 strings, are returned via the pmatch argument, which points to an array
4299 of nmatch structures of type regmatch_t, containing the members rm_so
4300 and rm_eo. These contain the offset to the first character of each sub-
4301 string and the offset to the first character after the end of each sub-
4302 string, respectively. The 0th element of the vector relates to the
4303 entire portion of string that was matched; subsequent elements relate
4304 to the capturing subpatterns of the regular expression. Unused entries
4305 in the array have both structure members set to -1.
4306
4307 A successful match yields a zero return; various error codes are
4308 defined in the header file, of which REG_NOMATCH is the "expected"
4309 failure code.
4310
4311
4312 ERROR MESSAGES
4313
4314 The regerror() function maps a non-zero errorcode from either regcomp()
4315 or regexec() to a printable message. If preg is not NULL, the error
4316 should have arisen from the use of that structure. A message terminated
4317 by a binary zero is placed in errbuf. The length of the message,
4318 including the zero, is limited to errbuf_size. The yield of the func-
4319 tion is the size of buffer needed to hold the whole message.
4320
4321
4322 MEMORY USAGE
4323
4324 Compiling a regular expression causes memory to be allocated and asso-
4325 ciated with the preg structure. The function regfree() frees all such
4326 memory, after which preg may no longer be used as a compiled expres-
4327 sion.
4328
4329
4330 AUTHOR
4331
4332 Philip Hazel
4333 University Computing Service,
4334 Cambridge CB2 3QG, England.
4335
4336 Last updated: 28 February 2005
4337 Copyright (c) 1997-2005 University of Cambridge.
4338 ------------------------------------------------------------------------------
4339
4340
4341 PCRECPP(3) PCRECPP(3)
4342
4343
4344 NAME
4345 PCRE - Perl-compatible regular expressions.
4346
4347
4348 SYNOPSIS OF C++ WRAPPER
4349
4350 #include <pcrecpp.h>
4351
4352
4353 DESCRIPTION
4354
4355 The C++ wrapper for PCRE was provided by Google Inc. Some additional
4356 functionality was added by Giuseppe Maxia. This brief man page was con-
4357 structed from the notes in the pcrecpp.h file, which should be con-
4358 sulted for further details.
4359
4360
4361 MATCHING INTERFACE
4362
4363 The "FullMatch" operation checks that supplied text matches a supplied
4364 pattern exactly. If pointer arguments are supplied, it copies matched
4365 sub-strings that match sub-patterns into them.
4366
4367 Example: successful match
4368 pcrecpp::RE re("h.*o");
4369 re.FullMatch("hello");
4370
4371 Example: unsuccessful match (requires full match):
4372 pcrecpp::RE re("e");
4373 !re.FullMatch("hello");
4374
4375 Example: creating a temporary RE object:
4376 pcrecpp::RE("h.*o").FullMatch("hello");
4377
4378 You can pass in a "const char*" or a "string" for "text". The examples
4379 below tend to use a const char*. You can, as in the different examples
4380 above, store the RE object explicitly in a variable or use a temporary
4381 RE object. The examples below use one mode or the other arbitrarily.
4382 Either could correctly be used for any of these examples.
4383
4384 You must supply extra pointer arguments to extract matched subpieces.
4385
4386 Example: extracts "ruby" into "s" and 1234 into "i"
4387 int i;
4388 string s;
4389 pcrecpp::RE re("(\\w+):(\\d+)");
4390 re.FullMatch("ruby:1234", &s, &i);
4391
4392 Example: does not try to extract any extra sub-patterns
4393 re.FullMatch("ruby:1234", &s);
4394
4395 Example: does not try to extract into NULL
4396 re.FullMatch("ruby:1234", NULL, &i);
4397
4398 Example: integer overflow causes failure
4399 !re.FullMatch("ruby:1234567891234", NULL, &i);
4400
4401 Example: fails because there aren't enough sub-patterns:
4402 !pcrecpp::RE("\\w+:\\d+").FullMatch("ruby:1234", &s);
4403
4404 Example: fails because string cannot be stored in integer
4405 !pcrecpp::RE("(.*)").FullMatch("ruby", &i);
4406
4407 The provided pointer arguments can be pointers to any scalar numeric
4408 type, or one of:
4409
4410 string (matched piece is copied to string)
4411 StringPiece (StringPiece is mutated to point to matched piece)
4412 T (where "bool T::ParseFrom(const char*, int)" exists)
4413 NULL (the corresponding matched sub-pattern is not copied)
4414
4415 The function returns true iff all of the following conditions are sat-
4416 isfied:
4417
4418 a. "text" matches "pattern" exactly;
4419
4420 b. The number of matched sub-patterns is >= number of supplied
4421 pointers;
4422
4423 c. The "i"th argument has a suitable type for holding the
4424 string captured as the "i"th sub-pattern. If you pass in
4425 NULL for the "i"th argument, or pass fewer arguments than
4426 number of sub-patterns, "i"th captured sub-pattern is
4427 ignored.
4428
4429 The matching interface supports at most 16 arguments per call. If you
4430 need more, consider using the more general interface
4431 pcrecpp::RE::DoMatch. See pcrecpp.h for the signature for DoMatch.
4432
4433
4434 PARTIAL MATCHES
4435
4436 You can use the "PartialMatch" operation when you want the pattern to
4437 match any substring of the text.
4438
4439 Example: simple search for a string:
4440 pcrecpp::RE("ell").PartialMatch("hello");
4441
4442 Example: find first number in a string:
4443 int number;
4444 pcrecpp::RE re("(\\d+)");
4445 re.PartialMatch("x*100 + 20", &number);
4446 assert(number == 100);
4447
4448
4449 UTF-8 AND THE MATCHING INTERFACE
4450
4451 By default, pattern and text are plain text, one byte per character.
4452 The UTF8 flag, passed to the constructor, causes both pattern and
4453 string to be treated as UTF-8 text, still a byte stream but potentially
4454 multiple bytes per character. In practice, the text is likelier to be
4455 UTF-8 than the pattern, but the match returned may depend on the UTF8
4456 flag, so always use it when matching UTF8 text. For example, "." will
4457 match one byte normally but with UTF8 set may match up to three bytes
4458 of a multi-byte character.
4459
4460 Example:
4461 pcrecpp::RE_Options options;
4462 options.set_utf8();
4463 pcrecpp::RE re(utf8_pattern, options);
4464 re.FullMatch(utf8_string);
4465
4466 Example: using the convenience function UTF8():
4467 pcrecpp::RE re(utf8_pattern, pcrecpp::UTF8());
4468 re.FullMatch(utf8_string);
4469
4470 NOTE: The UTF8 flag is ignored if pcre was not configured with the
4471 --enable-utf8 flag.
4472
4473
4474 PASSING MODIFIERS TO THE REGULAR EXPRESSION ENGINE
4475
4476 PCRE defines some modifiers to change the behavior of the regular
4477 expression engine. The C++ wrapper defines an auxiliary class,
4478 RE_Options, as a vehicle to pass such modifiers to a RE class. Cur-
4479 rently, the following modifiers are supported:
4480
4481 modifier description Perl corresponding
4482
4483 PCRE_CASELESS case insensitive match /i
4484 PCRE_MULTILINE multiple lines match /m
4485 PCRE_DOTALL dot matches newlines /s
4486 PCRE_DOLLAR_ENDONLY $ matches only at end N/A
4487 PCRE_EXTRA strict escape parsing N/A
4488 PCRE_EXTENDED ignore whitespaces /x
4489 PCRE_UTF8 handles UTF8 chars built-in
4490 PCRE_UNGREEDY reverses * and *? N/A
4491 PCRE_NO_AUTO_CAPTURE disables capturing parens N/A (*)
4492
4493 (*) Both Perl and PCRE allow non capturing parentheses by means of the
4494 "?:" modifier within the pattern itself. e.g. (?:ab|cd) does not cap-
4495 ture, while (ab|cd) does.
4496
4497 For a full account on how each modifier works, please check the PCRE
4498 API reference page.
4499
4500 For each modifier, there are two member functions whose name is made
4501 out of the modifier in lowercase, without the "PCRE_" prefix. For
4502 instance, PCRE_CASELESS is handled by
4503
4504 bool caseless()
4505
4506 which returns true if the modifier is set, and
4507
4508 RE_Options & set_caseless(bool)
4509
4510 which sets or unsets the modifier. Moreover, PCRE_CONFIG_MATCH_LIMIT
4511 can be accessed through the set_match_limit() and match_limit() member
4512 functions. Setting match_limit to a non-zero value will limit the exe-
4513 cution of pcre to keep it from doing bad things like blowing the stack
4514 or taking an eternity to return a result. A value of 5000 is good
4515 enough to stop stack blowup in a 2MB thread stack. Setting match_limit
4516 to zero disables match limiting.
4517
4518 Normally, to pass one or more modifiers to a RE class, you declare a
4519 RE_Options object, set the appropriate options, and pass this object to
4520 a RE constructor. Example:
4521
4522 RE_options opt;
4523 opt.set_caseless(true);
4524 if (RE("HELLO", opt).PartialMatch("hello world")) ...
4525
4526 RE_options has two constructors. The default constructor takes no argu-
4527 ments and creates a set of flags that are off by default. The optional
4528 parameter option_flags is to facilitate transfer of legacy code from C
4529 programs. This lets you do
4530
4531 RE(pattern,
4532 RE_Options(PCRE_CASELESS|PCRE_MULTILINE)).PartialMatch(str);
4533
4534 However, new code is better off doing
4535
4536 RE(pattern,
4537 RE_Options().set_caseless(true).set_multiline(true))
4538 .PartialMatch(str);
4539
4540 If you are going to pass one of the most used modifiers, there are some
4541 convenience functions that return a RE_Options class with the appropri-
4542 ate modifier already set: CASELESS(), UTF8(), MULTILINE(), DOTALL(),
4543 and EXTENDED().
4544
4545 If you need to set several options at once, and you don't want to go
4546 through the pains of declaring a RE_Options object and setting several
4547 options, there is a parallel method that give you such ability on the
4548 fly. You can concatenate several set_xxxxx() member functions, since
4549 each of them returns a reference to its class object. For example, to
4550 pass PCRE_CASELESS, PCRE_EXTENDED, and PCRE_MULTILINE to a RE with one
4551 statement, you may write:
4552
4553 RE(" ^ xyz \\s+ .* blah$",
4554 RE_Options()
4555 .set_caseless(true)
4556 .set_extended(true)
4557 .set_multiline(true)).PartialMatch(sometext);
4558
4559
4560 SCANNING TEXT INCREMENTALLY
4561
4562 The "Consume" operation may be useful if you want to repeatedly match
4563 regular expressions at the front of a string and skip over them as they
4564 match. This requires use of the "StringPiece" type, which represents a
4565 sub-range of a real string. Like RE, StringPiece is defined in the
4566 pcrecpp namespace.
4567
4568 Example: read lines of the form "var = value" from a string.
4569 string contents = ...; // Fill string somehow
4570 pcrecpp::StringPiece input(contents); // Wrap in a StringPiece
4571
4572 string var;
4573 int value;
4574 pcrecpp::RE re("(\\w+) = (\\d+)\n");
4575 while (re.Consume(&input, &var, &value)) {
4576 ...;
4577 }
4578
4579 Each successful call to "Consume" will set "var/value", and also
4580 advance "input" so it points past the matched text.
4581
4582 The "FindAndConsume" operation is similar to "Consume" but does not
4583 anchor your match at the beginning of the string. For example, you
4584 could extract all words from a string by repeatedly calling
4585
4586 pcrecpp::RE("(\\w+)").FindAndConsume(&input, &word)
4587
4588
4589 PARSING HEX/OCTAL/C-RADIX NUMBERS
4590
4591 By default, if you pass a pointer to a numeric value, the corresponding
4592 text is interpreted as a base-10 number. You can instead wrap the
4593 pointer with a call to one of the operators Hex(), Octal(), or CRadix()
4594 to interpret the text in another base. The CRadix operator interprets
4595 C-style "0" (base-8) and "0x" (base-16) prefixes, but defaults to
4596 base-10.
4597
4598 Example:
4599 int a, b, c, d;
4600 pcrecpp::RE re("(.*) (.*) (.*) (.*)");
4601 re.FullMatch("100 40 0100 0x40",
4602 pcrecpp::Octal(&a), pcrecpp::Hex(&b),
4603 pcrecpp::CRadix(&c), pcrecpp::CRadix(&d));
4604
4605 will leave 64 in a, b, c, and d.
4606
4607
4608 REPLACING PARTS OF STRINGS
4609
4610 You can replace the first match of "pattern" in "str" with "rewrite".
4611 Within "rewrite", backslash-escaped digits (\1 to \9) can be used to
4612 insert text matching corresponding parenthesized group from the pat-
4613 tern. \0 in "rewrite" refers to the entire matching text. For example:
4614
4615 string s = "yabba dabba doo";
4616 pcrecpp::RE("b+").Replace("d", &s);
4617
4618 will leave "s" containing "yada dabba doo". The result is true if the
4619 pattern matches and a replacement occurs, false otherwise.
4620
4621 GlobalReplace is like Replace except that it replaces all occurrences
4622 of the pattern in the string with the rewrite. Replacements are not
4623 subject to re-matching. For example:
4624
4625 string s = "yabba dabba doo";
4626 pcrecpp::RE("b+").GlobalReplace("d", &s);
4627
4628 will leave "s" containing "yada dada doo". It returns the number of
4629 replacements made.
4630
4631 Extract is like Replace, except that if the pattern matches, "rewrite"
4632 is copied into "out" (an additional argument) with substitutions. The
4633 non-matching portions of "text" are ignored. Returns true iff a match
4634 occurred and the extraction happened successfully; if no match occurs,
4635 the string is left unaffected.
4636
4637
4638 AUTHOR
4639
4640 The C++ wrapper was contributed by Google Inc.
4641 Copyright (c) 2005 Google Inc.
4642 ------------------------------------------------------------------------------
4643
4644
4645 PCRESAMPLE(3) PCRESAMPLE(3)
4646
4647
4648 NAME
4649 PCRE - Perl-compatible regular expressions
4650
4651
4652 PCRE SAMPLE PROGRAM
4653
4654 A simple, complete demonstration program, to get you started with using
4655 PCRE, is supplied in the file pcredemo.c in the PCRE distribution.
4656
4657 The program compiles the regular expression that is its first argument,
4658 and matches it against the subject string in its second argument. No
4659 PCRE options are set, and default character tables are used. If match-
4660 ing succeeds, the program outputs the portion of the subject that
4661 matched, together with the contents of any captured substrings.
4662
4663 If the -g option is given on the command line, the program then goes on
4664 to check for further matches of the same regular expression in the same
4665 subject string. The logic is a little bit tricky because of the possi-
4666 bility of matching an empty string. Comments in the code explain what
4667 is going on.
4668
4669 If PCRE is installed in the standard include and library directories
4670 for your system, you should be able to compile the demonstration pro-
4671 gram using this command:
4672
4673 gcc -o pcredemo pcredemo.c -lpcre
4674
4675 If PCRE is installed elsewhere, you may need to add additional options
4676 to the command line. For example, on a Unix-like system that has PCRE
4677 installed in /usr/local, you can compile the demonstration program
4678 using a command like this:
4679
4680 gcc -o pcredemo -I/usr/local/include pcredemo.c \
4681 -L/usr/local/lib -lpcre
4682
4683 Once you have compiled the demonstration program, you can run simple
4684 tests like this:
4685
4686 ./pcredemo 'cat|dog' 'the cat sat on the mat'
4687 ./pcredemo -g 'cat|dog' 'the dog sat on the cat'
4688
4689 Note that there is a much more comprehensive test program, called
4690 pcretest, which supports many more facilities for testing regular
4691 expressions and the PCRE library. The pcredemo program is provided as a
4692 simple coding example.
4693
4694 On some operating systems (e.g. Solaris), when PCRE is not installed in
4695 the standard library directory, you may get an error like this when you
4696 try to run pcredemo:
4697
4698 ld.so.1: a.out: fatal: libpcre.so.0: open failed: No such file or
4699 directory
4700
4701 This is caused by the way shared library support works on those sys-
4702 tems. You need to add
4703
4704 -R/usr/local/lib
4705
4706 (for example) to the compile command to get round this problem.
4707
4708 Last updated: 09 September 2004
4709 Copyright (c) 1997-2004 University of Cambridge.
4710 ------------------------------------------------------------------------------

  ViewVC Help
Powered by ViewVC 1.1.5