/[pcre]/code/trunk/doc/pcretest.txt
ViewVC logotype

Diff of /code/trunk/doc/pcretest.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 49 by nigel, Sat Feb 24 21:39:33 2007 UTC revision 71 by nigel, Sat Feb 24 21:40:24 2007 UTC
# Line 1  Line 1 
1  The pcretest program  NAME
2  --------------------       pcretest - a program  for  testing  Perl-compatible  regular
3         expressions.
4  This program is intended for testing PCRE, but it can also be used for  
5  experimenting with regular expressions.  
6    SYNOPSIS
7  If it is given two filename arguments, it reads from the first and writes to       pcretest [-d] [-i] [-m] [-o osize] [-p] [-t] [source]  [des-
8  the second. If it is given only one filename argument, it reads from that file       tination]
9  and writes to stdout. Otherwise, it reads from stdin and writes to stdout, and  
10  prompts for each line of input, using "re>" to prompt for regular expressions,       pcretest was written as a test program for the PCRE  regular
11  and "data>" to prompt for data lines.       expression  library  itself,  but  it  can  also be used for
12         experimenting  with  regular  expressions.   This   document
13  The program handles any number of sets of input on a single input file. Each       describes  the  features of the test program; for details of
14  set starts with a regular expression, and continues with any number of data       the regular  expressions  themselves,  see  the  pcrepattern
15  lines to be matched against the pattern. An empty line signals the end of the       documentation.  For details of PCRE and its options, see the
16  data lines, at which point a new regular expression is read. The regular       pcreapi documentation.
17  expressions are given enclosed in any non-alphameric delimiters other than  
18  backslash, for example  
19    OPTIONS
20    /(a|bc)x+yz/  
21    
22  White space before the initial delimiter is ignored. A regular expression may       -C        Output the version number of the PCRE library, and
23  be continued over several input lines, in which case the newline characters are                 all   available  information  about  the  optional
24  included within it. See the test input files in the testdata directory for many                 features that are included, and then exit.
25  examples. It is possible to include the delimiter within the pattern by  
26  escaping it, for example       -d        Behave as if each regex had the /D  modifier  (see
27                   below); the internal form is output after compila-
28    /abc\/def/                 tion.
29    
30  If you do so, the escape and the delimiter form part of the pattern, but since       -i        Behave as if  each  regex  had  the  /I  modifier;
31  delimiters are always non-alphameric, this does not affect its interpretation.                 information  about  the  compiled pattern is given
32  If the terminating delimiter is immediately followed by a backslash, for                 after compilation.
33  example,  
34         -m        Output the size of each compiled pattern after  it
35    /abc/\                 has been compiled. This is equivalent to adding /M
36                   to each regular expression. For compatibility with
37  then a backslash is added to the end of the pattern. This is done to provide a                 earlier  versions of pcretest, -s is a synonym for
38  way of testing the error condition that arises if a pattern finishes with a                 -m.
39  backslash, because  
40         -o osize  Set the number of elements in  the  output  vector
41    /abc\/                 that  is  used  when calling PCRE to be osize. The
42                   default value is 45, which is enough for  14  cap-
43  is interpreted as the first line of a pattern that starts with "abc/", causing                 turing  subexpressions.  The  vector  size  can be
44  pcretest to read the next line as a continuation of the regular expression.                 changed for individual matching calls by including
45                   \O in the data line (see below).
46    
47         -p        Behave as if each regex has /P modifier; the POSIX
48                   wrapper  API  is  used  to  call PCRE. None of the
49                   other options has any effect when -p is set.
50    
51         -t        Run each compile, study, and match many times with
52                   a  timer, and output resulting time per compile or
53                   match (in milliseconds). Do not set  -t  with  -m,
54                   because  you  will  then get the size output 20000
55                   times and the timing will be distorted.
56    
57    
58    DESCRIPTION
59    
60         If pcretest is given two filename arguments, it  reads  from
61         the  first and writes to the second. If it is given only one
62         filename argument, it reads from that  file  and  writes  to
63         stdout. Otherwise, it reads from stdin and writes to stdout,
64         and prompts for each line of input, using  "re>"  to  prompt
65         for  regular  expressions,  and  "data>"  to prompt for data
66         lines.
67    
68         The program handles any number of sets of input on a  single
69         input  file.  Each set starts with a regular expression, and
70         continues with any  number  of  data  lines  to  be  matched
71         against the pattern.
72    
73         Each line is matched separately and  independently.  If  you
74         want  to  do  multiple-line  matches, you have to use the \n
75         escape sequence in a single line of input to encode the new-
76         line  characters.  The maximum length of data line is 30,000
77         characters.
78    
79         An empty line signals the end of the data  lines,  at  which
80         point  a new regular expression is read. The regular expres-
81         sions are given enclosed in  any  non-alphameric  delimiters
82         other than backslash, for example
83    
84           /(a|bc)x+yz/
85    
86         White space before the initial delimiter is ignored. A regu-
87         lar expression may be continued over several input lines, in
88         which case the newline characters are included within it. It
89         is  possible  to include the delimiter within the pattern by
90         escaping it, for example
91    
92           /abc\/def/
93    
94         If you do so, the escape and the delimiter form part of  the
95         pattern,  but  since  delimiters  are always non-alphameric,
96         this does not affect its interpretation.  If the terminating
97         delimiter  is immediately followed by a backslash, for exam-
98         ple,
99    
100           /abc/\
101    
102         then a backslash is added to the end of the pattern. This is
103         done  to  provide  a way of testing the error condition that
104         arises if a pattern finishes with a backslash, because
105    
106           /abc\/
107    
108         is interpreted as the first line of a  pattern  that  starts
109         with  "abc/",  causing  pcretest  to read the next line as a
110         continuation of the regular expression.
111    
112    
113  PATTERN MODIFIERS  PATTERN MODIFIERS
 -----------------  
114    
115  The pattern may be followed by i, m, s, or x to set the PCRE_CASELESS,       The pattern may be followed by i, m, s,  or  x  to  set  the
116  PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED options, respectively. For       PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED
117  example:       options, respectively. For example:
118    
119    /caseless/i         /caseless/i
120    
121  These modifier letters have the same effect as they do in Perl. There are       These modifier letters have the same effect as  they  do  in
122  others which set PCRE options that do not correspond to anything in Perl: /A,       Perl.  There  are  others  that set PCRE options that do not
123  /E, and /X set PCRE_ANCHORED, PCRE_DOLLAR_ENDONLY, and PCRE_EXTRA respectively.       correspond to anything in Perl:  /A, /E, /N, /U, and /X  set
124         PCRE_ANCHORED,   PCRE_DOLLAR_ENDONLY,  PCRE_NO_AUTO_CAPTURE,
125  Searching for all possible matches within each subject string can be requested       PCRE_UNGREEDY, and PCRE_EXTRA respectively.
126  by the /g or /G modifier. After finding a match, PCRE is called again to search  
127  the remainder of the subject string. The difference between /g and /G is that       Searching for  all  possible  matches  within  each  subject
128  the former uses the startoffset argument to pcre_exec() to start searching at       string  can  be  requested  by  the /g or /G modifier. After
129  a new point within the entire string (which is in effect what Perl does),       finding  a  match,  PCRE  is  called  again  to  search  the
130  whereas the latter passes over a shortened substring. This makes a difference       remainder  of  the subject string. The difference between /g
131  to the matching process if the pattern begins with a lookbehind assertion       and /G is that the former uses the startoffset  argument  to
132  (including \b or \B).       pcre_exec()  to  start  searching  at a new point within the
133         entire string (which is in effect what Perl  does),  whereas
134  If any call to pcre_exec() in a /g or /G sequence matches an empty string, the       the  latter  passes over a shortened substring. This makes a
135  next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED flags set in order       difference to the matching process  if  the  pattern  begins
136  to search for another, non-empty, match at the same point. If this second match       with a lookbehind assertion (including \b or \B).
137  fails, the start offset is advanced by one, and the normal match is retried.  
138  This imitates the way Perl handles such cases when using the /g modifier or the       If any call to pcre_exec() in a /g or /G sequence matches an
139  split() function.       empty  string,  the next call is done with the PCRE_NOTEMPTY
140         and PCRE_ANCHORED flags set in order to search for  another,
141  There are a number of other modifiers for controlling the way pcretest       non-empty,  match  at  the same point.  If this second match
142  operates.       fails, the start offset is advanced by one, and  the  normal
143         match  is  retried.  This imitates the way Perl handles such
144  The /+ modifier requests that as well as outputting the substring that matched       cases when using the /g modifier or the split() function.
145  the entire pattern, pcretest should in addition output the remainder of the  
146  subject string. This is useful for tests where the subject contains multiple       There are a number of other modifiers  for  controlling  the
147  copies of the same substring.       way pcretest operates.
148    
149  The /L modifier must be followed directly by the name of a locale, for example,       The /+ modifier requests that as well as outputting the sub-
150         string  that  matched the entire pattern, pcretest should in
151    /pattern/Lfr       addition output the remainder of the subject string. This is
152         useful  for tests where the subject contains multiple copies
153  For this reason, it must be the last modifier letter. The given locale is set,       of the same substring.
154  pcre_maketables() is called to build a set of character tables for the locale,  
155  and this is then passed to pcre_compile() when compiling the regular       The /L modifier must be followed directly by the name  of  a
156  expression. Without an /L modifier, NULL is passed as the tables pointer; that       locale, for example,
157  is, /L applies only to the expression on which it appears.  
158           /pattern/Lfr
159  The /I modifier requests that pcretest output information about the compiled  
160  expression (whether it is anchored, has a fixed first character, and so on). It       For this reason, it must be the last  modifier  letter.  The
161  does this by calling pcre_fullinfo() after compiling an expression, and       given  locale is set, pcre_maketables() is called to build a
162  outputting the information it gets back. If the pattern is studied, the results       set of character tables for the locale,  and  this  is  then
163  of that are also output.       passed  to pcre_compile() when compiling the regular expres-
164         sion. Without an /L modifier, NULL is passed as  the  tables
165  The /D modifier is a PCRE debugging feature, which also assumes /I. It causes       pointer; that is, /L applies only to the expression on which
166  the internal form of compiled regular expressions to be output after       it appears.
167  compilation.  
168         The /I modifier requests that  pcretest  output  information
169  The /S modifier causes pcre_study() to be called after the expression has been       about the compiled expression (whether it is anchored, has a
170  compiled, and the results used when the expression is matched.       fixed first character, and so on). It does this  by  calling
171         pcre_fullinfo()  after  compiling an expression, and output-
172  The /M modifier causes the size of memory block used to hold the compiled       ting the information it gets back. If the  pattern  is  stu-
173  pattern to be output.       died, the results of that are also output.
174    
175  The /P modifier causes pcretest to call PCRE via the POSIX wrapper API rather       The /D modifier is a  PCRE  debugging  feature,  which  also
176  than its native API. When this is done, all other modifiers except /i, /m, and       assumes /I.  It causes the internal form of compiled regular
177  /+ are ignored. REG_ICASE is set if /i is present, and REG_NEWLINE is set if /m       expressions to be output after compilation. If  the  pattern
178  is present. The wrapper functions force PCRE_DOLLAR_ENDONLY always, and       was studied, the information returned is also output.
179  PCRE_DOTALL unless REG_NEWLINE is set.  
180         The /S modifier causes pcre_study() to be called  after  the
181  The /8 modifier causes pcretest to call PCRE with the PCRE_UTF8 option set.       expression  has been compiled, and the results used when the
182  This turns on the (currently incomplete) support for UTF-8 character handling       expression is matched.
183  in PCRE, provided that it was compiled with this support enabled. This modifier  
184  also causes any non-printing characters in output strings to be printed using       The /M modifier causes the size of memory block used to hold
185  the \x{hh...} notation if they are valid UTF-8 sequences.       the compiled pattern to be output.
186    
187         The /P modifier causes pcretest to call PCRE via  the  POSIX
188         wrapper  API  rather than its native API. When this is done,
189         all other modifiers except  /i,  /m,  and  /+  are  ignored.
190         REG_ICASE is set if /i is present, and REG_NEWLINE is set if
191         /m    is    present.    The    wrapper    functions    force
192         PCRE_DOLLAR_ENDONLY    always,    and   PCRE_DOTALL   unless
193         REG_NEWLINE is set.
194    
195         The /8 modifier  causes  pcretest  to  call  PCRE  with  the
196         PCRE_UTF8  option set. This turns on support for UTF-8 char-
197         acter handling in PCRE, provided that it was  compiled  with
198         this  support  enabled.  This  modifier also causes any non-
199         printing characters in output strings to  be  printed  using
200         the \x{hh...} notation if they are valid UTF-8 sequences.
201    
202         If the /? modifier is used with /8, it  causes  pcretest  to
203         call  pcre_compile()  with the PCRE_NO_UTF8_CHECK option, to
204         suppress the checking of the string for UTF-8 validity.
205    
206    
207    CALLOUTS
208    
209         If the pattern contains  any  callout  requests,  pcretest's
210         callout function will be called. By default, it displays the
211         callout number, and the start and current positions  in  the
212         text at the callout time. For example, the output
213    
214           --->pqrabcdef
215             0    ^  ^
216    
217         indicates that callout number 0 occurred for a match attempt
218         starting at the fourth character of the subject string, when
219         the pointer was at the seventh character. The callout  func-
220         tion returns zero (carry on matching) by default.
221    
222         Inserting callouts may be helpful  when  using  pcretest  to
223         check  complicated regular expressions. For further informa-
224         tion about callouts, see the pcrecallout documentation.
225    
226         For testing the PCRE library, additional control of  callout
227         behaviour  is available via escape sequences in the data, as
228         described in the following section.  In  particular,  it  is
229         possible to pass in a number as callout data (the default is
230         zero). If the callout function receives a  non-zero  number,
231         it returns that value instead of zero.
232    
233    
234  DATA LINES  DATA LINES
 ----------  
   
 Before each data line is passed to pcre_exec(), leading and trailing whitespace  
 is removed, and it is then scanned for \ escapes. The following are recognized:  
235    
236    \a         alarm (= BEL)       Before each data line is passed to pcre_exec(), leading  and
237    \b         backspace       trailing whitespace is removed, and it is then scanned for \
238    \e         escape       escapes.  Some  of  these  are  pretty  esoteric   features,
239    \f         formfeed       intended  for  checking  out  some  of  the more complicated
240    \n         newline       features of PCRE. If you are just testing "ordinary" regular
241    \r         carriage return       expressions,  you probably don't need any of these. The fol-
242    \t         tab       lowing escapes are recognized:
243    \v         vertical tab  
244    \nnn       octal character (up to 3 octal digits)         \a         alarm (= BEL)
245    \xhh       hexadecimal character (up to 2 hex digits)         \b         backspace
246    \x{hh...}  hexadecimal UTF-8 character         \e         escape
247           \f         formfeed
248    \A         pass the PCRE_ANCHORED option to pcre_exec()         \n         newline
249    \B         pass the PCRE_NOTBOL option to pcre_exec()         \r         carriage return
250    \Cdd       call pcre_copy_substring() for substring dd after a successful         \t         tab
251                 match (any decimal number less than 32)         \v         vertical tab
252    \Gdd       call pcre_get_substring() for substring dd after a successful         \nnn       octal character (up to 3 octal digits)
253                 match (any decimal number less than 32)         \xhh       hexadecimal character (up to 2 hex digits)
254    \L         call pcre_get_substringlist() after a successful match         \x{hh...}  hexadecimal character, any number of digits
255    \N         pass the PCRE_NOTEMPTY option to pcre_exec()                      in UTF-8 mode
256    \Odd       set the size of the output vector passed to pcre_exec() to dd         \A         pass the PCRE_ANCHORED option to pcre_exec()
257                 (any number of decimal digits)         \B         pass the PCRE_NOTBOL option to pcre_exec()
258    \Z         pass the PCRE_NOTEOL option to pcre_exec()         \Cdd       call pcre_copy_substring() for substring dd
259                        after a successful match (any decimal number
260  A backslash followed by anything else just escapes the anything else. If the                      less than 32)
261  very last character is a backslash, it is ignored. This gives a way of passing         \Cname     call pcre_copy_named_substring() for substring
262  an empty line as data, since a real empty line terminates the data input.  
263                        "name" after a successful match (name termin-
264  If /P was present on the regex, causing the POSIX wrapper API to be used, only                      ated by next non alphanumeric character)
265  \B, and \Z have any effect, causing REG_NOTBOL and REG_NOTEOL to be passed to         \C+        show the current captured substrings at callout
266  regexec() respectively.                      time
267           \C-        do not supply a callout function
268  The use of \x{hh...} to represent UTF-8 characters is not dependent on the use         \C!n       return 1 instead of 0 when callout number n is
269  of the /8 modifier on the pattern. It is recognized always. There may be any                      reached
270  number of hexadecimal digits inside the braces. The result is from one to six         \C!n!m     return 1 instead of 0 when callout number n is
271  bytes, encoded according to the UTF-8 rules.                      reached for the nth time
272           \C*n       pass the number n (may be negative) as callout
273                        data
274           \Gdd       call pcre_get_substring() for substring dd
275                        after a successful match (any decimal number
276                        less than 32)
277           \Gname     call pcre_get_named_substring() for substring
278                        "name" after a successful match (name termin-
279                        ated by next non-alphanumeric character)
280           \L         call pcre_get_substringlist() after a
281                        successful match
282           \M         discover the minimum MATCH_LIMIT setting
283           \N         pass the PCRE_NOTEMPTY option to pcre_exec()
284           \Odd       set the size of the output vector passed to
285                        pcre_exec() to dd (any number of decimal
286                        digits)
287           \Z         pass the PCRE_NOTEOL option to pcre_exec()
288           \?         pass the PCRE_NO_UTF8_CHECK option to
289                        pcre_exec()
290    
291         If \M is present, pcretest calls pcre_exec() several  times,
292         with  different  values  in  the  match_limit  field  of the
293         pcre_extra data structure, until it finds the minimum number
294         that is needed for pcre_exec() to complete. This number is a
295         measure of the amount of  recursion  and  backtracking  that
296         takes  place,  and  checking  it out can be instructive. For
297         most simple matches, the number is quite small, but for pat-
298         terns  with very large numbers of matching possibilities, it
299         can become large very quickly with increasing length of sub-
300         ject string.
301    
302         When \O is used, it may be higher or lower than the size set
303         by  the  -O  option (or defaulted to 45); \O applies only to
304         the call of pcre_exec() for the line in which it appears.
305    
306         A backslash followed by anything else just escapes the  any-
307         thing else. If the very last character is a backslash, it is
308         ignored. This gives a way of passing an empty line as  data,
309         since a real empty line terminates the data input.
310    
311         If /P was present on the regex, causing  the  POSIX  wrapper
312         API  to  be  used,  only  B,  and Z have any effect, causing
313         REG_NOTBOL and REG_NOTEOL to be passed to regexec()  respec-
314         tively.
315         The use of \x{hh...} to represent UTF-8  characters  is  not
316         dependent  on  the use of the /8 modifier on the pattern. It
317         is recognized always. There may be any number of hexadecimal
318         digits  inside  the  braces.  The  result is from one to six
319         bytes, encoded according to the UTF-8 rules.
320    
321    
322  OUTPUT FROM PCRETEST  OUTPUT FROM PCRETEST
 --------------------  
323    
324  When a match succeeds, pcretest outputs the list of captured substrings that       When a match succeeds, pcretest outputs the list of captured
325  pcre_exec() returns, starting with number 0 for the string that matched the       substrings  that pcre_exec() returns, starting with number 0
326  whole pattern. Here is an example of an interactive pcretest run.       for the string that matched the whole pattern.  Here  is  an
327         example of an interactive pcretest run.
328    $ pcretest  
329    PCRE version 2.06 08-Jun-1999         $ pcretest
330           PCRE version 4.00 08-Jan-2003
331      re> /^abc(\d+)/  
332    data> abc123           re> /^abc(\d+)/
333     0: abc123         data> abc123
334     1: 123          0: abc123
335    data> xyz          1: 123
336    No match         data> xyz
337           No match
338  If the strings contain any non-printing characters, they are output as \0x  
339  escapes, or as \x{...} escapes if the /8 modifier was present on the pattern.       If the strings contain any non-printing characters, they are
340  If the pattern has the /+ modifier, then the output for substring 0 is followed       output  as  \0x  escapes,  or  as  \x{...} escapes if the /8
341  by the the rest of the subject string, identified by "0+" like this:       modifier was present on the pattern. If the pattern has  the
342         /+  modifier, then the output for substring 0 is followed by
343      re> /cat/+       the the rest of the subject string, identified by "0+"  like
344    data> cataract       this:
345     0: cat  
346     0+ aract           re> /cat/+
347           data> cataract
348  If the pattern has the /g or /G modifier, the results of successive matching          0: cat
349  attempts are output in sequence, like this:          0+ aract
350    
351      re> /\Bi(\w\w)/g       If the pattern has the /g or /G  modifier,  the  results  of
352    data> Mississippi       successive  matching  attempts  are output in sequence, like
353     0: iss       this:
354     1: ss  
355     0: iss           re> /\Bi(\w\w)/g
356     1: ss         data> Mississippi
357     0: ipp          0: iss
358     1: pp          1: ss
359            0: iss
360  "No match" is output only if the first match attempt fails.          1: ss
361            0: ipp
362  If any of \C, \G, or \L are present in a data line that is successfully          1: pp
363  matched, the substrings extracted by the convenience functions are output with  
364  C, G, or L after the string number instead of a colon. This is in addition to       "No match" is output only if the first match attempt fails.
365  the normal full list. The string length (that is, the return from the  
366  extraction function) is given in parentheses after each string for \C and \G.       If any of the sequences \C, \G, or \L are present in a  data
367         line  that is successfully matched, the substrings extracted
368  Note that while patterns can be continued over several lines (a plain ">"       by the convenience functions are output  with  C,  G,  or  L
369  prompt is used for continuations), data lines may not. However newlines can be       after the string number instead of a colon. This is in addi-
370  included in data by means of the \n escape.       tion to the normal full list. The string  length  (that  is,
371         the  return  from  the  extraction  function)  is  given  in
372         parentheses after each string for \C and \G.
373  COMMAND LINE OPTIONS  
374  --------------------       Note that while patterns can be continued over several lines
375         (a  plain  ">" prompt is used for continuations), data lines
376  If the -p option is given to pcretest, it is equivalent to adding /P to each       may not. However newlines can be included in data  by  means
377  regular expression: the POSIX wrapper API is used to call PCRE. None of the       of the \n escape.
378  following flags has any effect in this case.  
379    
380  If the option -d is given to pcretest, it is equivalent to adding /D to each  AUTHOR
381  regular expression: the internal form is output after compilation.  
382         Philip Hazel <ph10@cam.ac.uk>
383  If the option -i is given to pcretest, it is equivalent to adding /I to each       University Computing Service,
384  regular expression: information about the compiled pattern is given after       Cambridge CB2 3QG, England.
 compilation.  
   
 If the option -m is given to pcretest, it outputs the size of each compiled  
 pattern after it has been compiled. It is equivalent to adding /M to each  
 regular expression. For compatibility with earlier versions of pcretest, -s is  
 a synonym for -m.  
   
 If the -t option is given, each compile, study, and match is run 20000 times  
 while being timed, and the resulting time per compile or match is output in  
 milliseconds. Do not set -t with -m, because you will then get the size output  
 20000 times and the timing will be distorted. If you want to change the number  
 of repetitions used for timing, edit the definition of LOOPREPEAT at the top of  
 pcretest.c  
385    
386  Philip Hazel <ph10@cam.ac.uk>  Last updated: 20 August 2003
387  August 2000  Copyright (c) 1997-2003 University of Cambridge.

Legend:
Removed from v.49  
changed lines
  Added in v.71

  ViewVC Help
Powered by ViewVC 1.1.5