ViewVC logotype

Contents of /code/trunk/doc/pcretest.1

Parent Directory Parent Directory | Revision Log Revision Log

Revision 77 - (show annotations)
Sat Feb 24 21:40:45 2007 UTC (14 years, 2 months ago) by nigel
File size: 22713 byte(s)
Error occurred while calculating annotation data.
Load pcre-6.0 into code/trunk.
3 pcretest - a program for testing Perl-compatible regular expressions.
5 .rs
6 .sp
7 .B pcretest "[-C] [-d] [-dfa] [-i] [-m] [-o osize] [-p] [-t] [source]"
8 .ti +5n
9 .B "[destination]"
10 .P
11 \fBpcretest\fP was written as a test program for the PCRE regular expression
12 library itself, but it can also be used for experimenting with regular
13 expressions. This document describes the features of the test program; for
14 details of the regular expressions themselves, see the
15 .\" HREF
16 \fBpcrepattern\fP
17 .\"
18 documentation. For details of the PCRE library function calls and their
19 options, see the
20 .\" HREF
21 \fBpcreapi\fP
22 .\"
23 documentation.
24 .
25 .
27 .rs
28 .TP 10
29 \fB-C\fP
30 Output the version number of the PCRE library, and all available information
31 about the optional features that are included, and then exit.
32 .TP 10
33 \fB-d\fP
34 Behave as if each regex has the \fB/D\fP (debug) modifier; the internal
35 form is output after compilation.
36 .TP 10
37 \fB-dfa\fP
38 Behave as if each data line contains the \eD escape sequence; this causes the
39 alternative matching function, \fBpcre_dfa_exec()\fP, to be used instead of the
40 standard \fBpcre_exec()\fP function (more detail is given below).
41 .TP 10
42 \fB-i\fP
43 Behave as if each regex has the \fB/I\fP modifier; information about the
44 compiled pattern is given after compilation.
45 .TP 10
46 \fB-m\fP
47 Output the size of each compiled pattern after it has been compiled. This is
48 equivalent to adding \fB/M\fP to each regular expression. For compatibility
49 with earlier versions of pcretest, \fB-s\fP is a synonym for \fB-m\fP.
50 .TP 10
51 \fB-o\fP \fIosize\fP
52 Set the number of elements in the output vector that is used when calling
53 \fBpcre_exec()\fP to be \fIosize\fP. The default value is 45, which is enough
54 for 14 capturing subexpressions. The vector size can be changed for individual
55 matching calls by including \eO in the data line (see below).
56 .TP 10
57 \fB-p\fP
58 Behave as if each regex has the \fB/P\fP modifier; the POSIX wrapper API is
59 used to call PCRE. None of the other options has any effect when \fB-p\fP is
60 set.
61 .TP 10
62 \fB-t\fP
63 Run each compile, study, and match many times with a timer, and output
64 resulting time per compile or match (in milliseconds). Do not set \fB-m\fP with
65 \fB-t\fP, because you will then get the size output a zillion times, and the
66 timing will be distorted.
67 .
68 .
70 .rs
71 .sp
72 If \fBpcretest\fP is given two filename arguments, it reads from the first and
73 writes to the second. If it is given only one filename argument, it reads from
74 that file and writes to stdout. Otherwise, it reads from stdin and writes to
75 stdout, and prompts for each line of input, using "re>" to prompt for regular
76 expressions, and "data>" to prompt for data lines.
77 .P
78 The program handles any number of sets of input on a single input file. Each
79 set starts with a regular expression, and continues with any number of data
80 lines to be matched against the pattern.
81 .P
82 Each data line is matched separately and independently. If you want to do
83 multiple-line matches, you have to use the \en escape sequence in a single line
84 of input to encode the newline characters. The maximum length of data line is
85 30,000 characters.
86 .P
87 An empty line signals the end of the data lines, at which point a new regular
88 expression is read. The regular expressions are given enclosed in any
89 non-alphanumeric delimiters other than backslash, for example
90 .sp
91 /(a|bc)x+yz/
92 .sp
93 White space before the initial delimiter is ignored. A regular expression may
94 be continued over several input lines, in which case the newline characters are
95 included within it. It is possible to include the delimiter within the pattern
96 by escaping it, for example
97 .sp
98 /abc\e/def/
99 .sp
100 If you do so, the escape and the delimiter form part of the pattern, but since
101 delimiters are always non-alphanumeric, this does not affect its interpretation.
102 If the terminating delimiter is immediately followed by a backslash, for
103 example,
104 .sp
105 /abc/\e
106 .sp
107 then a backslash is added to the end of the pattern. This is done to provide a
108 way of testing the error condition that arises if a pattern finishes with a
109 backslash, because
110 .sp
111 /abc\e/
112 .sp
113 is interpreted as the first line of a pattern that starts with "abc/", causing
114 pcretest to read the next line as a continuation of the regular expression.
115 .
116 .
118 .rs
119 .sp
120 A pattern may be followed by any number of modifiers, which are mostly single
121 characters. Following Perl usage, these are referred to below as, for example,
122 "the \fB/i\fP modifier", even though the delimiter of the pattern need not
123 always be a slash, and no slash is used when writing modifiers. Whitespace may
124 appear between the final pattern delimiter and the first modifier, and between
125 the modifiers themselves.
126 .P
127 The \fB/i\fP, \fB/m\fP, \fB/s\fP, and \fB/x\fP modifiers set the PCRE_CASELESS,
128 PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when
129 \fBpcre_compile()\fP is called. These four modifier letters have the same
130 effect as they do in Perl. For example:
131 .sp
132 /caseless/i
133 .sp
134 The following table shows additional modifiers for setting PCRE options that do
135 not correspond to anything in Perl:
136 .sp
144 .sp
145 Searching for all possible matches within each subject string can be requested
146 by the \fB/g\fP or \fB/G\fP modifier. After finding a match, PCRE is called
147 again to search the remainder of the subject string. The difference between
148 \fB/g\fP and \fB/G\fP is that the former uses the \fIstartoffset\fP argument to
149 \fBpcre_exec()\fP to start searching at a new point within the entire string
150 (which is in effect what Perl does), whereas the latter passes over a shortened
151 substring. This makes a difference to the matching process if the pattern
152 begins with a lookbehind assertion (including \eb or \eB).
153 .P
154 If any call to \fBpcre_exec()\fP in a \fB/g\fP or \fB/G\fP sequence matches an
155 empty string, the next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED
156 flags set in order to search for another, non-empty, match at the same point.
157 If this second match fails, the start offset is advanced by one, and the normal
158 match is retried. This imitates the way Perl handles such cases when using the
159 \fB/g\fP modifier or the \fBsplit()\fP function.
160 .P
161 There are yet more modifiers for controlling the way \fBpcretest\fP
162 operates.
163 .P
164 The \fB/+\fP modifier requests that as well as outputting the substring that
165 matched the entire pattern, pcretest should in addition output the remainder of
166 the subject string. This is useful for tests where the subject contains
167 multiple copies of the same substring.
168 .P
169 The \fB/L\fP modifier must be followed directly by the name of a locale, for
170 example,
171 .sp
172 /pattern/Lfr_FR
173 .sp
174 For this reason, it must be the last modifier. The given locale is set,
175 \fBpcre_maketables()\fP is called to build a set of character tables for the
176 locale, and this is then passed to \fBpcre_compile()\fP when compiling the
177 regular expression. Without an \fB/L\fP modifier, NULL is passed as the tables
178 pointer; that is, \fB/L\fP applies only to the expression on which it appears.
179 .P
180 The \fB/I\fP modifier requests that \fBpcretest\fP output information about the
181 compiled pattern (whether it is anchored, has a fixed first character, and
182 so on). It does this by calling \fBpcre_fullinfo()\fP after compiling a
183 pattern. If the pattern is studied, the results of that are also output.
184 .P
185 The \fB/D\fP modifier is a PCRE debugging feature, which also assumes \fB/I\fP.
186 It causes the internal form of compiled regular expressions to be output after
187 compilation. If the pattern was studied, the information returned is also
188 output.
189 .P
190 The \fB/F\fP modifier causes \fBpcretest\fP to flip the byte order of the
191 fields in the compiled pattern that contain 2-byte and 4-byte numbers. This
192 facility is for testing the feature in PCRE that allows it to execute patterns
193 that were compiled on a host with a different endianness. This feature is not
194 available when the POSIX interface to PCRE is being used, that is, when the
195 \fB/P\fP pattern modifier is specified. See also the section about saving and
196 reloading compiled patterns below.
197 .P
198 The \fB/S\fP modifier causes \fBpcre_study()\fP to be called after the
199 expression has been compiled, and the results used when the expression is
200 matched.
201 .P
202 The \fB/M\fP modifier causes the size of memory block used to hold the compiled
203 pattern to be output.
204 .P
205 The \fB/P\fP modifier causes \fBpcretest\fP to call PCRE via the POSIX wrapper
206 API rather than its native API. When this is done, all other modifiers except
207 \fB/i\fP, \fB/m\fP, and \fB/+\fP are ignored. REG_ICASE is set if \fB/i\fP is
208 present, and REG_NEWLINE is set if \fB/m\fP is present. The wrapper functions
209 force PCRE_DOLLAR_ENDONLY always, and PCRE_DOTALL unless REG_NEWLINE is set.
210 .P
211 The \fB/8\fP modifier causes \fBpcretest\fP to call PCRE with the PCRE_UTF8
212 option set. This turns on support for UTF-8 character handling in PCRE,
213 provided that it was compiled with this support enabled. This modifier also
214 causes any non-printing characters in output strings to be printed using the
215 \ex{hh...} notation if they are valid UTF-8 sequences.
216 .P
217 If the \fB/?\fP modifier is used with \fB/8\fP, it causes \fBpcretest\fP to
218 call \fBpcre_compile()\fP with the PCRE_NO_UTF8_CHECK option, to suppress the
219 checking of the string for UTF-8 validity.
220 .
221 .
223 .rs
224 .sp
225 Before each data line is passed to \fBpcre_exec()\fP, leading and trailing
226 whitespace is removed, and it is then scanned for \e escapes. Some of these are
227 pretty esoteric features, intended for checking out some of the more
228 complicated features of PCRE. If you are just testing "ordinary" regular
229 expressions, you probably don't need any of these. The following escapes are
230 recognized:
231 .sp
232 \ea alarm (= BEL)
233 \eb backspace
234 \ee escape
235 \ef formfeed
236 \en newline
237 \er carriage return
238 \et tab
239 \ev vertical tab
240 \ennn octal character (up to 3 octal digits)
241 \exhh hexadecimal character (up to 2 hex digits)
242 .\" JOIN
243 \ex{hh...} hexadecimal character, any number of digits
244 in UTF-8 mode
245 \eA pass the PCRE_ANCHORED option to \fBpcre_exec()\fP
246 \eB pass the PCRE_NOTBOL option to \fBpcre_exec()\fP
247 .\" JOIN
248 \eCdd call pcre_copy_substring() for substring dd
249 after a successful match (number less than 32)
250 .\" JOIN
251 \eCname call pcre_copy_named_substring() for substring
252 "name" after a successful match (name termin-
253 ated by next non alphanumeric character)
254 .\" JOIN
255 \eC+ show the current captured substrings at callout
256 time
257 \eC- do not supply a callout function
258 .\" JOIN
259 \eC!n return 1 instead of 0 when callout number n is
260 reached
261 .\" JOIN
262 \eC!n!m return 1 instead of 0 when callout number n is
263 reached for the nth time
264 .\" JOIN
265 \eC*n pass the number n (may be negative) as callout
266 data; this is used as the callout return value
267 \eD use the \fBpcre_dfa_exec()\fP match function
268 \eF only shortest match for \fBpcre_dfa_exec()\fP
269 .\" JOIN
270 \eGdd call pcre_get_substring() for substring dd
271 after a successful match (number less than 32)
272 .\" JOIN
273 \eGname call pcre_get_named_substring() for substring
274 "name" after a successful match (name termin-
275 ated by next non-alphanumeric character)
276 .\" JOIN
277 \eL call pcre_get_substringlist() after a
278 successful match
279 \eM discover the minimum MATCH_LIMIT setting
280 \eN pass the PCRE_NOTEMPTY option to \fBpcre_exec()\fP
281 .\" JOIN
282 \eOdd set the size of the output vector passed to
283 \fBpcre_exec()\fP to dd (any number of digits)
284 .\" JOIN
285 \eP pass the PCRE_PARTIAL option to \fBpcre_exec()\fP
286 or \fBpcre_dfa_exec()\fP
287 \eR pass the PCRE_DFA_RESTART option to \fBpcre_dfa_exec()\fP
288 \eS output details of memory get/free calls during matching
289 \eZ pass the PCRE_NOTEOL option to \fBpcre_exec()\fP
290 .\" JOIN
291 \e? pass the PCRE_NO_UTF8_CHECK option to
292 \fBpcre_exec()\fP
293 \e>dd start the match at offset dd (any number of digits);
294 this sets the \fIstartoffset\fP argument for \fBpcre_exec()\fP
295 .sp
296 A backslash followed by anything else just escapes the anything else. If the
297 very last character is a backslash, it is ignored. This gives a way of passing
298 an empty line as data, since a real empty line terminates the data input.
299 .P
300 If \eM is present, \fBpcretest\fP calls \fBpcre_exec()\fP several times, with
301 different values in the \fImatch_limit\fP field of the \fBpcre_extra\fP data
302 structure, until it finds the minimum number that is needed for
303 \fBpcre_exec()\fP to complete. This number is a measure of the amount of
304 recursion and backtracking that takes place, and checking it out can be
305 instructive. For most simple matches, the number is quite small, but for
306 patterns with very large numbers of matching possibilities, it can become large
307 very quickly with increasing length of subject string.
308 .P
309 When \eO is used, the value specified may be higher or lower than the size set
310 by the \fB-O\fP command line option (or defaulted to 45); \eO applies only to
311 the call of \fBpcre_exec()\fP for the line in which it appears.
312 .P
313 If the \fB/P\fP modifier was present on the pattern, causing the POSIX wrapper
314 API to be used, only \eB and \eZ have any effect, causing REG_NOTBOL and
315 REG_NOTEOL to be passed to \fBregexec()\fP respectively.
316 .P
317 The use of \ex{hh...} to represent UTF-8 characters is not dependent on the use
318 of the \fB/8\fP modifier on the pattern. It is recognized always. There may be
319 any number of hexadecimal digits inside the braces. The result is from one to
320 six bytes, encoded according to the UTF-8 rules.
321 .
322 .
324 .rs
325 .sp
326 By default, \fBpcretest\fP uses the standard PCRE matching function,
327 \fBpcre_exec()\fP to match each data line. From release 6.0, PCRE supports an
328 alternative matching function, \fBpcre_dfa_test()\fP, which operates in a
329 different way, and has some restrictions. The differences between the two
330 functions are described in the
331 .\" HREF
332 \fBpcrematching\fP
333 .\"
334 documentation.
335 .P
336 If a data line contains the \eD escape sequence, or if the command line
337 contains the \fB-dfa\fP option, the alternative matching function is called.
338 This function finds all possible matches at a given point. If, however, the \eF
339 escape sequence is present in the data line, it stops after the first match is
340 found. This is always the shortest possible match.
341 .
342 .
344 .rs
345 .sp
346 This section describes the output when the normal matching function,
347 \fBpcre_exec()\fP, is being used.
348 .P
349 When a match succeeds, pcretest outputs the list of captured substrings that
350 \fBpcre_exec()\fP returns, starting with number 0 for the string that matched
351 the whole pattern. Otherwise, it outputs "No match" or "Partial match"
352 when \fBpcre_exec()\fP returns PCRE_ERROR_NOMATCH or PCRE_ERROR_PARTIAL,
353 respectively, and otherwise the PCRE negative error number. Here is an example
354 of an interactive \fBpcretest\fP run.
355 .sp
356 $ pcretest
357 PCRE version 5.00 07-Sep-2004
358 .sp
359 re> /^abc(\ed+)/
360 data> abc123
361 0: abc123
362 1: 123
363 data> xyz
364 No match
365 .sp
366 If the strings contain any non-printing characters, they are output as \e0x
367 escapes, or as \ex{...} escapes if the \fB/8\fP modifier was present on the
368 pattern. If the pattern has the \fB/+\fP modifier, the output for substring 0
369 is followed by the the rest of the subject string, identified by "0+" like
370 this:
371 .sp
372 re> /cat/+
373 data> cataract
374 0: cat
375 0+ aract
376 .sp
377 If the pattern has the \fB/g\fP or \fB/G\fP modifier, the results of successive
378 matching attempts are output in sequence, like this:
379 .sp
380 re> /\eBi(\ew\ew)/g
381 data> Mississippi
382 0: iss
383 1: ss
384 0: iss
385 1: ss
386 0: ipp
387 1: pp
388 .sp
389 "No match" is output only if the first match attempt fails.
390 .P
391 If any of the sequences \fB\eC\fP, \fB\eG\fP, or \fB\eL\fP are present in a
392 data line that is successfully matched, the substrings extracted by the
393 convenience functions are output with C, G, or L after the string number
394 instead of a colon. This is in addition to the normal full list. The string
395 length (that is, the return from the extraction function) is given in
396 parentheses after each string for \fB\eC\fP and \fB\eG\fP.
397 .P
398 Note that while patterns can be continued over several lines (a plain ">"
399 prompt is used for continuations), data lines may not. However newlines can be
400 included in data by means of the \en escape.
401 .
402 .
404 .rs
405 .sp
406 When the alternative matching function, \fBpcre_dfa_exec()\fP, is used (by
407 means of the \eD escape sequence or the \fB-dfa\fP command line option), the
408 output consists of a list of all the matches that start at the first point in
409 the subject where there is at least one match. For example:
410 .sp
411 re> /(tang|tangerine|tan)/
412 data> yellow tangerine\eD
413 0: tangerine
414 1: tang
415 2: tan
416 .sp
417 (Using the normal matching function on this data finds only "tang".) The
418 longest matching string is always given first (and numbered zero).
419 .P
420 If \fB/g\P is present on the pattern, the search for further matches resumes
421 at the end of the longest match. For example:
422 .sp
423 re> /(tang|tangerine|tan)/g
424 data> yellow tangerine and tangy sultana\eD
425 0: tangerine
426 1: tang
427 2: tan
428 0: tang
429 1: tan
430 0: tan
431 .sp
432 Since the matching function does not support substring capture, the escape
433 sequences that are concerned with captured substrings are not relevant.
434 .
435 .
437 .rs
438 .sp
439 When the alternative matching function has given the PCRE_ERROR_PARTIAL return,
440 indicating that the subject partially matched the pattern, you can restart the
441 match with additional subject data by means of the \eR escape sequence. For
442 example:
443 .sp
444 re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
445 data> 23ja\eP\eD
446 Partial match: 23ja
447 data> n05\eR\eD
448 0: n05
449 .sp
450 For further information about partial matching, see the
451 .\" HREF
452 \fBpcrepartial\fP
453 .\"
454 documentation.
455 .
456 .
458 .rs
459 .sp
460 If the pattern contains any callout requests, \fBpcretest\fP's callout function
461 is called during matching. This works with both matching functions. By default,
462 the called function displays the callout number, the start and current
463 positions in the text at the callout time, and the next pattern item to be
464 tested. For example, the output
465 .sp
466 --->pqrabcdef
467 0 ^ ^ \ed
468 .sp
469 indicates that callout number 0 occurred for a match attempt starting at the
470 fourth character of the subject string, when the pointer was at the seventh
471 character of the data, and when the next pattern item was \ed. Just one
472 circumflex is output if the start and current positions are the same.
473 .P
474 Callouts numbered 255 are assumed to be automatic callouts, inserted as a
475 result of the \fB/C\fP pattern modifier. In this case, instead of showing the
476 callout number, the offset in the pattern, preceded by a plus, is output. For
477 example:
478 .sp
479 re> /\ed?[A-E]\e*/C
480 data> E*
481 --->E*
482 +0 ^ \ed?
483 +3 ^ [A-E]
484 +8 ^^ \e*
485 +10 ^ ^
486 0: E*
487 .sp
488 The callout function in \fBpcretest\fP returns zero (carry on matching) by
489 default, but you can use a \eC item in a data line (as described above) to
490 change this.
491 .P
492 Inserting callouts can be helpful when using \fBpcretest\fP to check
493 complicated regular expressions. For further information about callouts, see
494 the
495 .\" HREF
496 \fBpcrecallout\fP
497 .\"
498 documentation.
499 .
500 .
502 .rs
503 .sp
504 The facilities described in this section are not available when the POSIX
505 inteface to PCRE is being used, that is, when the \fB/P\fP pattern modifier is
506 specified.
507 .P
508 When the POSIX interface is not in use, you can cause \fBpcretest\fP to write a
509 compiled pattern to a file, by following the modifiers with > and a file name.
510 For example:
511 .sp
512 /pattern/im >/some/file
513 .sp
514 See the
515 .\" HREF
516 \fBpcreprecompile\fP
517 .\"
518 documentation for a discussion about saving and re-using compiled patterns.
519 .P
520 The data that is written is binary. The first eight bytes are the length of the
521 compiled pattern data followed by the length of the optional study data, each
522 written as four bytes in big-endian order (most significant byte first). If
523 there is no study data (either the pattern was not studied, or studying did not
524 return any data), the second length is zero. The lengths are followed by an
525 exact copy of the compiled pattern. If there is additional study data, this
526 follows immediately after the compiled pattern. After writing the file,
527 \fBpcretest\fP expects to read a new pattern.
528 .P
529 A saved pattern can be reloaded into \fBpcretest\fP by specifing < and a file
530 name instead of a pattern. The name of the file must not contain a < character,
531 as otherwise \fBpcretest\fP will interpret the line as a pattern delimited by <
532 characters.
533 For example:
534 .sp
535 re> </some/file
536 Compiled regex loaded from /some/file
537 No study data
538 .sp
539 When the pattern has been loaded, \fBpcretest\fP proceeds to read data lines in
540 the usual way.
541 .P
542 You can copy a file written by \fBpcretest\fP to a different host and reload it
543 there, even if the new host has opposite endianness to the one on which the
544 pattern was compiled. For example, you can compile on an i86 machine and run on
545 a SPARC machine.
546 .P
547 File names for saving and reloading can be absolute or relative, but note that
548 the shell facility of expanding a file name that starts with a tilde (~) is not
549 available.
550 .P
551 The ability to save and reload files in \fBpcretest\fP is intended for testing
552 and experimentation. It is not intended for production use because only a
553 single pattern can be written to a file. Furthermore, there is no facility for
554 supplying custom character tables for use with a reloaded pattern. If the
555 original pattern was compiled with custom tables, an attempt to match a subject
556 string using a reloaded pattern is likely to cause \fBpcretest\fP to crash.
557 Finally, if you attempt to load a file that is not in the correct format, the
558 result is undefined.
559 .
560 .
562 .rs
563 .sp
564 Philip Hazel
565 .br
566 University Computing Service,
567 .br
568 Cambridge CB2 3QG, England.
569 .P
570 .in 0
571 Last updated: 28 February 2005
572 .br
573 Copyright (c) 1997-2005 University of Cambridge.

  ViewVC Help
Powered by ViewVC 1.1.5