/[pcre]/code/trunk/ChangeLog
ViewVC logotype

Diff of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 61 by nigel, Sat Feb 24 21:39:58 2007 UTC revision 63 by nigel, Sat Feb 24 21:40:03 2007 UTC
# Line 1  Line 1 
1  ChangeLog for PCRE  ChangeLog for PCRE
2  ------------------  ------------------
3    
4  Version 3.0 02-Jan-02  Version 4.00 17-Feb-03
5    ----------------------
6    
7    1. If a comment in an extended regex that started immediately after a meta-item
8    extended to the end of string, PCRE compiled incorrect data. This could lead to
9    all kinds of weird effects. Example: /#/ was bad; /()#/ was bad; /a#/ was not.
10    
11    2. Moved to autoconf 2.53 and libtool 1.4.2.
12    
13    3. Perl 5.8 no longer needs "use utf8" for doing UTF-8 things. Consequently,
14    the special perltest8 script is no longer needed - all the tests can be run
15    from a single perltest script.
16    
17    4. From 5.004, Perl has not included the VT character (0x0b) in the set defined
18    by \s. It has now been removed in PCRE. This means it isn't recognized as
19    whitespace in /x regexes too, which is the same as Perl. Note that the POSIX
20    class [:space:] *does* include VT, thereby creating a mess.
21    
22    5. Added the class [:blank:] (a GNU extension from Perl 5.8) to match only
23    space and tab.
24    
25    6. Perl 5.005 was a long time ago. It's time to amalgamate the tests that use
26    its new features into the main test script, reducing the number of scripts.
27    
28    7. Perl 5.8 has changed the meaning of patterns like /a(?i)b/. Earlier versions
29    were backward compatible, and made the (?i) apply to the whole pattern, as if
30    /i were given. Now it behaves more logically, and applies the option setting
31    only to what follows. PCRE has been changed to follow suit. However, if it
32    finds options settings right at the start of the pattern, it extracts them into
33    the global options, as before. Thus, they show up in the info data.
34    
35    8. Added support for the \Q...\E escape sequence. Characters in between are
36    treated as literals. This is slightly different from Perl in that $ and @ are
37    also handled as literals inside the quotes. In Perl, they will cause variable
38    interpolation. Note the following examples:
39    
40        Pattern            PCRE matches      Perl matches
41    
42        \Qabc$xyz\E        abc$xyz           abc followed by the contents of $xyz
43        \Qabc\$xyz\E       abc\$xyz          abc\$xyz
44        \Qabc\E\$\Qxyz\E   abc$xyz           abc$xyz
45    
46    For compatibility with Perl, \Q...\E sequences are recognized inside character
47    classes as well as outside them.
48    
49    9. Re-organized 3 code statements in pcretest to avoid "overflow in
50    floating-point constant arithmetic" warnings from a Microsoft compiler. Added a
51    (size_t) cast to one statement in pcretest and one in pcreposix to avoid
52    signed/unsigned warnings.
53    
54    10. SunOS4 doesn't have strtoul(). This was used only for unpicking the -o
55    option for pcretest, so I've replaced it by a simple function that does just
56    that job.
57    
58    11. pcregrep was ending with code 0 instead of 2 for the commands "pcregrep" or
59    "pcregrep -".
60    
61    12. Added "possessive quantifiers" ?+, *+, ++, and {,}+ which come from Sun's
62    Java package. This provides some syntactic sugar for simple cases of what my
63    documentation calls "once-only subpatterns". A pattern such as x*+ is the same
64    as (?>x*). In other words, if what is inside (?>...) is just a single repeated
65    item, you can use this simplified notation. Note that only makes sense with
66    greedy quantifiers. Consequently, the use of the possessive quantifier forces
67    greediness, whatever the setting of the PCRE_UNGREEDY option.
68    
69    13. A change of greediness default within a pattern was not taking effect at
70    the current level for patterns like /(b+(?U)a+)/. It did apply to parenthesized
71    subpatterns that followed. Patterns like /b+(?U)a+/ worked because the option
72    was abstracted outside.
73    
74    14. PCRE now supports the \G assertion. It is true when the current matching
75    position is at the start point of the match. This differs from \A when the
76    starting offset is non-zero. Used with the /g option of pcretest (or similar
77    code), it works in the same way as it does for Perl's /g option. If all
78    alternatives of a regex begin with \G, the expression is anchored to the start
79    match position, and the "anchored" flag is set in the compiled expression.
80    
81    15. Some bugs concerning the handling of certain option changes within patterns
82    have been fixed. These applied to options other than (?ims). For example,
83    "a(?x: b c )d" did not match "XabcdY" but did match "Xa b c dY". It should have
84    been the other way round. Some of this was related to change 7 above.
85    
86    16. PCRE now gives errors for /[.x.]/ and /[=x=]/ as unsupported POSIX
87    features, as Perl does. Previously, PCRE gave the warnings only for /[[.x.]]/
88    and /[[=x=]]/. PCRE now also gives an error for /[:name:]/ because it supports
89    POSIX classes only within a class (e.g. /[[:alpha:]]/).
90    
91    17. Added support for Perl's \C escape. This matches one byte, even in UTF8
92    mode. Unlike ".", it always matches newline, whatever the setting of
93    PCRE_DOTALL. However, PCRE does not permit \C to appear in lookbehind
94    assertions. Perl allows it, but it doesn't (in general) work because it can't
95    calculate the length of the lookbehind. At least, that's the case for Perl
96    5.8.0 - I've been told they are going to document that it doesn't work in
97    future.
98    
99    18. Added an error diagnosis for escapes that PCRE does not support: these are
100    \L, \l, \N, \P, \p, \U, \u, and \X.
101    
102    19. Although correctly diagnosing a missing ']' in a character class, PCRE was
103    reading past the end of the pattern in cases such as /[abcd/.
104    
105    20. PCRE was getting more memory than necessary for patterns with classes that
106    contained both POSIX named classes and other characters, e.g. /[[:space:]abc/.
107    
108    21. Added some code, conditional on #ifdef VPCOMPAT, to make life easier for
109    compiling PCRE for use with Virtual Pascal.
110    
111    22. Small fix to the Makefile to make it work properly if the build is done
112    outside the source tree.
113    
114    23. Added a new extension: a condition to go with recursion. If a conditional
115    subpattern starts with (?(R) the "true" branch is used if recursion has
116    happened, whereas the "false" branch is used only at the top level.
117    
118    24. When there was a very long string of literal characters (over 255 bytes
119    without UTF support, over 250 bytes with UTF support), the computation of how
120    much memory was required could be incorrect, leading to segfaults or other
121    strange effects.
122    
123    25. PCRE was incorrectly assuming anchoring (either to start of subject or to
124    start of line for a non-DOTALL pattern) when a pattern started with (.*) and
125    there was a subsequent back reference to those brackets. This meant that, for
126    example, /(.*)\d+\1/ failed to match "abc123bc". Unfortunately, it isn't
127    possible to check for precisely this case. All we can do is abandon the
128    optimization if .* occurs inside capturing brackets when there are any back
129    references whatsoever. (See below for a better fix that came later.)
130    
131    26. The handling of the optimization for finding the first character of a
132    non-anchored pattern, and for finding a character that is required later in the
133    match were failing in some cases. This didn't break the matching; it just
134    failed to optimize when it could. The way this is done has been re-implemented.
135    
136    27. Fixed typo in error message for invalid (?R item (it said "(?p").
137    
138    28. Added a new feature that provides some of the functionality that Perl
139    provides with (?{...}). The facility is termed a "callout". The way it is done
140    in PCRE is for the caller to provide an optional function, by setting
141    pcre_callout to its entry point. Like pcre_malloc and pcre_free, this is a
142    global variable. By default it is unset, which disables all calling out. To get
143    the function called, the regex must include (?C) at appropriate points. This
144    is, in fact, equivalent to (?C0), and any number <= 255 may be given with (?C).
145    This provides a means of identifying different callout points. When PCRE
146    reaches such a point in the regex, if pcre_callout has been set, the external
147    function is called. It is provided with data in a structure called
148    pcre_callout_block, which is defined in pcre.h. If the function returns 0,
149    matching continues; if it returns a non-zero value, the match at the current
150    point fails. However, backtracking will occur if possible. [This was changed
151    later and other features added - see item 49 below.]
152    
153    29. pcretest is upgraded to test the callout functionality. It provides a
154    callout function that displays information. By default, it shows the start of
155    the match and the current position in the text. There are some new data escapes
156    to vary what happens:
157    
158        \C+         in addition, show current contents of captured substrings
159        \C-         do not supply a callout function
160        \C!n        return 1 when callout number n is reached
161        \C!n!m      return 1 when callout number n is reached for the mth time
162    
163    30. If pcregrep was called with the -l option and just a single file name, it
164    output "<stdin>" if a match was found, instead of the file name.
165    
166    31. Improve the efficiency of the POSIX API to PCRE. If the number of capturing
167    slots is less than POSIX_MALLOC_THRESHOLD, use a block on the stack to pass to
168    pcre_exec(). This saves a malloc/free per call. The default value of
169    POSIX_MALLOC_THRESHOLD is 10; it can be changed by --with-posix-malloc-threshold
170    when configuring.
171    
172    32. The default maximum size of a compiled pattern is 64K. There have been a
173    few cases of people hitting this limit. The code now uses macros to handle the
174    storing of links as offsets within the compiled pattern. It defaults to 2-byte
175    links, but this can be changed to 3 or 4 bytes by --with-link-size when
176    configuring. Tests 2 and 5 work only with 2-byte links because they output
177    debugging information about compiled patterns.
178    
179    33. Internal code re-arrangements:
180    
181    (a) Moved the debugging function for printing out a compiled regex into
182        its own source file (printint.c) and used #include to pull it into
183        pcretest.c and, when DEBUG is defined, into pcre.c, instead of having two
184        separate copies.
185    
186    (b) Defined the list of op-code names for debugging as a macro in
187        internal.h so that it is next to the definition of the opcodes.
188    
189    (c) Defined a table of op-code lengths for simpler skipping along compiled
190        code. This is again a macro in internal.h so that it is next to the
191        definition of the opcodes.
192    
193    34. Added support for recursive calls to individual subpatterns, along the
194    lines of Robin Houston's patch (but implemented somewhat differently).
195    
196    35. Further mods to the Makefile to help Win32. Also, added code to pcregrep to
197    allow it to read and process whole directories in Win32. This code was
198    contributed by Lionel Fourquaux; it has not been tested by me.
199    
200    36. Added support for named subpatterns. The Python syntax (?P<name>...) is
201    used to name a group. Names consist of alphanumerics and underscores, and must
202    be unique. Back references use the syntax (?P=name) and recursive calls use
203    (?P>name) which is a PCRE extension to the Python extension. Groups still have
204    numbers. The function pcre_fullinfo() can be used after compilation to extract
205    a name/number map. There are three relevant calls:
206    
207      PCRE_INFO_NAMEENTRYSIZE        yields the size of each entry in the map
208      PCRE_INFO_NAMECOUNT            yields the number of entries
209      PCRE_INFO_NAMETABLE            yields a pointer to the map.
210    
211    The map is a vector of fixed-size entries. The size of each entry depends on
212    the length of the longest name used. The first two bytes of each entry are the
213    group number, most significant byte first. There follows the corresponding
214    name, zero terminated. The names are in alphabetical order.
215    
216    37. Make the maximum literal string in the compiled code 250 for the non-UTF-8
217    case instead of 255. Making it the same both with and without UTF-8 support
218    means that the same test output works with both.
219    
220    38. There was a case of malloc(0) in the POSIX testing code in pcretest. Avoid
221    calling malloc() with a zero argument.
222    
223    39. Change 25 above had to resort to a heavy-handed test for the .* anchoring
224    optimization. I've improved things by keeping a bitmap of backreferences with
225    numbers 1-31 so that if .* occurs inside capturing brackets that are not in
226    fact referenced, the optimization can be applied. It is unlikely that a
227    relevant occurrence of .* (i.e. one which might indicate anchoring or forcing
228    the match to follow \n) will appear inside brackets with a number greater than
229    31, but if it does, any back reference > 31 suppresses the optimization.
230    
231    40. Added a new compile-time option PCRE_NO_AUTO_CAPTURE. This has the effect
232    of disabling numbered capturing parentheses. Any opening parenthesis that is
233    not followed by ? behaves as if it were followed by ?: but named parentheses
234    can still be used for capturing (and they will acquire numbers in the usual
235    way).
236    
237    41. Redesigned the return codes from the match() function into yes/no/error so
238    that errors can be passed back from deep inside the nested calls. A malloc
239    failure while inside a recursive subpattern call now causes the
240    PCRE_ERROR_NOMEMORY return instead of quietly going wrong.
241    
242    42. It is now possible to set a limit on the number of times the match()
243    function is called in a call to pcre_exec(). This facility makes it possible to
244    limit the amount of recursion and backtracking, though not in a directly
245    obvious way, because the match() function is used in a number of different
246    circumstances. The count starts from zero for each position in the subject
247    string (for non-anchored patterns). The default limit is, for compatibility, a
248    large number, namely 10 000 000. You can change this in two ways:
249    
250    (a) When configuring PCRE before making, you can use --with-match-limit=n
251        to set a default value for the compiled library.
252    
253    (b) For each call to pcre_exec(), you can pass a pcre_extra block in which
254        a different value is set. See 45 below.
255    
256    If the limit is exceeded, pcre_exec() returns PCRE_ERROR_MATCHLIMIT.
257    
258    43. Added a new function pcre_config(int, void *) to enable run-time extraction
259    of things that can be changed at compile time. The first argument specifies
260    what is wanted and the second points to where the information is to be placed.
261    The current list of available information is:
262    
263      PCRE_CONFIG_UTF8
264    
265    The output is an integer that is set to one if UTF-8 support is available;
266    otherwise it is set to zero.
267    
268      PCRE_CONFIG_NEWLINE
269    
270    The output is an integer that it set to the value of the code that is used for
271    newline. It is either LF (10) or CR (13).
272    
273      PCRE_CONFIG_LINK_SIZE
274    
275    The output is an integer that contains the number of bytes used for internal
276    linkage in compiled expressions. The value is 2, 3, or 4. See item 32 above.
277    
278      PCRE_CONFIG_POSIX_MALLOC_THRESHOLD
279    
280    The output is an integer that contains the threshold above which the POSIX
281    interface uses malloc() for output vectors. See item 31 above.
282    
283      PCRE_CONFIG_MATCH_LIMIT
284    
285    The output is an unsigned integer that contains the default limit of the number
286    of match() calls in a pcre_exec() execution. See 42 above.
287    
288    44. pcretest has been upgraded by the addition of the -C option. This causes it
289    to extract all the available output from the new pcre_config() function, and to
290    output it. The program then exits immediately.
291    
292    45. A need has arisen to pass over additional data with calls to pcre_exec() in
293    order to support additional features. One way would have been to define
294    pcre_exec2() (for example) with extra arguments, but this would not have been
295    extensible, and would also have required all calls to the original function to
296    be mapped to the new one. Instead, I have chosen to extend the mechanism that
297    is used for passing in "extra" data from pcre_study().
298    
299    The pcre_extra structure is now exposed and defined in pcre.h. It currently
300    contains the following fields:
301    
302      flags         a bitmap indicating which of the following fields are set
303      study_data    opaque data from pcre_study()
304      match_limit   a way of specifying a limit on match() calls for a specific
305                      call to pcre_exec()
306      callout_data  data for callouts (see 49 below)
307    
308    The flag bits are also defined in pcre.h, and are
309    
310      PCRE_EXTRA_STUDY_DATA
311      PCRE_EXTRA_MATCH_LIMIT
312      PCRE_EXTRA_CALLOUT_DATA
313    
314    The pcre_study() function now returns one of these new pcre_extra blocks, with
315    the actual study data pointed to by the study_data field, and the
316    PCRE_EXTRA_STUDY_DATA flag set. This can be passed directly to pcre_exec() as
317    before. That is, this change is entirely upwards-compatible and requires no
318    change to existing code.
319    
320    If you want to pass in additional data to pcre_exec(), you can either place it
321    in a pcre_extra block provided by pcre_study(), or create your own pcre_extra
322    block.
323    
324    46. pcretest has been extended to test the PCRE_EXTRA_MATCH_LIMIT feature. If a
325    data string contains the escape sequence \M, pcretest calls pcre_exec() several
326    times with different match limits, until it finds the minimum value needed for
327    pcre_exec() to complete. The value is then output. This can be instructive; for
328    most simple matches the number is quite small, but for pathological cases it
329    gets very large very quickly.
330    
331    47. There's a new option for pcre_fullinfo() called PCRE_INFO_STUDYSIZE. It
332    returns the size of the data block pointed to by the study_data field in a
333    pcre_extra block, that is, the value that was passed as the argument to
334    pcre_malloc() when PCRE was getting memory in which to place the information
335    created by pcre_study(). The fourth argument should point to a size_t variable.
336    pcretest has been extended so that this information is shown after a successful
337    pcre_study() call when information about the compiled regex is being displayed.
338    
339    48. Cosmetic change to Makefile: there's no need to have / after $(DESTDIR)
340    because what follows is always an absolute path. (Later: it turns out that this
341    is more than cosmetic for MinGW, because it doesn't like empty path
342    components.)
343    
344    49. Some changes have been made to the callout feature (see 28 above):
345    
346    (i)  A callout function now has three choices for what it returns:
347    
348           0  =>  success, carry on matching
349         > 0  =>  failure at this point, but backtrack if possible
350         < 0  =>  serious error, return this value from pcre_exec()
351    
352         Negative values should normally be chosen from the set of PCRE_ERROR_xxx
353         values. In particular, returning PCRE_ERROR_NOMATCH forces a standard
354         "match failed" error. The error number PCRE_ERROR_CALLOUT is reserved for
355         use by callout functions. It will never be used by PCRE itself.
356    
357    (ii) The pcre_extra structure (see 45 above) has a void * field called
358         callout_data, with corresponding flag bit PCRE_EXTRA_CALLOUT_DATA. The
359         pcre_callout_block structure has a field of the same name. The contents of
360         the field passed in the pcre_extra structure are passed to the callout
361         function in the corresponding field in the callout block. This makes it
362         easier to use the same callout-containing regex from multiple threads. For
363         testing, the pcretest program has a new data escape
364    
365           \C*n        pass the number n (may be negative) as callout_data
366    
367         If the callout function in pcretest receives a non-zero value as
368         callout_data, it returns that value.
369    
370    50. Makefile wasn't handling CFLAGS properly when compiling dftables. Also,
371    there were some redundant $(CFLAGS) in commands that are now specified as
372    $(LINK), which already includes $(CFLAGS).
373    
374    51. Extensions to UTF-8 support are listed below. These all apply when (a) PCRE
375    has been compiled with UTF-8 support *and* pcre_compile() has been compiled
376    with the PCRE_UTF8 flag. Patterns that are compiled without that flag assume
377    one-byte characters throughout. Note that case-insensitive matching applies
378    only to characters whose values are less than 256. PCRE doesn't support the
379    notion of cases for higher-valued characters.
380    
381    (i)   A character class whose characters are all within 0-255 is handled as
382          a bit map, and the map is inverted for negative classes. Previously, a
383          character > 255 always failed to match such a class; however it should
384          match if the class was a negative one (e.g. [^ab]). This has been fixed.
385    
386    (ii)  A negated character class with a single character < 255 is coded as
387          "not this character" (OP_NOT). This wasn't working properly when the test
388          character was multibyte, either singly or repeated.
389    
390    (iii) Repeats of multibyte characters are now handled correctly in UTF-8
391          mode, for example: \x{100}{2,3}.
392    
393    (iv)  The character escapes \b, \B, \d, \D, \s, \S, \w, and \W (either
394          singly or repeated) now correctly test multibyte characters. However,
395          PCRE doesn't recognize any characters with values greater than 255 as
396          digits, spaces, or word characters. Such characters always match \D, \S,
397          and \W, and never match \d, \s, or \w.
398    
399    (v)   Classes may now contain characters and character ranges with values
400          greater than 255. For example: [ab\x{100}-\x{400}].
401    
402    (vi)  pcregrep now has a --utf-8 option (synonym -u) which makes it call
403          PCRE in UTF-8 mode.
404    
405    52. The info request value PCRE_INFO_FIRSTCHAR has been renamed
406    PCRE_INFO_FIRSTBYTE because it is a byte value. However, the old name is
407    retained for backwards compatibility. (Note that LASTLITERAL is also a byte
408    value.)
409    
410    53. The single man page has become too large. I have therefore split it up into
411    a number of separate man pages. These also give rise to individual HTML pages;
412    these are now put in a separate directory, and there is an index.html page that
413    lists them all. Some hyperlinking between the pages has been installed.
414    
415    54. Added convenience functions for handling named capturing parentheses.
416    
417    55. Unknown escapes inside character classes (e.g. [\M]) and escapes that
418    aren't interpreted therein (e.g. [\C]) are literals in Perl. This is now also
419    true in PCRE, except when the PCRE_EXTENDED option is set, in which case they
420    are faulted.
421    
422    56. Introduced HOST_CC and HOST_CFLAGS which can be set in the environment when
423    calling configure. These values are used when compiling the dftables.c program
424    which is run to generate the source of the default character tables. They
425    default to the values of CC and CFLAGS. If you are cross-compiling PCRE,
426    you will need to set these values.
427    
428    57. Updated the building process for Windows DLL, as provided by Fred Cox.
429    
430    
431    Version 3.9 02-Jan-02
432  ---------------------  ---------------------
433    
434  1. A bit of extraneous text had somehow crept into the pcregrep documentation.  1. A bit of extraneous text had somehow crept into the pcregrep documentation.
Legend:
Removed from v.61  
changed lines
  Added in v.63
  ViewVC Help
Powered by ViewVC 1.1.5