ViewVC logotype

Diff of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 67 by nigel, Sat Feb 24 21:40:13 2007 UTC revision 81 by nigel, Sat Feb 24 21:40:59 2007 UTC
# Line 1  Line 1 
1  ChangeLog for PCRE  ChangeLog for PCRE
2  ------------------  ------------------
4    Version 6.2 01-Aug-05
5    ---------------------
7     1. There was no test for integer overflow of quantifier values. A construction
8        such as {1111111111111111} would give undefined results. What is worse, if
9        a minimum quantifier for a parenthesized subpattern overflowed and became
10        negative, the calculation of the memory size went wrong. This could have
11        led to memory overwriting.
13     2. Building PCRE using VPATH was broken. Hopefully it is now fixed.
15     3. Added "b" to the 2nd argument of fopen() in dftables.c, for non-Unix-like
16        operating environments where this matters.
18     4. Applied Giuseppe Maxia's patch to add additional features for controlling
19        PCRE options from within the C++ wrapper.
21     5. Named capturing subpatterns were not being correctly counted when a pattern
22        was compiled. This caused two problems: (a) If there were more than 100
23        such subpatterns, the calculation of the memory needed for the whole
24        compiled pattern went wrong, leading to an overflow error. (b) Numerical
25        back references of the form \12, where the number was greater than 9, were
26        not recognized as back references, even though there were sufficient
27        previous subpatterns.
29     6. Two minor patches to pcrecpp.cc in order to allow it to compile on older
30        versions of gcc, e.g. 2.95.4.
33    Version 6.1 21-Jun-05
34    ---------------------
36     1. There was one reference to the variable "posix" in pcretest.c that was not
37        surrounded by "#if !defined NOPOSIX".
39     2. Make it possible to compile pcretest without DFA support, UTF8 support, or
40        the cross-check on the old pcre_info() function, for the benefit of the
41        cut-down version of PCRE that is currently imported into Exim.
43     3. A (silly) pattern starting with (?i)(?-i) caused an internal space
44        allocation error. I've done the easy fix, which wastes 2 bytes for sensible
45        patterns that start (?i) but I don't think that matters. The use of (?i) is
46        just an example; this all applies to the other options as well.
48     4. Since libtool seems to echo the compile commands it is issuing, the output
49        from "make" can be reduced a bit by putting "@" in front of each libtool
50        compile command.
52     5. Patch from the folks at Google for configure.in to be a bit more thorough
53        in checking for a suitable C++ installation before trying to compile the
54        C++ stuff. This should fix a reported problem when a compiler was present,
55        but no suitable headers.
57     6. The man pages all had just "PCRE" as their title. I have changed them to
58        be the relevant file name. I have also arranged that these names are
59        retained in the file doc/pcre.txt, which is a concatenation in text format
60        of all the man pages except the little individual ones for each function.
62     7. The NON-UNIX-USE file had not been updated for the different set of source
63        files that come with release 6. I also added a few comments about the C++
64        wrapper.
67    Version 6.0 07-Jun-05
68    ---------------------
70     1. Some minor internal re-organization to help with my DFA experiments.
72     2. Some missing #ifdef SUPPORT_UCP conditionals in pcretest and printint that
73        didn't matter for the library itself when fully configured, but did matter
74        when compiling without UCP support, or within Exim, where the ucp files are
75        not imported.
77     3. Refactoring of the library code to split up the various functions into
78        different source modules. The addition of the new DFA matching code (see
79        below) to a single monolithic source would have made it really too
80        unwieldy, quite apart from causing all the code to be include in a
81        statically linked application, when only some functions are used. This is
82        relevant even without the DFA addition now that patterns can be compiled in
83        one application and matched in another.
85        The downside of splitting up is that there have to be some external
86        functions and data tables that are used internally in different modules of
87        the library but which are not part of the API. These have all had their
88        names changed to start with "_pcre_" so that they are unlikely to clash
89        with other external names.
91     4. Added an alternate matching function, pcre_dfa_exec(), which matches using
92        a different (DFA) algorithm. Although it is slower than the original
93        function, it does have some advantages for certain types of matching
94        problem.
96     5. Upgrades to pcretest in order to test the features of pcre_dfa_exec(),
97        including restarting after a partial match.
99     6. A patch for pcregrep that defines INVALID_FILE_ATTRIBUTES if it is not
100        defined when compiling for Windows was sent to me. I have put it into the
101        code, though I have no means of testing or verifying it.
103     7. Added the pcre_refcount() auxiliary function.
105     8. Added the PCRE_FIRSTLINE option. This constrains an unanchored pattern to
106        match before or at the first newline in the subject string. In pcretest,
107        the /f option on a pattern can be used to set this.
109     9. A repeated \w when used in UTF-8 mode with characters greater than 256
110        would behave wrongly. This has been present in PCRE since release 4.0.
112    10. A number of changes to the pcregrep command:
114        (a) Refactored how -x works; insert ^(...)$ instead of setting
115            PCRE_ANCHORED and checking the length, in preparation for adding
116            something similar for -w.
118        (b) Added the -w (match as a word) option.
120        (c) Refactored the way lines are read and buffered so as to have more
121            than one at a time available.
123        (d) Implemented a pcregrep test script.
125        (e) Added the -M (multiline match) option. This allows patterns to match
126            over several lines of the subject. The buffering ensures that at least
127            8K, or the rest of the document (whichever is the shorter) is available
128            for matching (and similarly the previous 8K for lookbehind assertions).
130        (f) Changed the --help output so that it now says
132              -w, --word-regex(p)
134            instead of two lines, one with "regex" and the other with "regexp"
135            because that confused at least one person since the short forms are the
136            same. (This required a bit of code, as the output is generated
137            automatically from a table. It wasn't just a text change.)
139        (g) -- can be used to terminate pcregrep options if the next thing isn't an
140            option but starts with a hyphen. Could be a pattern or a path name
141            starting with a hyphen, for instance.
143        (h) "-" can be given as a file name to represent stdin.
145        (i) When file names are being printed, "(standard input)" is used for
146            the standard input, for compatibility with GNU grep. Previously
147            "<stdin>" was used.
149        (j) The option --label=xxx can be used to supply a name to be used for
150            stdin when file names are being printed. There is no short form.
152        (k) Re-factored the options decoding logic because we are going to add
153            two more options that take data. Such options can now be given in four
154            different ways, e.g. "-fname", "-f name", "--file=name", "--file name".
156        (l) Added the -A, -B, and -C options for requesting that lines of context
157            around matches be printed.
159        (m) Added the -L option to print the names of files that do not contain
160            any matching lines, that is, the complement of -l.
162        (n) The return code is 2 if any file cannot be opened, but pcregrep does
163            continue to scan other files.
165        (o) The -s option was incorrectly implemented. For compatibility with other
166            greps, it now suppresses the error message for a non-existent or non-
167            accessible file (but not the return code). There is a new option called
168            -q that suppresses the output of matching lines, which was what -s was
169            previously doing.
171        (p) Added --include and --exclude options to specify files for inclusion
172            and exclusion when recursing.
174    11. The Makefile was not using the Autoconf-supported LDFLAGS macro properly.
175        Hopefully, it now does.
177    12. Missing cast in pcre_study().
179    13. Added an "uninstall" target to the makefile.
181    14. Replaced "extern" in the function prototypes in Makefile.in with
182        "PCRE_DATA_SCOPE", which defaults to 'extern' or 'extern "C"' in the Unix
183        world, but is set differently for Windows.
185    15. Added a second compiling function called pcre_compile2(). The only
186        difference is that it has an extra argument, which is a pointer to an
187        integer error code. When there is a compile-time failure, this is set
188        non-zero, in addition to the error test pointer being set to point to an
189        error message. The new argument may be NULL if no error number is required
190        (but then you may as well call pcre_compile(), which is now just a
191        wrapper). This facility is provided because some applications need a
192        numeric error indication, but it has also enabled me to tidy up the way
193        compile-time errors are handled in the POSIX wrapper.
195    16. Added VPATH=.libs to the makefile; this should help when building with one
196        prefix path and installing with another. (Or so I'm told by someone who
197        knows more about this stuff than I do.)
199    17. Added a new option, REG_DOTALL, to the POSIX function regcomp(). This
200        passes PCRE_DOTALL to the pcre_compile() function, making the "." character
201        match everything, including newlines. This is not POSIX-compatible, but
202        somebody wanted the feature. From pcretest it can be activated by using
203        both the P and the s flags.
205    18. AC_PROG_LIBTOOL appeared twice in Makefile.in. Removed one.
207    19. libpcre.pc was being incorrectly installed as executable.
209    20. A couple of places in pcretest check for end-of-line by looking for '\n';
210        it now also looks for '\r' so that it will work unmodified on Windows.
212    21. Added Google's contributed C++ wrapper to the distribution.
214    22. Added some untidy missing memory free() calls in pcretest, to keep
215        Electric Fence happy when testing.
219    Version 5.0 13-Sep-04
220    ---------------------
222     1. Internal change: literal characters are no longer packed up into items
223        containing multiple characters in a single byte-string. Each character
224        is now matched using a separate opcode. However, there may be more than one
225        byte in the character in UTF-8 mode.
227     2. The pcre_callout_block structure has two new fields: pattern_position and
228        next_item_length. These contain the offset in the pattern to the next match
229        item, and its length, respectively.
231     3. The PCRE_AUTO_CALLOUT option for pcre_compile() requests the automatic
232        insertion of callouts before each pattern item. Added the /C option to
233        pcretest to make use of this.
235     4. On the advice of a Windows user, the lines
237          #if defined(_WIN32) || defined(WIN32)
238          _setmode( _fileno( stdout ), 0x8000 );
239          #endif  /* defined(_WIN32) || defined(WIN32) */
241        have been added to the source of pcretest. This apparently does useful
242        magic in relation to line terminators.
244     5. Changed "r" and "w" in the calls to fopen() in pcretest to "rb" and "wb"
245        for the benefit of those environments where the "b" makes a difference.
247     6. The icc compiler has the same options as gcc, but "configure" doesn't seem
248        to know about it. I have put a hack into configure.in that adds in code
249        to set GCC=yes if CC=icc. This seems to end up at a point in the
250        generated configure script that is early enough to affect the setting of
251        compiler options, which is what is needed, but I have no means of testing
252        whether it really works. (The user who reported this had patched the
253        generated configure script, which of course I cannot do.)
255        LATER: After change 22 below (new libtool files), the configure script
256        seems to know about icc (and also ecc). Therefore, I have commented out
257        this hack in configure.in.
259     7. Added support for pkg-config (2 patches were sent in).
261     8. Negated POSIX character classes that used a combination of internal tables
262        were completely broken. These were [[:^alpha:]], [[:^alnum:]], and
263        [[:^ascii]]. Typically, they would match almost any characters. The other
264        POSIX classes were not broken in this way.
266     9. Matching the pattern "\b.*?" against "ab cd", starting at offset 1, failed
267        to find the match, as PCRE was deluded into thinking that the match had to
268        start at the start point or following a newline. The same bug applied to
269        patterns with negative forward assertions or any backward assertions
270        preceding ".*" at the start, unless the pattern required a fixed first
271        character. This was a failing pattern: "(?!.bcd).*". The bug is now fixed.
273    10. In UTF-8 mode, when moving forwards in the subject after a failed match
274        starting at the last subject character, bytes beyond the end of the subject
275        string were read.
277    11. Renamed the variable "class" as "classbits" to make life easier for C++
278        users. (Previously there was a macro definition, but it apparently wasn't
279        enough.)
281    12. Added the new field "tables" to the extra data so that tables can be passed
282        in at exec time, or the internal tables can be re-selected. This allows
283        a compiled regex to be saved and re-used at a later time by a different
284        program that might have everything at different addresses.
286    13. Modified the pcre-config script so that, when run on Solaris, it shows a
287        -R library as well as a -L library.
289    14. The debugging options of pcretest (-d on the command line or D on a
290        pattern) showed incorrect output for anything following an extended class
291        that contained multibyte characters and which was followed by a quantifier.
293    15. Added optional support for general category Unicode character properties
294        via the \p, \P, and \X escapes. Unicode property support implies UTF-8
295        support. It adds about 90K to the size of the library. The meanings of the
296        inbuilt class escapes such as \d and \s have NOT been changed.
298    16. Updated pcredemo.c to include calls to free() to release the memory for the
299        compiled pattern.
301    17. The generated file chartables.c was being created in the source directory
302        instead of in the building directory. This caused the build to fail if the
303        source directory was different from the building directory, and was
304        read-only.
306    18. Added some sample Win commands from Mark Tetrode into the NON-UNIX-USE
307        file. No doubt somebody will tell me if they don't make sense... Also added
308        Dan Mooney's comments about building on OpenVMS.
310    19. Added support for partial matching via the PCRE_PARTIAL option for
311        pcre_exec() and the \P data escape in pcretest.
313    20. Extended pcretest with 3 new pattern features:
315        (i)   A pattern option of the form ">rest-of-line" causes pcretest to
316              write the compiled pattern to the file whose name is "rest-of-line".
317              This is a straight binary dump of the data, with the saved pointer to
318              the character tables forced to be NULL. The study data, if any, is
319              written too. After writing, pcretest reads a new pattern.
321        (ii)  If, instead of a pattern, "<rest-of-line" is given, pcretest reads a
322              compiled pattern from the given file. There must not be any
323              occurrences of "<" in the file name (pretty unlikely); if there are,
324              pcretest will instead treat the initial "<" as a pattern delimiter.
325              After reading in the pattern, pcretest goes on to read data lines as
326              usual.
328        (iii) The F pattern option causes pcretest to flip the bytes in the 32-bit
329              and 16-bit fields in a compiled pattern, to simulate a pattern that
330              was compiled on a host of opposite endianness.
332    21. The pcre-exec() function can now cope with patterns that were compiled on
333        hosts of opposite endianness, with this restriction:
335          As for any compiled expression that is saved and used later, the tables
336          pointer field cannot be preserved; the extra_data field in the arguments
337          to pcre_exec() should be used to pass in a tables address if a value
338          other than the default internal tables were used at compile time.
340    22. Calling pcre_exec() with a negative value of the "ovecsize" parameter is
341        now diagnosed as an error. Previously, most of the time, a negative number
342        would have been treated as zero, but if in addition "ovector" was passed as
343        NULL, a crash could occur.
345    23. Updated the files ltmain.sh, config.sub, config.guess, and aclocal.m4 with
346        new versions from the libtool 1.5 distribution (the last one is a copy of
347        a file called libtool.m4). This seems to have fixed the need to patch
348        "configure" to support Darwin 1.3 (which I used to do). However, I still
349        had to patch ltmain.sh to ensure that ${SED} is set (it isn't on my
350        workstation).
352    24. Changed the PCRE licence to be the more standard "BSD" licence.
355    Version 4.5 01-Dec-03
356    ---------------------
358     1. There has been some re-arrangement of the code for the match() function so
359        that it can be compiled in a version that does not call itself recursively.
360        Instead, it keeps those local variables that need separate instances for
361        each "recursion" in a frame on the heap, and gets/frees frames whenever it
362        needs to "recurse". Keeping track of where control must go is done by means
363        of setjmp/longjmp. The whole thing is implemented by a set of macros that
364        hide most of the details from the main code, and operates only if
365        NO_RECURSE is defined while compiling pcre.c. If PCRE is built using the
366        "configure" mechanism, "--disable-stack-for-recursion" turns on this way of
367        operating.
369        To make it easier for callers to provide specially tailored get/free
370        functions for this usage, two new functions, pcre_stack_malloc, and
371        pcre_stack_free, are used. They are always called in strict stacking order,
372        and the size of block requested is always the same.
374        The PCRE_CONFIG_STACKRECURSE info parameter can be used to find out whether
375        PCRE has been compiled to use the stack or the heap for recursion. The
376        -C option of pcretest uses this to show which version is compiled.
378        A new data escape \S, is added to pcretest; it causes the amounts of store
379        obtained and freed by both kinds of malloc/free at match time to be added
380        to the output.
382     2. Changed the locale test to use "fr_FR" instead of "fr" because that's
383        what's available on my current Linux desktop machine.
385     3. When matching a UTF-8 string, the test for a valid string at the start has
386        been extended. If start_offset is not zero, PCRE now checks that it points
387        to a byte that is the start of a UTF-8 character. If not, it returns
388        PCRE_ERROR_BADUTF8_OFFSET (-11). Note: the whole string is still checked;
389        this is necessary because there may be backward assertions in the pattern.
390        When matching the same subject several times, it may save resources to use
391        PCRE_NO_UTF8_CHECK on all but the first call if the string is long.
393     4. The code for checking the validity of UTF-8 strings has been tightened so
394        that it rejects (a) strings containing 0xfe or 0xff bytes and (b) strings
395        containing "overlong sequences".
397     5. Fixed a bug (appearing twice) that I could not find any way of exploiting!
398        I had written "if ((digitab[*p++] && chtab_digit) == 0)" where the "&&"
399        should have been "&", but it just so happened that all the cases this let
400        through by mistake were picked up later in the function.
402     6. I had used a variable called "isblank" - this is a C99 function, causing
403        some compilers to warn. To avoid this, I renamed it (as "blankclass").
405     7. Cosmetic: (a) only output another newline at the end of pcretest if it is
406        prompting; (b) run "./pcretest /dev/null" at the start of the test script
407        so the version is shown; (c) stop "make test" echoing "./RunTest".
409     8. Added patches from David Burgess to enable PCRE to run on EBCDIC systems.
411     9. The prototype for memmove() for systems that don't have it was using
412        size_t, but the inclusion of the header that defines size_t was later. I've
413        moved the #includes for the C headers earlier to avoid this.
415    10. Added some adjustments to the code to make it easier to compiler on certain
416        special systems:
418          (a) Some "const" qualifiers were missing.
419          (b) Added the macro EXPORT before all exported functions; by default this
420              is defined to be empty.
421          (c) Changed the dftables auxiliary program (that builds chartables.c) so
422              that it reads its output file name as an argument instead of writing
423              to the standard output and assuming this can be redirected.
425    11. In UTF-8 mode, if a recursive reference (e.g. (?1)) followed a character
426        class containing characters with values greater than 255, PCRE compilation
427        went into a loop.
429    12. A recursive reference to a subpattern that was within another subpattern
430        that had a minimum quantifier of zero caused PCRE to crash. For example,
431        (x(y(?2))z)? provoked this bug with a subject that got as far as the
432        recursion. If the recursively-called subpattern itself had a zero repeat,
433        that was OK.
435    13. In pcretest, the buffer for reading a data line was set at 30K, but the
436        buffer into which it was copied (for escape processing) was still set at
437        1024, so long lines caused crashes.
439    14. A pattern such as /[ab]{1,3}+/ failed to compile, giving the error
440        "internal error: code overflow...". This applied to any character class
441        that was followed by a possessive quantifier.
443    15. Modified the Makefile to add libpcre.la as a prerequisite for
444        libpcreposix.la because I was told this is needed for a parallel build to
445        work.
447    16. If a pattern that contained .* following optional items at the start was
448        studied, the wrong optimizing data was generated, leading to matching
449        errors. For example, studying /[ab]*.*c/ concluded, erroneously, that any
450        matching string must start with a or b or c. The correct conclusion for
451        this pattern is that a match can start with any character.
454    Version 4.4 13-Aug-03
455    ---------------------
457     1. In UTF-8 mode, a character class containing characters with values between
458        127 and 255 was not handled correctly if the compiled pattern was studied.
459        In fixing this, I have also improved the studying algorithm for such
460        classes (slightly).
462     2. Three internal functions had redundant arguments passed to them. Removal
463        might give a very teeny performance improvement.
465     3. Documentation bug: the value of the capture_top field in a callout is *one
466        more than* the number of the hightest numbered captured substring.
468     4. The Makefile linked pcretest and pcregrep with -lpcre, which could result
469        in incorrectly linking with a previously installed version. They now link
470        explicitly with libpcre.la.
472     5. configure.in no longer needs to recognize Cygwin specially.
474     6. A problem in pcre.in for Windows platforms is fixed.
476     7. If a pattern was successfully studied, and the -d (or /D) flag was given to
477        pcretest, it used to include the size of the study block as part of its
478        output. Unfortunately, the structure contains a field that has a different
479        size on different hardware architectures. This meant that the tests that
480        showed this size failed. As the block is currently always of a fixed size,
481        this information isn't actually particularly useful in pcretest output, so
482        I have just removed it.
484     8. Three pre-processor statements accidentally did not start in column 1.
485        Sadly, there are *still* compilers around that complain, even though
486        standard C has not required this for well over a decade. Sigh.
488     9. In pcretest, the code for checking callouts passed small integers in the
489        callout_data field, which is a void * field. However, some picky compilers
490        complained about the casts involved for this on 64-bit systems. Now
491        pcretest passes the address of the small integer instead, which should get
492        rid of the warnings.
494    10. By default, when in UTF-8 mode, PCRE now checks for valid UTF-8 strings at
495        both compile and run time, and gives an error if an invalid UTF-8 sequence
496        is found. There is a option for disabling this check in cases where the
497        string is known to be correct and/or the maximum performance is wanted.
499    11. In response to a bug report, I changed one line in Makefile.in from
501            -Wl,--out-implib,.libs/lib@WIN_PREFIX@pcreposix.dll.a \
502        to
503            -Wl,--out-implib,.libs/@WIN_PREFIX@libpcreposix.dll.a \
505        to look similar to other lines, but I have no way of telling whether this
506        is the right thing to do, as I do not use Windows. No doubt I'll get told
507        if it's wrong...
510    Version 4.3 21-May-03
511    ---------------------
513    1. Two instances of @WIN_PREFIX@ omitted from the Windows targets in the
514       Makefile.
516    2. Some refactoring to improve the quality of the code:
518       (i)   The utf8_table... variables are now declared "const".
520       (ii)  The code for \cx, which used the "case flipping" table to upper case
521             lower case letters, now just substracts 32. This is ASCII-specific,
522             but the whole concept of \cx is ASCII-specific, so it seems
523             reasonable.
525       (iii) PCRE was using its character types table to recognize decimal and
526             hexadecimal digits in the pattern. This is silly, because it handles
527             only 0-9, a-f, and A-F, but the character types table is locale-
528             specific, which means strange things might happen. A private
529             table is now used for this - though it costs 256 bytes, a table is
530             much faster than multiple explicit tests. Of course, the standard
531             character types table is still used for matching digits in subject
532             strings against \d.
534       (iv)  Strictly, the identifier ESC_t is reserved by POSIX (all identifiers
535             ending in _t are). So I've renamed it as ESC_tee.
537    3. The first argument for regexec() in the POSIX wrapper should have been
538       defined as "const".
540    4. Changed pcretest to use malloc() for its buffers so that they can be
541       Electric Fenced for debugging.
543    5. There were several places in the code where, in UTF-8 mode, PCRE would try
544       to read one or more bytes before the start of the subject string. Often this
545       had no effect on PCRE's behaviour, but in some circumstances it could
546       provoke a segmentation fault.
548    6. A lookbehind at the start of a pattern in UTF-8 mode could also cause PCRE
549       to try to read one or more bytes before the start of the subject string.
551    7. A lookbehind in a pattern matched in non-UTF-8 mode on a PCRE compiled with
552       UTF-8 support could misbehave in various ways if the subject string
553       contained bytes with the 0x80 bit set and the 0x40 bit unset in a lookbehind
554       area. (PCRE was not checking for the UTF-8 mode flag, and trying to move
555       back over UTF-8 characters.)
558  Version 4.2 14-Apr-03  Version 4.2 14-Apr-03
559  ---------------------  ---------------------

Removed from v.67  
changed lines
  Added in v.81

  ViewVC Help
Powered by ViewVC 1.1.5