| 1 |
ChangeLog for PCRE |
ChangeLog for PCRE |
| 2 |
------------------ |
------------------ |
| 3 |
|
|
| 4 |
|
Version 7.0 19-Dec-06 |
| 5 |
|
--------------------- |
| 6 |
|
|
| 7 |
|
1. Fixed a signed/unsigned compiler warning in pcre_compile.c, shown up by |
| 8 |
|
moving to gcc 4.1.1. |
| 9 |
|
|
| 10 |
|
2. The -S option for pcretest uses setrlimit(); I had omitted to #include |
| 11 |
|
sys/time.h, which is documented as needed for this function. It doesn't |
| 12 |
|
seem to matter on Linux, but it showed up on some releases of OS X. |
| 13 |
|
|
| 14 |
|
3. It seems that there are systems where bytes whose values are greater than |
| 15 |
|
127 match isprint() in the "C" locale. The "C" locale should be the |
| 16 |
|
default when a C program starts up. In most systems, only ASCII printing |
| 17 |
|
characters match isprint(). This difference caused the output from pcretest |
| 18 |
|
to vary, making some of the tests fail. I have changed pcretest so that: |
| 19 |
|
|
| 20 |
|
(a) When it is outputting text in the compiled version of a pattern, bytes |
| 21 |
|
other than 32-126 are always shown as hex escapes. |
| 22 |
|
|
| 23 |
|
(b) When it is outputting text that is a matched part of a subject string, |
| 24 |
|
it does the same, unless a different locale has been set for the match |
| 25 |
|
(using the /L modifier). In this case, it uses isprint() to decide. |
| 26 |
|
|
| 27 |
|
4. Fixed a major bug that caused incorrect computation of the amount of memory |
| 28 |
|
required for a compiled pattern when options that changed within the |
| 29 |
|
pattern affected the logic of the preliminary scan that determines the |
| 30 |
|
length. The relevant options are -x, and -i in UTF-8 mode. The result was |
| 31 |
|
that the computed length was too small. The symptoms of this bug were |
| 32 |
|
either the PCRE error "internal error: code overflow" from pcre_compile(), |
| 33 |
|
or a glibc crash with a message such as "pcretest: free(): invalid next |
| 34 |
|
size (fast)". Examples of patterns that provoked this bug (shown in |
| 35 |
|
pcretest format) are: |
| 36 |
|
|
| 37 |
|
/(?-x: )/x |
| 38 |
|
/(?x)(?-x: \s*#\s*)/ |
| 39 |
|
/((?i)[\x{c0}])/8 |
| 40 |
|
/(?i:[\x{c0}])/8 |
| 41 |
|
|
| 42 |
|
HOWEVER: Change 17 below makes this fix obsolete as the memory computation |
| 43 |
|
is now done differently. |
| 44 |
|
|
| 45 |
|
5. Applied patches from Google to: (a) add a QuoteMeta function to the C++ |
| 46 |
|
wrapper classes; (b) implement a new function in the C++ scanner that is |
| 47 |
|
more efficient than the old way of doing things because it avoids levels of |
| 48 |
|
recursion in the regex matching; (c) add a paragraph to the documentation |
| 49 |
|
for the FullMatch() function. |
| 50 |
|
|
| 51 |
|
6. The escape sequence \n was being treated as whatever was defined as |
| 52 |
|
"newline". Not only was this contrary to the documentation, which states |
| 53 |
|
that \n is character 10 (hex 0A), but it also went horribly wrong when |
| 54 |
|
"newline" was defined as CRLF. This has been fixed. |
| 55 |
|
|
| 56 |
|
7. In pcre_dfa_exec.c the value of an unsigned integer (the variable called c) |
| 57 |
|
was being set to -1 for the "end of line" case (supposedly a value that no |
| 58 |
|
character can have). Though this value is never used (the check for end of |
| 59 |
|
line is "zero bytes in current character"), it caused compiler complaints. |
| 60 |
|
I've changed it to 0xffffffff. |
| 61 |
|
|
| 62 |
|
8. In pcre_version.c, the version string was being built by a sequence of |
| 63 |
|
C macros that, in the event of PCRE_PRERELEASE being defined as an empty |
| 64 |
|
string (as it is for production releases) called a macro with an empty |
| 65 |
|
argument. The C standard says the result of this is undefined. The gcc |
| 66 |
|
compiler treats it as an empty string (which was what was wanted) but it is |
| 67 |
|
reported that Visual C gives an error. The source has been hacked around to |
| 68 |
|
avoid this problem. |
| 69 |
|
|
| 70 |
|
9. On the advice of a Windows user, included <io.h> and <fcntl.h> in Windows |
| 71 |
|
builds of pcretest, and changed the call to _setmode() to use _O_BINARY |
| 72 |
|
instead of 0x8000. Made all the #ifdefs test both _WIN32 and WIN32 (not all |
| 73 |
|
of them did). |
| 74 |
|
|
| 75 |
|
10. Originally, pcretest opened its input and output without "b"; then I was |
| 76 |
|
told that "b" was needed in some environments, so it was added for release |
| 77 |
|
5.0 to both the input and output. (It makes no difference on Unix-like |
| 78 |
|
systems.) Later I was told that it is wrong for the input on Windows. I've |
| 79 |
|
now abstracted the modes into two macros, to make it easier to fiddle with |
| 80 |
|
them, and removed "b" from the input mode under Windows. |
| 81 |
|
|
| 82 |
|
11. Added pkgconfig support for the C++ wrapper library, libpcrecpp. |
| 83 |
|
|
| 84 |
|
12. Added -help and --help to pcretest as an official way of being reminded |
| 85 |
|
of the options. |
| 86 |
|
|
| 87 |
|
13. Removed some redundant semicolons after macro calls in pcrecpparg.h.in |
| 88 |
|
and pcrecpp.cc because they annoy compilers at high warning levels. |
| 89 |
|
|
| 90 |
|
14. A bit of tidying/refactoring in pcre_exec.c in the main bumpalong loop. |
| 91 |
|
|
| 92 |
|
15. Fixed an occurrence of == in configure.ac that should have been = (shell |
| 93 |
|
scripts are not C programs :-) and which was not noticed because it works |
| 94 |
|
on Linux. |
| 95 |
|
|
| 96 |
|
16. pcretest is supposed to handle any length of pattern and data line (as one |
| 97 |
|
line or as a continued sequence of lines) by extending its input buffer if |
| 98 |
|
necessary. This feature was broken for very long pattern lines, leading to |
| 99 |
|
a string of junk being passed to pcre_compile() if the pattern was longer |
| 100 |
|
than about 50K. |
| 101 |
|
|
| 102 |
|
17. I have done a major re-factoring of the way pcre_compile() computes the |
| 103 |
|
amount of memory needed for a compiled pattern. Previously, there was code |
| 104 |
|
that made a preliminary scan of the pattern in order to do this. That was |
| 105 |
|
OK when PCRE was new, but as the facilities have expanded, it has become |
| 106 |
|
harder and harder to keep it in step with the real compile phase, and there |
| 107 |
|
have been a number of bugs (see for example, 4 above). I have now found a |
| 108 |
|
cunning way of running the real compile function in a "fake" mode that |
| 109 |
|
enables it to compute how much memory it would need, while actually only |
| 110 |
|
ever using a few hundred bytes of working memory and without too many |
| 111 |
|
tests of the mode. This should make future maintenance and development |
| 112 |
|
easier. A side effect of this work is that the limit of 200 on the nesting |
| 113 |
|
depth of parentheses has been removed (though this was never a serious |
| 114 |
|
limitation, I suspect). However, there is a downside: pcre_compile() now |
| 115 |
|
runs more slowly than before (30% or more, depending on the pattern). I |
| 116 |
|
hope this isn't a big issue. There is no effect on runtime performance. |
| 117 |
|
|
| 118 |
|
18. Fixed a minor bug in pcretest: if a pattern line was not terminated by a |
| 119 |
|
newline (only possible for the last line of a file) and it was a |
| 120 |
|
pattern that set a locale (followed by /Lsomething), pcretest crashed. |
| 121 |
|
|
| 122 |
|
19. Added additional timing features to pcretest. (1) The -tm option now times |
| 123 |
|
matching only, not compiling. (2) Both -t and -tm can be followed, as a |
| 124 |
|
separate command line item, by a number that specifies the number of |
| 125 |
|
repeats to use when timing. The default is 50000; this gives better |
| 126 |
|
precision, but takes uncomfortably long for very large patterns. |
| 127 |
|
|
| 128 |
|
20. Extended pcre_study() to be more clever in cases where a branch of a |
| 129 |
|
subpattern has no definite first character. For example, (a*|b*)[cd] would |
| 130 |
|
previously give no result from pcre_study(). Now it recognizes that the |
| 131 |
|
first character must be a, b, c, or d. |
| 132 |
|
|
| 133 |
|
21. There was an incorrect error "recursive call could loop indefinitely" if |
| 134 |
|
a subpattern (or the entire pattern) that was being tested for matching an |
| 135 |
|
empty string contained only one non-empty item after a nested subpattern. |
| 136 |
|
For example, the pattern (?>\x{100}*)\d(?R) provoked this error |
| 137 |
|
incorrectly, because the \d was being skipped in the check. |
| 138 |
|
|
| 139 |
|
22. The pcretest program now has a new pattern option /B and a command line |
| 140 |
|
option -b, which is equivalent to adding /B to every pattern. This causes |
| 141 |
|
it to show the compiled bytecode, without the additional information that |
| 142 |
|
-d shows. The effect of -d is now the same as -b with -i (and similarly, /D |
| 143 |
|
is the same as /B/I). |
| 144 |
|
|
| 145 |
|
23. A new optimization is now able automatically to treat some sequences such |
| 146 |
|
as a*b as a*+b. More specifically, if something simple (such as a character |
| 147 |
|
or a simple class like \d) has an unlimited quantifier, and is followed by |
| 148 |
|
something that cannot possibly match the quantified thing, the quantifier |
| 149 |
|
is automatically "possessified". |
| 150 |
|
|
| 151 |
|
24. A recursive reference to a subpattern whose number was greater than 39 |
| 152 |
|
went wrong under certain circumstances in UTF-8 mode. This bug could also |
| 153 |
|
have affected the operation of pcre_study(). |
| 154 |
|
|
| 155 |
|
25. Realized that a little bit of performance could be had by replacing |
| 156 |
|
(c & 0xc0) == 0xc0 with c >= 0xc0 when processing UTF-8 characters. |
| 157 |
|
|
| 158 |
|
26. Timing data from pcretest is now shown to 4 decimal places instead of 3. |
| 159 |
|
|
| 160 |
|
27. Possessive quantifiers such as a++ were previously implemented by turning |
| 161 |
|
them into atomic groups such as ($>a+). Now they have their own opcodes, |
| 162 |
|
which improves performance. This includes the automatically created ones |
| 163 |
|
from 23 above. |
| 164 |
|
|
| 165 |
|
28. A pattern such as (?=(\w+))\1: which simulates an atomic group using a |
| 166 |
|
lookahead was broken if it was not anchored. PCRE was mistakenly expecting |
| 167 |
|
the first matched character to be a colon. This applied both to named and |
| 168 |
|
numbered groups. |
| 169 |
|
|
| 170 |
|
29. The ucpinternal.h header file was missing its idempotency #ifdef. |
| 171 |
|
|
| 172 |
|
30. I was sent a "project" file called libpcre.a.dev which I understand makes |
| 173 |
|
building PCRE on Windows easier, so I have included it in the distribution. |
| 174 |
|
|
| 175 |
|
31. There is now a check in pcretest against a ridiculously large number being |
| 176 |
|
returned by pcre_exec() or pcre_dfa_exec(). If this happens in a /g or /G |
| 177 |
|
loop, the loop is abandoned. |
| 178 |
|
|
| 179 |
|
32. Forward references to subpatterns in conditions such as (?(2)...) where |
| 180 |
|
subpattern 2 is defined later cause pcre_compile() to search forwards in |
| 181 |
|
the pattern for the relevant set of parentheses. This search went wrong |
| 182 |
|
when there were unescaped parentheses in a character class, parentheses |
| 183 |
|
escaped with \Q...\E, or parentheses in a #-comment in /x mode. |
| 184 |
|
|
| 185 |
|
33. "Subroutine" calls and backreferences were previously restricted to |
| 186 |
|
referencing subpatterns earlier in the regex. This restriction has now |
| 187 |
|
been removed. |
| 188 |
|
|
| 189 |
|
34. Added a number of extra features that are going to be in Perl 5.10. On the |
| 190 |
|
whole, these are just syntactic alternatives for features that PCRE had |
| 191 |
|
previously implemented using the Python syntax or my own invention. The |
| 192 |
|
other formats are all retained for compatibility. |
| 193 |
|
|
| 194 |
|
(a) Named groups can now be defined as (?<name>...) or (?'name'...) as well |
| 195 |
|
as (?P<name>...). The new forms, as well as being in Perl 5.10, are |
| 196 |
|
also .NET compatible. |
| 197 |
|
|
| 198 |
|
(b) A recursion or subroutine call to a named group can now be defined as |
| 199 |
|
(?&name) as well as (?P>name). |
| 200 |
|
|
| 201 |
|
(c) A backreference to a named group can now be defined as \k<name> or |
| 202 |
|
\k'name' as well as (?P=name). The new forms, as well as being in Perl |
| 203 |
|
5.10, are also .NET compatible. |
| 204 |
|
|
| 205 |
|
(d) A conditional reference to a named group can now use the syntax |
| 206 |
|
(?(<name>) or (?('name') as well as (?(name). |
| 207 |
|
|
| 208 |
|
(e) A "conditional group" of the form (?(DEFINE)...) can be used to define |
| 209 |
|
groups (named and numbered) that are never evaluated inline, but can be |
| 210 |
|
called as "subroutines" from elsewhere. In effect, the DEFINE condition |
| 211 |
|
is always false. There may be only one alternative in such a group. |
| 212 |
|
|
| 213 |
|
(f) A test for recursion can be given as (?(R1).. or (?(R&name)... as well |
| 214 |
|
as the simple (?(R). The condition is true only if the most recent |
| 215 |
|
recursion is that of the given number or name. It does not search out |
| 216 |
|
through the entire recursion stack. |
| 217 |
|
|
| 218 |
|
(g) The escape \gN or \g{N} has been added, where N is a positive or |
| 219 |
|
negative number, specifying an absolute or relative reference. |
| 220 |
|
|
| 221 |
|
35. Tidied to get rid of some further signed/unsigned compiler warnings and |
| 222 |
|
some "unreachable code" warnings. |
| 223 |
|
|
| 224 |
|
36. Updated the Unicode property tables to Unicode version 5.0.0. Amongst other |
| 225 |
|
things, this adds five new scripts. |
| 226 |
|
|
| 227 |
|
37. Perl ignores orphaned \E escapes completely. PCRE now does the same. |
| 228 |
|
There were also incompatibilities regarding the handling of \Q..\E inside |
| 229 |
|
character classes, for example with patterns like [\Qa\E-\Qz\E] where the |
| 230 |
|
hyphen was adjacent to \Q or \E. I hope I've cleared all this up now. |
| 231 |
|
|
| 232 |
|
38. Like Perl, PCRE detects when an indefinitely repeated parenthesized group |
| 233 |
|
matches an empty string, and forcibly breaks the loop. There were bugs in |
| 234 |
|
this code in non-simple cases. For a pattern such as ^(a()*)* matched |
| 235 |
|
against aaaa the result was just "a" rather than "aaaa", for example. Two |
| 236 |
|
separate and independent bugs (that affected different cases) have been |
| 237 |
|
fixed. |
| 238 |
|
|
| 239 |
|
39. Refactored the code to abolish the use of different opcodes for small |
| 240 |
|
capturing bracket numbers. This is a tidy that I avoided doing when I |
| 241 |
|
removed the limit on the number of capturing brackets for 3.5 back in 2001. |
| 242 |
|
The new approach is not only tidier, it makes it possible to reduce the |
| 243 |
|
memory needed to fix the previous bug (38). |
| 244 |
|
|
| 245 |
|
40. Implemented PCRE_NEWLINE_ANY to recognize any of the Unicode newline |
| 246 |
|
sequences (http://unicode.org/unicode/reports/tr18/) as "newline" when |
| 247 |
|
processing dot, circumflex, or dollar metacharacters, or #-comments in /x |
| 248 |
|
mode. |
| 249 |
|
|
| 250 |
|
41. Add \R to match any Unicode newline sequence, as suggested in the Unicode |
| 251 |
|
report. |
| 252 |
|
|
| 253 |
|
42. Applied patch, originally from Ari Pollak, modified by Google, to allow |
| 254 |
|
copy construction and assignment in the C++ wrapper. |
| 255 |
|
|
| 256 |
|
43. Updated pcregrep to support "--newline=any". In the process, I fixed a |
| 257 |
|
couple of bugs that could have given wrong results in the "--newline=crlf" |
| 258 |
|
case. |
| 259 |
|
|
| 260 |
|
44. Added a number of casts and did some reorganization of signed/unsigned int |
| 261 |
|
variables following suggestions from Dair Grant. Also renamed the variable |
| 262 |
|
"this" as "item" because it is a C++ keyword. |
| 263 |
|
|
| 264 |
|
45. Arranged for dftables to add |
| 265 |
|
|
| 266 |
|
#include "pcre_internal.h" |
| 267 |
|
|
| 268 |
|
to pcre_chartables.c because without it, gcc 4.x may remove the array |
| 269 |
|
definition from the final binary if PCRE is built into a static library and |
| 270 |
|
dead code stripping is activated. |
| 271 |
|
|
| 272 |
|
46. For an unanchored pattern, if a match attempt fails at the start of a |
| 273 |
|
newline sequence, and the newline setting is CRLF or ANY, and the next two |
| 274 |
|
characters are CRLF, advance by two characters instead of one. |
| 275 |
|
|
| 276 |
|
|
| 277 |
Version 6.7 04-Jul-06 |
Version 6.7 04-Jul-06 |
| 278 |
--------------------- |
--------------------- |
| 279 |
|
|
Parent Directory
| 
Revision Log
| 
Patch