/[pcre2]/code/trunk/ChangeLog
ViewVC logotype

Contents of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log


Revision 696 - (show annotations)
Tue Mar 21 17:46:21 2017 UTC (2 years, 4 months ago) by ph10
File size: 64447 byte(s)
Fix 32-bit error buffer size bug in pcre2test (Bugzilla 2079).
1 Change Log for PCRE2
2 --------------------
3
4
5 Version 10.30-DEV 09-March-2017
6 -------------------------------
7
8 1. The main interpreter, pcre2_match(), has been refactored into a new version
9 that does not use recursive function calls (and therefore the stack) for
10 remembering backtracking positions. This makes --disable-stack-for-recursion a
11 NOOP. The new implementation allows backtracking into recursive group calls in
12 patterns, making it more compatible with Perl, and also fixes some other
13 hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
14 the old code had a number of fudges to try to reduce stack usage. It seems to
15 run no slower than the old code.
16
17 A number of bugs in the refactored code were subsequently fixed during testing
18 before release, but after the code was made available in the repository. Many
19 of the bugs were discovered by fuzzing testing. These bugs were never in fully
20 released code, but are noted here for the record.
21
22 (a) If a pattern had fewer capturing parentheses than the ovector supplied in
23 the match data block, a memory error (detectable by ASAN) occurred after
24 a match, because the external block was being set from non-existent
25 internal ovector fields. Fixes oss-fuzz issue 781.
26
27 (b) A pattern with very many capturing parentheses (when the internal frame
28 size was greater than the initial frame vector on the stack) caused a
29 crash. A vector on the heap is now set up at the start of matching if the
30 vector on the stack is not big enough to handle at least 10 frames.
31 Fixes oss-fuzz issue 783.
32
33 2. Now that pcre2_match() no longer uses recursive function calls (see above),
34 the "match limit recursion" value seems misnamed. It still exists, and limits
35 the depth of tree that is searched. To avoid future confusion, it has been
36 renamed as "depth limit" in all relevant places (--with-depth-limit,
37 (*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
38 available for backwards compatibility.
39
40 3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:
41
42 (a) Check for malloc failures when getting memory for the ovector (POSIX) or
43 the match data block (non-POSIX).
44
45 4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
46 for a character with a code point greater than 0x10ffff (the Unicode maximum)
47 caused a crash.
48
49 5. If a lookbehind assertion that contained a back reference to a group
50 appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
51 undefined actions (often a segmentation fault) could occur, depending on what
52 other options were set. An example assertion is (?<!\1(abc)) where the
53 reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.
54
55 6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
56 pcre2test to use it to output the frame size when the "framesize" modifier is
57 given.
58
59 7. Reworked the recursive pattern matching in the JIT compiler to follow the
60 interpreter changes.
61
62 8. When the zero_terminate modifier was specified on a pcre2test subject line
63 for global matching, unpredictable things could happen. For example, in UTF-8
64 mode, the pattern //g,zero_terminate read random memory when matched against an
65 empty string with zero_terminate. This was a bug in pcre2test, not the library.
66
67 9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
68 of the section that is compiled when Unix-style directory scanning is
69 available, and into a new section that is always compiled for Windows.
70
71 10. In pcre2test, explicitly close the file after an error during serialization
72 or deserialization (the "load" or "save" commands).
73
74 11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.
75
76 12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
77 a NULL pattern pointer when Unicode support is available.
78
79 13. When the 32-bit library was being tested by pcre2test, error messages that
80 were longer than 64 code units could cause a buffer overflow. This was a bug in
81 pcre2test.
82
83
84 Version 10.23 14-February-2017
85 ------------------------------
86
87 1. Extended pcre2test with the utf8_input modifier so that it is able to
88 generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
89
90 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
91 PCRE2_UCP set, a negative character type such as \D in a positive class should
92 cause all characters greater than 255 to match, whatever else is in the class.
93 There was a bug that caused this not to happen if a Unicode property item was
94 added to such a class, for example [\D\P{Nd}] or [\W\pL].
95
96 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
97 checking is now done in the pre-pass that identifies capturing groups. This has
98 reduced the amount of duplication and made the code tidier. While doing this,
99 some minor bugs and Perl incompatibilities were fixed, including:
100
101 (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
102 of giving an invalid quantifier error.
103
104 (b) {0} can now be used after a group in a lookbehind assertion; previously
105 this caused an "assertion is not fixed length" error.
106
107 (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
108 the name "DEFINE" exists. PCRE2 now does likewise.
109
110 (d) A recursion condition test such as (?(R2)...) must now refer to an
111 existing subpattern.
112
113 (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
114 group whose name began with "R".
115
116 (f) When testing zero-terminated patterns under valgrind, the terminating
117 zero is now marked "no access". This catches bugs that would otherwise
118 show up only with non-zero-terminated patterns.
119
120 (g) A hyphen appearing immediately after a POSIX character class (for example
121 /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
122 literal, but gives a warning, so it seems best to fail it in PCRE.
123
124 (h) An empty \Q\E sequence may appear after a callout that precedes an
125 assertion condition (it is, of course, ignored).
126
127 One effect of the refactoring is that some error numbers and messages have
128 changed, and the pattern offset given for compiling errors is not always the
129 right-most character that has been read. In particular, for a variable-length
130 lookbehind assertion it now points to the start of the assertion. Another
131 change is that when a callout appears before a group, the "length of next
132 pattern item" that is passed now just gives the length of the opening
133 parenthesis item, not the length of the whole group. A length of zero is now
134 given only for a callout at the end of the pattern. Automatic callouts are no
135 longer inserted before and after explicit callouts in the pattern.
136
137 A number of bugs in the refactored code were subsequently fixed during testing
138 before release, but after the code was made available in the repository. Many
139 of the bugs were discovered by fuzzing testing. Several of them were related to
140 the change from assuming a zero-terminated pattern (which previously had
141 required non-zero terminated strings to be copied). These bugs were never in
142 fully released code, but are noted here for the record.
143
144 (a) An overall recursion such as (?0) inside a lookbehind assertion was not
145 being diagnosed as an error.
146
147 (b) In utf mode, the length of a *MARK (or other verb) name was being checked
148 in characters instead of code units, which could lead to bad code being
149 compiled, leading to unpredictable behaviour.
150
151 (c) In extended /x mode, characters whose code was greater than 255 caused
152 a lookup outside one of the global tables. A similar bug existed for wide
153 characters in *VERB names.
154
155 (d) The amount of memory needed for a compiled pattern was miscalculated if a
156 lookbehind contained more than one toplevel branch and the first branch
157 was of length zero.
158
159 (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
160 terminated pattern, if a # comment ran on to the end of the pattern, one
161 or more code units past the end were being read.
162
163 (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
164 "{2,2") could cause reading beyond the pattern.
165
166 (g) When reading a callout string, if the end delimiter was at the end of the
167 pattern one further code unit was read.
168
169 (h) An unterminated number after \g' could cause reading beyond the pattern.
170
171 (i) An insufficient memory size was being computed for compiling with
172 PCRE2_AUTO_CALLOUT.
173
174 (j) A conditional group with an assertion condition used more memory than was
175 allowed for it during parsing, so too many of them could therefore
176 overrun a buffer.
177
178 (k) If parsing a pattern exactly filled the buffer, the internal test for
179 overrun did not check when the final META_END item was added.
180
181 (l) If a lookbehind contained a subroutine call, and the called group
182 contained an option setting such as (?s), and the PCRE2_ANCHORED option
183 was set, unpredictable behaviour could occur. The underlying bug was
184 incorrect code and insufficient checking while searching for the end of
185 the called subroutine in the parsed pattern.
186
187 (m) Quantifiers following (*VERB)s were not being diagnosed as errors.
188
189 (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
190 PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
191
192 (o) If \Q was preceded by a quantified item, and the following \E was
193 followed by '?' or '+', and there was at least one literal character
194 between them, an internal error "unexpected repeat" occurred (example:
195 /.+\QX\E+/).
196
197 (p) A buffer overflow could occur while sorting the names in the group name
198 list (depending on the order in which the names were seen).
199
200 (q) A conditional group that started with a callout was not doing the right
201 check for a following assertion, leading to compiling bad code. Example:
202 /(?(C'XX))?!XX/
203
204 (r) If a character whose code point was greater than 0xffff appeared within
205 a lookbehind that was within another lookbehind, the calculation of the
206 lookbehind length went wrong and could provoke an internal error.
207
208 (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
209 an internal error. Now the hyphen is treated as a literal.
210
211 4. Back references are now permitted in lookbehind assertions when there are
212 no duplicated group numbers (that is, (?| has not been used), and, if the
213 reference is by name, there is only one group of that name. The referenced
214 group must, of course be of fixed length.
215
216 5. pcre2test has been upgraded so that, when run under valgrind with valgrind
217 support enabled, reading past the end of the pattern is detected, both when
218 compiling and during callout processing.
219
220 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
221 reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
222 not recognize this syntax.
223
224 7. Automatic callouts are no longer generated before and after callouts in the
225 pattern.
226
227 8. When pcre2test was outputing information from a callout, the caret indicator
228 for the current position in the subject line was incorrect if it was after an
229 escape sequence for a character whose code point was greater than \x{ff}.
230
231 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
232 PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
233
234 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
235 expansion when long lines are encountered. Original patch by Dmitry
236 Cherniachenko.
237
238 11. If pcre2grep was compiled with JIT support, but the library was compiled
239 without it (something that neither ./configure nor CMake allow, but it can be
240 done by editing config.h), pcre2grep was giving a JIT error. Now it detects
241 this situation and does not try to use JIT.
242
243 12. Added some "const" qualifiers to variables in pcre2grep.
244
245 13. Added Dmitry Cherniachenko's patch for colouring output in Windows
246 (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
247 variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
248
249 14. Add the -t (grand total) option to pcre2grep.
250
251 15. A number of bugs have been mended relating to match start-up optimizations
252 when the first thing in a pattern is a positive lookahead. These all applied
253 only when PCRE2_NO_START_OPTIMIZE was *not* set:
254
255 (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
256 both an initial 'X' and a following 'X'.
257 (b) Some patterns starting with an assertion that started with .* were
258 incorrectly optimized as having to match at the start of the subject or
259 after a newline. There are cases where this is not true, for example,
260 (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
261 start with spaces. Starting .* in an assertion is no longer taken as an
262 indication of matching at the start (or after a newline).
263
264 16. The "offset" modifier in pcre2test was not being ignored (as documented)
265 when the POSIX API was in use.
266
267 17. Added --enable-fuzz-support to "configure", causing an non-installed
268 library containing a test function that can be called by fuzzers to be
269 compiled. A non-installed binary to run the test function locally, called
270 pcre2fuzzcheck is also compiled.
271
272 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
273 which started with .* inside a positive lookahead was incorrectly being
274 compiled as implicitly anchored.
275
276 19. Removed all instances of "register" declarations, as they are considered
277 obsolete these days and in any case had become very haphazard.
278
279 20. Add strerror() to pcre2test for failed file opening.
280
281 21. Make pcre2test -C list valgrind support when it is enabled.
282
283 22. Add the use_length modifier to pcre2test.
284
285 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
286 'copy' modifiers.
287
288 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
289 is apparently needed there as well as in the function definitions. (Why did
290 nobody ask for this in PCRE1?)
291
292 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
293 PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
294 compliant and unique.
295
296 26. pcre2-config --libs-posix was listing -lpcre2posix instead of
297 -lpcre2-posix. Also, the CMake build process was building the library with the
298 wrong name.
299
300 27. In pcre2test, give some offset information for errors in hex patterns.
301 This uses the C99 formatting sequence %td, except for MSVC which doesn't
302 support it - %lu is used instead.
303
304 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
305 pcre2test for testing it.
306
307 29. Fix small memory leak in pcre2test.
308
309 30. Fix out-of-bounds read for partial matching of /./ against an empty string
310 when the newline type is CRLF.
311
312 31. Fix a bug in pcre2test that caused a crash when a locale was set either in
313 the current pattern or a previous one and a wide character was matched.
314
315 32. The appearance of \p, \P, or \X in a substitution string when
316 PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
317 dereference).
318
319 33. If the starting offset was specified as greater than the subject length in
320 a call to pcre2_substitute() an out-of-bounds memory reference could occur.
321
322 34. When PCRE2 was compiled to use the heap instead of the stack for recursive
323 calls to match(), a repeated minimizing caseless back reference, or a
324 maximizing one where the two cases had different numbers of code units,
325 followed by a caseful back reference, could lose the caselessness of the first
326 repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
327 but didn't).
328
329 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
330 matching length and just records zero. Typically this happens when there are
331 too many nested or recursive back references. If the limit was reached in
332 certain recursive cases it failed to be triggered and an internal error could
333 be the result.
334
335 36. The pcre2_dfa_match() function now takes note of the recursion limit for
336 the internal recursive calls that are used for lookrounds and recursions within
337 the pattern.
338
339 37. More refactoring has got rid of the internal could_be_empty_branch()
340 function (around 400 lines of code, including comments) by keeping track of
341 could-be-emptiness as the pattern is compiled instead of scanning compiled
342 groups. (This would have been much harder before the refactoring of #3 above.)
343 This lifts a restriction on the number of branches in a group (more than about
344 1100 would give "pattern is too complicated").
345
346 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
347 auto_callout".
348
349 39. In a library with Unicode support, incorrect data was compiled for a
350 pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
351 characters to match (for example, /[\s[:^ascii:]]/).
352
353 40. The callout_error modifier has been added to pcre2test to make it possible
354 to return PCRE2_ERROR_CALLOUT from a callout.
355
356 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
357 "<esc>[00m".
358
359 42. The limit in the auto-possessification code that was intended to catch
360 overly-complicated patterns and not spend too much time auto-possessifying was
361 being reset too often, resulting in very long compile times for some patterns.
362 Now such patterns are no longer completely auto-possessified.
363
364 43. Applied Jason Hood's revised patch for RunTest.bat.
365
366 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
367
368 45. Minor cosmetic fix to pcre2test: move a variable that is not used under
369 Windows into the "not Windows" code.
370
371 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
372 some of the code:
373
374 * normalised the Windows condition by ensuring WIN32 is defined;
375 * enables the callout feature under Windows;
376 * adds globbing (Microsoft's implementation expands quoted args),
377 using a tweaked opendirectory;
378 * implements the is_*_tty functions for Windows;
379 * --color=always will write the ANSI sequences to file;
380 * add sequences 4 (underline works on Win10) and 5 (blink as bright
381 background, relatively standard on DOS/Win);
382 * remove the (char *) casts for the now-const strings;
383 * remove GREP_COLOUR (grep's command line allowed the 'u', but not
384 the environment), parsing GREP_COLORS instead;
385 * uses the current colour if not set, rather than black;
386 * add print_match for the undefined case;
387 * fixes a typo.
388
389 In addition, colour settings containing anything other than digits and
390 semicolon are ignored, and the colour controls are no longer output for empty
391 strings.
392
393 47. Detecting patterns that are too large inside the length-measuring loop
394 saves processing ridiculously long patterns to their end.
395
396 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
397 just wastes time. In the UTF case it can also produce redundant entries in
398 XCLASS lists caused by characters with multiple other cases and pairs of
399 characters in the same "not-x" sublists.
400
401 49. A pattern such as /(?=(a\K))/ can report the end of the match being before
402 its start; pcre2test was not handling this correctly when using the POSIX
403 interface (it was OK with the native interface).
404
405 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
406 continue to work, falling back to interpretation if anything goes wrong with
407 JIT.
408
409 51. Applied patches from Christian Persch to configure.ac to make use of the
410 AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT
411 modules.
412
413 52. Minor fixes to pcre2grep from Jason Hood:
414 * fixed some spacing;
415 * Windows doesn't usually use single quotes, so I've added a define
416 to use appropriate quotes [in an example];
417 * LC_ALL was displayed as "LCC_ALL";
418 * numbers 11, 12 & 13 should end in "th";
419 * use double quotes in usage message.
420
421 53. When autopossessifying, skip empty branches without recursion, to reduce
422 stack usage for the benefit of clang with -fsanitize-address, which uses huge
423 stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553.
424
425 54. A pattern with very many explicit back references to a group that is a long
426 way from the start of the pattern could take a long time to compile because
427 searching for the referenced group in order to find the minimum length was
428 being done repeatedly. Now up to 128 group minimum lengths are cached and the
429 attempt to find a minimum length is abandoned if there is a back reference to a
430 group whose number is greater than 128. (In that case, the pattern is so
431 complicated that this optimization probably isn't worth it.) This fixes
432 oss-fuzz issue 557.
433
434 55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline
435 mode with --only-matching matched several lines, it restarted scanning at the
436 next line instead of moving on to the end of the matched string, which can be
437 several lines after the start.
438
439 56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line
440 with updates to the non-Windows version.
441
442
443
444 Version 10.22 29-July-2016
445 --------------------------
446
447 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
448 to fix problems with running the tests under Windows.
449
450 2. Implemented a facility for quoting literal characters within hexadecimal
451 patterns in pcre2test, to make it easier to create patterns with just a few
452 non-printing characters.
453
454 3. Binary zeros are not supported in pcre2test input files. It now detects them
455 and gives an error.
456
457 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
458 smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
459 that it matches only unknown objects.
460
461 5. Updated the maintenance script maint/ManyConfigTests to make it easier to
462 select individual groups of tests.
463
464 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
465 used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
466 disables the use of back references (and subroutine calls), which are supported
467 by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
468 longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
469 and pmatch when regexec() is called.
470
471 7. Because of 6 above, pcre2test has been modified with a new modifier called
472 posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
473 modifier had this effect. That option is now ignored when the POSIX API is in
474 use.
475
476 8. Minor tidies to the pcre2demo.c sample program, including more comments
477 about its 8-bit-ness.
478
479 9. Detect unmatched closing parentheses and give the error in the pre-scan
480 instead of later. Previously the pre-scan carried on and could give a
481 misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
482 message about invalid duplicate group names.
483
484 10. It has happened that pcre2test was accidentally linked with another POSIX
485 regex library instead of libpcre2-posix. In this situation, a call to regcomp()
486 (in the other library) may succeed, returning zero, but of course putting its
487 own data into the regex_t block. In one example the re_pcre2_code field was
488 left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
489 so it treated the next line as another pattern line, resulting in a confusing
490 error message. A check has been added to pcre2test to see if the data returned
491 from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
492 are not, an error message is output and the pcre2test run is abandoned. The
493 message points out the possibility of a mis-linking. Hopefully this will avoid
494 some head-scratching the next time this happens.
495
496 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
497 assertion, caused pcre2test to output a very large number of spaces when the
498 callout was taken, making the program appearing to loop.
499
500 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
501 nested set of parentheses of sufficient size caused an overflow of the
502 compiling workspace (which was diagnosed, but of course is not desirable).
503
504 13. Detect missing closing parentheses during the pre-pass for group
505 identification.
506
507 14. Changed some integer variable types and put in a number of casts, following
508 a report of compiler warnings from Visual Studio 2013 and a few tests with
509 gcc's -Wconversion (which still throws up a lot).
510
511 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
512 for testing it.
513
514 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
515 regerror(). When the error buffer is too small, my version of snprintf() puts a
516 binary zero in the final byte. Bug #1801 seems to show that other versions do
517 not do this, leading to bad output from pcre2test when it was checking for
518 buffer overflow. It no longer assumes a binary zero at the end of a too-small
519 regerror() buffer.
520
521 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
522 actually affect anything, by sheer luck.
523
524 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
525 "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
526 older MSVC compilers. This has been done both in src/pcre2_internal.h for most
527 of the library, and also in src/pcre2posix.c, which no longer includes
528 pcre2_internal.h (see 24 below).
529
530 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
531 static compilation. Subsequently applied Chris Wilson's second patch, putting
532 the first patch under a new option instead of being unconditional when
533 PCRE_STATIC is set.
534
535 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
536 to convert \r\n at the ends of reflected lines into \r\r\n. This required
537 ensuring that other output that is written to stdout (e.g. file names) uses the
538 appropriate line terminator: \r\n for Windows, \n otherwise.
539
540 21. When a line is too long for pcre2grep's internal buffer, show the maximum
541 length in the error message.
542
543 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
544 additions).
545
546 23. RunTest.bat was missing a "set type" line for test 22.
547
548 24. The pcre2posix.c file was including pcre2_internal.h, and using some
549 "private" knowledge of the data structures. This is unnecessary; the code has
550 been re-factored and no longer includes pcre2_internal.h.
551
552 25. A racing condition is fixed in JIT reported by Mozilla.
553
554 26. Minor code refactor to avoid "array subscript is below array bounds"
555 compiler warning.
556
557 27. Minor code refactor to avoid "left shift of negative number" warning.
558
559 28. Add a bit more sanity checking to pcre2_serialize_decode() and document
560 that it expects trusted data.
561
562 29. Fix typo in pcre2_jit_test.c
563
564 30. Due to an oversight, pcre2grep was not making use of JIT when available.
565 This is now fixed.
566
567 31. The RunGrepTest script is updated to use the valgrind suppressions file
568 when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
569 file is updated so that is now the same as for PCRE1: it suppresses the
570 Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
571 code). Also changed smc-check=all to smc-check=all-non-file as was done for
572 RunTest (see 4 above).
573
574 32. Implemented the PCRE2_NO_JIT option for pcre2_match().
575
576 33. Fix typo that gave a compiler error when JIT not supported.
577
578 34. Fix comment describing the returns from find_fixedlength().
579
580 35. Fix potential negative index in pcre2test.
581
582 36. Calls to pcre2_get_error_message() with error numbers that are never
583 returned by PCRE2 functions were returning empty strings. Now the error code
584 PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
585 show the texts for given error numbers (i.e. to call pcre2_get_error_message()
586 and display what it returns) and a few representative error codes are now
587 checked in RunTest.
588
589 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
590 pcre2_match.c, in anticipation that this is needed for the same reason it was
591 recently added to pcrecpp.cc in PCRE1.
592
593 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
594 the match extended over a line boundary, as it tried to find more matches "on
595 the same line" - but it was already over the end.
596
597 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
598 to the same code as '.' when PCRE2_DOTALL is set).
599
600 40. Fix two clang compiler warnings in pcre2test when only one code unit width
601 is supported.
602
603 41. Upgrade RunTest to automatically re-run test 2 with a large (64M) stack if
604 it fails when running the interpreter with a 16M stack (and if changing the
605 stack size via pcre2test is possible). This avoids having to manually set a
606 large stack size when testing with clang.
607
608 42. Fix register overwite in JIT when SSE2 acceleration is enabled.
609
610 43. Detect integer overflow in pcre2test pattern and data repetition counts.
611
612 44. In pcre2test, ignore "allcaptures" after DFA matching.
613
614 45. Fix unaligned accesses on x86. Patch by Marc Mutz.
615
616 46. Fix some more clang compiler warnings.
617
618
619 Version 10.21 12-January-2016
620 -----------------------------
621
622 1. Improve matching speed of patterns starting with + or * in JIT.
623
624 2. Use memchr() to find the first character in an unanchored match in 8-bit
625 mode in the interpreter. This gives a significant speed improvement.
626
627 3. Removed a redundant copy of the opcode_possessify table in the
628 pcre2_auto_possessify.c source.
629
630 4. Fix typos in dftables.c for z/OS.
631
632 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
633 processing them could involve a buffer overflow if the following character was
634 an opening parenthesis.
635
636 6. Change 36 for 10.20 also introduced a bug in processing this pattern:
637 /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
638 setting (which (*:0) is), then (?x) did not get unset at the end of its group
639 during the scan for named groups, and hence the external # was incorrectly
640 treated as a comment and the invalid (?' at the end of the pattern was not
641 diagnosed. This caused a buffer overflow during the real compile. This bug was
642 discovered by Karl Skomski with the LLVM fuzzer.
643
644 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
645 own source module to avoid a circular dependency between src/pcre2_compile.c
646 and src/pcre2_study.c
647
648 8. A callout with a string argument containing an opening square bracket, for
649 example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
650 overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
651
652 9. The handling of callouts during the pre-pass for named group identification
653 has been tightened up.
654
655 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
656 possessive. This is a very minor optimization.
657
658 11. A possessively repeated conditional group that could match an empty string,
659 for example, /(?(R))*+/, was incorrectly compiled.
660
661 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
662 Persch).
663
664 13. An empty comment (?#) in a pattern was incorrectly processed and could
665 provoke a buffer overflow. This bug was discovered by Karl Skomski with the
666 LLVM fuzzer.
667
668 14. Fix infinite recursion in the JIT compiler when certain patterns such as
669 /(?:|a|){100}x/ are analysed.
670
671 15. Some patterns with character classes involving [: and \\ were incorrectly
672 compiled and could cause reading from uninitialized memory or an incorrect
673 error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
674 first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
675
676 16. Pathological patterns containing many nested occurrences of [: caused
677 pcre2_compile() to run for a very long time. This bug was found by the LLVM
678 fuzzer.
679
680 17. A missing closing parenthesis for a callout with a string argument was not
681 being diagnosed, possibly leading to a buffer overflow. This bug was found by
682 the LLVM fuzzer.
683
684 18. A conditional group with only one branch has an implicit empty alternative
685 branch and must therefore be treated as potentially matching an empty string.
686
687 19. If (?R was followed by - or + incorrect behaviour happened instead of a
688 diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
689
690 20. Another bug that was introduced by change 36 for 10.20: conditional groups
691 whose condition was an assertion preceded by an explicit callout with a string
692 argument might be incorrectly processed, especially if the string contained \Q.
693 This bug was discovered by Karl Skomski with the LLVM fuzzer.
694
695 21. Compiling PCRE2 with the sanitize options of clang showed up a number of
696 very pedantic coding infelicities and a buffer overflow while checking a UTF-8
697 string if the final multi-byte UTF-8 character was truncated.
698
699 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
700 class, where both values are literal letters in the same case, omit the
701 non-letter EBCDIC code points within the range.
702
703 23. Finding the minimum matching length of complex patterns with back
704 references and/or recursions can take a long time. There is now a cut-off that
705 gives up trying to find a minimum length when things get too complex.
706
707 24. An optimization has been added that speeds up finding the minimum matching
708 length for patterns containing repeated capturing groups or recursions.
709
710 25. If a pattern contained a back reference to a group whose number was
711 duplicated as a result of appearing in a (?|...) group, the computation of the
712 minimum matching length gave a wrong result, which could cause incorrect "no
713 match" errors. For such patterns, a minimum matching length cannot at present
714 be computed.
715
716 26. Added a check for integer overflow in conditions (?(<digits>) and
717 (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
718 fuzzer.
719
720 27. Fixed an issue when \p{Any} inside an xclass did not read the current
721 character.
722
723 28. If pcre2grep was given the -q option with -c or -l, or when handling a
724 binary file, it incorrectly wrote output to stdout.
725
726 29. The JIT compiler did not restore the control verb head in case of *THEN
727 control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
728
729 30. The way recursive references such as (?3) are compiled has been re-written
730 because the old way was the cause of many issues. Now, conversion of the group
731 number into a pattern offset does not happen until the pattern has been
732 completely compiled. This does mean that detection of all infinitely looping
733 recursions is postponed till match time. In the past, some easy ones were
734 detected at compile time. This re-writing was done in response to yet another
735 bug found by the LLVM fuzzer.
736
737 31. A test for a back reference to a non-existent group was missing for items
738 such as \987. This caused incorrect code to be compiled. This issue was found
739 by Karl Skomski with a custom LLVM fuzzer.
740
741 32. Error messages for syntax errors following \g and \k were giving inaccurate
742 offsets in the pattern.
743
744 33. Improve the performance of starting single character repetitions in JIT.
745
746 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
747
748 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
749 give the right offset instead of zero.
750
751 36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
752 This issue was found by Karl Skomski with a custom LLVM fuzzer.
753
754 37. The JIT compiler should restore the control chain for empty possessive
755 repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
756
757 38. A bug which was introduced by the single character repetition optimization
758 was fixed.
759
760 39. Match limit check added to recursion. This issue was found by Karl Skomski
761 with a custom LLVM fuzzer.
762
763 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
764 only at the part of the subject that is relevant when the starting offset is
765 non-zero.
766
767 41. Improve first character match in JIT with SSE2 on x86.
768
769 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
770 with a custom LLVM fuzzer.
771
772 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
773 III).
774
775 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
776 test (there are now 20 in total).
777
778 45. Fixed a corner case of range optimization in JIT.
779
780 46. Add the ${*MARK} facility to pcre2_substitute().
781
782 47. Modifier lists in pcre2test were splitting at spaces without the required
783 commas.
784
785 48. Implemented PCRE2_ALT_VERBNAMES.
786
787 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
788 LLVM fuzzer.
789
790 50. The pcre2test program has been extended by adding the #newline_default
791 command. This has made it possible to run the standard tests when PCRE2 is
792 compiled with either CR or CRLF as the default newline convention. As part of
793 this work, the new command was added to several test files and the testing
794 scripts were modified. The pcre2grep tests can now also be run when there is no
795 LF in the default newline convention.
796
797 51. The RunTest script has been modified so that, when JIT is used and valgrind
798 is specified, a valgrind suppressions file is set up to ignore "Invalid read of
799 size 16" errors because these are false positives when the hardware supports
800 the SSE2 instruction set.
801
802 52. It is now possible to have comment lines amid the subject strings in
803 pcre2test (and perltest.sh) input.
804
805 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
806
807 54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
808 and the matching functions with NULL contexts can be tested.
809
810 55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
811
812 56. In a character class such as [\W\p{Any}] where both a negative-type escape
813 ("not a word character") and a property escape were present, the property
814 escape was being ignored.
815
816 57. Fixed integer overflow for patterns whose minimum matching length is very,
817 very large.
818
819 58. Implemented --never-backslash-C.
820
821 59. Change 55 above introduced a bug by which certain patterns provoked the
822 erroneous error "\ at end of pattern".
823
824 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
825 errors or other strange effects if compiled in UCP mode. Found with libFuzzer
826 and AddressSanitizer.
827
828 61. Whitespace at the end of a pcre2test pattern line caused a spurious error
829 message if there were only single-character modifiers. It should be ignored.
830
831 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
832 or segmentation errors for some patterns. Found with libFuzzer and
833 AddressSanitizer.
834
835 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
836 overflow.
837
838 64. Improve error message for overly-complicated patterns.
839
840 65. Implemented an optional replication feature for patterns in pcre2test, to
841 make it easier to test long repetitive patterns. The tests for 63 above are
842 converted to use the new feature.
843
844 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
845 misbehave.
846
847 67. In pcre2_substitute() in UTF mode, the UTF validity check on the
848 replacement string was happening before the length setting when the replacement
849 string was zero-terminated.
850
851 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the
852 second and subsequent calls to pcre2_match().
853
854 69. There was no check for integer overflow for a replacement group number in
855 pcre2_substitute(). An added check for a number greater than the largest group
856 number in the pattern means this is not now needed.
857
858 70. The PCRE2-specific VERSION condition didn't work correctly if only one
859 digit was given after the decimal point, or if more than two digits were given.
860 It now works with one or two digits, and gives a compile time error if more are
861 given.
862
863 71. In pcre2_substitute() there was the possibility of reading one code unit
864 beyond the end of the replacement string.
865
866 72. The code for checking a subject's UTF-32 validity for a pattern with a
867 lookbehind involved an out-of-bounds pointer, which could potentially cause
868 trouble in some environments.
869
870 73. The maximum lookbehind length was incorrectly calculated for patterns such
871 as /(?<=(a)(?-1))x/ which have a recursion within a backreference.
872
873 74. Give an error if a lookbehind assertion is longer than 65535 code units.
874
875 75. Give an error in pcre2_substitute() if a match ends before it starts (as a
876 result of the use of \K).
877
878 76. Check the length of subpattern names and the names in (*MARK:xx) etc.
879 dynamically to avoid the possibility of integer overflow.
880
881 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the
882 size of patterns that they are prepared to handle.
883
884 78. (*NO_AUTO_POSSESS) was not working.
885
886 79. Adding group information caching improves the speed of compiling when
887 checking whether a group has a fixed length and/or could match an empty string,
888 especially when recursion or subroutine calls are involved. However, this
889 cannot be used when (?| is present in the pattern because the same number may
890 be used for groups of different sizes. To catch runaway patterns in this
891 situation, counts have been introduced to the functions that scan for empty
892 branches or compute fixed lengths.
893
894 80. Allow for the possibility of the size of the nest_save structure not being
895 a factor of the size of the compiling workspace (it currently is).
896
897 81. Check for integer overflow in minimum length calculation and cap it at
898 65535.
899
900 82. Small optimizations in code for finding the minimum matching length.
901
902 83. Lock out configuring for EBCDIC with non-8-bit libraries.
903
904 84. Test for error code <= 0 in regerror().
905
906 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute().
907
908 86. Avoid the possibility of computing with an out-of-bounds pointer (though
909 not dereferencing it) while handling lookbehind assertions.
910
911 87. Failure to get memory for the match data in regcomp() is now given as a
912 regcomp() error instead of waiting for regexec() to pick it up.
913
914 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid
915 newline sequence.
916
917 89. Paranoid check in regcomp() for bad error code from pcre2_compile().
918
919 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well
920 as for link size 2.
921
922 91. Document that JIT has a limit on pattern size, and give more information
923 about JIT compile failures in pcre2test.
924
925 92. Implement PCRE2_INFO_HASBACKSLASHC.
926
927 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports
928 with JIT (possibly caused by SSE2?).
929
930 94. Support offset_limit in JIT.
931
932 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed
933 by a single ASCII character in a class item, was incorrectly compiled in UCP
934 mode. The POSIX class got lost, but only if the single character followed it.
935
936 96. [:punct:] in UCP mode was matching some characters in the range 128-255
937 that should not have been matched.
938
939 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all
940 characters with code points greater than 255 are in the class. When a Unicode
941 property was also in the class (if PCRE2_UCP is set, escapes such as \w are
942 turned into Unicode properties), wide characters were not correctly handled,
943 and could fail to match.
944
945 98. In pcre2test, make the "startoffset" modifier a synonym of "offset",
946 because it sets the "startoffset" parameter for pcre2_match().
947
948 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between
949 an item and its qualifier (for example, A(?#comment)?B) pcre2_compile()
950 misbehaved. This bug was found by the LLVM fuzzer.
951
952 100. The error for an invalid UTF pattern string always gave the code unit
953 offset as zero instead of where the invalidity was found.
954
955 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not
956 working correctly in UCP mode.
957
958 102. Similar to 99 above, if an isolated \E was present between an item and its
959 qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug
960 was found by the LLVM fuzzer.
961
962 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND
963 was set when the pmatch argument was NULL. It now returns REG_INVARG.
964
965 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep.
966
967 105. An empty \Q\E sequence between an item and its qualifier caused
968 pcre2_compile() to misbehave when auto callouts were enabled. This bug
969 was found by the LLVM fuzzer.
970
971 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or
972 other verb "name" ended with whitespace immediately before the closing
973 parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when
974 both those options were set.
975
976 107. In a number of places pcre2_compile() was not handling NULL characters
977 correctly, and pcre2test with the "bincode" modifier was not always correctly
978 displaying fields containing NULLS:
979
980 (a) Within /x extended #-comments
981 (b) Within the "name" part of (*MARK) and other *verbs
982 (c) Within the text argument of a callout
983
984 108. If a pattern that was compiled with PCRE2_EXTENDED started with white
985 space or a #-type comment that was followed by (?-x), which turns off
986 PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again,
987 pcre2_compile() assumed that (?-x) applied to the whole pattern and
988 consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix
989 for this bug means that a setting of any of the (?imsxJU) options at the start
990 of a pattern is no longer transferred to the options that are returned by
991 PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have
992 changed when the effects of those options were all moved to compile time.
993
994 109. An escaped closing parenthesis in the "name" part of a (*verb) when
995 PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug
996 was found by the LLVM fuzzer.
997
998 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it
999 possible to test it.
1000
1001 111. "Harden" pcre2test against ridiculously large values in modifiers and
1002 command line arguments.
1003
1004 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_
1005 LENGTH.
1006
1007 113. Fix printing of *MARK names that contain binary zeroes in pcre2test.
1008
1009
1010 Version 10.20 30-June-2015
1011 --------------------------
1012
1013 1. Callouts with string arguments have been added.
1014
1015 2. Assertion code generator in JIT has been optimized.
1016
1017 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
1018 pcre2_compile() function read past the end of the input before diagnosing an
1019 error. This bug was discovered by the LLVM fuzzer.
1020
1021 4. Implemented pcre2_callout_enumerate().
1022
1023 5. Fix JIT compilation of conditional blocks whose assertion is converted to
1024 (*FAIL). E.g: /(?(?!))/.
1025
1026 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
1027 discovered by the LLVM fuzzer.
1028
1029 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
1030 when this assertion was used as a condition, for example (?(?!)a|b). In
1031 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
1032 error about an unsupported item.
1033
1034 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
1035 possessification code could take exponential time to complete. A recursion
1036 depth limit of 1000 has been imposed to limit the resources used by this
1037 optimization. This infelicity was discovered by the LLVM fuzzer.
1038
1039 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
1040 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
1041 because \S ensures they are all in the class. The code for doing this was
1042 interacting badly with the code for computing the amount of space needed to
1043 compile the pattern, leading to a buffer overflow. This bug was discovered by
1044 the LLVM fuzzer.
1045
1046 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
1047 other kinds of group caused stack overflow at compile time. This bug was
1048 discovered by the LLVM fuzzer.
1049
1050 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment
1051 between a subroutine call and its quantifier was incorrectly compiled, leading
1052 to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer.
1053
1054 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an
1055 assertion after (?(. The code was failing to check the character after (?(?<
1056 for the ! or = that would indicate a lookbehind assertion. This bug was
1057 discovered by the LLVM fuzzer.
1058
1059 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with
1060 a fixed maximum following a group that contains a subroutine reference was
1061 incorrectly compiled and could trigger buffer overflow. This bug was discovered
1062 by the LLVM fuzzer.
1063
1064 14. Negative relative recursive references such as (?-7) to non-existent
1065 subpatterns were not being diagnosed and could lead to unpredictable behaviour.
1066 This bug was discovered by the LLVM fuzzer.
1067
1068 15. The bug fixed in 14 was due to an integer variable that was unsigned when
1069 it should have been signed. Some other "int" variables, having been checked,
1070 have either been changed to uint32_t or commented as "must be signed".
1071
1072 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1)))
1073 caused a stack overflow instead of the diagnosis of a non-fixed length
1074 lookbehind assertion. This bug was discovered by the LLVM fuzzer.
1075
1076 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern
1077 (e.g. /(?<=\Ka)/) could make pcre2grep loop.
1078
1079 18. There was a similar problem to 17 in pcre2test for global matches, though
1080 the code there did catch the loop.
1081
1082 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*),
1083 and a subsequent item in the pattern caused a non-match, backtracking over the
1084 repeated \X did not stop, but carried on past the start of the subject, causing
1085 reference to random memory and/or a segfault. There were also some other cases
1086 where backtracking after \C could crash. This set of bugs was discovered by the
1087 LLVM fuzzer.
1088
1089 20. The function for finding the minimum length of a matching string could take
1090 a very long time if mutual recursion was present many times in a pattern, for
1091 example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has
1092 been implemented. This infelicity was discovered by the LLVM fuzzer.
1093
1094 21. Implemented PCRE2_NEVER_BACKSLASH_C.
1095
1096 22. The feature for string replication in pcre2test could read from freed
1097 memory if the replication required a buffer to be extended, and it was not
1098 working properly in 16-bit and 32-bit modes. This issue was discovered by a
1099 fuzzer: see http://lcamtuf.coredump.cx/afl/.
1100
1101 23. Added the PCRE2_ALT_CIRCUMFLEX option.
1102
1103 24. Adjust the treatment of \8 and \9 to be the same as the current Perl
1104 behaviour.
1105
1106 25. Static linking against the PCRE2 library using the pkg-config module was
1107 failing on missing pthread symbols.
1108
1109 26. If a group that contained a recursive back reference also contained a
1110 forward reference subroutine call followed by a non-forward-reference
1111 subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
1112 compile correct code, leading to undefined behaviour or an internally detected
1113 error. This bug was discovered by the LLVM fuzzer.
1114
1115 27. Quantification of certain items (e.g. atomic back references) could cause
1116 incorrect code to be compiled when recursive forward references were involved.
1117 For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was
1118 discovered by the LLVM fuzzer.
1119
1120 28. A repeated conditional group whose condition was a reference by name caused
1121 a buffer overflow if there was more than one group with the given name. This
1122 bug was discovered by the LLVM fuzzer.
1123
1124 29. A recursive back reference by name within a group that had the same name as
1125 another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/.
1126 This bug was discovered by the LLVM fuzzer.
1127
1128 30. A forward reference by name to a group whose number is the same as the
1129 current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a
1130 buffer overflow at compile time. This bug was discovered by the LLVM fuzzer.
1131
1132 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1
1133 as an int; fixed by writing it as 1u).
1134
1135 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives
1136 a warning for "fileno" unless -std=gnu99 us used.
1137
1138 33. A lookbehind assertion within a set of mutually recursive subpatterns could
1139 provoke a buffer overflow. This bug was discovered by the LLVM fuzzer.
1140
1141 34. Give an error for an empty subpattern name such as (?'').
1142
1143 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains
1144 \P, \p, or \X.
1145
1146 36. The way named subpatterns are handled has been refactored. There is now a
1147 pre-pass over the regex which does nothing other than identify named
1148 subpatterns and count the total captures. This means that information about
1149 named patterns is known before the rest of the compile. In particular, it means
1150 that forward references can be checked as they are encountered. Previously, the
1151 code for handling forward references was contorted and led to several errors in
1152 computing the memory requirements for some patterns, leading to buffer
1153 overflows.
1154
1155 37. There was no check for integer overflow in subroutine calls such as (?123).
1156
1157 38. The table entry for \l in EBCDIC environments was incorrect, leading to its
1158 being treated as a literal 'l' instead of causing an error.
1159
1160 39. If a non-capturing group containing a conditional group that could match
1161 an empty string was repeated, it was not identified as matching an empty string
1162 itself. For example: /^(?:(?(1)x|)+)+$()/.
1163
1164 40. In an EBCDIC environment, pcretest was mishandling the escape sequences
1165 \a and \e in test subject lines.
1166
1167 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII
1168 instead of the EBCDIC value.
1169
1170 42. The handling of \c in an EBCDIC environment has been revised so that it is
1171 now compatible with the specification in Perl's perlebcdic page.
1172
1173 43. Single character repetition in JIT has been improved. 20-30% speedup
1174 was achieved on certain patterns.
1175
1176 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in
1177 ASCII/Unicode. This has now been added to the list of characters that are
1178 recognized as white space in EBCDIC.
1179
1180 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave
1181 an error (correctly) when used outside a class, but did not give an error
1182 within a class.
1183
1184 46. \h within a class was incorrectly compiled in EBCDIC environments.
1185
1186 47. JIT should return with error when the compiled pattern requires
1187 more stack space than the maximum.
1188
1189 48. Fixed a memory leak in pcre2grep when a locale is set.
1190
1191
1192 Version 10.10 06-March-2015
1193 ---------------------------
1194
1195 1. When a pattern is compiled, it remembers the highest back reference so that
1196 when matching, if the ovector is too small, extra memory can be obtained to
1197 use instead. A conditional subpattern whose condition is a check on a capture
1198 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
1199 another kind of back reference, but it was not setting the highest
1200 backreference number. This mattered only if pcre2_match() was called with an
1201 ovector that was too small to hold the capture, and there was no other kind of
1202 back reference (a situation which is probably quite rare). The effect of the
1203 bug was that the condition was always treated as FALSE when the capture could
1204 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
1205 has been fixed.
1206
1207 2. Functions for serialization and deserialization of sets of compiled patterns
1208 have been added.
1209
1210 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
1211 excess code units at the end of the data block that may occasionally occur if
1212 the code for calculating the size over-estimates. This change stops the
1213 serialization code copying uninitialized data, to which valgrind objects. The
1214 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
1215 include the general overhead. This has been corrected.
1216
1217 4. All code units in every slot in the table of group names are now set, again
1218 in order to avoid accessing uninitialized data when serializing.
1219
1220 5. The (*NO_JIT) feature is implemented.
1221
1222 6. If a bug that caused pcre2_compile() to use more memory than allocated was
1223 triggered when using valgrind, the code in (3) above passed a stupidly large
1224 value to valgrind. This caused a crash instead of an "internal error" return.
1225
1226 7. A reference to a duplicated named group (either a back reference or a test
1227 for being set in a conditional) that occurred in a part of the pattern where
1228 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
1229 to be incorrectly calculated, leading to overwriting.
1230
1231 8. A mutually recursive set of back references such as (\2)(\1) caused a
1232 segfault at compile time (while trying to find the minimum matching length).
1233 The infinite loop is now broken (with the minimum length unset, that is, zero).
1234
1235 9. If an assertion that was used as a condition was quantified with a minimum
1236 of zero, matching went wrong. In particular, if the whole group had unlimited
1237 repetition and could match an empty string, a segfault was likely. The pattern
1238 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
1239 quantified, but not if they are being used as conditions, so the above pattern
1240 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
1241 patterns.
1242
1243 10. The error message for an invalid quantifier has been changed from "nothing
1244 to repeat" to "quantifier does not follow a repeatable item".
1245
1246 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
1247 scanning the compiled pattern in subsequent auto-possessification can get out
1248 of step and lead to an unknown opcode. Previously this could have caused an
1249 infinite loop. Now it generates an "internal error" error. This is a tidyup,
1250 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
1251 undefined outcome.
1252
1253 12. A UTF pattern containing a "not" match of a non-ASCII character and a
1254 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
1255
1256 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
1257 that is found in the output of "locale -a" can actually be set by pcre2test
1258 before it is accepted. Previously, in an environment where a locale was listed
1259 but would not set (an example does exist), the test would "pass" without
1260 actually doing anything. Also the fr_CA locale has been added to the list of
1261 locales that can be used.
1262
1263 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
1264 capturing group number without parentheses, the last character was incorrectly
1265 literally included at the end of the replacement string.
1266
1267 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
1268 failed to allow the zero-repeat case if pcre2_match() was called with an
1269 ovector too small to capture the group.
1270
1271 16. Improved error message in pcre2test when setting the stack size (-S) fails.
1272
1273 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
1274 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
1275 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
1276 of PCRE2 sources, which caused a failure to build pcre2test.
1277
1278 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
1279 only on Windows.
1280
1281 19. Use binary input when reading back saved serialized patterns in pcre2test.
1282
1283 20. Added RunTest.bat for running the tests under Windows.
1284
1285 21. "make distclean" was not removing config.h, a file that may be created for
1286 use with CMake.
1287
1288 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
1289 forward reference repeated a large (but limited) number of times within a
1290 repeated outer group that has a zero minimum quantifier, caused incorrect code
1291 to be compiled, leading to the error "internal error: previously-checked
1292 referenced subpattern not found" when an incorrect memory address was read.
1293 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
1294 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
1295
1296 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
1297 call within a group that also contained a recursive back reference caused
1298 incorrect code to be compiled. This bug was reported as "heap overflow",
1299 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
1300 CVE-2015-2326 was given to this.)
1301
1302 24. Computing the size of the JIT read-only data in advance has been a source
1303 of various issues, and new ones are still appear unfortunately. To fix
1304 existing and future issues, size computation is eliminated from the code,
1305 and replaced by on-demand memory allocation.
1306
1307 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
1308 adjacent to the end of the range, and the range contained characters with more
1309 than one other case, caused incorrect behaviour when compiled in UTF mode. In
1310 that example, the range a-j was left out of the class.
1311
1312
1313 Version 10.00 05-January-2015
1314 -----------------------------
1315
1316 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
1317 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
1318 API, up to item 20 for release 8.36.
1319
1320 The code of the library was heavily revised as part of the new API
1321 implementation. Details of each and every modification were not individually
1322 logged. In addition to the API changes, the following changes were made. They
1323 are either new functionality, or bug fixes and other noticeable changes of
1324 behaviour that were implemented after the code had been forked.
1325
1326 1. Including Unicode support at build time is now enabled by default, but it
1327 can optionally be disabled. It is not enabled by default at run time (no
1328 change).
1329
1330 2. The test program, now called pcre2test, was re-specified and almost
1331 completely re-written. Its input is not compatible with input for pcretest.
1332
1333 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
1334 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
1335 matched by that pattern.
1336
1337 4. For the benefit of those who use PCRE2 via some other application, that is,
1338 not writing the function calls themselves, it is possible to check the PCRE2
1339 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
1340 string such as "yesno".
1341
1342 5. There are case-equivalent Unicode characters whose encodings use different
1343 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
1344 theoretically possible for this to happen in UTF-16 too.) If a backreference to
1345 a group containing one of these characters was greedily repeated, and during
1346 the match a backtrack occurred, the subject might be backtracked by the wrong
1347 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
1348 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
1349 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
1350 Incorrect backtracking meant that group 2 captured only the last two bytes.
1351 This bug has been fixed; the new code is slower, but it is used only when the
1352 strings matched by the repetition are not all the same length.
1353
1354 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
1355 information. This applied to any pattern with a group that matched no
1356 characters, for example: /(?:(?=.)|(?<!x))a/.
1357
1358 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
1359 those parentheses to be closed with whatever has been captured so far. However,
1360 it was failing to mark any other groups between the highest capture so far and
1361 the currrent group as "unset". Thus, the ovector for those groups contained
1362 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
1363 matched against "abcd".
1364
1365 8. The pcre2_substitute() function has been implemented.
1366
1367 9. If an assertion used as a condition was quantified with a minimum of zero
1368 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
1369 occur.
1370
1371 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.
1372
1373 ****

  ViewVC Help
Powered by ViewVC 1.1.5