/[pcre2]/code/trunk/ChangeLog
ViewVC logotype

Contents of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log


Revision 651 - (show annotations)
Wed Jan 11 17:10:28 2017 UTC (4 days, 22 hours ago) by ph10
File size: 58630 byte(s)
Ignore all JIT compile errors in pcre2grep.
1 Change Log for PCRE2
2 --------------------
3
4
5 Version 10.23 xx-xxxxxx-2016
6 ----------------------------
7
8 1. Extended pcre2test with the utf8_input modifier so that it is able to
9 generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
10
11 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
12 PCRE2_UCP set, a negative character type such as \D in a positive class should
13 cause all characters greater than 255 to match, whatever else is in the class.
14 There was a bug that caused this not to happen if a Unicode property item was
15 added to such a class, for example [\D\P{Nd}] or [\W\pL].
16
17 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
18 checking is now done in the pre-pass that identifies capturing groups. This has
19 reduced the amount of duplication and made the code tidier. While doing this,
20 some minor bugs and Perl incompatibilities were fixed, including:
21
22 (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
23 of giving an invalid quantifier error.
24
25 (b) {0} can now be used after a group in a lookbehind assertion; previously
26 this caused an "assertion is not fixed length" error.
27
28 (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
29 the name "DEFINE" exists. PCRE2 now does likewise.
30
31 (d) A recursion condition test such as (?(R2)...) must now refer to an
32 existing subpattern.
33
34 (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
35 group whose name began with "R".
36
37 (f) When testing zero-terminated patterns under valgrind, the terminating
38 zero is now marked "no access". This catches bugs that would otherwise
39 show up only with non-zero-terminated patterns.
40
41 (g) A hyphen appearing immediately after a POSIX character class (for example
42 /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
43 literal, but gives a warning, so it seems best to fail it in PCRE.
44
45 (h) An empty \Q\E sequence may appear after a callout that precedes an
46 assertion condition (it is, of course, ignored).
47
48 One effect of the refactoring is that some error numbers and messages have
49 changed, and the pattern offset given for compiling errors is not always the
50 right-most character that has been read. In particular, for a variable-length
51 lookbehind assertion it now points to the start of the assertion. Another
52 change is that when a callout appears before a group, the "length of next
53 pattern item" that is passed now just gives the length of the opening
54 parenthesis item, not the length of the whole group. A length of zero is now
55 given only for a callout at the end of the pattern. Automatic callouts are no
56 longer inserted before and after explicit callouts in the pattern.
57
58 A number of bugs in the refactored code were subsequently fixed during testing
59 before release, but after the code was made available in the repository. Many
60 of the bugs were discovered by fuzzing testing. Several of them were related to
61 the change from assuming a zero-terminated pattern (which previously had
62 required non-zero terminated strings to be copied). These bugs were never in
63 fully released code, but are noted here for the record.
64
65 (a) An overall recursion such as (?0) inside a lookbehind assertion was not
66 being diagnosed as an error.
67
68 (b) In utf mode, the length of a *MARK (or other verb) name was being checked
69 in characters instead of code units, which could lead to bad code being
70 compiled, leading to unpredictable behaviour.
71
72 (c) In extended /x mode, characters whose code was greater than 255 caused
73 a lookup outside one of the global tables. A similar bug existed for wide
74 characters in *VERB names.
75
76 (d) The amount of memory needed for a compiled pattern was miscalculated if a
77 lookbehind contained more than one toplevel branch and the first branch
78 was of length zero.
79
80 (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
81 terminated pattern, if a # comment ran on to the end of the pattern, one
82 or more code units past the end were being read.
83
84 (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
85 "{2,2") could cause reading beyond the pattern.
86
87 (g) When reading a callout string, if the end delimiter was at the end of the
88 pattern one further code unit was read.
89
90 (h) An unterminated number after \g' could cause reading beyond the pattern.
91
92 (i) An insufficient memory size was being computed for compiling with
93 PCRE2_AUTO_CALLOUT.
94
95 (j) A conditional group with an assertion condition used more memory than was
96 allowed for it during parsing, so too many of them could therefore
97 overrun a buffer.
98
99 (k) If parsing a pattern exactly filled the buffer, the internal test for
100 overrun did not check when the final META_END item was added.
101
102 (l) If a lookbehind contained a subroutine call, and the called group
103 contained an option setting such as (?s), and the PCRE2_ANCHORED option
104 was set, unpredictable behaviour could occur. The underlying bug was
105 incorrect code and insufficient checking while searching for the end of
106 the called subroutine in the parsed pattern.
107
108 (m) Quantifiers following (*VERB)s were not being diagnosed as errors.
109
110 (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
111 PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
112
113 (o) If \Q was preceded by a quantified item, and the following \E was
114 followed by '?' or '+', and there was at least one literal character
115 between them, an internal error "unexpected repeat" occurred (example:
116 /.+\QX\E+/).
117
118 (p) A buffer overflow could occur while sorting the names in the group name
119 list (depending on the order in which the names were seen).
120
121 (q) A conditional group that started with a callout was not doing the right
122 check for a following assertion, leading to compiling bad code. Example:
123 /(?(C'XX))?!XX/
124
125 (r) If a character whose code point was greater than 0xffff appeared within
126 a lookbehind that was within another lookbehind, the calculation of the
127 lookbehind length went wrong and could provoke an internal error.
128
129 (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
130 an internal error. Now the hyphen is treated as a literal.
131
132 4. Back references are now permitted in lookbehind assertions when there are
133 no duplicated group numbers (that is, (?| has not been used), and, if the
134 reference is by name, there is only one group of that name. The referenced
135 group must, of course be of fixed length.
136
137 5. pcre2test has been upgraded so that, when run under valgrind with valgrind
138 support enabled, reading past the end of the pattern is detected, both when
139 compiling and during callout processing.
140
141 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
142 reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
143 not recognize this syntax.
144
145 7. Automatic callouts are no longer generated before and after callouts in the
146 pattern.
147
148 8. When pcre2test was outputing information from a callout, the caret indicator
149 for the current position in the subject line was incorrect if it was after an
150 escape sequence for a character whose code point was greater than \x{ff}.
151
152 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
153 PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
154
155 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
156 expansion when long lines are encountered. Original patch by Dmitry
157 Cherniachenko.
158
159 11. If pcre2grep was compiled with JIT support, but the library was compiled
160 without it (something that neither ./configure nor CMake allow, but it can be
161 done by editing config.h), pcre2grep was giving a JIT error. Now it detects
162 this situation and does not try to use JIT.
163
164 12. Added some "const" qualifiers to variables in pcre2grep.
165
166 13. Added Dmitry Cherniachenko's patch for colouring output in Windows
167 (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
168 variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
169
170 14. Add the -t (grand total) option to pcre2grep.
171
172 15. A number of bugs have been mended relating to match start-up optimizations
173 when the first thing in a pattern is a positive lookahead. These all applied
174 only when PCRE2_NO_START_OPTIMIZE was *not* set:
175
176 (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
177 both an initial 'X' and a following 'X'.
178 (b) Some patterns starting with an assertion that started with .* were
179 incorrectly optimized as having to match at the start of the subject or
180 after a newline. There are cases where this is not true, for example,
181 (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
182 start with spaces. Starting .* in an assertion is no longer taken as an
183 indication of matching at the start (or after a newline).
184
185 16. The "offset" modifier in pcre2test was not being ignored (as documented)
186 when the POSIX API was in use.
187
188 17. Added --enable-fuzz-support to "configure", causing an non-installed
189 library containing a test function that can be called by fuzzers to be
190 compiled. A non-installed binary to run the test function locally, called
191 pcre2fuzzcheck is also compiled.
192
193 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
194 which started with .* inside a positive lookahead was incorrectly being
195 compiled as implicitly anchored.
196
197 19. Removed all instances of "register" declarations, as they are considered
198 obsolete these days and in any case had become very haphazard.
199
200 20. Add strerror() to pcre2test for failed file opening.
201
202 21. Make pcre2test -C list valgrind support when it is enabled.
203
204 22. Add the use_length modifier to pcre2test.
205
206 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
207 'copy' modifiers.
208
209 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
210 is apparently needed there as well as in the function definitions. (Why did
211 nobody ask for this in PCRE1?)
212
213 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
214 PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
215 compliant and unique.
216
217 26. pcre2-config --libs-posix was listing -lpcre2posix instead of
218 -lpcre2-posix. Also, the CMake build process was building the library with the
219 wrong name.
220
221 27. In pcre2test, give some offset information for errors in hex patterns.
222
223 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
224 pcre2test for testing it.
225
226 29. Fix small memory leak in pcre2test.
227
228 30. Fix out-of-bounds read for partial matching of /./ against an empty string
229 when the newline type is CRLF.
230
231 31. Fix a bug in pcre2test that caused a crash when a locale was set either in
232 the current pattern or a previous one and a wide character was matched.
233
234 32. The appearance of \p, \P, or \X in a substitution string when
235 PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
236 dereference).
237
238 33. If the starting offset was specified as greater than the subject length in
239 a call to pcre2_substitute() an out-of-bounds memory reference could occur.
240
241 34. When PCRE2 was compiled to use the heap instead of the stack for recursive
242 calls to match(), a repeated minimizing caseless back reference, or a
243 maximizing one where the two cases had different numbers of code units,
244 followed by a caseful back reference, could lose the caselessness of the first
245 repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
246 but didn't).
247
248 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
249 matching length and just records zero. Typically this happens when there are
250 too many nested or recursive back references. If the limit was reached in
251 certain recursive cases it failed to be triggered and an internal error could
252 be the result.
253
254 36. The pcre2_dfa_match() function now takes note of the recursion limit for
255 the internal recursive calls that are used for lookrounds and recursions within
256 the pattern.
257
258 37. More refactoring has got rid of the internal could_be_empty_branch()
259 function (around 400 lines of code, including comments) by keeping track of
260 could-be-emptiness as the pattern is compiled instead of scanning compiled
261 groups. (This would have been much harder before the refactoring of #3 above.)
262 This lifts a restriction on the number of branches in a group (more than about
263 1100 would give "pattern is too complicated").
264
265 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
266 auto_callout".
267
268 39. In a library with Unicode support, incorrect data was compiled for a
269 pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
270 characters to match (for example, /[\s[:^ascii:]]/).
271
272 40. The callout_error modifier has been added to pcre2test to make it possible
273 to return PCRE2_ERROR_CALLOUT from a callout.
274
275 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
276 "<esc>[00m".
277
278 42. The limit in the auto-possessification code that was intended to catch
279 overly-complicated patterns and not spend too much time auto-possessifying was
280 being reset too often, resulting in very long compile times for some patterns.
281 Now such patterns are no longer completely auto-possessified.
282
283 43. Applied Jason Hood's revised patch for RunTest.bat.
284
285 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
286
287 45. Minor cosmetic fix to pcre2test: move a variable that is not used under
288 Windows into the "not Windows" code.
289
290 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
291 some of the code:
292
293 * normalised the Windows condition by ensuring WIN32 is defined;
294 * enables the callout feature under Windows;
295 * adds globbing (Microsoft's implementation expands quoted args),
296 using a tweaked opendirectory;
297 * implements the is_*_tty functions for Windows;
298 * --color=always will write the ANSI sequences to file;
299 * add sequences 4 (underline works on Win10) and 5 (blink as bright
300 background, relatively standard on DOS/Win);
301 * remove the (char *) casts for the now-const strings;
302 * remove GREP_COLOUR (grep's command line allowed the 'u', but not
303 the environment), parsing GREP_COLORS instead;
304 * uses the current colour if not set, rather than black;
305 * add print_match for the undefined case;
306 * fixes a typo.
307
308 In addition, colour settings containing anything other than digits and
309 semicolon are ignored, and the colour controls are no longer output for empty
310 strings.
311
312 47. Detecting patterns that are too large inside the length-measuring loop
313 saves processing ridiculously long patterns to their end.
314
315 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
316 just wastes time. In the UTF case it can also produce redundant entries in
317 XCLASS lists caused by characters with multiple other cases and pairs of
318 characters in the same "not-x" sublists.
319
320 49. A pattern such as /(?=(a\K))/ can report the end of the match being before
321 its start; pcre2test was not handling this correctly when using the POSIX
322 interface (it was OK with the native interface).
323
324 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
325 continue to work, falling back to interpretation if anything goes wrong with
326 JIT.
327
328
329 Version 10.22 29-July-2016
330 --------------------------
331
332 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
333 to fix problems with running the tests under Windows.
334
335 2. Implemented a facility for quoting literal characters within hexadecimal
336 patterns in pcre2test, to make it easier to create patterns with just a few
337 non-printing characters.
338
339 3. Binary zeros are not supported in pcre2test input files. It now detects them
340 and gives an error.
341
342 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
343 smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
344 that it matches only unknown objects.
345
346 5. Updated the maintenance script maint/ManyConfigTests to make it easier to
347 select individual groups of tests.
348
349 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
350 used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
351 disables the use of back references (and subroutine calls), which are supported
352 by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
353 longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
354 and pmatch when regexec() is called.
355
356 7. Because of 6 above, pcre2test has been modified with a new modifier called
357 posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
358 modifier had this effect. That option is now ignored when the POSIX API is in
359 use.
360
361 8. Minor tidies to the pcre2demo.c sample program, including more comments
362 about its 8-bit-ness.
363
364 9. Detect unmatched closing parentheses and give the error in the pre-scan
365 instead of later. Previously the pre-scan carried on and could give a
366 misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
367 message about invalid duplicate group names.
368
369 10. It has happened that pcre2test was accidentally linked with another POSIX
370 regex library instead of libpcre2-posix. In this situation, a call to regcomp()
371 (in the other library) may succeed, returning zero, but of course putting its
372 own data into the regex_t block. In one example the re_pcre2_code field was
373 left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
374 so it treated the next line as another pattern line, resulting in a confusing
375 error message. A check has been added to pcre2test to see if the data returned
376 from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
377 are not, an error message is output and the pcre2test run is abandoned. The
378 message points out the possibility of a mis-linking. Hopefully this will avoid
379 some head-scratching the next time this happens.
380
381 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
382 assertion, caused pcre2test to output a very large number of spaces when the
383 callout was taken, making the program appearing to loop.
384
385 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
386 nested set of parentheses of sufficient size caused an overflow of the
387 compiling workspace (which was diagnosed, but of course is not desirable).
388
389 13. Detect missing closing parentheses during the pre-pass for group
390 identification.
391
392 14. Changed some integer variable types and put in a number of casts, following
393 a report of compiler warnings from Visual Studio 2013 and a few tests with
394 gcc's -Wconversion (which still throws up a lot).
395
396 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
397 for testing it.
398
399 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
400 regerror(). When the error buffer is too small, my version of snprintf() puts a
401 binary zero in the final byte. Bug #1801 seems to show that other versions do
402 not do this, leading to bad output from pcre2test when it was checking for
403 buffer overflow. It no longer assumes a binary zero at the end of a too-small
404 regerror() buffer.
405
406 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
407 actually affect anything, by sheer luck.
408
409 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
410 "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
411 older MSVC compilers. This has been done both in src/pcre2_internal.h for most
412 of the library, and also in src/pcre2posix.c, which no longer includes
413 pcre2_internal.h (see 24 below).
414
415 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
416 static compilation. Subsequently applied Chris Wilson's second patch, putting
417 the first patch under a new option instead of being unconditional when
418 PCRE_STATIC is set.
419
420 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
421 to convert \r\n at the ends of reflected lines into \r\r\n. This required
422 ensuring that other output that is written to stdout (e.g. file names) uses the
423 appropriate line terminator: \r\n for Windows, \n otherwise.
424
425 21. When a line is too long for pcre2grep's internal buffer, show the maximum
426 length in the error message.
427
428 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
429 additions).
430
431 23. RunTest.bat was missing a "set type" line for test 22.
432
433 24. The pcre2posix.c file was including pcre2_internal.h, and using some
434 "private" knowledge of the data structures. This is unnecessary; the code has
435 been re-factored and no longer includes pcre2_internal.h.
436
437 25. A racing condition is fixed in JIT reported by Mozilla.
438
439 26. Minor code refactor to avoid "array subscript is below array bounds"
440 compiler warning.
441
442 27. Minor code refactor to avoid "left shift of negative number" warning.
443
444 28. Add a bit more sanity checking to pcre2_serialize_decode() and document
445 that it expects trusted data.
446
447 29. Fix typo in pcre2_jit_test.c
448
449 30. Due to an oversight, pcre2grep was not making use of JIT when available.
450 This is now fixed.
451
452 31. The RunGrepTest script is updated to use the valgrind suppressions file
453 when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
454 file is updated so that is now the same as for PCRE1: it suppresses the
455 Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
456 code). Also changed smc-check=all to smc-check=all-non-file as was done for
457 RunTest (see 4 above).
458
459 32. Implemented the PCRE2_NO_JIT option for pcre2_match().
460
461 33. Fix typo that gave a compiler error when JIT not supported.
462
463 34. Fix comment describing the returns from find_fixedlength().
464
465 35. Fix potential negative index in pcre2test.
466
467 36. Calls to pcre2_get_error_message() with error numbers that are never
468 returned by PCRE2 functions were returning empty strings. Now the error code
469 PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
470 show the texts for given error numbers (i.e. to call pcre2_get_error_message()
471 and display what it returns) and a few representative error codes are now
472 checked in RunTest.
473
474 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
475 pcre2_match.c, in anticipation that this is needed for the same reason it was
476 recently added to pcrecpp.cc in PCRE1.
477
478 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
479 the match extended over a line boundary, as it tried to find more matches "on
480 the same line" - but it was already over the end.
481
482 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
483 to the same code as '.' when PCRE2_DOTALL is set).
484
485 40. Fix two clang compiler warnings in pcre2test when only one code unit width
486 is supported.
487
488 41. Upgrade RunTest to automatically re-run test 2 with a large (64M) stack if
489 it fails when running the interpreter with a 16M stack (and if changing the
490 stack size via pcre2test is possible). This avoids having to manually set a
491 large stack size when testing with clang.
492
493 42. Fix register overwite in JIT when SSE2 acceleration is enabled.
494
495 43. Detect integer overflow in pcre2test pattern and data repetition counts.
496
497 44. In pcre2test, ignore "allcaptures" after DFA matching.
498
499 45. Fix unaligned accesses on x86. Patch by Marc Mutz.
500
501 46. Fix some more clang compiler warnings.
502
503
504 Version 10.21 12-January-2016
505 -----------------------------
506
507 1. Improve matching speed of patterns starting with + or * in JIT.
508
509 2. Use memchr() to find the first character in an unanchored match in 8-bit
510 mode in the interpreter. This gives a significant speed improvement.
511
512 3. Removed a redundant copy of the opcode_possessify table in the
513 pcre2_auto_possessify.c source.
514
515 4. Fix typos in dftables.c for z/OS.
516
517 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
518 processing them could involve a buffer overflow if the following character was
519 an opening parenthesis.
520
521 6. Change 36 for 10.20 also introduced a bug in processing this pattern:
522 /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
523 setting (which (*:0) is), then (?x) did not get unset at the end of its group
524 during the scan for named groups, and hence the external # was incorrectly
525 treated as a comment and the invalid (?' at the end of the pattern was not
526 diagnosed. This caused a buffer overflow during the real compile. This bug was
527 discovered by Karl Skomski with the LLVM fuzzer.
528
529 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
530 own source module to avoid a circular dependency between src/pcre2_compile.c
531 and src/pcre2_study.c
532
533 8. A callout with a string argument containing an opening square bracket, for
534 example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
535 overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
536
537 9. The handling of callouts during the pre-pass for named group identification
538 has been tightened up.
539
540 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
541 possessive. This is a very minor optimization.
542
543 11. A possessively repeated conditional group that could match an empty string,
544 for example, /(?(R))*+/, was incorrectly compiled.
545
546 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
547 Persch).
548
549 13. An empty comment (?#) in a pattern was incorrectly processed and could
550 provoke a buffer overflow. This bug was discovered by Karl Skomski with the
551 LLVM fuzzer.
552
553 14. Fix infinite recursion in the JIT compiler when certain patterns such as
554 /(?:|a|){100}x/ are analysed.
555
556 15. Some patterns with character classes involving [: and \\ were incorrectly
557 compiled and could cause reading from uninitialized memory or an incorrect
558 error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
559 first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
560
561 16. Pathological patterns containing many nested occurrences of [: caused
562 pcre2_compile() to run for a very long time. This bug was found by the LLVM
563 fuzzer.
564
565 17. A missing closing parenthesis for a callout with a string argument was not
566 being diagnosed, possibly leading to a buffer overflow. This bug was found by
567 the LLVM fuzzer.
568
569 18. A conditional group with only one branch has an implicit empty alternative
570 branch and must therefore be treated as potentially matching an empty string.
571
572 19. If (?R was followed by - or + incorrect behaviour happened instead of a
573 diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
574
575 20. Another bug that was introduced by change 36 for 10.20: conditional groups
576 whose condition was an assertion preceded by an explicit callout with a string
577 argument might be incorrectly processed, especially if the string contained \Q.
578 This bug was discovered by Karl Skomski with the LLVM fuzzer.
579
580 21. Compiling PCRE2 with the sanitize options of clang showed up a number of
581 very pedantic coding infelicities and a buffer overflow while checking a UTF-8
582 string if the final multi-byte UTF-8 character was truncated.
583
584 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
585 class, where both values are literal letters in the same case, omit the
586 non-letter EBCDIC code points within the range.
587
588 23. Finding the minimum matching length of complex patterns with back
589 references and/or recursions can take a long time. There is now a cut-off that
590 gives up trying to find a minimum length when things get too complex.
591
592 24. An optimization has been added that speeds up finding the minimum matching
593 length for patterns containing repeated capturing groups or recursions.
594
595 25. If a pattern contained a back reference to a group whose number was
596 duplicated as a result of appearing in a (?|...) group, the computation of the
597 minimum matching length gave a wrong result, which could cause incorrect "no
598 match" errors. For such patterns, a minimum matching length cannot at present
599 be computed.
600
601 26. Added a check for integer overflow in conditions (?(<digits>) and
602 (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
603 fuzzer.
604
605 27. Fixed an issue when \p{Any} inside an xclass did not read the current
606 character.
607
608 28. If pcre2grep was given the -q option with -c or -l, or when handling a
609 binary file, it incorrectly wrote output to stdout.
610
611 29. The JIT compiler did not restore the control verb head in case of *THEN
612 control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
613
614 30. The way recursive references such as (?3) are compiled has been re-written
615 because the old way was the cause of many issues. Now, conversion of the group
616 number into a pattern offset does not happen until the pattern has been
617 completely compiled. This does mean that detection of all infinitely looping
618 recursions is postponed till match time. In the past, some easy ones were
619 detected at compile time. This re-writing was done in response to yet another
620 bug found by the LLVM fuzzer.
621
622 31. A test for a back reference to a non-existent group was missing for items
623 such as \987. This caused incorrect code to be compiled. This issue was found
624 by Karl Skomski with a custom LLVM fuzzer.
625
626 32. Error messages for syntax errors following \g and \k were giving inaccurate
627 offsets in the pattern.
628
629 33. Improve the performance of starting single character repetitions in JIT.
630
631 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
632
633 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
634 give the right offset instead of zero.
635
636 36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
637 This issue was found by Karl Skomski with a custom LLVM fuzzer.
638
639 37. The JIT compiler should restore the control chain for empty possessive
640 repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
641
642 38. A bug which was introduced by the single character repetition optimization
643 was fixed.
644
645 39. Match limit check added to recursion. This issue was found by Karl Skomski
646 with a custom LLVM fuzzer.
647
648 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
649 only at the part of the subject that is relevant when the starting offset is
650 non-zero.
651
652 41. Improve first character match in JIT with SSE2 on x86.
653
654 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
655 with a custom LLVM fuzzer.
656
657 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
658 III).
659
660 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
661 test (there are now 20 in total).
662
663 45. Fixed a corner case of range optimization in JIT.
664
665 46. Add the ${*MARK} facility to pcre2_substitute().
666
667 47. Modifier lists in pcre2test were splitting at spaces without the required
668 commas.
669
670 48. Implemented PCRE2_ALT_VERBNAMES.
671
672 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
673 LLVM fuzzer.
674
675 50. The pcre2test program has been extended by adding the #newline_default
676 command. This has made it possible to run the standard tests when PCRE2 is
677 compiled with either CR or CRLF as the default newline convention. As part of
678 this work, the new command was added to several test files and the testing
679 scripts were modified. The pcre2grep tests can now also be run when there is no
680 LF in the default newline convention.
681
682 51. The RunTest script has been modified so that, when JIT is used and valgrind
683 is specified, a valgrind suppressions file is set up to ignore "Invalid read of
684 size 16" errors because these are false positives when the hardware supports
685 the SSE2 instruction set.
686
687 52. It is now possible to have comment lines amid the subject strings in
688 pcre2test (and perltest.sh) input.
689
690 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
691
692 54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
693 and the matching functions with NULL contexts can be tested.
694
695 55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
696
697 56. In a character class such as [\W\p{Any}] where both a negative-type escape
698 ("not a word character") and a property escape were present, the property
699 escape was being ignored.
700
701 57. Fixed integer overflow for patterns whose minimum matching length is very,
702 very large.
703
704 58. Implemented --never-backslash-C.
705
706 59. Change 55 above introduced a bug by which certain patterns provoked the
707 erroneous error "\ at end of pattern".
708
709 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
710 errors or other strange effects if compiled in UCP mode. Found with libFuzzer
711 and AddressSanitizer.
712
713 61. Whitespace at the end of a pcre2test pattern line caused a spurious error
714 message if there were only single-character modifiers. It should be ignored.
715
716 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
717 or segmentation errors for some patterns. Found with libFuzzer and
718 AddressSanitizer.
719
720 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
721 overflow.
722
723 64. Improve error message for overly-complicated patterns.
724
725 65. Implemented an optional replication feature for patterns in pcre2test, to
726 make it easier to test long repetitive patterns. The tests for 63 above are
727 converted to use the new feature.
728
729 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
730 misbehave.
731
732 67. In pcre2_substitute() in UTF mode, the UTF validity check on the
733 replacement string was happening before the length setting when the replacement
734 string was zero-terminated.
735
736 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the
737 second and subsequent calls to pcre2_match().
738
739 69. There was no check for integer overflow for a replacement group number in
740 pcre2_substitute(). An added check for a number greater than the largest group
741 number in the pattern means this is not now needed.
742
743 70. The PCRE2-specific VERSION condition didn't work correctly if only one
744 digit was given after the decimal point, or if more than two digits were given.
745 It now works with one or two digits, and gives a compile time error if more are
746 given.
747
748 71. In pcre2_substitute() there was the possibility of reading one code unit
749 beyond the end of the replacement string.
750
751 72. The code for checking a subject's UTF-32 validity for a pattern with a
752 lookbehind involved an out-of-bounds pointer, which could potentially cause
753 trouble in some environments.
754
755 73. The maximum lookbehind length was incorrectly calculated for patterns such
756 as /(?<=(a)(?-1))x/ which have a recursion within a backreference.
757
758 74. Give an error if a lookbehind assertion is longer than 65535 code units.
759
760 75. Give an error in pcre2_substitute() if a match ends before it starts (as a
761 result of the use of \K).
762
763 76. Check the length of subpattern names and the names in (*MARK:xx) etc.
764 dynamically to avoid the possibility of integer overflow.
765
766 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the
767 size of patterns that they are prepared to handle.
768
769 78. (*NO_AUTO_POSSESS) was not working.
770
771 79. Adding group information caching improves the speed of compiling when
772 checking whether a group has a fixed length and/or could match an empty string,
773 especially when recursion or subroutine calls are involved. However, this
774 cannot be used when (?| is present in the pattern because the same number may
775 be used for groups of different sizes. To catch runaway patterns in this
776 situation, counts have been introduced to the functions that scan for empty
777 branches or compute fixed lengths.
778
779 80. Allow for the possibility of the size of the nest_save structure not being
780 a factor of the size of the compiling workspace (it currently is).
781
782 81. Check for integer overflow in minimum length calculation and cap it at
783 65535.
784
785 82. Small optimizations in code for finding the minimum matching length.
786
787 83. Lock out configuring for EBCDIC with non-8-bit libraries.
788
789 84. Test for error code <= 0 in regerror().
790
791 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute().
792
793 86. Avoid the possibility of computing with an out-of-bounds pointer (though
794 not dereferencing it) while handling lookbehind assertions.
795
796 87. Failure to get memory for the match data in regcomp() is now given as a
797 regcomp() error instead of waiting for regexec() to pick it up.
798
799 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid
800 newline sequence.
801
802 89. Paranoid check in regcomp() for bad error code from pcre2_compile().
803
804 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well
805 as for link size 2.
806
807 91. Document that JIT has a limit on pattern size, and give more information
808 about JIT compile failures in pcre2test.
809
810 92. Implement PCRE2_INFO_HASBACKSLASHC.
811
812 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports
813 with JIT (possibly caused by SSE2?).
814
815 94. Support offset_limit in JIT.
816
817 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed
818 by a single ASCII character in a class item, was incorrectly compiled in UCP
819 mode. The POSIX class got lost, but only if the single character followed it.
820
821 96. [:punct:] in UCP mode was matching some characters in the range 128-255
822 that should not have been matched.
823
824 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all
825 characters with code points greater than 255 are in the class. When a Unicode
826 property was also in the class (if PCRE2_UCP is set, escapes such as \w are
827 turned into Unicode properties), wide characters were not correctly handled,
828 and could fail to match.
829
830 98. In pcre2test, make the "startoffset" modifier a synonym of "offset",
831 because it sets the "startoffset" parameter for pcre2_match().
832
833 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between
834 an item and its qualifier (for example, A(?#comment)?B) pcre2_compile()
835 misbehaved. This bug was found by the LLVM fuzzer.
836
837 100. The error for an invalid UTF pattern string always gave the code unit
838 offset as zero instead of where the invalidity was found.
839
840 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not
841 working correctly in UCP mode.
842
843 102. Similar to 99 above, if an isolated \E was present between an item and its
844 qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug
845 was found by the LLVM fuzzer.
846
847 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND
848 was set when the pmatch argument was NULL. It now returns REG_INVARG.
849
850 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep.
851
852 105. An empty \Q\E sequence between an item and its qualifier caused
853 pcre2_compile() to misbehave when auto callouts were enabled. This bug
854 was found by the LLVM fuzzer.
855
856 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or
857 other verb "name" ended with whitespace immediately before the closing
858 parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when
859 both those options were set.
860
861 107. In a number of places pcre2_compile() was not handling NULL characters
862 correctly, and pcre2test with the "bincode" modifier was not always correctly
863 displaying fields containing NULLS:
864
865 (a) Within /x extended #-comments
866 (b) Within the "name" part of (*MARK) and other *verbs
867 (c) Within the text argument of a callout
868
869 108. If a pattern that was compiled with PCRE2_EXTENDED started with white
870 space or a #-type comment that was followed by (?-x), which turns off
871 PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again,
872 pcre2_compile() assumed that (?-x) applied to the whole pattern and
873 consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix
874 for this bug means that a setting of any of the (?imsxJU) options at the start
875 of a pattern is no longer transferred to the options that are returned by
876 PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have
877 changed when the effects of those options were all moved to compile time.
878
879 109. An escaped closing parenthesis in the "name" part of a (*verb) when
880 PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug
881 was found by the LLVM fuzzer.
882
883 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it
884 possible to test it.
885
886 111. "Harden" pcre2test against ridiculously large values in modifiers and
887 command line arguments.
888
889 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_
890 LENGTH.
891
892 113. Fix printing of *MARK names that contain binary zeroes in pcre2test.
893
894
895 Version 10.20 30-June-2015
896 --------------------------
897
898 1. Callouts with string arguments have been added.
899
900 2. Assertion code generator in JIT has been optimized.
901
902 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
903 pcre2_compile() function read past the end of the input before diagnosing an
904 error. This bug was discovered by the LLVM fuzzer.
905
906 4. Implemented pcre2_callout_enumerate().
907
908 5. Fix JIT compilation of conditional blocks whose assertion is converted to
909 (*FAIL). E.g: /(?(?!))/.
910
911 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
912 discovered by the LLVM fuzzer.
913
914 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
915 when this assertion was used as a condition, for example (?(?!)a|b). In
916 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
917 error about an unsupported item.
918
919 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
920 possessification code could take exponential time to complete. A recursion
921 depth limit of 1000 has been imposed to limit the resources used by this
922 optimization. This infelicity was discovered by the LLVM fuzzer.
923
924 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
925 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
926 because \S ensures they are all in the class. The code for doing this was
927 interacting badly with the code for computing the amount of space needed to
928 compile the pattern, leading to a buffer overflow. This bug was discovered by
929 the LLVM fuzzer.
930
931 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
932 other kinds of group caused stack overflow at compile time. This bug was
933 discovered by the LLVM fuzzer.
934
935 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment
936 between a subroutine call and its quantifier was incorrectly compiled, leading
937 to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer.
938
939 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an
940 assertion after (?(. The code was failing to check the character after (?(?<
941 for the ! or = that would indicate a lookbehind assertion. This bug was
942 discovered by the LLVM fuzzer.
943
944 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with
945 a fixed maximum following a group that contains a subroutine reference was
946 incorrectly compiled and could trigger buffer overflow. This bug was discovered
947 by the LLVM fuzzer.
948
949 14. Negative relative recursive references such as (?-7) to non-existent
950 subpatterns were not being diagnosed and could lead to unpredictable behaviour.
951 This bug was discovered by the LLVM fuzzer.
952
953 15. The bug fixed in 14 was due to an integer variable that was unsigned when
954 it should have been signed. Some other "int" variables, having been checked,
955 have either been changed to uint32_t or commented as "must be signed".
956
957 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1)))
958 caused a stack overflow instead of the diagnosis of a non-fixed length
959 lookbehind assertion. This bug was discovered by the LLVM fuzzer.
960
961 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern
962 (e.g. /(?<=\Ka)/) could make pcre2grep loop.
963
964 18. There was a similar problem to 17 in pcre2test for global matches, though
965 the code there did catch the loop.
966
967 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*),
968 and a subsequent item in the pattern caused a non-match, backtracking over the
969 repeated \X did not stop, but carried on past the start of the subject, causing
970 reference to random memory and/or a segfault. There were also some other cases
971 where backtracking after \C could crash. This set of bugs was discovered by the
972 LLVM fuzzer.
973
974 20. The function for finding the minimum length of a matching string could take
975 a very long time if mutual recursion was present many times in a pattern, for
976 example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has
977 been implemented. This infelicity was discovered by the LLVM fuzzer.
978
979 21. Implemented PCRE2_NEVER_BACKSLASH_C.
980
981 22. The feature for string replication in pcre2test could read from freed
982 memory if the replication required a buffer to be extended, and it was not
983 working properly in 16-bit and 32-bit modes. This issue was discovered by a
984 fuzzer: see http://lcamtuf.coredump.cx/afl/.
985
986 23. Added the PCRE2_ALT_CIRCUMFLEX option.
987
988 24. Adjust the treatment of \8 and \9 to be the same as the current Perl
989 behaviour.
990
991 25. Static linking against the PCRE2 library using the pkg-config module was
992 failing on missing pthread symbols.
993
994 26. If a group that contained a recursive back reference also contained a
995 forward reference subroutine call followed by a non-forward-reference
996 subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
997 compile correct code, leading to undefined behaviour or an internally detected
998 error. This bug was discovered by the LLVM fuzzer.
999
1000 27. Quantification of certain items (e.g. atomic back references) could cause
1001 incorrect code to be compiled when recursive forward references were involved.
1002 For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was
1003 discovered by the LLVM fuzzer.
1004
1005 28. A repeated conditional group whose condition was a reference by name caused
1006 a buffer overflow if there was more than one group with the given name. This
1007 bug was discovered by the LLVM fuzzer.
1008
1009 29. A recursive back reference by name within a group that had the same name as
1010 another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/.
1011 This bug was discovered by the LLVM fuzzer.
1012
1013 30. A forward reference by name to a group whose number is the same as the
1014 current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a
1015 buffer overflow at compile time. This bug was discovered by the LLVM fuzzer.
1016
1017 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1
1018 as an int; fixed by writing it as 1u).
1019
1020 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives
1021 a warning for "fileno" unless -std=gnu99 us used.
1022
1023 33. A lookbehind assertion within a set of mutually recursive subpatterns could
1024 provoke a buffer overflow. This bug was discovered by the LLVM fuzzer.
1025
1026 34. Give an error for an empty subpattern name such as (?'').
1027
1028 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains
1029 \P, \p, or \X.
1030
1031 36. The way named subpatterns are handled has been refactored. There is now a
1032 pre-pass over the regex which does nothing other than identify named
1033 subpatterns and count the total captures. This means that information about
1034 named patterns is known before the rest of the compile. In particular, it means
1035 that forward references can be checked as they are encountered. Previously, the
1036 code for handling forward references was contorted and led to several errors in
1037 computing the memory requirements for some patterns, leading to buffer
1038 overflows.
1039
1040 37. There was no check for integer overflow in subroutine calls such as (?123).
1041
1042 38. The table entry for \l in EBCDIC environments was incorrect, leading to its
1043 being treated as a literal 'l' instead of causing an error.
1044
1045 39. If a non-capturing group containing a conditional group that could match
1046 an empty string was repeated, it was not identified as matching an empty string
1047 itself. For example: /^(?:(?(1)x|)+)+$()/.
1048
1049 40. In an EBCDIC environment, pcretest was mishandling the escape sequences
1050 \a and \e in test subject lines.
1051
1052 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII
1053 instead of the EBCDIC value.
1054
1055 42. The handling of \c in an EBCDIC environment has been revised so that it is
1056 now compatible with the specification in Perl's perlebcdic page.
1057
1058 43. Single character repetition in JIT has been improved. 20-30% speedup
1059 was achieved on certain patterns.
1060
1061 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in
1062 ASCII/Unicode. This has now been added to the list of characters that are
1063 recognized as white space in EBCDIC.
1064
1065 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave
1066 an error (correctly) when used outside a class, but did not give an error
1067 within a class.
1068
1069 46. \h within a class was incorrectly compiled in EBCDIC environments.
1070
1071 47. JIT should return with error when the compiled pattern requires
1072 more stack space than the maximum.
1073
1074 48. Fixed a memory leak in pcre2grep when a locale is set.
1075
1076
1077 Version 10.10 06-March-2015
1078 ---------------------------
1079
1080 1. When a pattern is compiled, it remembers the highest back reference so that
1081 when matching, if the ovector is too small, extra memory can be obtained to
1082 use instead. A conditional subpattern whose condition is a check on a capture
1083 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
1084 another kind of back reference, but it was not setting the highest
1085 backreference number. This mattered only if pcre2_match() was called with an
1086 ovector that was too small to hold the capture, and there was no other kind of
1087 back reference (a situation which is probably quite rare). The effect of the
1088 bug was that the condition was always treated as FALSE when the capture could
1089 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
1090 has been fixed.
1091
1092 2. Functions for serialization and deserialization of sets of compiled patterns
1093 have been added.
1094
1095 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
1096 excess code units at the end of the data block that may occasionally occur if
1097 the code for calculating the size over-estimates. This change stops the
1098 serialization code copying uninitialized data, to which valgrind objects. The
1099 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
1100 include the general overhead. This has been corrected.
1101
1102 4. All code units in every slot in the table of group names are now set, again
1103 in order to avoid accessing uninitialized data when serializing.
1104
1105 5. The (*NO_JIT) feature is implemented.
1106
1107 6. If a bug that caused pcre2_compile() to use more memory than allocated was
1108 triggered when using valgrind, the code in (3) above passed a stupidly large
1109 value to valgrind. This caused a crash instead of an "internal error" return.
1110
1111 7. A reference to a duplicated named group (either a back reference or a test
1112 for being set in a conditional) that occurred in a part of the pattern where
1113 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
1114 to be incorrectly calculated, leading to overwriting.
1115
1116 8. A mutually recursive set of back references such as (\2)(\1) caused a
1117 segfault at compile time (while trying to find the minimum matching length).
1118 The infinite loop is now broken (with the minimum length unset, that is, zero).
1119
1120 9. If an assertion that was used as a condition was quantified with a minimum
1121 of zero, matching went wrong. In particular, if the whole group had unlimited
1122 repetition and could match an empty string, a segfault was likely. The pattern
1123 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
1124 quantified, but not if they are being used as conditions, so the above pattern
1125 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
1126 patterns.
1127
1128 10. The error message for an invalid quantifier has been changed from "nothing
1129 to repeat" to "quantifier does not follow a repeatable item".
1130
1131 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
1132 scanning the compiled pattern in subsequent auto-possessification can get out
1133 of step and lead to an unknown opcode. Previously this could have caused an
1134 infinite loop. Now it generates an "internal error" error. This is a tidyup,
1135 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
1136 undefined outcome.
1137
1138 12. A UTF pattern containing a "not" match of a non-ASCII character and a
1139 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
1140
1141 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
1142 that is found in the output of "locale -a" can actually be set by pcre2test
1143 before it is accepted. Previously, in an environment where a locale was listed
1144 but would not set (an example does exist), the test would "pass" without
1145 actually doing anything. Also the fr_CA locale has been added to the list of
1146 locales that can be used.
1147
1148 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
1149 capturing group number without parentheses, the last character was incorrectly
1150 literally included at the end of the replacement string.
1151
1152 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
1153 failed to allow the zero-repeat case if pcre2_match() was called with an
1154 ovector too small to capture the group.
1155
1156 16. Improved error message in pcre2test when setting the stack size (-S) fails.
1157
1158 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
1159 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
1160 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
1161 of PCRE2 sources, which caused a failure to build pcre2test.
1162
1163 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
1164 only on Windows.
1165
1166 19. Use binary input when reading back saved serialized patterns in pcre2test.
1167
1168 20. Added RunTest.bat for running the tests under Windows.
1169
1170 21. "make distclean" was not removing config.h, a file that may be created for
1171 use with CMake.
1172
1173 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
1174 forward reference repeated a large (but limited) number of times within a
1175 repeated outer group that has a zero minimum quantifier, caused incorrect code
1176 to be compiled, leading to the error "internal error: previously-checked
1177 referenced subpattern not found" when an incorrect memory address was read.
1178 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
1179 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
1180
1181 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
1182 call within a group that also contained a recursive back reference caused
1183 incorrect code to be compiled. This bug was reported as "heap overflow",
1184 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
1185 CVE-2015-2326 was given to this.)
1186
1187 24. Computing the size of the JIT read-only data in advance has been a source
1188 of various issues, and new ones are still appear unfortunately. To fix
1189 existing and future issues, size computation is eliminated from the code,
1190 and replaced by on-demand memory allocation.
1191
1192 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
1193 adjacent to the end of the range, and the range contained characters with more
1194 than one other case, caused incorrect behaviour when compiled in UTF mode. In
1195 that example, the range a-j was left out of the class.
1196
1197
1198 Version 10.00 05-January-2015
1199 -----------------------------
1200
1201 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
1202 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
1203 API, up to item 20 for release 8.36.
1204
1205 The code of the library was heavily revised as part of the new API
1206 implementation. Details of each and every modification were not individually
1207 logged. In addition to the API changes, the following changes were made. They
1208 are either new functionality, or bug fixes and other noticeable changes of
1209 behaviour that were implemented after the code had been forked.
1210
1211 1. Including Unicode support at build time is now enabled by default, but it
1212 can optionally be disabled. It is not enabled by default at run time (no
1213 change).
1214
1215 2. The test program, now called pcre2test, was re-specified and almost
1216 completely re-written. Its input is not compatible with input for pcretest.
1217
1218 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
1219 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
1220 matched by that pattern.
1221
1222 4. For the benefit of those who use PCRE2 via some other application, that is,
1223 not writing the function calls themselves, it is possible to check the PCRE2
1224 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
1225 string such as "yesno".
1226
1227 5. There are case-equivalent Unicode characters whose encodings use different
1228 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
1229 theoretically possible for this to happen in UTF-16 too.) If a backreference to
1230 a group containing one of these characters was greedily repeated, and during
1231 the match a backtrack occurred, the subject might be backtracked by the wrong
1232 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
1233 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
1234 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
1235 Incorrect backtracking meant that group 2 captured only the last two bytes.
1236 This bug has been fixed; the new code is slower, but it is used only when the
1237 strings matched by the repetition are not all the same length.
1238
1239 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
1240 information. This applied to any pattern with a group that matched no
1241 characters, for example: /(?:(?=.)|(?<!x))a/.
1242
1243 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
1244 those parentheses to be closed with whatever has been captured so far. However,
1245 it was failing to mark any other groups between the highest capture so far and
1246 the currrent group as "unset". Thus, the ovector for those groups contained
1247 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
1248 matched against "abcd".
1249
1250 8. The pcre2_substitute() function has been implemented.
1251
1252 9. If an assertion used as a condition was quantified with a minimum of zero
1253 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
1254 occur.
1255
1256 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.
1257
1258 ****

  ViewVC Help
Powered by ViewVC 1.1.5