/[pcre2]/code/trunk/ChangeLog
ViewVC logotype

Contents of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1091 - (show annotations)
Mon May 13 16:26:17 2019 UTC (21 months, 2 weeks ago) by ph10
File size: 95761 byte(s)
Fix crash when \X is used without UTF in JIT.
1 Change Log for PCRE2
2 --------------------
3
4
5 Version 10.34 22-April-2019
6 ---------------------------
7
8 1. The maximum number of capturing subpatterns is 65535 (documented), but no
9 check on this was ever implemented. This omission has been rectified; it fixes
10 ClusterFuzz 14376.
11
12 2. Improved the invalid utf32 support of the JIT compiler. Now it correctly
13 detects invalid characters in the 0xd800-0xdfff range.
14
15 3. Fix minor typo bug in JIT compile when \X is used in a non-UTF string.
16
17
18 Version 10.33 16-April-2019
19 ---------------------------
20
21 1. Added "allvector" to pcre2test to make it easy to check the part of the
22 ovector that shouldn't be changed, in particular after substitute and failed or
23 partial matches.
24
25 2. Fix subject buffer overread in JIT when UTF is disabled and \X or \R has
26 a greater than 1 fixed quantifier. This issue was found by Yunho Kim.
27
28 3. Added support for callouts from pcre2_substitute(). After 10.33-RC1, but
29 prior to release, fixed a bug that caused a crash if pcre2_substitute() was
30 called with a NULL match context.
31
32 4. The POSIX functions are now all called pcre2_regcomp() etc., with wrapper
33 functions that use the standard POSIX names. However, in pcre2posix.h the POSIX
34 names are defined as macros. This should help avoid linking with the wrong
35 library in some environments while still exporting the POSIX names for
36 pre-existing programs that use them. (The Debian alternative names are also
37 defined as macros, but not documented.)
38
39 5. Fix an xclass matching issue in JIT.
40
41 6. Implement PCRE2_EXTRA_ESCAPED_CR_IS_LF (see Bugzilla 2315).
42
43 7. Implement the Perl 5.28 experimental alphabetic names for atomic groups and
44 lookaround assertions, for example, (*pla:...) and (*atomic:...). These are
45 characterized by a lower case letter following (* and to simplify coding for
46 this, the character tables created by pcre2_maketables() were updated to add a
47 new "is lower case letter" bit. At the same time, the now unused "is
48 hexadecimal digit" bit was removed. The default tables in
49 src/pcre2_chartables.c.dist are updated.
50
51 8. Implement the new Perl "script run" features (*script_run:...) and
52 (*atomic_script_run:...) aka (*sr:...) and (*asr:...).
53
54 9. Fixed two typos in change 22 for 10.21, which added special handling for
55 ranges such as a-z in EBCDIC environments. The original code probably never
56 worked, though there were no bug reports.
57
58 10. Implement PCRE2_COPY_MATCHED_SUBJECT for pcre2_match() (including JIT via
59 pcre2_match()) and pcre2_dfa_match(), but *not* the pcre2_jit_match() fast
60 path. Also, when a match fails, set the subject field in the match data to NULL
61 for tidiness - none of the substring extractors should reference this after
62 match failure.
63
64 11. If a pattern started with a subroutine call that had a quantifier with a
65 minimum of zero, an incorrect "match must start with this character" could be
66 recorded. Example: /(?&xxx)*ABC(?<xxx>XYZ)/ would (incorrectly) expect 'A' to
67 be the first character of a match.
68
69 12. The heap limit checking code in pcre2_dfa_match() could suffer from
70 overflow if the heap limit was set very large. This could cause incorrect "heap
71 limit exceeded" errors.
72
73 13. Add "kibibytes" to the heap limit output from pcre2test -C to make the
74 units clear.
75
76 14. Add a call to pcre2_jit_free_unused_memory() in pcre2grep, for tidiness.
77
78 15. Updated the VMS-specific code in pcre2test on the advice of a VMS user.
79
80 16. Removed the unnecessary inclusion of stdint.h (or inttypes.h) from
81 pcre2_internal.h as it is now included by pcre2.h. Also, change 17 for 10.32
82 below was unnecessarily complicated, as inttypes.h is a Standard C header,
83 which is defined to be a superset of stdint.h. Instead of conditionally
84 including stdint.h or inttypes.h, pcre2.h now unconditionally includes
85 inttypes.h. This supports environments that do not have stdint.h but do have
86 inttypes.h, which are known to exist. A note in the autotools documentation
87 says (November 2018) that there are none known that are the other way round.
88
89 17. Added --disable-percent-zt to "configure" (and equivalent to CMake) to
90 forcibly disable the use of %zu and %td in formatting strings because there is
91 at least one version of VMS that claims to be C99 but does not support these
92 modifiers.
93
94 18. Added --disable-pcre2grep-callout-fork, which restricts the callout support
95 in pcre2grep to the inbuilt echo facility. This may be useful in environments
96 that do not support fork().
97
98 19. Fix two instances of <= 0 being applied to unsigned integers (the VMS
99 compiler complains).
100
101 20. Added "fork" support for VMS to pcre2grep, for running an external program
102 via a string callout.
103
104 21. Improve MAP_JIT flag usage on MacOS. Patch by Rich Siegel.
105
106 22. If a pattern started with (*MARK), (*COMMIT), (*PRUNE), (*SKIP), or (*THEN)
107 followed by ^ it was not recognized as anchored.
108
109 23. The RunGrepTest script used to cut out the test of NUL characters for
110 Solaris and MacOS as printf and sed can't handle them. It seems that the *BSD
111 systems can't either. I've inverted the test so that only those OS that are
112 known to work (currently only Linux) try to run this test.
113
114 24. Some tests in RunGrepTest appended to testtrygrep from two different file
115 descriptors instead of redirecting stderr to stdout. This worked on Linux, but
116 it was reported not to on other systems, causing the tests to fail.
117
118 25. In the RunTest script, make the test for stack setting use the same value
119 for the stack as it needs for -bigstack.
120
121 26. Insert a cast in pcre2_dfa_match.c to suppress a compiler warning.
122
123 26. With PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL set, escape sequences such as \s
124 which are valid in character classes, but not as the end of ranges, were being
125 treated as literals. An example is [_-\s] (but not [\s-_] because that gave an
126 error at the *start* of a range). Now an "invalid range" error is given
127 independently of PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
128
129 27. Related to 26 above, PCRE2_BAD_ESCAPE_IS_LITERAL was affecting known escape
130 sequences such as \eX when they appeared invalidly in a character class. Now
131 the option applies only to unrecognized or malformed escape sequences.
132
133 28. Fix word boundary in JIT compiler. Patch by Mike Munday.
134
135 29. The pcre2_dfa_match() function was incorrectly handling conditional version
136 tests such as (?(VERSION>=0)...) when the version test was true. Incorrect
137 processing or a crash could result.
138
139 30. When PCRE2_UTF is set, allow non-ASCII letters and decimal digits in group
140 names, as Perl does. There was a small bug in this new code, found by
141 ClusterFuzz 12950, fixed before release.
142
143 31. Implemented PCRE2_EXTRA_ALT_BSUX to support ECMAScript 6's \u{hhh}
144 construct.
145
146 32. Compile \p{Any} to be the same as . in DOTALL mode, so that it benefits
147 from auto-anchoring if \p{Any}* starts a pattern.
148
149 33. Compile invalid UTF check in JIT test when only pcre32 is enabled.
150
151 34. For some time now, CMake has been warning about the setting of policy
152 CMP0026 to "OLD" in CmakeLists.txt, and hinting that the feature might be
153 removed in a future version. A request for CMake expertise on the list produced
154 no result, so I have now hacked CMakeLists.txt along the lines of some changes
155 I found on the Internet. The new code no longer needs the policy setting, and
156 it appears to work fine on Linux.
157
158 35. Setting --enable-jit=auto for an out-of-tree build failed because the
159 source directory wasn't in the search path for AC_TRY_COMPILE always. Patch
160 from Ross Burton.
161
162 36. Disable SSE2 JIT optimizations in x86 CPUs when SSE2 is not available.
163 Patch by Guillem Jover.
164
165 37. Changed expressions such as 1<<10 to 1u<<10 in many places because compiler
166 warnings were reported.
167
168 38. Using the clang compiler with sanitizing options causes runtime complaints
169 about truncation for statments such as x = ~x when x is an 8-bit value; it
170 seems to compute ~x as a 32-bit value. Changing such statements to x = 255 ^ x
171 gets rid of the warnings. There were also two missing casts in pcre2test.
172
173
174 Version 10.32 10-September-2018
175 -------------------------------
176
177 1. When matching using the the REG_STARTEND feature of the POSIX API with a
178 non-zero starting offset, unset capturing groups with lower numbers than a
179 group that did capture something were not being correctly returned as "unset"
180 (that is, with offset values of -1).
181
182 2. When matching using the POSIX API, pcre2test used to omit listing unset
183 groups altogether. Now it shows those that come before any actual captures as
184 "<unset>", as happens for non-POSIX matching.
185
186 3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",
187 whatever the build configuration was. It now correctly says "\R matches all
188 Unicode newlines" in the default case when --enable-bsr-anycrlf has not been
189 specified. Similarly, running "pcre2test -C bsr" never produced the result
190 ANY.
191
192 4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing
193 multi-code-unit characters caused bad behaviour and possibly a crash. This
194 issue was fixed for other kinds of repeat in release 10.20 by change 19, but
195 repeating character classes were overlooked.
196
197 5. pcre2grep now supports the inclusion of binary zeros in patterns that are
198 read from files via the -f option.
199
200 6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.
201
202 7. Added --enable-jit=auto support to configure.ac.
203
204 8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit
205 modes for the benefit of m68k, where pointers can be 16-bit aligned. The
206 dummies force 32-bit alignment and this ensures that the structure is a
207 multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other
208 architectures, alignment requirements take care of this automatically.
209
210 9. When returning an error from pcre2_pattern_convert(), ensure the error
211 offset is set zero for early errors.
212
213 10. A number of patches for Windows support from Daniel Richard G:
214
215 (a) List of error numbers in Runtest.bat corrected (it was not the same as in
216 Runtest).
217
218 (b) pcre2grep snprintf() workaround as used elsewhere in the tree.
219
220 (c) Support for non-C99 snprintf() that returns -1 in the overflow case.
221
222 11. Minor tidy of pcre2_dfa_match() code.
223
224 12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer
225 use the stack for local workspace and local ovectors. Instead, an initial block
226 of stack is reserved, but if this is insufficient, heap memory is used. The
227 heap limit parameter now applies to pcre2_dfa_match().
228
229 13. If a "find limits" test of DFA matching in pcre2test resulted in too many
230 matches for the ovector, no matches were displayed.
231
232 14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as
233 EOF. The test looks to have come from a fuzzer.
234
235 15. If PCRE2 was built with a default match limit a lot greater than the
236 default default of 10 000 000, some JIT tests of the match limit no longer
237 failed. All such tests now set 10 000 000 as the upper limit.
238
239 16. Another Windows related patch for pcregrep to ensure that WIN32 is
240 undefined under Cygwin.
241
242 17. Test for the presence of stdint.h and inttypes.h in configure and CMake and
243 include whichever exists (stdint preferred) instead of unconditionally
244 including stdint. This makes life easier for old and non-standard systems.
245
246 18. Further changes to improve portability, especially to old and or non-
247 standard systems:
248
249 (a) Put all printf arguments in RunGrepTest into single, not double, quotes,
250 and use \0 not \x00 for binary zero.
251
252 (b) Avoid the use of C++ (i.e. BCPL) // comments.
253
254 (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of
255 these now, if using MSVC or a standard C before C99, %lu is used with a
256 cast if necessary.
257
258 19. Applied a contributed patch to CMakeLists.txt to increase the stack size
259 when linking pcre2test with MSVC. This gets rid of a stack overflow error in
260 the standard set of tests.
261
262 20. Output a warning in pcre2test when ignoring the "altglobal" modifier when
263 it is given with the "replace" modifier.
264
265 21. In both pcre2test and pcre2_substitute(), with global matching, a pattern
266 that matched an empty string, but never at the starting match offset, was not
267 handled in a Perl-compatible way. The pattern /(<?=\G.)/ is an example of such
268 a pattern. Because \G is in a lookbehind assertion, there has to be a
269 "bumpalong" before there can be a match. The automatic "advance by one
270 character after an empty string match" rule is therefore inappropriate. A more
271 complicated algorithm has now been implemented.
272
273 22. When checking to see if a lookbehind is of fixed length, lookaheads were
274 correctly ignored, but qualifiers on lookaheads were not being ignored, leading
275 to an incorrect "lookbehind assertion is not fixed length" error.
276
277 23. The VERSION condition test was reading fractional PCRE2 version numbers
278 such as the 04 in 10.04 incorrectly and hence giving wrong results.
279
280 24. Updated to Unicode version 11.0.0. As well as the usual addition of new
281 scripts and characters, this involved re-jigging the grapheme break property
282 algorithm because Unicode has changed the way emojis are handled.
283
284 25. Fixed an obscure bug that struck when there were two atomic groups not
285 separated by something with a backtracking point. There could be an incorrect
286 backtrack into the first of the atomic groups. A complicated example is
287 /(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP
288 shouldn't find a MARK (because is in an atomic group), but it did.
289
290 26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set
291 a list of modifiers for all subsequent patterns - only those that the script
292 recognizes are meaningful; (2) #subject lines can be used to set or unset a
293 default "mark" modifier; (3) Unsupported #command lines give a warning when
294 they are ignored; (4) Mark data is output only if the "mark" modifier is
295 present.
296
297 27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported.
298
299 28. A (*MARK) name was not being passed back for positive assertions that were
300 terminated by (*ACCEPT).
301
302 29. Add support for \N{U+dddd}, but only in Unicode mode.
303
304 30. Add support for (?^) for unsetting all imnsx options.
305
306 31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose
307 code point was less than 256 and that were recognized by the lookup table
308 generated by pcre2_maketables(), which uses isspace() to identify white space.
309 Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085,
310 U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by
311 Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl.
312
313 32. In certain circumstances, option settings within patterns were not being
314 correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly
315 matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the
316 end of its group during the parse process, but without another setting such as
317 (?m) the compile phase got it right.) This bug was introduced by the
318 refactoring in release 10.23.
319
320 33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to
321 define memmove() as function call to bcopy(). This hasn't been tested for a
322 long time because in pcre2test the result of memmove() was being used, whereas
323 bcopy() doesn't return a result. This feature is now refactored always to call
324 an emulation function when there is no memmove(). The emulation makes use of
325 bcopy() when available.
326
327 34. When serializing a pattern, set the memctl, executable_jit, and tables
328 fields (that is, all the fields that contain pointers) to zeros so that the
329 result of serializing is always the same. These fields are re-set when the
330 pattern is deserialized.
331
332 35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated
333 negative class with no characters less than 0x100 followed by a positive class
334 with only characters less than 0x100, the first class was incorrectly being
335 auto-possessified, causing incorrect match failures.
336
337 36. Removed the character type bit ctype_meta, which dates from PCRE1 and is
338 not used in PCRE2.
339
340 37. Tidied up unnecessarily complicated macros used in the escapes table.
341
342 38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted
343 from distribution tarballs, owing to a typo in Makefile.am which had
344 testoutput8-16-3 twice. Now fixed.
345
346 39. If the only branch in a conditional subpattern was anchored, the whole
347 subpattern was treated as anchored, when it should not have been, since the
348 assumed empty second branch cannot be anchored. Demonstrated by test patterns
349 such as /(?(1)^())b/ or /(?(?=^))b/.
350
351 40. A repeated conditional subpattern that could match an empty string was
352 always assumed to be unanchored. Now it it checked just like any other
353 repeated conditional subpattern, and can be found to be anchored if the minimum
354 quantifier is one or more. I can't see much use for a repeated anchored
355 pattern, but the behaviour is now consistent.
356
357 41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint
358 (for an event that could never occur but you had to have external information
359 to know that).
360
361 42. If before the first match in a file that was being searched by pcre2grep
362 there was a line that was sufficiently long to cause the input buffer to be
363 expanded, the variable holding the location of the end of the previous match
364 was being adjusted incorrectly, and could cause an overflow warning from a code
365 sanitizer. However, as the value is used only to print pending "after" lines
366 when the next match is reached (and there are no such lines in this case) this
367 bug could do no damage.
368
369
370 Version 10.31 12-February-2018
371 ------------------------------
372
373 1. Fix typo (missing ]) in VMS code in pcre2test.c.
374
375 2. Replace the replicated code for matching extended Unicode grapheme sequences
376 (which got a lot more complicated by change 10.30/49) by a single subroutine
377 that is called by both pcre2_match() and pcre2_dfa_match().
378
379 3. Add idempotent guard to pcre2_internal.h.
380
381 4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and
382 PCRE2_CONFIG_COMPILED_WIDTHS.
383
384 5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is
385 defined (e.g. by --enable-never-backslash-C).
386
387 6. Defined public names for all the pcre2_compile() error numbers, and used
388 the public names in pcre2_convert.c.
389
390 7. Fixed a small memory leak in pcre2test (convert contexts).
391
392 8. Added two casts to compile.c and one to match.c to avoid compiler warnings.
393
394 9. Added code to pcre2grep when compiled under VMS to set the symbol
395 PCRE2GREP_RC to the exit status, because VMS does not distinguish between
396 exit(0) and exit(1).
397
398 10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain
399 about a bad option only if the following argument item does not start with a
400 hyphen.
401
402 11. pcre2grep was truncating components of file names to 128 characters when
403 processing files with the -r option, and also (some very odd code) truncating
404 path names to 512 characters. There is now a check on the absolute length of
405 full path file names, which may be up to 2047 characters long.
406
407 12. When an assertion contained (*ACCEPT) it caused all open capturing groups
408 to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to
409 misbehaviour for subsequent references to groups that started outside the
410 assertion. ACCEPT in an assertion now closes only those groups that were
411 started within that assertion. Fixes oss-fuzz issues 3852 and 3891.
412
413 13. Multiline matching in pcre2grep was misbehaving if the pattern matched
414 within a line, and then matched again at the end of the line and over into
415 subsequent lines. Behaviour was different with and without colouring, and
416 sometimes context lines were incorrectly printed and/or line endings were lost.
417 All these issues should now be fixed.
418
419 14. If --line-buffered was specified for pcre2grep when input was from a
420 compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be
421 ignored for compressed files.)
422
423 15. Although pcre2_jit_match checks whether the pattern is compiled
424 in a given mode, it was also expected that at least one mode is available.
425 This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION
426 when the pattern is not optimized by JIT at all.
427
428 16. The line number and related variables such as match counts in pcre2grep
429 were all int variables, causing overflow when files with more than 2147483647
430 lines were processed (assuming 32-bit ints). They have all been changed to
431 unsigned long ints.
432
433 17. If a backreference with a minimum repeat count of zero was first in a
434 pattern, apart from assertions, an incorrect first matching character could be
435 recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set
436 as the first character of a match.
437
438 18. Characters in a leading positive assertion are considered for recording a
439 first character of a match when the rest of the pattern does not provide one.
440 However, a character in a non-assertive group within a leading assertion such
441 as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an
442 infelicity rather than an outright bug, because it did not affect the result of
443 a match, just its speed. (In fact, in this case, the starting 'a' was
444 subsequently picked up in the study.)
445
446 19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return"
447 instead of "RRETURN" saves unwinding the backtracks in these cases (only one
448 didn't).
449
450 20. Allocate a single callout block on the stack at the start of pcre2_match()
451 and set its never-changing fields once only. Do the same for pcre2_dfa_match().
452
453 21. Save the extra compile options (set in the compile context) with the
454 compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS
455 to retrieve them, and update pcre2test to show them.
456
457 22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new
458 field callout_flags in callout blocks. The bits are set by pcre2_match(), but
459 not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts
460 if the callout_extra subject modifier is set. These bits are provided to help
461 with tracking how a backtracking match is proceeding.
462
463 23. Updated the pcre2demo.c demonstration program, which was missing the extra
464 code for -g that handles the case when \K in an assertion causes the match to
465 end at the original start point. Also arranged for it to detect when \K causes
466 the end of a match to be before its start.
467
468 24. Similar to 23 above, strange things (including loops) could happen in
469 pcre2grep when \K was used in an assertion when --colour was used or in
470 multiline mode. The "end at original start point" bug is fixed, and if the end
471 point is found to be before the start point, they are swapped.
472
473 25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT
474 matching (both pcre2_match() and pcre2_dfa_match()) and the matched string
475 started with the first code unit of a newline sequence, matching failed because
476 it was not tried at the newline.
477
478 26. Code for giving up a non-partial match after failing to find a starting
479 code unit anywhere in the subject was missing when searching for one of a
480 number of code units (the bitmap case) in both pcre2_match() and
481 pcre2_dfa_match(). This was a missing optimization rather than a bug.
482
483 27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a
484 pointer argument rather than a code unit value. This should not have affected
485 the generated code.
486
487 28. The JIT compiler has been updated.
488
489 29. Avoid pointer overflow for unset captures in pcre2_substring_list_get().
490 This could not actually cause a crash because it was always used in a memcpy()
491 call with zero length.
492
493 30. Some internal structures have a variable-length ovector[] as their last
494 element. Their actual memory is obtained dynamically, giving an ovector of
495 appropriate length. However, they are defined in the structure as
496 ovector[NUMBER], where NUMBER is large so that array bound checkers don't
497 grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing
498 groups, making the ovector larger than this. The number has been increased to
499 131072, which allows for the maximum number of captures (65535) plus the
500 overall match. This fixes oss-fuzz issue 5415.
501
502 31. Auto-possessification at the end of a capturing group was dependent on what
503 follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused
504 incorrect behaviour when the group was called recursively from elsewhere in the
505 pattern where something different might follow. This bug is an unforseen
506 consequence of change #1 for 10.30 - the implementation of backtracking into
507 recursions. Iterators at the ends of capturing groups are no longer considered
508 for auto-possessification if the pattern contains any recursions. Fixes
509 Bugzilla #2232.
510
511
512 Version 10.30 14-August-2017
513 ----------------------------
514
515 1. The main interpreter, pcre2_match(), has been refactored into a new version
516 that does not use recursive function calls (and therefore the stack) for
517 remembering backtracking positions. This makes --disable-stack-for-recursion a
518 NOOP. The new implementation allows backtracking into recursive group calls in
519 patterns, making it more compatible with Perl, and also fixes some other
520 hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
521 the old code had a number of fudges to try to reduce stack usage. It seems to
522 run no slower than the old code.
523
524 A number of bugs in the refactored code were subsequently fixed during testing
525 before release, but after the code was made available in the repository. These
526 bugs were never in fully released code, but are noted here for the record.
527
528 (a) If a pattern had fewer capturing parentheses than the ovector supplied in
529 the match data block, a memory error (detectable by ASAN) occurred after
530 a match, because the external block was being set from non-existent
531 internal ovector fields. Fixes oss-fuzz issue 781.
532
533 (b) A pattern with very many capturing parentheses (when the internal frame
534 size was greater than the initial frame vector on the stack) caused a
535 crash. A vector on the heap is now set up at the start of matching if the
536 vector on the stack is not big enough to handle at least 10 frames.
537 Fixes oss-fuzz issue 783.
538
539 (c) Handling of (*VERB)s in recursions was wrong in some cases.
540
541 (d) Captures in negative assertions that were used as conditions were not
542 happening if the assertion matched via (*ACCEPT).
543
544 (e) Mark values were not being passed out of recursions.
545
546 (f) Refactor some code in do_callout() to avoid picky compiler warnings about
547 negative indices. Fixes oss-fuzz issue 1454.
548
549 (g) Similarly refactor the way the variable length ovector is addressed for
550 similar reasons. Fixes oss-fuzz issue 1465.
551
552 2. Now that pcre2_match() no longer uses recursive function calls (see above),
553 the "match limit recursion" value seems misnamed. It still exists, and limits
554 the depth of tree that is searched. To avoid future confusion, it has been
555 renamed as "depth limit" in all relevant places (--with-depth-limit,
556 (*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
557 available for backwards compatibility.
558
559 3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:
560
561 (a) Check for malloc failures when getting memory for the ovector (POSIX) or
562 the match data block (non-POSIX).
563
564 4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
565 for a character with a code point greater than 0x10ffff (the Unicode maximum)
566 caused a crash.
567
568 5. If a lookbehind assertion that contained a back reference to a group
569 appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
570 undefined actions (often a segmentation fault) could occur, depending on what
571 other options were set. An example assertion is (?<!\1(abc)) where the
572 reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.
573
574 6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
575 pcre2test to use it to output the frame size when the "framesize" modifier is
576 given.
577
578 7. Reworked the recursive pattern matching in the JIT compiler to follow the
579 interpreter changes.
580
581 8. When the zero_terminate modifier was specified on a pcre2test subject line
582 for global matching, unpredictable things could happen. For example, in UTF-8
583 mode, the pattern //g,zero_terminate read random memory when matched against an
584 empty string with zero_terminate. This was a bug in pcre2test, not the library.
585
586 9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
587 of the section that is compiled when Unix-style directory scanning is
588 available, and into a new section that is always compiled for Windows.
589
590 10. In pcre2test, explicitly close the file after an error during serialization
591 or deserialization (the "load" or "save" commands).
592
593 11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.
594
595 12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
596 a NULL pattern pointer when Unicode support is available.
597
598 13. When the 32-bit library was being tested by pcre2test, error messages that
599 were longer than 64 code units could cause a buffer overflow. This was a bug in
600 pcre2test.
601
602 14. The alternative matching function, pcre2_dfa_match() misbehaved if it
603 encountered a character class with a possessive repeat, for example [a-f]{3}+.
604
605 15. The depth (formerly recursion) limit now applies to DFA matching (as
606 of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA
607 matching to find the minimum value for this limit.
608
609 16. Since 10.21, if pcre2_match() was called with a null context, default
610 memory allocation functions were used instead of whatever was used when the
611 pattern was compiled.
612
613 17. Changes to the pcre2test "memory" modifier on a subject line. These apply
614 only to pcre2_match():
615
616 (a) Warn if null_context is set on both pattern and subject, because the
617 memory details cannot then be shown.
618
619 (b) Remember (up to a certain number of) memory allocations and their
620 lengths, and list only the lengths, so as to be system-independent.
621 (In practice, the new interpreter never has more than 2 blocks allocated
622 simultaneously.)
623
624 18. Make pcre2test detect an error return from pcre2_get_error_message(), give
625 a message, and abandon the run (this would have detected #13 above).
626
627 19. Implemented PCRE2_ENDANCHORED.
628
629 20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement
630 the --output=text (-O) option and the inbuilt callout echo.
631
632 21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and
633 single-branch conditions with a false condition (e.g. DEFINE) at the start of a
634 branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as
635 anchored.
636
637 22. Added an explicit limit on the amount of heap used by pcre2_match(), set by
638 pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the
639 heap limit along with other pattern information, and to find the minimum when
640 the find_limits modifier is set.
641
642 23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled
643 pattern is set up so as to initialize any padding the compiler might have
644 included. This avoids valgrind warnings when a compiled pattern is copied, in
645 particular when it is serialized.
646
647 24. Remove a redundant line of code left in accidentally a long time ago.
648
649 25. Remove a duplication typo in pcre2_tables.c
650
651 26. Correct an incorrect cast in pcre2_valid_utf.c
652
653 27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the
654 tests to improve coverage.
655
656 28. Some fixes/tidies as a result of looking at Coverity Scan output:
657
658 (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.
659 (b) Added some casts to avoid "suspicious implicit sign extension".
660 (c) Resource leaks in pcre2test in rare error cases.
661 (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge
662 for checking at compile time that tables are the right size.
663 (e) Add missing "fall through" comment.
664
665 29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.
666
667 30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.
668
669 31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in
670 pcre2test, a crash could occur.
671
672 32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so
673 that all the tests can run with clang's sanitizing options.
674
675 33. Implement extra compile options in the compile context and add the first
676 one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.
677
678 34. Implement newline type PCRE2_NEWLINE_NUL.
679
680 35. A lookbehind assertion that had a zero-length branch caused undefined
681 behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.
682
683 36. The match limit value now also applies to pcre2_dfa_match() as there are
684 patterns that can use up a lot of resources without necessarily recursing very
685 deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.
686
687 37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
688
689 38. Fix returned offsets from regexec() when REG_STARTEND is used with a
690 starting offset greater than zero.
691
692 39. Implement REG_PEND (GNU extension) for the POSIX wrapper.
693
694 40. Implement the subject_literal modifier in pcre2test, and allow jitstack on
695 pattern lines.
696
697 41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.
698
699 42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit
700 of pcre2grep.
701
702 43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,
703 PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:
704
705 (a) The -F option did not work for fixed strings containing \E.
706 (b) The -w option did not work for patterns with multiple branches.
707
708 44. Added configuration options for the SELinux compatible execmem allocator in
709 JIT.
710
711 45. Increased the limit for searching for a "must be present" code unit in
712 subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are
713 much faster.
714
715 46. Arrange for anchored patterns to record and use "first code unit" data,
716 because this can give a fast "no match" without searching for a "required code
717 unit". Previously only non-anchored patterns did this.
718
719 47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.
720
721 48. Add the callout_no_where modifier to pcre2test.
722
723 49. Update extended grapheme breaking rules to the latest set that are in
724 Unicode Standard Annex #29.
725
726 50. Added experimental foreign pattern conversion facilities
727 (pcre2_pattern_convert() and friends).
728
729 51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE
730 is defined in a system header in cygwin. Also modified some of the #ifdefs in
731 pcre2grep related to Windows and Cygwin support.
732
733 52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a
734 character class is the last character in the class, Perl does not give a
735 warning. PCRE2 now also treats this as a literal.
736
737 53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was
738 not doing so for [\d-X] (and similar escapes), as is documented.
739
740 54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard.
741
742 55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in
743 pcre2_compile() which could never actually trigger (code should have been cut
744 out when Unicode support is disabled).
745
746
747 Version 10.23 14-February-2017
748 ------------------------------
749
750 1. Extended pcre2test with the utf8_input modifier so that it is able to
751 generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
752
753 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
754 PCRE2_UCP set, a negative character type such as \D in a positive class should
755 cause all characters greater than 255 to match, whatever else is in the class.
756 There was a bug that caused this not to happen if a Unicode property item was
757 added to such a class, for example [\D\P{Nd}] or [\W\pL].
758
759 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
760 checking is now done in the pre-pass that identifies capturing groups. This has
761 reduced the amount of duplication and made the code tidier. While doing this,
762 some minor bugs and Perl incompatibilities were fixed, including:
763
764 (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
765 of giving an invalid quantifier error.
766
767 (b) {0} can now be used after a group in a lookbehind assertion; previously
768 this caused an "assertion is not fixed length" error.
769
770 (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
771 the name "DEFINE" exists. PCRE2 now does likewise.
772
773 (d) A recursion condition test such as (?(R2)...) must now refer to an
774 existing subpattern.
775
776 (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
777 group whose name began with "R".
778
779 (f) When testing zero-terminated patterns under valgrind, the terminating
780 zero is now marked "no access". This catches bugs that would otherwise
781 show up only with non-zero-terminated patterns.
782
783 (g) A hyphen appearing immediately after a POSIX character class (for example
784 /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
785 literal, but gives a warning, so it seems best to fail it in PCRE.
786
787 (h) An empty \Q\E sequence may appear after a callout that precedes an
788 assertion condition (it is, of course, ignored).
789
790 One effect of the refactoring is that some error numbers and messages have
791 changed, and the pattern offset given for compiling errors is not always the
792 right-most character that has been read. In particular, for a variable-length
793 lookbehind assertion it now points to the start of the assertion. Another
794 change is that when a callout appears before a group, the "length of next
795 pattern item" that is passed now just gives the length of the opening
796 parenthesis item, not the length of the whole group. A length of zero is now
797 given only for a callout at the end of the pattern. Automatic callouts are no
798 longer inserted before and after explicit callouts in the pattern.
799
800 A number of bugs in the refactored code were subsequently fixed during testing
801 before release, but after the code was made available in the repository. Many
802 of the bugs were discovered by fuzzing testing. Several of them were related to
803 the change from assuming a zero-terminated pattern (which previously had
804 required non-zero terminated strings to be copied). These bugs were never in
805 fully released code, but are noted here for the record.
806
807 (a) An overall recursion such as (?0) inside a lookbehind assertion was not
808 being diagnosed as an error.
809
810 (b) In utf mode, the length of a *MARK (or other verb) name was being checked
811 in characters instead of code units, which could lead to bad code being
812 compiled, leading to unpredictable behaviour.
813
814 (c) In extended /x mode, characters whose code was greater than 255 caused
815 a lookup outside one of the global tables. A similar bug existed for wide
816 characters in *VERB names.
817
818 (d) The amount of memory needed for a compiled pattern was miscalculated if a
819 lookbehind contained more than one toplevel branch and the first branch
820 was of length zero.
821
822 (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
823 terminated pattern, if a # comment ran on to the end of the pattern, one
824 or more code units past the end were being read.
825
826 (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
827 "{2,2") could cause reading beyond the pattern.
828
829 (g) When reading a callout string, if the end delimiter was at the end of the
830 pattern one further code unit was read.
831
832 (h) An unterminated number after \g' could cause reading beyond the pattern.
833
834 (i) An insufficient memory size was being computed for compiling with
835 PCRE2_AUTO_CALLOUT.
836
837 (j) A conditional group with an assertion condition used more memory than was
838 allowed for it during parsing, so too many of them could therefore
839 overrun a buffer.
840
841 (k) If parsing a pattern exactly filled the buffer, the internal test for
842 overrun did not check when the final META_END item was added.
843
844 (l) If a lookbehind contained a subroutine call, and the called group
845 contained an option setting such as (?s), and the PCRE2_ANCHORED option
846 was set, unpredictable behaviour could occur. The underlying bug was
847 incorrect code and insufficient checking while searching for the end of
848 the called subroutine in the parsed pattern.
849
850 (m) Quantifiers following (*VERB)s were not being diagnosed as errors.
851
852 (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
853 PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
854
855 (o) If \Q was preceded by a quantified item, and the following \E was
856 followed by '?' or '+', and there was at least one literal character
857 between them, an internal error "unexpected repeat" occurred (example:
858 /.+\QX\E+/).
859
860 (p) A buffer overflow could occur while sorting the names in the group name
861 list (depending on the order in which the names were seen).
862
863 (q) A conditional group that started with a callout was not doing the right
864 check for a following assertion, leading to compiling bad code. Example:
865 /(?(C'XX))?!XX/
866
867 (r) If a character whose code point was greater than 0xffff appeared within
868 a lookbehind that was within another lookbehind, the calculation of the
869 lookbehind length went wrong and could provoke an internal error.
870
871 (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
872 an internal error. Now the hyphen is treated as a literal.
873
874 4. Back references are now permitted in lookbehind assertions when there are
875 no duplicated group numbers (that is, (?| has not been used), and, if the
876 reference is by name, there is only one group of that name. The referenced
877 group must, of course be of fixed length.
878
879 5. pcre2test has been upgraded so that, when run under valgrind with valgrind
880 support enabled, reading past the end of the pattern is detected, both when
881 compiling and during callout processing.
882
883 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
884 reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
885 not recognize this syntax.
886
887 7. Automatic callouts are no longer generated before and after callouts in the
888 pattern.
889
890 8. When pcre2test was outputing information from a callout, the caret indicator
891 for the current position in the subject line was incorrect if it was after an
892 escape sequence for a character whose code point was greater than \x{ff}.
893
894 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
895 PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
896
897 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
898 expansion when long lines are encountered. Original patch by Dmitry
899 Cherniachenko.
900
901 11. If pcre2grep was compiled with JIT support, but the library was compiled
902 without it (something that neither ./configure nor CMake allow, but it can be
903 done by editing config.h), pcre2grep was giving a JIT error. Now it detects
904 this situation and does not try to use JIT.
905
906 12. Added some "const" qualifiers to variables in pcre2grep.
907
908 13. Added Dmitry Cherniachenko's patch for colouring output in Windows
909 (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
910 variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
911
912 14. Add the -t (grand total) option to pcre2grep.
913
914 15. A number of bugs have been mended relating to match start-up optimizations
915 when the first thing in a pattern is a positive lookahead. These all applied
916 only when PCRE2_NO_START_OPTIMIZE was *not* set:
917
918 (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
919 both an initial 'X' and a following 'X'.
920 (b) Some patterns starting with an assertion that started with .* were
921 incorrectly optimized as having to match at the start of the subject or
922 after a newline. There are cases where this is not true, for example,
923 (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
924 start with spaces. Starting .* in an assertion is no longer taken as an
925 indication of matching at the start (or after a newline).
926
927 16. The "offset" modifier in pcre2test was not being ignored (as documented)
928 when the POSIX API was in use.
929
930 17. Added --enable-fuzz-support to "configure", causing an non-installed
931 library containing a test function that can be called by fuzzers to be
932 compiled. A non-installed binary to run the test function locally, called
933 pcre2fuzzcheck is also compiled.
934
935 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
936 which started with .* inside a positive lookahead was incorrectly being
937 compiled as implicitly anchored.
938
939 19. Removed all instances of "register" declarations, as they are considered
940 obsolete these days and in any case had become very haphazard.
941
942 20. Add strerror() to pcre2test for failed file opening.
943
944 21. Make pcre2test -C list valgrind support when it is enabled.
945
946 22. Add the use_length modifier to pcre2test.
947
948 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
949 'copy' modifiers.
950
951 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
952 is apparently needed there as well as in the function definitions. (Why did
953 nobody ask for this in PCRE1?)
954
955 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
956 PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
957 compliant and unique.
958
959 26. pcre2-config --libs-posix was listing -lpcre2posix instead of
960 -lpcre2-posix. Also, the CMake build process was building the library with the
961 wrong name.
962
963 27. In pcre2test, give some offset information for errors in hex patterns.
964 This uses the C99 formatting sequence %td, except for MSVC which doesn't
965 support it - %lu is used instead.
966
967 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
968 pcre2test for testing it.
969
970 29. Fix small memory leak in pcre2test.
971
972 30. Fix out-of-bounds read for partial matching of /./ against an empty string
973 when the newline type is CRLF.
974
975 31. Fix a bug in pcre2test that caused a crash when a locale was set either in
976 the current pattern or a previous one and a wide character was matched.
977
978 32. The appearance of \p, \P, or \X in a substitution string when
979 PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
980 dereference).
981
982 33. If the starting offset was specified as greater than the subject length in
983 a call to pcre2_substitute() an out-of-bounds memory reference could occur.
984
985 34. When PCRE2 was compiled to use the heap instead of the stack for recursive
986 calls to match(), a repeated minimizing caseless back reference, or a
987 maximizing one where the two cases had different numbers of code units,
988 followed by a caseful back reference, could lose the caselessness of the first
989 repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
990 but didn't).
991
992 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
993 matching length and just records zero. Typically this happens when there are
994 too many nested or recursive back references. If the limit was reached in
995 certain recursive cases it failed to be triggered and an internal error could
996 be the result.
997
998 36. The pcre2_dfa_match() function now takes note of the recursion limit for
999 the internal recursive calls that are used for lookrounds and recursions within
1000 the pattern.
1001
1002 37. More refactoring has got rid of the internal could_be_empty_branch()
1003 function (around 400 lines of code, including comments) by keeping track of
1004 could-be-emptiness as the pattern is compiled instead of scanning compiled
1005 groups. (This would have been much harder before the refactoring of #3 above.)
1006 This lifts a restriction on the number of branches in a group (more than about
1007 1100 would give "pattern is too complicated").
1008
1009 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
1010 auto_callout".
1011
1012 39. In a library with Unicode support, incorrect data was compiled for a
1013 pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
1014 characters to match (for example, /[\s[:^ascii:]]/).
1015
1016 40. The callout_error modifier has been added to pcre2test to make it possible
1017 to return PCRE2_ERROR_CALLOUT from a callout.
1018
1019 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
1020 "<esc>[00m".
1021
1022 42. The limit in the auto-possessification code that was intended to catch
1023 overly-complicated patterns and not spend too much time auto-possessifying was
1024 being reset too often, resulting in very long compile times for some patterns.
1025 Now such patterns are no longer completely auto-possessified.
1026
1027 43. Applied Jason Hood's revised patch for RunTest.bat.
1028
1029 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
1030
1031 45. Minor cosmetic fix to pcre2test: move a variable that is not used under
1032 Windows into the "not Windows" code.
1033
1034 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
1035 some of the code:
1036
1037 * normalised the Windows condition by ensuring WIN32 is defined;
1038 * enables the callout feature under Windows;
1039 * adds globbing (Microsoft's implementation expands quoted args),
1040 using a tweaked opendirectory;
1041 * implements the is_*_tty functions for Windows;
1042 * --color=always will write the ANSI sequences to file;
1043 * add sequences 4 (underline works on Win10) and 5 (blink as bright
1044 background, relatively standard on DOS/Win);
1045 * remove the (char *) casts for the now-const strings;
1046 * remove GREP_COLOUR (grep's command line allowed the 'u', but not
1047 the environment), parsing GREP_COLORS instead;
1048 * uses the current colour if not set, rather than black;
1049 * add print_match for the undefined case;
1050 * fixes a typo.
1051
1052 In addition, colour settings containing anything other than digits and
1053 semicolon are ignored, and the colour controls are no longer output for empty
1054 strings.
1055
1056 47. Detecting patterns that are too large inside the length-measuring loop
1057 saves processing ridiculously long patterns to their end.
1058
1059 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
1060 just wastes time. In the UTF case it can also produce redundant entries in
1061 XCLASS lists caused by characters with multiple other cases and pairs of
1062 characters in the same "not-x" sublists.
1063
1064 49. A pattern such as /(?=(a\K))/ can report the end of the match being before
1065 its start; pcre2test was not handling this correctly when using the POSIX
1066 interface (it was OK with the native interface).
1067
1068 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
1069 continue to work, falling back to interpretation if anything goes wrong with
1070 JIT.
1071
1072 51. Applied patches from Christian Persch to configure.ac to make use of the
1073 AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT
1074 modules.
1075
1076 52. Minor fixes to pcre2grep from Jason Hood:
1077 * fixed some spacing;
1078 * Windows doesn't usually use single quotes, so I've added a define
1079 to use appropriate quotes [in an example];
1080 * LC_ALL was displayed as "LCC_ALL";
1081 * numbers 11, 12 & 13 should end in "th";
1082 * use double quotes in usage message.
1083
1084 53. When autopossessifying, skip empty branches without recursion, to reduce
1085 stack usage for the benefit of clang with -fsanitize-address, which uses huge
1086 stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553.
1087
1088 54. A pattern with very many explicit back references to a group that is a long
1089 way from the start of the pattern could take a long time to compile because
1090 searching for the referenced group in order to find the minimum length was
1091 being done repeatedly. Now up to 128 group minimum lengths are cached and the
1092 attempt to find a minimum length is abandoned if there is a back reference to a
1093 group whose number is greater than 128. (In that case, the pattern is so
1094 complicated that this optimization probably isn't worth it.) This fixes
1095 oss-fuzz issue 557.
1096
1097 55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline
1098 mode with --only-matching matched several lines, it restarted scanning at the
1099 next line instead of moving on to the end of the matched string, which can be
1100 several lines after the start.
1101
1102 56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line
1103 with updates to the non-Windows version.
1104
1105
1106
1107 Version 10.22 29-July-2016
1108 --------------------------
1109
1110 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
1111 to fix problems with running the tests under Windows.
1112
1113 2. Implemented a facility for quoting literal characters within hexadecimal
1114 patterns in pcre2test, to make it easier to create patterns with just a few
1115 non-printing characters.
1116
1117 3. Binary zeros are not supported in pcre2test input files. It now detects them
1118 and gives an error.
1119
1120 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
1121 smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
1122 that it matches only unknown objects.
1123
1124 5. Updated the maintenance script maint/ManyConfigTests to make it easier to
1125 select individual groups of tests.
1126
1127 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
1128 used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
1129 disables the use of back references (and subroutine calls), which are supported
1130 by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
1131 longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
1132 and pmatch when regexec() is called.
1133
1134 7. Because of 6 above, pcre2test has been modified with a new modifier called
1135 posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
1136 modifier had this effect. That option is now ignored when the POSIX API is in
1137 use.
1138
1139 8. Minor tidies to the pcre2demo.c sample program, including more comments
1140 about its 8-bit-ness.
1141
1142 9. Detect unmatched closing parentheses and give the error in the pre-scan
1143 instead of later. Previously the pre-scan carried on and could give a
1144 misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
1145 message about invalid duplicate group names.
1146
1147 10. It has happened that pcre2test was accidentally linked with another POSIX
1148 regex library instead of libpcre2-posix. In this situation, a call to regcomp()
1149 (in the other library) may succeed, returning zero, but of course putting its
1150 own data into the regex_t block. In one example the re_pcre2_code field was
1151 left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
1152 so it treated the next line as another pattern line, resulting in a confusing
1153 error message. A check has been added to pcre2test to see if the data returned
1154 from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
1155 are not, an error message is output and the pcre2test run is abandoned. The
1156 message points out the possibility of a mis-linking. Hopefully this will avoid
1157 some head-scratching the next time this happens.
1158
1159 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
1160 assertion, caused pcre2test to output a very large number of spaces when the
1161 callout was taken, making the program appearing to loop.
1162
1163 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
1164 nested set of parentheses of sufficient size caused an overflow of the
1165 compiling workspace (which was diagnosed, but of course is not desirable).
1166
1167 13. Detect missing closing parentheses during the pre-pass for group
1168 identification.
1169
1170 14. Changed some integer variable types and put in a number of casts, following
1171 a report of compiler warnings from Visual Studio 2013 and a few tests with
1172 gcc's -Wconversion (which still throws up a lot).
1173
1174 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
1175 for testing it.
1176
1177 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
1178 regerror(). When the error buffer is too small, my version of snprintf() puts a
1179 binary zero in the final byte. Bug #1801 seems to show that other versions do
1180 not do this, leading to bad output from pcre2test when it was checking for
1181 buffer overflow. It no longer assumes a binary zero at the end of a too-small
1182 regerror() buffer.
1183
1184 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
1185 actually affect anything, by sheer luck.
1186
1187 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
1188 "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
1189 older MSVC compilers. This has been done both in src/pcre2_internal.h for most
1190 of the library, and also in src/pcre2posix.c, which no longer includes
1191 pcre2_internal.h (see 24 below).
1192
1193 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
1194 static compilation. Subsequently applied Chris Wilson's second patch, putting
1195 the first patch under a new option instead of being unconditional when
1196 PCRE_STATIC is set.
1197
1198 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
1199 to convert \r\n at the ends of reflected lines into \r\r\n. This required
1200 ensuring that other output that is written to stdout (e.g. file names) uses the
1201 appropriate line terminator: \r\n for Windows, \n otherwise.
1202
1203 21. When a line is too long for pcre2grep's internal buffer, show the maximum
1204 length in the error message.
1205
1206 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
1207 additions).
1208
1209 23. RunTest.bat was missing a "set type" line for test 22.
1210
1211 24. The pcre2posix.c file was including pcre2_internal.h, and using some
1212 "private" knowledge of the data structures. This is unnecessary; the code has
1213 been re-factored and no longer includes pcre2_internal.h.
1214
1215 25. A racing condition is fixed in JIT reported by Mozilla.
1216
1217 26. Minor code refactor to avoid "array subscript is below array bounds"
1218 compiler warning.
1219
1220 27. Minor code refactor to avoid "left shift of negative number" warning.
1221
1222 28. Add a bit more sanity checking to pcre2_serialize_decode() and document
1223 that it expects trusted data.
1224
1225 29. Fix typo in pcre2_jit_test.c
1226
1227 30. Due to an oversight, pcre2grep was not making use of JIT when available.
1228 This is now fixed.
1229
1230 31. The RunGrepTest script is updated to use the valgrind suppressions file
1231 when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
1232 file is updated so that is now the same as for PCRE1: it suppresses the
1233 Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
1234 code). Also changed smc-check=all to smc-check=all-non-file as was done for
1235 RunTest (see 4 above).
1236
1237 32. Implemented the PCRE2_NO_JIT option for pcre2_match().
1238
1239 33. Fix typo that gave a compiler error when JIT not supported.
1240
1241 34. Fix comment describing the returns from find_fixedlength().
1242
1243 35. Fix potential negative index in pcre2test.
1244
1245 36. Calls to pcre2_get_error_message() with error numbers that are never
1246 returned by PCRE2 functions were returning empty strings. Now the error code
1247 PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
1248 show the texts for given error numbers (i.e. to call pcre2_get_error_message()
1249 and display what it returns) and a few representative error codes are now
1250 checked in RunTest.
1251
1252 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
1253 pcre2_match.c, in anticipation that this is needed for the same reason it was
1254 recently added to pcrecpp.cc in PCRE1.
1255
1256 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
1257 the match extended over a line boundary, as it tried to find more matches "on
1258 the same line" - but it was already over the end.
1259
1260 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
1261 to the same code as '.' when PCRE2_DOTALL is set).
1262
1263 40. Fix two clang compiler warnings in pcre2test when only one code unit width
1264 is supported.
1265
1266 41. Upgrade RunTest to automatically re-run test 2 with a large (64MiB) stack
1267 if it fails when running the interpreter with a 16MiB stack (and if changing
1268 the stack size via pcre2test is possible). This avoids having to manually set a
1269 large stack size when testing with clang.
1270
1271 42. Fix register overwite in JIT when SSE2 acceleration is enabled.
1272
1273 43. Detect integer overflow in pcre2test pattern and data repetition counts.
1274
1275 44. In pcre2test, ignore "allcaptures" after DFA matching.
1276
1277 45. Fix unaligned accesses on x86. Patch by Marc Mutz.
1278
1279 46. Fix some more clang compiler warnings.
1280
1281
1282 Version 10.21 12-January-2016
1283 -----------------------------
1284
1285 1. Improve matching speed of patterns starting with + or * in JIT.
1286
1287 2. Use memchr() to find the first character in an unanchored match in 8-bit
1288 mode in the interpreter. This gives a significant speed improvement.
1289
1290 3. Removed a redundant copy of the opcode_possessify table in the
1291 pcre2_auto_possessify.c source.
1292
1293 4. Fix typos in dftables.c for z/OS.
1294
1295 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
1296 processing them could involve a buffer overflow if the following character was
1297 an opening parenthesis.
1298
1299 6. Change 36 for 10.20 also introduced a bug in processing this pattern:
1300 /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
1301 setting (which (*:0) is), then (?x) did not get unset at the end of its group
1302 during the scan for named groups, and hence the external # was incorrectly
1303 treated as a comment and the invalid (?' at the end of the pattern was not
1304 diagnosed. This caused a buffer overflow during the real compile. This bug was
1305 discovered by Karl Skomski with the LLVM fuzzer.
1306
1307 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
1308 own source module to avoid a circular dependency between src/pcre2_compile.c
1309 and src/pcre2_study.c
1310
1311 8. A callout with a string argument containing an opening square bracket, for
1312 example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
1313 overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1314
1315 9. The handling of callouts during the pre-pass for named group identification
1316 has been tightened up.
1317
1318 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
1319 possessive. This is a very minor optimization.
1320
1321 11. A possessively repeated conditional group that could match an empty string,
1322 for example, /(?(R))*+/, was incorrectly compiled.
1323
1324 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
1325 Persch).
1326
1327 13. An empty comment (?#) in a pattern was incorrectly processed and could
1328 provoke a buffer overflow. This bug was discovered by Karl Skomski with the
1329 LLVM fuzzer.
1330
1331 14. Fix infinite recursion in the JIT compiler when certain patterns such as
1332 /(?:|a|){100}x/ are analysed.
1333
1334 15. Some patterns with character classes involving [: and \\ were incorrectly
1335 compiled and could cause reading from uninitialized memory or an incorrect
1336 error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
1337 first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
1338
1339 16. Pathological patterns containing many nested occurrences of [: caused
1340 pcre2_compile() to run for a very long time. This bug was found by the LLVM
1341 fuzzer.
1342
1343 17. A missing closing parenthesis for a callout with a string argument was not
1344 being diagnosed, possibly leading to a buffer overflow. This bug was found by
1345 the LLVM fuzzer.
1346
1347 18. A conditional group with only one branch has an implicit empty alternative
1348 branch and must therefore be treated as potentially matching an empty string.
1349
1350 19. If (?R was followed by - or + incorrect behaviour happened instead of a
1351 diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1352
1353 20. Another bug that was introduced by change 36 for 10.20: conditional groups
1354 whose condition was an assertion preceded by an explicit callout with a string
1355 argument might be incorrectly processed, especially if the string contained \Q.
1356 This bug was discovered by Karl Skomski with the LLVM fuzzer.
1357
1358 21. Compiling PCRE2 with the sanitize options of clang showed up a number of
1359 very pedantic coding infelicities and a buffer overflow while checking a UTF-8
1360 string if the final multi-byte UTF-8 character was truncated.
1361
1362 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
1363 class, where both values are literal letters in the same case, omit the
1364 non-letter EBCDIC code points within the range.
1365
1366 23. Finding the minimum matching length of complex patterns with back
1367 references and/or recursions can take a long time. There is now a cut-off that
1368 gives up trying to find a minimum length when things get too complex.
1369
1370 24. An optimization has been added that speeds up finding the minimum matching
1371 length for patterns containing repeated capturing groups or recursions.
1372
1373 25. If a pattern contained a back reference to a group whose number was
1374 duplicated as a result of appearing in a (?|...) group, the computation of the
1375 minimum matching length gave a wrong result, which could cause incorrect "no
1376 match" errors. For such patterns, a minimum matching length cannot at present
1377 be computed.
1378
1379 26. Added a check for integer overflow in conditions (?(<digits>) and
1380 (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
1381 fuzzer.
1382
1383 27. Fixed an issue when \p{Any} inside an xclass did not read the current
1384 character.
1385
1386 28. If pcre2grep was given the -q option with -c or -l, or when handling a
1387 binary file, it incorrectly wrote output to stdout.
1388
1389 29. The JIT compiler did not restore the control verb head in case of *THEN
1390 control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1391
1392 30. The way recursive references such as (?3) are compiled has been re-written
1393 because the old way was the cause of many issues. Now, conversion of the group
1394 number into a pattern offset does not happen until the pattern has been
1395 completely compiled. This does mean that detection of all infinitely looping
1396 recursions is postponed till match time. In the past, some easy ones were
1397 detected at compile time. This re-writing was done in response to yet another
1398 bug found by the LLVM fuzzer.
1399
1400 31. A test for a back reference to a non-existent group was missing for items
1401 such as \987. This caused incorrect code to be compiled. This issue was found
1402 by Karl Skomski with a custom LLVM fuzzer.
1403
1404 32. Error messages for syntax errors following \g and \k were giving inaccurate
1405 offsets in the pattern.
1406
1407 33. Improve the performance of starting single character repetitions in JIT.
1408
1409 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
1410
1411 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
1412 give the right offset instead of zero.
1413
1414 36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
1415 This issue was found by Karl Skomski with a custom LLVM fuzzer.
1416
1417 37. The JIT compiler should restore the control chain for empty possessive
1418 repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1419
1420 38. A bug which was introduced by the single character repetition optimization
1421 was fixed.
1422
1423 39. Match limit check added to recursion. This issue was found by Karl Skomski
1424 with a custom LLVM fuzzer.
1425
1426 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
1427 only at the part of the subject that is relevant when the starting offset is
1428 non-zero.
1429
1430 41. Improve first character match in JIT with SSE2 on x86.
1431
1432 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
1433 with a custom LLVM fuzzer.
1434
1435 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
1436 III).
1437
1438 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
1439 test (there are now 20 in total).
1440
1441 45. Fixed a corner case of range optimization in JIT.
1442
1443 46. Add the ${*MARK} facility to pcre2_substitute().
1444
1445 47. Modifier lists in pcre2test were splitting at spaces without the required
1446 commas.
1447
1448 48. Implemented PCRE2_ALT_VERBNAMES.
1449
1450 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
1451 LLVM fuzzer.
1452
1453 50. The pcre2test program has been extended by adding the #newline_default
1454 command. This has made it possible to run the standard tests when PCRE2 is
1455 compiled with either CR or CRLF as the default newline convention. As part of
1456 this work, the new command was added to several test files and the testing
1457 scripts were modified. The pcre2grep tests can now also be run when there is no
1458 LF in the default newline convention.
1459
1460 51. The RunTest script has been modified so that, when JIT is used and valgrind
1461 is specified, a valgrind suppressions file is set up to ignore "Invalid read of
1462 size 16" errors because these are false positives when the hardware supports
1463 the SSE2 instruction set.
1464
1465 52. It is now possible to have comment lines amid the subject strings in
1466 pcre2test (and perltest.sh) input.
1467
1468 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
1469
1470 54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
1471 and the matching functions with NULL contexts can be tested.
1472
1473 55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
1474
1475 56. In a character class such as [\W\p{Any}] where both a negative-type escape
1476 ("not a word character") and a property escape were present, the property
1477 escape was being ignored.
1478
1479 57. Fixed integer overflow for patterns whose minimum matching length is very,
1480 very large.
1481
1482 58. Implemented --never-backslash-C.
1483
1484 59. Change 55 above introduced a bug by which certain patterns provoked the
1485 erroneous error "\ at end of pattern".
1486
1487 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
1488 errors or other strange effects if compiled in UCP mode. Found with libFuzzer
1489 and AddressSanitizer.
1490
1491 61. Whitespace at the end of a pcre2test pattern line caused a spurious error
1492 message if there were only single-character modifiers. It should be ignored.
1493
1494 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
1495 or segmentation errors for some patterns. Found with libFuzzer and
1496 AddressSanitizer.
1497
1498 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
1499 overflow.
1500
1501 64. Improve error message for overly-complicated patterns.
1502
1503 65. Implemented an optional replication feature for patterns in pcre2test, to
1504 make it easier to test long repetitive patterns. The tests for 63 above are
1505 converted to use the new feature.
1506
1507 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
1508 misbehave.
1509
1510 67. In pcre2_substitute() in UTF mode, the UTF validity check on the
1511 replacement string was happening before the length setting when the replacement
1512 string was zero-terminated.
1513
1514 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the
1515 second and subsequent calls to pcre2_match().
1516
1517 69. There was no check for integer overflow for a replacement group number in
1518 pcre2_substitute(). An added check for a number greater than the largest group
1519 number in the pattern means this is not now needed.
1520
1521 70. The PCRE2-specific VERSION condition didn't work correctly if only one
1522 digit was given after the decimal point, or if more than two digits were given.
1523 It now works with one or two digits, and gives a compile time error if more are
1524 given.
1525
1526 71. In pcre2_substitute() there was the possibility of reading one code unit
1527 beyond the end of the replacement string.
1528
1529 72. The code for checking a subject's UTF-32 validity for a pattern with a
1530 lookbehind involved an out-of-bounds pointer, which could potentially cause
1531 trouble in some environments.
1532
1533 73. The maximum lookbehind length was incorrectly calculated for patterns such
1534 as /(?<=(a)(?-1))x/ which have a recursion within a backreference.
1535
1536 74. Give an error if a lookbehind assertion is longer than 65535 code units.
1537
1538 75. Give an error in pcre2_substitute() if a match ends before it starts (as a
1539 result of the use of \K).
1540
1541 76. Check the length of subpattern names and the names in (*MARK:xx) etc.
1542 dynamically to avoid the possibility of integer overflow.
1543
1544 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the
1545 size of patterns that they are prepared to handle.
1546
1547 78. (*NO_AUTO_POSSESS) was not working.
1548
1549 79. Adding group information caching improves the speed of compiling when
1550 checking whether a group has a fixed length and/or could match an empty string,
1551 especially when recursion or subroutine calls are involved. However, this
1552 cannot be used when (?| is present in the pattern because the same number may
1553 be used for groups of different sizes. To catch runaway patterns in this
1554 situation, counts have been introduced to the functions that scan for empty
1555 branches or compute fixed lengths.
1556
1557 80. Allow for the possibility of the size of the nest_save structure not being
1558 a factor of the size of the compiling workspace (it currently is).
1559
1560 81. Check for integer overflow in minimum length calculation and cap it at
1561 65535.
1562
1563 82. Small optimizations in code for finding the minimum matching length.
1564
1565 83. Lock out configuring for EBCDIC with non-8-bit libraries.
1566
1567 84. Test for error code <= 0 in regerror().
1568
1569 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute().
1570
1571 86. Avoid the possibility of computing with an out-of-bounds pointer (though
1572 not dereferencing it) while handling lookbehind assertions.
1573
1574 87. Failure to get memory for the match data in regcomp() is now given as a
1575 regcomp() error instead of waiting for regexec() to pick it up.
1576
1577 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid
1578 newline sequence.
1579
1580 89. Paranoid check in regcomp() for bad error code from pcre2_compile().
1581
1582 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well
1583 as for link size 2.
1584
1585 91. Document that JIT has a limit on pattern size, and give more information
1586 about JIT compile failures in pcre2test.
1587
1588 92. Implement PCRE2_INFO_HASBACKSLASHC.
1589
1590 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports
1591 with JIT (possibly caused by SSE2?).
1592
1593 94. Support offset_limit in JIT.
1594
1595 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed
1596 by a single ASCII character in a class item, was incorrectly compiled in UCP
1597 mode. The POSIX class got lost, but only if the single character followed it.
1598
1599 96. [:punct:] in UCP mode was matching some characters in the range 128-255
1600 that should not have been matched.
1601
1602 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all
1603 characters with code points greater than 255 are in the class. When a Unicode
1604 property was also in the class (if PCRE2_UCP is set, escapes such as \w are
1605 turned into Unicode properties), wide characters were not correctly handled,
1606 and could fail to match.
1607
1608 98. In pcre2test, make the "startoffset" modifier a synonym of "offset",
1609 because it sets the "startoffset" parameter for pcre2_match().
1610
1611 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between
1612 an item and its qualifier (for example, A(?#comment)?B) pcre2_compile()
1613 misbehaved. This bug was found by the LLVM fuzzer.
1614
1615 100. The error for an invalid UTF pattern string always gave the code unit
1616 offset as zero instead of where the invalidity was found.
1617
1618 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not
1619 working correctly in UCP mode.
1620
1621 102. Similar to 99 above, if an isolated \E was present between an item and its
1622 qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug
1623 was found by the LLVM fuzzer.
1624
1625 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND
1626 was set when the pmatch argument was NULL. It now returns REG_INVARG.
1627
1628 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep.
1629
1630 105. An empty \Q\E sequence between an item and its qualifier caused
1631 pcre2_compile() to misbehave when auto callouts were enabled. This bug
1632 was found by the LLVM fuzzer.
1633
1634 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or
1635 other verb "name" ended with whitespace immediately before the closing
1636 parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when
1637 both those options were set.
1638
1639 107. In a number of places pcre2_compile() was not handling NULL characters
1640 correctly, and pcre2test with the "bincode" modifier was not always correctly
1641 displaying fields containing NULLS:
1642
1643 (a) Within /x extended #-comments
1644 (b) Within the "name" part of (*MARK) and other *verbs
1645 (c) Within the text argument of a callout
1646
1647 108. If a pattern that was compiled with PCRE2_EXTENDED started with white
1648 space or a #-type comment that was followed by (?-x), which turns off
1649 PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again,
1650 pcre2_compile() assumed that (?-x) applied to the whole pattern and
1651 consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix
1652 for this bug means that a setting of any of the (?imsxJU) options at the start
1653 of a pattern is no longer transferred to the options that are returned by
1654 PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have
1655 changed when the effects of those options were all moved to compile time.
1656
1657 109. An escaped closing parenthesis in the "name" part of a (*verb) when
1658 PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug
1659 was found by the LLVM fuzzer.
1660
1661 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it
1662 possible to test it.
1663
1664 111. "Harden" pcre2test against ridiculously large values in modifiers and
1665 command line arguments.
1666
1667 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_
1668 LENGTH.
1669
1670 113. Fix printing of *MARK names that contain binary zeroes in pcre2test.
1671
1672
1673 Version 10.20 30-June-2015
1674 --------------------------
1675
1676 1. Callouts with string arguments have been added.
1677
1678 2. Assertion code generator in JIT has been optimized.
1679
1680 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
1681 pcre2_compile() function read past the end of the input before diagnosing an
1682 error. This bug was discovered by the LLVM fuzzer.
1683
1684 4. Implemented pcre2_callout_enumerate().
1685
1686 5. Fix JIT compilation of conditional blocks whose assertion is converted to
1687 (*FAIL). E.g: /(?(?!))/.
1688
1689 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
1690 discovered by the LLVM fuzzer.
1691
1692 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
1693 when this assertion was used as a condition, for example (?(?!)a|b). In
1694 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
1695 error about an unsupported item.
1696
1697 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
1698 possessification code could take exponential time to complete. A recursion
1699 depth limit of 1000 has been imposed to limit the resources used by this
1700 optimization. This infelicity was discovered by the LLVM fuzzer.
1701
1702 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
1703 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
1704 because \S ensures they are all in the class. The code for doing this was
1705 interacting badly with the code for computing the amount of space needed to
1706 compile the pattern, leading to a buffer overflow. This bug was discovered by
1707 the LLVM fuzzer.
1708
1709 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
1710 other kinds of group caused stack overflow at compile time. This bug was
1711 discovered by the LLVM fuzzer.
1712
1713 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment
1714 between a subroutine call and its quantifier was incorrectly compiled, leading
1715 to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer.
1716
1717 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an
1718 assertion after (?(. The code was failing to check the character after (?(?<
1719 for the ! or = that would indicate a lookbehind assertion. This bug was
1720 discovered by the LLVM fuzzer.
1721
1722 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with
1723 a fixed maximum following a group that contains a subroutine reference was
1724 incorrectly compiled and could trigger buffer overflow. This bug was discovered
1725 by the LLVM fuzzer.
1726
1727 14. Negative relative recursive references such as (?-7) to non-existent
1728 subpatterns were not being diagnosed and could lead to unpredictable behaviour.
1729 This bug was discovered by the LLVM fuzzer.
1730
1731 15. The bug fixed in 14 was due to an integer variable that was unsigned when
1732 it should have been signed. Some other "int" variables, having been checked,
1733 have either been changed to uint32_t or commented as "must be signed".
1734
1735 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1)))
1736 caused a stack overflow instead of the diagnosis of a non-fixed length
1737 lookbehind assertion. This bug was discovered by the LLVM fuzzer.
1738
1739 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern
1740 (e.g. /(?<=\Ka)/) could make pcre2grep loop.
1741
1742 18. There was a similar problem to 17 in pcre2test for global matches, though
1743 the code there did catch the loop.
1744
1745 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*),
1746 and a subsequent item in the pattern caused a non-match, backtracking over the
1747 repeated \X did not stop, but carried on past the start of the subject, causing
1748 reference to random memory and/or a segfault. There were also some other cases
1749 where backtracking after \C could crash. This set of bugs was discovered by the
1750 LLVM fuzzer.
1751
1752 20. The function for finding the minimum length of a matching string could take
1753 a very long time if mutual recursion was present many times in a pattern, for
1754 example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has
1755 been implemented. This infelicity was discovered by the LLVM fuzzer.
1756
1757 21. Implemented PCRE2_NEVER_BACKSLASH_C.
1758
1759 22. The feature for string replication in pcre2test could read from freed
1760 memory if the replication required a buffer to be extended, and it was not
1761 working properly in 16-bit and 32-bit modes. This issue was discovered by a
1762 fuzzer: see http://lcamtuf.coredump.cx/afl/.
1763
1764 23. Added the PCRE2_ALT_CIRCUMFLEX option.
1765
1766 24. Adjust the treatment of \8 and \9 to be the same as the current Perl
1767 behaviour.
1768
1769 25. Static linking against the PCRE2 library using the pkg-config module was
1770 failing on missing pthread symbols.
1771
1772 26. If a group that contained a recursive back reference also contained a
1773 forward reference subroutine call followed by a non-forward-reference
1774 subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
1775 compile correct code, leading to undefined behaviour or an internally detected
1776 error. This bug was discovered by the LLVM fuzzer.
1777
1778 27. Quantification of certain items (e.g. atomic back references) could cause
1779 incorrect code to be compiled when recursive forward references were involved.
1780 For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was
1781 discovered by the LLVM fuzzer.
1782
1783 28. A repeated conditional group whose condition was a reference by name caused
1784 a buffer overflow if there was more than one group with the given name. This
1785 bug was discovered by the LLVM fuzzer.
1786
1787 29. A recursive back reference by name within a group that had the same name as
1788 another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/.
1789 This bug was discovered by the LLVM fuzzer.
1790
1791 30. A forward reference by name to a group whose number is the same as the
1792 current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a
1793 buffer overflow at compile time. This bug was discovered by the LLVM fuzzer.
1794
1795 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1
1796 as an int; fixed by writing it as 1u).
1797
1798 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives
1799 a warning for "fileno" unless -std=gnu99 us used.
1800
1801 33. A lookbehind assertion within a set of mutually recursive subpatterns could
1802 provoke a buffer overflow. This bug was discovered by the LLVM fuzzer.
1803
1804 34. Give an error for an empty subpattern name such as (?'').
1805
1806 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains
1807 \P, \p, or \X.
1808
1809 36. The way named subpatterns are handled has been refactored. There is now a
1810 pre-pass over the regex which does nothing other than identify named
1811 subpatterns and count the total captures. This means that information about
1812 named patterns is known before the rest of the compile. In particular, it means
1813 that forward references can be checked as they are encountered. Previously, the
1814 code for handling forward references was contorted and led to several errors in
1815 computing the memory requirements for some patterns, leading to buffer
1816 overflows.
1817
1818 37. There was no check for integer overflow in subroutine calls such as (?123).
1819
1820 38. The table entry for \l in EBCDIC environments was incorrect, leading to its
1821 being treated as a literal 'l' instead of causing an error.
1822
1823 39. If a non-capturing group containing a conditional group that could match
1824 an empty string was repeated, it was not identified as matching an empty string
1825 itself. For example: /^(?:(?(1)x|)+)+$()/.
1826
1827 40. In an EBCDIC environment, pcretest was mishandling the escape sequences
1828 \a and \e in test subject lines.
1829
1830 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII
1831 instead of the EBCDIC value.
1832
1833 42. The handling of \c in an EBCDIC environment has been revised so that it is
1834 now compatible with the specification in Perl's perlebcdic page.
1835
1836 43. Single character repetition in JIT has been improved. 20-30% speedup
1837 was achieved on certain patterns.
1838
1839 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in
1840 ASCII/Unicode. This has now been added to the list of characters that are
1841 recognized as white space in EBCDIC.
1842
1843 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave
1844 an error (correctly) when used outside a class, but did not give an error
1845 within a class.
1846
1847 46. \h within a class was incorrectly compiled in EBCDIC environments.
1848
1849 47. JIT should return with error when the compiled pattern requires
1850 more stack space than the maximum.
1851
1852 48. Fixed a memory leak in pcre2grep when a locale is set.
1853
1854
1855 Version 10.10 06-March-2015
1856 ---------------------------
1857
1858 1. When a pattern is compiled, it remembers the highest back reference so that
1859 when matching, if the ovector is too small, extra memory can be obtained to
1860 use instead. A conditional subpattern whose condition is a check on a capture
1861 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
1862 another kind of back reference, but it was not setting the highest
1863 backreference number. This mattered only if pcre2_match() was called with an
1864 ovector that was too small to hold the capture, and there was no other kind of
1865 back reference (a situation which is probably quite rare). The effect of the
1866 bug was that the condition was always treated as FALSE when the capture could
1867 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
1868 has been fixed.
1869
1870 2. Functions for serialization and deserialization of sets of compiled patterns
1871 have been added.
1872
1873 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
1874 excess code units at the end of the data block that may occasionally occur if
1875 the code for calculating the size over-estimates. This change stops the
1876 serialization code copying uninitialized data, to which valgrind objects. The
1877 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
1878 include the general overhead. This has been corrected.
1879
1880 4. All code units in every slot in the table of group names are now set, again
1881 in order to avoid accessing uninitialized data when serializing.
1882
1883 5. The (*NO_JIT) feature is implemented.
1884
1885 6. If a bug that caused pcre2_compile() to use more memory than allocated was
1886 triggered when using valgrind, the code in (3) above passed a stupidly large
1887 value to valgrind. This caused a crash instead of an "internal error" return.
1888
1889 7. A reference to a duplicated named group (either a back reference or a test
1890 for being set in a conditional) that occurred in a part of the pattern where
1891 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
1892 to be incorrectly calculated, leading to overwriting.
1893
1894 8. A mutually recursive set of back references such as (\2)(\1) caused a
1895 segfault at compile time (while trying to find the minimum matching length).
1896 The infinite loop is now broken (with the minimum length unset, that is, zero).
1897
1898 9. If an assertion that was used as a condition was quantified with a minimum
1899 of zero, matching went wrong. In particular, if the whole group had unlimited
1900 repetition and could match an empty string, a segfault was likely. The pattern
1901 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
1902 quantified, but not if they are being used as conditions, so the above pattern
1903 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
1904 patterns.
1905
1906 10. The error message for an invalid quantifier has been changed from "nothing
1907 to repeat" to "quantifier does not follow a repeatable item".
1908
1909 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
1910 scanning the compiled pattern in subsequent auto-possessification can get out
1911 of step and lead to an unknown opcode. Previously this could have caused an
1912 infinite loop. Now it generates an "internal error" error. This is a tidyup,
1913 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
1914 undefined outcome.
1915
1916 12. A UTF pattern containing a "not" match of a non-ASCII character and a
1917 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
1918
1919 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
1920 that is found in the output of "locale -a" can actually be set by pcre2test
1921 before it is accepted. Previously, in an environment where a locale was listed
1922 but would not set (an example does exist), the test would "pass" without
1923 actually doing anything. Also the fr_CA locale has been added to the list of
1924 locales that can be used.
1925
1926 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
1927 capturing group number without parentheses, the last character was incorrectly
1928 literally included at the end of the replacement string.
1929
1930 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
1931 failed to allow the zero-repeat case if pcre2_match() was called with an
1932 ovector too small to capture the group.
1933
1934 16. Improved error message in pcre2test when setting the stack size (-S) fails.
1935
1936 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
1937 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
1938 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
1939 of PCRE2 sources, which caused a failure to build pcre2test.
1940
1941 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
1942 only on Windows.
1943
1944 19. Use binary input when reading back saved serialized patterns in pcre2test.
1945
1946 20. Added RunTest.bat for running the tests under Windows.
1947
1948 21. "make distclean" was not removing config.h, a file that may be created for
1949 use with CMake.
1950
1951 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
1952 forward reference repeated a large (but limited) number of times within a
1953 repeated outer group that has a zero minimum quantifier, caused incorrect code
1954 to be compiled, leading to the error "internal error: previously-checked
1955 referenced subpattern not found" when an incorrect memory address was read.
1956 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
1957 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
1958
1959 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
1960 call within a group that also contained a recursive back reference caused
1961 incorrect code to be compiled. This bug was reported as "heap overflow",
1962 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
1963 CVE-2015-2326 was given to this.)
1964
1965 24. Computing the size of the JIT read-only data in advance has been a source
1966 of various issues, and new ones are still appear unfortunately. To fix
1967 existing and future issues, size computation is eliminated from the code,
1968 and replaced by on-demand memory allocation.
1969
1970 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
1971 adjacent to the end of the range, and the range contained characters with more
1972 than one other case, caused incorrect behaviour when compiled in UTF mode. In
1973 that example, the range a-j was left out of the class.
1974
1975
1976 Version 10.00 05-January-2015
1977 -----------------------------
1978
1979 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
1980 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
1981 API, up to item 20 for release 8.36.
1982
1983 The code of the library was heavily revised as part of the new API
1984 implementation. Details of each and every modification were not individually
1985 logged. In addition to the API changes, the following changes were made. They
1986 are either new functionality, or bug fixes and other noticeable changes of
1987 behaviour that were implemented after the code had been forked.
1988
1989 1. Including Unicode support at build time is now enabled by default, but it
1990 can optionally be disabled. It is not enabled by default at run time (no
1991 change).
1992
1993 2. The test program, now called pcre2test, was re-specified and almost
1994 completely re-written. Its input is not compatible with input for pcretest.
1995
1996 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
1997 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
1998 matched by that pattern.
1999
2000 4. For the benefit of those who use PCRE2 via some other application, that is,
2001 not writing the function calls themselves, it is possible to check the PCRE2
2002 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
2003 string such as "yesno".
2004
2005 5. There are case-equivalent Unicode characters whose encodings use different
2006 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
2007 theoretically possible for this to happen in UTF-16 too.) If a backreference to
2008 a group containing one of these characters was greedily repeated, and during
2009 the match a backtrack occurred, the subject might be backtracked by the wrong
2010 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
2011 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
2012 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
2013 Incorrect backtracking meant that group 2 captured only the last two bytes.
2014 This bug has been fixed; the new code is slower, but it is used only when the
2015 strings matched by the repetition are not all the same length.
2016
2017 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
2018 information. This applied to any pattern with a group that matched no
2019 characters, for example: /(?:(?=.)|(?<!x))a/.
2020
2021 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
2022 those parentheses to be closed with whatever has been captured so far. However,
2023 it was failing to mark any other groups between the highest capture so far and
2024 the currrent group as "unset". Thus, the ovector for those groups contained
2025 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
2026 matched against "abcd".
2027
2028 8. The pcre2_substitute() function has been implemented.
2029
2030 9. If an assertion used as a condition was quantified with a minimum of zero
2031 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
2032 occur.
2033
2034 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.
2035
2036 ****

  ViewVC Help
Powered by ViewVC 1.1.5