/[pcre2]/code/trunk/ChangeLog
ViewVC logotype

Contents of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1304 - (show annotations)
Fri Feb 19 12:05:57 2021 UTC (8 weeks, 1 day ago) by ph10
File size: 116947 byte(s)
Add another test, tidy ChangeLog.
1 Change Log for PCRE2
2 --------------------
3
4
5 Version 10.37-RC1 xx-xxxx-2021
6 ---------------------------------
7
8 1. Change RunGrepTest to use tr instead of sed when testing with binary
9 zero bytes, because sed varies a lot from system to system and has problems
10 with binary zeros. This is from Bugzilla #2681. Patch from Jeremie
11 Courreges-Anglas via Nam Nguyen. This fixes RunGrepTest for OpenBSD. Later:
12 it broke it for at least one version of Solaris, where tr can't handle binary
13 zeros. However, that system had /usr/xpg4/bin/tr installed, which works OK, so
14 RunGrepTest now checks for that command and use it if found.
15
16 2. Compiling with gcc 10.2's -fanalyzer option showed up a hypothetical problem
17 with a NULL dereference. I don't think this case could ever occur in practice,
18 but I have put in a check in order to get rid of the compiler error.
19
20 3. An alternative patch for CMakeLists.txt because 10.36 # 4 breaks CMake on
21 Windows. Patch from email@cs-ware.de fixes bugzilla #2688.
22
23 4. Two bugs related to over-large numbers have been fixed so the behaviour is
24 now the same as Perl.
25
26 (a) A pattern such as /\214748364/ gave an overflow error instead of being
27 treated as the octal number \214 followed by literal digits.
28
29 (b) A sequence such as {65536 that has no terminating } so is not a
30 quantifier was nevertheless complaining that a quantifier number was too big.
31
32 5. A run of autoconf suggested that configure.ac was out-of-date with respect
33 to the lastest autoconf. Running autoupdate made some valid changes, some valid
34 suggestions, and also some invalid changes, which were fixed by hand. Autoconf
35 now runs clean and the resulting "configure" seems to work, so I hope nothing
36 is broken. Later: the requirement for autoconf 2.70 broke some automatic test
37 robots. It doesn't seem to be necessary: trying a reduction to 2.60.
38
39 6. The pattern /a\K.(?0)*/ when matched against "abac" by the interpreter gave
40 the answer "bac", whereas Perl and JIT both yield "c". This was because the
41 effect of \K was not propagating back from the full pattern recursion. Other
42 recursions such as /(a\K.(?1)*)/ did not have this problem.
43
44 7. Restore single character repetition optimization in JIT. Currently fewer
45 character repetitions are optimized than in 10.34.
46
47
48
49 Version 10.36 04-December-2020
50 ------------------------------
51
52 1. Add CET_CFLAGS so that when Intel CET is enabled, pass -mshstk to
53 compiler. This fixes https://bugs.exim.org/show_bug.cgi?id=2578. Patch for
54 Makefile.am and configure.ac by H.J. Lu. Equivalent patch for CMakeLists.txt
55 invented by PH.
56
57 2. Fix inifinite loop when a single byte newline is searched in JIT when
58 invalid utf8 mode is enabled.
59
60 3. Updated CMakeLists.txt with patch from Wolfgang Stöggl (Bugzilla #2584):
61
62 - Include GNUInstallDirs and use ${CMAKE_INSTALL_LIBDIR} instead of hardcoded
63 lib. This allows differentiation between lib and lib64.
64 CMAKE_INSTALL_LIBDIR is used for installation of libraries and also for
65 pkgconfig file generation.
66
67 - Add the version of PCRE2 to the configuration summary like ./configure
68 does.
69
70 - Fix typo: MACTHED_STRING->MATCHED_STRING
71
72 4. Updated CMakeLists.txt with another patch from Wolfgang Stöggl (Bugzilla
73 #2588):
74
75 - Add escaped double quotes around include directory in CMakeLists.txt to
76 allow spaces in directory names.
77
78 - This fixes a cmake error, if the path of the pcre2 source contains a space.
79
80 5. Updated CMakeLists.txt with a patch from B. Scott Michel: CMake's
81 documentation suggests using CHECK_SYMBOL_EXISTS over CHECK_FUNCTION_EXIST.
82 Moreover, these functions come from specific header files, which need to be
83 specified (and, thankfully, are the same on both the Linux and WinXX
84 platforms.)
85
86 6. Added a (uint32_t) cast to prevent a compiler warning in pcre2_compile.c.
87
88 7. Applied a patch from Wolfgang Stöggl (Bugzilla #2600) to fix postfix for
89 debug Windows builds using CMake. This also updated configure so that it
90 generates *.pc files and pcre2-config with the same content, as in the past.
91
92 8. If a pattern ended with (?(VERSION=n.d where n is any number but d is just a
93 single digit, the code unit beyond d was being read (i.e. there was a read
94 buffer overflow). Fixes ClusterFuzz 23779.
95
96 9. After the rework in r1235, certain character ranges were incorrectly
97 handled by an optimization in JIT. Furthermore a wrong offset was used to
98 read a value from a buffer which could lead to memory overread.
99
100 10. Unnoticed for many years was the fact that delimiters other than / in the
101 testinput1 and testinput4 files could cause incorrect behaviour when these
102 files were processed by perltest.sh. There were several tests that used quotes
103 as delimiters, and it was just luck that they didn't go wrong with perltest.sh.
104 All the patterns in testinput1 and testinput4 now use / as their delimiter.
105 This fixes Bugzilla #2641.
106
107 11. Perl has started to give an error for \K within lookarounds (though there
108 are cases where it doesn't). PCRE2 still allows this, so the tests that include
109 this case have been moved from test 1 to test 2.
110
111 12. Further to 10 above, pcre2test has been updated to detect and grumble if a
112 delimiter other than / is used after #perltest.
113
114 13. Fixed a bug with PCRE2_MATCH_INVALID_UTF in 8-bit mode when PCRE2_CASELESS
115 was set and PCRE2_NO_START_OPTIMIZE was not set. The optimization for finding
116 the start of a match was not resetting correctly after a failed match on the
117 first valid fragment of the subject, possibly causing incorrect "no match"
118 returns on subsequent fragments. For example, the pattern /A/ failed to match
119 the subject \xe5A. Fixes Bugzilla #2642.
120
121 14. Fixed a bug in character set matching when JIT is enabled and both unicode
122 scripts and unicode classes are present at the same time.
123
124 15. Added GNU grep's -m (aka --max-count) option to pcre2grep.
125
126 16. Refactored substitution processing in pcre2grep strings, both for the -O
127 option and when dealing with callouts. There is now a single function that
128 handles $ expansion in all cases (instead of multiple copies of almost
129 identical code). This means that the same escape sequences are available
130 everywhere, which was not previously the case. At the same time, the escape
131 sequences $x{...} and $o{...} have been introduced, to allow for characters
132 whose code points are greater than 255 in Unicode mode.
133
134 17. Applied the patch from Bugzilla #2628 to RunGrepTest. This does an explicit
135 test for a version of sed that can handle binary zero, instead of assuming that
136 any Linux version will work. Later: replaced $(...) by `...` because not all
137 shells recognize the former.
138
139 18. Fixed a word boundary check bug in JIT when partial matching is enabled.
140
141 19. Fix ARM64 compilation warning in JIT. Patch by Carlo.
142
143 20. A bug in the RunTest script meant that if the first part of test 2 failed,
144 the failure was not reported.
145
146 21. Test 2 was failing when run from a directory other than the source
147 directory. This failure was previously missed in RunTest because of 20 above.
148 Fixes added to both RunTest and RunTest.bat.
149
150 22. Patch to CMakeLists.txt from Daniel to fix problem with testing under
151 Windows.
152
153
154 Version 10.35 09-May-2020
155 ---------------------------
156
157 1. Use PCRE2_MATCH_EMPTY flag to detect empty matches in JIT.
158
159 2. Fix ARMv5 JIT improper handling of labels right after a constant pool.
160
161 3. A JIT bug is fixed which allowed to read the fields of the compiled
162 pattern before its existence is checked.
163
164 4. Back in the PCRE1 day, capturing groups that contained recursive back
165 references to themselves were made atomic (version 8.01, change 18) because
166 after the end a repeated group, the captured substrings had their values from
167 the final repetition, not from an earlier repetition that might be the
168 destination of a backtrack. This feature was documented, and was carried over
169 into PCRE2. However, it has now been realized that the major refactoring that
170 was done for 10.30 has made this atomicizing unnecessary, and it is confusing
171 when users are unaware of it, making some patterns appear not to be working as
172 expected. Capture values of recursive back references in repeated groups are
173 now correctly backtracked, so this unnecessary restriction has been removed.
174
175 5. Added PCRE2_SUBSTITUTE_LITERAL.
176
177 6. Avoid some VS compiler warnings.
178
179 7. Added PCRE2_SUBSTITUTE_MATCHED.
180
181 8. Added (?* and (?<* as synonms for (*napla: and (*naplb: to match another
182 regex engine. The Perl regex folks are aware of this usage and have made a note
183 about it.
184
185 9. When an assertion is repeated, PCRE2 used to limit the maximum repetition to
186 1, believing that repeating an assertion is pointless. However, if a positive
187 assertion contains capturing groups, repetition can be useful. In any case, an
188 assertion could always be wrapped in a repeated group. The only restriction
189 that is now imposed is that an unlimited maximum is changed to one more than
190 the minimum.
191
192 10. Fix *THEN verbs in lookahead assertions in JIT.
193
194 11. Added PCRE2_SUBSTITUTE_REPLACEMENT_ONLY.
195
196 12. The JIT stack should be freed when the low-level stack allocation fails.
197
198 13. In pcre2grep, if the final line in a scanned file is output but does not
199 end with a newline sequence, add a newline according to the --newline setting.
200
201 14. (?(DEFINE)...) groups were not being handled correctly when checking for
202 the fixed length of a lookbehind assertion. Such a group within a lookbehind
203 should be skipped, as it does not contribute to the length of the group.
204 Instead, the (DEFINE) group was being processed, and if at the end of the
205 lookbehind, that end was not correctly recognized. Errors such as "lookbehind
206 assertion is not fixed length" and also "internal error: bad code value in
207 parsed_skip()" could result.
208
209 15. Put a limit of 1000 on recursive calls in pcre2_study() when searching
210 nested groups for starting code units, in order to avoid stack overflow issues.
211 If the limit is reached, it just gives up trying for this optimization.
212
213 16. The control verb chain list must always be restored when exiting from a
214 recurse function in JIT.
215
216 17. Fix a crash which occurs when the character type of an invalid UTF
217 character is decoded in JIT.
218
219 18. Changes in many areas of the code so that when Unicode is supported and
220 PCRE2_UCP is set without PCRE2_UTF, Unicode character properties are used for
221 upper/lower case computations on characters whose code points are greater than
222 127.
223
224 19. The function for checking UTF-16 validity was returning an incorrect offset
225 for the start of the error when a high surrogate was not followed by a valid
226 low surrogate. This caused incorrect behaviour, for example when
227 PCRE2_MATCH_INVALID_UTF was set and a match started immediately following the
228 invalid high surrogate, such as /aa/ matching "\x{d800}aa".
229
230 20. If a DEFINE group immediately preceded a lookbehind assertion, the pattern
231 could be mis-compiled and therefore not match correctly. This is the example
232 that found this: /(?(DEFINE)(?<foo>bar))(?<![-a-z0-9])word/ which failed to
233 match "word" because the "move back" value was set to zero.
234
235 21. Following a request from a user, some extensions and tidies to the
236 character tables handling have been done:
237
238 (a) The dftables auxiliary program is renamed pcre2_dftables, but it is still
239 not installed for public use.
240
241 (b) There is now a -b option for pcre2_dftables, which causes the tables to
242 be written in binary. There is also a -help option.
243
244 (c) PCRE2_CONFIG_TABLES_LENGTH is added to pcre2_config() so that an
245 application that wants to save tables in binary knows how long they are.
246
247 22. Changed setting of CMAKE_MODULE_PATH in CMakeLists.txt from SET to
248 LIST(APPEND...) to allow a setting from the command line to be included.
249
250 23. Updated to Unicode 13.0.0.
251
252 24. CMake build now checks for secure_getenv() and strerror(). Patch by Carlo.
253
254 25. Avoid using [-1] as a suffix in pcre2test because it can provoke a compiler
255 warning.
256
257 26. Added tests for __attribute__((uninitialized)) to both the configure and
258 CMake build files, and then applied this attribute to the variable called
259 stack_frames_vector[] in pcre2_match(). When implemented, this disables
260 automatic initialization (a facility in clang), which can take time on big
261 variables.
262
263 27. Updated CMakeLists.txt (patches by Uwe Korn) to add support for
264 pcre2-config, the libpcre*.pc files, SOVERSION, VERSION and the
265 MACHO_*_VERSIONS settings for CMake builds.
266
267 28. Another patch to CMakeLists.txt to check for mkostemp (configure already
268 does). Patch by Carlo Marcelo Arenas Belon.
269
270 29. Check for the existence of memfd_create in both CMake and configure
271 configurations. Patch by Carlo Marcelo Arenas Belon.
272
273 30. Restrict the configuration setting for the SELinux compatible execmem
274 allocator (change 10.30/44) to Linux and NetBSD.
275
276
277 Version 10.34 21-November-2019
278 ------------------------------
279
280 1. The maximum number of capturing subpatterns is 65535 (documented), but no
281 check on this was ever implemented. This omission has been rectified; it fixes
282 ClusterFuzz 14376.
283
284 2. Improved the invalid utf32 support of the JIT compiler. Now it correctly
285 detects invalid characters in the 0xd800-0xdfff range.
286
287 3. Fix minor typo bug in JIT compile when \X is used in a non-UTF string.
288
289 4. Add support for matching in invalid UTF strings to the pcre2_match()
290 interpreter, and integrate with the existing JIT support via the new
291 PCRE2_MATCH_INVALID_UTF compile-time option.
292
293 5. Give more error detail for invalid UTF-8 when detected in pcre2grep.
294
295 6. Add support for invalid UTF-8 to pcre2grep.
296
297 7. Adjust the limit for "must have" code unit searching, in particular,
298 increase it substantially for non-anchored patterns.
299
300 8. Allow (*ACCEPT) to be quantified, because an ungreedy quantifier with a zero
301 minimum is potentially useful.
302
303 9. Some changes to the way the minimum subject length is handled:
304
305 * When PCRE2_NO_START_OPTIMIZE is set, no minimum length is computed;
306 pcre2test now omits this item instead of showing a value of zero.
307
308 * An incorrect minimum length could be calculated for a pattern that
309 contained (*ACCEPT) inside a qualified group whose minimum repetition was
310 zero, for example /A(?:(*ACCEPT))?B/, which incorrectly computed a minimum
311 of 2. The minimum length scan no longer happens for a pattern that
312 contains (*ACCEPT).
313
314 * When no minimum length is set by the normal scan, but a first and/or last
315 code unit is recorded, set the minimum to 1 or 2 as appropriate.
316
317 * When a pattern contains multiple groups with the same number, a back
318 reference cannot know which one to scan for a minimum length. This used to
319 cause the minimum length finder to give up with no result. Now it treats
320 such references as not adding to the minimum length (which it should have
321 done all along).
322
323 * Furthermore, the above action now happens only if the back reference is to
324 a group that exists more than once in a pattern instead of any back
325 reference in a pattern with duplicate numbers.
326
327 10. A (*MARK) value inside a successful condition was not being returned by the
328 interpretive matcher (it was returned by JIT). This bug has been mended.
329
330 11. A bug in pcre2grep meant that -o without an argument (or -o0) didn't work
331 if the pattern had more than 32 capturing parentheses. This is fixed. In
332 addition (a) the default limit for groups requested by -o<n> has been raised to
333 50, (b) the new --om-capture option changes the limit, (c) an error is raised
334 if -o asks for a group that is above the limit.
335
336 12. The quantifier {1} was always being ignored, but this is incorrect when it
337 is made possessive and applied to an item in parentheses, because a
338 parenthesized item may contain multiple branches or other backtracking points,
339 for example /(a|ab){1}+c/ or /(a+){1}+a/.
340
341 13. For partial matches, pcre2test was always showing the maximum lookbehind
342 characters, flagged with "<", which is misleading when the lookbehind didn't
343 actually look behind the start (because it was later in the pattern). Showing
344 all consulted preceding characters for partial matches is now controlled by the
345 existing "allusedtext" modifier and, as for complete matches, this facility is
346 available only for non-JIT matching, because JIT does not maintain the first
347 and last consulted characters.
348
349 14. DFA matching (using pcre2_dfa_match()) was not recognising a partial match
350 if the end of the subject was encountered in a lookahead (conditional or
351 otherwise), an atomic group, or a recursion.
352
353 15. Give error if pcre2test -t, -T, -tm or -TM is given an argument of zero.
354
355 16. Check for integer overflow when computing lookbehind lengths. Fixes
356 Clusterfuzz issue 15636.
357
358 17. Implemented non-atomic positive lookaround assertions.
359
360 18. If a lookbehind contained a lookahead that contained another lookbehind
361 within it, the nested lookbehind was not correctly processed. For example, if
362 /(?<=(?=(?<=a)))b/ was matched to "ab" it gave no match instead of matching
363 "b".
364
365 19. Implemented pcre2_get_match_data_size().
366
367 20. Two alterations to partial matching:
368
369 (a) The definition of a partial match is slightly changed: if a pattern
370 contains any lookbehinds, an empty partial match may be given, because this
371 is another situation where adding characters to the current subject can
372 lead to a full match. Example: /c*+(?<=[bc])/ with subject "ab".
373
374 (b) Similarly, if a pattern could match an empty string, an empty partial
375 match may be given. Example: /(?![ab]).*/ with subject "ab". This case
376 applies only to PCRE2_PARTIAL_HARD.
377
378 (c) An empty string partial hard match can be returned for \z and \Z as it
379 is documented that they shouldn't match.
380
381 21. A branch that started with (*ACCEPT) was not being recognized as one that
382 could match an empty string.
383
384 22. Corrected pcre2_set_character_tables() tables data type: was const unsigned
385 char * instead of const uint8_t *, as generated by pcre2_maketables().
386
387 23. Upgraded to Unicode 12.1.0.
388
389 24. Add -jitfast command line option to pcre2test (to make all the jit options
390 available directly).
391
392 25. Make pcre2test -C show if libreadline or libedit is supported.
393
394 26. If the length of one branch of a group exceeded 65535 (the maximum value
395 that is remembered as a minimum length), the whole group's length was
396 incorrectly recorded as 65535, leading to incorrect "no match" when start-up
397 optimizations were in force.
398
399 27. The "rightmost consulted character" value was not always correct; in
400 particular, if a pattern ended with a negative lookahead, characters that were
401 inspected in that lookahead were not included.
402
403 28. Add the pcre2_maketables_free() function.
404
405 29. The start-up optimization that looks for a unique initial matching
406 code unit in the interpretive engines uses memchr() in 8-bit mode. When the
407 search is caseless, it was doing so inefficiently, which ended up slowing down
408 the match drastically when the subject was very long. The revised code (a)
409 remembers if one case is not found, so it never repeats the search for that
410 case after a bumpalong and (b) when one case has been found, it searches only
411 up to that position for an earlier occurrence of the other case. This fix
412 applies to both interpretive pcre2_match() and to pcre2_dfa_match().
413
414 30. While scanning to find the minimum length of a group, if any branch has
415 minimum length zero, there is no need to scan any subsequent branches (a small
416 compile-time performance improvement).
417
418 31. Installed a .gitignore file on a user's suggestion. When using the svn
419 repository with git (through git svn) this helps keep it tidy.
420
421 32. Add underflow check in JIT which may occur when the value of subject
422 string pointer is close to 0.
423
424 33. Arrange for classes such as [Aa] which contain just the two cases of the
425 same character, to be treated as a single caseless character. This causes the
426 first and required code unit optimizations to kick in where relevant.
427
428 34. Improve the bitmap of starting bytes for positive classes that include wide
429 characters, but no property types, in UTF-8 mode. Previously, on encountering
430 such a class, the bits for all bytes greater than \xc4 were set, thus
431 specifying any character with codepoint >= 0x100. Now the only bits that are
432 set are for the relevant bytes that start the wide characters. This can give a
433 noticeable performance improvement.
434
435 35. If the bitmap of starting code units contains only 1 or 2 bits, replace it
436 with a single starting code unit (1 bit) or a caseless single starting code
437 unit if the two relevant characters are case-partners. This is particularly
438 relevant to the 8-bit library, though it applies to all. It can give a
439 performance boost for patterns such as [Ww]ord and (word|WORD). However, this
440 optimization doesn't happen if there is a "required" code unit of the same
441 value (because the search for a "required" code unit starts at the match start
442 for non-unique first code unit patterns, but after a unique first code unit,
443 and patterns such as a*a need the former action).
444
445 36. Small patch to pcre2posix.c to set the erroroffset field to -1 immediately
446 after a successful compile, instead of at the start of matching to avoid a
447 sanitizer complaint (regexec is supposed to be thread safe).
448
449 37. Add NEON vectorization to JIT to speed up matching of first character and
450 pairs of characters on ARM64 CPUs.
451
452 38. If a non-ASCII character was the first in a starting assertion in a
453 caseless match, the "first code unit" optimization did not get the casing
454 right, and the assertion failed to match a character in the other case if it
455 did not start with the same code unit.
456
457 39. Fixed the incorrect computation of jump sizes on x86 CPUs in JIT. A masking
458 operation was incorrectly removed in r1136. Reported by Ralf Junker.
459
460
461 Version 10.33 16-April-2019
462 ---------------------------
463
464 1. Added "allvector" to pcre2test to make it easy to check the part of the
465 ovector that shouldn't be changed, in particular after substitute and failed or
466 partial matches.
467
468 2. Fix subject buffer overread in JIT when UTF is disabled and \X or \R has
469 a greater than 1 fixed quantifier. This issue was found by Yunho Kim.
470
471 3. Added support for callouts from pcre2_substitute(). After 10.33-RC1, but
472 prior to release, fixed a bug that caused a crash if pcre2_substitute() was
473 called with a NULL match context.
474
475 4. The POSIX functions are now all called pcre2_regcomp() etc., with wrapper
476 functions that use the standard POSIX names. However, in pcre2posix.h the POSIX
477 names are defined as macros. This should help avoid linking with the wrong
478 library in some environments while still exporting the POSIX names for
479 pre-existing programs that use them. (The Debian alternative names are also
480 defined as macros, but not documented.)
481
482 5. Fix an xclass matching issue in JIT.
483
484 6. Implement PCRE2_EXTRA_ESCAPED_CR_IS_LF (see Bugzilla 2315).
485
486 7. Implement the Perl 5.28 experimental alphabetic names for atomic groups and
487 lookaround assertions, for example, (*pla:...) and (*atomic:...). These are
488 characterized by a lower case letter following (* and to simplify coding for
489 this, the character tables created by pcre2_maketables() were updated to add a
490 new "is lower case letter" bit. At the same time, the now unused "is
491 hexadecimal digit" bit was removed. The default tables in
492 src/pcre2_chartables.c.dist are updated.
493
494 8. Implement the new Perl "script run" features (*script_run:...) and
495 (*atomic_script_run:...) aka (*sr:...) and (*asr:...).
496
497 9. Fixed two typos in change 22 for 10.21, which added special handling for
498 ranges such as a-z in EBCDIC environments. The original code probably never
499 worked, though there were no bug reports.
500
501 10. Implement PCRE2_COPY_MATCHED_SUBJECT for pcre2_match() (including JIT via
502 pcre2_match()) and pcre2_dfa_match(), but *not* the pcre2_jit_match() fast
503 path. Also, when a match fails, set the subject field in the match data to NULL
504 for tidiness - none of the substring extractors should reference this after
505 match failure.
506
507 11. If a pattern started with a subroutine call that had a quantifier with a
508 minimum of zero, an incorrect "match must start with this character" could be
509 recorded. Example: /(?&xxx)*ABC(?<xxx>XYZ)/ would (incorrectly) expect 'A' to
510 be the first character of a match.
511
512 12. The heap limit checking code in pcre2_dfa_match() could suffer from
513 overflow if the heap limit was set very large. This could cause incorrect "heap
514 limit exceeded" errors.
515
516 13. Add "kibibytes" to the heap limit output from pcre2test -C to make the
517 units clear.
518
519 14. Add a call to pcre2_jit_free_unused_memory() in pcre2grep, for tidiness.
520
521 15. Updated the VMS-specific code in pcre2test on the advice of a VMS user.
522
523 16. Removed the unnecessary inclusion of stdint.h (or inttypes.h) from
524 pcre2_internal.h as it is now included by pcre2.h. Also, change 17 for 10.32
525 below was unnecessarily complicated, as inttypes.h is a Standard C header,
526 which is defined to be a superset of stdint.h. Instead of conditionally
527 including stdint.h or inttypes.h, pcre2.h now unconditionally includes
528 inttypes.h. This supports environments that do not have stdint.h but do have
529 inttypes.h, which are known to exist. A note in the autotools documentation
530 says (November 2018) that there are none known that are the other way round.
531
532 17. Added --disable-percent-zt to "configure" (and equivalent to CMake) to
533 forcibly disable the use of %zu and %td in formatting strings because there is
534 at least one version of VMS that claims to be C99 but does not support these
535 modifiers.
536
537 18. Added --disable-pcre2grep-callout-fork, which restricts the callout support
538 in pcre2grep to the inbuilt echo facility. This may be useful in environments
539 that do not support fork().
540
541 19. Fix two instances of <= 0 being applied to unsigned integers (the VMS
542 compiler complains).
543
544 20. Added "fork" support for VMS to pcre2grep, for running an external program
545 via a string callout.
546
547 21. Improve MAP_JIT flag usage on MacOS. Patch by Rich Siegel.
548
549 22. If a pattern started with (*MARK), (*COMMIT), (*PRUNE), (*SKIP), or (*THEN)
550 followed by ^ it was not recognized as anchored.
551
552 23. The RunGrepTest script used to cut out the test of NUL characters for
553 Solaris and MacOS as printf and sed can't handle them. It seems that the *BSD
554 systems can't either. I've inverted the test so that only those OS that are
555 known to work (currently only Linux) try to run this test.
556
557 24. Some tests in RunGrepTest appended to testtrygrep from two different file
558 descriptors instead of redirecting stderr to stdout. This worked on Linux, but
559 it was reported not to on other systems, causing the tests to fail.
560
561 25. In the RunTest script, make the test for stack setting use the same value
562 for the stack as it needs for -bigstack.
563
564 26. Insert a cast in pcre2_dfa_match.c to suppress a compiler warning.
565
566 26. With PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL set, escape sequences such as \s
567 which are valid in character classes, but not as the end of ranges, were being
568 treated as literals. An example is [_-\s] (but not [\s-_] because that gave an
569 error at the *start* of a range). Now an "invalid range" error is given
570 independently of PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
571
572 27. Related to 26 above, PCRE2_BAD_ESCAPE_IS_LITERAL was affecting known escape
573 sequences such as \eX when they appeared invalidly in a character class. Now
574 the option applies only to unrecognized or malformed escape sequences.
575
576 28. Fix word boundary in JIT compiler. Patch by Mike Munday.
577
578 29. The pcre2_dfa_match() function was incorrectly handling conditional version
579 tests such as (?(VERSION>=0)...) when the version test was true. Incorrect
580 processing or a crash could result.
581
582 30. When PCRE2_UTF is set, allow non-ASCII letters and decimal digits in group
583 names, as Perl does. There was a small bug in this new code, found by
584 ClusterFuzz 12950, fixed before release.
585
586 31. Implemented PCRE2_EXTRA_ALT_BSUX to support ECMAScript 6's \u{hhh}
587 construct.
588
589 32. Compile \p{Any} to be the same as . in DOTALL mode, so that it benefits
590 from auto-anchoring if \p{Any}* starts a pattern.
591
592 33. Compile invalid UTF check in JIT test when only pcre32 is enabled.
593
594 34. For some time now, CMake has been warning about the setting of policy
595 CMP0026 to "OLD" in CmakeLists.txt, and hinting that the feature might be
596 removed in a future version. A request for CMake expertise on the list produced
597 no result, so I have now hacked CMakeLists.txt along the lines of some changes
598 I found on the Internet. The new code no longer needs the policy setting, and
599 it appears to work fine on Linux.
600
601 35. Setting --enable-jit=auto for an out-of-tree build failed because the
602 source directory wasn't in the search path for AC_TRY_COMPILE always. Patch
603 from Ross Burton.
604
605 36. Disable SSE2 JIT optimizations in x86 CPUs when SSE2 is not available.
606 Patch by Guillem Jover.
607
608 37. Changed expressions such as 1<<10 to 1u<<10 in many places because compiler
609 warnings were reported.
610
611 38. Using the clang compiler with sanitizing options causes runtime complaints
612 about truncation for statments such as x = ~x when x is an 8-bit value; it
613 seems to compute ~x as a 32-bit value. Changing such statements to x = 255 ^ x
614 gets rid of the warnings. There were also two missing casts in pcre2test.
615
616
617 Version 10.32 10-September-2018
618 -------------------------------
619
620 1. When matching using the the REG_STARTEND feature of the POSIX API with a
621 non-zero starting offset, unset capturing groups with lower numbers than a
622 group that did capture something were not being correctly returned as "unset"
623 (that is, with offset values of -1).
624
625 2. When matching using the POSIX API, pcre2test used to omit listing unset
626 groups altogether. Now it shows those that come before any actual captures as
627 "<unset>", as happens for non-POSIX matching.
628
629 3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",
630 whatever the build configuration was. It now correctly says "\R matches all
631 Unicode newlines" in the default case when --enable-bsr-anycrlf has not been
632 specified. Similarly, running "pcre2test -C bsr" never produced the result
633 ANY.
634
635 4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing
636 multi-code-unit characters caused bad behaviour and possibly a crash. This
637 issue was fixed for other kinds of repeat in release 10.20 by change 19, but
638 repeating character classes were overlooked.
639
640 5. pcre2grep now supports the inclusion of binary zeros in patterns that are
641 read from files via the -f option.
642
643 6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.
644
645 7. Added --enable-jit=auto support to configure.ac.
646
647 8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit
648 modes for the benefit of m68k, where pointers can be 16-bit aligned. The
649 dummies force 32-bit alignment and this ensures that the structure is a
650 multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other
651 architectures, alignment requirements take care of this automatically.
652
653 9. When returning an error from pcre2_pattern_convert(), ensure the error
654 offset is set zero for early errors.
655
656 10. A number of patches for Windows support from Daniel Richard G:
657
658 (a) List of error numbers in Runtest.bat corrected (it was not the same as in
659 Runtest).
660
661 (b) pcre2grep snprintf() workaround as used elsewhere in the tree.
662
663 (c) Support for non-C99 snprintf() that returns -1 in the overflow case.
664
665 11. Minor tidy of pcre2_dfa_match() code.
666
667 12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer
668 use the stack for local workspace and local ovectors. Instead, an initial block
669 of stack is reserved, but if this is insufficient, heap memory is used. The
670 heap limit parameter now applies to pcre2_dfa_match().
671
672 13. If a "find limits" test of DFA matching in pcre2test resulted in too many
673 matches for the ovector, no matches were displayed.
674
675 14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as
676 EOF. The test looks to have come from a fuzzer.
677
678 15. If PCRE2 was built with a default match limit a lot greater than the
679 default default of 10 000 000, some JIT tests of the match limit no longer
680 failed. All such tests now set 10 000 000 as the upper limit.
681
682 16. Another Windows related patch for pcregrep to ensure that WIN32 is
683 undefined under Cygwin.
684
685 17. Test for the presence of stdint.h and inttypes.h in configure and CMake and
686 include whichever exists (stdint preferred) instead of unconditionally
687 including stdint. This makes life easier for old and non-standard systems.
688
689 18. Further changes to improve portability, especially to old and or non-
690 standard systems:
691
692 (a) Put all printf arguments in RunGrepTest into single, not double, quotes,
693 and use \0 not \x00 for binary zero.
694
695 (b) Avoid the use of C++ (i.e. BCPL) // comments.
696
697 (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of
698 these now, if using MSVC or a standard C before C99, %lu is used with a
699 cast if necessary.
700
701 19. Applied a contributed patch to CMakeLists.txt to increase the stack size
702 when linking pcre2test with MSVC. This gets rid of a stack overflow error in
703 the standard set of tests.
704
705 20. Output a warning in pcre2test when ignoring the "altglobal" modifier when
706 it is given with the "replace" modifier.
707
708 21. In both pcre2test and pcre2_substitute(), with global matching, a pattern
709 that matched an empty string, but never at the starting match offset, was not
710 handled in a Perl-compatible way. The pattern /(<?=\G.)/ is an example of such
711 a pattern. Because \G is in a lookbehind assertion, there has to be a
712 "bumpalong" before there can be a match. The automatic "advance by one
713 character after an empty string match" rule is therefore inappropriate. A more
714 complicated algorithm has now been implemented.
715
716 22. When checking to see if a lookbehind is of fixed length, lookaheads were
717 correctly ignored, but qualifiers on lookaheads were not being ignored, leading
718 to an incorrect "lookbehind assertion is not fixed length" error.
719
720 23. The VERSION condition test was reading fractional PCRE2 version numbers
721 such as the 04 in 10.04 incorrectly and hence giving wrong results.
722
723 24. Updated to Unicode version 11.0.0. As well as the usual addition of new
724 scripts and characters, this involved re-jigging the grapheme break property
725 algorithm because Unicode has changed the way emojis are handled.
726
727 25. Fixed an obscure bug that struck when there were two atomic groups not
728 separated by something with a backtracking point. There could be an incorrect
729 backtrack into the first of the atomic groups. A complicated example is
730 /(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP
731 shouldn't find a MARK (because is in an atomic group), but it did.
732
733 26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set
734 a list of modifiers for all subsequent patterns - only those that the script
735 recognizes are meaningful; (2) #subject lines can be used to set or unset a
736 default "mark" modifier; (3) Unsupported #command lines give a warning when
737 they are ignored; (4) Mark data is output only if the "mark" modifier is
738 present.
739
740 27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported.
741
742 28. A (*MARK) name was not being passed back for positive assertions that were
743 terminated by (*ACCEPT).
744
745 29. Add support for \N{U+dddd}, but only in Unicode mode.
746
747 30. Add support for (?^) for unsetting all imnsx options.
748
749 31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose
750 code point was less than 256 and that were recognized by the lookup table
751 generated by pcre2_maketables(), which uses isspace() to identify white space.
752 Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085,
753 U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by
754 Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl.
755
756 32. In certain circumstances, option settings within patterns were not being
757 correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly
758 matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the
759 end of its group during the parse process, but without another setting such as
760 (?m) the compile phase got it right.) This bug was introduced by the
761 refactoring in release 10.23.
762
763 33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to
764 define memmove() as function call to bcopy(). This hasn't been tested for a
765 long time because in pcre2test the result of memmove() was being used, whereas
766 bcopy() doesn't return a result. This feature is now refactored always to call
767 an emulation function when there is no memmove(). The emulation makes use of
768 bcopy() when available.
769
770 34. When serializing a pattern, set the memctl, executable_jit, and tables
771 fields (that is, all the fields that contain pointers) to zeros so that the
772 result of serializing is always the same. These fields are re-set when the
773 pattern is deserialized.
774
775 35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated
776 negative class with no characters less than 0x100 followed by a positive class
777 with only characters less than 0x100, the first class was incorrectly being
778 auto-possessified, causing incorrect match failures.
779
780 36. Removed the character type bit ctype_meta, which dates from PCRE1 and is
781 not used in PCRE2.
782
783 37. Tidied up unnecessarily complicated macros used in the escapes table.
784
785 38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted
786 from distribution tarballs, owing to a typo in Makefile.am which had
787 testoutput8-16-3 twice. Now fixed.
788
789 39. If the only branch in a conditional subpattern was anchored, the whole
790 subpattern was treated as anchored, when it should not have been, since the
791 assumed empty second branch cannot be anchored. Demonstrated by test patterns
792 such as /(?(1)^())b/ or /(?(?=^))b/.
793
794 40. A repeated conditional subpattern that could match an empty string was
795 always assumed to be unanchored. Now it it checked just like any other
796 repeated conditional subpattern, and can be found to be anchored if the minimum
797 quantifier is one or more. I can't see much use for a repeated anchored
798 pattern, but the behaviour is now consistent.
799
800 41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint
801 (for an event that could never occur but you had to have external information
802 to know that).
803
804 42. If before the first match in a file that was being searched by pcre2grep
805 there was a line that was sufficiently long to cause the input buffer to be
806 expanded, the variable holding the location of the end of the previous match
807 was being adjusted incorrectly, and could cause an overflow warning from a code
808 sanitizer. However, as the value is used only to print pending "after" lines
809 when the next match is reached (and there are no such lines in this case) this
810 bug could do no damage.
811
812
813 Version 10.31 12-February-2018
814 ------------------------------
815
816 1. Fix typo (missing ]) in VMS code in pcre2test.c.
817
818 2. Replace the replicated code for matching extended Unicode grapheme sequences
819 (which got a lot more complicated by change 10.30/49) by a single subroutine
820 that is called by both pcre2_match() and pcre2_dfa_match().
821
822 3. Add idempotent guard to pcre2_internal.h.
823
824 4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and
825 PCRE2_CONFIG_COMPILED_WIDTHS.
826
827 5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is
828 defined (e.g. by --enable-never-backslash-C).
829
830 6. Defined public names for all the pcre2_compile() error numbers, and used
831 the public names in pcre2_convert.c.
832
833 7. Fixed a small memory leak in pcre2test (convert contexts).
834
835 8. Added two casts to compile.c and one to match.c to avoid compiler warnings.
836
837 9. Added code to pcre2grep when compiled under VMS to set the symbol
838 PCRE2GREP_RC to the exit status, because VMS does not distinguish between
839 exit(0) and exit(1).
840
841 10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain
842 about a bad option only if the following argument item does not start with a
843 hyphen.
844
845 11. pcre2grep was truncating components of file names to 128 characters when
846 processing files with the -r option, and also (some very odd code) truncating
847 path names to 512 characters. There is now a check on the absolute length of
848 full path file names, which may be up to 2047 characters long.
849
850 12. When an assertion contained (*ACCEPT) it caused all open capturing groups
851 to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to
852 misbehaviour for subsequent references to groups that started outside the
853 assertion. ACCEPT in an assertion now closes only those groups that were
854 started within that assertion. Fixes oss-fuzz issues 3852 and 3891.
855
856 13. Multiline matching in pcre2grep was misbehaving if the pattern matched
857 within a line, and then matched again at the end of the line and over into
858 subsequent lines. Behaviour was different with and without colouring, and
859 sometimes context lines were incorrectly printed and/or line endings were lost.
860 All these issues should now be fixed.
861
862 14. If --line-buffered was specified for pcre2grep when input was from a
863 compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be
864 ignored for compressed files.)
865
866 15. Although pcre2_jit_match checks whether the pattern is compiled
867 in a given mode, it was also expected that at least one mode is available.
868 This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION
869 when the pattern is not optimized by JIT at all.
870
871 16. The line number and related variables such as match counts in pcre2grep
872 were all int variables, causing overflow when files with more than 2147483647
873 lines were processed (assuming 32-bit ints). They have all been changed to
874 unsigned long ints.
875
876 17. If a backreference with a minimum repeat count of zero was first in a
877 pattern, apart from assertions, an incorrect first matching character could be
878 recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set
879 as the first character of a match.
880
881 18. Characters in a leading positive assertion are considered for recording a
882 first character of a match when the rest of the pattern does not provide one.
883 However, a character in a non-assertive group within a leading assertion such
884 as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an
885 infelicity rather than an outright bug, because it did not affect the result of
886 a match, just its speed. (In fact, in this case, the starting 'a' was
887 subsequently picked up in the study.)
888
889 19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return"
890 instead of "RRETURN" saves unwinding the backtracks in these cases (only one
891 didn't).
892
893 20. Allocate a single callout block on the stack at the start of pcre2_match()
894 and set its never-changing fields once only. Do the same for pcre2_dfa_match().
895
896 21. Save the extra compile options (set in the compile context) with the
897 compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS
898 to retrieve them, and update pcre2test to show them.
899
900 22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new
901 field callout_flags in callout blocks. The bits are set by pcre2_match(), but
902 not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts
903 if the callout_extra subject modifier is set. These bits are provided to help
904 with tracking how a backtracking match is proceeding.
905
906 23. Updated the pcre2demo.c demonstration program, which was missing the extra
907 code for -g that handles the case when \K in an assertion causes the match to
908 end at the original start point. Also arranged for it to detect when \K causes
909 the end of a match to be before its start.
910
911 24. Similar to 23 above, strange things (including loops) could happen in
912 pcre2grep when \K was used in an assertion when --colour was used or in
913 multiline mode. The "end at original start point" bug is fixed, and if the end
914 point is found to be before the start point, they are swapped.
915
916 25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT
917 matching (both pcre2_match() and pcre2_dfa_match()) and the matched string
918 started with the first code unit of a newline sequence, matching failed because
919 it was not tried at the newline.
920
921 26. Code for giving up a non-partial match after failing to find a starting
922 code unit anywhere in the subject was missing when searching for one of a
923 number of code units (the bitmap case) in both pcre2_match() and
924 pcre2_dfa_match(). This was a missing optimization rather than a bug.
925
926 27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a
927 pointer argument rather than a code unit value. This should not have affected
928 the generated code.
929
930 28. The JIT compiler has been updated.
931
932 29. Avoid pointer overflow for unset captures in pcre2_substring_list_get().
933 This could not actually cause a crash because it was always used in a memcpy()
934 call with zero length.
935
936 30. Some internal structures have a variable-length ovector[] as their last
937 element. Their actual memory is obtained dynamically, giving an ovector of
938 appropriate length. However, they are defined in the structure as
939 ovector[NUMBER], where NUMBER is large so that array bound checkers don't
940 grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing
941 groups, making the ovector larger than this. The number has been increased to
942 131072, which allows for the maximum number of captures (65535) plus the
943 overall match. This fixes oss-fuzz issue 5415.
944
945 31. Auto-possessification at the end of a capturing group was dependent on what
946 follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused
947 incorrect behaviour when the group was called recursively from elsewhere in the
948 pattern where something different might follow. This bug is an unforseen
949 consequence of change #1 for 10.30 - the implementation of backtracking into
950 recursions. Iterators at the ends of capturing groups are no longer considered
951 for auto-possessification if the pattern contains any recursions. Fixes
952 Bugzilla #2232.
953
954
955 Version 10.30 14-August-2017
956 ----------------------------
957
958 1. The main interpreter, pcre2_match(), has been refactored into a new version
959 that does not use recursive function calls (and therefore the stack) for
960 remembering backtracking positions. This makes --disable-stack-for-recursion a
961 NOOP. The new implementation allows backtracking into recursive group calls in
962 patterns, making it more compatible with Perl, and also fixes some other
963 hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
964 the old code had a number of fudges to try to reduce stack usage. It seems to
965 run no slower than the old code.
966
967 A number of bugs in the refactored code were subsequently fixed during testing
968 before release, but after the code was made available in the repository. These
969 bugs were never in fully released code, but are noted here for the record.
970
971 (a) If a pattern had fewer capturing parentheses than the ovector supplied in
972 the match data block, a memory error (detectable by ASAN) occurred after
973 a match, because the external block was being set from non-existent
974 internal ovector fields. Fixes oss-fuzz issue 781.
975
976 (b) A pattern with very many capturing parentheses (when the internal frame
977 size was greater than the initial frame vector on the stack) caused a
978 crash. A vector on the heap is now set up at the start of matching if the
979 vector on the stack is not big enough to handle at least 10 frames.
980 Fixes oss-fuzz issue 783.
981
982 (c) Handling of (*VERB)s in recursions was wrong in some cases.
983
984 (d) Captures in negative assertions that were used as conditions were not
985 happening if the assertion matched via (*ACCEPT).
986
987 (e) Mark values were not being passed out of recursions.
988
989 (f) Refactor some code in do_callout() to avoid picky compiler warnings about
990 negative indices. Fixes oss-fuzz issue 1454.
991
992 (g) Similarly refactor the way the variable length ovector is addressed for
993 similar reasons. Fixes oss-fuzz issue 1465.
994
995 2. Now that pcre2_match() no longer uses recursive function calls (see above),
996 the "match limit recursion" value seems misnamed. It still exists, and limits
997 the depth of tree that is searched. To avoid future confusion, it has been
998 renamed as "depth limit" in all relevant places (--with-depth-limit,
999 (*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
1000 available for backwards compatibility.
1001
1002 3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:
1003
1004 (a) Check for malloc failures when getting memory for the ovector (POSIX) or
1005 the match data block (non-POSIX).
1006
1007 4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
1008 for a character with a code point greater than 0x10ffff (the Unicode maximum)
1009 caused a crash.
1010
1011 5. If a lookbehind assertion that contained a back reference to a group
1012 appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
1013 undefined actions (often a segmentation fault) could occur, depending on what
1014 other options were set. An example assertion is (?<!\1(abc)) where the
1015 reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.
1016
1017 6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
1018 pcre2test to use it to output the frame size when the "framesize" modifier is
1019 given.
1020
1021 7. Reworked the recursive pattern matching in the JIT compiler to follow the
1022 interpreter changes.
1023
1024 8. When the zero_terminate modifier was specified on a pcre2test subject line
1025 for global matching, unpredictable things could happen. For example, in UTF-8
1026 mode, the pattern //g,zero_terminate read random memory when matched against an
1027 empty string with zero_terminate. This was a bug in pcre2test, not the library.
1028
1029 9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
1030 of the section that is compiled when Unix-style directory scanning is
1031 available, and into a new section that is always compiled for Windows.
1032
1033 10. In pcre2test, explicitly close the file after an error during serialization
1034 or deserialization (the "load" or "save" commands).
1035
1036 11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.
1037
1038 12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
1039 a NULL pattern pointer when Unicode support is available.
1040
1041 13. When the 32-bit library was being tested by pcre2test, error messages that
1042 were longer than 64 code units could cause a buffer overflow. This was a bug in
1043 pcre2test.
1044
1045 14. The alternative matching function, pcre2_dfa_match() misbehaved if it
1046 encountered a character class with a possessive repeat, for example [a-f]{3}+.
1047
1048 15. The depth (formerly recursion) limit now applies to DFA matching (as
1049 of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA
1050 matching to find the minimum value for this limit.
1051
1052 16. Since 10.21, if pcre2_match() was called with a null context, default
1053 memory allocation functions were used instead of whatever was used when the
1054 pattern was compiled.
1055
1056 17. Changes to the pcre2test "memory" modifier on a subject line. These apply
1057 only to pcre2_match():
1058
1059 (a) Warn if null_context is set on both pattern and subject, because the
1060 memory details cannot then be shown.
1061
1062 (b) Remember (up to a certain number of) memory allocations and their
1063 lengths, and list only the lengths, so as to be system-independent.
1064 (In practice, the new interpreter never has more than 2 blocks allocated
1065 simultaneously.)
1066
1067 18. Make pcre2test detect an error return from pcre2_get_error_message(), give
1068 a message, and abandon the run (this would have detected #13 above).
1069
1070 19. Implemented PCRE2_ENDANCHORED.
1071
1072 20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement
1073 the --output=text (-O) option and the inbuilt callout echo.
1074
1075 21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and
1076 single-branch conditions with a false condition (e.g. DEFINE) at the start of a
1077 branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as
1078 anchored.
1079
1080 22. Added an explicit limit on the amount of heap used by pcre2_match(), set by
1081 pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the
1082 heap limit along with other pattern information, and to find the minimum when
1083 the find_limits modifier is set.
1084
1085 23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled
1086 pattern is set up so as to initialize any padding the compiler might have
1087 included. This avoids valgrind warnings when a compiled pattern is copied, in
1088 particular when it is serialized.
1089
1090 24. Remove a redundant line of code left in accidentally a long time ago.
1091
1092 25. Remove a duplication typo in pcre2_tables.c
1093
1094 26. Correct an incorrect cast in pcre2_valid_utf.c
1095
1096 27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the
1097 tests to improve coverage.
1098
1099 28. Some fixes/tidies as a result of looking at Coverity Scan output:
1100
1101 (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.
1102 (b) Added some casts to avoid "suspicious implicit sign extension".
1103 (c) Resource leaks in pcre2test in rare error cases.
1104 (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge
1105 for checking at compile time that tables are the right size.
1106 (e) Add missing "fall through" comment.
1107
1108 29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.
1109
1110 30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.
1111
1112 31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in
1113 pcre2test, a crash could occur.
1114
1115 32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so
1116 that all the tests can run with clang's sanitizing options.
1117
1118 33. Implement extra compile options in the compile context and add the first
1119 one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.
1120
1121 34. Implement newline type PCRE2_NEWLINE_NUL.
1122
1123 35. A lookbehind assertion that had a zero-length branch caused undefined
1124 behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.
1125
1126 36. The match limit value now also applies to pcre2_dfa_match() as there are
1127 patterns that can use up a lot of resources without necessarily recursing very
1128 deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.
1129
1130 37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
1131
1132 38. Fix returned offsets from regexec() when REG_STARTEND is used with a
1133 starting offset greater than zero.
1134
1135 39. Implement REG_PEND (GNU extension) for the POSIX wrapper.
1136
1137 40. Implement the subject_literal modifier in pcre2test, and allow jitstack on
1138 pattern lines.
1139
1140 41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.
1141
1142 42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit
1143 of pcre2grep.
1144
1145 43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,
1146 PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:
1147
1148 (a) The -F option did not work for fixed strings containing \E.
1149 (b) The -w option did not work for patterns with multiple branches.
1150
1151 44. Added configuration options for the SELinux compatible execmem allocator in
1152 JIT.
1153
1154 45. Increased the limit for searching for a "must be present" code unit in
1155 subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are
1156 much faster.
1157
1158 46. Arrange for anchored patterns to record and use "first code unit" data,
1159 because this can give a fast "no match" without searching for a "required code
1160 unit". Previously only non-anchored patterns did this.
1161
1162 47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.
1163
1164 48. Add the callout_no_where modifier to pcre2test.
1165
1166 49. Update extended grapheme breaking rules to the latest set that are in
1167 Unicode Standard Annex #29.
1168
1169 50. Added experimental foreign pattern conversion facilities
1170 (pcre2_pattern_convert() and friends).
1171
1172 51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE
1173 is defined in a system header in cygwin. Also modified some of the #ifdefs in
1174 pcre2grep related to Windows and Cygwin support.
1175
1176 52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a
1177 character class is the last character in the class, Perl does not give a
1178 warning. PCRE2 now also treats this as a literal.
1179
1180 53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was
1181 not doing so for [\d-X] (and similar escapes), as is documented.
1182
1183 54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard.
1184
1185 55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in
1186 pcre2_compile() which could never actually trigger (code should have been cut
1187 out when Unicode support is disabled).
1188
1189
1190 Version 10.23 14-February-2017
1191 ------------------------------
1192
1193 1. Extended pcre2test with the utf8_input modifier so that it is able to
1194 generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
1195
1196 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
1197 PCRE2_UCP set, a negative character type such as \D in a positive class should
1198 cause all characters greater than 255 to match, whatever else is in the class.
1199 There was a bug that caused this not to happen if a Unicode property item was
1200 added to such a class, for example [\D\P{Nd}] or [\W\pL].
1201
1202 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
1203 checking is now done in the pre-pass that identifies capturing groups. This has
1204 reduced the amount of duplication and made the code tidier. While doing this,
1205 some minor bugs and Perl incompatibilities were fixed, including:
1206
1207 (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
1208 of giving an invalid quantifier error.
1209
1210 (b) {0} can now be used after a group in a lookbehind assertion; previously
1211 this caused an "assertion is not fixed length" error.
1212
1213 (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
1214 the name "DEFINE" exists. PCRE2 now does likewise.
1215
1216 (d) A recursion condition test such as (?(R2)...) must now refer to an
1217 existing subpattern.
1218
1219 (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
1220 group whose name began with "R".
1221
1222 (f) When testing zero-terminated patterns under valgrind, the terminating
1223 zero is now marked "no access". This catches bugs that would otherwise
1224 show up only with non-zero-terminated patterns.
1225
1226 (g) A hyphen appearing immediately after a POSIX character class (for example
1227 /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
1228 literal, but gives a warning, so it seems best to fail it in PCRE.
1229
1230 (h) An empty \Q\E sequence may appear after a callout that precedes an
1231 assertion condition (it is, of course, ignored).
1232
1233 One effect of the refactoring is that some error numbers and messages have
1234 changed, and the pattern offset given for compiling errors is not always the
1235 right-most character that has been read. In particular, for a variable-length
1236 lookbehind assertion it now points to the start of the assertion. Another
1237 change is that when a callout appears before a group, the "length of next
1238 pattern item" that is passed now just gives the length of the opening
1239 parenthesis item, not the length of the whole group. A length of zero is now
1240 given only for a callout at the end of the pattern. Automatic callouts are no
1241 longer inserted before and after explicit callouts in the pattern.
1242
1243 A number of bugs in the refactored code were subsequently fixed during testing
1244 before release, but after the code was made available in the repository. Many
1245 of the bugs were discovered by fuzzing testing. Several of them were related to
1246 the change from assuming a zero-terminated pattern (which previously had
1247 required non-zero terminated strings to be copied). These bugs were never in
1248 fully released code, but are noted here for the record.
1249
1250 (a) An overall recursion such as (?0) inside a lookbehind assertion was not
1251 being diagnosed as an error.
1252
1253 (b) In utf mode, the length of a *MARK (or other verb) name was being checked
1254 in characters instead of code units, which could lead to bad code being
1255 compiled, leading to unpredictable behaviour.
1256
1257 (c) In extended /x mode, characters whose code was greater than 255 caused
1258 a lookup outside one of the global tables. A similar bug existed for wide
1259 characters in *VERB names.
1260
1261 (d) The amount of memory needed for a compiled pattern was miscalculated if a
1262 lookbehind contained more than one toplevel branch and the first branch
1263 was of length zero.
1264
1265 (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
1266 terminated pattern, if a # comment ran on to the end of the pattern, one
1267 or more code units past the end were being read.
1268
1269 (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
1270 "{2,2") could cause reading beyond the pattern.
1271
1272 (g) When reading a callout string, if the end delimiter was at the end of the
1273 pattern one further code unit was read.
1274
1275 (h) An unterminated number after \g' could cause reading beyond the pattern.
1276
1277 (i) An insufficient memory size was being computed for compiling with
1278 PCRE2_AUTO_CALLOUT.
1279
1280 (j) A conditional group with an assertion condition used more memory than was
1281 allowed for it during parsing, so too many of them could therefore
1282 overrun a buffer.
1283
1284 (k) If parsing a pattern exactly filled the buffer, the internal test for
1285 overrun did not check when the final META_END item was added.
1286
1287 (l) If a lookbehind contained a subroutine call, and the called group
1288 contained an option setting such as (?s), and the PCRE2_ANCHORED option
1289 was set, unpredictable behaviour could occur. The underlying bug was
1290 incorrect code and insufficient checking while searching for the end of
1291 the called subroutine in the parsed pattern.
1292
1293 (m) Quantifiers following (*VERB)s were not being diagnosed as errors.
1294
1295 (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
1296 PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
1297
1298 (o) If \Q was preceded by a quantified item, and the following \E was
1299 followed by '?' or '+', and there was at least one literal character
1300 between them, an internal error "unexpected repeat" occurred (example:
1301 /.+\QX\E+/).
1302
1303 (p) A buffer overflow could occur while sorting the names in the group name
1304 list (depending on the order in which the names were seen).
1305
1306 (q) A conditional group that started with a callout was not doing the right
1307 check for a following assertion, leading to compiling bad code. Example:
1308 /(?(C'XX))?!XX/
1309
1310 (r) If a character whose code point was greater than 0xffff appeared within
1311 a lookbehind that was within another lookbehind, the calculation of the
1312 lookbehind length went wrong and could provoke an internal error.
1313
1314 (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
1315 an internal error. Now the hyphen is treated as a literal.
1316
1317 4. Back references are now permitted in lookbehind assertions when there are
1318 no duplicated group numbers (that is, (?| has not been used), and, if the
1319 reference is by name, there is only one group of that name. The referenced
1320 group must, of course be of fixed length.
1321
1322 5. pcre2test has been upgraded so that, when run under valgrind with valgrind
1323 support enabled, reading past the end of the pattern is detected, both when
1324 compiling and during callout processing.
1325
1326 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
1327 reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
1328 not recognize this syntax.
1329
1330 7. Automatic callouts are no longer generated before and after callouts in the
1331 pattern.
1332
1333 8. When pcre2test was outputing information from a callout, the caret indicator
1334 for the current position in the subject line was incorrect if it was after an
1335 escape sequence for a character whose code point was greater than \x{ff}.
1336
1337 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
1338 PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
1339
1340 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
1341 expansion when long lines are encountered. Original patch by Dmitry
1342 Cherniachenko.
1343
1344 11. If pcre2grep was compiled with JIT support, but the library was compiled
1345 without it (something that neither ./configure nor CMake allow, but it can be
1346 done by editing config.h), pcre2grep was giving a JIT error. Now it detects
1347 this situation and does not try to use JIT.
1348
1349 12. Added some "const" qualifiers to variables in pcre2grep.
1350
1351 13. Added Dmitry Cherniachenko's patch for colouring output in Windows
1352 (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
1353 variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
1354
1355 14. Add the -t (grand total) option to pcre2grep.
1356
1357 15. A number of bugs have been mended relating to match start-up optimizations
1358 when the first thing in a pattern is a positive lookahead. These all applied
1359 only when PCRE2_NO_START_OPTIMIZE was *not* set:
1360
1361 (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
1362 both an initial 'X' and a following 'X'.
1363 (b) Some patterns starting with an assertion that started with .* were
1364 incorrectly optimized as having to match at the start of the subject or
1365 after a newline. There are cases where this is not true, for example,
1366 (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
1367 start with spaces. Starting .* in an assertion is no longer taken as an
1368 indication of matching at the start (or after a newline).
1369
1370 16. The "offset" modifier in pcre2test was not being ignored (as documented)
1371 when the POSIX API was in use.
1372
1373 17. Added --enable-fuzz-support to "configure", causing an non-installed
1374 library containing a test function that can be called by fuzzers to be
1375 compiled. A non-installed binary to run the test function locally, called
1376 pcre2fuzzcheck is also compiled.
1377
1378 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
1379 which started with .* inside a positive lookahead was incorrectly being
1380 compiled as implicitly anchored.
1381
1382 19. Removed all instances of "register" declarations, as they are considered
1383 obsolete these days and in any case had become very haphazard.
1384
1385 20. Add strerror() to pcre2test for failed file opening.
1386
1387 21. Make pcre2test -C list valgrind support when it is enabled.
1388
1389 22. Add the use_length modifier to pcre2test.
1390
1391 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
1392 'copy' modifiers.
1393
1394 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
1395 is apparently needed there as well as in the function definitions. (Why did
1396 nobody ask for this in PCRE1?)
1397
1398 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
1399 PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
1400 compliant and unique.
1401
1402 26. pcre2-config --libs-posix was listing -lpcre2posix instead of
1403 -lpcre2-posix. Also, the CMake build process was building the library with the
1404 wrong name.
1405
1406 27. In pcre2test, give some offset information for errors in hex patterns.
1407 This uses the C99 formatting sequence %td, except for MSVC which doesn't
1408 support it - %lu is used instead.
1409
1410 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
1411 pcre2test for testing it.
1412
1413 29. Fix small memory leak in pcre2test.
1414
1415 30. Fix out-of-bounds read for partial matching of /./ against an empty string
1416 when the newline type is CRLF.
1417
1418 31. Fix a bug in pcre2test that caused a crash when a locale was set either in
1419 the current pattern or a previous one and a wide character was matched.
1420
1421 32. The appearance of \p, \P, or \X in a substitution string when
1422 PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
1423 dereference).
1424
1425 33. If the starting offset was specified as greater than the subject length in
1426 a call to pcre2_substitute() an out-of-bounds memory reference could occur.
1427
1428 34. When PCRE2 was compiled to use the heap instead of the stack for recursive
1429 calls to match(), a repeated minimizing caseless back reference, or a
1430 maximizing one where the two cases had different numbers of code units,
1431 followed by a caseful back reference, could lose the caselessness of the first
1432 repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
1433 but didn't).
1434
1435 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
1436 matching length and just records zero. Typically this happens when there are
1437 too many nested or recursive back references. If the limit was reached in
1438 certain recursive cases it failed to be triggered and an internal error could
1439 be the result.
1440
1441 36. The pcre2_dfa_match() function now takes note of the recursion limit for
1442 the internal recursive calls that are used for lookrounds and recursions within
1443 the pattern.
1444
1445 37. More refactoring has got rid of the internal could_be_empty_branch()
1446 function (around 400 lines of code, including comments) by keeping track of
1447 could-be-emptiness as the pattern is compiled instead of scanning compiled
1448 groups. (This would have been much harder before the refactoring of #3 above.)
1449 This lifts a restriction on the number of branches in a group (more than about
1450 1100 would give "pattern is too complicated").
1451
1452 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
1453 auto_callout".
1454
1455 39. In a library with Unicode support, incorrect data was compiled for a
1456 pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
1457 characters to match (for example, /[\s[:^ascii:]]/).
1458
1459 40. The callout_error modifier has been added to pcre2test to make it possible
1460 to return PCRE2_ERROR_CALLOUT from a callout.
1461
1462 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
1463 "<esc>[00m".
1464
1465 42. The limit in the auto-possessification code that was intended to catch
1466 overly-complicated patterns and not spend too much time auto-possessifying was
1467 being reset too often, resulting in very long compile times for some patterns.
1468 Now such patterns are no longer completely auto-possessified.
1469
1470 43. Applied Jason Hood's revised patch for RunTest.bat.
1471
1472 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
1473
1474 45. Minor cosmetic fix to pcre2test: move a variable that is not used under
1475 Windows into the "not Windows" code.
1476
1477 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
1478 some of the code:
1479
1480 * normalised the Windows condition by ensuring WIN32 is defined;
1481 * enables the callout feature under Windows;
1482 * adds globbing (Microsoft's implementation expands quoted args),
1483 using a tweaked opendirectory;
1484 * implements the is_*_tty functions for Windows;
1485 * --color=always will write the ANSI sequences to file;
1486 * add sequences 4 (underline works on Win10) and 5 (blink as bright
1487 background, relatively standard on DOS/Win);
1488 * remove the (char *) casts for the now-const strings;
1489 * remove GREP_COLOUR (grep's command line allowed the 'u', but not
1490 the environment), parsing GREP_COLORS instead;
1491 * uses the current colour if not set, rather than black;
1492 * add print_match for the undefined case;
1493 * fixes a typo.
1494
1495 In addition, colour settings containing anything other than digits and
1496 semicolon are ignored, and the colour controls are no longer output for empty
1497 strings.
1498
1499 47. Detecting patterns that are too large inside the length-measuring loop
1500 saves processing ridiculously long patterns to their end.
1501
1502 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
1503 just wastes time. In the UTF case it can also produce redundant entries in
1504 XCLASS lists caused by characters with multiple other cases and pairs of
1505 characters in the same "not-x" sublists.
1506
1507 49. A pattern such as /(?=(a\K))/ can report the end of the match being before
1508 its start; pcre2test was not handling this correctly when using the POSIX
1509 interface (it was OK with the native interface).
1510
1511 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
1512 continue to work, falling back to interpretation if anything goes wrong with
1513 JIT.
1514
1515 51. Applied patches from Christian Persch to configure.ac to make use of the
1516 AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT
1517 modules.
1518
1519 52. Minor fixes to pcre2grep from Jason Hood:
1520 * fixed some spacing;
1521 * Windows doesn't usually use single quotes, so I've added a define
1522 to use appropriate quotes [in an example];
1523 * LC_ALL was displayed as "LCC_ALL";
1524 * numbers 11, 12 & 13 should end in "th";
1525 * use double quotes in usage message.
1526
1527 53. When autopossessifying, skip empty branches without recursion, to reduce
1528 stack usage for the benefit of clang with -fsanitize-address, which uses huge
1529 stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553.
1530
1531 54. A pattern with very many explicit back references to a group that is a long
1532 way from the start of the pattern could take a long time to compile because
1533 searching for the referenced group in order to find the minimum length was
1534 being done repeatedly. Now up to 128 group minimum lengths are cached and the
1535 attempt to find a minimum length is abandoned if there is a back reference to a
1536 group whose number is greater than 128. (In that case, the pattern is so
1537 complicated that this optimization probably isn't worth it.) This fixes
1538 oss-fuzz issue 557.
1539
1540 55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline
1541 mode with --only-matching matched several lines, it restarted scanning at the
1542 next line instead of moving on to the end of the matched string, which can be
1543 several lines after the start.
1544
1545 56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line
1546 with updates to the non-Windows version.
1547
1548
1549
1550 Version 10.22 29-July-2016
1551 --------------------------
1552
1553 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
1554 to fix problems with running the tests under Windows.
1555
1556 2. Implemented a facility for quoting literal characters within hexadecimal
1557 patterns in pcre2test, to make it easier to create patterns with just a few
1558 non-printing characters.
1559
1560 3. Binary zeros are not supported in pcre2test input files. It now detects them
1561 and gives an error.
1562
1563 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
1564 smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
1565 that it matches only unknown objects.
1566
1567 5. Updated the maintenance script maint/ManyConfigTests to make it easier to
1568 select individual groups of tests.
1569
1570 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
1571 used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
1572 disables the use of back references (and subroutine calls), which are supported
1573 by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
1574 longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
1575 and pmatch when regexec() is called.
1576
1577 7. Because of 6 above, pcre2test has been modified with a new modifier called
1578 posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
1579 modifier had this effect. That option is now ignored when the POSIX API is in
1580 use.
1581
1582 8. Minor tidies to the pcre2demo.c sample program, including more comments
1583 about its 8-bit-ness.
1584
1585 9. Detect unmatched closing parentheses and give the error in the pre-scan
1586 instead of later. Previously the pre-scan carried on and could give a
1587 misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
1588 message about invalid duplicate group names.
1589
1590 10. It has happened that pcre2test was accidentally linked with another POSIX
1591 regex library instead of libpcre2-posix. In this situation, a call to regcomp()
1592 (in the other library) may succeed, returning zero, but of course putting its
1593 own data into the regex_t block. In one example the re_pcre2_code field was
1594 left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
1595 so it treated the next line as another pattern line, resulting in a confusing
1596 error message. A check has been added to pcre2test to see if the data returned
1597 from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
1598 are not, an error message is output and the pcre2test run is abandoned. The
1599 message points out the possibility of a mis-linking. Hopefully this will avoid
1600 some head-scratching the next time this happens.
1601
1602 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
1603 assertion, caused pcre2test to output a very large number of spaces when the
1604 callout was taken, making the program appearing to loop.
1605
1606 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
1607 nested set of parentheses of sufficient size caused an overflow of the
1608 compiling workspace (which was diagnosed, but of course is not desirable).
1609
1610 13. Detect missing closing parentheses during the pre-pass for group
1611 identification.
1612
1613 14. Changed some integer variable types and put in a number of casts, following
1614 a report of compiler warnings from Visual Studio 2013 and a few tests with
1615 gcc's -Wconversion (which still throws up a lot).
1616
1617 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
1618 for testing it.
1619
1620 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
1621 regerror(). When the error buffer is too small, my version of snprintf() puts a
1622 binary zero in the final byte. Bug #1801 seems to show that other versions do
1623 not do this, leading to bad output from pcre2test when it was checking for
1624 buffer overflow. It no longer assumes a binary zero at the end of a too-small
1625 regerror() buffer.
1626
1627 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
1628 actually affect anything, by sheer luck.
1629
1630 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
1631 "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
1632 older MSVC compilers. This has been done both in src/pcre2_internal.h for most
1633 of the library, and also in src/pcre2posix.c, which no longer includes
1634 pcre2_internal.h (see 24 below).
1635
1636 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
1637 static compilation. Subsequently applied Chris Wilson's second patch, putting
1638 the first patch under a new option instead of being unconditional when
1639 PCRE_STATIC is set.
1640
1641 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
1642 to convert \r\n at the ends of reflected lines into \r\r\n. This required
1643 ensuring that other output that is written to stdout (e.g. file names) uses the
1644 appropriate line terminator: \r\n for Windows, \n otherwise.
1645
1646 21. When a line is too long for pcre2grep's internal buffer, show the maximum
1647 length in the error message.
1648
1649 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
1650 additions).
1651
1652 23. RunTest.bat was missing a "set type" line for test 22.
1653
1654 24. The pcre2posix.c file was including pcre2_internal.h, and using some
1655 "private" knowledge of the data structures. This is unnecessary; the code has
1656 been re-factored and no longer includes pcre2_internal.h.
1657
1658 25. A racing condition is fixed in JIT reported by Mozilla.
1659
1660 26. Minor code refactor to avoid "array subscript is below array bounds"
1661 compiler warning.
1662
1663 27. Minor code refactor to avoid "left shift of negative number" warning.
1664
1665 28. Add a bit more sanity checking to pcre2_serialize_decode() and document
1666 that it expects trusted data.
1667
1668 29. Fix typo in pcre2_jit_test.c
1669
1670 30. Due to an oversight, pcre2grep was not making use of JIT when available.
1671 This is now fixed.
1672
1673 31. The RunGrepTest script is updated to use the valgrind suppressions file
1674 when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
1675 file is updated so that is now the same as for PCRE1: it suppresses the
1676 Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
1677 code). Also changed smc-check=all to smc-check=all-non-file as was done for
1678 RunTest (see 4 above).
1679
1680 32. Implemented the PCRE2_NO_JIT option for pcre2_match().
1681
1682 33. Fix typo that gave a compiler error when JIT not supported.
1683
1684 34. Fix comment describing the returns from find_fixedlength().
1685
1686 35. Fix potential negative index in pcre2test.
1687
1688 36. Calls to pcre2_get_error_message() with error numbers that are never
1689 returned by PCRE2 functions were returning empty strings. Now the error code
1690 PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
1691 show the texts for given error numbers (i.e. to call pcre2_get_error_message()
1692 and display what it returns) and a few representative error codes are now
1693 checked in RunTest.
1694
1695 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
1696 pcre2_match.c, in anticipation that this is needed for the same reason it was
1697 recently added to pcrecpp.cc in PCRE1.
1698
1699 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
1700 the match extended over a line boundary, as it tried to find more matches "on
1701 the same line" - but it was already over the end.
1702
1703 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
1704 to the same code as '.' when PCRE2_DOTALL is set).
1705
1706 40. Fix two clang compiler warnings in pcre2test when only one code unit width
1707 is supported.
1708
1709 41. Upgrade RunTest to automatically re-run test 2 with a large (64MiB) stack
1710 if it fails when running the interpreter with a 16MiB stack (and if changing
1711 the stack size via pcre2test is possible). This avoids having to manually set a
1712 large stack size when testing with clang.
1713
1714 42. Fix register overwite in JIT when SSE2 acceleration is enabled.
1715
1716 43. Detect integer overflow in pcre2test pattern and data repetition counts.
1717
1718 44. In pcre2test, ignore "allcaptures" after DFA matching.
1719
1720 45. Fix unaligned accesses on x86. Patch by Marc Mutz.
1721
1722 46. Fix some more clang compiler warnings.
1723
1724
1725 Version 10.21 12-January-2016
1726 -----------------------------
1727
1728 1. Improve matching speed of patterns starting with + or * in JIT.
1729
1730 2. Use memchr() to find the first character in an unanchored match in 8-bit
1731 mode in the interpreter. This gives a significant speed improvement.
1732
1733 3. Removed a redundant copy of the opcode_possessify table in the
1734 pcre2_auto_possessify.c source.
1735
1736 4. Fix typos in dftables.c for z/OS.
1737
1738 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
1739 processing them could involve a buffer overflow if the following character was
1740 an opening parenthesis.
1741
1742 6. Change 36 for 10.20 also introduced a bug in processing this pattern:
1743 /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
1744 setting (which (*:0) is), then (?x) did not get unset at the end of its group
1745 during the scan for named groups, and hence the external # was incorrectly
1746 treated as a comment and the invalid (?' at the end of the pattern was not
1747 diagnosed. This caused a buffer overflow during the real compile. This bug was
1748 discovered by Karl Skomski with the LLVM fuzzer.
1749
1750 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
1751 own source module to avoid a circular dependency between src/pcre2_compile.c
1752 and src/pcre2_study.c
1753
1754 8. A callout with a string argument containing an opening square bracket, for
1755 example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
1756 overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1757
1758 9. The handling of callouts during the pre-pass for named group identification
1759 has been tightened up.
1760
1761 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
1762 possessive. This is a very minor optimization.
1763
1764 11. A possessively repeated conditional group that could match an empty string,
1765 for example, /(?(R))*+/, was incorrectly compiled.
1766
1767 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
1768 Persch).
1769
1770 13. An empty comment (?#) in a pattern was incorrectly processed and could
1771 provoke a buffer overflow. This bug was discovered by Karl Skomski with the
1772 LLVM fuzzer.
1773
1774 14. Fix infinite recursion in the JIT compiler when certain patterns such as
1775 /(?:|a|){100}x/ are analysed.
1776
1777 15. Some patterns with character classes involving [: and \\ were incorrectly
1778 compiled and could cause reading from uninitialized memory or an incorrect
1779 error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
1780 first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
1781
1782 16. Pathological patterns containing many nested occurrences of [: caused
1783 pcre2_compile() to run for a very long time. This bug was found by the LLVM
1784 fuzzer.
1785
1786 17. A missing closing parenthesis for a callout with a string argument was not
1787 being diagnosed, possibly leading to a buffer overflow. This bug was found by
1788 the LLVM fuzzer.
1789
1790 18. A conditional group with only one branch has an implicit empty alternative
1791 branch and must therefore be treated as potentially matching an empty string.
1792
1793 19. If (?R was followed by - or + incorrect behaviour happened instead of a
1794 diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1795
1796 20. Another bug that was introduced by change 36 for 10.20: conditional groups
1797 whose condition was an assertion preceded by an explicit callout with a string
1798 argument might be incorrectly processed, especially if the string contained \Q.
1799 This bug was discovered by Karl Skomski with the LLVM fuzzer.
1800
1801 21. Compiling PCRE2 with the sanitize options of clang showed up a number of
1802 very pedantic coding infelicities and a buffer overflow while checking a UTF-8
1803 string if the final multi-byte UTF-8 character was truncated.
1804
1805 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
1806 class, where both values are literal letters in the same case, omit the
1807 non-letter EBCDIC code points within the range.
1808
1809 23. Finding the minimum matching length of complex patterns with back
1810 references and/or recursions can take a long time. There is now a cut-off that
1811 gives up trying to find a minimum length when things get too complex.
1812
1813 24. An optimization has been added that speeds up finding the minimum matching
1814 length for patterns containing repeated capturing groups or recursions.
1815
1816 25. If a pattern contained a back reference to a group whose number was
1817 duplicated as a result of appearing in a (?|...) group, the computation of the
1818 minimum matching length gave a wrong result, which could cause incorrect "no
1819 match" errors. For such patterns, a minimum matching length cannot at present
1820 be computed.
1821
1822 26. Added a check for integer overflow in conditions (?(<digits>) and
1823 (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
1824 fuzzer.
1825
1826 27. Fixed an issue when \p{Any} inside an xclass did not read the current
1827 character.
1828
1829 28. If pcre2grep was given the -q option with -c or -l, or when handling a
1830 binary file, it incorrectly wrote output to stdout.
1831
1832 29. The JIT compiler did not restore the control verb head in case of *THEN
1833 control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1834
1835 30. The way recursive references such as (?3) are compiled has been re-written
1836 because the old way was the cause of many issues. Now, conversion of the group
1837 number into a pattern offset does not happen until the pattern has been
1838 completely compiled. This does mean that detection of all infinitely looping
1839 recursions is postponed till match time. In the past, some easy ones were
1840 detected at compile time. This re-writing was done in response to yet another
1841 bug found by the LLVM fuzzer.
1842
1843 31. A test for a back reference to a non-existent group was missing for items
1844 such as \987. This caused incorrect code to be compiled. This issue was found
1845 by Karl Skomski with a custom LLVM fuzzer.
1846
1847 32. Error messages for syntax errors following \g and \k were giving inaccurate
1848 offsets in the pattern.
1849
1850 33. Improve the performance of starting single character repetitions in JIT.
1851
1852 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
1853
1854 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
1855 give the right offset instead of zero.
1856
1857 36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
1858 This issue was found by Karl Skomski with a custom LLVM fuzzer.
1859
1860 37. The JIT compiler should restore the control chain for empty possessive
1861 repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1862
1863 38. A bug which was introduced by the single character repetition optimization
1864 was fixed.
1865
1866 39. Match limit check added to recursion. This issue was found by Karl Skomski
1867 with a custom LLVM fuzzer.
1868
1869 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
1870 only at the part of the subject that is relevant when the starting offset is
1871 non-zero.
1872
1873 41. Improve first character match in JIT with SSE2 on x86.
1874
1875 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
1876 with a custom LLVM fuzzer.
1877
1878 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
1879 III).
1880
1881 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
1882 test (there are now 20 in total).
1883
1884 45. Fixed a corner case of range optimization in JIT.
1885
1886 46. Add the ${*MARK} facility to pcre2_substitute().
1887
1888 47. Modifier lists in pcre2test were splitting at spaces without the required
1889 commas.
1890
1891 48. Implemented PCRE2_ALT_VERBNAMES.
1892
1893 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
1894 LLVM fuzzer.
1895
1896 50. The pcre2test program has been extended by adding the #newline_default
1897 command. This has made it possible to run the standard tests when PCRE2 is
1898 compiled with either CR or CRLF as the default newline convention. As part of
1899 this work, the new command was added to several test files and the testing
1900 scripts were modified. The pcre2grep tests can now also be run when there is no
1901 LF in the default newline convention.
1902
1903 51. The RunTest script has been modified so that, when JIT is used and valgrind
1904 is specified, a valgrind suppressions file is set up to ignore "Invalid read of
1905 size 16" errors because these are false positives when the hardware supports
1906 the SSE2 instruction set.
1907
1908 52. It is now possible to have comment lines amid the subject strings in
1909 pcre2test (and perltest.sh) input.
1910
1911 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
1912
1913 54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
1914 and the matching functions with NULL contexts can be tested.
1915
1916 55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
1917
1918 56. In a character class such as [\W\p{Any}] where both a negative-type escape
1919 ("not a word character") and a property escape were present, the property
1920 escape was being ignored.
1921
1922 57. Fixed integer overflow for patterns whose minimum matching length is very,
1923 very large.
1924
1925 58. Implemented --never-backslash-C.
1926
1927 59. Change 55 above introduced a bug by which certain patterns provoked the
1928 erroneous error "\ at end of pattern".
1929
1930 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
1931 errors or other strange effects if compiled in UCP mode. Found with libFuzzer
1932 and AddressSanitizer.
1933
1934 61. Whitespace at the end of a pcre2test pattern line caused a spurious error
1935 message if there were only single-character modifiers. It should be ignored.
1936
1937 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
1938 or segmentation errors for some patterns. Found with libFuzzer and
1939 AddressSanitizer.
1940
1941 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
1942 overflow.
1943
1944 64. Improve error message for overly-complicated patterns.
1945
1946 65. Implemented an optional replication feature for patterns in pcre2test, to
1947 make it easier to test long repetitive patterns. The tests for 63 above are
1948 converted to use the new feature.
1949
1950 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
1951 misbehave.
1952
1953 67. In pcre2_substitute() in UTF mode, the UTF validity check on the
1954 replacement string was happening before the length setting when the replacement
1955 string was zero-terminated.
1956
1957 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the
1958 second and subsequent calls to pcre2_match().
1959
1960 69. There was no check for integer overflow for a replacement group number in
1961 pcre2_substitute(). An added check for a number greater than the largest group
1962 number in the pattern means this is not now needed.
1963
1964 70. The PCRE2-specific VERSION condition didn't work correctly if only one
1965 digit was given after the decimal point, or if more than two digits were given.
1966 It now works with one or two digits, and gives a compile time error if more are
1967 given.
1968
1969 71. In pcre2_substitute() there was the possibility of reading one code unit
1970 beyond the end of the replacement string.
1971
1972 72. The code for checking a subject's UTF-32 validity for a pattern with a
1973 lookbehind involved an out-of-bounds pointer, which could potentially cause
1974 trouble in some environments.
1975
1976 73. The maximum lookbehind length was incorrectly calculated for patterns such
1977 as /(?<=(a)(?-1))x/ which have a recursion within a backreference.
1978
1979 74. Give an error if a lookbehind assertion is longer than 65535 code units.
1980
1981 75. Give an error in pcre2_substitute() if a match ends before it starts (as a
1982 result of the use of \K).
1983
1984 76. Check the length of subpattern names and the names in (*MARK:xx) etc.
1985 dynamically to avoid the possibility of integer overflow.
1986
1987 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the
1988 size of patterns that they are prepared to handle.
1989
1990 78. (*NO_AUTO_POSSESS) was not working.
1991
1992 79. Adding group information caching improves the speed of compiling when
1993 checking whether a group has a fixed length and/or could match an empty string,
1994 especially when recursion or subroutine calls are involved. However, this
1995 cannot be used when (?| is present in the pattern because the same number may
1996 be used for groups of different sizes. To catch runaway patterns in this
1997 situation, counts have been introduced to the functions that scan for empty
1998 branches or compute fixed lengths.
1999
2000 80. Allow for the possibility of the size of the nest_save structure not being
2001 a factor of the size of the compiling workspace (it currently is).
2002
2003 81. Check for integer overflow in minimum length calculation and cap it at
2004 65535.
2005
2006 82. Small optimizations in code for finding the minimum matching length.
2007
2008 83. Lock out configuring for EBCDIC with non-8-bit libraries.
2009
2010 84. Test for error code <= 0 in regerror().
2011
2012 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute().
2013
2014 86. Avoid the possibility of computing with an out-of-bounds pointer (though
2015 not dereferencing it) while handling lookbehind assertions.
2016
2017 87. Failure to get memory for the match data in regcomp() is now given as a
2018 regcomp() error instead of waiting for regexec() to pick it up.
2019
2020 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid
2021 newline sequence.
2022
2023 89. Paranoid check in regcomp() for bad error code from pcre2_compile().
2024
2025 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well
2026 as for link size 2.
2027
2028 91. Document that JIT has a limit on pattern size, and give more information
2029 about JIT compile failures in pcre2test.
2030
2031 92. Implement PCRE2_INFO_HASBACKSLASHC.
2032
2033 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports
2034 with JIT (possibly caused by SSE2?).
2035
2036 94. Support offset_limit in JIT.
2037
2038 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed
2039 by a single ASCII character in a class item, was incorrectly compiled in UCP
2040 mode. The POSIX class got lost, but only if the single character followed it.
2041
2042 96. [:punct:] in UCP mode was matching some characters in the range 128-255
2043 that should not have been matched.
2044
2045 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all
2046 characters with code points greater than 255 are in the class. When a Unicode
2047 property was also in the class (if PCRE2_UCP is set, escapes such as \w are
2048 turned into Unicode properties), wide characters were not correctly handled,
2049 and could fail to match.
2050
2051 98. In pcre2test, make the "startoffset" modifier a synonym of "offset",
2052 because it sets the "startoffset" parameter for pcre2_match().
2053
2054 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between
2055 an item and its qualifier (for example, A(?#comment)?B) pcre2_compile()
2056 misbehaved. This bug was found by the LLVM fuzzer.
2057
2058 100. The error for an invalid UTF pattern string always gave the code unit
2059 offset as zero instead of where the invalidity was found.
2060
2061 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not
2062 working correctly in UCP mode.
2063
2064 102. Similar to 99 above, if an isolated \E was present between an item and its
2065 qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug
2066 was found by the LLVM fuzzer.
2067
2068 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND
2069 was set when the pmatch argument was NULL. It now returns REG_INVARG.
2070
2071 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep.
2072
2073 105. An empty \Q\E sequence between an item and its qualifier caused
2074 pcre2_compile() to misbehave when auto callouts were enabled. This bug
2075 was found by the LLVM fuzzer.
2076
2077 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or
2078 other verb "name" ended with whitespace immediately before the closing
2079 parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when
2080 both those options were set.
2081
2082 107. In a number of places pcre2_compile() was not handling NULL characters
2083 correctly, and pcre2test with the "bincode" modifier was not always correctly
2084 displaying fields containing NULLS:
2085
2086 (a) Within /x extended #-comments
2087 (b) Within the "name" part of (*MARK) and other *verbs
2088 (c) Within the text argument of a callout
2089
2090 108. If a pattern that was compiled with PCRE2_EXTENDED started with white
2091 space or a #-type comment that was followed by (?-x), which turns off
2092 PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again,
2093 pcre2_compile() assumed that (?-x) applied to the whole pattern and
2094 consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix
2095 for this bug means that a setting of any of the (?imsxJU) options at the start
2096 of a pattern is no longer transferred to the options that are returned by
2097 PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have
2098 changed when the effects of those options were all moved to compile time.
2099
2100 109. An escaped closing parenthesis in the "name" part of a (*verb) when
2101 PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug
2102 was found by the LLVM fuzzer.
2103
2104 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it
2105 possible to test it.
2106
2107 111. "Harden" pcre2test against ridiculously large values in modifiers and
2108 command line arguments.
2109
2110 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_
2111 LENGTH.
2112
2113 113. Fix printing of *MARK names that contain binary zeroes in pcre2test.
2114
2115
2116 Version 10.20 30-June-2015
2117 --------------------------
2118
2119 1. Callouts with string arguments have been added.
2120
2121 2. Assertion code generator in JIT has been optimized.
2122
2123 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
2124 pcre2_compile() function read past the end of the input before diagnosing an
2125 error. This bug was discovered by the LLVM fuzzer.
2126
2127 4. Implemented pcre2_callout_enumerate().
2128
2129 5. Fix JIT compilation of conditional blocks whose assertion is converted to
2130 (*FAIL). E.g: /(?(?!))/.
2131
2132 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
2133 discovered by the LLVM fuzzer.
2134
2135 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
2136 when this assertion was used as a condition, for example (?(?!)a|b). In
2137 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
2138 error about an unsupported item.
2139
2140 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
2141 possessification code could take exponential time to complete. A recursion
2142 depth limit of 1000 has been imposed to limit the resources used by this
2143 optimization. This infelicity was discovered by the LLVM fuzzer.
2144
2145 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
2146 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
2147 because \S ensures they are all in the class. The code for doing this was
2148 interacting badly with the code for computing the amount of space needed to
2149 compile the pattern, leading to a buffer overflow. This bug was discovered by
2150 the LLVM fuzzer.
2151
2152 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
2153 other kinds of group caused stack overflow at compile time. This bug was
2154 discovered by the LLVM fuzzer.
2155
2156 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment
2157 between a subroutine call and its quantifier was incorrectly compiled, leading
2158 to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer.
2159
2160 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an
2161 assertion after (?(. The code was failing to check the character after (?(?<
2162 for the ! or = that would indicate a lookbehind assertion. This bug was
2163 discovered by the LLVM fuzzer.
2164
2165 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with
2166 a fixed maximum following a group that contains a subroutine reference was
2167 incorrectly compiled and could trigger buffer overflow. This bug was discovered
2168 by the LLVM fuzzer.
2169
2170 14. Negative relative recursive references such as (?-7) to non-existent
2171 subpatterns were not being diagnosed and could lead to unpredictable behaviour.
2172 This bug was discovered by the LLVM fuzzer.
2173
2174 15. The bug fixed in 14 was due to an integer variable that was unsigned when
2175 it should have been signed. Some other "int" variables, having been checked,
2176 have either been changed to uint32_t or commented as "must be signed".
2177
2178 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1)))
2179 caused a stack overflow instead of the diagnosis of a non-fixed length
2180 lookbehind assertion. This bug was discovered by the LLVM fuzzer.
2181
2182 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern
2183 (e.g. /(?<=\Ka)/) could make pcre2grep loop.
2184
2185 18. There was a similar problem to 17 in pcre2test for global matches, though
2186 the code there did catch the loop.
2187
2188 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*),
2189 and a subsequent item in the pattern caused a non-match, backtracking over the
2190 repeated \X did not stop, but carried on past the start of the subject, causing
2191 reference to random memory and/or a segfault. There were also some other cases
2192 where backtracking after \C could crash. This set of bugs was discovered by the
2193 LLVM fuzzer.
2194
2195 20. The function for finding the minimum length of a matching string could take
2196 a very long time if mutual recursion was present many times in a pattern, for
2197 example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has
2198 been implemented. This infelicity was discovered by the LLVM fuzzer.
2199
2200 21. Implemented PCRE2_NEVER_BACKSLASH_C.
2201
2202 22. The feature for string replication in pcre2test could read from freed
2203 memory if the replication required a buffer to be extended, and it was not
2204 working properly in 16-bit and 32-bit modes. This issue was discovered by a
2205 fuzzer: see http://lcamtuf.coredump.cx/afl/.
2206
2207 23. Added the PCRE2_ALT_CIRCUMFLEX option.
2208
2209 24. Adjust the treatment of \8 and \9 to be the same as the current Perl
2210 behaviour.
2211
2212 25. Static linking against the PCRE2 library using the pkg-config module was
2213 failing on missing pthread symbols.
2214
2215 26. If a group that contained a recursive back reference also contained a
2216 forward reference subroutine call followed by a non-forward-reference
2217 subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
2218 compile correct code, leading to undefined behaviour or an internally detected
2219 error. This bug was discovered by the LLVM fuzzer.
2220
2221 27. Quantification of certain items (e.g. atomic back references) could cause
2222 incorrect code to be compiled when recursive forward references were involved.
2223 For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was
2224 discovered by the LLVM fuzzer.
2225
2226 28. A repeated conditional group whose condition was a reference by name caused
2227 a buffer overflow if there was more than one group with the given name. This
2228 bug was discovered by the LLVM fuzzer.
2229
2230 29. A recursive back reference by name within a group that had the same name as
2231 another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/.
2232 This bug was discovered by the LLVM fuzzer.
2233
2234 30. A forward reference by name to a group whose number is the same as the
2235 current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a
2236 buffer overflow at compile time. This bug was discovered by the LLVM fuzzer.
2237
2238 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1
2239 as an int; fixed by writing it as 1u).
2240
2241 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives
2242 a warning for "fileno" unless -std=gnu99 us used.
2243
2244 33. A lookbehind assertion within a set of mutually recursive subpatterns could
2245 provoke a buffer overflow. This bug was discovered by the LLVM fuzzer.
2246
2247 34. Give an error for an empty subpattern name such as (?'').
2248
2249 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains
2250 \P, \p, or \X.
2251
2252 36. The way named subpatterns are handled has been refactored. There is now a
2253 pre-pass over the regex which does nothing other than identify named
2254 subpatterns and count the total captures. This means that information about
2255 named patterns is known before the rest of the compile. In particular, it means
2256 that forward references can be checked as they are encountered. Previously, the
2257 code for handling forward references was contorted and led to several errors in
2258 computing the memory requirements for some patterns, leading to buffer
2259 overflows.
2260
2261 37. There was no check for integer overflow in subroutine calls such as (?123).
2262
2263 38. The table entry for \l in EBCDIC environments was incorrect, leading to its
2264 being treated as a literal 'l' instead of causing an error.
2265
2266 39. If a non-capturing group containing a conditional group that could match
2267 an empty string was repeated, it was not identified as matching an empty string
2268 itself. For example: /^(?:(?(1)x|)+)+$()/.
2269
2270 40. In an EBCDIC environment, pcretest was mishandling the escape sequences
2271 \a and \e in test subject lines.
2272
2273 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII
2274 instead of the EBCDIC value.
2275
2276 42. The handling of \c in an EBCDIC environment has been revised so that it is
2277 now compatible with the specification in Perl's perlebcdic page.
2278
2279 43. Single character repetition in JIT has been improved. 20-30% speedup
2280 was achieved on certain patterns.
2281
2282 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in
2283 ASCII/Unicode. This has now been added to the list of characters that are
2284 recognized as white space in EBCDIC.
2285
2286 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave
2287 an error (correctly) when used outside a class, but did not give an error
2288 within a class.
2289
2290 46. \h within a class was incorrectly compiled in EBCDIC environments.
2291
2292 47. JIT should return with error when the compiled pattern requires
2293 more stack space than the maximum.
2294
2295 48. Fixed a memory leak in pcre2grep when a locale is set.
2296
2297
2298 Version 10.10 06-March-2015
2299 ---------------------------
2300
2301 1. When a pattern is compiled, it remembers the highest back reference so that
2302 when matching, if the ovector is too small, extra memory can be obtained to
2303 use instead. A conditional subpattern whose condition is a check on a capture
2304 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
2305 another kind of back reference, but it was not setting the highest
2306 backreference number. This mattered only if pcre2_match() was called with an
2307 ovector that was too small to hold the capture, and there was no other kind of
2308 back reference (a situation which is probably quite rare). The effect of the
2309 bug was that the condition was always treated as FALSE when the capture could
2310 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
2311 has been fixed.
2312
2313 2. Functions for serialization and deserialization of sets of compiled patterns
2314 have been added.
2315
2316 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
2317 excess code units at the end of the data block that may occasionally occur if
2318 the code for calculating the size over-estimates. This change stops the
2319 serialization code copying uninitialized data, to which valgrind objects. The
2320 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
2321 include the general overhead. This has been corrected.
2322
2323 4. All code units in every slot in the table of group names are now set, again
2324 in order to avoid accessing uninitialized data when serializing.
2325
2326 5. The (*NO_JIT) feature is implemented.
2327
2328 6. If a bug that caused pcre2_compile() to use more memory than allocated was
2329 triggered when using valgrind, the code in (3) above passed a stupidly large
2330 value to valgrind. This caused a crash instead of an "internal error" return.
2331
2332 7. A reference to a duplicated named group (either a back reference or a test
2333 for being set in a conditional) that occurred in a part of the pattern where
2334 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
2335 to be incorrectly calculated, leading to overwriting.
2336
2337 8. A mutually recursive set of back references such as (\2)(\1) caused a
2338 segfault at compile time (while trying to find the minimum matching length).
2339 The infinite loop is now broken (with the minimum length unset, that is, zero).
2340
2341 9. If an assertion that was used as a condition was quantified with a minimum
2342 of zero, matching went wrong. In particular, if the whole group had unlimited
2343 repetition and could match an empty string, a segfault was likely. The pattern
2344 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
2345 quantified, but not if they are being used as conditions, so the above pattern
2346 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
2347 patterns.
2348
2349 10. The error message for an invalid quantifier has been changed from "nothing
2350 to repeat" to "quantifier does not follow a repeatable item".
2351
2352 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
2353 scanning the compiled pattern in subsequent auto-possessification can get out
2354 of step and lead to an unknown opcode. Previously this could have caused an
2355 infinite loop. Now it generates an "internal error" error. This is a tidyup,
2356 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
2357 undefined outcome.
2358
2359 12. A UTF pattern containing a "not" match of a non-ASCII character and a
2360 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
2361
2362 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
2363 that is found in the output of "locale -a" can actually be set by pcre2test
2364 before it is accepted. Previously, in an environment where a locale was listed
2365 but would not set (an example does exist), the test would "pass" without
2366 actually doing anything. Also the fr_CA locale has been added to the list of
2367 locales that can be used.
2368
2369 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
2370 capturing group number without parentheses, the last character was incorrectly
2371 literally included at the end of the replacement string.
2372
2373 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
2374 failed to allow the zero-repeat case if pcre2_match() was called with an
2375 ovector too small to capture the group.
2376
2377 16. Improved error message in pcre2test when setting the stack size (-S) fails.
2378
2379 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
2380 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
2381 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
2382 of PCRE2 sources, which caused a failure to build pcre2test.
2383
2384 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
2385 only on Windows.
2386
2387 19. Use binary input when reading back saved serialized patterns in pcre2test.
2388
2389 20. Added RunTest.bat for running the tests under Windows.
2390
2391 21. "make distclean" was not removing config.h, a file that may be created for
2392 use with CMake.
2393
2394 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
2395 forward reference repeated a large (but limited) number of times within a
2396 repeated outer group that has a zero minimum quantifier, caused incorrect code
2397 to be compiled, leading to the error "internal error: previously-checked
2398 referenced subpattern not found" when an incorrect memory address was read.
2399 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
2400 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
2401
2402 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
2403 call within a group that also contained a recursive back reference caused
2404 incorrect code to be compiled. This bug was reported as "heap overflow",
2405 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
2406 CVE-2015-2326 was given to this.)
2407
2408 24. Computing the size of the JIT read-only data in advance has been a source
2409 of various issues, and new ones are still appear unfortunately. To fix
2410 existing and future issues, size computation is eliminated from the code,
2411 and replaced by on-demand memory allocation.
2412
2413 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
2414 adjacent to the end of the range, and the range contained characters with more
2415 than one other case, caused incorrect behaviour when compiled in UTF mode. In
2416 that example, the range a-j was left out of the class.
2417
2418
2419 Version 10.00 05-January-2015
2420 -----------------------------
2421
2422 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
2423 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
2424 API, up to item 20 for release 8.36.
2425
2426 The code of the library was heavily revised as part of the new API
2427 implementation. Details of each and every modification were not individually
2428 logged. In addition to the API changes, the following changes were made. They
2429 are either new functionality, or bug fixes and other noticeable changes of
2430 behaviour that were implemented after the code had been forked.
2431
2432 1. Including Unicode support at build time is now enabled by default, but it
2433 can optionally be disabled. It is not enabled by default at run time (no
2434 change).
2435
2436 2. The test program, now called pcre2test, was re-specified and almost
2437 completely re-written. Its input is not compatible with input for pcretest.
2438
2439 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
2440 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
2441 matched by that pattern.
2442
2443 4. For the benefit of those who use PCRE2 via some other application, that is,
2444 not writing the function calls themselves, it is possible to check the PCRE2
2445 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
2446 string such as "yesno".
2447
2448 5. There are case-equivalent Unicode characters whose encodings use different
2449 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
2450 theoretically possible for this to happen in UTF-16 too.) If a backreference to
2451 a group containing one of these characters was greedily repeated, and during
2452 the match a backtrack occurred, the subject might be backtracked by the wrong
2453 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
2454 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
2455 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
2456 Incorrect backtracking meant that group 2 captured only the last two bytes.
2457 This bug has been fixed; the new code is slower, but it is used only when the
2458 strings matched by the repetition are not all the same length.
2459
2460 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
2461 information. This applied to any pattern with a group that matched no
2462 characters, for example: /(?:(?=.)|(?<!x))a/.
2463
2464 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
2465 those parentheses to be closed with whatever has been captured so far. However,
2466 it was failing to mark any other groups between the highest capture so far and
2467 the currrent group as "unset". Thus, the ovector for those groups contained
2468 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
2469 matched against "abcd".
2470
2471 8. The pcre2_substitute() function has been implemented.
2472
2473 9. If an assertion used as a condition was quantified with a minimum of zero
2474 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
2475 occur.
2476
2477 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.
2478
2479 ****

  ViewVC Help
Powered by ViewVC 1.1.5