/[pcre2]/code/trunk/ChangeLog
ViewVC logotype

Contents of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1317 - (show annotations)
Sun Jul 11 15:15:24 2021 UTC (2 months, 2 weeks ago) by ph10
File size: 118936 byte(s)
Apply Jan-Willem Blokland's CMake patches to build both static and shared 
libraries, plus some other CMake extensions.
1 Change Log for PCRE2
2 --------------------
3
4 Version 10.38-RC1 xx-xxx-2021
5 -----------------------------
6
7 1. Fix invalid single character repetition issues in JIT when the repetition
8 is inside a capturing bracket and the bracket is preceeded by character
9 literals.
10
11 2. Installed revised CMake configuration files provided by Jan-Willem Blokland.
12 This extends the CMake build system to build both static and shared libraries
13 in one go, builds the static library with PIC, and exposes PCRE2 libraries
14 using the CMake config files. JWB provided these notes:
15
16 - Introduced CMake variable BUILD_STATIC_LIBS to build the static library.
17
18 - Make a small modification to config-cmake.h.in by removing the PCRE2_STATIC
19 variable. Added PCRE2_STATIC variable to the static build using the
20 target_compile_definitions() function.
21
22 - Extended the CMake config files.
23
24 - Introduced CMake variable PCRE2_USE_STATIC_LIBS to easily switch between
25 the static and shared libraries.
26
27 - Added the PCRE_STATIC variable to the target compile definitions for the
28 import of the static library.
29
30 Building static and shared libraries using MSVC results in a name clash of
31 the libraries. Both static and shared library builds create, for example, the
32 file pcre2-8.lib. Therefore, I decided to change the static library names by
33 adding "-static". For example, pcre2-8.lib has become pcre2-8-static.lib.
34 [Comment by PH: this seems to be MSVC-specific. It doesn't happen on Linux.]
35
36
37
38 Version 10.37 26-May-2021
39 -------------------------
40
41 1. Change RunGrepTest to use tr instead of sed when testing with binary
42 zero bytes, because sed varies a lot from system to system and has problems
43 with binary zeros. This is from Bugzilla #2681. Patch from Jeremie
44 Courreges-Anglas via Nam Nguyen. This fixes RunGrepTest for OpenBSD. Later:
45 it broke it for at least one version of Solaris, where tr can't handle binary
46 zeros. However, that system had /usr/xpg4/bin/tr installed, which works OK, so
47 RunGrepTest now checks for that command and uses it if found.
48
49 2. Compiling with gcc 10.2's -fanalyzer option showed up a hypothetical problem
50 with a NULL dereference. I don't think this case could ever occur in practice,
51 but I have put in a check in order to get rid of the compiler error.
52
53 3. An alternative patch for CMakeLists.txt because 10.36 #4 breaks CMake on
54 Windows. Patch from email@cs-ware.de fixes bugzilla #2688.
55
56 4. Two bugs related to over-large numbers have been fixed so the behaviour is
57 now the same as Perl.
58
59 (a) A pattern such as /\214748364/ gave an overflow error instead of being
60 treated as the octal number \214 followed by literal digits.
61
62 (b) A sequence such as {65536 that has no terminating } so is not a
63 quantifier was nevertheless complaining that a quantifier number was too big.
64
65 5. A run of autoconf suggested that configure.ac was out-of-date with respect
66 to the lastest autoconf. Running autoupdate made some valid changes, some valid
67 suggestions, and also some invalid changes, which were fixed by hand. Autoconf
68 now runs clean and the resulting "configure" seems to work, so I hope nothing
69 is broken. Later: the requirement for autoconf 2.70 broke some automatic test
70 robots. It doesn't seem to be necessary: trying a reduction to 2.60.
71
72 6. The pattern /a\K.(?0)*/ when matched against "abac" by the interpreter gave
73 the answer "bac", whereas Perl and JIT both yield "c". This was because the
74 effect of \K was not propagating back from the full pattern recursion. Other
75 recursions such as /(a\K.(?1)*)/ did not have this problem.
76
77 7. Restore single character repetition optimization in JIT. Currently fewer
78 character repetitions are optimized than in 10.34.
79
80 8. When the names of the functions in the POSIX wrapper were changed to
81 pcre2_regcomp() etc. (see change 10.33 #4 below), functions with the original
82 names were left in the library so that pre-compiled programs would still work.
83 However, this has proved troublesome when programs link with several libraries,
84 some of which use PCRE2 via the POSIX interface while others use a native POSIX
85 library. For this reason, the POSIX function names are removed in this release.
86 The macros in pcre2posix.h should ensure that re-compiling fixes any programs
87 that haven't been compiled since before 10.33.
88
89
90 Version 10.36 04-December-2020
91 ------------------------------
92
93 1. Add CET_CFLAGS so that when Intel CET is enabled, pass -mshstk to
94 compiler. This fixes https://bugs.exim.org/show_bug.cgi?id=2578. Patch for
95 Makefile.am and configure.ac by H.J. Lu. Equivalent patch for CMakeLists.txt
96 invented by PH.
97
98 2. Fix inifinite loop when a single byte newline is searched in JIT when
99 invalid utf8 mode is enabled.
100
101 3. Updated CMakeLists.txt with patch from Wolfgang Stöggl (Bugzilla #2584):
102
103 - Include GNUInstallDirs and use ${CMAKE_INSTALL_LIBDIR} instead of hardcoded
104 lib. This allows differentiation between lib and lib64.
105 CMAKE_INSTALL_LIBDIR is used for installation of libraries and also for
106 pkgconfig file generation.
107
108 - Add the version of PCRE2 to the configuration summary like ./configure
109 does.
110
111 - Fix typo: MACTHED_STRING->MATCHED_STRING
112
113 4. Updated CMakeLists.txt with another patch from Wolfgang Stöggl (Bugzilla
114 #2588):
115
116 - Add escaped double quotes around include directory in CMakeLists.txt to
117 allow spaces in directory names.
118
119 - This fixes a cmake error, if the path of the pcre2 source contains a space.
120
121 5. Updated CMakeLists.txt with a patch from B. Scott Michel: CMake's
122 documentation suggests using CHECK_SYMBOL_EXISTS over CHECK_FUNCTION_EXIST.
123 Moreover, these functions come from specific header files, which need to be
124 specified (and, thankfully, are the same on both the Linux and WinXX
125 platforms.)
126
127 6. Added a (uint32_t) cast to prevent a compiler warning in pcre2_compile.c.
128
129 7. Applied a patch from Wolfgang Stöggl (Bugzilla #2600) to fix postfix for
130 debug Windows builds using CMake. This also updated configure so that it
131 generates *.pc files and pcre2-config with the same content, as in the past.
132
133 8. If a pattern ended with (?(VERSION=n.d where n is any number but d is just a
134 single digit, the code unit beyond d was being read (i.e. there was a read
135 buffer overflow). Fixes ClusterFuzz 23779.
136
137 9. After the rework in r1235, certain character ranges were incorrectly
138 handled by an optimization in JIT. Furthermore a wrong offset was used to
139 read a value from a buffer which could lead to memory overread.
140
141 10. Unnoticed for many years was the fact that delimiters other than / in the
142 testinput1 and testinput4 files could cause incorrect behaviour when these
143 files were processed by perltest.sh. There were several tests that used quotes
144 as delimiters, and it was just luck that they didn't go wrong with perltest.sh.
145 All the patterns in testinput1 and testinput4 now use / as their delimiter.
146 This fixes Bugzilla #2641.
147
148 11. Perl has started to give an error for \K within lookarounds (though there
149 are cases where it doesn't). PCRE2 still allows this, so the tests that include
150 this case have been moved from test 1 to test 2.
151
152 12. Further to 10 above, pcre2test has been updated to detect and grumble if a
153 delimiter other than / is used after #perltest.
154
155 13. Fixed a bug with PCRE2_MATCH_INVALID_UTF in 8-bit mode when PCRE2_CASELESS
156 was set and PCRE2_NO_START_OPTIMIZE was not set. The optimization for finding
157 the start of a match was not resetting correctly after a failed match on the
158 first valid fragment of the subject, possibly causing incorrect "no match"
159 returns on subsequent fragments. For example, the pattern /A/ failed to match
160 the subject \xe5A. Fixes Bugzilla #2642.
161
162 14. Fixed a bug in character set matching when JIT is enabled and both unicode
163 scripts and unicode classes are present at the same time.
164
165 15. Added GNU grep's -m (aka --max-count) option to pcre2grep.
166
167 16. Refactored substitution processing in pcre2grep strings, both for the -O
168 option and when dealing with callouts. There is now a single function that
169 handles $ expansion in all cases (instead of multiple copies of almost
170 identical code). This means that the same escape sequences are available
171 everywhere, which was not previously the case. At the same time, the escape
172 sequences $x{...} and $o{...} have been introduced, to allow for characters
173 whose code points are greater than 255 in Unicode mode.
174
175 17. Applied the patch from Bugzilla #2628 to RunGrepTest. This does an explicit
176 test for a version of sed that can handle binary zero, instead of assuming that
177 any Linux version will work. Later: replaced $(...) by `...` because not all
178 shells recognize the former.
179
180 18. Fixed a word boundary check bug in JIT when partial matching is enabled.
181
182 19. Fix ARM64 compilation warning in JIT. Patch by Carlo.
183
184 20. A bug in the RunTest script meant that if the first part of test 2 failed,
185 the failure was not reported.
186
187 21. Test 2 was failing when run from a directory other than the source
188 directory. This failure was previously missed in RunTest because of 20 above.
189 Fixes added to both RunTest and RunTest.bat.
190
191 22. Patch to CMakeLists.txt from Daniel to fix problem with testing under
192 Windows.
193
194
195 Version 10.35 09-May-2020
196 ---------------------------
197
198 1. Use PCRE2_MATCH_EMPTY flag to detect empty matches in JIT.
199
200 2. Fix ARMv5 JIT improper handling of labels right after a constant pool.
201
202 3. A JIT bug is fixed which allowed to read the fields of the compiled
203 pattern before its existence is checked.
204
205 4. Back in the PCRE1 day, capturing groups that contained recursive back
206 references to themselves were made atomic (version 8.01, change 18) because
207 after the end a repeated group, the captured substrings had their values from
208 the final repetition, not from an earlier repetition that might be the
209 destination of a backtrack. This feature was documented, and was carried over
210 into PCRE2. However, it has now been realized that the major refactoring that
211 was done for 10.30 has made this atomicizing unnecessary, and it is confusing
212 when users are unaware of it, making some patterns appear not to be working as
213 expected. Capture values of recursive back references in repeated groups are
214 now correctly backtracked, so this unnecessary restriction has been removed.
215
216 5. Added PCRE2_SUBSTITUTE_LITERAL.
217
218 6. Avoid some VS compiler warnings.
219
220 7. Added PCRE2_SUBSTITUTE_MATCHED.
221
222 8. Added (?* and (?<* as synonms for (*napla: and (*naplb: to match another
223 regex engine. The Perl regex folks are aware of this usage and have made a note
224 about it.
225
226 9. When an assertion is repeated, PCRE2 used to limit the maximum repetition to
227 1, believing that repeating an assertion is pointless. However, if a positive
228 assertion contains capturing groups, repetition can be useful. In any case, an
229 assertion could always be wrapped in a repeated group. The only restriction
230 that is now imposed is that an unlimited maximum is changed to one more than
231 the minimum.
232
233 10. Fix *THEN verbs in lookahead assertions in JIT.
234
235 11. Added PCRE2_SUBSTITUTE_REPLACEMENT_ONLY.
236
237 12. The JIT stack should be freed when the low-level stack allocation fails.
238
239 13. In pcre2grep, if the final line in a scanned file is output but does not
240 end with a newline sequence, add a newline according to the --newline setting.
241
242 14. (?(DEFINE)...) groups were not being handled correctly when checking for
243 the fixed length of a lookbehind assertion. Such a group within a lookbehind
244 should be skipped, as it does not contribute to the length of the group.
245 Instead, the (DEFINE) group was being processed, and if at the end of the
246 lookbehind, that end was not correctly recognized. Errors such as "lookbehind
247 assertion is not fixed length" and also "internal error: bad code value in
248 parsed_skip()" could result.
249
250 15. Put a limit of 1000 on recursive calls in pcre2_study() when searching
251 nested groups for starting code units, in order to avoid stack overflow issues.
252 If the limit is reached, it just gives up trying for this optimization.
253
254 16. The control verb chain list must always be restored when exiting from a
255 recurse function in JIT.
256
257 17. Fix a crash which occurs when the character type of an invalid UTF
258 character is decoded in JIT.
259
260 18. Changes in many areas of the code so that when Unicode is supported and
261 PCRE2_UCP is set without PCRE2_UTF, Unicode character properties are used for
262 upper/lower case computations on characters whose code points are greater than
263 127.
264
265 19. The function for checking UTF-16 validity was returning an incorrect offset
266 for the start of the error when a high surrogate was not followed by a valid
267 low surrogate. This caused incorrect behaviour, for example when
268 PCRE2_MATCH_INVALID_UTF was set and a match started immediately following the
269 invalid high surrogate, such as /aa/ matching "\x{d800}aa".
270
271 20. If a DEFINE group immediately preceded a lookbehind assertion, the pattern
272 could be mis-compiled and therefore not match correctly. This is the example
273 that found this: /(?(DEFINE)(?<foo>bar))(?<![-a-z0-9])word/ which failed to
274 match "word" because the "move back" value was set to zero.
275
276 21. Following a request from a user, some extensions and tidies to the
277 character tables handling have been done:
278
279 (a) The dftables auxiliary program is renamed pcre2_dftables, but it is still
280 not installed for public use.
281
282 (b) There is now a -b option for pcre2_dftables, which causes the tables to
283 be written in binary. There is also a -help option.
284
285 (c) PCRE2_CONFIG_TABLES_LENGTH is added to pcre2_config() so that an
286 application that wants to save tables in binary knows how long they are.
287
288 22. Changed setting of CMAKE_MODULE_PATH in CMakeLists.txt from SET to
289 LIST(APPEND...) to allow a setting from the command line to be included.
290
291 23. Updated to Unicode 13.0.0.
292
293 24. CMake build now checks for secure_getenv() and strerror(). Patch by Carlo.
294
295 25. Avoid using [-1] as a suffix in pcre2test because it can provoke a compiler
296 warning.
297
298 26. Added tests for __attribute__((uninitialized)) to both the configure and
299 CMake build files, and then applied this attribute to the variable called
300 stack_frames_vector[] in pcre2_match(). When implemented, this disables
301 automatic initialization (a facility in clang), which can take time on big
302 variables.
303
304 27. Updated CMakeLists.txt (patches by Uwe Korn) to add support for
305 pcre2-config, the libpcre*.pc files, SOVERSION, VERSION and the
306 MACHO_*_VERSIONS settings for CMake builds.
307
308 28. Another patch to CMakeLists.txt to check for mkostemp (configure already
309 does). Patch by Carlo Marcelo Arenas Belon.
310
311 29. Check for the existence of memfd_create in both CMake and configure
312 configurations. Patch by Carlo Marcelo Arenas Belon.
313
314 30. Restrict the configuration setting for the SELinux compatible execmem
315 allocator (change 10.30/44) to Linux and NetBSD.
316
317
318 Version 10.34 21-November-2019
319 ------------------------------
320
321 1. The maximum number of capturing subpatterns is 65535 (documented), but no
322 check on this was ever implemented. This omission has been rectified; it fixes
323 ClusterFuzz 14376.
324
325 2. Improved the invalid utf32 support of the JIT compiler. Now it correctly
326 detects invalid characters in the 0xd800-0xdfff range.
327
328 3. Fix minor typo bug in JIT compile when \X is used in a non-UTF string.
329
330 4. Add support for matching in invalid UTF strings to the pcre2_match()
331 interpreter, and integrate with the existing JIT support via the new
332 PCRE2_MATCH_INVALID_UTF compile-time option.
333
334 5. Give more error detail for invalid UTF-8 when detected in pcre2grep.
335
336 6. Add support for invalid UTF-8 to pcre2grep.
337
338 7. Adjust the limit for "must have" code unit searching, in particular,
339 increase it substantially for non-anchored patterns.
340
341 8. Allow (*ACCEPT) to be quantified, because an ungreedy quantifier with a zero
342 minimum is potentially useful.
343
344 9. Some changes to the way the minimum subject length is handled:
345
346 * When PCRE2_NO_START_OPTIMIZE is set, no minimum length is computed;
347 pcre2test now omits this item instead of showing a value of zero.
348
349 * An incorrect minimum length could be calculated for a pattern that
350 contained (*ACCEPT) inside a qualified group whose minimum repetition was
351 zero, for example /A(?:(*ACCEPT))?B/, which incorrectly computed a minimum
352 of 2. The minimum length scan no longer happens for a pattern that
353 contains (*ACCEPT).
354
355 * When no minimum length is set by the normal scan, but a first and/or last
356 code unit is recorded, set the minimum to 1 or 2 as appropriate.
357
358 * When a pattern contains multiple groups with the same number, a back
359 reference cannot know which one to scan for a minimum length. This used to
360 cause the minimum length finder to give up with no result. Now it treats
361 such references as not adding to the minimum length (which it should have
362 done all along).
363
364 * Furthermore, the above action now happens only if the back reference is to
365 a group that exists more than once in a pattern instead of any back
366 reference in a pattern with duplicate numbers.
367
368 10. A (*MARK) value inside a successful condition was not being returned by the
369 interpretive matcher (it was returned by JIT). This bug has been mended.
370
371 11. A bug in pcre2grep meant that -o without an argument (or -o0) didn't work
372 if the pattern had more than 32 capturing parentheses. This is fixed. In
373 addition (a) the default limit for groups requested by -o<n> has been raised to
374 50, (b) the new --om-capture option changes the limit, (c) an error is raised
375 if -o asks for a group that is above the limit.
376
377 12. The quantifier {1} was always being ignored, but this is incorrect when it
378 is made possessive and applied to an item in parentheses, because a
379 parenthesized item may contain multiple branches or other backtracking points,
380 for example /(a|ab){1}+c/ or /(a+){1}+a/.
381
382 13. For partial matches, pcre2test was always showing the maximum lookbehind
383 characters, flagged with "<", which is misleading when the lookbehind didn't
384 actually look behind the start (because it was later in the pattern). Showing
385 all consulted preceding characters for partial matches is now controlled by the
386 existing "allusedtext" modifier and, as for complete matches, this facility is
387 available only for non-JIT matching, because JIT does not maintain the first
388 and last consulted characters.
389
390 14. DFA matching (using pcre2_dfa_match()) was not recognising a partial match
391 if the end of the subject was encountered in a lookahead (conditional or
392 otherwise), an atomic group, or a recursion.
393
394 15. Give error if pcre2test -t, -T, -tm or -TM is given an argument of zero.
395
396 16. Check for integer overflow when computing lookbehind lengths. Fixes
397 Clusterfuzz issue 15636.
398
399 17. Implemented non-atomic positive lookaround assertions.
400
401 18. If a lookbehind contained a lookahead that contained another lookbehind
402 within it, the nested lookbehind was not correctly processed. For example, if
403 /(?<=(?=(?<=a)))b/ was matched to "ab" it gave no match instead of matching
404 "b".
405
406 19. Implemented pcre2_get_match_data_size().
407
408 20. Two alterations to partial matching:
409
410 (a) The definition of a partial match is slightly changed: if a pattern
411 contains any lookbehinds, an empty partial match may be given, because this
412 is another situation where adding characters to the current subject can
413 lead to a full match. Example: /c*+(?<=[bc])/ with subject "ab".
414
415 (b) Similarly, if a pattern could match an empty string, an empty partial
416 match may be given. Example: /(?![ab]).*/ with subject "ab". This case
417 applies only to PCRE2_PARTIAL_HARD.
418
419 (c) An empty string partial hard match can be returned for \z and \Z as it
420 is documented that they shouldn't match.
421
422 21. A branch that started with (*ACCEPT) was not being recognized as one that
423 could match an empty string.
424
425 22. Corrected pcre2_set_character_tables() tables data type: was const unsigned
426 char * instead of const uint8_t *, as generated by pcre2_maketables().
427
428 23. Upgraded to Unicode 12.1.0.
429
430 24. Add -jitfast command line option to pcre2test (to make all the jit options
431 available directly).
432
433 25. Make pcre2test -C show if libreadline or libedit is supported.
434
435 26. If the length of one branch of a group exceeded 65535 (the maximum value
436 that is remembered as a minimum length), the whole group's length was
437 incorrectly recorded as 65535, leading to incorrect "no match" when start-up
438 optimizations were in force.
439
440 27. The "rightmost consulted character" value was not always correct; in
441 particular, if a pattern ended with a negative lookahead, characters that were
442 inspected in that lookahead were not included.
443
444 28. Add the pcre2_maketables_free() function.
445
446 29. The start-up optimization that looks for a unique initial matching
447 code unit in the interpretive engines uses memchr() in 8-bit mode. When the
448 search is caseless, it was doing so inefficiently, which ended up slowing down
449 the match drastically when the subject was very long. The revised code (a)
450 remembers if one case is not found, so it never repeats the search for that
451 case after a bumpalong and (b) when one case has been found, it searches only
452 up to that position for an earlier occurrence of the other case. This fix
453 applies to both interpretive pcre2_match() and to pcre2_dfa_match().
454
455 30. While scanning to find the minimum length of a group, if any branch has
456 minimum length zero, there is no need to scan any subsequent branches (a small
457 compile-time performance improvement).
458
459 31. Installed a .gitignore file on a user's suggestion. When using the svn
460 repository with git (through git svn) this helps keep it tidy.
461
462 32. Add underflow check in JIT which may occur when the value of subject
463 string pointer is close to 0.
464
465 33. Arrange for classes such as [Aa] which contain just the two cases of the
466 same character, to be treated as a single caseless character. This causes the
467 first and required code unit optimizations to kick in where relevant.
468
469 34. Improve the bitmap of starting bytes for positive classes that include wide
470 characters, but no property types, in UTF-8 mode. Previously, on encountering
471 such a class, the bits for all bytes greater than \xc4 were set, thus
472 specifying any character with codepoint >= 0x100. Now the only bits that are
473 set are for the relevant bytes that start the wide characters. This can give a
474 noticeable performance improvement.
475
476 35. If the bitmap of starting code units contains only 1 or 2 bits, replace it
477 with a single starting code unit (1 bit) or a caseless single starting code
478 unit if the two relevant characters are case-partners. This is particularly
479 relevant to the 8-bit library, though it applies to all. It can give a
480 performance boost for patterns such as [Ww]ord and (word|WORD). However, this
481 optimization doesn't happen if there is a "required" code unit of the same
482 value (because the search for a "required" code unit starts at the match start
483 for non-unique first code unit patterns, but after a unique first code unit,
484 and patterns such as a*a need the former action).
485
486 36. Small patch to pcre2posix.c to set the erroroffset field to -1 immediately
487 after a successful compile, instead of at the start of matching to avoid a
488 sanitizer complaint (regexec is supposed to be thread safe).
489
490 37. Add NEON vectorization to JIT to speed up matching of first character and
491 pairs of characters on ARM64 CPUs.
492
493 38. If a non-ASCII character was the first in a starting assertion in a
494 caseless match, the "first code unit" optimization did not get the casing
495 right, and the assertion failed to match a character in the other case if it
496 did not start with the same code unit.
497
498 39. Fixed the incorrect computation of jump sizes on x86 CPUs in JIT. A masking
499 operation was incorrectly removed in r1136. Reported by Ralf Junker.
500
501
502 Version 10.33 16-April-2019
503 ---------------------------
504
505 1. Added "allvector" to pcre2test to make it easy to check the part of the
506 ovector that shouldn't be changed, in particular after substitute and failed or
507 partial matches.
508
509 2. Fix subject buffer overread in JIT when UTF is disabled and \X or \R has
510 a greater than 1 fixed quantifier. This issue was found by Yunho Kim.
511
512 3. Added support for callouts from pcre2_substitute(). After 10.33-RC1, but
513 prior to release, fixed a bug that caused a crash if pcre2_substitute() was
514 called with a NULL match context.
515
516 4. The POSIX functions are now all called pcre2_regcomp() etc., with wrapper
517 functions that use the standard POSIX names. However, in pcre2posix.h the POSIX
518 names are defined as macros. This should help avoid linking with the wrong
519 library in some environments while still exporting the POSIX names for
520 pre-existing programs that use them. (The Debian alternative names are also
521 defined as macros, but not documented.)
522
523 5. Fix an xclass matching issue in JIT.
524
525 6. Implement PCRE2_EXTRA_ESCAPED_CR_IS_LF (see Bugzilla 2315).
526
527 7. Implement the Perl 5.28 experimental alphabetic names for atomic groups and
528 lookaround assertions, for example, (*pla:...) and (*atomic:...). These are
529 characterized by a lower case letter following (* and to simplify coding for
530 this, the character tables created by pcre2_maketables() were updated to add a
531 new "is lower case letter" bit. At the same time, the now unused "is
532 hexadecimal digit" bit was removed. The default tables in
533 src/pcre2_chartables.c.dist are updated.
534
535 8. Implement the new Perl "script run" features (*script_run:...) and
536 (*atomic_script_run:...) aka (*sr:...) and (*asr:...).
537
538 9. Fixed two typos in change 22 for 10.21, which added special handling for
539 ranges such as a-z in EBCDIC environments. The original code probably never
540 worked, though there were no bug reports.
541
542 10. Implement PCRE2_COPY_MATCHED_SUBJECT for pcre2_match() (including JIT via
543 pcre2_match()) and pcre2_dfa_match(), but *not* the pcre2_jit_match() fast
544 path. Also, when a match fails, set the subject field in the match data to NULL
545 for tidiness - none of the substring extractors should reference this after
546 match failure.
547
548 11. If a pattern started with a subroutine call that had a quantifier with a
549 minimum of zero, an incorrect "match must start with this character" could be
550 recorded. Example: /(?&xxx)*ABC(?<xxx>XYZ)/ would (incorrectly) expect 'A' to
551 be the first character of a match.
552
553 12. The heap limit checking code in pcre2_dfa_match() could suffer from
554 overflow if the heap limit was set very large. This could cause incorrect "heap
555 limit exceeded" errors.
556
557 13. Add "kibibytes" to the heap limit output from pcre2test -C to make the
558 units clear.
559
560 14. Add a call to pcre2_jit_free_unused_memory() in pcre2grep, for tidiness.
561
562 15. Updated the VMS-specific code in pcre2test on the advice of a VMS user.
563
564 16. Removed the unnecessary inclusion of stdint.h (or inttypes.h) from
565 pcre2_internal.h as it is now included by pcre2.h. Also, change 17 for 10.32
566 below was unnecessarily complicated, as inttypes.h is a Standard C header,
567 which is defined to be a superset of stdint.h. Instead of conditionally
568 including stdint.h or inttypes.h, pcre2.h now unconditionally includes
569 inttypes.h. This supports environments that do not have stdint.h but do have
570 inttypes.h, which are known to exist. A note in the autotools documentation
571 says (November 2018) that there are none known that are the other way round.
572
573 17. Added --disable-percent-zt to "configure" (and equivalent to CMake) to
574 forcibly disable the use of %zu and %td in formatting strings because there is
575 at least one version of VMS that claims to be C99 but does not support these
576 modifiers.
577
578 18. Added --disable-pcre2grep-callout-fork, which restricts the callout support
579 in pcre2grep to the inbuilt echo facility. This may be useful in environments
580 that do not support fork().
581
582 19. Fix two instances of <= 0 being applied to unsigned integers (the VMS
583 compiler complains).
584
585 20. Added "fork" support for VMS to pcre2grep, for running an external program
586 via a string callout.
587
588 21. Improve MAP_JIT flag usage on MacOS. Patch by Rich Siegel.
589
590 22. If a pattern started with (*MARK), (*COMMIT), (*PRUNE), (*SKIP), or (*THEN)
591 followed by ^ it was not recognized as anchored.
592
593 23. The RunGrepTest script used to cut out the test of NUL characters for
594 Solaris and MacOS as printf and sed can't handle them. It seems that the *BSD
595 systems can't either. I've inverted the test so that only those OS that are
596 known to work (currently only Linux) try to run this test.
597
598 24. Some tests in RunGrepTest appended to testtrygrep from two different file
599 descriptors instead of redirecting stderr to stdout. This worked on Linux, but
600 it was reported not to on other systems, causing the tests to fail.
601
602 25. In the RunTest script, make the test for stack setting use the same value
603 for the stack as it needs for -bigstack.
604
605 26. Insert a cast in pcre2_dfa_match.c to suppress a compiler warning.
606
607 26. With PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL set, escape sequences such as \s
608 which are valid in character classes, but not as the end of ranges, were being
609 treated as literals. An example is [_-\s] (but not [\s-_] because that gave an
610 error at the *start* of a range). Now an "invalid range" error is given
611 independently of PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
612
613 27. Related to 26 above, PCRE2_BAD_ESCAPE_IS_LITERAL was affecting known escape
614 sequences such as \eX when they appeared invalidly in a character class. Now
615 the option applies only to unrecognized or malformed escape sequences.
616
617 28. Fix word boundary in JIT compiler. Patch by Mike Munday.
618
619 29. The pcre2_dfa_match() function was incorrectly handling conditional version
620 tests such as (?(VERSION>=0)...) when the version test was true. Incorrect
621 processing or a crash could result.
622
623 30. When PCRE2_UTF is set, allow non-ASCII letters and decimal digits in group
624 names, as Perl does. There was a small bug in this new code, found by
625 ClusterFuzz 12950, fixed before release.
626
627 31. Implemented PCRE2_EXTRA_ALT_BSUX to support ECMAScript 6's \u{hhh}
628 construct.
629
630 32. Compile \p{Any} to be the same as . in DOTALL mode, so that it benefits
631 from auto-anchoring if \p{Any}* starts a pattern.
632
633 33. Compile invalid UTF check in JIT test when only pcre32 is enabled.
634
635 34. For some time now, CMake has been warning about the setting of policy
636 CMP0026 to "OLD" in CmakeLists.txt, and hinting that the feature might be
637 removed in a future version. A request for CMake expertise on the list produced
638 no result, so I have now hacked CMakeLists.txt along the lines of some changes
639 I found on the Internet. The new code no longer needs the policy setting, and
640 it appears to work fine on Linux.
641
642 35. Setting --enable-jit=auto for an out-of-tree build failed because the
643 source directory wasn't in the search path for AC_TRY_COMPILE always. Patch
644 from Ross Burton.
645
646 36. Disable SSE2 JIT optimizations in x86 CPUs when SSE2 is not available.
647 Patch by Guillem Jover.
648
649 37. Changed expressions such as 1<<10 to 1u<<10 in many places because compiler
650 warnings were reported.
651
652 38. Using the clang compiler with sanitizing options causes runtime complaints
653 about truncation for statments such as x = ~x when x is an 8-bit value; it
654 seems to compute ~x as a 32-bit value. Changing such statements to x = 255 ^ x
655 gets rid of the warnings. There were also two missing casts in pcre2test.
656
657
658 Version 10.32 10-September-2018
659 -------------------------------
660
661 1. When matching using the the REG_STARTEND feature of the POSIX API with a
662 non-zero starting offset, unset capturing groups with lower numbers than a
663 group that did capture something were not being correctly returned as "unset"
664 (that is, with offset values of -1).
665
666 2. When matching using the POSIX API, pcre2test used to omit listing unset
667 groups altogether. Now it shows those that come before any actual captures as
668 "<unset>", as happens for non-POSIX matching.
669
670 3. Running "pcre2test -C" always stated "\R matches CR, LF, or CRLF only",
671 whatever the build configuration was. It now correctly says "\R matches all
672 Unicode newlines" in the default case when --enable-bsr-anycrlf has not been
673 specified. Similarly, running "pcre2test -C bsr" never produced the result
674 ANY.
675
676 4. Matching the pattern /(*UTF)\C[^\v]+\x80/ against an 8-bit string containing
677 multi-code-unit characters caused bad behaviour and possibly a crash. This
678 issue was fixed for other kinds of repeat in release 10.20 by change 19, but
679 repeating character classes were overlooked.
680
681 5. pcre2grep now supports the inclusion of binary zeros in patterns that are
682 read from files via the -f option.
683
684 6. A small fix to pcre2grep to avoid compiler warnings for -Wformat-overflow=2.
685
686 7. Added --enable-jit=auto support to configure.ac.
687
688 8. Added some dummy variables to the heapframe structure in 16-bit and 32-bit
689 modes for the benefit of m68k, where pointers can be 16-bit aligned. The
690 dummies force 32-bit alignment and this ensures that the structure is a
691 multiple of PCRE2_SIZE, a requirement that is tested at compile time. In other
692 architectures, alignment requirements take care of this automatically.
693
694 9. When returning an error from pcre2_pattern_convert(), ensure the error
695 offset is set zero for early errors.
696
697 10. A number of patches for Windows support from Daniel Richard G:
698
699 (a) List of error numbers in Runtest.bat corrected (it was not the same as in
700 Runtest).
701
702 (b) pcre2grep snprintf() workaround as used elsewhere in the tree.
703
704 (c) Support for non-C99 snprintf() that returns -1 in the overflow case.
705
706 11. Minor tidy of pcre2_dfa_match() code.
707
708 12. Refactored pcre2_dfa_match() so that the internal recursive calls no longer
709 use the stack for local workspace and local ovectors. Instead, an initial block
710 of stack is reserved, but if this is insufficient, heap memory is used. The
711 heap limit parameter now applies to pcre2_dfa_match().
712
713 13. If a "find limits" test of DFA matching in pcre2test resulted in too many
714 matches for the ovector, no matches were displayed.
715
716 14. Removed an occurrence of ctrl/Z from test 6 because Windows treats it as
717 EOF. The test looks to have come from a fuzzer.
718
719 15. If PCRE2 was built with a default match limit a lot greater than the
720 default default of 10 000 000, some JIT tests of the match limit no longer
721 failed. All such tests now set 10 000 000 as the upper limit.
722
723 16. Another Windows related patch for pcregrep to ensure that WIN32 is
724 undefined under Cygwin.
725
726 17. Test for the presence of stdint.h and inttypes.h in configure and CMake and
727 include whichever exists (stdint preferred) instead of unconditionally
728 including stdint. This makes life easier for old and non-standard systems.
729
730 18. Further changes to improve portability, especially to old and or non-
731 standard systems:
732
733 (a) Put all printf arguments in RunGrepTest into single, not double, quotes,
734 and use \0 not \x00 for binary zero.
735
736 (b) Avoid the use of C++ (i.e. BCPL) // comments.
737
738 (c) Parameterize the use of %zu in pcre2test to make it like %td. For both of
739 these now, if using MSVC or a standard C before C99, %lu is used with a
740 cast if necessary.
741
742 19. Applied a contributed patch to CMakeLists.txt to increase the stack size
743 when linking pcre2test with MSVC. This gets rid of a stack overflow error in
744 the standard set of tests.
745
746 20. Output a warning in pcre2test when ignoring the "altglobal" modifier when
747 it is given with the "replace" modifier.
748
749 21. In both pcre2test and pcre2_substitute(), with global matching, a pattern
750 that matched an empty string, but never at the starting match offset, was not
751 handled in a Perl-compatible way. The pattern /(<?=\G.)/ is an example of such
752 a pattern. Because \G is in a lookbehind assertion, there has to be a
753 "bumpalong" before there can be a match. The automatic "advance by one
754 character after an empty string match" rule is therefore inappropriate. A more
755 complicated algorithm has now been implemented.
756
757 22. When checking to see if a lookbehind is of fixed length, lookaheads were
758 correctly ignored, but qualifiers on lookaheads were not being ignored, leading
759 to an incorrect "lookbehind assertion is not fixed length" error.
760
761 23. The VERSION condition test was reading fractional PCRE2 version numbers
762 such as the 04 in 10.04 incorrectly and hence giving wrong results.
763
764 24. Updated to Unicode version 11.0.0. As well as the usual addition of new
765 scripts and characters, this involved re-jigging the grapheme break property
766 algorithm because Unicode has changed the way emojis are handled.
767
768 25. Fixed an obscure bug that struck when there were two atomic groups not
769 separated by something with a backtracking point. There could be an incorrect
770 backtrack into the first of the atomic groups. A complicated example is
771 /(?>a(*:1))(?>b)(*SKIP:1)x|.*/ matched against "abc", where the *SKIP
772 shouldn't find a MARK (because is in an atomic group), but it did.
773
774 26. Upgraded the perltest.sh script: (1) #pattern lines can now be used to set
775 a list of modifiers for all subsequent patterns - only those that the script
776 recognizes are meaningful; (2) #subject lines can be used to set or unset a
777 default "mark" modifier; (3) Unsupported #command lines give a warning when
778 they are ignored; (4) Mark data is output only if the "mark" modifier is
779 present.
780
781 27. (*ACCEPT:ARG), (*FAIL:ARG), and (*COMMIT:ARG) are now supported.
782
783 28. A (*MARK) name was not being passed back for positive assertions that were
784 terminated by (*ACCEPT).
785
786 29. Add support for \N{U+dddd}, but only in Unicode mode.
787
788 30. Add support for (?^) for unsetting all imnsx options.
789
790 31. The PCRE2_EXTENDED (/x) option only ever discarded space characters whose
791 code point was less than 256 and that were recognized by the lookup table
792 generated by pcre2_maketables(), which uses isspace() to identify white space.
793 Now, when Unicode support is compiled, PCRE2_EXTENDED also discards U+0085,
794 U+200E, U+200F, U+2028, and U+2029, which are additional characters defined by
795 Unicode as "Pattern White Space". This makes PCRE2 compatible with Perl.
796
797 32. In certain circumstances, option settings within patterns were not being
798 correctly processed. For example, the pattern /((?i)A)(?m)B/ incorrectly
799 matched "ab". (The (?m) setting lost the fact that (?i) should be reset at the
800 end of its group during the parse process, but without another setting such as
801 (?m) the compile phase got it right.) This bug was introduced by the
802 refactoring in release 10.23.
803
804 33. PCRE2 uses bcopy() if available when memmove() is not, and it used just to
805 define memmove() as function call to bcopy(). This hasn't been tested for a
806 long time because in pcre2test the result of memmove() was being used, whereas
807 bcopy() doesn't return a result. This feature is now refactored always to call
808 an emulation function when there is no memmove(). The emulation makes use of
809 bcopy() when available.
810
811 34. When serializing a pattern, set the memctl, executable_jit, and tables
812 fields (that is, all the fields that contain pointers) to zeros so that the
813 result of serializing is always the same. These fields are re-set when the
814 pattern is deserialized.
815
816 35. In a pattern such as /[^\x{100}-\x{ffff}]*[\x80-\xff]/ which has a repeated
817 negative class with no characters less than 0x100 followed by a positive class
818 with only characters less than 0x100, the first class was incorrectly being
819 auto-possessified, causing incorrect match failures.
820
821 36. Removed the character type bit ctype_meta, which dates from PCRE1 and is
822 not used in PCRE2.
823
824 37. Tidied up unnecessarily complicated macros used in the escapes table.
825
826 38. Since 10.21, the new testoutput8-16-4 file has accidentally been omitted
827 from distribution tarballs, owing to a typo in Makefile.am which had
828 testoutput8-16-3 twice. Now fixed.
829
830 39. If the only branch in a conditional subpattern was anchored, the whole
831 subpattern was treated as anchored, when it should not have been, since the
832 assumed empty second branch cannot be anchored. Demonstrated by test patterns
833 such as /(?(1)^())b/ or /(?(?=^))b/.
834
835 40. A repeated conditional subpattern that could match an empty string was
836 always assumed to be unanchored. Now it it checked just like any other
837 repeated conditional subpattern, and can be found to be anchored if the minimum
838 quantifier is one or more. I can't see much use for a repeated anchored
839 pattern, but the behaviour is now consistent.
840
841 41. Minor addition to pcre2_jit_compile.c to avoid static analyzer complaint
842 (for an event that could never occur but you had to have external information
843 to know that).
844
845 42. If before the first match in a file that was being searched by pcre2grep
846 there was a line that was sufficiently long to cause the input buffer to be
847 expanded, the variable holding the location of the end of the previous match
848 was being adjusted incorrectly, and could cause an overflow warning from a code
849 sanitizer. However, as the value is used only to print pending "after" lines
850 when the next match is reached (and there are no such lines in this case) this
851 bug could do no damage.
852
853
854 Version 10.31 12-February-2018
855 ------------------------------
856
857 1. Fix typo (missing ]) in VMS code in pcre2test.c.
858
859 2. Replace the replicated code for matching extended Unicode grapheme sequences
860 (which got a lot more complicated by change 10.30/49) by a single subroutine
861 that is called by both pcre2_match() and pcre2_dfa_match().
862
863 3. Add idempotent guard to pcre2_internal.h.
864
865 4. Add new pcre2_config() options: PCRE2_CONFIG_NEVER_BACKSLASH_C and
866 PCRE2_CONFIG_COMPILED_WIDTHS.
867
868 5. Cut out \C tests in the JIT regression tests when NEVER_BACKSLASH_C is
869 defined (e.g. by --enable-never-backslash-C).
870
871 6. Defined public names for all the pcre2_compile() error numbers, and used
872 the public names in pcre2_convert.c.
873
874 7. Fixed a small memory leak in pcre2test (convert contexts).
875
876 8. Added two casts to compile.c and one to match.c to avoid compiler warnings.
877
878 9. Added code to pcre2grep when compiled under VMS to set the symbol
879 PCRE2GREP_RC to the exit status, because VMS does not distinguish between
880 exit(0) and exit(1).
881
882 10. Added the -LM (list modifiers) option to pcre2test. Also made -C complain
883 about a bad option only if the following argument item does not start with a
884 hyphen.
885
886 11. pcre2grep was truncating components of file names to 128 characters when
887 processing files with the -r option, and also (some very odd code) truncating
888 path names to 512 characters. There is now a check on the absolute length of
889 full path file names, which may be up to 2047 characters long.
890
891 12. When an assertion contained (*ACCEPT) it caused all open capturing groups
892 to be closed (as for a non-assertion ACCEPT), which was wrong and could lead to
893 misbehaviour for subsequent references to groups that started outside the
894 assertion. ACCEPT in an assertion now closes only those groups that were
895 started within that assertion. Fixes oss-fuzz issues 3852 and 3891.
896
897 13. Multiline matching in pcre2grep was misbehaving if the pattern matched
898 within a line, and then matched again at the end of the line and over into
899 subsequent lines. Behaviour was different with and without colouring, and
900 sometimes context lines were incorrectly printed and/or line endings were lost.
901 All these issues should now be fixed.
902
903 14. If --line-buffered was specified for pcre2grep when input was from a
904 compressed file (.gz or .bz2) a segfault occurred. (Line buffering should be
905 ignored for compressed files.)
906
907 15. Although pcre2_jit_match checks whether the pattern is compiled
908 in a given mode, it was also expected that at least one mode is available.
909 This is fixed and pcre2_jit_match returns with PCRE2_ERROR_JIT_BADOPTION
910 when the pattern is not optimized by JIT at all.
911
912 16. The line number and related variables such as match counts in pcre2grep
913 were all int variables, causing overflow when files with more than 2147483647
914 lines were processed (assuming 32-bit ints). They have all been changed to
915 unsigned long ints.
916
917 17. If a backreference with a minimum repeat count of zero was first in a
918 pattern, apart from assertions, an incorrect first matching character could be
919 recorded. For example, for the pattern /(?=(a))\1?b/, "b" was incorrectly set
920 as the first character of a match.
921
922 18. Characters in a leading positive assertion are considered for recording a
923 first character of a match when the rest of the pattern does not provide one.
924 However, a character in a non-assertive group within a leading assertion such
925 as in the pattern /(?=(a))\1?b/ caused this process to fail. This was an
926 infelicity rather than an outright bug, because it did not affect the result of
927 a match, just its speed. (In fact, in this case, the starting 'a' was
928 subsequently picked up in the study.)
929
930 19. A minor tidy in pcre2_match(): making all PCRE2_ERROR_ returns use "return"
931 instead of "RRETURN" saves unwinding the backtracks in these cases (only one
932 didn't).
933
934 20. Allocate a single callout block on the stack at the start of pcre2_match()
935 and set its never-changing fields once only. Do the same for pcre2_dfa_match().
936
937 21. Save the extra compile options (set in the compile context) with the
938 compiled pattern (they were not previously saved), add PCRE2_INFO_EXTRAOPTIONS
939 to retrieve them, and update pcre2test to show them.
940
941 22. Added PCRE2_CALLOUT_STARTMATCH and PCRE2_CALLOUT_BACKTRACK bits to a new
942 field callout_flags in callout blocks. The bits are set by pcre2_match(), but
943 not by JIT or pcre2_dfa_match(). Their settings are shown in pcre2test callouts
944 if the callout_extra subject modifier is set. These bits are provided to help
945 with tracking how a backtracking match is proceeding.
946
947 23. Updated the pcre2demo.c demonstration program, which was missing the extra
948 code for -g that handles the case when \K in an assertion causes the match to
949 end at the original start point. Also arranged for it to detect when \K causes
950 the end of a match to be before its start.
951
952 24. Similar to 23 above, strange things (including loops) could happen in
953 pcre2grep when \K was used in an assertion when --colour was used or in
954 multiline mode. The "end at original start point" bug is fixed, and if the end
955 point is found to be before the start point, they are swapped.
956
957 25. When PCRE2_FIRSTLINE without PCRE2_NO_START_OPTIMIZE was used in non-JIT
958 matching (both pcre2_match() and pcre2_dfa_match()) and the matched string
959 started with the first code unit of a newline sequence, matching failed because
960 it was not tried at the newline.
961
962 26. Code for giving up a non-partial match after failing to find a starting
963 code unit anywhere in the subject was missing when searching for one of a
964 number of code units (the bitmap case) in both pcre2_match() and
965 pcre2_dfa_match(). This was a missing optimization rather than a bug.
966
967 27. Tidied up the ACROSSCHAR macro to be like FORWARDCHAR and BACKCHAR, using a
968 pointer argument rather than a code unit value. This should not have affected
969 the generated code.
970
971 28. The JIT compiler has been updated.
972
973 29. Avoid pointer overflow for unset captures in pcre2_substring_list_get().
974 This could not actually cause a crash because it was always used in a memcpy()
975 call with zero length.
976
977 30. Some internal structures have a variable-length ovector[] as their last
978 element. Their actual memory is obtained dynamically, giving an ovector of
979 appropriate length. However, they are defined in the structure as
980 ovector[NUMBER], where NUMBER is large so that array bound checkers don't
981 grumble. The value of NUMBER was 10000, but a fuzzer exceeded 5000 capturing
982 groups, making the ovector larger than this. The number has been increased to
983 131072, which allows for the maximum number of captures (65535) plus the
984 overall match. This fixes oss-fuzz issue 5415.
985
986 31. Auto-possessification at the end of a capturing group was dependent on what
987 follows the group (e.g. /(a+)b/ would auto-possessify the a+) but this caused
988 incorrect behaviour when the group was called recursively from elsewhere in the
989 pattern where something different might follow. This bug is an unforseen
990 consequence of change #1 for 10.30 - the implementation of backtracking into
991 recursions. Iterators at the ends of capturing groups are no longer considered
992 for auto-possessification if the pattern contains any recursions. Fixes
993 Bugzilla #2232.
994
995
996 Version 10.30 14-August-2017
997 ----------------------------
998
999 1. The main interpreter, pcre2_match(), has been refactored into a new version
1000 that does not use recursive function calls (and therefore the stack) for
1001 remembering backtracking positions. This makes --disable-stack-for-recursion a
1002 NOOP. The new implementation allows backtracking into recursive group calls in
1003 patterns, making it more compatible with Perl, and also fixes some other
1004 hard-to-do issues such as #1887 in Bugzilla. The code is also cleaner because
1005 the old code had a number of fudges to try to reduce stack usage. It seems to
1006 run no slower than the old code.
1007
1008 A number of bugs in the refactored code were subsequently fixed during testing
1009 before release, but after the code was made available in the repository. These
1010 bugs were never in fully released code, but are noted here for the record.
1011
1012 (a) If a pattern had fewer capturing parentheses than the ovector supplied in
1013 the match data block, a memory error (detectable by ASAN) occurred after
1014 a match, because the external block was being set from non-existent
1015 internal ovector fields. Fixes oss-fuzz issue 781.
1016
1017 (b) A pattern with very many capturing parentheses (when the internal frame
1018 size was greater than the initial frame vector on the stack) caused a
1019 crash. A vector on the heap is now set up at the start of matching if the
1020 vector on the stack is not big enough to handle at least 10 frames.
1021 Fixes oss-fuzz issue 783.
1022
1023 (c) Handling of (*VERB)s in recursions was wrong in some cases.
1024
1025 (d) Captures in negative assertions that were used as conditions were not
1026 happening if the assertion matched via (*ACCEPT).
1027
1028 (e) Mark values were not being passed out of recursions.
1029
1030 (f) Refactor some code in do_callout() to avoid picky compiler warnings about
1031 negative indices. Fixes oss-fuzz issue 1454.
1032
1033 (g) Similarly refactor the way the variable length ovector is addressed for
1034 similar reasons. Fixes oss-fuzz issue 1465.
1035
1036 2. Now that pcre2_match() no longer uses recursive function calls (see above),
1037 the "match limit recursion" value seems misnamed. It still exists, and limits
1038 the depth of tree that is searched. To avoid future confusion, it has been
1039 renamed as "depth limit" in all relevant places (--with-depth-limit,
1040 (*LIMIT_DEPTH), pcre2_set_depth_limit(), etc) but the old names are still
1041 available for backwards compatibility.
1042
1043 3. Hardened pcre2test so as to reduce the number of bugs reported by fuzzers:
1044
1045 (a) Check for malloc failures when getting memory for the ovector (POSIX) or
1046 the match data block (non-POSIX).
1047
1048 4. In the 32-bit library in non-UTF mode, an attempt to find a Unicode property
1049 for a character with a code point greater than 0x10ffff (the Unicode maximum)
1050 caused a crash.
1051
1052 5. If a lookbehind assertion that contained a back reference to a group
1053 appearing later in the pattern was compiled with the PCRE2_ANCHORED option,
1054 undefined actions (often a segmentation fault) could occur, depending on what
1055 other options were set. An example assertion is (?<!\1(abc)) where the
1056 reference \1 precedes the group (abc). This fixes oss-fuzz issue 865.
1057
1058 6. Added the PCRE2_INFO_FRAMESIZE item to pcre2_pattern_info() and arranged for
1059 pcre2test to use it to output the frame size when the "framesize" modifier is
1060 given.
1061
1062 7. Reworked the recursive pattern matching in the JIT compiler to follow the
1063 interpreter changes.
1064
1065 8. When the zero_terminate modifier was specified on a pcre2test subject line
1066 for global matching, unpredictable things could happen. For example, in UTF-8
1067 mode, the pattern //g,zero_terminate read random memory when matched against an
1068 empty string with zero_terminate. This was a bug in pcre2test, not the library.
1069
1070 9. Moved some Windows-specific code in pcre2grep (introduced in 10.23/13) out
1071 of the section that is compiled when Unix-style directory scanning is
1072 available, and into a new section that is always compiled for Windows.
1073
1074 10. In pcre2test, explicitly close the file after an error during serialization
1075 or deserialization (the "load" or "save" commands).
1076
1077 11. Fix memory leak in pcre2_serialize_decode() when the input is invalid.
1078
1079 12. Fix potential NULL dereference in pcre2_callout_enumerate() if called with
1080 a NULL pattern pointer when Unicode support is available.
1081
1082 13. When the 32-bit library was being tested by pcre2test, error messages that
1083 were longer than 64 code units could cause a buffer overflow. This was a bug in
1084 pcre2test.
1085
1086 14. The alternative matching function, pcre2_dfa_match() misbehaved if it
1087 encountered a character class with a possessive repeat, for example [a-f]{3}+.
1088
1089 15. The depth (formerly recursion) limit now applies to DFA matching (as
1090 of 10.23/36); pcre2test has been upgraded so that \=find_limits works with DFA
1091 matching to find the minimum value for this limit.
1092
1093 16. Since 10.21, if pcre2_match() was called with a null context, default
1094 memory allocation functions were used instead of whatever was used when the
1095 pattern was compiled.
1096
1097 17. Changes to the pcre2test "memory" modifier on a subject line. These apply
1098 only to pcre2_match():
1099
1100 (a) Warn if null_context is set on both pattern and subject, because the
1101 memory details cannot then be shown.
1102
1103 (b) Remember (up to a certain number of) memory allocations and their
1104 lengths, and list only the lengths, so as to be system-independent.
1105 (In practice, the new interpreter never has more than 2 blocks allocated
1106 simultaneously.)
1107
1108 18. Make pcre2test detect an error return from pcre2_get_error_message(), give
1109 a message, and abandon the run (this would have detected #13 above).
1110
1111 19. Implemented PCRE2_ENDANCHORED.
1112
1113 20. Applied Jason Hood's patches (slightly modified) to pcre2grep, to implement
1114 the --output=text (-O) option and the inbuilt callout echo.
1115
1116 21. Extend auto-anchoring etc. to ignore groups with a zero qualifier and
1117 single-branch conditions with a false condition (e.g. DEFINE) at the start of a
1118 branch. For example, /(?(DEFINE)...)^A/ and /(...){0}^B/ are now flagged as
1119 anchored.
1120
1121 22. Added an explicit limit on the amount of heap used by pcre2_match(), set by
1122 pcre2_set_heap_limit() or (*LIMIT_HEAP=xxx). Upgraded pcre2test to show the
1123 heap limit along with other pattern information, and to find the minimum when
1124 the find_limits modifier is set.
1125
1126 23. Write to the last 8 bytes of the pcre2_real_code structure when a compiled
1127 pattern is set up so as to initialize any padding the compiler might have
1128 included. This avoids valgrind warnings when a compiled pattern is copied, in
1129 particular when it is serialized.
1130
1131 24. Remove a redundant line of code left in accidentally a long time ago.
1132
1133 25. Remove a duplication typo in pcre2_tables.c
1134
1135 26. Correct an incorrect cast in pcre2_valid_utf.c
1136
1137 27. Update pcre2test, remove some unused code in pcre2_match(), and upgrade the
1138 tests to improve coverage.
1139
1140 28. Some fixes/tidies as a result of looking at Coverity Scan output:
1141
1142 (a) Typo: ">" should be ">=" in opcode check in pcre2_auto_possess.c.
1143 (b) Added some casts to avoid "suspicious implicit sign extension".
1144 (c) Resource leaks in pcre2test in rare error cases.
1145 (d) Avoid warning for never-use case OP_TABLE_LENGTH which is just a fudge
1146 for checking at compile time that tables are the right size.
1147 (e) Add missing "fall through" comment.
1148
1149 29. Implemented PCRE2_EXTENDED_MORE and related /xx and (?xx) features.
1150
1151 30. Implement (?n: for PCRE2_NO_AUTO_CAPTURE, because Perl now has this.
1152
1153 31. If more than one of "push", "pushcopy", or "pushtablescopy" were set in
1154 pcre2test, a crash could occur.
1155
1156 32. Make -bigstack in RunTest allocate a 64MiB stack (instead of 16MiB) so
1157 that all the tests can run with clang's sanitizing options.
1158
1159 33. Implement extra compile options in the compile context and add the first
1160 one: PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES.
1161
1162 34. Implement newline type PCRE2_NEWLINE_NUL.
1163
1164 35. A lookbehind assertion that had a zero-length branch caused undefined
1165 behaviour when processed by pcre2_dfa_match(). This is oss-fuzz issue 1859.
1166
1167 36. The match limit value now also applies to pcre2_dfa_match() as there are
1168 patterns that can use up a lot of resources without necessarily recursing very
1169 deeply. (Compare item 10.23/36.) This should fix oss-fuzz #1761.
1170
1171 37. Implement PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL.
1172
1173 38. Fix returned offsets from regexec() when REG_STARTEND is used with a
1174 starting offset greater than zero.
1175
1176 39. Implement REG_PEND (GNU extension) for the POSIX wrapper.
1177
1178 40. Implement the subject_literal modifier in pcre2test, and allow jitstack on
1179 pattern lines.
1180
1181 41. Implement PCRE2_LITERAL and use it to support REG_NOSPEC.
1182
1183 42. Implement PCRE2_EXTRA_MATCH_LINE and PCRE2_EXTRA_MATCH_WORD for the benefit
1184 of pcre2grep.
1185
1186 43. Re-implement pcre2grep's -F, -w, and -x options using PCRE2_LITERAL,
1187 PCRE2_EXTRA_MATCH_WORD, and PCRE2_EXTRA_MATCH_LINE. This fixes two bugs:
1188
1189 (a) The -F option did not work for fixed strings containing \E.
1190 (b) The -w option did not work for patterns with multiple branches.
1191
1192 44. Added configuration options for the SELinux compatible execmem allocator in
1193 JIT.
1194
1195 45. Increased the limit for searching for a "must be present" code unit in
1196 subjects from 1000 to 2000 for 8-bit searches, since they use memchr() and are
1197 much faster.
1198
1199 46. Arrange for anchored patterns to record and use "first code unit" data,
1200 because this can give a fast "no match" without searching for a "required code
1201 unit". Previously only non-anchored patterns did this.
1202
1203 47. Upgraded the Unicode tables from Unicode 8.0.0 to Unicode 10.0.0.
1204
1205 48. Add the callout_no_where modifier to pcre2test.
1206
1207 49. Update extended grapheme breaking rules to the latest set that are in
1208 Unicode Standard Annex #29.
1209
1210 50. Added experimental foreign pattern conversion facilities
1211 (pcre2_pattern_convert() and friends).
1212
1213 51. Change the macro FWRITE, used in pcre2grep, to FWRITE_IGNORE because FWRITE
1214 is defined in a system header in cygwin. Also modified some of the #ifdefs in
1215 pcre2grep related to Windows and Cygwin support.
1216
1217 52. Change 3(g) for 10.23 was a bit too zealous. If a hyphen that follows a
1218 character class is the last character in the class, Perl does not give a
1219 warning. PCRE2 now also treats this as a literal.
1220
1221 53. Related to 52, though PCRE2 was throwing an error for [[:digit:]-X] it was
1222 not doing so for [\d-X] (and similar escapes), as is documented.
1223
1224 54. Fixed a MIPS issue in the JIT compiler reported by Joshua Kinard.
1225
1226 55. Fixed a "maybe uninitialized" warning for class_uchardata in \p handling in
1227 pcre2_compile() which could never actually trigger (code should have been cut
1228 out when Unicode support is disabled).
1229
1230
1231 Version 10.23 14-February-2017
1232 ------------------------------
1233
1234 1. Extended pcre2test with the utf8_input modifier so that it is able to
1235 generate all possible 16-bit and 32-bit code unit values in non-UTF modes.
1236
1237 2. In any wide-character mode (8-bit UTF or any 16-bit or 32-bit mode), without
1238 PCRE2_UCP set, a negative character type such as \D in a positive class should
1239 cause all characters greater than 255 to match, whatever else is in the class.
1240 There was a bug that caused this not to happen if a Unicode property item was
1241 added to such a class, for example [\D\P{Nd}] or [\W\pL].
1242
1243 3. There has been a major re-factoring of the pcre2_compile.c file. Most syntax
1244 checking is now done in the pre-pass that identifies capturing groups. This has
1245 reduced the amount of duplication and made the code tidier. While doing this,
1246 some minor bugs and Perl incompatibilities were fixed, including:
1247
1248 (a) \Q\E in the middle of a quantifier such as A+\Q\E+ is now ignored instead
1249 of giving an invalid quantifier error.
1250
1251 (b) {0} can now be used after a group in a lookbehind assertion; previously
1252 this caused an "assertion is not fixed length" error.
1253
1254 (c) Perl always treats (?(DEFINE) as a "define" group, even if a group with
1255 the name "DEFINE" exists. PCRE2 now does likewise.
1256
1257 (d) A recursion condition test such as (?(R2)...) must now refer to an
1258 existing subpattern.
1259
1260 (e) A conditional recursion test such as (?(R)...) misbehaved if there was a
1261 group whose name began with "R".
1262
1263 (f) When testing zero-terminated patterns under valgrind, the terminating
1264 zero is now marked "no access". This catches bugs that would otherwise
1265 show up only with non-zero-terminated patterns.
1266
1267 (g) A hyphen appearing immediately after a POSIX character class (for example
1268 /[[:ascii:]-z]/) now generates an error. Perl does accept this as a
1269 literal, but gives a warning, so it seems best to fail it in PCRE.
1270
1271 (h) An empty \Q\E sequence may appear after a callout that precedes an
1272 assertion condition (it is, of course, ignored).
1273
1274 One effect of the refactoring is that some error numbers and messages have
1275 changed, and the pattern offset given for compiling errors is not always the
1276 right-most character that has been read. In particular, for a variable-length
1277 lookbehind assertion it now points to the start of the assertion. Another
1278 change is that when a callout appears before a group, the "length of next
1279 pattern item" that is passed now just gives the length of the opening
1280 parenthesis item, not the length of the whole group. A length of zero is now
1281 given only for a callout at the end of the pattern. Automatic callouts are no
1282 longer inserted before and after explicit callouts in the pattern.
1283
1284 A number of bugs in the refactored code were subsequently fixed during testing
1285 before release, but after the code was made available in the repository. Many
1286 of the bugs were discovered by fuzzing testing. Several of them were related to
1287 the change from assuming a zero-terminated pattern (which previously had
1288 required non-zero terminated strings to be copied). These bugs were never in
1289 fully released code, but are noted here for the record.
1290
1291 (a) An overall recursion such as (?0) inside a lookbehind assertion was not
1292 being diagnosed as an error.
1293
1294 (b) In utf mode, the length of a *MARK (or other verb) name was being checked
1295 in characters instead of code units, which could lead to bad code being
1296 compiled, leading to unpredictable behaviour.
1297
1298 (c) In extended /x mode, characters whose code was greater than 255 caused
1299 a lookup outside one of the global tables. A similar bug existed for wide
1300 characters in *VERB names.
1301
1302 (d) The amount of memory needed for a compiled pattern was miscalculated if a
1303 lookbehind contained more than one toplevel branch and the first branch
1304 was of length zero.
1305
1306 (e) In UTF-8 or UTF-16 modes with PCRE2_EXTENDED (/x) set and a non-zero-
1307 terminated pattern, if a # comment ran on to the end of the pattern, one
1308 or more code units past the end were being read.
1309
1310 (f) An unterminated repeat at the end of a non-zero-terminated pattern (e.g.
1311 "{2,2") could cause reading beyond the pattern.
1312
1313 (g) When reading a callout string, if the end delimiter was at the end of the
1314 pattern one further code unit was read.
1315
1316 (h) An unterminated number after \g' could cause reading beyond the pattern.
1317
1318 (i) An insufficient memory size was being computed for compiling with
1319 PCRE2_AUTO_CALLOUT.
1320
1321 (j) A conditional group with an assertion condition used more memory than was
1322 allowed for it during parsing, so too many of them could therefore
1323 overrun a buffer.
1324
1325 (k) If parsing a pattern exactly filled the buffer, the internal test for
1326 overrun did not check when the final META_END item was added.
1327
1328 (l) If a lookbehind contained a subroutine call, and the called group
1329 contained an option setting such as (?s), and the PCRE2_ANCHORED option
1330 was set, unpredictable behaviour could occur. The underlying bug was
1331 incorrect code and insufficient checking while searching for the end of
1332 the called subroutine in the parsed pattern.
1333
1334 (m) Quantifiers following (*VERB)s were not being diagnosed as errors.
1335
1336 (n) The use of \Q...\E in a (*VERB) name when PCRE2_ALT_VERBNAMES and
1337 PCRE2_AUTO_CALLOUT were both specified caused undetermined behaviour.
1338
1339 (o) If \Q was preceded by a quantified item, and the following \E was
1340 followed by '?' or '+', and there was at least one literal character
1341 between them, an internal error "unexpected repeat" occurred (example:
1342 /.+\QX\E+/).
1343
1344 (p) A buffer overflow could occur while sorting the names in the group name
1345 list (depending on the order in which the names were seen).
1346
1347 (q) A conditional group that started with a callout was not doing the right
1348 check for a following assertion, leading to compiling bad code. Example:
1349 /(?(C'XX))?!XX/
1350
1351 (r) If a character whose code point was greater than 0xffff appeared within
1352 a lookbehind that was within another lookbehind, the calculation of the
1353 lookbehind length went wrong and could provoke an internal error.
1354
1355 (t) The sequence \E- or \Q\E- after a POSIX class in a character class caused
1356 an internal error. Now the hyphen is treated as a literal.
1357
1358 4. Back references are now permitted in lookbehind assertions when there are
1359 no duplicated group numbers (that is, (?| has not been used), and, if the
1360 reference is by name, there is only one group of that name. The referenced
1361 group must, of course be of fixed length.
1362
1363 5. pcre2test has been upgraded so that, when run under valgrind with valgrind
1364 support enabled, reading past the end of the pattern is detected, both when
1365 compiling and during callout processing.
1366
1367 6. \g{+<number>} (e.g. \g{+2} ) is now supported. It is a "forward back
1368 reference" and can be useful in repetitions (compare \g{-<number>} ). Perl does
1369 not recognize this syntax.
1370
1371 7. Automatic callouts are no longer generated before and after callouts in the
1372 pattern.
1373
1374 8. When pcre2test was outputing information from a callout, the caret indicator
1375 for the current position in the subject line was incorrect if it was after an
1376 escape sequence for a character whose code point was greater than \x{ff}.
1377
1378 9. Change 19 for 10.22 had a typo (PCRE_STATIC_RUNTIME should be
1379 PCRE2_STATIC_RUNTIME). Fix from David Gaussmann.
1380
1381 10. Added --max-buffer-size to pcre2grep, to allow for automatic buffer
1382 expansion when long lines are encountered. Original patch by Dmitry
1383 Cherniachenko.
1384
1385 11. If pcre2grep was compiled with JIT support, but the library was compiled
1386 without it (something that neither ./configure nor CMake allow, but it can be
1387 done by editing config.h), pcre2grep was giving a JIT error. Now it detects
1388 this situation and does not try to use JIT.
1389
1390 12. Added some "const" qualifiers to variables in pcre2grep.
1391
1392 13. Added Dmitry Cherniachenko's patch for colouring output in Windows
1393 (untested by me). Also, look for GREP_COLOUR or GREP_COLOR if the environment
1394 variables PCRE2GREP_COLOUR and PCRE2GREP_COLOR are not found.
1395
1396 14. Add the -t (grand total) option to pcre2grep.
1397
1398 15. A number of bugs have been mended relating to match start-up optimizations
1399 when the first thing in a pattern is a positive lookahead. These all applied
1400 only when PCRE2_NO_START_OPTIMIZE was *not* set:
1401
1402 (a) A pattern such as (?=.*X)X$ was incorrectly optimized as if it needed
1403 both an initial 'X' and a following 'X'.
1404 (b) Some patterns starting with an assertion that started with .* were
1405 incorrectly optimized as having to match at the start of the subject or
1406 after a newline. There are cases where this is not true, for example,
1407 (?=.*[A-Z])(?=.{8,16})(?!.*[\s]) matches after the start in lines that
1408 start with spaces. Starting .* in an assertion is no longer taken as an
1409 indication of matching at the start (or after a newline).
1410
1411 16. The "offset" modifier in pcre2test was not being ignored (as documented)
1412 when the POSIX API was in use.
1413
1414 17. Added --enable-fuzz-support to "configure", causing an non-installed
1415 library containing a test function that can be called by fuzzers to be
1416 compiled. A non-installed binary to run the test function locally, called
1417 pcre2fuzzcheck is also compiled.
1418
1419 18. A pattern with PCRE2_DOTALL (/s) set but not PCRE2_NO_DOTSTAR_ANCHOR, and
1420 which started with .* inside a positive lookahead was incorrectly being
1421 compiled as implicitly anchored.
1422
1423 19. Removed all instances of "register" declarations, as they are considered
1424 obsolete these days and in any case had become very haphazard.
1425
1426 20. Add strerror() to pcre2test for failed file opening.
1427
1428 21. Make pcre2test -C list valgrind support when it is enabled.
1429
1430 22. Add the use_length modifier to pcre2test.
1431
1432 23. Fix an off-by-one bug in pcre2test for the list of names for 'get' and
1433 'copy' modifiers.
1434
1435 24. Add PCRE2_CALL_CONVENTION into the prototype declarations in pcre2.h as it
1436 is apparently needed there as well as in the function definitions. (Why did
1437 nobody ask for this in PCRE1?)
1438
1439 25. Change the _PCRE2_H and _PCRE2_UCP_H guard macros in the header files to
1440 PCRE2_H_IDEMPOTENT_GUARD and PCRE2_UCP_H_IDEMPOTENT_GUARD to be more standard
1441 compliant and unique.
1442
1443 26. pcre2-config --libs-posix was listing -lpcre2posix instead of
1444 -lpcre2-posix. Also, the CMake build process was building the library with the
1445 wrong name.
1446
1447 27. In pcre2test, give some offset information for errors in hex patterns.
1448 This uses the C99 formatting sequence %td, except for MSVC which doesn't
1449 support it - %lu is used instead.
1450
1451 28. Implemented pcre2_code_copy_with_tables(), and added pushtablescopy to
1452 pcre2test for testing it.
1453
1454 29. Fix small memory leak in pcre2test.
1455
1456 30. Fix out-of-bounds read for partial matching of /./ against an empty string
1457 when the newline type is CRLF.
1458
1459 31. Fix a bug in pcre2test that caused a crash when a locale was set either in
1460 the current pattern or a previous one and a wide character was matched.
1461
1462 32. The appearance of \p, \P, or \X in a substitution string when
1463 PCRE2_SUBSTITUTE_EXTENDED was set caused a segmentation fault (NULL
1464 dereference).
1465
1466 33. If the starting offset was specified as greater than the subject length in
1467 a call to pcre2_substitute() an out-of-bounds memory reference could occur.
1468
1469 34. When PCRE2 was compiled to use the heap instead of the stack for recursive
1470 calls to match(), a repeated minimizing caseless back reference, or a
1471 maximizing one where the two cases had different numbers of code units,
1472 followed by a caseful back reference, could lose the caselessness of the first
1473 repeated back reference (example: /(Z)(a)\2{1,2}?(?-i)\1X/i should match ZaAAZX
1474 but didn't).
1475
1476 35. When a pattern is too complicated, PCRE2 gives up trying to find a minimum
1477 matching length and just records zero. Typically this happens when there are
1478 too many nested or recursive back references. If the limit was reached in
1479 certain recursive cases it failed to be triggered and an internal error could
1480 be the result.
1481
1482 36. The pcre2_dfa_match() function now takes note of the recursion limit for
1483 the internal recursive calls that are used for lookrounds and recursions within
1484 the pattern.
1485
1486 37. More refactoring has got rid of the internal could_be_empty_branch()
1487 function (around 400 lines of code, including comments) by keeping track of
1488 could-be-emptiness as the pattern is compiled instead of scanning compiled
1489 groups. (This would have been much harder before the refactoring of #3 above.)
1490 This lifts a restriction on the number of branches in a group (more than about
1491 1100 would give "pattern is too complicated").
1492
1493 38. Add the "-ac" command line option to pcre2test as a synonym for "-pattern
1494 auto_callout".
1495
1496 39. In a library with Unicode support, incorrect data was compiled for a
1497 pattern with PCRE2_UCP set without PCRE2_UTF if a class required all wide
1498 characters to match (for example, /[\s[:^ascii:]]/).
1499
1500 40. The callout_error modifier has been added to pcre2test to make it possible
1501 to return PCRE2_ERROR_CALLOUT from a callout.
1502
1503 41. A minor change to pcre2grep: colour reset is now "<esc>[0m" instead of
1504 "<esc>[00m".
1505
1506 42. The limit in the auto-possessification code that was intended to catch
1507 overly-complicated patterns and not spend too much time auto-possessifying was
1508 being reset too often, resulting in very long compile times for some patterns.
1509 Now such patterns are no longer completely auto-possessified.
1510
1511 43. Applied Jason Hood's revised patch for RunTest.bat.
1512
1513 44. Added a new Windows script RunGrepTest.bat, courtesy of Jason Hood.
1514
1515 45. Minor cosmetic fix to pcre2test: move a variable that is not used under
1516 Windows into the "not Windows" code.
1517
1518 46. Applied Jason Hood's patches to upgrade pcre2grep under Windows and tidy
1519 some of the code:
1520
1521 * normalised the Windows condition by ensuring WIN32 is defined;
1522 * enables the callout feature under Windows;
1523 * adds globbing (Microsoft's implementation expands quoted args),
1524 using a tweaked opendirectory;
1525 * implements the is_*_tty functions for Windows;
1526 * --color=always will write the ANSI sequences to file;
1527 * add sequences 4 (underline works on Win10) and 5 (blink as bright
1528 background, relatively standard on DOS/Win);
1529 * remove the (char *) casts for the now-const strings;
1530 * remove GREP_COLOUR (grep's command line allowed the 'u', but not
1531 the environment), parsing GREP_COLORS instead;
1532 * uses the current colour if not set, rather than black;
1533 * add print_match for the undefined case;
1534 * fixes a typo.
1535
1536 In addition, colour settings containing anything other than digits and
1537 semicolon are ignored, and the colour controls are no longer output for empty
1538 strings.
1539
1540 47. Detecting patterns that are too large inside the length-measuring loop
1541 saves processing ridiculously long patterns to their end.
1542
1543 48. Ignore PCRE2_CASELESS when processing \h, \H, \v, and \V in classes as it
1544 just wastes time. In the UTF case it can also produce redundant entries in
1545 XCLASS lists caused by characters with multiple other cases and pairs of
1546 characters in the same "not-x" sublists.
1547
1548 49. A pattern such as /(?=(a\K))/ can report the end of the match being before
1549 its start; pcre2test was not handling this correctly when using the POSIX
1550 interface (it was OK with the native interface).
1551
1552 50. In pcre2grep, ignore all JIT compile errors. This means that pcre2grep will
1553 continue to work, falling back to interpretation if anything goes wrong with
1554 JIT.
1555
1556 51. Applied patches from Christian Persch to configure.ac to make use of the
1557 AC_USE_SYSTEM_EXTENSIONS macro and to test for functions used by the JIT
1558 modules.
1559
1560 52. Minor fixes to pcre2grep from Jason Hood:
1561 * fixed some spacing;
1562 * Windows doesn't usually use single quotes, so I've added a define
1563 to use appropriate quotes [in an example];
1564 * LC_ALL was displayed as "LCC_ALL";
1565 * numbers 11, 12 & 13 should end in "th";
1566 * use double quotes in usage message.
1567
1568 53. When autopossessifying, skip empty branches without recursion, to reduce
1569 stack usage for the benefit of clang with -fsanitize-address, which uses huge
1570 stack frames. Example pattern: /X?(R||){3335}/. Fixes oss-fuzz issue 553.
1571
1572 54. A pattern with very many explicit back references to a group that is a long
1573 way from the start of the pattern could take a long time to compile because
1574 searching for the referenced group in order to find the minimum length was
1575 being done repeatedly. Now up to 128 group minimum lengths are cached and the
1576 attempt to find a minimum length is abandoned if there is a back reference to a
1577 group whose number is greater than 128. (In that case, the pattern is so
1578 complicated that this optimization probably isn't worth it.) This fixes
1579 oss-fuzz issue 557.
1580
1581 55. Issue 32 for 10.22 below was not correctly fixed. If pcre2grep in multiline
1582 mode with --only-matching matched several lines, it restarted scanning at the
1583 next line instead of moving on to the end of the matched string, which can be
1584 several lines after the start.
1585
1586 56. Applied Jason Hood's new patch for RunGrepTest.bat that updates it in line
1587 with updates to the non-Windows version.
1588
1589
1590
1591 Version 10.22 29-July-2016
1592 --------------------------
1593
1594 1. Applied Jason Hood's patches to RunTest.bat and testdata/wintestoutput3
1595 to fix problems with running the tests under Windows.
1596
1597 2. Implemented a facility for quoting literal characters within hexadecimal
1598 patterns in pcre2test, to make it easier to create patterns with just a few
1599 non-printing characters.
1600
1601 3. Binary zeros are not supported in pcre2test input files. It now detects them
1602 and gives an error.
1603
1604 4. Updated the valgrind parameters in RunTest: (a) changed smc-check=all to
1605 smc-check=all-non-file; (b) changed obj:* in the suppression file to obj:??? so
1606 that it matches only unknown objects.
1607
1608 5. Updated the maintenance script maint/ManyConfigTests to make it easier to
1609 select individual groups of tests.
1610
1611 6. When the POSIX wrapper function regcomp() is called, the REG_NOSUB option
1612 used to set PCRE2_NO_AUTO_CAPTURE when calling pcre2_compile(). However, this
1613 disables the use of back references (and subroutine calls), which are supported
1614 by other implementations of regcomp() with RE_NOSUB. Therefore, REG_NOSUB no
1615 longer causes PCRE2_NO_AUTO_CAPTURE to be set, though it still ignores nmatch
1616 and pmatch when regexec() is called.
1617
1618 7. Because of 6 above, pcre2test has been modified with a new modifier called
1619 posix_nosub, to call regcomp() with REG_NOSUB. Previously the no_auto_capture
1620 modifier had this effect. That option is now ignored when the POSIX API is in
1621 use.
1622
1623 8. Minor tidies to the pcre2demo.c sample program, including more comments
1624 about its 8-bit-ness.
1625
1626 9. Detect unmatched closing parentheses and give the error in the pre-scan
1627 instead of later. Previously the pre-scan carried on and could give a
1628 misleading incorrect error message. For example, /(?J)(?'a'))(?'a')/ gave a
1629 message about invalid duplicate group names.
1630
1631 10. It has happened that pcre2test was accidentally linked with another POSIX
1632 regex library instead of libpcre2-posix. In this situation, a call to regcomp()
1633 (in the other library) may succeed, returning zero, but of course putting its
1634 own data into the regex_t block. In one example the re_pcre2_code field was
1635 left as NULL, which made pcre2test think it had not got a compiled POSIX regex,
1636 so it treated the next line as another pattern line, resulting in a confusing
1637 error message. A check has been added to pcre2test to see if the data returned
1638 from a successful call of regcomp() are valid for PCRE2's regcomp(). If they
1639 are not, an error message is output and the pcre2test run is abandoned. The
1640 message points out the possibility of a mis-linking. Hopefully this will avoid
1641 some head-scratching the next time this happens.
1642
1643 11. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
1644 assertion, caused pcre2test to output a very large number of spaces when the
1645 callout was taken, making the program appearing to loop.
1646
1647 12. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
1648 nested set of parentheses of sufficient size caused an overflow of the
1649 compiling workspace (which was diagnosed, but of course is not desirable).
1650
1651 13. Detect missing closing parentheses during the pre-pass for group
1652 identification.
1653
1654 14. Changed some integer variable types and put in a number of casts, following
1655 a report of compiler warnings from Visual Studio 2013 and a few tests with
1656 gcc's -Wconversion (which still throws up a lot).
1657
1658 15. Implemented pcre2_code_copy(), and added pushcopy and #popcopy to pcre2test
1659 for testing it.
1660
1661 16. Change 66 for 10.21 introduced the use of snprintf() in PCRE2's version of
1662 regerror(). When the error buffer is too small, my version of snprintf() puts a
1663 binary zero in the final byte. Bug #1801 seems to show that other versions do
1664 not do this, leading to bad output from pcre2test when it was checking for
1665 buffer overflow. It no longer assumes a binary zero at the end of a too-small
1666 regerror() buffer.
1667
1668 17. Fixed typo ("&&" for "&") in pcre2_study(). Fortunately, this could not
1669 actually affect anything, by sheer luck.
1670
1671 18. Two minor fixes for MSVC compilation: (a) removal of apparently incorrect
1672 "const" qualifiers in pcre2test and (b) defining snprintf as _snprintf for
1673 older MSVC compilers. This has been done both in src/pcre2_internal.h for most
1674 of the library, and also in src/pcre2posix.c, which no longer includes
1675 pcre2_internal.h (see 24 below).
1676
1677 19. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
1678 static compilation. Subsequently applied Chris Wilson's second patch, putting
1679 the first patch under a new option instead of being unconditional when
1680 PCRE_STATIC is set.
1681
1682 20. Updated pcre2grep to set stdout as binary when run under Windows, so as not
1683 to convert \r\n at the ends of reflected lines into \r\r\n. This required
1684 ensuring that other output that is written to stdout (e.g. file names) uses the
1685 appropriate line terminator: \r\n for Windows, \n otherwise.
1686
1687 21. When a line is too long for pcre2grep's internal buffer, show the maximum
1688 length in the error message.
1689
1690 22. Added support for string callouts to pcre2grep (Zoltan's patch with PH
1691 additions).
1692
1693 23. RunTest.bat was missing a "set type" line for test 22.
1694
1695 24. The pcre2posix.c file was including pcre2_internal.h, and using some
1696 "private" knowledge of the data structures. This is unnecessary; the code has
1697 been re-factored and no longer includes pcre2_internal.h.
1698
1699 25. A racing condition is fixed in JIT reported by Mozilla.
1700
1701 26. Minor code refactor to avoid "array subscript is below array bounds"
1702 compiler warning.
1703
1704 27. Minor code refactor to avoid "left shift of negative number" warning.
1705
1706 28. Add a bit more sanity checking to pcre2_serialize_decode() and document
1707 that it expects trusted data.
1708
1709 29. Fix typo in pcre2_jit_test.c
1710
1711 30. Due to an oversight, pcre2grep was not making use of JIT when available.
1712 This is now fixed.
1713
1714 31. The RunGrepTest script is updated to use the valgrind suppressions file
1715 when testing with JIT under valgrind (compare 10.21/51 below). The suppressions
1716 file is updated so that is now the same as for PCRE1: it suppresses the
1717 Memcheck warnings Addr16 and Cond in unknown objects (that is, JIT-compiled
1718 code). Also changed smc-check=all to smc-check=all-non-file as was done for
1719 RunTest (see 4 above).
1720
1721 32. Implemented the PCRE2_NO_JIT option for pcre2_match().
1722
1723 33. Fix typo that gave a compiler error when JIT not supported.
1724
1725 34. Fix comment describing the returns from find_fixedlength().
1726
1727 35. Fix potential negative index in pcre2test.
1728
1729 36. Calls to pcre2_get_error_message() with error numbers that are never
1730 returned by PCRE2 functions were returning empty strings. Now the error code
1731 PCRE2_ERROR_BADDATA is returned. A facility has been added to pcre2test to
1732 show the texts for given error numbers (i.e. to call pcre2_get_error_message()
1733 and display what it returns) and a few representative error codes are now
1734 checked in RunTest.
1735
1736 37. Added "&& !defined(__INTEL_COMPILER)" to the test for __GNUC__ in
1737 pcre2_match.c, in anticipation that this is needed for the same reason it was
1738 recently added to pcrecpp.cc in PCRE1.
1739
1740 38. Using -o with -M in pcre2grep could cause unnecessary repeated output when
1741 the match extended over a line boundary, as it tried to find more matches "on
1742 the same line" - but it was already over the end.
1743
1744 39. Allow \C in lookbehinds and DFA matching in UTF-32 mode (by converting it
1745 to the same code as '.' when PCRE2_DOTALL is set).
1746
1747 40. Fix two clang compiler warnings in pcre2test when only one code unit width
1748 is supported.
1749
1750 41. Upgrade RunTest to automatically re-run test 2 with a large (64MiB) stack
1751 if it fails when running the interpreter with a 16MiB stack (and if changing
1752 the stack size via pcre2test is possible). This avoids having to manually set a
1753 large stack size when testing with clang.
1754
1755 42. Fix register overwite in JIT when SSE2 acceleration is enabled.
1756
1757 43. Detect integer overflow in pcre2test pattern and data repetition counts.
1758
1759 44. In pcre2test, ignore "allcaptures" after DFA matching.
1760
1761 45. Fix unaligned accesses on x86. Patch by Marc Mutz.
1762
1763 46. Fix some more clang compiler warnings.
1764
1765
1766 Version 10.21 12-January-2016
1767 -----------------------------
1768
1769 1. Improve matching speed of patterns starting with + or * in JIT.
1770
1771 2. Use memchr() to find the first character in an unanchored match in 8-bit
1772 mode in the interpreter. This gives a significant speed improvement.
1773
1774 3. Removed a redundant copy of the opcode_possessify table in the
1775 pcre2_auto_possessify.c source.
1776
1777 4. Fix typos in dftables.c for z/OS.
1778
1779 5. Change 36 for 10.20 broke the handling of [[:>:]] and [[:<:]] in that
1780 processing them could involve a buffer overflow if the following character was
1781 an opening parenthesis.
1782
1783 6. Change 36 for 10.20 also introduced a bug in processing this pattern:
1784 /((?x)(*:0))#(?'/. Specifically: if a setting of (?x) was followed by a (*MARK)
1785 setting (which (*:0) is), then (?x) did not get unset at the end of its group
1786 during the scan for named groups, and hence the external # was incorrectly
1787 treated as a comment and the invalid (?' at the end of the pattern was not
1788 diagnosed. This caused a buffer overflow during the real compile. This bug was
1789 discovered by Karl Skomski with the LLVM fuzzer.
1790
1791 7. Moved the pcre2_find_bracket() function from src/pcre2_compile.c into its
1792 own source module to avoid a circular dependency between src/pcre2_compile.c
1793 and src/pcre2_study.c
1794
1795 8. A callout with a string argument containing an opening square bracket, for
1796 example /(?C$[$)(?<]/, was incorrectly processed and could provoke a buffer
1797 overflow. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1798
1799 9. The handling of callouts during the pre-pass for named group identification
1800 has been tightened up.
1801
1802 10. The quantifier {1} can be ignored, whether greedy, non-greedy, or
1803 possessive. This is a very minor optimization.
1804
1805 11. A possessively repeated conditional group that could match an empty string,
1806 for example, /(?(R))*+/, was incorrectly compiled.
1807
1808 12. The Unicode tables have been updated to Unicode 8.0.0 (thanks to Christian
1809 Persch).
1810
1811 13. An empty comment (?#) in a pattern was incorrectly processed and could
1812 provoke a buffer overflow. This bug was discovered by Karl Skomski with the
1813 LLVM fuzzer.
1814
1815 14. Fix infinite recursion in the JIT compiler when certain patterns such as
1816 /(?:|a|){100}x/ are analysed.
1817
1818 15. Some patterns with character classes involving [: and \\ were incorrectly
1819 compiled and could cause reading from uninitialized memory or an incorrect
1820 error diagnosis. Examples are: /[[:\\](?<[::]/ and /[[:\\](?'abc')[a:]. The
1821 first of these bugs was discovered by Karl Skomski with the LLVM fuzzer.
1822
1823 16. Pathological patterns containing many nested occurrences of [: caused
1824 pcre2_compile() to run for a very long time. This bug was found by the LLVM
1825 fuzzer.
1826
1827 17. A missing closing parenthesis for a callout with a string argument was not
1828 being diagnosed, possibly leading to a buffer overflow. This bug was found by
1829 the LLVM fuzzer.
1830
1831 18. A conditional group with only one branch has an implicit empty alternative
1832 branch and must therefore be treated as potentially matching an empty string.
1833
1834 19. If (?R was followed by - or + incorrect behaviour happened instead of a
1835 diagnostic. This bug was discovered by Karl Skomski with the LLVM fuzzer.
1836
1837 20. Another bug that was introduced by change 36 for 10.20: conditional groups
1838 whose condition was an assertion preceded by an explicit callout with a string
1839 argument might be incorrectly processed, especially if the string contained \Q.
1840 This bug was discovered by Karl Skomski with the LLVM fuzzer.
1841
1842 21. Compiling PCRE2 with the sanitize options of clang showed up a number of
1843 very pedantic coding infelicities and a buffer overflow while checking a UTF-8
1844 string if the final multi-byte UTF-8 character was truncated.
1845
1846 22. For Perl compatibility in EBCDIC environments, ranges such as a-z in a
1847 class, where both values are literal letters in the same case, omit the
1848 non-letter EBCDIC code points within the range.
1849
1850 23. Finding the minimum matching length of complex patterns with back
1851 references and/or recursions can take a long time. There is now a cut-off that
1852 gives up trying to find a minimum length when things get too complex.
1853
1854 24. An optimization has been added that speeds up finding the minimum matching
1855 length for patterns containing repeated capturing groups or recursions.
1856
1857 25. If a pattern contained a back reference to a group whose number was
1858 duplicated as a result of appearing in a (?|...) group, the computation of the
1859 minimum matching length gave a wrong result, which could cause incorrect "no
1860 match" errors. For such patterns, a minimum matching length cannot at present
1861 be computed.
1862
1863 26. Added a check for integer overflow in conditions (?(<digits>) and
1864 (?(R<digits>). This omission was discovered by Karl Skomski with the LLVM
1865 fuzzer.
1866
1867 27. Fixed an issue when \p{Any} inside an xclass did not read the current
1868 character.
1869
1870 28. If pcre2grep was given the -q option with -c or -l, or when handling a
1871 binary file, it incorrectly wrote output to stdout.
1872
1873 29. The JIT compiler did not restore the control verb head in case of *THEN
1874 control verbs. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1875
1876 30. The way recursive references such as (?3) are compiled has been re-written
1877 because the old way was the cause of many issues. Now, conversion of the group
1878 number into a pattern offset does not happen until the pattern has been
1879 completely compiled. This does mean that detection of all infinitely looping
1880 recursions is postponed till match time. In the past, some easy ones were
1881 detected at compile time. This re-writing was done in response to yet another
1882 bug found by the LLVM fuzzer.
1883
1884 31. A test for a back reference to a non-existent group was missing for items
1885 such as \987. This caused incorrect code to be compiled. This issue was found
1886 by Karl Skomski with a custom LLVM fuzzer.
1887
1888 32. Error messages for syntax errors following \g and \k were giving inaccurate
1889 offsets in the pattern.
1890
1891 33. Improve the performance of starting single character repetitions in JIT.
1892
1893 34. (*LIMIT_MATCH=) now gives an error instead of setting the value to 0.
1894
1895 35. Error messages for syntax errors in *LIMIT_MATCH and *LIMIT_RECURSION now
1896 give the right offset instead of zero.
1897
1898 36. The JIT compiler should not check repeats after a {0,1} repeat byte code.
1899 This issue was found by Karl Skomski with a custom LLVM fuzzer.
1900
1901 37. The JIT compiler should restore the control chain for empty possessive
1902 repeats. This issue was found by Karl Skomski with a custom LLVM fuzzer.
1903
1904 38. A bug which was introduced by the single character repetition optimization
1905 was fixed.
1906
1907 39. Match limit check added to recursion. This issue was found by Karl Skomski
1908 with a custom LLVM fuzzer.
1909
1910 40. Arrange for the UTF check in pcre2_match() and pcre2_dfa_match() to look
1911 only at the part of the subject that is relevant when the starting offset is
1912 non-zero.
1913
1914 41. Improve first character match in JIT with SSE2 on x86.
1915
1916 42. Fix two assertion fails in JIT. These issues were found by Karl Skomski
1917 with a custom LLVM fuzzer.
1918
1919 43. Correct the setting of CMAKE_C_FLAGS in CMakeLists.txt (patch from Roy Ivy
1920 III).
1921
1922 44. Fix bug in RunTest.bat for new test 14, and adjust the script for the added
1923 test (there are now 20 in total).
1924
1925 45. Fixed a corner case of range optimization in JIT.
1926
1927 46. Add the ${*MARK} facility to pcre2_substitute().
1928
1929 47. Modifier lists in pcre2test were splitting at spaces without the required
1930 commas.
1931
1932 48. Implemented PCRE2_ALT_VERBNAMES.
1933
1934 49. Fixed two issues in JIT. These were found by Karl Skomski with a custom
1935 LLVM fuzzer.
1936
1937 50. The pcre2test program has been extended by adding the #newline_default
1938 command. This has made it possible to run the standard tests when PCRE2 is
1939 compiled with either CR or CRLF as the default newline convention. As part of
1940 this work, the new command was added to several test files and the testing
1941 scripts were modified. The pcre2grep tests can now also be run when there is no
1942 LF in the default newline convention.
1943
1944 51. The RunTest script has been modified so that, when JIT is used and valgrind
1945 is specified, a valgrind suppressions file is set up to ignore "Invalid read of
1946 size 16" errors because these are false positives when the hardware supports
1947 the SSE2 instruction set.
1948
1949 52. It is now possible to have comment lines amid the subject strings in
1950 pcre2test (and perltest.sh) input.
1951
1952 53. Implemented PCRE2_USE_OFFSET_LIMIT and pcre2_set_offset_limit().
1953
1954 54. Add the null_context modifier to pcre2test so that calling pcre2_compile()
1955 and the matching functions with NULL contexts can be tested.
1956
1957 55. Implemented PCRE2_SUBSTITUTE_EXTENDED.
1958
1959 56. In a character class such as [\W\p{Any}] where both a negative-type escape
1960 ("not a word character") and a property escape were present, the property
1961 escape was being ignored.
1962
1963 57. Fixed integer overflow for patterns whose minimum matching length is very,
1964 very large.
1965
1966 58. Implemented --never-backslash-C.
1967
1968 59. Change 55 above introduced a bug by which certain patterns provoked the
1969 erroneous error "\ at end of pattern".
1970
1971 60. The special sequences [[:<:]] and [[:>:]] gave rise to incorrect compiling
1972 errors or other strange effects if compiled in UCP mode. Found with libFuzzer
1973 and AddressSanitizer.
1974
1975 61. Whitespace at the end of a pcre2test pattern line caused a spurious error
1976 message if there were only single-character modifiers. It should be ignored.
1977
1978 62. The use of PCRE2_NO_AUTO_CAPTURE could cause incorrect compilation results
1979 or segmentation errors for some patterns. Found with libFuzzer and
1980 AddressSanitizer.
1981
1982 63. Very long names in (*MARK) or (*THEN) etc. items could provoke a buffer
1983 overflow.
1984
1985 64. Improve error message for overly-complicated patterns.
1986
1987 65. Implemented an optional replication feature for patterns in pcre2test, to
1988 make it easier to test long repetitive patterns. The tests for 63 above are
1989 converted to use the new feature.
1990
1991 66. In the POSIX wrapper, if regerror() was given too small a buffer, it could
1992 misbehave.
1993
1994 67. In pcre2_substitute() in UTF mode, the UTF validity check on the
1995 replacement string was happening before the length setting when the replacement
1996 string was zero-terminated.
1997
1998 68. In pcre2_substitute() in UTF mode, PCRE2_NO_UTF_CHECK can be set for the
1999 second and subsequent calls to pcre2_match().
2000
2001 69. There was no check for integer overflow for a replacement group number in
2002 pcre2_substitute(). An added check for a number greater than the largest group
2003 number in the pattern means this is not now needed.
2004
2005 70. The PCRE2-specific VERSION condition didn't work correctly if only one
2006 digit was given after the decimal point, or if more than two digits were given.
2007 It now works with one or two digits, and gives a compile time error if more are
2008 given.
2009
2010 71. In pcre2_substitute() there was the possibility of reading one code unit
2011 beyond the end of the replacement string.
2012
2013 72. The code for checking a subject's UTF-32 validity for a pattern with a
2014 lookbehind involved an out-of-bounds pointer, which could potentially cause
2015 trouble in some environments.
2016
2017 73. The maximum lookbehind length was incorrectly calculated for patterns such
2018 as /(?<=(a)(?-1))x/ which have a recursion within a backreference.
2019
2020 74. Give an error if a lookbehind assertion is longer than 65535 code units.
2021
2022 75. Give an error in pcre2_substitute() if a match ends before it starts (as a
2023 result of the use of \K).
2024
2025 76. Check the length of subpattern names and the names in (*MARK:xx) etc.
2026 dynamically to avoid the possibility of integer overflow.
2027
2028 77. Implement pcre2_set_max_pattern_length() so that programs can restrict the
2029 size of patterns that they are prepared to handle.
2030
2031 78. (*NO_AUTO_POSSESS) was not working.
2032
2033 79. Adding group information caching improves the speed of compiling when
2034 checking whether a group has a fixed length and/or could match an empty string,
2035 especially when recursion or subroutine calls are involved. However, this
2036 cannot be used when (?| is present in the pattern because the same number may
2037 be used for groups of different sizes. To catch runaway patterns in this
2038 situation, counts have been introduced to the functions that scan for empty
2039 branches or compute fixed lengths.
2040
2041 80. Allow for the possibility of the size of the nest_save structure not being
2042 a factor of the size of the compiling workspace (it currently is).
2043
2044 81. Check for integer overflow in minimum length calculation and cap it at
2045 65535.
2046
2047 82. Small optimizations in code for finding the minimum matching length.
2048
2049 83. Lock out configuring for EBCDIC with non-8-bit libraries.
2050
2051 84. Test for error code <= 0 in regerror().
2052
2053 85. Check for too many replacements (more than INT_MAX) in pcre2_substitute().
2054
2055 86. Avoid the possibility of computing with an out-of-bounds pointer (though
2056 not dereferencing it) while handling lookbehind assertions.
2057
2058 87. Failure to get memory for the match data in regcomp() is now given as a
2059 regcomp() error instead of waiting for regexec() to pick it up.
2060
2061 88. In pcre2_substitute(), ensure that CRLF is not split when it is a valid
2062 newline sequence.
2063
2064 89. Paranoid check in regcomp() for bad error code from pcre2_compile().
2065
2066 90. Run test 8 (internal offsets and code sizes) for link sizes 3 and 4 as well
2067 as for link size 2.
2068
2069 91. Document that JIT has a limit on pattern size, and give more information
2070 about JIT compile failures in pcre2test.
2071
2072 92. Implement PCRE2_INFO_HASBACKSLASHC.
2073
2074 93. Re-arrange valgrind support code in pcre2test to avoid spurious reports
2075 with JIT (possibly caused by SSE2?).
2076
2077 94. Support offset_limit in JIT.
2078
2079 95. A sequence such as [[:punct:]b] that is, a POSIX character class followed
2080 by a single ASCII character in a class item, was incorrectly compiled in UCP
2081 mode. The POSIX class got lost, but only if the single character followed it.
2082
2083 96. [:punct:] in UCP mode was matching some characters in the range 128-255
2084 that should not have been matched.
2085
2086 97. If [:^ascii:] or [:^xdigit:] are present in a non-negated class, all
2087 characters with code points greater than 255 are in the class. When a Unicode
2088 property was also in the class (if PCRE2_UCP is set, escapes such as \w are
2089 turned into Unicode properties), wide characters were not correctly handled,
2090 and could fail to match.
2091
2092 98. In pcre2test, make the "startoffset" modifier a synonym of "offset",
2093 because it sets the "startoffset" parameter for pcre2_match().
2094
2095 99. If PCRE2_AUTO_CALLOUT was set on a pattern that had a (?# comment between
2096 an item and its qualifier (for example, A(?#comment)?B) pcre2_compile()
2097 misbehaved. This bug was found by the LLVM fuzzer.
2098
2099 100. The error for an invalid UTF pattern string always gave the code unit
2100 offset as zero instead of where the invalidity was found.
2101
2102 101. Further to 97 above, negated classes such as [^[:^ascii:]\d] were also not
2103 working correctly in UCP mode.
2104
2105 102. Similar to 99 above, if an isolated \E was present between an item and its
2106 qualifier when PCRE2_AUTO_CALLOUT was set, pcre2_compile() misbehaved. This bug
2107 was found by the LLVM fuzzer.
2108
2109 103. The POSIX wrapper function regexec() crashed if the option REG_STARTEND
2110 was set when the pmatch argument was NULL. It now returns REG_INVARG.
2111
2112 104. Allow for up to 32-bit numbers in the ordin() function in pcre2grep.
2113
2114 105. An empty \Q\E sequence between an item and its qualifier caused
2115 pcre2_compile() to misbehave when auto callouts were enabled. This bug
2116 was found by the LLVM fuzzer.
2117
2118 106. If both PCRE2_ALT_VERBNAMES and PCRE2_EXTENDED were set, and a (*MARK) or
2119 other verb "name" ended with whitespace immediately before the closing
2120 parenthesis, pcre2_compile() misbehaved. Example: /(*:abc )/, but only when
2121 both those options were set.
2122
2123 107. In a number of places pcre2_compile() was not handling NULL characters
2124 correctly, and pcre2test with the "bincode" modifier was not always correctly
2125 displaying fields containing NULLS:
2126
2127 (a) Within /x extended #-comments
2128 (b) Within the "name" part of (*MARK) and other *verbs
2129 (c) Within the text argument of a callout
2130
2131 108. If a pattern that was compiled with PCRE2_EXTENDED started with white
2132 space or a #-type comment that was followed by (?-x), which turns off
2133 PCRE2_EXTENDED, and there was no subsequent (?x) to turn it on again,
2134 pcre2_compile() assumed that (?-x) applied to the whole pattern and
2135 consequently mis-compiled it. This bug was found by the LLVM fuzzer. The fix
2136 for this bug means that a setting of any of the (?imsxJU) options at the start
2137 of a pattern is no longer transferred to the options that are returned by
2138 PCRE2_INFO_ALLOPTIONS. In fact, this was an anachronism that should have
2139 changed when the effects of those options were all moved to compile time.
2140
2141 109. An escaped closing parenthesis in the "name" part of a (*verb) when
2142 PCRE2_ALT_VERBNAMES was set caused pcre2_compile() to malfunction. This bug
2143 was found by the LLVM fuzzer.
2144
2145 110. Implemented PCRE2_SUBSTITUTE_UNSET_EMPTY, and updated pcre2test to make it
2146 possible to test it.
2147
2148 111. "Harden" pcre2test against ridiculously large values in modifiers and
2149 command line arguments.
2150
2151 112. Implemented PCRE2_SUBSTITUTE_UNKNOWN_UNSET and PCRE2_SUBSTITUTE_OVERFLOW_
2152 LENGTH.
2153
2154 113. Fix printing of *MARK names that contain binary zeroes in pcre2test.
2155
2156
2157 Version 10.20 30-June-2015
2158 --------------------------
2159
2160 1. Callouts with string arguments have been added.
2161
2162 2. Assertion code generator in JIT has been optimized.
2163
2164 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
2165 pcre2_compile() function read past the end of the input before diagnosing an
2166 error. This bug was discovered by the LLVM fuzzer.
2167
2168 4. Implemented pcre2_callout_enumerate().
2169
2170 5. Fix JIT compilation of conditional blocks whose assertion is converted to
2171 (*FAIL). E.g: /(?(?!))/.
2172
2173 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
2174 discovered by the LLVM fuzzer.
2175
2176 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
2177 when this assertion was used as a condition, for example (?(?!)a|b). In
2178 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
2179 error about an unsupported item.
2180
2181 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
2182 possessification code could take exponential time to complete. A recursion
2183 depth limit of 1000 has been imposed to limit the resources used by this
2184 optimization. This infelicity was discovered by the LLVM fuzzer.
2185
2186 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
2187 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
2188 because \S ensures they are all in the class. The code for doing this was
2189 interacting badly with the code for computing the amount of space needed to
2190 compile the pattern, leading to a buffer overflow. This bug was discovered by
2191 the LLVM fuzzer.
2192
2193 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
2194 other kinds of group caused stack overflow at compile time. This bug was
2195 discovered by the LLVM fuzzer.
2196
2197 11. A pattern such as /(?1)(?#?'){8}(a)/ which had a parenthesized comment
2198 between a subroutine call and its quantifier was incorrectly compiled, leading
2199 to buffer overflow or other errors. This bug was discovered by the LLVM fuzzer.
2200
2201 12. The illegal pattern /(?(?<E>.*!.*)?)/ was not being diagnosed as missing an
2202 assertion after (?(. The code was failing to check the character after (?(?<
2203 for the ! or = that would indicate a lookbehind assertion. This bug was
2204 discovered by the LLVM fuzzer.
2205
2206 13. A pattern such as /X((?2)()*+){2}+/ which has a possessive quantifier with
2207 a fixed maximum following a group that contains a subroutine reference was
2208 incorrectly compiled and could trigger buffer overflow. This bug was discovered
2209 by the LLVM fuzzer.
2210
2211 14. Negative relative recursive references such as (?-7) to non-existent
2212 subpatterns were not being diagnosed and could lead to unpredictable behaviour.
2213 This bug was discovered by the LLVM fuzzer.
2214
2215 15. The bug fixed in 14 was due to an integer variable that was unsigned when
2216 it should have been signed. Some other "int" variables, having been checked,
2217 have either been changed to uint32_t or commented as "must be signed".
2218
2219 16. A mutual recursion within a lookbehind assertion such as (?<=((?2))((?1)))
2220 caused a stack overflow instead of the diagnosis of a non-fixed length
2221 lookbehind assertion. This bug was discovered by the LLVM fuzzer.
2222
2223 17. The use of \K in a positive lookbehind assertion in a non-anchored pattern
2224 (e.g. /(?<=\Ka)/) could make pcre2grep loop.
2225
2226 18. There was a similar problem to 17 in pcre2test for global matches, though
2227 the code there did catch the loop.
2228
2229 19. If a greedy quantified \X was preceded by \C in UTF mode (e.g. \C\X*),
2230 and a subsequent item in the pattern caused a non-match, backtracking over the
2231 repeated \X did not stop, but carried on past the start of the subject, causing
2232 reference to random memory and/or a segfault. There were also some other cases
2233 where backtracking after \C could crash. This set of bugs was discovered by the
2234 LLVM fuzzer.
2235
2236 20. The function for finding the minimum length of a matching string could take
2237 a very long time if mutual recursion was present many times in a pattern, for
2238 example, /((?2){73}(?2))((?1))/. A better mutual recursion detection method has
2239 been implemented. This infelicity was discovered by the LLVM fuzzer.
2240
2241 21. Implemented PCRE2_NEVER_BACKSLASH_C.
2242
2243 22. The feature for string replication in pcre2test could read from freed
2244 memory if the replication required a buffer to be extended, and it was not
2245 working properly in 16-bit and 32-bit modes. This issue was discovered by a
2246 fuzzer: see http://lcamtuf.coredump.cx/afl/.
2247
2248 23. Added the PCRE2_ALT_CIRCUMFLEX option.
2249
2250 24. Adjust the treatment of \8 and \9 to be the same as the current Perl
2251 behaviour.
2252
2253 25. Static linking against the PCRE2 library using the pkg-config module was
2254 failing on missing pthread symbols.
2255
2256 26. If a group that contained a recursive back reference also contained a
2257 forward reference subroutine call followed by a non-forward-reference
2258 subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
2259 compile correct code, leading to undefined behaviour or an internally detected
2260 error. This bug was discovered by the LLVM fuzzer.
2261
2262 27. Quantification of certain items (e.g. atomic back references) could cause
2263 incorrect code to be compiled when recursive forward references were involved.
2264 For example, in this pattern: /(?1)()((((((\1++))\x85)+)|))/. This bug was
2265 discovered by the LLVM fuzzer.
2266
2267 28. A repeated conditional group whose condition was a reference by name caused
2268 a buffer overflow if there was more than one group with the given name. This
2269 bug was discovered by the LLVM fuzzer.
2270
2271 29. A recursive back reference by name within a group that had the same name as
2272 another group caused a buffer overflow. For example: /(?J)(?'d'(?'d'\g{d}))/.
2273 This bug was discovered by the LLVM fuzzer.
2274
2275 30. A forward reference by name to a group whose number is the same as the
2276 current group, for example in this pattern: /(?|(\k'Pm')|(?'Pm'))/, caused a
2277 buffer overflow at compile time. This bug was discovered by the LLVM fuzzer.
2278
2279 31. Fix -fsanitize=undefined warnings for left shifts of 1 by 31 (it treats 1
2280 as an int; fixed by writing it as 1u).
2281
2282 32. Fix pcre2grep compile when -std=c99 is used with gcc, though it still gives
2283 a warning for "fileno" unless -std=gnu99 us used.
2284
2285 33. A lookbehind assertion within a set of mutually recursive subpatterns could
2286 provoke a buffer overflow. This bug was discovered by the LLVM fuzzer.
2287
2288 34. Give an error for an empty subpattern name such as (?'').
2289
2290 35. Make pcre2test give an error if a pattern that follows #forbud_utf contains
2291 \P, \p, or \X.
2292
2293 36. The way named subpatterns are handled has been refactored. There is now a
2294 pre-pass over the regex which does nothing other than identify named
2295 subpatterns and count the total captures. This means that information about
2296 named patterns is known before the rest of the compile. In particular, it means
2297 that forward references can be checked as they are encountered. Previously, the
2298 code for handling forward references was contorted and led to several errors in
2299 computing the memory requirements for some patterns, leading to buffer
2300 overflows.
2301
2302 37. There was no check for integer overflow in subroutine calls such as (?123).
2303
2304 38. The table entry for \l in EBCDIC environments was incorrect, leading to its
2305 being treated as a literal 'l' instead of causing an error.
2306
2307 39. If a non-capturing group containing a conditional group that could match
2308 an empty string was repeated, it was not identified as matching an empty string
2309 itself. For example: /^(?:(?(1)x|)+)+$()/.
2310
2311 40. In an EBCDIC environment, pcretest was mishandling the escape sequences
2312 \a and \e in test subject lines.
2313
2314 41. In an EBCDIC environment, \a in a pattern was converted to the ASCII
2315 instead of the EBCDIC value.
2316
2317 42. The handling of \c in an EBCDIC environment has been revised so that it is
2318 now compatible with the specification in Perl's perlebcdic page.
2319
2320 43. Single character repetition in JIT has been improved. 20-30% speedup
2321 was achieved on certain patterns.
2322
2323 44. The EBCDIC character 0x41 is a non-breaking space, equivalent to 0xa0 in
2324 ASCII/Unicode. This has now been added to the list of characters that are
2325 recognized as white space in EBCDIC.
2326
2327 45. When PCRE2 was compiled without Unicode support, the use of \p and \P gave
2328 an error (correctly) when used outside a class, but did not give an error
2329 within a class.
2330
2331 46. \h within a class was incorrectly compiled in EBCDIC environments.
2332
2333 47. JIT should return with error when the compiled pattern requires
2334 more stack space than the maximum.
2335
2336 48. Fixed a memory leak in pcre2grep when a locale is set.
2337
2338
2339 Version 10.10 06-March-2015
2340 ---------------------------
2341
2342 1. When a pattern is compiled, it remembers the highest back reference so that
2343 when matching, if the ovector is too small, extra memory can be obtained to
2344 use instead. A conditional subpattern whose condition is a check on a capture
2345 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
2346 another kind of back reference, but it was not setting the highest
2347 backreference number. This mattered only if pcre2_match() was called with an
2348 ovector that was too small to hold the capture, and there was no other kind of
2349 back reference (a situation which is probably quite rare). The effect of the
2350 bug was that the condition was always treated as FALSE when the capture could
2351 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
2352 has been fixed.
2353
2354 2. Functions for serialization and deserialization of sets of compiled patterns
2355 have been added.
2356
2357 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
2358 excess code units at the end of the data block that may occasionally occur if
2359 the code for calculating the size over-estimates. This change stops the
2360 serialization code copying uninitialized data, to which valgrind objects. The
2361 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
2362 include the general overhead. This has been corrected.
2363
2364 4. All code units in every slot in the table of group names are now set, again
2365 in order to avoid accessing uninitialized data when serializing.
2366
2367 5. The (*NO_JIT) feature is implemented.
2368
2369 6. If a bug that caused pcre2_compile() to use more memory than allocated was
2370 triggered when using valgrind, the code in (3) above passed a stupidly large
2371 value to valgrind. This caused a crash instead of an "internal error" return.
2372
2373 7. A reference to a duplicated named group (either a back reference or a test
2374 for being set in a conditional) that occurred in a part of the pattern where
2375 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
2376 to be incorrectly calculated, leading to overwriting.
2377
2378 8. A mutually recursive set of back references such as (\2)(\1) caused a
2379 segfault at compile time (while trying to find the minimum matching length).
2380 The infinite loop is now broken (with the minimum length unset, that is, zero).
2381
2382 9. If an assertion that was used as a condition was quantified with a minimum
2383 of zero, matching went wrong. In particular, if the whole group had unlimited
2384 repetition and could match an empty string, a segfault was likely. The pattern
2385 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
2386 quantified, but not if they are being used as conditions, so the above pattern
2387 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
2388 patterns.
2389
2390 10. The error message for an invalid quantifier has been changed from "nothing
2391 to repeat" to "quantifier does not follow a repeatable item".
2392
2393 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
2394 scanning the compiled pattern in subsequent auto-possessification can get out
2395 of step and lead to an unknown opcode. Previously this could have caused an
2396 infinite loop. Now it generates an "internal error" error. This is a tidyup,
2397 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
2398 undefined outcome.
2399
2400 12. A UTF pattern containing a "not" match of a non-ASCII character and a
2401 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
2402
2403 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
2404 that is found in the output of "locale -a" can actually be set by pcre2test
2405 before it is accepted. Previously, in an environment where a locale was listed
2406 but would not set (an example does exist), the test would "pass" without
2407 actually doing anything. Also the fr_CA locale has been added to the list of
2408 locales that can be used.
2409
2410 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
2411 capturing group number without parentheses, the last character was incorrectly
2412 literally included at the end of the replacement string.
2413
2414 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
2415 failed to allow the zero-repeat case if pcre2_match() was called with an
2416 ovector too small to capture the group.
2417
2418 16. Improved error message in pcre2test when setting the stack size (-S) fails.
2419
2420 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
2421 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
2422 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
2423 of PCRE2 sources, which caused a failure to build pcre2test.
2424
2425 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
2426 only on Windows.
2427
2428 19. Use binary input when reading back saved serialized patterns in pcre2test.
2429
2430 20. Added RunTest.bat for running the tests under Windows.
2431
2432 21. "make distclean" was not removing config.h, a file that may be created for
2433 use with CMake.
2434
2435 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
2436 forward reference repeated a large (but limited) number of times within a
2437 repeated outer group that has a zero minimum quantifier, caused incorrect code
2438 to be compiled, leading to the error "internal error: previously-checked
2439 referenced subpattern not found" when an incorrect memory address was read.
2440 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
2441 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
2442
2443 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
2444 call within a group that also contained a recursive back reference caused
2445 incorrect code to be compiled. This bug was reported as "heap overflow",
2446 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
2447 CVE-2015-2326 was given to this.)
2448
2449 24. Computing the size of the JIT read-only data in advance has been a source
2450 of various issues, and new ones are still appear unfortunately. To fix
2451 existing and future issues, size computation is eliminated from the code,
2452 and replaced by on-demand memory allocation.
2453
2454 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
2455 adjacent to the end of the range, and the range contained characters with more
2456 than one other case, caused incorrect behaviour when compiled in UTF mode. In
2457 that example, the range a-j was left out of the class.
2458
2459
2460 Version 10.00 05-January-2015
2461 -----------------------------
2462
2463 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
2464 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
2465 API, up to item 20 for release 8.36.
2466
2467 The code of the library was heavily revised as part of the new API
2468 implementation. Details of each and every modification were not individually
2469 logged. In addition to the API changes, the following changes were made. They
2470 are either new functionality, or bug fixes and other noticeable changes of
2471 behaviour that were implemented after the code had been forked.
2472
2473 1. Including Unicode support at build time is now enabled by default, but it
2474 can optionally be disabled. It is not enabled by default at run time (no
2475 change).
2476
2477 2. The test program, now called pcre2test, was re-specified and almost
2478 completely re-written. Its input is not compatible with input for pcretest.
2479
2480 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
2481 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
2482 matched by that pattern.
2483
2484 4. For the benefit of those who use PCRE2 via some other application, that is,
2485 not writing the function calls themselves, it is possible to check the PCRE2
2486 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
2487 string such as "yesno".
2488
2489 5. There are case-equivalent Unicode characters whose encodings use different
2490 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
2491 theoretically possible for this to happen in UTF-16 too.) If a backreference to
2492 a group containing one of these characters was greedily repeated, and during
2493 the match a backtrack occurred, the subject might be backtracked by the wrong
2494 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
2495 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
2496 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
2497 Incorrect backtracking meant that group 2 captured only the last two bytes.
2498 This bug has been fixed; the new code is slower, but it is used only when the
2499 strings matched by the repetition are not all the same length.
2500
2501 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
2502 information. This applied to any pattern with a group that matched no
2503 characters, for example: /(?:(?=.)|(?<!x))a/.
2504
2505 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
2506 those parentheses to be closed with whatever has been captured so far. However,
2507 it was failing to mark any other groups between the highest capture so far and
2508 the currrent group as "unset". Thus, the ovector for those groups contained
2509 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
2510 matched against "abcd".
2511
2512 8. The pcre2_substitute() function has been implemented.
2513
2514 9. If an assertion used as a condition was quantified with a minimum of zero
2515 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
2516 occur.
2517
2518 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.
2519
2520 ****

  ViewVC Help
Powered by ViewVC 1.1.5