/[pcre2]/code/trunk/ChangeLog
ViewVC logotype

Contents of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log


Revision 234 - (show annotations)
Fri Mar 27 17:45:02 2015 UTC (2 years, 8 months ago) by ph10
File size: 11315 byte(s)
Error occurred while calculating annotation data.
Fix mutual recursion inside other parentheses stack overflow bug.
1 Change Log for PCRE2
2 --------------------
3
4 Version 10.20 xx-xx-2015
5 ------------------------
6
7 1. Callouts with string arguments have been added.
8
9 2. Assertion code generator in JIT has been optimized.
10
11 3. The invalid pattern (?(?C) has a missing assertion condition at the end. The
12 pcre2_compile() function read past the end of the input before diagnosing an
13 error. This bug was discovered by the LLVM fuzzer.
14
15 4. Implemented pcre2_callout_enumerate().
16
17 5. Fix JIT compilation of conditional blocks whose assertion is converted to
18 (*FAIL). E.g: /(?(?!))/.
19
20 6. The pattern /(?(?!)^)/ caused references to random memory. This bug was
21 discovered by the LLVM fuzzer.
22
23 7. The assertion (?!) is optimized to (*FAIL). This was not handled correctly
24 when this assertion was used as a condition, for example (?(?!)a|b). In
25 pcre2_match() it worked by luck; in pcre2_dfa_match() it gave an incorrect
26 error about an unsupported item.
27
28 8. For some types of pattern, for example /Z*(|d*){216}/, the auto-
29 possessification code could take exponential time to complete. A recursion
30 depth limit of 10000 has been imposed to limit the resources used by this
31 optimization. This infelicity was discovered by the LLVM fuzzer.
32
33 9. A pattern such as /(*UTF)[\S\V\H]/, which contains a negated special class
34 such as \S in non-UCP mode, explicit wide characters (> 255) can be ignored
35 because \S ensures they are all in the class. The code for doing this was
36 interacting badly with the code for computing the amount of space needed to
37 compile the pattern, leading to a buffer overflow. This bug was discovered by
38 the LLVM fuzzer.
39
40 10. A pattern such as /((?2)+)((?1))/ which has mutual recursion nested inside
41 other kinds of group caused stack overflow at compile time. This bug was
42 discovered by the LLVM fuzzer.
43
44
45 Version 10.10 06-March-2015
46 ---------------------------
47
48 1. When a pattern is compiled, it remembers the highest back reference so that
49 when matching, if the ovector is too small, extra memory can be obtained to
50 use instead. A conditional subpattern whose condition is a check on a capture
51 having happened, such as, for example in the pattern /^(?:(a)|b)(?(1)A|B)/, is
52 another kind of back reference, but it was not setting the highest
53 backreference number. This mattered only if pcre2_match() was called with an
54 ovector that was too small to hold the capture, and there was no other kind of
55 back reference (a situation which is probably quite rare). The effect of the
56 bug was that the condition was always treated as FALSE when the capture could
57 not be consulted, leading to a incorrect behaviour by pcre2_match(). This bug
58 has been fixed.
59
60 2. Functions for serialization and deserialization of sets of compiled patterns
61 have been added.
62
63 3. The value that is returned by PCRE2_INFO_SIZE has been corrected to remove
64 excess code units at the end of the data block that may occasionally occur if
65 the code for calculating the size over-estimates. This change stops the
66 serialization code copying uninitialized data, to which valgrind objects. The
67 documentation of PCRE2_INFO_SIZE was incorrect in stating that the size did not
68 include the general overhead. This has been corrected.
69
70 4. All code units in every slot in the table of group names are now set, again
71 in order to avoid accessing uninitialized data when serializing.
72
73 5. The (*NO_JIT) feature is implemented.
74
75 6. If a bug that caused pcre2_compile() to use more memory than allocated was
76 triggered when using valgrind, the code in (3) above passed a stupidly large
77 value to valgrind. This caused a crash instead of an "internal error" return.
78
79 7. A reference to a duplicated named group (either a back reference or a test
80 for being set in a conditional) that occurred in a part of the pattern where
81 PCRE2_DUPNAMES was not set caused the amount of memory needed for the pattern
82 to be incorrectly calculated, leading to overwriting.
83
84 8. A mutually recursive set of back references such as (\2)(\1) caused a
85 segfault at compile time (while trying to find the minimum matching length).
86 The infinite loop is now broken (with the minimum length unset, that is, zero).
87
88 9. If an assertion that was used as a condition was quantified with a minimum
89 of zero, matching went wrong. In particular, if the whole group had unlimited
90 repetition and could match an empty string, a segfault was likely. The pattern
91 (?(?=0)?)+ is an example that caused this. Perl allows assertions to be
92 quantified, but not if they are being used as conditions, so the above pattern
93 is faulted by Perl. PCRE2 has now been changed so that it also rejects such
94 patterns.
95
96 10. The error message for an invalid quantifier has been changed from "nothing
97 to repeat" to "quantifier does not follow a repeatable item".
98
99 11. If a bad UTF string is compiled with NO_UTF_CHECK, it may succeed, but
100 scanning the compiled pattern in subsequent auto-possessification can get out
101 of step and lead to an unknown opcode. Previously this could have caused an
102 infinite loop. Now it generates an "internal error" error. This is a tidyup,
103 not a bug fix; passing bad UTF with NO_UTF_CHECK is documented as having an
104 undefined outcome.
105
106 12. A UTF pattern containing a "not" match of a non-ASCII character and a
107 subroutine reference could loop at compile time. Example: /[^\xff]((?1))/.
108
109 13. The locale test (RunTest 3) has been upgraded. It now checks that a locale
110 that is found in the output of "locale -a" can actually be set by pcre2test
111 before it is accepted. Previously, in an environment where a locale was listed
112 but would not set (an example does exist), the test would "pass" without
113 actually doing anything. Also the fr_CA locale has been added to the list of
114 locales that can be used.
115
116 14. Fixed a bug in pcre2_substitute(). If a replacement string ended in a
117 capturing group number without parentheses, the last character was incorrectly
118 literally included at the end of the replacement string.
119
120 15. A possessive capturing group such as (a)*+ with a minimum repeat of zero
121 failed to allow the zero-repeat case if pcre2_match() was called with an
122 ovector too small to capture the group.
123
124 16. Improved error message in pcre2test when setting the stack size (-S) fails.
125
126 17. Fixed two bugs in CMakeLists.txt: (1) Some lines had got lost in the
127 transfer from PCRE1, meaning that CMake configuration failed if "build tests"
128 was selected. (2) The file src/pcre2_serialize.c had not been added to the list
129 of PCRE2 sources, which caused a failure to build pcre2test.
130
131 18. Fixed typo in pcre2_serialize.c (DECL instead of DEFN) that causes problems
132 only on Windows.
133
134 19. Use binary input when reading back saved serialized patterns in pcre2test.
135
136 20. Added RunTest.bat for running the tests under Windows.
137
138 21. "make distclean" was not removing config.h, a file that may be created for
139 use with CMake.
140
141 22. A pattern such as "((?2){0,1999}())?", which has a group containing a
142 forward reference repeated a large (but limited) number of times within a
143 repeated outer group that has a zero minimum quantifier, caused incorrect code
144 to be compiled, leading to the error "internal error: previously-checked
145 referenced subpattern not found" when an incorrect memory address was read.
146 This bug was reported as "heap overflow", discovered by Kai Lu of Fortinet's
147 FortiGuard Labs. (Added 24-March-2015: CVE-2015-2325 was given to this.)
148
149 23. A pattern such as "((?+1)(\1))/" containing a forward reference subroutine
150 call within a group that also contained a recursive back reference caused
151 incorrect code to be compiled. This bug was reported as "heap overflow",
152 discovered by Kai Lu of Fortinet's FortiGuard Labs. (Added 24-March-2015:
153 CVE-2015-2326 was given to this.)
154
155 24. Computing the size of the JIT read-only data in advance has been a source
156 of various issues, and new ones are still appear unfortunately. To fix
157 existing and future issues, size computation is eliminated from the code,
158 and replaced by on-demand memory allocation.
159
160 25. A pattern such as /(?i)[A-`]/, where characters in the other case are
161 adjacent to the end of the range, and the range contained characters with more
162 than one other case, caused incorrect behaviour when compiled in UTF mode. In
163 that example, the range a-j was left out of the class.
164
165
166 Version 10.00 05-January-2015
167 -----------------------------
168
169 Version 10.00 is the first release of PCRE2, a revised API for the PCRE
170 library. Changes prior to 10.00 are logged in the ChangeLog file for the old
171 API, up to item 20 for release 8.36.
172
173 The code of the library was heavily revised as part of the new API
174 implementation. Details of each and every modification were not individually
175 logged. In addition to the API changes, the following changes were made. They
176 are either new functionality, or bug fixes and other noticeable changes of
177 behaviour that were implemented after the code had been forked.
178
179 1. Including Unicode support at build time is now enabled by default, but it
180 can optionally be disabled. It is not enabled by default at run time (no
181 change).
182
183 2. The test program, now called pcre2test, was re-specified and almost
184 completely re-written. Its input is not compatible with input for pcretest.
185
186 3. Patterns may start with (*NOTEMPTY) or (*NOTEMPTY_ATSTART) to set the
187 PCRE2_NOTEMPTY or PCRE2_NOTEMPTY_ATSTART options for every subject line that is
188 matched by that pattern.
189
190 4. For the benefit of those who use PCRE2 via some other application, that is,
191 not writing the function calls themselves, it is possible to check the PCRE2
192 version by matching a pattern such as /(?(VERSION>=10)yes|no)/ against a
193 string such as "yesno".
194
195 5. There are case-equivalent Unicode characters whose encodings use different
196 numbers of code units in UTF-8. U+023A and U+2C65 are one example. (It is
197 theoretically possible for this to happen in UTF-16 too.) If a backreference to
198 a group containing one of these characters was greedily repeated, and during
199 the match a backtrack occurred, the subject might be backtracked by the wrong
200 number of code units. For example, if /^(\x{23a})\1*(.)/ is matched caselessly
201 (and in UTF-8 mode) against "\x{23a}\x{2c65}\x{2c65}\x{2c65}", group 2 should
202 capture the final character, which is the three bytes E2, B1, and A5 in UTF-8.
203 Incorrect backtracking meant that group 2 captured only the last two bytes.
204 This bug has been fixed; the new code is slower, but it is used only when the
205 strings matched by the repetition are not all the same length.
206
207 6. A pattern such as /()a/ was not setting the "first character must be 'a'"
208 information. This applied to any pattern with a group that matched no
209 characters, for example: /(?:(?=.)|(?<!x))a/.
210
211 7. When an (*ACCEPT) is triggered inside capturing parentheses, it arranges for
212 those parentheses to be closed with whatever has been captured so far. However,
213 it was failing to mark any other groups between the highest capture so far and
214 the currrent group as "unset". Thus, the ovector for those groups contained
215 whatever was previously there. An example is the pattern /(x)|((*ACCEPT))/ when
216 matched against "abcd".
217
218 8. The pcre2_substitute() function has been implemented.
219
220 9. If an assertion used as a condition was quantified with a minimum of zero
221 (an odd thing to do, but it happened), SIGSEGV or other misbehaviour could
222 occur.
223
224 10. The PCRE2_NO_DOTSTAR_ANCHOR option has been implemented.
225
226 ****

  ViewVC Help
Powered by ViewVC 1.1.5