1 |
MAINTENANCE README FOR PCRE
|
2 |
---------------------------
|
3 |
|
4 |
The files in the "maint" directory of the PCRE source contain data, scripts,
|
5 |
and programs that are used for the maintenance of PCRE, but which do not form
|
6 |
part of the PCRE distribution tarballs. This document describes these files and
|
7 |
also contains some notes for maintainers. Its contents are:
|
8 |
|
9 |
Files in the maint directory
|
10 |
Updating to a new Unicode release
|
11 |
Preparing for a PCRE release
|
12 |
Making a PCRE release
|
13 |
Long-term ideas (wish list)
|
14 |
|
15 |
|
16 |
Files in the maint directory
|
17 |
----------------------------
|
18 |
|
19 |
----------------- This file is now OBSOLETE and no longer used ----------------
|
20 |
Builducptable A Perl script that creates the contents of the ucptable.h file
|
21 |
from two Unicode data files, which themselves are downloaded
|
22 |
from the Unicode web site. Run this script in the "maint"
|
23 |
directory.
|
24 |
----------------- This file is now OBSOLETE and no longer used ----------------
|
25 |
|
26 |
GenerateUtt.py A Python script to generate part of the pcre_tables.c file
|
27 |
that contains Unicode script names in a long string with
|
28 |
offsets, which is tedious to maintain by hand.
|
29 |
|
30 |
ManyConfigTests A shell script that runs "configure, make, test" a number of
|
31 |
times with different configuration settings.
|
32 |
|
33 |
MultiStage2.py A Python script that generates the file pcre_ucd.c from three
|
34 |
Unicode data tables, which are themselves downloaded from the
|
35 |
Unicode web site. Run this script in the "maint" directory.
|
36 |
The generated file contains the tables for a 2-stage lookup
|
37 |
of Unicode properties.
|
38 |
|
39 |
README This file.
|
40 |
|
41 |
Unicode.tables The files in this directory, DerivedGeneralCategory.txt,
|
42 |
Scripts.txt and UnicodeData.txt, were downloaded from the
|
43 |
Unicode web site. They contain information about Unicode
|
44 |
characters and scripts.
|
45 |
|
46 |
ucptest.c A short C program for testing the Unicode property macros
|
47 |
that do lookups in the pcre_ucd.c data, mainly useful after
|
48 |
rebuilding the Unicode property table. Compile and run this in
|
49 |
the "maint" directory (see comments at its head).
|
50 |
|
51 |
ucptestdata A directory containing two files, testinput1 and testoutput1,
|
52 |
to use in conjunction with the ucptest program.
|
53 |
|
54 |
utf8.c A short, freestanding C program for converting a Unicode code
|
55 |
point into a sequence of bytes in the UTF-8 encoding, and vice
|
56 |
versa. If its argument is a hex number such as 0x1234, it
|
57 |
outputs a list of the equivalent UTF-8 bytes. If its argument
|
58 |
is sequence of concatenated UTF-8 bytes (e.g. e188b4) it
|
59 |
treats them as a UTF-8 character and outputs the equivalent
|
60 |
code point in hex.
|
61 |
|
62 |
|
63 |
Updating to a new Unicode release
|
64 |
---------------------------------
|
65 |
|
66 |
When there is a new release of Unicode, the files in Unicode.tables must be
|
67 |
refreshed from the web site. If the new version of Unicode adds new character
|
68 |
scripts, the source file ucp.h and both the MultiStage2.py and the
|
69 |
GenerateUtt.py scripts must be edited to add the new names. Then MultiStage2.py
|
70 |
can be run to generate a new version of pcre_ucd.c, and GenerateUtt.py can be
|
71 |
run to generate the tricky tables for inclusion in pcre_tables.c.
|
72 |
|
73 |
If MultiStage2.py gives the error "ValueError: list.index(x): x not in list",
|
74 |
the cause is usually a missing (or misspelt) name in the list of scripts. I
|
75 |
couldn't find a straightforward list of scripts on the Unicode site, but
|
76 |
there's a useful Wikipedia page that list them, and notes the Unicode version
|
77 |
in which they were introduced:
|
78 |
|
79 |
http://en.wikipedia.org/wiki/Unicode_scripts#Table_of_Unicode_scripts
|
80 |
|
81 |
The ucptest program can be compiled and used to check that the new tables in
|
82 |
pcre_ucd.c work properly, using the data files in ucptestdata to check a number
|
83 |
of test characters. The source file ucptest.c must be updated whenever new
|
84 |
Unicode script names are added.
|
85 |
|
86 |
Note also that both the pcresyntax.3 and pcrepattern.3 man pages contain lists
|
87 |
of Unicode script names.
|
88 |
|
89 |
|
90 |
Preparing for a PCRE release
|
91 |
----------------------------
|
92 |
|
93 |
This section contains a checklist of things that I consult before building a
|
94 |
distribution for a new release.
|
95 |
|
96 |
. Ensure that the version number and version date are correct in configure.ac.
|
97 |
|
98 |
. If new build options have been added, ensure that they are added to the CMake
|
99 |
files as well as to the autoconf files.
|
100 |
|
101 |
. Run ./autogen.sh to ensure everything is up-to-date.
|
102 |
|
103 |
. Compile and test with many different config options, and combinations of
|
104 |
options. The maint/ManyConfigTests script now encapsulates this testing.
|
105 |
|
106 |
. Run perltest.pl on the test data for tests 1, 4, 6, and 11. The first two can
|
107 |
be run with Perl 5.8 or 5.10; the last two require Perl 5.10. The output
|
108 |
should match the PCRE test output, apart from the version identification at
|
109 |
the start of each test. The other tests are not Perl-compatible (they use
|
110 |
various PCRE-specific features or options).
|
111 |
|
112 |
. Test with valgrind by running "RunTest valgrind". There is also "RunGrepTest
|
113 |
valgrind", though that takes quite a long time.
|
114 |
|
115 |
. It may also useful to test with Electric Fence, though the fact that it
|
116 |
grumbles for missing free() calls can be a nuisance. (A missing free() in
|
117 |
pcretest is hardly a big problem.) To build with EF, use:
|
118 |
|
119 |
LIBS='/usr/lib/libefence.a -lpthread' with ./configure.
|
120 |
|
121 |
Then all normal runs use it to check for buffer overflow. Also run everything
|
122 |
with:
|
123 |
|
124 |
EF_PROTECT_BELOW=1 <whatever>
|
125 |
|
126 |
because there have been problems with lookbehinds that looked too far.
|
127 |
|
128 |
. Test with the emulated memmove() function by undefining HAVE_MEMMOVE and
|
129 |
HAVE_BCOPY in config.h. You may see a number of "pcre_memmove defined but not
|
130 |
used" warnings for the modules in which there is no call to memmove(). These
|
131 |
can be ignored.
|
132 |
|
133 |
. Documentation: check AUTHORS, COPYING, ChangeLog (check version and date),
|
134 |
INSTALL, LICENCE, NEWS (check version and date), NON-UNIX-USE, and README.
|
135 |
Many of these won't need changing, but over the long term things do change.
|
136 |
|
137 |
. Man pages: Check all man pages for \ not followed by e or f or " because
|
138 |
that indicates a markup error.
|
139 |
|
140 |
. When the release is built, test it on a number of different operating
|
141 |
systems if possible, and using different compilers as well. For example,
|
142 |
on Solaris it is helpful to test using Sun's cc compiler as a change from
|
143 |
gcc. Adding -xarch=v9 to the cc options does a 64-bit test, but it also
|
144 |
needs -S 64 for pcretest to increase the stack size for test 2.
|
145 |
|
146 |
|
147 |
Making a PCRE release
|
148 |
---------------------
|
149 |
|
150 |
Run PrepareRelease and commit the files that it changes (by removing trailing
|
151 |
spaces). Then run "make distcheck" to create the tarballs and the zipball.
|
152 |
Double-check with "svn status", then create an SVN tagged copy:
|
153 |
|
154 |
svn copy svn://vcs.exim.org/pcre/code/trunk \
|
155 |
svn://vcs.exim.org/pcre/code/tags/pcre-8.xx
|
156 |
|
157 |
Don't forget to update Freshmeat when the new release is out, and to tell
|
158 |
webmaster@pcre.org and the mailing list.
|
159 |
|
160 |
|
161 |
Future ideas (wish list)
|
162 |
------------------------
|
163 |
|
164 |
This section records a list of ideas so that they do not get forgotten. They
|
165 |
vary enormously in their usefulness and potential for implementation. Some are
|
166 |
very sensible; some are rather wacky. Some have been on this list for years;
|
167 |
others are relatively new.
|
168 |
|
169 |
. Optimization
|
170 |
|
171 |
There are always ideas for new optimizations so as to speed up pattern
|
172 |
matching. Most of them try to save work by recognizing a non-match without
|
173 |
having to scan all the possibilities. These are some that I've recorded:
|
174 |
|
175 |
* /((A{0,5}){0,5}){0,5}(something complex)/ on a non-matching string is very
|
176 |
slow, though Perl is fast. Can we speed up somehow? Convert to {0,125}?
|
177 |
OTOH, this is pathological - the user could easily fix it.
|
178 |
|
179 |
* Turn ={4} into ==== ? (for speed). I once did an experiment, and it seems
|
180 |
to have little effect, and maybe makes things worse.
|
181 |
|
182 |
* "Ends with literal string" - note that a single character doesn't gain much
|
183 |
over the existing "required byte" (reqbyte) feature that just remembers one
|
184 |
byte.
|
185 |
|
186 |
* These probably need to go in study():
|
187 |
|
188 |
o Remember an initial string rather than just 1 char?
|
189 |
|
190 |
o A required byte from alternatives - not just the last char, but an
|
191 |
earlier one if common to all alternatives.
|
192 |
|
193 |
o Friedl contains other ideas.
|
194 |
|
195 |
. If Perl gets to a consistent state over the settings of capturing sub-
|
196 |
patterns inside repeats, see if we can match it. One example of the
|
197 |
difference is the matching of /(main(O)?)+/ against mainOmain, where PCRE
|
198 |
leaves $2 set. In Perl, it's unset. Changing this in PCRE will be very hard
|
199 |
because I think it needs much more state to be remembered.
|
200 |
|
201 |
. Perl 6 will be a revolution. Is it a revolution too far for PCRE?
|
202 |
|
203 |
. Unicode
|
204 |
|
205 |
* Note that in Perl, \s matches \pZ and similarly for \d, \w and the POSIX
|
206 |
character classes. For the moment, I've chosen not to support this for
|
207 |
backward compatibility, for speed, and because it would be messy to
|
208 |
implement.
|
209 |
|
210 |
* A different approach to Unicode might be to use a typedef to do everything
|
211 |
in unsigned shorts instead of unsigned chars. Actually, we'd have to have a
|
212 |
new typedef to distinguish data from bits of compiled pattern that are in
|
213 |
bytes, I think. There would need to be conversion functions in and out. I
|
214 |
don't think this is particularly trivial - and anyway, Unicode now has
|
215 |
characters that need more than 16 bits, so is this at all sensible?
|
216 |
|
217 |
* There has been a request for direct support of 16-bit characters and
|
218 |
UTF-16. However, since Unicode is moving beyond purely 16-bit characters,
|
219 |
is this worth it at all? One possible way of handling 16-bit characters
|
220 |
would be to "load" them in the same way that UTF-8 characters are loaded.
|
221 |
|
222 |
. Allow errorptr and erroroffset to be NULL. I don't like this idea.
|
223 |
|
224 |
. Line endings:
|
225 |
|
226 |
* Option to use NUL as a line terminator in subject strings. This could now
|
227 |
be done relatively easily since the extension to support LF, CR, and CRLF.
|
228 |
If it is done, a suitable option for pcregrep is also required.
|
229 |
|
230 |
. Option to provide the pattern with a length instead of with a NUL terminator.
|
231 |
This affects quite a few places in the code and is not trivial.
|
232 |
|
233 |
. Catch SIGSEGV for stack overflows?
|
234 |
|
235 |
. A feature to suspend a match via a callout was once requested.
|
236 |
|
237 |
. Option to convert results into character offsets and character lengths.
|
238 |
|
239 |
. Option for pcregrep to scan only the start of a file. I am not keen - this is
|
240 |
the job of "head".
|
241 |
|
242 |
. A (non-Unix) user wanted pcregrep options to (a) list a file name just once,
|
243 |
preceded by a blank line, instead of adding it to every matched line, and (b)
|
244 |
support --outputfile=name.
|
245 |
|
246 |
. Consider making UTF-8 and UCP the default for PCRE n.0 for some n > 8.
|
247 |
|
248 |
. Add a user pointer to pcre_malloc/free functions -- some option would be
|
249 |
needed to retain backward compatibility.
|
250 |
|
251 |
. Define a union for the results from pcre_fullinfo().
|
252 |
|
253 |
. Provide a "random access to the subject" facility so that the way in which it
|
254 |
is stored is independent of PCRE. For efficiency, it probably isn't possible
|
255 |
to switch this dynamically. It would have to be specified when PCRE was
|
256 |
compiled. PCRE would then call a function every time it wanted a character.
|
257 |
|
258 |
. Wild thought: the ability to compile from PCRE's internal byte code to a real
|
259 |
FSM and a very fast (third) matcher to process the result. There would be
|
260 |
even more restrictions than for pcre_dfa_exec(), however. This is not easy.
|
261 |
|
262 |
. Should pcretest have some private locale data, to avoid relying on the
|
263 |
available locales for the test data, since different OS have different ideas?
|
264 |
This won't be as thorough a test, but perhaps that doesn't really matter.
|
265 |
|
266 |
. pcregrep: add -rs for a sorted recurse? Having to store file names and sort
|
267 |
them will of course slow it down.
|
268 |
|
269 |
. Someone suggested --disable-callout to save code space when callouts are
|
270 |
never wanted. This seems rather marginal.
|
271 |
|
272 |
. Check names that consist entirely of digits: PCRE allows, but do Perl and
|
273 |
Python, etc?
|
274 |
|
275 |
. A user suggested a parameter to limit the length of string matched, for
|
276 |
example if the parameter is N, the current match should fail if the matched
|
277 |
substring exceeds N. This could apply to both match functions. The value
|
278 |
could be a new field in the extra block.
|
279 |
|
280 |
. Callouts with arguments: (?Cn:ARG) for instance.
|
281 |
|
282 |
. A user is going to supply a patch to generalize the API for user-specific
|
283 |
memory allocation so that it is more flexible in threaded environments. Thiw
|
284 |
was promised a long time ago, and never appeared...
|
285 |
|
286 |
. Write a function that generates random matching strings for a compiled regex.
|
287 |
|
288 |
. Write a wrapper to maintain a structure with specified runtime parameters,
|
289 |
such as recurse limit, and pass these to PCRE each time it is called. Also
|
290 |
maybe malloc and free. A user sent a prototype.
|
291 |
|
292 |
. Pcregrep: an option to specify the output line separator, either as a string
|
293 |
or select from a fixed list. This is not dead easy, because at the moment it
|
294 |
outputs whatever is in the input file.
|
295 |
|
296 |
. Improve the code for duplicate checking in pcre_dfa_exec(). An incomplete,
|
297 |
non-thread-safe patch showed that this can help performance for patterns
|
298 |
where there are many alternatives. However, a simple thread-safe
|
299 |
implementation that I tried made things worse in many simple cases, so this
|
300 |
is not an obviously good thing.
|
301 |
|
302 |
. Make the longest lookbehind available via pcre_fullinfo(). This is not
|
303 |
straightforward because lookbehinds can be nested inside lookbehinds. This
|
304 |
case will have to be identified, and the amounts added. This should then give
|
305 |
the maximum possible lookbehind length. The reason for wanting this is to
|
306 |
help when implementing multi-segment matching using pcre_exec() with partial
|
307 |
matching and overlapping segments.
|
308 |
|
309 |
. PCRE cannot at present distinguish between subpatterns with different names,
|
310 |
but the same number (created by the use of ?|). In order to do so, a way of
|
311 |
remembering *which* subpattern numbered n matched is needed. Bugzilla #760.
|
312 |
|
313 |
Philip Hazel
|
314 |
Email local part: ph10
|
315 |
Email domain: cam.ac.uk
|
316 |
Last updated: 26 September 2009
|