| 1 |
MAINTENANCE README FOR PCRE
|
| 2 |
===========================
|
| 3 |
|
| 4 |
The files in the "maint" directory of the PCRE source contain data, scripts,
|
| 5 |
and programs that are used for the maintenance of PCRE, but which do not form
|
| 6 |
part of the PCRE distribution tarballs. This document describes these files and
|
| 7 |
also contains some notes for maintainers. Its contents are:
|
| 8 |
|
| 9 |
Files in the maint directory
|
| 10 |
Updating to a new Unicode release
|
| 11 |
Preparing for a PCRE release
|
| 12 |
Making a PCRE release
|
| 13 |
Long-term ideas (wish list)
|
| 14 |
|
| 15 |
|
| 16 |
Files in the maint directory
|
| 17 |
============================
|
| 18 |
|
| 19 |
GenerateUtt.py A Python script to generate part of the pcre_tables.c file
|
| 20 |
that contains Unicode script names in a long string with
|
| 21 |
offsets, which is tedious to maintain by hand.
|
| 22 |
|
| 23 |
ManyConfigTests A shell script that runs "configure, make, test" a number of
|
| 24 |
times with different configuration settings.
|
| 25 |
|
| 26 |
MultiStage2.py A Python script that generates the file pcre_ucd.c from three
|
| 27 |
Unicode data tables, which are themselves downloaded from the
|
| 28 |
Unicode web site. Run this script in the "maint" directory.
|
| 29 |
The generated file contains the tables for a 2-stage lookup
|
| 30 |
of Unicode properties.
|
| 31 |
|
| 32 |
pcre_chartables.c.non-standard
|
| 33 |
This is a set of character tables that came from a Windows
|
| 34 |
system. It has characters greater than 128 that are set as
|
| 35 |
spaces, amongst other things. I kept it so that it can be
|
| 36 |
used for testing from time to time.
|
| 37 |
|
| 38 |
README This file.
|
| 39 |
|
| 40 |
Unicode.tables The files in this directory (CaseFolding.txt,
|
| 41 |
DerivedGeneralCategory.txt, GraphemeBreakProperty.txt,
|
| 42 |
Scripts.txt and UnicodeData.txt) were downloaded from the
|
| 43 |
Unicode web site. They contain information about Unicode
|
| 44 |
characters and scripts.
|
| 45 |
|
| 46 |
ucptest.c A short C program for testing the Unicode property macros
|
| 47 |
that do lookups in the pcre_ucd.c data, mainly useful after
|
| 48 |
rebuilding the Unicode property table. Compile and run this in
|
| 49 |
the "maint" directory (see comments at its head).
|
| 50 |
|
| 51 |
ucptestdata A directory containing two files, testinput1 and testoutput1,
|
| 52 |
to use in conjunction with the ucptest program.
|
| 53 |
|
| 54 |
utf8.c A short, freestanding C program for converting a Unicode code
|
| 55 |
point into a sequence of bytes in the UTF-8 encoding, and vice
|
| 56 |
versa. If its argument is a hex number such as 0x1234, it
|
| 57 |
outputs a list of the equivalent UTF-8 bytes. If its argument
|
| 58 |
is sequence of concatenated UTF-8 bytes (e.g. e188b4) it
|
| 59 |
treats them as a UTF-8 character and outputs the equivalent
|
| 60 |
code point in hex.
|
| 61 |
|
| 62 |
|
| 63 |
Updating to a new Unicode release
|
| 64 |
=================================
|
| 65 |
|
| 66 |
When there is a new release of Unicode, the files in Unicode.tables must be
|
| 67 |
refreshed from the web site. If the new version of Unicode adds new character
|
| 68 |
scripts, the source file ucp.h and both the MultiStage2.py and the
|
| 69 |
GenerateUtt.py scripts must be edited to add the new names. Then MultiStage2.py
|
| 70 |
can be run to generate a new version of pcre_ucd.c, and GenerateUtt.py can be
|
| 71 |
run to generate the tricky tables for inclusion in pcre_tables.c.
|
| 72 |
|
| 73 |
If MultiStage2.py gives the error "ValueError: list.index(x): x not in list",
|
| 74 |
the cause is usually a missing (or misspelt) name in the list of scripts. I
|
| 75 |
couldn't find a straightforward list of scripts on the Unicode site, but
|
| 76 |
there's a useful Wikipedia page that list them, and notes the Unicode version
|
| 77 |
in which they were introduced:
|
| 78 |
|
| 79 |
http://en.wikipedia.org/wiki/Unicode_scripts#Table_of_Unicode_scripts
|
| 80 |
|
| 81 |
The ucptest program can be compiled and used to check that the new tables in
|
| 82 |
pcre_ucd.c work properly, using the data files in ucptestdata to check a number
|
| 83 |
of test characters. The source file ucptest.c must be updated whenever new
|
| 84 |
Unicode script names are added.
|
| 85 |
|
| 86 |
Note also that both the pcresyntax.3 and pcrepattern.3 man pages contain lists
|
| 87 |
of Unicode script names.
|
| 88 |
|
| 89 |
|
| 90 |
Preparing for a PCRE release
|
| 91 |
============================
|
| 92 |
|
| 93 |
This section contains a checklist of things that I consult before building a
|
| 94 |
distribution for a new release.
|
| 95 |
|
| 96 |
. Ensure that the version number and version date are correct in configure.ac.
|
| 97 |
|
| 98 |
. Update the library version numbers in configure.ac according to the rules
|
| 99 |
given below.
|
| 100 |
|
| 101 |
. If new build options have been added, ensure that they are added to the CMake
|
| 102 |
files as well as to the autoconf files. The relevant files are CMakeLists.txt
|
| 103 |
and config-cmake.h.in. After making a release tarball, test it out with CMake
|
| 104 |
if there have been changes here.
|
| 105 |
|
| 106 |
. Run ./autogen.sh to ensure everything is up-to-date.
|
| 107 |
|
| 108 |
. Compile and test with many different config options, and combinations of
|
| 109 |
options. Also, test with valgrind by running "RunTest valgrind" and
|
| 110 |
"RunGrepTest valgrind" (which takes quite a long time). The script
|
| 111 |
maint/ManyConfigTests now encapsulates this testing. It runs tests with
|
| 112 |
different configurations, and it also runs some of them with valgrind, all of
|
| 113 |
which can take quite some time.
|
| 114 |
|
| 115 |
. Run perltest.pl on the test data for tests 1, 4, and 6. The output
|
| 116 |
should match the PCRE test output, apart from the version identification at
|
| 117 |
the start of each test. The other tests are not Perl-compatible (they use
|
| 118 |
various PCRE-specific features or options).
|
| 119 |
|
| 120 |
. It is possible to test with the emulated memmove() function by undefining
|
| 121 |
HAVE_MEMMOVE and HAVE_BCOPY in config.h, though I do not do this often. You
|
| 122 |
may see a number of "pcre_memmove defined but not used" warnings for the
|
| 123 |
modules in which there is no call to memmove(). These can be ignored.
|
| 124 |
|
| 125 |
. Documentation: check AUTHORS, ChangeLog (check version and date), LICENCE,
|
| 126 |
NEWS (check version and date), NON-AUTOTOOLS-BUILD, and README. Many of these
|
| 127 |
won't need changing, but over the long term things do change.
|
| 128 |
|
| 129 |
. I used to test new releases myself on a number of different operating
|
| 130 |
systems, using different compilers as well. For example, on Solaris it is
|
| 131 |
helpful to test using Sun's cc compiler as a change from gcc. Adding
|
| 132 |
-xarch=v9 to the cc options does a 64-bit test, but it also needs -S 64 for
|
| 133 |
pcretest to increase the stack size for test 2. Since I retired I can no
|
| 134 |
longer do this, but instead I rely on putting out release candidates for
|
| 135 |
folks on the pcre-dev list to test.
|
| 136 |
|
| 137 |
|
| 138 |
Updating version info for libtool
|
| 139 |
=================================
|
| 140 |
|
| 141 |
This set of rules for updating library version information came from a web page
|
| 142 |
whose URL I have forgotten. The version information consists of three parts:
|
| 143 |
(current, revision, age).
|
| 144 |
|
| 145 |
1. Start with version information of 0:0:0 for each libtool library.
|
| 146 |
|
| 147 |
2. Update the version information only immediately before a public release of
|
| 148 |
your software. More frequent updates are unnecessary, and only guarantee
|
| 149 |
that the current interface number gets larger faster.
|
| 150 |
|
| 151 |
3. If the library source code has changed at all since the last update, then
|
| 152 |
increment revision; c:r:a becomes c:r+1:a.
|
| 153 |
|
| 154 |
4. If any interfaces have been added, removed, or changed since the last
|
| 155 |
update, increment current, and set revision to 0.
|
| 156 |
|
| 157 |
5. If any interfaces have been added since the last public release, then
|
| 158 |
increment age.
|
| 159 |
|
| 160 |
6. If any interfaces have been removed or changed since the last public
|
| 161 |
release, then set age to 0.
|
| 162 |
|
| 163 |
The following explanation may help in understanding the above rules a bit
|
| 164 |
better. Consider that there are three possible kinds of reaction from users to
|
| 165 |
changes in a shared library:
|
| 166 |
|
| 167 |
1. Programs using the previous version may use the new version as a drop-in
|
| 168 |
replacement, and programs using the new version can also work with the
|
| 169 |
previous one. In other words, no recompiling nor relinking is needed. In
|
| 170 |
this case, increment revision only, don't touch current or age.
|
| 171 |
|
| 172 |
2. Programs using the previous version may use the new version as a drop-in
|
| 173 |
replacement, but programs using the new version may use APIs not present in
|
| 174 |
the previous one. In other words, a program linking against the new version
|
| 175 |
may fail if linked against the old version at run time. In this case, set
|
| 176 |
revision to 0, increment current and age.
|
| 177 |
|
| 178 |
3. Programs may need to be changed, recompiled, relinked in order to use the
|
| 179 |
new version. Increment current, set revision and age to 0.
|
| 180 |
|
| 181 |
|
| 182 |
Making a PCRE release
|
| 183 |
=====================
|
| 184 |
|
| 185 |
Run PrepareRelease and commit the files that it changes (by removing trailing
|
| 186 |
spaces). The first thing this script does is to run CheckMan on the man pages;
|
| 187 |
if it finds any markup errors, it reports them and then aborts.
|
| 188 |
|
| 189 |
Once PrepareRelease has run clean, run "make distcheck" to create the tarballs
|
| 190 |
and the zipball. Double-check with "svn status", then create an SVN tagged
|
| 191 |
copy:
|
| 192 |
|
| 193 |
svn copy svn://vcs.exim.org/pcre/code/trunk \
|
| 194 |
svn://vcs.exim.org/pcre/code/tags/pcre-8.xx
|
| 195 |
|
| 196 |
Don't forget to update Freshmeat when the new release is out, and to tell
|
| 197 |
webmaster@pcre.org and the mailing list. Also, update the list of version
|
| 198 |
numbers in Bugzilla (edit products).
|
| 199 |
|
| 200 |
|
| 201 |
Future ideas (wish list)
|
| 202 |
========================
|
| 203 |
|
| 204 |
This section records a list of ideas so that they do not get forgotten. They
|
| 205 |
vary enormously in their usefulness and potential for implementation. Some are
|
| 206 |
very sensible; some are rather wacky. Some have been on this list for years;
|
| 207 |
others are relatively new.
|
| 208 |
|
| 209 |
. Optimization
|
| 210 |
|
| 211 |
There are always ideas for new optimizations so as to speed up pattern
|
| 212 |
matching. Most of them try to save work by recognizing a non-match without
|
| 213 |
having to scan all the possibilities. These are some that I've recorded:
|
| 214 |
|
| 215 |
* /((A{0,5}){0,5}){0,5}(something complex)/ on a non-matching string is very
|
| 216 |
slow, though Perl is fast. Can we speed up somehow? Convert to {0,125}?
|
| 217 |
OTOH, this is pathological - the user could easily fix it.
|
| 218 |
|
| 219 |
* Turn ={4} into ==== ? (for speed). I once did an experiment, and it seems
|
| 220 |
to have little effect, and maybe makes things worse.
|
| 221 |
|
| 222 |
* "Ends with literal string" - note that a single character doesn't gain much
|
| 223 |
over the existing "required byte" (reqbyte) feature that just remembers one
|
| 224 |
data unit.
|
| 225 |
|
| 226 |
* These probably need to go in pcre_study():
|
| 227 |
|
| 228 |
o Remember an initial string rather than just 1 char?
|
| 229 |
|
| 230 |
o A required data unit from alternatives - not just the last unit, but an
|
| 231 |
earlier one if common to all alternatives.
|
| 232 |
|
| 233 |
o Friedl contains other ideas.
|
| 234 |
|
| 235 |
* pcre_study() does not set initial byte flags for Unicode property types
|
| 236 |
such as \p; I don't know how much benefit there would be for, for example,
|
| 237 |
setting the bits for 0-9 and all bytes >= xC0 when a pattern starts with
|
| 238 |
\p{N}.
|
| 239 |
|
| 240 |
* There is scope for more "auto-possessifying" in connection with \p and \P.
|
| 241 |
|
| 242 |
. If Perl gets to a consistent state over the settings of capturing sub-
|
| 243 |
patterns inside repeats, see if we can match it. One example of the
|
| 244 |
difference is the matching of /(main(O)?)+/ against mainOmain, where PCRE
|
| 245 |
leaves $2 set. In Perl, it's unset. Changing this in PCRE will be very hard
|
| 246 |
because I think it needs much more state to be remembered.
|
| 247 |
|
| 248 |
. Perl 6 will be a revolution. Is it a revolution too far for PCRE?
|
| 249 |
|
| 250 |
. Allow errorptr and erroroffset to be NULL. I don't like this idea.
|
| 251 |
|
| 252 |
. Line endings:
|
| 253 |
|
| 254 |
* Option to use NUL as a line terminator in subject strings. This could now
|
| 255 |
be done relatively easily since the extension to support LF, CR, and CRLF.
|
| 256 |
If it is done, a suitable option for pcregrep is also required.
|
| 257 |
|
| 258 |
. Option to provide the pattern with a length instead of with a NUL terminator.
|
| 259 |
This affects quite a few places in the code and is not trivial.
|
| 260 |
|
| 261 |
. Catch SIGSEGV for stack overflows?
|
| 262 |
|
| 263 |
. A feature to suspend a match via a callout was once requested.
|
| 264 |
|
| 265 |
. Option to convert results into character offsets and character lengths.
|
| 266 |
|
| 267 |
. Option for pcregrep to scan only the start of a file. I am not keen - this is
|
| 268 |
the job of "head".
|
| 269 |
|
| 270 |
. A (non-Unix) user wanted pcregrep options to (a) list a file name just once,
|
| 271 |
preceded by a blank line, instead of adding it to every matched line, and (b)
|
| 272 |
support --outputfile=name.
|
| 273 |
|
| 274 |
. Consider making UTF-8 and UCP the default for PCRE n.0 for some n > 8.
|
| 275 |
(And now presumably UTF-16 and UCP for the 16-bit library, and UTF-32 and UCP
|
| 276 |
for the 32-bit library.)
|
| 277 |
|
| 278 |
. Add a user pointer to pcre_malloc/free functions -- some option would be
|
| 279 |
needed to retain backward compatibility.
|
| 280 |
|
| 281 |
. Define a union for the results from pcre_fullinfo().
|
| 282 |
|
| 283 |
. Provide a "random access to the subject" facility so that the way in which it
|
| 284 |
is stored is independent of PCRE. For efficiency, it probably isn't possible
|
| 285 |
to switch this dynamically. It would have to be specified when PCRE was
|
| 286 |
compiled. PCRE would then call a function every time it wanted a character.
|
| 287 |
|
| 288 |
. Wild thought: the ability to compile from PCRE's internal byte code to a real
|
| 289 |
FSM and a very fast (third) matcher to process the result. There would be
|
| 290 |
even more restrictions than for pcre_dfa_exec(), however. This is not easy.
|
| 291 |
This is probably obsolete now that we have the JIT support.
|
| 292 |
|
| 293 |
. Should pcretest have some private locale data, to avoid relying on the
|
| 294 |
available locales for the test data, since different OS have different ideas?
|
| 295 |
This won't be as thorough a test, but perhaps that doesn't really matter.
|
| 296 |
|
| 297 |
. pcregrep: add -rs for a sorted recurse? Having to store file names and sort
|
| 298 |
them will of course slow it down.
|
| 299 |
|
| 300 |
. Someone suggested --disable-callout to save code space when callouts are
|
| 301 |
never wanted. This seems rather marginal.
|
| 302 |
|
| 303 |
. Check names that consist entirely of digits: PCRE allows, but do Perl and
|
| 304 |
Python, etc?
|
| 305 |
|
| 306 |
. A user suggested a parameter to limit the length of string matched, for
|
| 307 |
example if the parameter is N, the current match should fail if the matched
|
| 308 |
substring exceeds N. This could apply to both match functions. The value
|
| 309 |
could be a new field in the extra block.
|
| 310 |
|
| 311 |
. Callouts with arguments: (?Cn:ARG) for instance.
|
| 312 |
|
| 313 |
. A user is going to supply a patch to generalize the API for user-specific
|
| 314 |
memory allocation so that it is more flexible in threaded environments. This
|
| 315 |
was promised a long time ago, and never appeared. However, this is a live
|
| 316 |
issue not only for threaded environments, but for libraries that use PCRE and
|
| 317 |
want not to be beholden to their caller's memory allocation.
|
| 318 |
|
| 319 |
. Write a wrapper to maintain a structure with specified runtime parameters,
|
| 320 |
such as recurse limit, and pass these to PCRE each time it is called. Also
|
| 321 |
maybe malloc and free. A user sent a prototype. This relates the the previous
|
| 322 |
item.
|
| 323 |
|
| 324 |
. Write a function that generates random matching strings for a compiled regex.
|
| 325 |
|
| 326 |
. Pcregrep: an option to specify the output line separator, either as a string
|
| 327 |
or select from a fixed list. This is not dead easy, because at the moment it
|
| 328 |
outputs whatever is in the input file.
|
| 329 |
|
| 330 |
. Improve the code for duplicate checking in pcre_dfa_exec(). An incomplete,
|
| 331 |
non-thread-safe patch showed that this can help performance for patterns
|
| 332 |
where there are many alternatives. However, a simple thread-safe
|
| 333 |
implementation that I tried made things worse in many simple cases, so this
|
| 334 |
is not an obviously good thing.
|
| 335 |
|
| 336 |
. PCRE cannot at present distinguish between subpatterns with different names,
|
| 337 |
but the same number (created by the use of ?|). In order to do so, a way of
|
| 338 |
remembering *which* subpattern numbered n matched is needed. Bugzilla #760.
|
| 339 |
Now that (*MARK) has been implemented, it can perhaps be used as a way round
|
| 340 |
this problem.
|
| 341 |
|
| 342 |
. Instead of having #ifdef HAVE_CONFIG_H in each module, put #include
|
| 343 |
"something" and the the #ifdef appears only in one place, in "something".
|
| 344 |
|
| 345 |
Philip Hazel
|
| 346 |
Email local part: ph10
|
| 347 |
Email domain: cam.ac.uk
|
| 348 |
Last updated: 07 December 2012
|
Parent Directory
| 
Revision Log