--- code/trunk/README 2007/02/24 21:38:01 3 +++ code/trunk/README 2007/02/24 21:38:49 27 @@ -1,16 +1,34 @@ README file for PCRE (Perl-compatible regular expressions) ---------------------------------------------------------- +******************************************************************************* +* IMPORTANT FOR THOSE UPGRADING FROM VERSIONS BEFORE 2.00 * +* * +* Please note that there has been a change in the API such that a larger * +* ovector is required at matching time, to provide some additional workspace. * +* The new man page has details. This change was necessary in order to support * +* some of the new functionality in Perl 5.005. * +* * +* IMPORTANT FOR THOSE UPGRADING FROM VERSION 2.00 * +* * +* Another (I hope this is the last!) change has been made to the API for the * +* pcre_compile() function. An additional argument has been added to make it * +* possible to pass over a pointer to character tables built in the current * +* locale by pcre_maketables(). To use the default tables, this new arguement * +* should be passed as NULL. * +******************************************************************************* + The distribution should contain the following files: ChangeLog log of changes to the code Makefile for building PCRE - Performance notes on performance README this file + RunTest a shell script for running tests Tech.Notes notes on the encoding pcre.3 man page for the functions pcreposix.3 man page for the POSIX wrapper API - maketables.c auxiliary program for building chartables.c + dftables.c auxiliary program for building chartables.c + maketables.c ) study.c ) source of pcre.c ) the functions pcreposix.c ) @@ -21,31 +39,51 @@ pgrep.1 man page for pgrep pgrep.c source of a grep utility that uses PCRE perltest Perl test program - testinput test data, compatible with Perl + testinput test data, compatible with Perl 5.004 and 5.005 testinput2 test data for error messages and non-Perl things + testinput3 test data, compatible with Perl 5.005 + testinput4 test data for locale-specific tests testoutput test results corresponding to testinput testoutput2 test results corresponding to testinput2 + testoutput3 test results corresponding to testinput3 + testoutput4 test results corresponding to testinput4 + +To build PCRE, edit Makefile for your system (it is a fairly simple make file, +and there are some comments at the top) and then run it. It builds two +libraries called libpcre.a and libpcreposix.a, a test program called pcretest, +and the pgrep command. + +To test PCRE, run the RunTest script in the pcre directory. This runs pcretest +on each of the testinput files in turn, and compares the output with the +contents of the corresponding testoutput file. A file called testtry is used to +hold the output from pcretest (which is documented below). + +To run pcretest on just one of the test files, give its number as an argument +to RunTest, for example: + + RunTest 3 + +The first and third test files can also be fed directly into the perltest +program to check that Perl gives the same results. The third file requires the +additional features of release 5.005, which is why it is kept separate from the +main test input, which needs only Perl 5.004. In the long run, when 5.005 is +widespread, these two test files may get amalgamated. + +The second set of tests check pcre_info(), pcre_study(), error detection and +run-time flags that are specific to PCRE, as well as the POSIX wrapper API. + +The fourth set of tests checks pcre_maketables(), the facility for building a +set of character tables for a specific locale and using them instead of the +default tables. The tests make use of the "fr" (French) locale. Before running +the test, the script checks for the presence of this locale by running the +"locale" command. If that command fails, or if it doesn't include "fr" in the +list of available locales, the fourth test cannot be run, and a comment is +output to say why. If running this test produces instances of the error -To build PCRE, edit Makefile for your system (it is a fairly simple make file) -and then run it. It builds a two libraries called libpcre.a and libpcreposix.a, -a test program called pcretest, and the pgrep command. - -To test PCRE, run pcretest on the file testinput, and compare the output with -the contents of testoutput. There should be no differences. For example: - - pcretest testinput /tmp/anything - diff /tmp/anything testoutput - -Do the same with testinput2, comparing the output with testoutput2, but this -time using the -i flag for pcretest, i.e. - - pcretest -i testinput2 /tmp/anything - diff /tmp/anything testoutput2 - -There are two sets of tests because the first set can also be fed directly into -the perltest program to check that Perl gives the same results. The second set -of tests check pcre_info(), pcre_study(), error detection and run-time flags -that are specific to PCRE, as well as the POSIX wrapper API. + ** Failed to set locale "fr" + +in the comparison output, it means that locale is not available on your system, +despite being listed by "locale". This does not mean that PCRE is broken. To install PCRE, copy libpcre.a to any suitable library directory (e.g. /usr/local/lib), pcre.h to any suitable include directory (e.g. @@ -63,29 +101,34 @@ for the POSIX-style functions is called pcreposix.h. The official POSIX name is regex.h, but I didn't want to risk possible problems with existing files of that name by distributing it that way. To use it with an existing program that -uses the POSIX API it will have to be renamed or pointed at by a link. +uses the POSIX API, it will have to be renamed or pointed at by a link. Character tables ---------------- -PCRE uses four tables for manipulating and identifying characters. These are -compiled from a source file called chartables.c. This is not supplied in -the distribution, but is built by the program maketables (compiled from -maketables.c), which uses the ANSI C character handling functions such as -isalnum(), isalpha(), isupper(), islower(), etc. to build the table sources. -This means that the default C locale set in your system may affect the contents -of the tables. You can change the tables by editing chartables.c and then -re-building PCRE. If you do this, you should probably also edit Makefile to -ensure that the file doesn't ever get re-generated. - -The first two tables pcre_lcc[] and pcre_fcc[] provide lower casing and a -case flipping functions, respectively. The pcre_cbits[] table consists of four -32-byte bit maps which identify digits, letters, "word" characters, and white -space, respectively. These are used when building 32-byte bit maps that -represent character classes. +PCRE uses four tables for manipulating and identifying characters. The final +argument of the pcre_compile() function is a pointer to a block of memory +containing the concatenated tables. A call to pcre_maketables() is used to +generate a set of tables in the current locale. However, if the final argument +is passed as NULL, a set of default tables that is built into the binary is +used. + +The source file called chartables.c contains the default set of tables. This is +not supplied in the distribution, but is built by the program dftables +(compiled from dftables.c), which uses the ANSI C character handling functions +such as isalnum(), isalpha(), isupper(), islower(), etc. to build the table +sources. This means that the default C locale set your system will control the +contents of the tables. You can change the default tables by editing +chartables.c and then re-building PCRE. If you do this, you should probably +also edit Makefile to ensure that the file doesn't ever get re-generated. + +The first two 256-byte tables provide lower casing and case flipping functions, +respectively. The next table consists of three 32-byte bit maps which identify +digits, "word" characters, and white space, respectively. These are used when +building 32-byte bit maps that represent character classes. -The pcre_ctypes[] table has bits indicating various character types, as +The final 256-byte table has bits indicating various character types, as follows: 1 white space character @@ -124,11 +167,28 @@ There are also some upper case options that do not match Perl options: /A, /E, and /X set PCRE_ANCHORED, PCRE_DOLLAR_ENDONLY, and PCRE_EXTRA respectively. -The /D option is a PCRE debugging feature. It causes the internal form of -compiled regular expressions to be output after compilation. The /S option -causes pcre_study() to be called after the expression has been compiled, and -the results used when the expression is matched. If /I is present as well as -/S, then pcre_study() is called with the PCRE_CASELESS option. + +The /L option must be followed directly by the name of a locale, for example, + + /pattern/Lfr + +For this reason, it must be the last option letter. The given locale is set, +pcre_maketables() is called to build a set of character tables for the locale, +and this is then passed to pcre_compile() when compiling the regular +expression. Without an /L option, NULL is passed as the tables pointer; that +is, /L applies only to the expression on which it appears. + +The /I option requests that pcretest output information about the compiled +expression (whether it is anchored, has a fixed first character, and so on). It +does this by calling pcre_info() after compiling an expression, and outputting +the information it gets back. If the pattern is studied, the results of that +are also output. + +The /D option is a PCRE debugging feature, which also assumes /I. It causes the +internal form of compiled regular expressions to be output after compilation. + +The /S option causes pcre_study() to be called after the expression has been +compiled, and the results used when the expression is matched. Finally, the /P option causes pcretest to call PCRE via the POSIX wrapper API rather than its native API. When this is done, all other options except /i and @@ -137,7 +197,7 @@ PCRE_DOTALL unless REG_NEWLINE is set. A regular expression can extend over several lines of input; the newlines are -included in it. See the testinput file for many examples. +included in it. See the testinput files for many examples. Before each data line is passed to pcre_exec(), leading and trailing whitespace is removed, and it is then scanned for \ escapes. The following are recognized: @@ -155,10 +215,6 @@ \A pass the PCRE_ANCHORED option to pcre_exec() \B pass the PCRE_NOTBOL option to pcre_exec() - \E pass the PCRE_DOLLAR_ENDONLY option to pcre_exec() - \I pass the PCRE_CASELESS option to pcre_exec() - \M pass the PCRE_MULTILINE option to pcre_exec() - \S pass the PCRE_DOTALL option to pcre_exec() \Odd set the size of the output vector passed to pcre_exec() to dd (any number of decimal digits) \Z pass the PCRE_NOTEOL option to pcre_exec() @@ -179,11 +235,11 @@ Testing Perl-Compatible Regular Expressions PCRE version 0.90 08-Sep-1997 - re> /^abc(\d+)/ - data> abc123 - 0: abc123 - 1: 123 - data> xyz + re> /^abc(\d+)/ + data> abc123 + 0: abc123 + 1: 123 + data> xyz No match Note that while patterns can be continued over several lines (a plain ">" @@ -197,17 +253,19 @@ If the option -d is given to pcretest, it is equivalent to adding /D to each regular expression: the internal form is output after compilation. -If the option -i (for "information") is given to pcretest, it calls pcre_info() -after compiling an expression, and outputs the information it gets back. If the -pattern is studied, the results of that are also output. +If the option -i is given to pcretest, it is equivalent to adding /I to each +regular expression: information about the compiled pattern is given after +compilation. If the option -s is given to pcretest, it outputs the size of each compiled pattern after it has been compiled. -If the -t option is given, each compile, study, and match is run 2000 times +If the -t option is given, each compile, study, and match is run 10000 times while being timed, and the resulting time per compile or match is output in milliseconds. Do not set -t with -s, because you will then get the size output -2000 times and the timing will be distorted. +10000 times and the timing will be distorted. If you want to change the number +of repetitions used for timing, edit the definition of LOOPREPEAT at the top of +pcretest.c @@ -216,7 +274,8 @@ The perltest program tests Perl's regular expressions; it has the same specification as pcretest, and so can be given identical input, except that -input patterns can be followed only by Perl's lower case options. +input patterns can be followed only by Perl's lower case options. The contents +of testinput and testinput3 meet this condition. The data lines are processed as Perl strings, so if they contain $ or @ characters, these have to be escaped. For this reason, all such characters in @@ -225,9 +284,11 @@ recognizes are not used in this file. The output should be identical, apart from the initial identifying banner. -The testinput2 file is not suitable for feeding to Perltest, since it does -make use of the special upper case options and escapes that pcretest uses to -test additional features of PCRE. +The testinput2 and testinput4 files are not suitable for feeding to Perltest, +since they do make use of the special upper case options and escapes that +pcretest uses to test some features of PCRE. The first of these files also +contains malformed regular expressions, in order to check that PCRE diagnoses +them correctly. Philip Hazel -October 1997 +January 1999