/[pcre]/code/trunk/README
ViewVC logotype

Diff of /code/trunk/README

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 836 by ph10, Wed Dec 28 17:16:11 2011 UTC revision 936 by ph10, Sat Feb 25 17:02:23 2012 UTC
# Line 34  The contents of this README file are: Line 34  The contents of this README file are:
34  The PCRE APIs  The PCRE APIs
35  -------------  -------------
36    
37  PCRE is written in C, and it has its own API. The distribution also includes a  PCRE is written in C, and it has its own API. There are two sets of functions,
38  set of C++ wrapper functions (see the pcrecpp man page for details), courtesy  one for the 8-bit library, which processes strings of bytes, and one for the
39  of Google Inc.  16-bit library, which processes strings of 16-bit values. The distribution also
40    includes a set of C++ wrapper functions (see the pcrecpp man page for details),
41  In addition, there is a set of C wrapper functions that are based on the POSIX  courtesy of Google Inc., which can be used to call the 8-bit PCRE library from
42  regular expression API (see the pcreposix man page). These end up in the  C++.
43  library called libpcreposix. Note that this just provides a POSIX calling  
44  interface to PCRE; the regular expressions themselves still follow Perl syntax  In addition, there is a set of C wrapper functions (again, just for the 8-bit
45  and semantics. The POSIX API is restricted, and does not give full access to  library) that are based on the POSIX regular expression API (see the pcreposix
46  all of PCRE's facilities.  man page). These end up in the library called libpcreposix. Note that this just
47    provides a POSIX calling interface to PCRE; the regular expressions themselves
48    still follow Perl syntax and semantics. The POSIX API is restricted, and does
49    not give full access to all of PCRE's facilities.
50    
51  The header file for the POSIX-style functions is called pcreposix.h. The  The header file for the POSIX-style functions is called pcreposix.h. The
52  official POSIX name is regex.h, but I did not want to risk possible problems  official POSIX name is regex.h, but I did not want to risk possible problems
# Line 143  the usual methods of changing standard d Line 146  the usual methods of changing standard d
146    
147  CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local  CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local
148    
149  specifies that the C compiler should be run with the flags '-O2 -Wall' instead  This command specifies that the C compiler should be run with the flags '-O2
150  of the default, and that "make install" should install PCRE under /opt/local  -Wall' instead of the default, and that "make install" should install PCRE
151  instead of the default /usr/local.  under /opt/local instead of the default /usr/local.
152    
153  If you want to build in a different directory, just run "configure" with that  If you want to build in a different directory, just run "configure" with that
154  directory as current. For example, suppose you have unpacked the PCRE source  directory as current. For example, suppose you have unpacked the PCRE source
# Line 169  library. They are also documented in the Line 172  library. They are also documented in the
172    
173    (See also "Shared libraries on Unix-like systems" below.)    (See also "Shared libraries on Unix-like systems" below.)
174    
175  . If you want to suppress the building of the C++ wrapper library, you can add  . By default, only the 8-bit library is built. If you add --enable-pcre16 to
176    --disable-cpp to the "configure" command. Otherwise, when "configure" is run,    the "configure" command, the 16-bit library is also built. If you want only
177    it will try to find a C++ compiler and C++ header files, and if it succeeds,    the 16-bit library, use "./configure --enable-pcre16 --disable-pcre8".
178    it will try to build the C++ wrapper.  
179    . If you are building the 8-bit library and want to suppress the building of
180      the C++ wrapper library, you can add --disable-cpp to the "configure"
181      command. Otherwise, when "configure" is run without --disable-pcre8, it will
182      try to find a C++ compiler and C++ header files, and if it succeeds, it will
183      try to build the C++ wrapper.
184    
185  . If you want to include support for just-in-time compiling, which can give  . If you want to include support for just-in-time compiling, which can give
186    large performance improvements on certain platforms, add --enable-jit to the    large performance improvements on certain platforms, add --enable-jit to the
# Line 184  library. They are also documented in the Line 192  library. They are also documented in the
192    you add --disable-pcregrep-jit to the "configure" command.    you add --disable-pcregrep-jit to the "configure" command.
193    
194  . If you want to make use of the support for UTF-8 Unicode character strings in  . If you want to make use of the support for UTF-8 Unicode character strings in
195    PCRE, you must add --enable-utf8 to the "configure" command. Without it, the    the 8-bit library, or UTF-16 Unicode character strings in the 16-bit library,
196    code for handling UTF-8 is not included in the library. Even when included,    you must add --enable-utf to the "configure" command. Without it, the code
197    it still has to be enabled by an option at run time. When PCRE is compiled    for handling UTF-8 and UTF-16 is not included in the relevant library. Even
198    with this option, its input can only either be ASCII or UTF-8, even when    when --enable-utf is included, the use of a UTF encoding still has to be
199    running on EBCDIC platforms. It is not possible to use both --enable-utf8 and    enabled by an option at run time. When PCRE is compiled with this option, its
200    --enable-ebcdic at the same time.    input can only either be ASCII or UTF-8/16, even when running on EBCDIC
201      platforms. It is not possible to use both --enable-utf and --enable-ebcdic at
202  . If, in addition to support for UTF-8 character strings, you want to include    the same time.
203    support for the \P, \p, and \X sequences that recognize Unicode character  
204    properties, you must add --enable-unicode-properties to the "configure"  . There are no separate options for enabling UTF-8 and UTF-16 independently
205    command. This adds about 30K to the size of the library (in the form of a    because that would allow ridiculous settings such as requesting UTF-16
206    property table); only the basic two-letter properties such as Lu are    support while building only the 8-bit library. However, the option
207    supported.    --enable-utf8 is retained for backwards compatibility with earlier releases
208      that did not support 16-bit character strings. It is synonymous with
209      --enable-utf. It is not possible to configure one library with UTF support
210      and the other without in the same configuration.
211    
212    . If, in addition to support for UTF-8/16 character strings, you want to
213      include support for the \P, \p, and \X sequences that recognize Unicode
214      character properties, you must add --enable-unicode-properties to the
215      "configure" command. This adds about 30K to the size of the library (in the
216      form of a property table); only the basic two-letter properties such as Lu
217      are supported.
218    
219  . You can build PCRE to recognize either CR or LF or the sequence CRLF or any  . You can build PCRE to recognize either CR or LF or the sequence CRLF or any
220    of the preceding, or any of the Unicode newline sequences as indicating the    of the preceding, or any of the Unicode newline sequences as indicating the
# Line 249  library. They are also documented in the Line 267  library. They are also documented in the
267    sizes in the pcrestack man page.    sizes in the pcrestack man page.
268    
269  . The default maximum compiled pattern size is around 64K. You can increase  . The default maximum compiled pattern size is around 64K. You can increase
270    this by adding --with-link-size=3 to the "configure" command. You can    this by adding --with-link-size=3 to the "configure" command. In the 8-bit
271    increase it even more by setting --with-link-size=4, but this is unlikely    library, PCRE then uses three bytes instead of two for offsets to different
272    ever to be necessary. Increasing the internal link size will reduce    parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is
273    performance.    the same as --with-link-size=4, which (in both libraries) uses four-byte
274      offsets. Increasing the internal link size reduces performance.
275    
276  . You can build PCRE so that its internal match() function that is called from  . You can build PCRE so that its internal match() function that is called from
277    pcre_exec() does not call itself recursively. Instead, it uses memory blocks    pcre_exec() does not call itself recursively. Instead, it uses memory blocks
# Line 287  library. They are also documented in the Line 306  library. They are also documented in the
306    
307    This automatically implies --enable-rebuild-chartables (see above). However,    This automatically implies --enable-rebuild-chartables (see above). However,
308    when PCRE is built this way, it always operates in EBCDIC. It cannot support    when PCRE is built this way, it always operates in EBCDIC. It cannot support
309    both EBCDIC and UTF-8.    both EBCDIC and UTF-8/16.
310    
311  . It is possible to compile pcregrep to use libz and/or libbz2, in order to  . The pcregrep program currently supports only 8-bit data files, and so
312    read .gz and .bz2 files (respectively), by specifying one or both of    requires the 8-bit PCRE library. It is possible to compile pcregrep to use
313      libz and/or libbz2, in order to read .gz and .bz2 files (respectively), by
314      specifying one or both of
315    
316    --enable-pcregrep-libz    --enable-pcregrep-libz
317    --enable-pcregrep-libbz2    --enable-pcregrep-libbz2
# Line 305  library. They are also documented in the Line 326  library. They are also documented in the
326    The default value is 20K.    The default value is 20K.
327    
328  . It is possible to compile pcretest so that it links with the libreadline  . It is possible to compile pcretest so that it links with the libreadline
329    library, by specifying    or libedit libraries, by specifying, respectively,
330    
331    --enable-pcretest-libreadline    --enable-pcretest-libreadline or --enable-pcretest-libedit
332    
333    If this is done, when pcretest's input is from a terminal, it reads it using    If this is done, when pcretest's input is from a terminal, it reads it using
334    the readline() function. This provides line-editing and history facilities.    the readline() function. This provides line-editing and history facilities.
335    Note that libreadline is GPL-licenced, so if you distribute a binary of    Note that libreadline is GPL-licenced, so if you distribute a binary of
336    pcretest linked in this way, there may be licensing issues.    pcretest linked in this way, there may be licensing issues. These can be
337      avoided by linking with libedit instead.
338    
339    Setting this option causes the -lreadline option to be added to the pcretest    Enabling libreadline causes the -lreadline option to be added to the pcretest
340    build. In many operating environments with a sytem-installed readline    build. In many operating environments with a sytem-installed readline
341    library this is sufficient. However, in some environments (e.g. if an    library this is sufficient. However, in some environments (e.g. if an
342    unmodified distribution version of readline is in use), it may be necessary    unmodified distribution version of readline is in use), it may be necessary
# Line 333  The "configure" script builds the follow Line 355  The "configure" script builds the follow
355  . pcre-config          script that shows the building settings such as CFLAGS  . pcre-config          script that shows the building settings such as CFLAGS
356                           that were set for "configure"                           that were set for "configure"
357  . libpcre.pc         ) data for the pkg-config command  . libpcre.pc         ) data for the pkg-config command
358    . libpcre16.pc       )
359  . libpcreposix.pc    )  . libpcreposix.pc    )
360  . libtool              script that builds shared and/or static libraries  . libtool              script that builds shared and/or static libraries
361  . RunTest              script for running tests on the basic C library  . RunTest              script for running tests on the basic C library
# Line 343  names config.h.generic and pcre.h.generi Line 366  names config.h.generic and pcre.h.generi
366  have to built PCRE without using "configure" or CMake. If you use "configure"  have to built PCRE without using "configure" or CMake. If you use "configure"
367  or CMake, the .generic versions are not used.  or CMake, the .generic versions are not used.
368    
369  If a C++ compiler is found, the following files are also built:  When building the 8-bit library, if a C++ compiler is found, the following
370    files are also built:
371    
372  . libpcrecpp.pc        data for the pkg-config command  . libpcrecpp.pc        data for the pkg-config command
373  . pcrecpparg.h         header file for calling PCRE via the C++ wrapper  . pcrecpparg.h         header file for calling PCRE via the C++ wrapper
# Line 353  The "configure" script also creates conf Line 377  The "configure" script also creates conf
377  script that can be run to recreate the configuration, and config.log, which  script that can be run to recreate the configuration, and config.log, which
378  contains compiler output from tests that "configure" runs.  contains compiler output from tests that "configure" runs.
379    
380  Once "configure" has run, you can run "make". It builds two libraries, called  Once "configure" has run, you can run "make". This builds either or both of the
381  libpcre and libpcreposix, a test program called pcretest, and the pcregrep  libraries libpcre and libpcre16, and a test program called pcretest. If you
382  command. If a C++ compiler was found on your system, and you did not disable it  enabled JIT support with --enable-jit, a test program called pcre_jit_test is
383  with --disable-cpp, "make" also builds the C++ wrapper library, which is called  built as well.
384  libpcrecpp, and some test programs called pcrecpp_unittest,  
385  pcre_scanner_unittest, and pcre_stringpiece_unittest. If you enabled JIT  If the 8-bit library is built, libpcreposix and the pcregrep command are also
386  support with --enable-jit, a test program called pcre_jit_test is also built.  built, and if a C++ compiler was found on your system, and you did not disable
387    it with --disable-cpp, "make" builds the C++ wrapper library, which is called
388    libpcrecpp, as well as some test programs called pcrecpp_unittest,
389    pcre_scanner_unittest, and pcre_stringpiece_unittest.
390    
391  The command "make check" runs all the appropriate tests. Details of the PCRE  The command "make check" runs all the appropriate tests. Details of the PCRE
392  tests are given below in a separate section of this document.  tests are given below in a separate section of this document.
# Line 370  system. The following are installed (fil Line 397  system. The following are installed (fil
397    
398    Commands (bin):    Commands (bin):
399      pcretest      pcretest
400      pcregrep      pcregrep (if 8-bit support is enabled)
401      pcre-config      pcre-config
402    
403    Libraries (lib):    Libraries (lib):
404      libpcre      libpcre16     (if 16-bit support is enabled)
405      libpcreposix      libpcre       (if 8-bit support is enabled)
406      libpcrecpp (if C++ support is enabled)      libpcreposix  (if 8-bit support is enabled)
407        libpcrecpp    (if 8-bit and C++ support is enabled)
408    
409    Configuration information (lib/pkgconfig):    Configuration information (lib/pkgconfig):
410        libpcre16.pc
411      libpcre.pc      libpcre.pc
412      libpcreposix.pc      libpcreposix.pc
413      libpcrecpp.pc (if C++ support is enabled)      libpcrecpp.pc (if C++ support is enabled)
# Line 558  The RunTest script runs the pcretest tes Line 587  The RunTest script runs the pcretest tes
587  own man page) on each of the relevant testinput files in the testdata  own man page) on each of the relevant testinput files in the testdata
588  directory, and compares the output with the contents of the corresponding  directory, and compares the output with the contents of the corresponding
589  testoutput files. Some tests are relevant only when certain build-time options  testoutput files. Some tests are relevant only when certain build-time options
590  were selected. For example, the tests for UTF-8 support are run only if  were selected. For example, the tests for UTF-8/16 support are run only if
591  --enable-utf8 was used. RunTest outputs a comment when it skips a test.  --enable-utf was used. RunTest outputs a comment when it skips a test.
592    
593  Many of the tests that are not skipped are run up to three times. The second  Many of the tests that are not skipped are run up to three times. The second
594  run forces pcre_study() to be called for all patterns except for a few in some  run forces pcre_study() to be called for all patterns except for a few in some
# Line 567  tests that are marked "never study" (see Line 596  tests that are marked "never study" (see
596  done). If JIT support is available, the non-DFA tests are run a third time,  done). If JIT support is available, the non-DFA tests are run a third time,
597  this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option.  this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option.
598    
599  RunTest uses a file called testtry to hold the main output from pcretest  When both 8-bit and 16-bit support is enabled, the entire set of tests is run
600  (testsavedregex is also used as a working file). To run pcretest on just one of  twice, once for each library. If you want to run just one set of tests, call
601  the test files, give its number as an argument to RunTest, for example:  RunTest with either the -8 or -16 option.
602    
603    RunTest uses a file called testtry to hold the main output from pcretest.
604    Other files whose names begin with "test" are used as working files in some
605    tests. To run pcretest on just one or more specific test files, give their
606    numbers as arguments to RunTest, for example:
607    
608    RunTest 2    RunTest 2 7 11
609    
610  The first test file can be fed directly into the perltest.pl script to check  The first test file can be fed directly into the perltest.pl script to check
611  that Perl gives the same results. The only difference you should see is in the  that Perl gives the same results. The only difference you should see is in the
612  first few lines, where the Perl version is given instead of the PCRE version.  first few lines, where the Perl version is given instead of the PCRE version.
613    
614  The second set of tests check pcre_fullinfo(), pcre_info(), pcre_study(),  The second set of tests check pcre_fullinfo(), pcre_study(),
615  pcre_copy_substring(), pcre_get_substring(), pcre_get_substring_list(), error  pcre_copy_substring(), pcre_get_substring(), pcre_get_substring_list(), error
616  detection, and run-time flags that are specific to PCRE, as well as the POSIX  detection, and run-time flags that are specific to PCRE, as well as the POSIX
617  wrapper API. It also uses the debugging flags to check some of the internals of  wrapper API. It also uses the debugging flags to check some of the internals of
# Line 612  RunTest.bat. The version of RunTest.bat Line 646  RunTest.bat. The version of RunTest.bat
646  Windows versions of test 2. More info on using RunTest.bat is included in the  Windows versions of test 2. More info on using RunTest.bat is included in the
647  document entitled NON-UNIX-USE.]  document entitled NON-UNIX-USE.]
648    
649  The fourth test checks the UTF-8 support. This file can be also fed directly to  The fourth and fifth tests check the UTF-8/16 support and error handling and
650  the perltest.pl script, provided you are running Perl 5.8 or higher.  internal UTF features of PCRE that are not relevant to Perl, respectively. The
651    sixth and seventh tests do the same for Unicode character properties support.
652    
653    The eighth, ninth, and tenth tests check the pcre_dfa_exec() alternative
654    matching function, in non-UTF-8/16 mode, UTF-8/16 mode, and UTF-8/16 mode with
655    Unicode property support, respectively.
656    
657  The fifth test checks error handling with UTF-8 encoding, and internal UTF-8  The eleventh test checks some internal offsets and code size features; it is
658  features of PCRE that are not relevant to Perl.  run only when the default "link size" of 2 is set (in other cases the sizes
659    change) and when Unicode property support is enabled.
660    
661  The sixth test (which is Perl-5.10 compatible) checks the support for Unicode  The twelfth test is run only when JIT support is available, and the thirteenth
662  character properties. This file can be also fed directly to the perltest.pl  test is run only when JIT support is not available. They test some JIT-specific
663  script, provided you are running Perl 5.10 or higher.  features such as information output from pcretest about JIT compilation.
664    
665  The seventh, eighth, and ninth tests check the pcre_dfa_exec() alternative  The fourteenth, fifteenth, and sixteenth tests are run only in 8-bit mode, and
666  matching function, in non-UTF-8 mode, UTF-8 mode, and UTF-8 mode with Unicode  the seventeenth, eighteenth, and nineteenth tests are run only in 16-bit mode.
667  property support, respectively.  These are tests that generate different output in the two modes. They are for
668    general cases, UTF-8/16 support, and Unicode property support, respectively.
669    
670  The tenth test checks some internal offsets and code size features; it is run  The twentieth test is run only in 16-bit mode. It tests some specific 16-bit
671  only when the default "link size" of 2 is set (in other cases the sizes  features of the DFA matching engine.
 change) and when Unicode property support is enabled.  
672    
673  The eleventh and twelfth tests check out features that are new in Perl 5.10,  The twenty-first and twenty-second tests are run only in 16-bit mode, when the
674  without and with UTF-8 support, respectively. This file can be also fed  link size is set to 2. They test reloading pre-compiled patterns.
 directly to the perltest.pl script, provided you are running Perl 5.10 or  
 higher.  
   
 The thirteenth test checks a number internals and non-Perl features concerned  
 with Unicode property support.  
   
 The fourteenth test is run only when JIT support is available, and the  
 fifteenth test is run only when JIT support is not available. They test some  
 JIT-specific features such as information output from pcretest about JIT  
 compilation.  
675    
676    
677  Character tables  Character tables
# Line 701  will cause PCRE to malfunction. Line 731  will cause PCRE to malfunction.
731  File manifest  File manifest
732  -------------  -------------
733    
734  The distribution should contain the following files:  The distribution should contain the files listed below. Where a file name is
735    given as pcre[16]_xxx it means that there are two files, one with the name
736    pcre_xxx and the other with the name pcre16_xxx.
737    
738  (A) Source files of the PCRE library functions and their headers:  (A) Source files of the PCRE library functions and their headers:
739    
# Line 710  The distribution should contain the foll Line 742  The distribution should contain the foll
742    
743    pcre_chartables.c.dist  a default set of character tables that assume ASCII    pcre_chartables.c.dist  a default set of character tables that assume ASCII
744                              coding; used, unless --enable-rebuild-chartables is                              coding; used, unless --enable-rebuild-chartables is
745                              specified, by copying to pcre_chartables.c                              specified, by copying to pcre[16]_chartables.c
746    
747    pcreposix.c             )    pcreposix.c             )
748    pcre_byte_order.c       )    pcre[16]_byte_order.c   )
749    pcre_compile.c          )    pcre[16]_compile.c      )
750    pcre_config.c           )    pcre[16]_config.c       )
751    pcre_dfa_exec.c         )    pcre[16]_dfa_exec.c     )
752    pcre_exec.c             )    pcre[16]_exec.c         )
753    pcre_fullinfo.c         )    pcre[16]_fullinfo.c     )
754    pcre_get.c              ) sources for the functions in the library,    pcre[16]_get.c          ) sources for the functions in the library,
755    pcre_globals.c          )   and some internal functions that they use    pcre[16]_globals.c      )   and some internal functions that they use
756    pcre_info.c             )    pcre[16]_jit_compile.c  )
757    pcre_jit_compile.c      )    pcre[16]_maketables.c   )
758    pcre_maketables.c       )    pcre[16]_newline.c      )
759    pcre_newline.c          )    pcre[16]_refcount.c     )
760      pcre[16]_string_utils.c )
761      pcre[16]_study.c        )
762      pcre[16]_tables.c       )
763      pcre[16]_ucd.c          )
764      pcre[16]_version.c      )
765      pcre[16]_xclass.c       )
766    pcre_ord2utf8.c         )    pcre_ord2utf8.c         )
   pcre_refcount.c         )  
   pcre_study.c            )  
   pcre_tables.c           )  
   pcre_ucd.c              )  
767    pcre_valid_utf8.c       )    pcre_valid_utf8.c       )
768    pcre_version.c          )    pcre16_ord2utf16.c      )
769    pcre_xclass.c           )    pcre16_utf16_utils.c    )
770    pcre_printint.src       ) debugging function that is #included in pcretest,    pcre16_valid_utf16.c    )
771    
772      pcre[16]_printint.c     ) debugging function that is used by pcretest,
773                            )   and can also be #included in pcre_compile()                            )   and can also be #included in pcre_compile()
774    
775    pcre.h.in               template for pcre.h when built by "configure"    pcre.h.in               template for pcre.h when built by "configure"
776    pcreposix.h             header for the external POSIX wrapper API    pcreposix.h             header for the external POSIX wrapper API
777    pcre_internal.h         header for internal use    pcre_internal.h         header for internal use
# Line 796  The distribution should contain the foll Line 833  The distribution should contain the foll
833    doc/pcretest.txt        plain text documentation of test program    doc/pcretest.txt        plain text documentation of test program
834    doc/perltest.txt        plain text documentation of Perl test program    doc/perltest.txt        plain text documentation of Perl test program
835    install-sh              a shell script for installing files    install-sh              a shell script for installing files
836      libpcre16.pc.in         template for libpcre16.pc for pkg-config
837    libpcre.pc.in           template for libpcre.pc for pkg-config    libpcre.pc.in           template for libpcre.pc for pkg-config
838    libpcreposix.pc.in      template for libpcreposix.pc for pkg-config    libpcreposix.pc.in      template for libpcreposix.pc for pkg-config
839    libpcrecpp.pc.in        template for libpcrecpp.pc for pkg-config    libpcrecpp.pc.in        template for libpcrecpp.pc for pkg-config
# Line 812  The distribution should contain the foll Line 850  The distribution should contain the foll
850    testdata/testinput*     test data for main library tests    testdata/testinput*     test data for main library tests
851    testdata/testoutput*    expected test results    testdata/testoutput*    expected test results
852    testdata/grep*          input and output for pcregrep tests    testdata/grep*          input and output for pcregrep tests
853      testdata/*              other supporting test files
854    
855  (D) Auxiliary files for cmake support  (D) Auxiliary files for cmake support
856    
# Line 842  The distribution should contain the foll Line 881  The distribution should contain the foll
881  Philip Hazel  Philip Hazel
882  Email local part: ph10  Email local part: ph10
883  Email domain: cam.ac.uk  Email domain: cam.ac.uk
884  Last updated: 06 September 2011  Last updated: 25 February 2012

Legend:
Removed from v.836  
changed lines
  Added in v.936

  ViewVC Help
Powered by ViewVC 1.1.5