1 |
<html>
|
2 |
<head>
|
3 |
<title>pcrejit specification</title>
|
4 |
</head>
|
5 |
<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
|
6 |
<h1>pcrejit man page</h1>
|
7 |
<p>
|
8 |
Return to the <a href="index.html">PCRE index page</a>.
|
9 |
</p>
|
10 |
<p>
|
11 |
This page is part of the PCRE HTML documentation. It was generated automatically
|
12 |
from the original man page. If there is any nonsense in it, please consult the
|
13 |
man page, in case the conversion went wrong.
|
14 |
<br>
|
15 |
<ul>
|
16 |
<li><a name="TOC1" href="#SEC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a>
|
17 |
<li><a name="TOC2" href="#SEC2">AVAILABILITY OF JIT SUPPORT</a>
|
18 |
<li><a name="TOC3" href="#SEC3">SIMPLE USE OF JIT</a>
|
19 |
<li><a name="TOC4" href="#SEC4">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a>
|
20 |
<li><a name="TOC5" href="#SEC5">RETURN VALUES FROM JIT EXECUTION</a>
|
21 |
<li><a name="TOC6" href="#SEC6">SAVING AND RESTORING COMPILED PATTERNS</a>
|
22 |
<li><a name="TOC7" href="#SEC7">CONTROLLING THE JIT STACK</a>
|
23 |
<li><a name="TOC8" href="#SEC8">EXAMPLE CODE</a>
|
24 |
<li><a name="TOC9" href="#SEC9">SEE ALSO</a>
|
25 |
<li><a name="TOC10" href="#SEC10">AUTHOR</a>
|
26 |
<li><a name="TOC11" href="#SEC11">REVISION</a>
|
27 |
</ul>
|
28 |
<br><a name="SEC1" href="#TOC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a><br>
|
29 |
<P>
|
30 |
Just-in-time compiling is a heavyweight optimization that can greatly speed up
|
31 |
pattern matching. However, it comes at the cost of extra processing before the
|
32 |
match is performed. Therefore, it is of most benefit when the same pattern is
|
33 |
going to be matched many times. This does not necessarily mean many calls of
|
34 |
\fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take
|
35 |
place many times at various positions in the subject, even for a single call to
|
36 |
<b>pcre_exec()</b>. If the subject string is very long, it may still pay to use
|
37 |
JIT for one-off matches.
|
38 |
</P>
|
39 |
<P>
|
40 |
JIT support applies only to the traditional matching function,
|
41 |
<b>pcre_exec()</b>. It does not apply when <b>pcre_dfa_exec()</b> is being used.
|
42 |
The code for this support was written by Zoltan Herczeg.
|
43 |
</P>
|
44 |
<br><a name="SEC2" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br>
|
45 |
<P>
|
46 |
JIT support is an optional feature of PCRE. The "configure" option --enable-jit
|
47 |
(or equivalent CMake option) must be set when PCRE is built if you want to use
|
48 |
JIT. The support is limited to the following hardware platforms:
|
49 |
<pre>
|
50 |
ARM v5, v7, and Thumb2
|
51 |
Intel x86 32-bit and 64-bit
|
52 |
MIPS 32-bit
|
53 |
Power PC 32-bit and 64-bit (experimental)
|
54 |
</pre>
|
55 |
The Power PC support is designated as experimental because it has not been
|
56 |
fully tested. If --enable-jit is set on an unsupported platform, compilation
|
57 |
fails.
|
58 |
</P>
|
59 |
<P>
|
60 |
A program can tell if JIT support is available by calling <b>pcre_config()</b>
|
61 |
with the PCRE_CONFIG_JIT option. The result is 1 when JIT is available, and 0
|
62 |
otherwise. However, a simple program does not need to check this in order to
|
63 |
use JIT. The API is implemented in a way that falls back to the ordinary PCRE
|
64 |
code if JIT is not available.
|
65 |
</P>
|
66 |
<br><a name="SEC3" href="#TOC1">SIMPLE USE OF JIT</a><br>
|
67 |
<P>
|
68 |
You have to do two things to make use of the JIT support in the simplest way:
|
69 |
<pre>
|
70 |
(1) Call <b>pcre_study()</b> with the PCRE_STUDY_JIT_COMPILE option for
|
71 |
each compiled pattern, and pass the resulting <b>pcre_extra</b> block to
|
72 |
<b>pcre_exec()</b>.
|
73 |
|
74 |
(2) Use <b>pcre_free_study()</b> to free the <b>pcre_extra</b> block when it is
|
75 |
no longer needed instead of just freeing it yourself. This
|
76 |
ensures that any JIT data is also freed.
|
77 |
</pre>
|
78 |
In some circumstances you may need to call additional functions. These are
|
79 |
described in the section entitled
|
80 |
<a href="#stackcontrol">"Controlling the JIT stack"</a>
|
81 |
below.
|
82 |
</P>
|
83 |
<P>
|
84 |
If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
|
85 |
data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
|
86 |
which turns it into machine code that executes much faster than the normal
|
87 |
interpretive code. When <b>pcre_exec()</b> is passed a <b>pcre_extra</b> block
|
88 |
containing a pointer to JIT code, it obeys that instead of the normal code. The
|
89 |
result is identical, but the code runs much faster.
|
90 |
</P>
|
91 |
<P>
|
92 |
There are some <b>pcre_exec()</b> options that are not supported for JIT
|
93 |
execution. There are also some pattern items that JIT cannot handle. Details
|
94 |
are given below. In both cases, execution automatically falls back to the
|
95 |
interpretive code.
|
96 |
</P>
|
97 |
<P>
|
98 |
If the JIT compiler finds an unsupported item, no JIT data is generated. You
|
99 |
can find out if JIT execution is available after studying a pattern by calling
|
100 |
<b>pcre_fullinfo()</b> with the PCRE_INFO_JIT option. A result of 1 means that
|
101 |
JIT compilation was successful. A result of 0 means that JIT support is not
|
102 |
available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the
|
103 |
JIT compiler was not able to handle the pattern.
|
104 |
</P>
|
105 |
<P>
|
106 |
Once a pattern has been studied, with or without JIT, it can be used as many
|
107 |
times as you like for matching different subject strings.
|
108 |
</P>
|
109 |
<br><a name="SEC4" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br>
|
110 |
<P>
|
111 |
The only <b>pcre_exec()</b> options that are supported for JIT execution are
|
112 |
PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
|
113 |
PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
|
114 |
supported.
|
115 |
</P>
|
116 |
<P>
|
117 |
The unsupported pattern items are:
|
118 |
<pre>
|
119 |
\C match a single byte; not supported in UTF-8 mode
|
120 |
(?Cn) callouts
|
121 |
(?(<name>)... conditional test on setting of a named subpattern
|
122 |
(?(R)... conditional test on whole pattern recursion
|
123 |
(?(Rn)... conditional test on recursion, by number
|
124 |
(?(R&name)... conditional test on recursion, by name
|
125 |
(*COMMIT) )
|
126 |
(*MARK) )
|
127 |
(*PRUNE) ) the backtracking control verbs
|
128 |
(*SKIP) )
|
129 |
(*THEN) )
|
130 |
</pre>
|
131 |
Support for some of these may be added in future.
|
132 |
</P>
|
133 |
<br><a name="SEC5" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br>
|
134 |
<P>
|
135 |
When a pattern is matched using JIT execution, the return values are the same
|
136 |
as those given by the interpretive <b>pcre_exec()</b> code, with the addition of
|
137 |
one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
|
138 |
for the JIT stack was insufficient. See
|
139 |
<a href="#stackcontrol">"Controlling the JIT stack"</a>
|
140 |
below for a discussion of JIT stack usage. For compatibility with the
|
141 |
interpretive <b>pcre_exec()</b> code, no more than two-thirds of the
|
142 |
<i>ovector</i> argument is used for passing back captured substrings.
|
143 |
</P>
|
144 |
<P>
|
145 |
The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
|
146 |
very large pattern tree goes on for too long, as it is in the same circumstance
|
147 |
when JIT is not used, but the details of exactly what is counted are not the
|
148 |
same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
|
149 |
execution.
|
150 |
</P>
|
151 |
<br><a name="SEC6" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br>
|
152 |
<P>
|
153 |
The code that is generated by the JIT compiler is architecture-specific, and is
|
154 |
also position dependent. For those reasons it cannot be saved (in a file or
|
155 |
database) and restored later like the bytecode and other data of a compiled
|
156 |
pattern. Saving and restoring compiled patterns is not something many people
|
157 |
do. More detail about this facility is given in the
|
158 |
<a href="pcreprecompile.html"><b>pcreprecompile</b></a>
|
159 |
documentation. It should be possible to run <b>pcre_study()</b> on a saved and
|
160 |
restored pattern, and thereby recreate the JIT data, but because JIT
|
161 |
compilation uses significant resources, it is probably not worth doing this;
|
162 |
you might as well recompile the original pattern.
|
163 |
<a name="stackcontrol"></a></P>
|
164 |
<br><a name="SEC7" href="#TOC1">CONTROLLING THE JIT STACK</a><br>
|
165 |
<P>
|
166 |
When the compiled JIT code runs, it needs a block of memory to use as a stack.
|
167 |
By default, it uses 32K on the machine stack. However, some large or
|
168 |
complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
|
169 |
is given when there is not enough stack. Three functions are provided for
|
170 |
managing blocks of memory for use as JIT stacks.
|
171 |
</P>
|
172 |
<P>
|
173 |
The <b>pcre_jit_stack_alloc()</b> function creates a JIT stack. Its arguments
|
174 |
are a starting size and a maximum size, and it returns a pointer to an opaque
|
175 |
structure of type <b>pcre_jit_stack</b>, or NULL if there is an error. The
|
176 |
<b>pcre_jit_stack_free()</b> function can be used to free a stack that is no
|
177 |
longer needed. (For the technically minded: the address space is allocated by
|
178 |
mmap or VirtualAlloc.)
|
179 |
</P>
|
180 |
<P>
|
181 |
JIT uses far less memory for recursion than the interpretive code,
|
182 |
and a maximum stack size of 512K to 1M should be more than enough for any
|
183 |
pattern.
|
184 |
</P>
|
185 |
<P>
|
186 |
The <b>pcre_assign_jit_stack()</b> function specifies which stack JIT code
|
187 |
should use. Its arguments are as follows:
|
188 |
<pre>
|
189 |
pcre_extra *extra
|
190 |
pcre_jit_callback callback
|
191 |
void *data
|
192 |
</pre>
|
193 |
The <i>extra</i> argument must be the result of studying a pattern with
|
194 |
PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
|
195 |
options:
|
196 |
<pre>
|
197 |
(1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32K block
|
198 |
on the machine stack is used.
|
199 |
|
200 |
(2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be
|
201 |
a valid JIT stack, the result of calling <b>pcre_jit_stack_alloc()</b>.
|
202 |
|
203 |
(3) If <i>callback</i> not NULL, it must point to a function that is called
|
204 |
with <i>data</i> as an argument at the start of matching, in order to
|
205 |
set up a JIT stack. If the result is NULL, the internal 32K stack
|
206 |
is used; otherwise the return value must be a valid JIT stack,
|
207 |
the result of calling <b>pcre_jit_stack_alloc()</b>.
|
208 |
</pre>
|
209 |
You may safely assign the same JIT stack to more than one pattern, as long as
|
210 |
they are all matched sequentially in the same thread. In a multithread
|
211 |
application, each thread must use its own JIT stack.
|
212 |
</P>
|
213 |
<P>
|
214 |
Strictly speaking, even more is allowed. You can assign the same stack to any
|
215 |
number of patterns as long as they are not used for matching by multiple
|
216 |
threads at the same time. For example, you can assign the same stack to all
|
217 |
compiled patterns, and use a global mutex in the callback to wait until the
|
218 |
stack is available for use. However, this is an inefficient solution, and
|
219 |
not recommended.
|
220 |
</P>
|
221 |
<P>
|
222 |
This is a suggestion for how a typical multithreaded program might operate:
|
223 |
<pre>
|
224 |
During thread initalization
|
225 |
thread_local_var = pcre_jit_stack_alloc(...)
|
226 |
|
227 |
During thread exit
|
228 |
pcre_jit_stack_free(thread_local_var)
|
229 |
|
230 |
Use a one-line callback function
|
231 |
return thread_local_var
|
232 |
</pre>
|
233 |
All the functions described in this section do nothing if JIT is not available,
|
234 |
and <b>pcre_assign_jit_stack()</b> does nothing unless the <b>extra</b> argument
|
235 |
is non-NULL and points to a <b>pcre_extra</b> block that is the result of a
|
236 |
successful study with PCRE_STUDY_JIT_COMPILE.
|
237 |
</P>
|
238 |
<br><a name="SEC8" href="#TOC1">EXAMPLE CODE</a><br>
|
239 |
<P>
|
240 |
This is a single-threaded example that specifies a JIT stack without using a
|
241 |
callback.
|
242 |
<pre>
|
243 |
int rc;
|
244 |
int ovector[30];
|
245 |
pcre *re;
|
246 |
pcre_extra *extra;
|
247 |
pcre_jit_stack *jit_stack;
|
248 |
|
249 |
re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
|
250 |
/* Check for errors */
|
251 |
extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
|
252 |
jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
|
253 |
/* Check for error (NULL) */
|
254 |
pcre_assign_jit_stack(extra, NULL, jit_stack);
|
255 |
rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
|
256 |
/* Check results */
|
257 |
pcre_free(re);
|
258 |
pcre_free_study(extra);
|
259 |
pcre_jit_stack_free(jit_stack);
|
260 |
|
261 |
</PRE>
|
262 |
</P>
|
263 |
<br><a name="SEC9" href="#TOC1">SEE ALSO</a><br>
|
264 |
<P>
|
265 |
<b>pcreapi</b>(3)
|
266 |
</P>
|
267 |
<br><a name="SEC10" href="#TOC1">AUTHOR</a><br>
|
268 |
<P>
|
269 |
Philip Hazel
|
270 |
<br>
|
271 |
University Computing Service
|
272 |
<br>
|
273 |
Cambridge CB2 3QH, England.
|
274 |
<br>
|
275 |
</P>
|
276 |
<br><a name="SEC11" href="#TOC1">REVISION</a><br>
|
277 |
<P>
|
278 |
Last updated: 19 October 2011
|
279 |
<br>
|
280 |
Copyright © 1997-2011 University of Cambridge.
|
281 |
<br>
|
282 |
<p>
|
283 |
Return to the <a href="index.html">PCRE index page</a>.
|
284 |
</p>
|