144 |
\fBpcrestack\fP |
\fBpcrestack\fP |
145 |
.\" |
.\" |
146 |
documentation. |
documentation. |
147 |
.sp |
. |
148 |
.\" HTML <a name="utf8support"></a> |
.\" HTML <a name="utf8support"></a> |
149 |
. |
. |
150 |
. |
. |
182 |
\ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported. |
\ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported. |
183 |
Furthermore, in Perl, many properties may optionally be prefixed by "Is", for |
Furthermore, in Perl, many properties may optionally be prefixed by "Is", for |
184 |
compatibility with Perl 5.6. PCRE does not support this. |
compatibility with Perl 5.6. PCRE does not support this. |
185 |
.P |
. |
186 |
The following comments apply when PCRE is running in UTF-8 mode: |
.\" HTML <a name="utf8strings"></a> |
187 |
.P |
. |
188 |
1. When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects |
.SS "Validity of UTF-8 strings" |
189 |
are checked for validity on entry to the relevant functions. Note that the |
.rs |
190 |
check is for a syntactically valid UTF-8 byte string, as defined by RFC 2279. |
.sp |
191 |
It is \fInot\fP a check for a UTF-8 string of assigned or allowable Unicode |
When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects |
192 |
code points. For example, the byte sequence \exED\exB2\ex94 is a valid UTF-8 |
are (by default) checked for validity on entry to the relevant functions. From |
193 |
encoding of the code point U+DC94, and is not rejected by PCRE. However, that |
release 7.3 of PCRE, the check is according the rules of RFC 3629, which are |
194 |
code point is in the "Low Surrogate Area" of Unicode, of which the Unicode |
themselves derived from the Unicode specification. Earlier releases of PCRE |
195 |
Standard says this: "The Low Surrogate Area does not contain any character |
followed the rules of RFC 2279, which allows the full range of 31-bit values (0 |
196 |
assignments, consequently no character code charts or namelists are provided |
to 0x7FFFFFFF). The current check allows only values in the range U+0 to |
197 |
for this area. Surrogates are reserved for use with UTF-16 and then must be |
U+10FFFF, excluding U+D800 to U+DFFF. |
198 |
used in pairs." |
.P |
199 |
.P |
The excluded code points are the "Low Surrogate Area" of Unicode, of which the |
200 |
The reason for the UTF-8 check at the start is so that the rest of PCRE can |
Unicode Standard says this: "The Low Surrogate Area does not contain any |
201 |
assume that UTF-8 strings are well formed. There is no intention of |
character assignments, consequently no character code charts or namelists are |
202 |
interpreting the values of the code points, which would involve more processing |
provided for this area. Surrogates are reserved for use with UTF-16 and then |
203 |
and affect performance. |
must be used in pairs." The code points that are encoded by UTF-16 pairs are |
204 |
.P |
available as independent code points in the UTF-8 encoding. (In other words, |
205 |
If a syntactically invalid UTF-8 string is passed, an error return is given. In |
the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up |
206 |
some situations, you may already know that your strings are valid, and |
UTF-8.) |
207 |
therefore want to skip these checks in order to improve performance. If you set |
.P |
208 |
the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that |
If an invalid UTF-8 string is passed to PCRE, an error return |
209 |
the pattern or subject it is given (respectively) contains only valid UTF-8 |
(PCRE_ERROR_BADUTF8) is given. In some situations, you may already know that |
210 |
codes. In this case, it does not diagnose an invalid UTF-8 string. If you pass |
your strings are valid, and therefore want to skip these checks in order to |
211 |
an invalid UTF-8 string to PCRE when PCRE_NO_UTF8_CHECK is set, the results are |
improve performance. If you set the PCRE_NO_UTF8_CHECK flag at compile time or |
212 |
undefined. Your program may crash. |
at run time, PCRE assumes that the pattern or subject it is given |
213 |
.P |
(respectively) contains only valid UTF-8 codes. In this case, it does not |
214 |
2. An unbraced hexadecimal escape sequence (such as \exb3) matches a two-byte |
diagnose an invalid UTF-8 string. |
215 |
|
.P |
216 |
|
If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what |
217 |
|
happens depends on why the string is invalid. If the string conforms to the |
218 |
|
"old" definition of UTF-8 (RFC 2279), it is processed as a string of characters |
219 |
|
in the range 0 to 0x7FFFFFFF. In other words, apart from the initial validity |
220 |
|
test, PCRE (when in UTF-8 mode) handles strings according to the more liberal |
221 |
|
rules of RFC 2279. However, if the string does not even conform to RFC 2279, |
222 |
|
the result is undefined. Your program may crash. |
223 |
|
.P |
224 |
|
If you want to process strings of values in the full range 0 to 0x7FFFFFFF, |
225 |
|
encoded in a UTF-8-like manner as per the old RFC, you can set |
226 |
|
PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this |
227 |
|
situation, you will have to apply your own validity check. |
228 |
|
. |
229 |
|
.SS "General comments about UTF-8 mode" |
230 |
|
.rs |
231 |
|
.sp |
232 |
|
1. An unbraced hexadecimal escape sequence (such as \exb3) matches a two-byte |
233 |
UTF-8 character if the value is greater than 127. |
UTF-8 character if the value is greater than 127. |
234 |
.P |
.P |
235 |
3. Octal numbers up to \e777 are recognized, and match two-byte UTF-8 |
2. Octal numbers up to \e777 are recognized, and match two-byte UTF-8 |
236 |
characters for values greater than \e177. |
characters for values greater than \e177. |
237 |
.P |
.P |
238 |
4. Repeat quantifiers apply to complete UTF-8 characters, not to individual |
3. Repeat quantifiers apply to complete UTF-8 characters, not to individual |
239 |
bytes, for example: \ex{100}{3}. |
bytes, for example: \ex{100}{3}. |
240 |
.P |
.P |
241 |
5. The dot metacharacter matches one UTF-8 character instead of a single byte. |
4. The dot metacharacter matches one UTF-8 character instead of a single byte. |
242 |
.P |
.P |
243 |
6. The escape sequence \eC can be used to match a single byte in UTF-8 mode, |
5. The escape sequence \eC can be used to match a single byte in UTF-8 mode, |
244 |
but its use can lead to some strange effects. This facility is not available in |
but its use can lead to some strange effects. This facility is not available in |
245 |
the alternative matching function, \fBpcre_dfa_exec()\fP. |
the alternative matching function, \fBpcre_dfa_exec()\fP. |
246 |
.P |
.P |
247 |
7. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly |
6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly |
248 |
test characters of any code value, but the characters that PCRE recognizes as |
test characters of any code value, but the characters that PCRE recognizes as |
249 |
digits, spaces, or word characters remain the same set as before, all with |
digits, spaces, or word characters remain the same set as before, all with |
250 |
values less than 256. This remains true even when PCRE includes Unicode |
values less than 256. This remains true even when PCRE includes Unicode |
252 |
cases. If you really want to test for a wider sense of, say, "digit", you |
cases. If you really want to test for a wider sense of, say, "digit", you |
253 |
must use Unicode property tests such as \ep{Nd}. |
must use Unicode property tests such as \ep{Nd}. |
254 |
.P |
.P |
255 |
8. Similarly, characters that match the POSIX named character classes are all |
7. Similarly, characters that match the POSIX named character classes are all |
256 |
low-valued characters. |
low-valued characters. |
257 |
.P |
.P |
258 |
9. However, the Perl 5.10 horizontal and vertical whitespace matching escapes |
8. However, the Perl 5.10 horizontal and vertical whitespace matching escapes |
259 |
(\eh, \eH, \ev, and \eV) do match all the appropriate Unicode characters. |
(\eh, \eH, \ev, and \eV) do match all the appropriate Unicode characters. |
260 |
.P |
.P |
261 |
10. Case-insensitive matching applies only to characters whose values are less |
9. Case-insensitive matching applies only to characters whose values are less |
262 |
than 128, unless PCRE is built with Unicode property support. Even when Unicode |
than 128, unless PCRE is built with Unicode property support. Even when Unicode |
263 |
property support is available, PCRE still uses its own character tables when |
property support is available, PCRE still uses its own character tables when |
264 |
checking the case of low-valued characters, so as not to degrade performance. |
checking the case of low-valued characters, so as not to degrade performance. |
287 |
.rs |
.rs |
288 |
.sp |
.sp |
289 |
.nf |
.nf |
290 |
Last updated: 07 August 2007 |
Last updated: 09 August 2007 |
291 |
Copyright (c) 1997-2007 University of Cambridge. |
Copyright (c) 1997-2007 University of Cambridge. |
292 |
.fi |
.fi |