perlunicook - cookbookish examples of handling Unicode in Perl
This manpage contains short recipes demonstrating how to handle common Unicode operations in Perl, plus one complete program at the end. Any undeclared variables in individual recipes are assumed to have a previous appropriate value in them.
Unless otherwise notes, all examples below require this standard preamble
to work correctly, with the #!
adjusted to work on your system:
- #!/usr/bin/env perl
- use utf8; # so literals and identifiers can be in UTF-8
- use v5.12; # or later to get "unicode_strings" feature
- use strict; # quote strings, declare variables
- use warnings; # on by default
- use warnings qw(FATAL utf8); # fatalize encoding glitches
- use open qw(:std :utf8); # undeclared streams in UTF-8
- use charnames qw(:full :short); # unneeded in v5.16
This does make even Unix programmers binmode
your binary streams,
or open them with :raw
, but that's the only way to get at them
portably anyway.
WARNING: use autodie
(pre 2.26) and use open
do not get along with each
other.
Always decompose on the way in, then recompose on the way out.
As of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings.
Without the all-critical use utf8
declaration, putting UTF‑8 in your
literals and identifiers won’t work right. If you used the standard
preamble just given above, this already happened. If you did, you can
do things like this:
If you forget use utf8
, high bytes will be misunderstood as
separate characters, and nothing will work right.
The ord
and chr
functions work transparently on all codepoints,
not just on ASCII alone — nor in fact, not even just on Unicode alone.
In an interpolated literal, whether a double-quoted string or a
regex, you may specify a character by its number using the
\x{HHHHHH}
escape.
- String: "\x{3a3}"
- Regex: /\x{3a3}/
- String: "\x{1d45b}"
- Regex: /\x{1d45b}/
- # even non-BMP ranges in regex work fine
- /[\x{1D434}-\x{1D467}]/
Use the \N{charname}
notation to get the character
by that name for use in interpolated literals (double-quoted
strings and regexes). In v5.16, there is an implicit
- use charnames qw(:full :short);
But prior to v5.16, you must be explicit about which set of charnames you
want. The :full
names are the official Unicode character name, alias, or
sequence, which all share a namespace.
- use charnames qw(:full :short latin greek);
- "\N{MATHEMATICAL ITALIC SMALL N}" # :full
- "\N{GREEK CAPITAL LETTER SIGMA}" # :full
Anything else is a Perl-specific convenience abbreviation. Specify one or more scripts by names if you want short names that are script-specific.
- "\N{Greek:Sigma}" # :short
- "\N{ae}" # latin
- "\N{epsilon}" # greek
The v5.16 release also supports a :loose
import for loose matching of
character names, which works just like loose matching of property names:
that is, it disregards case, whitespace, and underscores:
- "\N{euro sign}" # :loose (from v5.16)
These look just like character names but return multiple codepoints.
Notice the %vx
vector-print functionality in printf
.
Use :alias
to give your own lexically scoped nicknames to existing
characters, or even to give unnamed private-use characters useful names.
- use charnames ":full", ":alias" => {
- ecute => "LATIN SMALL LETTER E WITH ACUTE",
- "APPLE LOGO" => 0xF8FF, # private use character
- };
- "\N{ecute}"
- "\N{APPLE LOGO}"
Sinograms like “東京” come back with character names of
CJK UNIFIED IDEOGRAPH-6771
and CJK UNIFIED IDEOGRAPH-4EAC
,
because their “names” vary. The CPAN Unicode::Unihan
module
has a large database for decoding these (and a whole lot more), provided you
know how to understand its output.
prints:
- CJK 東京 in Mandarin is DONG1JING1
- CJK 東京 in Cantonese is dung1ging1
- CJK 東京 in Korean is TONGKYENG
- CJK 東京 in JapaneseOn is TOUKYOU KEI KIN
- CJK 東京 in JapaneseKun is HIGASHI AZUMAMIYAKO
If you have a specific romanization scheme in mind, use the specific module:
prints
- Japanese for 東京 is toukyou
On rare occasion, such as a database read, you may be given encoded text you need to decode.
For streams all in the same encoding, don't use encode/decode; instead
set the file encoding when you open the file or immediately after with
binmode
as described later below.
Use a command-line option, an environment variable, or else
call binmode
explicitly:
Files opened without an encoding argument will be in UTF-8:
Specify stream encoding. This is the normal way to deal with encoded text, not by calling low-level functions.
- # input file
- open(my $in_file, "< :encoding(UTF-16)", "wintext");
- OR
- open(my $in_file, "<", "wintext");
- binmode($in_file, ":encoding(UTF-16)");
- THEN
- my $line = <$in_file>;
- # output file
- open($out_file, "> :encoding(cp1252)", "wintext");
- OR
- open(my $out_file, ">", "wintext");
- binmode($out_file, ":encoding(cp1252)");
- THEN
- print $out_file "some text\n";
More layers than just the encoding can be specified here. For example,
the incantation ":raw :encoding(UTF-16LE) :crlf"
includes implicit
CRLF handling.
Unicode casing is very different from ASCII casing.
Also available in the CPAN Unicode::CaseFold module,
the new fc
“foldcase” function from v5.16 grants
access to the same Unicode casefolding as the /i
pattern modifier has always used:
A Unicode linebreak matches the two-character CRLF grapheme or any of seven vertical whitespace characters. Good for dealing with textfiles coming from different operating systems.
- \R
- s/\R/\n/g; # normalize all linebreaks to \n
Find the general category of a numeric codepoint.
Disable \w
, \b
, \s
, \d
, and the POSIX
classes from working correctly on Unicode either in this
scope, or in just one regex.
Or use specific un-Unicode properties, like \p{ahex}
and \p{POSIX_Digit
}. Properties still work normally
no matter what charset modifiers (/d /u /l /a /aa
)
should be effect.
These all match a single codepoint with the given
property. Use \P
in place of \p
to match
one codepoint lacking that property.
- \pL, \pN, \pS, \pP, \pM, \pZ, \pC
- \p{Sk}, \p{Ps}, \p{Lt}
- \p{alpha}, \p{upper}, \p{lower}
- \p{Latin}, \p{Greek}
- \p{script=Latin}, \p{script=Greek}
- \p{East_Asian_Width=Wide}, \p{EA=W}
- \p{Line_Break=Hyphen}, \p{LB=HY}
- \p{Numeric_Value=4}, \p{NV=4}
Define at compile-time your own custom character properties for use in regexes.
- # using private-use characters
- sub In_Tengwar { "E000\tE07F\n" }
- if (/\p{In_Tengwar}/) { ... }
- # blending existing properties
- sub Is_GraecoRoman_Title {<<'END_OF_SET'}
- +utf8::IsLatin
- +utf8::IsGreek
- &utf8::IsTitle
- END_OF_SET
- if (/\p{Is_GraecoRoman_Title}/ { ... }
Typically render into NFD on input and NFC on output. Using NFKC or NFKD functions improves recall on searches, assuming you've already done to the same text to be searched. Note that this is about much more than just pre- combined compatibility glyphs; it also reorders marks according to their canonical combining classes and weeds out singletons.
Unless you’ve used /a
or /aa
, \d
matches more than
ASCII digits only, but Perl’s implicit string-to-number
conversion does not current recognize these. Here’s how to
convert such strings manually.
- use v5.14; # needed for num() function
- use Unicode::UCD qw(num);
- my $str = "got Ⅻ and ४५६७ and ⅞ and here";
- my @nums = ();
- while ($str =~ /(\d+|\N)/g) { # not just ASCII!
- push @nums, num($1);
- }
- say "@nums"; # 12 4567 0.875
- use charnames qw(:full);
- my $nv = num("\N{RUMI DIGIT ONE}\N{RUMI DIGIT TWO}");
Programmer-visible “characters” are codepoints matched by /./s
,
but user-visible “characters” are graphemes matched by /\X/
.
- # Find vowel *plus* any combining diacritics,underlining,etc.
- my $nfd = NFD($orig);
- $nfd =~ / (?=[aeiou]) \X /xi
- # match and grab five first graphemes
- my($first_five) = $str =~ /^ ( \X{5} ) /x;
Reversing by codepoint messes up diacritics, mistakenly converting
crème brûlée
into éel̂urb em̀erc
instead of into eélûrb emèrc
;
so reverse by grapheme instead. Both these approaches work
right no matter what normalization the string is in:
The string brûlée
has six graphemes but up to eight codepoints.
This counts by grapheme, not by codepoint:
Perl’s printf
, sprintf
, and format
think all
codepoints take up 1 print column, but many take 0 or 2.
Here to show that normalization makes no difference,
we print out both forms:
generates this to show that it pads correctly no matter the normalization:
- crème |
- crème |
- brûlée |
- brûlée |
Text sorted by numeric codepoint follows no reasonable alphabetic order; use the UCA for sorting text.
See the ucsort program from the Unicode::Tussle CPAN module for a convenient command-line interface to this module.
Specify a collation strength of level 1 to ignore case and diacritics, only looking at the basic character.
Some locales have special sorting rules.
The ucsort program mentioned above accepts a --locale
parameter.
cmp
work on text instead of codepointsInstead of this:
Use this:
Use a collator object to compare Unicode text by character instead of by codepoint.
Same, but in a specific locale.
- my $de = Unicode::Collate::Locale->new(
- locale => "de__phonebook",
- );
- # now this is true:
- $de->eq("tschüß", "TSCHUESS"); # notice ü => UE, ß => SS
Break up text into lines according to Unicode rules.
Using a regular Perl string as a key or value for a DBM hash will trigger a wide character exception if any codepoints won’t fit into a byte. Here’s how to manually manage the translation:
- use DB_File;
- use Encode qw(encode decode);
- tie %dbhash, "DB_File", "pathname";
- # STORE
- # assume $uni_key and $uni_value are abstract Unicode strings
- my $enc_key = encode("UTF-8", $uni_key, 1);
- my $enc_value = encode("UTF-8", $uni_value, 1);
- $dbhash{$enc_key} = $enc_value;
- # FETCH
- # assume $uni_key holds a normal Perl string (abstract Unicode)
- my $enc_key = encode("UTF-8", $uni_key, 1);
- my $enc_value = $dbhash{$enc_key};
- my $uni_value = decode("UTF-8", $enc_value, 1);
Here’s how to implicitly manage the translation; all encoding and decoding is done automatically, just as with streams that have a particular encoding attached to them:
- use DB_File;
- use DBM_Filter;
- my $dbobj = tie %dbhash, "DB_File", "pathname";
- $dbobj->Filter_Value("utf8"); # this is the magic bit
- # STORE
- # assume $uni_key and $uni_value are abstract Unicode strings
- $dbhash{$uni_key} = $uni_value;
- # FETCH
- # $uni_key holds a normal Perl string (abstract Unicode)
- my $uni_value = $dbhash{$uni_key};
Here’s a full program showing how to make use of locale-sensitive sorting, Unicode casing, and managing print widths when some of the characters take up zero or two columns, not just one column each time. When run, the following program produces this nicely aligned output:
- Crème Brûlée....... €2.00
- Éclair............. €1.60
- Fideuà............. €4.20
- Hamburger.......... €6.00
- Jamón Serrano...... €4.45
- Linguiça........... €7.00
- Pâté............... €4.15
- Pears.............. €2.00
- Pêches............. €2.25
- Smørbrød........... €5.75
- Spätzle............ €5.50
- Xoriço............. €3.00
- Γύρος.............. €6.50
- 막걸리............. €4.00
- おもち............. €2.65
- お好み焼き......... €8.00
- シュークリーム..... €1.85
- 寿司............... €9.99
- 包子............... €7.50
Here's that program; tested on v5.14.
- #!/usr/bin/env perl
- # umenu - demo sorting and printing of Unicode food
- #
- # (obligatory and increasingly long preamble)
- #
- use utf8;
- use v5.14; # for locale sorting
- use strict;
- use warnings;
- use warnings qw(FATAL utf8); # fatalize encoding faults
- use open qw(:std :utf8); # undeclared streams in UTF-8
- use charnames qw(:full :short); # unneeded in v5.16
- # std modules
- use Unicode::Normalize; # std perl distro as of v5.8
- use List::Util qw(max); # std perl distro as of v5.10
- use Unicode::Collate::Locale; # std perl distro as of v5.14
- # cpan modules
- use Unicode::GCString; # from CPAN
- # forward defs
- sub pad($$$);
- sub colwidth(_);
- sub entitle(_);
- my %price = (
- "γύρος" => 6.50, # gyros
- "pears" => 2.00, # like um, pears
- "linguiça" => 7.00, # spicy sausage, Portuguese
- "xoriço" => 3.00, # chorizo sausage, Catalan
- "hamburger" => 6.00, # burgermeister meisterburger
- "éclair" => 1.60, # dessert, French
- "smørbrød" => 5.75, # sandwiches, Norwegian
- "spätzle" => 5.50, # Bayerisch noodles, little sparrows
- "包子" => 7.50, # bao1 zi5, steamed pork buns, Mandarin
- "jamón serrano" => 4.45, # country ham, Spanish
- "pêches" => 2.25, # peaches, French
- "シュークリーム" => 1.85, # cream-filled pastry like eclair
- "막걸리" => 4.00, # makgeolli, Korean rice wine
- "寿司" => 9.99, # sushi, Japanese
- "おもち" => 2.65, # omochi, rice cakes, Japanese
- "crème brûlée" => 2.00, # crema catalana
- "fideuà" => 4.20, # more noodles, Valencian
- # (Catalan=fideuada)
- "pâté" => 4.15, # gooseliver paste, French
- "お好み焼き" => 8.00, # okonomiyaki, Japanese
- );
- my $width = 5 + max map { colwidth } keys %price;
- # So the Asian stuff comes out in an order that someone
- # who reads those scripts won't freak out over; the
- # CJK stuff will be in JIS X 0208 order that way.
- my $coll = new Unicode::Collate::Locale locale => "ja";
- for my $item ($coll->sort(keys %price)) {
- print pad(entitle($item), $width, ".");
- printf " €%.2f\n", $price{$item};
- }
- sub pad($$$) {
- my($str, $width, $padchar) = @_;
- return $str . ($padchar x ($width - colwidth($str)));
- }
- sub colwidth(_) {
- my($str) = @_;
- return Unicode::GCString->new($str)->columns;
- }
- sub entitle(_) {
- my($str) = @_;
- $str =~ s{ (?=\pL)(\S) (\S*) }
- { ucfirst($1) . lc($2) }xge;
- return $str;
- }
See these manpages, some of which are CPAN modules: perlunicode, perluniprops, perlre, perlrecharclass, perluniintro, perlunitut, perlunifaq, PerlIO, DB_File, DBM_Filter, DBM_Filter::utf8, Encode, Encode::Locale, Unicode::UCD, Unicode::Normalize, Unicode::GCString, Unicode::LineBreak, Unicode::Collate, Unicode::Collate::Locale, Unicode::Unihan, Unicode::CaseFold, Unicode::Tussle, Lingua::JA::Romanize::Japanese, Lingua::ZH::Romanize::Pinyin, Lingua::KO::Romanize::Hangul.
The Unicode::Tussle CPAN module includes many programs to help with working with Unicode, including these programs to fully or partly replace standard utilities: tcgrep instead of egrep, uniquote instead of cat -v or hexdump, uniwc instead of wc, unilook instead of look, unifmt instead of fmt, and ucsort instead of sort. For exploring Unicode character names and character properties, see its uniprops, unichars, and uninames programs. It also supplies these programs, all of which are general filters that do Unicode-y things: unititle and unicaps; uniwide and uninarrow; unisupers and unisubs; nfd, nfc, nfkd, and nfkc; and uc, lc, and tc.
Finally, see the published Unicode Standard (page numbers are from version 6.0.0), including these specific annexes and technical reports:
Tom Christiansen <tchrist@perl.com> wrote this, with occasional kibbitzing from Larry Wall and Jeffrey Friedl in the background.
Copyright © 2012 Tom Christiansen.
This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself.
Most of these examples taken from the current edition of the “Camel Book”; that is, from the 4ᵗʰ Edition of Programming Perl, Copyright © 2012 Tom Christiansen <et al.>, 2012-02-13 by O’Reilly Media. The code itself is freely redistributable, and you are encouraged to transplant, fold, spindle, and mutilate any of the examples in this manpage however you please for inclusion into your own programs without any encumbrance whatsoever. Acknowledgement via code comment is polite but not required.
v1.0.0 – first public release, 2012-02-27