Сравнение кодировок Unicode: UTF-8, UTF-16 и UTF-32

None

This article compares Unicode encodings. Two situations are considered: 8 bit clean environments (which can be assumed), and environments that forbid use of byte values that have the high bit set. Originally such prohibitions were to allow for links that used only seven data bits, but they remain in some standards and so some standard conforming software must generate messages that comply with the restrictions. Standard Compression Scheme for Unicode and Binary Ordered Compression for Unicode are excluded from the comparison tables because it is difficult to simply quantify their size.

A UTF 8 file that contains only ASCII characters is identical to an ASCII file. Legacy programs can generally handle UTF 8 encoded files, even if they contain non ASCII characters. For instance, the C printf function can print a UTF 8 string, as it only looks for the ASCII '%' character to define a formatting string, and prints all other bytes unchanged, thus non ASCII characters will be output unchanged. UTF 16 and UTF 32 are incompatible with ASCII files, and thus require Unicode aware programs to display, print and manipulate them, even if the file is known to contain only characters in the ASCII subset. Because they contain many zero bytes, the strings cannot be manipulated by normal null terminated string handling for even simple operations such as copy. Therefore, even on most UTF 16 systems such as Windows and Java, UTF 16 text files are not common; older 8 bit encodings such as ASCII or ISO 8859 1 are still used, forgoing Unicode support; or UTF 8 is used for Unicode. One rare counter example is the "strings" file used by macOS (Mac OS X 10.3 Panther and later) applications for lookup of internationalized versions of messages which defaults to UTF 16, with "files encoded using UTF 8 not guaranteed to work." XML is, by convention, encoded as UTF 8, and all XML processors must at least support UTF 8 (including US ASCII by definition) and UTF 16.

UTF 8 requires 8, 16, 24 or 32 bits (one to four bytes) to encode a Unicode character, UTF 16 requires either 16 or 32 bits to encode a character, and UTF 32 always requires 32 bits to encode a character. The first 128 Unicode code points, U+0000 to U+007F, used for the C0 Controls and Basic Latin characters and which correspond one to one to their ASCII code equivalents, are encoded using 8 bits in UTF 8, 16 bits in UTF 16, and 32 bits in UTF 32. The next 1,920 characters, U+0080 to U+07FF (encompassing the remainder of almost all Latin script alphabets, and also Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac, Tāna and N'Ko), require 16 bits to encode in both UTF 8 and UTF 16, and 32 bits in UTF 32. For U+0800 to U+FFFF, i. e. the remainder of the characters in the Basic Multilingual Plane (BMP, plane 0, U+0000 to U+FFFF), which encompasses the rest of the characters of most of the world's living languages, UTF 8 needs 24 bits to encode a character, while UTF 16 needs 16 bits and UTF 32 needs 32. Code points U+010000 to U+10FFFF, which represent characters in the supplementary planes (planes 1–16), require 32 bits in UTF 8, UTF 16 and UTF 32. Therefore, a file is shorter in UTF 8 than in UTF 16 if there are more ASCII code points than there are code points in the range U+0800 to U+FFFF. A surprising result is that real world documents written in languages that use characters only in the high range are still often shorter in UTF 8, due to the extensive use of spaces, digits, punctuation, newlines, html markup, and embedded words and acronyms written with Latin letters. UTF 32 is always longer unless there are no code points less than U+10000. All printable characters in UTF EBCDIC use at least as many bytes as in UTF 8, and most use more, due to a decision made to allow encoding the C1 control codes as single bytes. For seven bit environments, UTF 7 is more space efficient than the combination of other Unicode encodings with quoted printable or base64 for almost all types of text (see "Seven bit environments" below).

Text with variable length encoding such as UTF 8 or UTF 16 is harder to process if there is a need to work with individual code units, as opposed to working with sequences of code units. Searching is unaffected by whether the characters are variable sized, since a search for a sequence of code units does not care about the divisions (it does require that the encoding be self synchronizing, which both UTF 8 and UTF 16 are). A common misconception is that there is a need to "find the nth character" and that this requires a fixed length encoding; however, in real use the number n is only derived from examining the n−1 characters, thus sequential access is needed anyway. When character sequences in one endian order are loaded onto a machine with a different endian order, the characters need to be converted before they can be processed efficiently (or two processors are needed). Byte based encodings such as UTF 8 do not have this problem. UTF 16BE and UTF 32BE are big endian, UTF 16LE and UTF 32LE are little endian.

For processing, a format should be easy to search, truncate, and generally process safely. All normal Unicode encodings use some form of fixed size code unit. Depending on the format and the code point to be encoded, one or more of these code units will represent a Unicode code point. To allow easy searching and truncation, a sequence must not occur within a longer sequence or across the boundary of two other sequences. UTF 8, UTF 16, UTF 32 and UTF EBCDIC have these important properties but UTF 7 and GB 18030 do not. Fixed size characters can be helpful, but even if there is a fixed byte count per code point (as in UTF 32), there is not a fixed byte count per displayed character due to combining characters. Considering these incompatibilities and other quirks among different encoding schemes, handling unicode data with the same (or compatible) protocol throughout and across the interfaces (e. g. using an API/library, handling unicode characters in client/server model, etc.) can in general simplify the whole pipeline while eliminating a potential source of bugs at the same time. UTF 16 is popular because many APIs date to the time when Unicode was 16 bit fixed width (referred as UCS 2). However, using UTF 16 makes characters outside the Basic Multilingual Plane a special case which increases the risk of oversights related to their handling. That said, programs that mishandle surrogate pairs probably also have problems with combining sequences, so using UTF 32 is unlikely to solve the more general problem of poor handling of multi code unit characters. If any stored data is in UTF 8 (such as file contents or names), it is very difficult to write a system that uses UTF 16 or UTF 32 as an API. This is due to the oft overlooked fact that the byte array used by UTF 8 can physically contain invalid sequences. For instance, it is impossible to fix an invalid UTF 8 filename using a UTF 16 API, as no possible UTF 16 string will translate to that invalid filename. The opposite is not true: it is trivial to translate invalid UTF 16 to a unique (though technically invalid) UTF 8 string, so a UTF 8 API can control both UTF 8 and UTF 16 files and names, making UTF 8 preferred in any such mixed environment. An unfortunate but far more common workaround used by UTF 16 systems is to interpret the UTF 8 as some other encoding such as CP 1252 and ignore the mojibake for any non ASCII data.

UTF 16 and UTF 32 do not have endianness defined, so a byte order must be selected when receiving them over a byte oriented network or reading them from a byte oriented storage. This may be achieved by using a byte order mark at the start of the text or assuming big endian (RFC 2781). UTF 8, UTF 16BE, UTF 32BE, UTF 16LE and UTF 32LE are standardised on a single byte order and do not have this problem. If the byte stream is subject to corruption then some encodings recover better than others. UTF 8 and UTF EBCDIC are best in this regard as they can always resynchronize after a corrupt or missing byte at the start of the next code point; GB 18030 is unable to recover until the next ASCII non number. UTF 16 can handle altered bytes, but not an odd number of missing bytes, which will garble all the following text (though it will produce uncommon and/or unassigned characters). If bits can be lost all of them will garble the following text, though UTF 8 can be resynchronized as incorrect byte boundaries will produce invalid UTF 8 in almost all text longer than a few bytes.

The tables below list the number of bytes per code point for different Unicode ranges. Any additional comments needed are included in the table. The figures assume that overheads at the start and end of the block of text are negligible. N. B. The tables below list numbers of bytes per code point, not per user visible "character" (or "grapheme cluster"). It can take multiple code points to describe a single grapheme cluster, so even in UTF 32, care must be taken when splitting or concatenating strings.

Code range (hexadecimal) UTF 8 UTF 16 UTF 32 UTF EBCDIC GB 18030000000 – 00007F12411000080 – 00009F22 for characters inherited fromGB 2312/GBK (e. g. mostChinese characters) 4 foreverything else.0000A0 – 0003FF2000400 – 0007FF3000800 – 003FFF3004000 – 00FFFF4010000 – 03FFFF444040000 – 10FFFF5

This table may not cover every special case and so should be used for estimation and comparison only. To accurately determine the size of text in an encoding, see the actual specifications. Code range (hexadecimal)UTF 7UTF 8 quoted printableUTF 8 base64UTF 16 q. p. UTF 16 base64GB 18030 q. p. GB 18030 base64ASCIIgraphic characters(except U+003D "=")1 for "direct characters" (depends on the encoder setting for some code points), 2 for U+002B "+", otherwise same as for 000080 – 00FFFF11 1/342 2/311 1/300003D (equals sign)363ASCIIcontrol characters:000000 – 00001Fand 00007F1 or 3 depending on directness1 or 3 depending on directness000080 – 0007FF5 for an isolated case inside a run of single byte characters. For runs 2 2/3 per character plus padding to make it a whole number of bytes plus two to start and finish the run62 2/32–6 depending on if the byte values need to be escaped4–6 for characters inherited from GB2312/GBK (e. g. most Chinese characters) 8 for everything else.2 2/3 for characters inherited from GB2312/GBK (e. g. most Chinese characters) 5 1/3 for everything else.000800 – 00FFFF94010000 – 10FFFF8 for isolated case, 5 1/3 per character plus padding to integer plus 2 for a run125 1/38–12 depending on if the low bytes of the surrogates need to be escaped.5 1/385 1/3
Endianness does not affect sizes (UTF 16BE and UTF 32BE have the same size as UTF 16LE and UTF 32LE, respectively). The use of UTF 32 under quoted printable is highly impractical, but if implemented, will result in 8–12 bytes per code point (about 10 bytes in average), namely for BMP, each code point will occupy exactly 6 bytes more than the same code in quoted printable/UTF 16. Base64/UTF 32 gets 5 1/3 bytes for any code point. An ASCII control character under quoted printable or UTF 7 may be represented either directly or encoded (escaped). The need to escape a given control character depends on many circumstances, but newlines in text data are usually coded directly.

BOCU 1 and SCSU are two ways to compress Unicode data. Their encoding relies on how frequently the text is used. Most runs of text use the same script; for example, Latin, Cyrillic, Greek and so on. This normal use allows many runs of text to compress down to about 1 byte per code point. These stateful encodings make it more difficult to randomly access text at any position of a string. These two compression schemes are not as efficient as other compression schemes, like zip or bzip2. Those general purpose compression schemes can compress longer runs of bytes to just a few bytes. The SCSU and BOCU 1 compression schemes will not compress more than the theoretical 25% of text encoded as UTF 8, UTF 16 or UTF 32. Other general purpose compression schemes can easily compress to 10% of original text size. The general purpose schemes require more complicated algorithms and longer chunks of text for a good compression ratio. Unicode Technical Note #14 contains a more detailed comparison of compression schemes.

Proposals have been made for a UTF 5 and UTF 6 for the internationalization of domain names (IDN). The UTF 5 proposal used a base 32 encoding, where Punycode is (among other things, and not exactly) a base 36 encoding. The name UTF 5 for a code unit of 5 bits is explained by the equation 25 = 32. The UTF 6 proposal added a running length encoding to UTF 5, here 6 simply stands for UTF 5 plus 1. The IETF IDN WG later adopted the more efficient Punycode for this purpose.

UTF 1 never gained serious acceptance. UTF 8 is much more frequently used. The nonet encodings UTF 9 and UTF 18 are April Fools' Day RFC joke specifications, although UTF 9 is a functioning nonet Unicode transformation format, and UTF 18 is a functioning nonet encoding for all non Private Use code points in Unicode 12 and below, although not for Supplementary Private Use Areas or portions of Unicode 13 and later.