UTF-16: Переменная ширина кодирования Unicode

Variable width encoding of Unicode, using one or two 16 bit code units

UTF 16 (16 bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF 16). The encoding is variable length, as code points are encoded with one or two 16 bit code units. UTF 16 arose from an earlier obsolete fixed width 16 bit encoding now known as UCS 2 (for 2 byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names. UTF 16 is used by systems such as the Microsoft Windows API, the Java programming language and JavaScript/ECMAScript. It is also sometimes used for plain text and word processing data files on Microsoft Windows. It is used by SMS (the SMS standard specifies UCS 2, but almost all users actually implement UTF 16 so that emojis work). UTF 16 is the only encoding (still) allowed on the web that is incompatible with ASCII and never gained popularity on the web, where it is declared by under 0.004% of web pages (and many of these are actually UTF 8 but incorrectly marked). UTF 8, by comparison, accounts for over 98% of all web pages. The Web Hypertext Application Technology Working Group (WHATWG) considers UTF 8 "the mandatory encoding for all [text]" and that for security reasons browser applications should not use UTF 16.

In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" (UCS) that would replace earlier language specific encodings with one coordinated system. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256 character encodings, which required 1 byte per character, with an encoding using 65,536 (216) values, which would require 2 bytes (16 bits) per character. Two groups worked on this in parallel, ISO/IEC JTC 1/SC 2 and the Unicode Consortium, the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2 byte encoding was originally called "Unicode", but is now called "UCS 2". When it became increasingly clear that 216 characters would not suffice, IEEE introduced a larger 31 bit space and an encoding (UCS 4) that would require 4 bytes per character. This was resisted by the Unicode Consortium, both because 4 bytes per character wasted a lot of memory and disk space, and because some manufacturers were already heavily invested in 2 byte per character technology. The UTF 16 encoding scheme was developed as a compromise and introduced with version 2.0 of the Unicode standard in July 1996. It is fully specified in RFC 2781, published in 2000 by the IETF. UTF 16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard. "UCS 2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard." (Any scheme that remains a self synchronizing code would require allocating at least one Basic Multilingual Plane (BMP) code point to start a sequence. Changing the purpose of a code point is disallowed.)

Each Unicode code point is encoded either as one or two 16 bit code units. Code points less than 216 ("in the BMP") are encoded with a single 16 bit code unit equal to the numerical value of the code point, as in the older UCS 2. Code points greater than or equal to 216 ("above the BMP") are encoded using two 16 bit code units. These two 16 bit code units are chosen from the UTF 16 surrogate range which had not previously been assigned to characters. Values in this range are not used as characters, and UTF 16 provides no legal way to code them as individual code points. A UTF 16 stream, therefore, consists of single 16 bit codes outside the surrogate range, and pairs of 16 bit values that are within the surrogate range.

Both UTF 16 and UCS 2 encode code points in this range as single 16 bit code units that are numerically equal to the corresponding code points. These code points in the Basic Multilingual Plane (BMP) are the only code points that can be represented in UCS 2. As of Unicode 9.0, some modern non Latin Asian, Middle Eastern, and African scripts fall outside this range, as do most emoji characters.