How are text files encoded, and what are the pros and cons of different encodings? This document introduces the ASCII and Unicode encodings to answer these questions. Some short pieces of Go code are provided which you can run yourself to see how strings are being encoded. Go is used because it has built-in support for Unicode, but you should be able to follow without any knowledge of Go.
ASCII is a character encoding which uses a 7-bit number to encode a single character. In ASCII, we associate each character with a unique number between 0 and 127. On modern machines, each character is stored as a byte where the most significant bit is always 0. Let's see it in action!
// Go uses UTF-8 for string encoding, but this string will be encoded in ASCII,
// trust me (and keep reading to learn why!).
hello := "Hello!"
fmt.Println([]byte(hello)) // Prints the string hello as an array of bytes
// Prints:
// [72 101 108 108 111 33]We can deduce from this test program that the character H is encoded as 72,
e is 101, etc...
For the full ASCII table showing every character and its corresponding
encoding, see man ascii, or http://www.asciitable.com.
The ASCII encoding is very compact, with 1 byte per character, but it only has space for 128 different characters. This is enough for digits, the English alphabet (lowercase and uppercase), certain punctuation, and spacing/control characters. For representing languages other than English, this is not good enough.
Enter Unicode, the one code to rule them all. Unicode is a standard for text encoding and representation. The Unicode standard defines a way to represent 1,112,064 possible characters. Of these, there are roughly 140,000 characters in use currently. The Unicode standard defines several different encodings: UTF-32, UTF-16, and UTF-8 (and some other ones we won't discuss). UTF-8 is the most widely used, but we'll talk about UTF-32 first.
Things get confusing because it's no longer clear what a "character" means. In
C, a char is 1 byte. In text, a character is a symbol, and not necessarily
encoded as one byte. To make things clearer we will use the following
terminology:
- A "byte" is one byte (8 bits).
- A "code point" or "rune" is one of the 1,112,064 possible symbols defined by
the Unicode standard. To refer to a code point, we use the notation
U+xxxxxx, wherexxxxxxis a hexadecimal number referring to that code point's index in the overall list of code points. - A "grapheme" is one cell of text. For example, as we will see later certain symbols such as characters with accents can be one grapheme but multiple code points.
Since we only need to encode 1,112,064 possible code points, the simplest choice is to use a 32-bit integer to encode a code point. This is clearly large enough to encode any possible code point. In Go, a "rune" refers to a Unicode code point, and is exactly the same as the "int32" datatype, which means an array of runes is using a UTF-32 encoding.
Let's try it out:
symbols := "αβ"
fmt.Println([]rune(symbols)) // Prints symbols as an array of runes/code points
// Prints:
// [945 946]Great, we can encode any Unicode code point now! Plus, it is a fixed width encoding, so determining the location of the nth code point in a string is simple. There are some downsides though:
- Every symbol takes 4 bytes to encode, which means every file encoded in UTF-32 will be 4 times larger than the same file encoded in ASCII (if it's possible to encode the file in ASCII).
- Any file that was encoded in ASCII before will have to be converted to UTF-32. Likewise, UTF-32 encoded files cannot be viewed by a program that can only decode ASCII (even if the UTF-32 file only contains code points that can be represented by ASCII).
With UTF-8, we will try to fix the two downsides incurred by UTF-32. UTF-8 is a variable-width encoding. This means that some code points are represented using anywhere between 1 and 4 bytes of space. Let's look at how this works (using a table from Wikipedia). The table shows how various ranges of code points are represented in the UTF-8 encoding.
| Number of bytes | First code point | Last code point | Byte 1 | Byte 2 | Byte 3 | Byte 4 |
|---|---|---|---|---|---|---|
| 1 | U+0000 | U+007F | 0xxxxxxx | |||
| 2 | U+0080 | U+07FF | 110xxxxx | 10xxxxxx | ||
| 3 | U+0800 | U+FFFF | 1110xxxx | 10xxxxxx | 10xxxxxx | |
| 4 | U+10000 | U+10FFFF | 11110xxx | 10xxxxxx | 10xxxxxx | 10xxxxxx |
What does this mean? To encode a code point from U+0000 to U+007F (0 to 127), we just use one byte. This is also extremely convenient because it is the same encoding as ASCII for the first 127 code points, so any ASCII file will also be a valid UTF-8 file.
For later code points, we use prefixes on the first byte to know how many bytes
this code point will use for encoding. Since all prefixes uniquely identify the
kind of byte being encoded (e.g. first byte of a 2-byte encoded code point, or
first byte of a 3-byte encoded code point...) there is never ambiguity during
decoding about how many bytes to look at. For code points with multi-byte
encodings, each later byte begins with 10. This is to prevent problems where
starting to read the text in the middle of a code point (because of an error)
would completely throw off the decoding of later code points as well.
Let's see it in action:
symbols := "αβ" // in Go, strings are encoded in UTF-8 by default
fmt.Println([]byte(symbols))
// Prints:
// [206 177 206 178]For the two symbols αβ, we see that bytes with values greater than 127 are
used. For a string like "Hello", we would see one byte per code point, since
all of the code points in that string are between U+0000 and U+007F.
This is great because it allows our files to be compact in the common case (most files use English characters), and it is backwards compatible with ASCII! The downside is that figuring out where the nth code point begins requires decoding until that code point is found. Finding the number of code points in a string is also an O(n) algorithm, even if the length in bytes of the string is known.
One difficulty in Unicode is encoding accents. Should we make a separate code
point for every character with an accent? For example, should é, á, ´
a, e all have one unique code point? Surely they need to be encoded
differently, but dedicating a codepoint for every possible combination of
character and accent starts using up many code points. Unicode solves this with
the concept of a "combining code point." When the code point for a is
followed by the code point for ´, the accent is combined with the character
by the program that is displaying the text. This is because the code point for
the accent is defined to be a "combining" code point.
Let's look at the bytes in Go:
symbols := "X̿" // in Go, strings are encoded in UTF-8 by default
fmt.Println([]byte(symbols))
fmt.Println([]rune(symbols))
// Prints:
// [88 204 191]
// [88 831]We convert to UTF-32 by casting to a rune array to look at each individual
code point. Here's what we can determine: this string has two code points even
though it displays as one cell, with the accent symbol above the symbol encoded
before it, and the accent symbol's code point is encoded using two bytes. Cool!
Why did I suddenly pick X̿ for my example instead of á like I was talking
about previously? This is because for á the story is more complicated: this
accent is more common, so for space efficiency it actually does get a unique
code point. This means there are two ways of encoding the character á: using
an a and a combining ´, or using the dedicated code point á. Let's see it
in Go:
combiningAccent := "á"
fmt.Println("Combining:")
fmt.Println([]byte(combiningAccent))
fmt.Println([]rune(combiningAccent))
// Prints:
// Combining:
// [97 204 129]
// [97 769]
dedicatedAccent := "á"
fmt.Println("Dedicated:")
fmt.Println([]byte(dedicatedAccent))
fmt.Println([]rune(dedicatedAccent))
// Prints:
// Dedicated:
// [195 161]
// [225]Even though these two strings look exactly the same when displayed by your editor, they are encoded differently! To read more about this, look up "text normalization." If you try pasting this code into your editor, you might see different results if your computer is automatically changing the combined accent to the dedicated one. You can try directly printing the bytes from the combined accent above to see the combining accent in action:
b := []byte{97, 204, 129}
fmt.Println(string(b))
// Prints:
// áMore than one combining character can be applied to form a single grapheme. For example, here is an interesting string:
interesting := "H̼̥̯͇͙̕͘͞e̸̦̞̠̣̰͙̼̥̦̼̖̬͕͕̰̯̫͇̕ĺ̜̠̩̯̯͙̼̭̠͕̮̞͜l̶͓̫̞̮͈͞ͅo̸͔͙̳̠͈̮̼̳͙̥̲͜͠"
fmt.Printf("Number of bytes: %d\n", len(interesting))
fmt.Printf("Number of code points: %d\n", utf8.RuneCountInString(interesting))
fmt.Printf("Number of graphemes: %d\n", uniseg.GraphemeClusterCount(interesting))
// Prints:
// Number of bytes: 132
// Number of code points: 68
// Number of graphemes: 5Wow! This string is 132 bytes, 68 code points, and only 5 graphemes! To count
the number of graphemes, I used the uniseg library, which you can find at
https://github.com/rivo/uniseg.
There are other ways characters can be marked to display differently from
normal. Some characters are "double-width" which means they take up two
text cells when displayed. For example, many chinese characters are double
width: ⾖.
UTF-16 is an encoding similar to UTF-8 but it uses blocks of 16 bits instead of 8 bits. Back when Unicode only supported fewer than 65536 code points, this worked fairly well, but since then the standard has grown. This means it incurs most of the downsides of both UTF-32 and UTF-8. It still sees use as a legacy encoding format because it existed before UTF-8.