How Unicode and UTF-8 Work: Code Points, Bytes and Why Text Breaks

Characters, code points and bytes: three different things

The confusion around text mostly comes from mixing up three layers that sound alike:

• Character: an abstract symbol, such as the letter A, the euro sign or a smiling-face emoji.
• Code point: the number Unicode assigns to that character, written like U+0041 for A or U+20AC for €.
• Encoding: the recipe that turns code points into actual bytes in a file, a network packet or a database row. UTF-8, UTF-16 and UTF-32 are three such recipes for the same code points.

Unicode is the catalog; UTF-8 is a shipping format. Keeping those apart makes almost every text bug easier to reason about.

How big is Unicode?

Unicode's code space runs from U+0000 to U+10FFFF: 17 "planes" of 65,536 code points each, or 1,114,112 slots in total. Most everyday text lives in the first plane, the Basic Multilingual Plane. Emoji, many historic scripts and rarer CJK ideographs live above it. According to the Unicode Consortium's official counts, version 17.0 (2025) has 297,334 assigned code points, a figure that includes 137,468 reserved for private use.

How UTF-8 encodes a code point

UTF-8 is a variable-length encoding: each code point takes between one and four bytes, depending on how large its number is. The scheme is defined in IETF RFC 3629, and the whole thing fits in a small table:

The design has some clever properties that explain why it took over:

1. ASCII stays ASCII. The first 128 code points encode as the same single bytes they always had, so any plain ASCII file is already valid UTF-8.
2. No false matches. Bytes below 0x80 never appear inside a multi-byte sequence, so code that splits on commas, slashes or newlines can't accidentally cut a character in half.
3. Self-synchronizing. Lead bytes and continuation bytes (10xxxxxx) look different, so a decoder dropped into the middle of a stream can find the next character boundary.
4. No byte-order problem. UTF-8 is a sequence of single bytes, so unlike UTF-16 it doesn't need to declare big-endian or little-endian.

The format has a fun origin story. In an email Rob Pike later shared, he recalls that he and Ken Thompson worked out the scheme in September 1992 over dinner at a New Jersey diner, with Thompson sketching the bit-packing on a placemat.

Rules a decoder must enforce

RFC 3629 also tightened what counts as valid. UTF-8 stops at U+10FFFF (earlier versions allowed up to six bytes), the range U+D800–U+DFFF is forbidden because those values are reserved for UTF-16 surrogates, and "overlong" encodings, such as writing a plain character with more bytes than necessary, must be rejected. That last rule matters for security: lenient decoders have historically let attackers sneak characters like / past filters.

Why string length lies

Here's where developers get bitten. "How long is this string?" has at least four honest answers, and different languages pick different ones:

// JavaScript: length counts UTF-16 code units
"😄".length;                 // 2
[..."😄"].length;            // 1  (code points)

// User-perceived characters (grapheme clusters)
const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
[...seg.segment("👨‍👩‍👧")].length;  // 1

# Python: len() counts code points
len("😄")                    # 1
len("😄".encode("utf-8"))    # 4 bytes

MDN's documentation is explicit that a JavaScript string's length is the number of UTF-16 code units, not visible characters. Characters above the first plane, including most emoji, take two units (a "surrogate pair"). It gets stranger with combined emoji: the family emoji above is three people glued together with invisible zero-width joiners, so it's five code points and 18 UTF-8 bytes while looking like one symbol. Intl.Segmenter counts what a human would call one character.

Mojibake: what garbled text really means

The "Zoë" in our intro has a name: mojibake, from the Japanese for "character transformation." It happens when bytes written in one encoding are read as another. The UTF-8 bytes for "ë" are C3 AB; read those two bytes as the old Windows-1252 code page and you get two separate characters, "Ã" and "«". The text wasn't corrupted when it was saved. It was misread later.

Common culprits include:

• A web page or API response that doesn't declare its charset, leaving the client to guess.
• A database connection using a different character set from the tables it writes to.
• Spreadsheet exports and imports that assume a legacy regional encoding.
• Text that was decoded and re-encoded twice somewhere in a pipeline.

UTF-8 everywhere: practical habits

UTF-8 has effectively won. W3Techs reports that 99.1% of websites whose encoding it can detect use UTF-8. The remaining job is making sure every layer of your stack agrees. A short checklist:

• Declare it on the web. Put <meta charset="utf-8"> early in your HTML head (part of good semantic HTML hygiene) and send Content-Type: text/html; charset=utf-8 from the server.
• Configure the database end to end. In MySQL, prefer utf8mb4; the older utf8 alias stores at most three bytes per character and can't hold emoji. Check the connection setting, not just the table.
• Decode at the edges. Convert bytes to text as soon as data enters your program, work with text internally, and encode back to UTF-8 only when writing out.
• Skip the BOM unless required. RFC 3629 notes that protocols which always use UTF-8 should forbid the byte-order mark EF BB BF; it tends to confuse scripts, CSV parsers and shells.
• Measure the right thing. Enforce length limits in the unit your storage uses, and never truncate in the middle of a multi-byte sequence or grapheme.

When you're tracking down slow queries rather than strange characters, our guide to how database indexes work covers the next most common database surprise.

Frequently asked questions