Unicode in JavaScript strings

Strings showed that "😀".length is 2 and that normalize() makes two spellings of é equal. This lesson explains why, and gives you the tools to get “first character”, “length” and “cut to N” right. The general ideas of encodings and text are in the fundamentals lesson on text and strings. Here the focus is on how JavaScript represents text.

Four ways to count text

The same piece of text has different sizes depending on what you count.

A JavaScript string is a sequence of UTF-16 code units. Characters above U+FFFF, including most emoji, take two code units, called a surrogate pair.

Code units and code points

The escape \u{...} writes a code point by number, with any number of digits.

const smile = "\u{1F600}";
console.log(smile.length, smile === "😀");

2 true

console.log(smile.charCodeAt(0).toString(16), smile.charCodeAt(1).toString(16));
console.log(smile.codePointAt(0).toString(16));
console.log(smile.codePointAt(1).toString(16));
console.log(String.fromCodePoint(0x1f600) === smile);

d83d de00
1f600
de00
true

• The emoji is stored as two code units, d83d then de00, in hex.
• codePointAt(0) reads the whole pair and gives 1f600. codePointAt(1) points into the middle of the pair and returns only the second half, de00. Use it at positions you know are starts.
• String.fromCodePoint is the reverse of codePointAt. The older String.fromCharCode only handles one code unit.

Indexing, slice and length all work on code units. for...of and spread work on code points.

const word = "a😀b";
console.log(word.length, [...word].length);
console.log(word.slice(0, 2) === "a😀", word.slice(0, 2).length);

4 3
false 2

for (const ch of word) {
  console.log(ch, ch.codePointAt(0));
}

a 97
😀 128512
b 98

slice(0, 2) kept a and only half of the emoji, which is a broken string with length 2. Cutting by code unit can split a pair. Loop with for...of or spread first when you take parts of text that may contain emoji.

Normalization

Many characters have more than one spelling. é can be one code point (U+00E9) or e followed by a combining acute accent (U+0301). They look the same and compare as different.

const composed = "café";
const decomposed = "café";
console.log(composed === decomposed, composed.length, decomposed.length);

false 4 5

console.log(composed.normalize("NFD") === decomposed);
console.log(decomposed.normalize("NFC") === composed);

true
true

normalize takes a form name:

• NFC composes where it can (the default). Use it for storing and comparing.
• NFD decomposes into base letters and combining marks.
• NFKC and NFKD also replace “compatibility” characters, such as ligatures, with plain equivalents.

Decomposing is also the standard trick for stripping accents. After NFD, the marks are separate code points that the Unicode property \p{M} (mark) finds:

const stripAccents = (s) => s.normalize("NFD").replace(/\p{M}/gu, "");
console.log(stripAccents("Crème brûlée, señor"));

Creme brulee, senor

The compatibility forms are for matching and search, not for storage, because they lose information:

console.log("find".normalize("NFKC"), "①".normalize("NFKC"));
console.log("find".normalize("NFC") === "find");

find 1
false

The ligature fi becomes the two letters fi and the circled digit becomes 1 under NFKC, but NFC leaves the ligature alone.

Well-formed strings

JavaScript does not stop you from creating a string with a lone surrogate, one half of a pair with no partner. It is not valid Unicode text and cannot be encoded as UTF-8. It usually comes from slicing in the middle of an emoji. isWellFormed() checks for it and toWellFormed() replaces each lone surrogate with the replacement character U+FFFD.

const good = "ok \u{1F600}";
const bad = "ok \uD83D";
console.log(good.isWellFormed(), bad.isWellFormed());
console.log(bad.toWellFormed() === "ok �");

true false
true

Some APIs refuse broken strings, others quietly change them.

try {
  encodeURIComponent(bad);
} catch (error) {
  console.log(error.name);
}
console.log(encodeURIComponent(bad.toWellFormed()));
console.log(JSON.stringify(bad));

URIError
ok%20%EF%BF%BD
"ok \ud83d"

• encodeURIComponent throws a URIError on a lone surrogate.
• After toWellFormed, it encodes the replacement character as %EF%BF%BD.
• JSON.stringify writes the lone surrogate as the escape \ud83d, so the JSON stays valid.

Call toWellFormed() before sending text to something that insists on UTF-8, if the text could have been cut by code unit.

Graphemes with Intl.Segmenter

Even code points are not “characters” in the reader’s sense. A family emoji is three people joined by invisible zero-width joiners. A flag is two regional-indicator code points. é as e plus a combining mark is two code points. A reader sees one symbol in each case.

Intl.Segmenter splits text by the rules of a locale. With granularity: "grapheme" it splits into grapheme clusters, the user-visible characters.

const family = "👨‍👩‍👧";
const flag = "🇯🇵";
const accented = "é";

for (const s of [family, flag, accented]) {
  console.log(s.length, [...s].length);
}

8 5
4 2
2 2

Each line shows the length and the code point count. The family emoji is 8 units and 5 code points, but one symbol.

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
const graphemes = (s) => Array.from(seg.segment(s), (part) => part.segment);

for (const s of [family, flag, accented]) {
  console.log(graphemes(s).length);
}
console.log(graphemes("a👨‍👩‍👧é"));

1
1
1
[ 'a', '👨‍👩‍👧', 'é' ]

• segment returns an iterable of parts, and each part has a segment string (and an index).
• Array.from with a mapping function turns the parts into strings.
• All three samples count as one grapheme, and the mixed string splits into three.

The segmenter has two more granularities. "word" marks the pieces of text that are words, via isWordLike:

const words = new Intl.Segmenter("en", { granularity: "word" });
const text = "Well, it's 3.5 days!";

const found = [];
for (const part of words.segment(text)) {
  if (part.isWordLike) {
    found.push(part.segment);
  }
}
console.log(found);

[ 'Well', "it's", '3.5', 'days' ]

Punctuation and spaces are segments too, but isWordLike is false for them. The segmenter knows that it's and 3.5 are each one word, which a plain split on spaces and punctuation would break. The third granularity is "sentence".

Creating a segmenter has a cost, so make one and reuse it.

Bytes and encodings

Files and networks carry bytes, and JavaScript text is turned into them with an encoding. UTF-8 is the common choice. The built-in TextEncoder always produces UTF-8, and TextDecoder reads it.

const encoder = new TextEncoder();
for (const s of ["a", "é", "€", "😀"]) {
  console.log(s, s.length, encoder.encode(s).length);
}

a 1 1
é 1 2
€ 1 3
😀 2 4

const bytes = encoder.encode("é");
console.log([...bytes]);
console.log(new TextDecoder().decode(bytes));
console.log(new TextDecoder().decode(Uint8Array.of(0xff, 0x41)));

[ 195, 169 ]
é
�A

• A string of one character takes between one and four bytes. The same emoji is 2 code units but 4 bytes.
• Decoding bytes that are not valid UTF-8 does not throw by default. Each bad byte becomes the replacement character �.

So the length you need depends on the question. A database column with a byte limit needs TextEncoder. A tweet-like counter needs graphemes. An array index needs code units.

Percent-encoding for URLs

A URL may only carry a small set of ASCII characters. Anything else is written as its UTF-8 bytes, each as % and two hex digits. encodeURIComponent does that for one piece of a URL, such as a query value, and decodeURIComponent reverses it.

const query = "café & crème?";
const encoded = encodeURIComponent(query);

console.log(encoded);
console.log(decodeURIComponent(encoded) === query);
console.log(encodeURIComponent("😀"));
console.log(encodeURI("/search?q=café"));
console.log(encodeURIComponent("/search?q=café"));

caf%C3%A9%20%26%20cr%C3%A8me%3F
true
%F0%9F%98%80
/search?q=caf%C3%A9
%2Fsearch%3Fq%3Dcaf%C3%A9

The emoji becomes four bytes, as in the table above. The last two lines show the difference between the two encoders. encodeURI is for a whole URL, so it keeps the characters that give a URL its structure (/, ?, &, =, #). encodeURIComponent is for a single value, so it escapes them too. Use it for any text that a user typed, or the & in a name will split your query in two.

Both functions throw a URIError for input that has no valid encoding, and so does decodeURIComponent for a stray %. A lone surrogate is the same broken text that isWellFormed finds:

for (const attempt of [
  () => decodeURIComponent("100%"),
  () => encodeURIComponent("\uD800"),
]) {
  try {
    attempt();
  } catch (error) {
    console.log(error.name, error.message);
  }
}

URIError URI malformed
URIError URI malformed

Decoding text from a request or a link is a place where this crash reaches a server, so wrap it in try/catch. When you build or read whole URLs, the URL and URLSearchParams classes in the files and paths lesson do the encoding for you.

Bit operators you will meet

Bytes bring the bitwise operators. You will not need them often, but you will read them in other people’s code: & (and), | (or), ^ (exclusive or), ~ (not), <<, >> and >>> (shifts). They convert their operands to 32-bit integers first. A common use is a set of on/off flags stored in one number:

const READ = 1;
const WRITE = 2;
const EXEC = 4;

let mode = READ | WRITE;
console.log(mode, (mode & WRITE) !== 0, (mode & EXEC) !== 0);
mode ^= READ;
console.log(mode);

console.log(3.7 | 0, ~~-3.7, -1 >>> 0, 1 << 3, -16 >> 2, 2 ** 10);

3 true false
2
3 -3 4294967295 8 -4 1024

| sets a flag, & tests it, and ^ toggles it. The last line shows idioms from older code. x | 0 and ~~x cut off the fraction, as Math.trunc does for numbers in the 32-bit range, and x >>> 0 turns a number into an unsigned 32-bit one. 1 << 3 is 8, and -16 >> 2 is -4. The ** operator in the same line is not a bit operator: it raises to a power. Prefer Math.trunc and ** when you write new code.

With bytes, & can look at the top bits. In UTF-8 a continuation byte starts with the bits 10, so counting the bytes that do not is a way to count characters without decoding:

const bytes = new TextEncoder().encode("a€😀");

// A continuation byte looks like 10xxxxxx. Every other byte starts a character.
const starts = bytes.filter((byte) => (byte & 0b11000000) !== 0b10000000);
console.log(bytes.length, starts.length);

8 3

The text is 8 bytes and 3 code points.

Common pitfalls

• Using length as the number of characters. It counts code units. Count graphemes for user-visible length.
• Taking text[0] or slice(0, n) of text with emoji. It can cut a pair in half. Use Intl.Segmenter to take the first user-visible character.
• Reversing with split("").reverse().join(""). It scrambles surrogate pairs and combining marks. Reverse graphemes instead.
• Comparing accented text without normalize(). Two equal-looking strings can differ.
• Using NFKC for storage. It loses distinctions. Use NFC for storage and comparisons, and the K forms for search.
• Using \w or [a-z] for names. They only match ASCII letters. Use \p{L} with the u or v flag.
• Mixing up characters and bytes. length is not a size in bytes.

Exercises

const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });

function firstChar(text) {
  const [first] = segmenter.segment(text);
  return first?.segment ?? "";
}

function charCount(text) {
  return [...segmenter.segment(text)].length;
}

for (const s of ["😀 party", "👩‍💻 dev", "🇯🇵 Tokyo", "école"]) {
  console.log(firstChar(s), charCount(s), s.length);
}

😀 7 8
👩‍💻 5 9
🇯🇵 7 10
é 5 6

function slugify(title) {
  return title
    .normalize("NFD")
    .replace(/\p{M}/gu, "")
    .toLowerCase()
    .replace(/[^a-z0-9]+/g, "-")
    .replace(/^-+|-+$/g, "");
}

console.log(slugify("Crème Brûlée à la Carte!"));
console.log(slugify("  Año nuevo -- 2027  "));

creme-brulee-a-la-carte
ano-nuevo-2027

const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });

function truncate(text, max) {
  const parts = Array.from(segmenter.segment(text), (p) => p.segment);
  if (parts.length <= max) {
    return text;
  }
  return parts.slice(0, max - 1).join("") + "…";
}

console.log(truncate("Hello, world", 8));
console.log(truncate("👨‍👩‍👧👨‍👩‍👧👨‍👩‍👧👨‍👩‍👧", 3));
console.log(truncate("short", 8));

Hello, …
👨‍👩‍👧👨‍👩‍👧…
short

const encoder = new TextEncoder();

function cutBytes(text, maxBytes) {
  let used = 0;
  let result = "";
  for (const ch of text) {
    const size = encoder.encode(ch).length;
    if (used + size > maxBytes) {
      break;
    }
    used += size;
    result += ch;
  }
  return result;
}

const cut = cutBytes("Héllo wörld 😀", 10);
console.log(cut, encoder.encode(cut).length);
console.log(cutBytes("ab😀", 5), cutBytes("ab😀", 6));

Héllo wö 10
ab ab😀

Summary

• A string is a sequence of UTF-16 code units. length and indexing use them; for...of, spread and codePointAt use code points.
• A visible character can be several code points. Intl.Segmenter with granularity: "grapheme" counts and splits them correctly.
• normalize("NFC") makes spellings comparable. NFD plus \p{M} strips accents. The K forms are for search.
• isWellFormed and toWellFormed find and repair lone surrogates, which some APIs reject.
• TextEncoder gives UTF-8 bytes. Characters, code units and bytes are three different counts.
• encodeURIComponent percent-encodes the UTF-8 bytes of one URL value, and throws a URIError on broken text. Bit operators work on 32-bit integers.

Next: Intl, for numbers, dates, lists and plurals.