Text, characters and encodings

Text feels like the simplest kind of data and is among the trickiest. The word “length” turns out to have three plausible meanings, and a mismatch between them has caused garbled names and broken file reads in nearly every system ever written.

Strings

A string is a sequence of characters treated as one value. You write one with quotes. Strings have an order, so each character has a position, its index. Most languages count from zero.

text:   h   e   l   l   o
index:  0   1   2   3   4

The first character is at index 0 and the last at index length minus one. Asking for an index beyond the end is an error in most languages. Several (Python, for example) also allow negative indexes from the end: -1 is the last character.

text = "hello"
print(text[0], text[4], len(text))
print(text[-1])

h o 5
o

const text = "hello";
console.log(text[0], text[4], text.length);
console.log(text.at(-1));

h o 5
o

public class Indexing {
    public static void main(String[] args) {
        String text = "hello";
        System.out.println(text.charAt(0) + " " + text.charAt(4) + " " + text.length());
        System.out.println(text.charAt(text.length() - 1));
    }
}

h o 5
o

#include <stdio.h>
#include <string.h>

int main(void) {
    const char *text = "hello";
    printf("%c %c %zu\n", text[0], text[4], strlen(text));
    printf("%c\n", text[strlen(text) - 1]);
    return 0;
}

h o 5
o

Immutability

In most languages strings are immutable: you cannot change a character in place. Operations that appear to modify a string actually build a new one. To turn cat into bat, build a new string and rebind the name:

word = "cat"
# word[0] = "b"        # TypeError: strings do not support item assignment
word = "b" + word[1:]
print(word)

let word = "cat";
word[0] = "x";   // silently ignored in sloppy mode; an error in strict mode
word = "b" + word.slice(1);
console.log(word);

public class Immutable {
    public static void main(String[] args) {
        String word = "cat";
        word = "b" + word.substring(1);   // a new String; the original is untouched
        System.out.println(word);
    }
}

#include <stdio.h>

int main(void) {
    char word[] = "cat";   /* a mutable array of characters */
    word[0] = 'b';
    printf("%s\n", word);
    return 0;
}

All four print bat.

Characters, code points and bytes

Here is the heart of the lesson. Three different units are in play:

Unicode is a catalogue that assigns every symbol, from Latin to Chinese to emoji, a unique code point. It says nothing about bytes. An encoding is the rule for turning code points into bytes.

The dominant encoding is UTF-8. It uses 1 byte for the original ASCII characters (plain English letters, digits, punctuation) and 2 to 4 bytes for everything else. It is compact for English, handles every language, and is the standard for files and the web.

character   code point   UTF-8 bytes (hex)
a           U+0061       61
é           U+00E9       C3 A9
€           U+20AC       E2 82 AC
😀          U+1F600      F0 9F 98 80

How languages count

The text “naïve café” has 10 characters, but ï and é each need two bytes in UTF-8, so it occupies 12 bytes. The same visible text yields different numbers depending on how the language defines string length. Each program counts “naïve café” and one emoji. The source files must be saved as UTF-8.

word = "naïve café"
print(len(word))
print(len(word.encode("utf-8")))
print(len("😀"))
print(len("😀".encode("utf-8")))

10
12
1
4

const word = "naïve café";
console.log(word.length);
console.log(new TextEncoder().encode(word).length);
console.log("😀".length);
console.log(new TextEncoder().encode("😀").length);
console.log([..."😀"].length);

10
12
2
4
1

import java.nio.charset.StandardCharsets;

public class TextCounts {
    public static void main(String[] args) {
        String word = "naïve café";
        String emoji = "😀";
        System.out.println(word.length());
        System.out.println(word.getBytes(StandardCharsets.UTF_8).length);
        System.out.println(emoji.length());
        System.out.println(emoji.getBytes(StandardCharsets.UTF_8).length);
        System.out.println(emoji.codePointCount(0, emoji.length()));
    }
}

10
12
2
4
1

#include <stdio.h>
#include <string.h>

/* Count code points by skipping UTF-8 continuation bytes (10xxxxxx). */
static int code_points(const char *s) {
    int count = 0;
    for (; *s; s++) {
        if (((unsigned char)*s & 0xC0) != 0x80) {
            count++;
        }
    }
    return count;
}

int main(void) {
    const char *word = "naïve café";
    const char *emoji = "😀";
    printf("%d\n", code_points(word));
    printf("%zu\n", strlen(word));
    printf("%d\n", code_points(emoji));
    printf("%zu\n", strlen(emoji));
    return 0;
}

10
12
1
4

So the emoji is 1 code point, 2 UTF-16 units, and 4 UTF-8 bytes. None of those is wrong; they answer different questions.

One more wrinkle: what a reader sees as one character can be several code points. The accented “é” can be the single code point U+00E9 or an “e” followed by a combining accent U+0301. They look identical, yet compare as unequal. Text from different sources should be normalised to a single form before comparing, which most languages support in their standard library.

import unicodedata

composed = "\u00e9"
decomposed = "e\u0301"
print(composed == decomposed, len(composed), len(decomposed))
print(composed == unicodedata.normalize("NFC", decomposed))

False 1 2
True

const composed = "\u00e9";
const decomposed = "e\u0301";
console.log(composed === decomposed, composed.length, decomposed.length);
console.log(composed === decomposed.normalize("NFC"));

false 1 2
true

import java.text.Normalizer;

public class Normalise {
    public static void main(String[] args) {
        String composed = "\u00e9";
        String decomposed = "e\u0301";
        System.out.println(composed.equals(decomposed) + " " + composed.length() + " " + decomposed.length());
        System.out.println(composed.equals(Normalizer.normalize(decomposed, Normalizer.Form.NFC)));
    }
}

false 1 2
true

#include <stdio.h>
#include <string.h>

int main(void) {
    const char *composed = "\xc3\xa9";       /* U+00E9 as UTF-8 */
    const char *decomposed = "e\xcc\x81";    /* e + U+0301 as UTF-8 */
    printf("%d %zu %zu\n", strcmp(composed, decomposed) == 0, strlen(composed), strlen(decomposed));
    return 0;
}

0 2 3

Why encodings matter

A file is just bytes. To read it as text you must know the encoding it was written in. Decode UTF-8 bytes as an older encoding such as Latin-1, and é (the two bytes C3 A9) turns into the two characters é. That garbling is called mojibake. Decode the wrong way round, and you may even hit an outright error because some byte sequences are not valid in the encoding.

The practical rule: use UTF-8 for everything you write, and state the encoding explicitly whenever you read or write a file or network data rather than trusting a default.

Escaping

Some characters cannot be typed directly inside quotes. A backslash begins an escape sequence:

Backslashes are also why Windows paths in code surprise people: "C:\new" contains a newline. Double the backslashes or use your language’s raw-string form.

Joining and formatting

Joining strings is called concatenation, usually with +. It works for a few pieces, but it gets clumsy, and in most languages you must convert numbers to strings first. C has no + for text at all.

Formatting inserts values into a template and controls layout. Python has f-strings, JavaScript has template literals plus padding methods, and Java and C share the printf family of templates. Each program builds one receipt line: the name left-aligned in 10 columns, the quantity right-aligned in 3, and two prices with two decimals.

name, qty, price = "Tea", 3, 2.5
print(f"{name:<10}{qty:>3} x {price:>6.2f} = {qty * price:>7.2f}")

const name = "Tea";
const qty = 3;
const price = 2.5;
console.log(
  `${name.padEnd(10)}${String(qty).padStart(3)} x ${price.toFixed(2).padStart(6)} = ${(qty * price).toFixed(2).padStart(7)}`
);

public class Receipt {
    public static void main(String[] args) {
        String name = "Tea";
        int qty = 3;
        double price = 2.5;
        System.out.println(String.format("%-10s%3d x %6.2f = %7.2f", name, qty, price, qty * price));
    }
}

#include <stdio.h>

int main(void) {
    const char *name = "Tea";
    int qty = 3;
    double price = 2.5;
    printf("%-10s%3d x %6.2f = %7.2f\n", name, qty, price, qty * price);
    return 0;
}

All four print:

Tea         3 x   2.50 =    7.50

Prefer formatting once you are inserting more than one value or care about alignment and decimals. If you build a very long string by adding one piece at a time in a loop, collect the pieces and join them once at the end; many languages copy the whole string at every +.

Common pitfalls

• Assuming one character is one byte. True only for ASCII.
• Using byte length to mean display length (or the reverse), for example when truncating a name to fit a database column.
• Not stating the encoding when opening files, so code works on your machine and fails on a colleague’s.
• Comparing visually identical strings that use different code point sequences.
• Treating a string as editable in a language where it is immutable.
• Forgetting to convert numbers before concatenating.

name, qty, price = "Pen", 12, 0.75
print(f"{name:<10}{qty:>3} x {price:>6.2f} = {qty * price:>7.2f}")

const name = "Pen";
const qty = 12;
const price = 0.75;
console.log(
  `${name.padEnd(10)}${String(qty).padStart(3)} x ${price.toFixed(2).padStart(6)} = ${(qty * price).toFixed(2).padStart(7)}`
);

public class ReceiptPen {
    public static void main(String[] args) {
        String name = "Pen";
        int qty = 12;
        double price = 0.75;
        System.out.println(String.format("%-10s%3d x %6.2f = %7.2f", name, qty, price, qty * price));
    }
}

#include <stdio.h>

int main(void) {
    const char *name = "Pen";
    int qty = 12;
    double price = 0.75;
    printf("%-10s%3d x %6.2f = %7.2f\n", name, qty, price, qty * price);
    return 0;
}

s = "hello"
s = "j" + s[1:]
print(s)

let s = "hello";
s = "j" + s.slice(1);
console.log(s);

public class Jello {
    public static void main(String[] args) {
        String s = "hello";
        s = "j" + s.substring(1);
        System.out.println(s);
    }
}

#include <stdio.h>

int main(void) {
    char s[] = "hello";
    s[0] = 'j';
    printf("%s\n", s);
    return 0;
}

Summary

• A string is an ordered, usually immutable sequence; indexes normally start at 0.
• Characters, code points and bytes are different units: Unicode assigns code points, and an encoding such as UTF-8 turns them into bytes.
• Always state the encoding when reading or writing, and use formatting to assemble text.

This completes the first module of the shared track. The next module covers control flow and functions.