Unicode, bytes and encodings

A str holds text. A bytes object holds raw numbers from 0 to 255. Files, networks and terminals only deal in bytes, so every time text goes in or out of your program something converts between the two. When that conversion is wrong, you get garbled characters.

The fundamentals lesson on text explains code points and UTF-8 in general. This lesson shows the Python tools.

str and bytes

A Python str is a sequence of Unicode code points. encode() turns a str into bytes, and decode() turns bytes back into a str. Both take an encoding name, and UTF-8 is the right choice almost everywhere.

word = "café"
data = word.encode("utf-8")
print(len(word), len(data))
print(data)
print(data.decode("utf-8"))

4 5
b'caf\xc3\xa9'
café

"café" has 4 characters but 5 bytes in UTF-8, because é takes two bytes (\xc3\xa9). “Length” depends on what you count.

A code point is the number Unicode gives to a character. ord() gives a character’s code point, and chr() goes the other way. You can write a character in source directly, or by name with \N{...}.

print(ord("a"), ord("é"), ord("€"))
print(chr(233), chr(0x20AC), "\N{GREEK SMALL LETTER LAMDA}")

97 233 8364
é € λ

Indexing a bytes object gives an int, not a one-character string. Slicing gives bytes.

data = "hi".encode("utf-8")
print(data[0], data[1])
print(b"abc"[:2])

104 105
b'ab'

Other encodings and errors

Encoding with a different codec produces different bytes. Latin-1 stores é in one byte:

print("café".encode("latin-1"))

b'caf\xe9'

But Latin-1 cannot represent € at all. By default that stops the program with an error:

print("€".encode("latin-1"))

UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

The errors="replace" option substitutes ? instead, which loses information, so use it only when you accept that.

print("€".encode("latin-1", errors="replace"))

b'?'

Mojibake: decoding with the wrong codec

Decoding bytes with a different encoding than the one that wrote them produces mojibake, text such as é where é should be. The damage is often reversible, as long as no step lost data.

original = "naïve café"
data = original.encode("utf-8")
wrong = data.decode("latin-1")
print(wrong)

naïve café

Latin-1 maps every byte to a character, so it never fails, which is exactly why it hides mistakes. Encoding the wrong text back with Latin-1 recovers the original bytes, and decoding those as UTF-8 repairs it:

original = "naïve café"
wrong = original.encode("utf-8").decode("latin-1")
fixed = wrong.encode("latin-1").decode("utf-8")
print(fixed == original)
print(len(original), len(original.encode("utf-8")))

True
10 12

The better fix is to decode correctly in the first place.

Files and the default encoding

When you open a text file, Python decodes the bytes for you, and it needs to know the encoding. Always say so with encoding="utf-8". The pathlib module gives you a Path object for a file, with write_text and read_text methods. This example uses a file in your system’s temporary folder and deletes it at the end.

import tempfile
from pathlib import Path

path = Path(tempfile.gettempdir()) / "unicode_lesson_note.txt"
path.write_text("naïve café\n", encoding="utf-8")
print(path.read_text(encoding="utf-8").strip())
print(path.stat().st_size)
path.unlink()

naïve café
13

The file holds 13 bytes for 11 characters including the newline: two characters, ï and é, use two bytes each. unlink() deletes the file again so the example leaves nothing behind.

Common pitfalls

• Mixing str and bytes. "abc" + b"def" raises TypeError. Decide which one you have and convert at the boundary.
• Omitting encoding= when opening files. Code that works on your machine may fail or garble text on another, unless it runs on 3.15 or later.
• Counting bytes when you meant characters. Use len(text) for characters and len(text.encode("utf-8")) for bytes.
• Silencing errors. errors="ignore" and errors="replace" hide problems. Prefer fixing the encoding.
• Believing one character is one code point. Some visible characters, such as certain accented letters and emoji sequences, are several code points. Counting code points with len() is usually good enough for beginners.

text = "naïve café"
print(len(text), len(text.encode("utf-8")))

10 12

garbled = "naïve café"
print(garbled.encode("latin-1").decode("utf-8"))

naïve café

Summary

• str is text, bytes is data. encode() and decode() convert, and UTF-8 is the default choice.
• Wrong-codec decoding gives mojibake. Pass encoding="utf-8" whenever you open or write a text file.
• 3.15+Requires version 3.15 or later open() defaults to UTF-8 everywhere, but explicit is still better.

That completes the module on numbers and text. Next comes control flow, starting with booleans and truthiness.