Generators (Python) | Programming Tutorials

The iterator classes in the previous lesson work, but they take a dozen lines to say “count down from n”. A generator gets the same result from an ordinary-looking function. It is also the tool that lets a program handle a million log lines, or a stream that never ends, without ever holding them all at once.

yield turns a function into a generator

If a function body contains yield anywhere, calling the function does not run the body. It returns a generator, which is an iterator that will run the body for you, a piece at a time:

def countdown(n):
    print("starting")
    while n > 0:
        yield n
        n -= 1
    print("done")


gen = countdown(3)
print(type(gen).__name__)
print("created")

Output

generator
created

Notice that starting was not printed. Nothing in the body has run yet. The first next() runs the body up to the first yield, hands that value out, and pauses right there, with all local variables intact:

print(next(gen))
print(next(gen))
print(list(gen))

Output

starting
3
2
done
[1]

Read the output in order. The first next printed starting and gave back 3. The second resumed after the yield, ran n -= 1, looped, and gave back 2. Then list(gen) resumed once more: it produced 1, ran off the end of the function (which printed done), and the generator signalled StopIteration for you. That is why done appears before the list [1] is printed, since the list is only printed once it is complete.

Because a generator is an iterator, everything from the previous lesson applies, including the rule that it can be walked only once:

def numbers():
    yield 1
    yield 2
    yield 3


gen = numbers()
print(sum(gen))
print(sum(gen))
print(list(numbers()), list(numbers()))

Output

6
0
[1, 2, 3] [1, 2, 3]

The function is reusable, since each call makes a new generator. The generator object is not.

Lazy and infinite sequences

Because values appear only when asked for, a generator can describe a sequence with no end. This one produces the Fibonacci numbers for as long as anyone keeps asking:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


for position, value in zip(range(10), fibonacci()):
    print(position, value)

Output

while True would hang a normal function. Here it is safe, because the loop is only advanced when zip asks for another value, and zip stops after range(10) runs out. The consumer decides when to stop. The itertools lesson has islice, which takes the first few items of an iterator more directly.

Generator expressions

You met comprehensions in the comprehensions lesson, including a short mention of generator expressions. Swap the square brackets of a list comprehension for parentheses and you get a generator instead of a list. The difference is when the work happens:

def noisy(n):
    for i in range(n):
        print("making", i)
        yield i


squares_list = [x * x for x in noisy(3)]
print("list built")

squares_gen = (x * x for x in noisy(3))
print("generator built")
print(next(squares_gen))

Output

making 0
making 1
making 2
list built
generator built
making 0
0

The list comprehension did all its work before list built was printed. The generator expression did none until next asked for one value. That has two practical effects: it uses almost no memory however long the source is, and it can stop early.

When a generator expression is the only argument to a function, the extra pair of parentheses can be dropped:

print("sum:", sum(x * x for x in range(1, 4)))
print(any(x > 2 for x in noisy(10)))

Output

sum: 14
making 0
making 1
making 2
making 3
True

any stopped as soon as it saw a value above 2, so only four of the ten numbers were ever made. A list comprehension inside any([...]) would have made all ten first. Use a generator expression when the result is consumed once by sum, min, max, any, all, join and similar. Use a list when you need to index it, measure it with len, or loop over it again.

yield from

A generator often needs to hand out everything from another iterable. You could write for item in other: yield item, but yield from other says it in one line:

def header_then_rows(header, rows):
    yield header
    yield from rows


print(list(header_then_rows("name", ["ada", "grace"])))

Output

['name', 'ada', 'grace']

yield from also works with another generator, which makes recursive generators short. This one flattens lists nested to any depth:

def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item


print(list(flatten([1, [2, [3, 4]], 5, [[6]]])))

Output

[1, 2, 3, 4, 5, 6]

Each nested list is handled by its own flatten generator, and its values pass straight through to the caller.

Pipelines

The real strength comes from chaining generators, each doing one job and passing values to the next. Below, three small generators read, filter and extract. The read_lines stage prints what it has read, so you can see the order in which the work happens:

LOG = """\
2026-10-09 08:00:01 INFO server started
2026-10-09 08:00:05 ERROR disk almost full
2026-10-09 08:01:12 INFO request ok
2026-10-09 08:02:40 ERROR timeout talking to db
2026-10-09 08:03:00 WARN slow request
2026-10-09 08:03:30 ERROR timeout talking to db
"""


def read_lines(text):
    for line in text.splitlines():
        print("  read:", line[:19])
        yield line


def only_level(lines, level):
    for line in lines:
        if f" {level} " in line:
            yield line


def messages(lines):
    for line in lines:
        yield line.split(" ", 3)[3]


errors = messages(only_level(read_lines(LOG), "ERROR"))
print(next(errors))
print(next(errors))

Output

  read: 2026-10-09 08:00:01
  read: 2026-10-09 08:00:05
disk almost full
  read: 2026-10-09 08:01:12
  read: 2026-10-09 08:02:40
timeout talking to db

Building errors does no work at all. Each next(errors) pulls one line at a time through all three stages. The first call had to read two lines to find an error, and the last two lines of the log were never touched. With a real log file in place of the string, you could process a file larger than your memory this way, because only one line exists at a time.

Common pitfalls

Using a generator twice. The second pass is silently empty (see the numbers example above). If you need two passes, build a list, or call the generator function again.
Expecting len() or indexing. len(gen) and gen[0] raise TypeError. Convert with list(gen) if you really need them, remembering that this uses the memory you were saving.
Surprise timing of side effects. Code in a generator body runs when the value is requested, not when the generator is created. Errors in the body, such as a missing file, appear at the first next, perhaps far from the line that built the generator.
Calling list() on an infinite generator. list(fibonacci()) never finishes and eats memory. Always put a limit on infinite sources.

ExerciseCount error messages lazily

Using the three generators from this lesson (read_lines, only_level, messages, without the print in read_lines), count how many times each error message occurs in LOG, and print the counts from most to least frequent as count message. Use collections.Counter. Expected output:

2 timeout talking to db
1 disk almost full

Show solution

from collections import Counter

LOG = """\
2026-10-09 08:00:01 INFO server started
2026-10-09 08:00:05 ERROR disk almost full
2026-10-09 08:01:12 INFO request ok
2026-10-09 08:02:40 ERROR timeout talking to db
2026-10-09 08:03:00 WARN slow request
2026-10-09 08:03:30 ERROR timeout talking to db
"""


def read_lines(text):
    for line in text.splitlines():
        yield line


def only_level(lines, level):
    for line in lines:
        if f" {level} " in line:
            yield line


def messages(lines):
    for line in lines:
        yield line.split(" ", 3)[3]


errors = messages(only_level(read_lines(LOG), "ERROR"))
counts = Counter(errors)
for message, count in counts.most_common():
    print(count, message)

Output

2 timeout talking to db
1 disk almost full

Counter accepts any iterable, so the pipeline feeds it directly and no list of all messages is ever built. most_common() sorts by count, highest first.

ExercisePredict the order

What does this print? Write down the lines before you run it.

def letters():
    print("a")
    yield "x"
    print("b")
    yield "y"
    print("c")


gen = letters()
print("start")
print(next(gen))
print("middle")
print(next(gen))

Show solution

start
a
x
middle
b
y

Creating gen prints nothing. start comes first. The first next runs the body up to the first yield, printing a, and gives back x. After middle, the second next resumes, prints b and gives back y. The print("c") never runs, because nobody asked for a third value.

ExerciseShorten with yield from

This generator flattens a list of pairs into single values. Write pairs_from(rows) that does the same job for rows of any length, using yield from.

Input and expected output, printing list(...) of each call:

list(pairs([(1, 2), (3, 4)]))       -> [1, 2, 3, 4]
list(pairs_from([(1, 2), (3, 4)]))  -> [1, 2, 3, 4]
list(pairs_from([(1, 2, 3), (), (4,)]))  -> [1, 2, 3, 4]

The last call has rows of length 3, 0 and 1, which pairs cannot handle.

def pairs(rows):
    for a, b in rows:
        yield a
        yield b

Show solution

def pairs(rows):
    for a, b in rows:
        yield a
        yield b


def pairs_from(rows):
    for row in rows:
        yield from row


print(list(pairs([(1, 2), (3, 4)])))
print(list(pairs_from([(1, 2), (3, 4)])))
print(list(pairs_from([(1, 2, 3), (), (4,)])))

Output

[1, 2, 3, 4]
[1, 2, 3, 4]
[1, 2, 3, 4]

yield from row hands out each element of the row in turn, so the function no longer depends on every row having exactly two items.

Summary

A function with yield returns a generator, whose body runs in pieces and pauses at each yield, keeping its local variables.
Generator expressions, (expr for x in source), are the lazy form of a list comprehension. yield from hands over everything from another iterable.
Chains of generators form pipelines that process one item at a time. Next: writing your own context managers with a generator.
Going further: the advanced lesson asyncio basics shows how the same pause-and-resume idea powers coroutines.