Closures and higher-order functions

In the beginner course you passed functions around and wrote small lambda expressions (see lambdas and higher-order functions), and you learned where names are looked up in the LEGB rule. Put those two ideas together and you get closures: functions that carry some of their surroundings with them. The general concept is described in the scope and lifetime lesson of the fundamentals track. This lesson covers how Python does it, and the one bug it is famous for.

A function that remembers

When a function is defined inside another function and uses a variable of the outer one, the inner function keeps that variable alive after the outer function has returned:

def make_multiplier(factor):
    def multiply(x):
        return x * factor

    return multiply


double = make_multiplier(2)
triple = make_multiplier(3)
print(double(10), triple(10))

20 30

factor is a free variable of multiply: it is used there but defined in the enclosing function. Each call to make_multiplier creates a separate factor, so double and triple do not interfere. You can inspect what a function has captured:

print(double.__code__.co_freevars)
print(double.__closure__[0].cell_contents)
print(triple.__closure__[0].cell_contents)

('factor',)
2
3

This is for understanding, not for production code. The point is that the captured thing is the variable itself (stored in a “cell”), not a copy of its value at the time of definition. The next sections depend on that difference.

def outer():
    x = "before"

    def show():
        return x

    x = "after"
    return show


print(outer()())

after

show looks up x when it runs, and by then x has been rebound to "after".

nonlocal: changing a captured variable

A closure can read a free variable freely. Assigning to it is another matter, because an assignment inside a function makes the name local, as you saw in the scope lesson. The nonlocal statement says “this name belongs to the nearest enclosing function, so rebind that one”:

def make_counter_broken():
    count = 0

    def increment():
        count += 1
        return count

    return increment


broken = make_counter_broken()
try:
    broken()
except UnboundLocalError:
    print("UnboundLocalError")

UnboundLocalError

count += 1 makes count local to increment, and reading it before it has a value fails. Declaring it nonlocal fixes that:

def make_counter():
    count = 0

    def increment():
        nonlocal count
        count += 1
        return count

    return increment


counter = make_counter()
print(counter(), counter(), counter())
other = make_counter()
print(other())

1 2 3
1

Each counter has its own private count, with no global variable and no class. Closures are a light way to hold a little state. If the state grows past one or two variables, a class is usually clearer.

nonlocal is not global. It refuses names that do not exist in an enclosing function, so a typo is a SyntaxError and not a silent new global. It is needed only for rebinding: changing a captured list with .append needs no declaration, because that is mutation and not assignment.

Functions that take and return functions

A higher-order function takes a function as an argument, returns one, or both. You already use sorted(key=...) and map. Writing your own is mostly a matter of closures:

def compose(*functions):
    """Return a function that applies the given functions left to right."""

    def run(value):
        for function in functions:
            value = function(value)
        return value

    return run


def apply_twice(function, value):
    return function(function(value))


clean = compose(str.strip, str.lower, str.title)
print(clean("  hELLO wORLD  "))
print(apply_twice(lambda s: s + "!", "hi"))
print(apply_twice(clean, "  x  "))

Hello World
hi!!
X

compose returns a closure over the functions tuple. Methods such as str.strip can be passed as plain functions, because str.strip(text) is the same as text.strip(). The next lesson uses exactly this shape to write decorators.

The late-binding trap

Because a closure captures the variable and not its value, creating functions in a loop gives a surprise. A lambda in a list comprehension looks up i when it is called, and by then the loop has finished:

adders = [lambda x: x + i for i in range(3)]
print([f(10) for f in adders])

[12, 12, 12]

All three functions share one i, whose final value is 2. Most people expect [10, 11, 12]. There are three common fixes, and each makes the value get stored at definition time. The shortest binds the current value as a default argument:

adders = [lambda x, i=i: x + i for i in range(3)]
print([f(10) for f in adders])

[10, 11, 12]

A default argument is evaluated when the lambda is created, so each function gets its own copy. The risk is that callers can overwrite i by passing a second argument. A cleaner fix is a factory function, where each call creates a new variable of its own:

def make_adder(i):
    return lambda x: x + i


adders = [make_adder(i) for i in range(3)]
print([f(10) for f in adders])

[10, 11, 12]

The third option is functools.partial, from the previous lesson, which stores the argument immediately and has no such risk of being overridden by position. Writing partial(add, i=i) gives the same output.

Common pitfalls

• Expecting a snapshot. A closure sees the variable’s latest value, not its value when the function was defined. Rebinding after the definition changes the result (see predict_closure.py).
• Assigning without nonlocal. count += 1 in the inner function fails with UnboundLocalError. Add nonlocal count, or restructure.
• Lambdas in loops. Use a default argument, a factory or partial.
• Hidden shared state. A closure over a mutable object, such as a list, shares it with everyone who got the closure. Create the list inside the factory so that each closure has its own.

0 2 4
0
10 20

evens = make_counter(0, 2)
print(evens(), evens(), evens())
evens.reset()
print(evens())
tens = make_counter(10, 10)
print(tens(), tens())

def make_counter(start=0, step=1):
    value = start - step

    def next_value():
        nonlocal value
        value += step
        return value

    def reset():
        nonlocal value
        value = start - step

    next_value.reset = reset
    return next_value


evens = make_counter(0, 2)
print(evens(), evens(), evens())
evens.reset()
print(evens())
tens = make_counter(10, 10)
print(tens(), tens())

0 2 4
0
10 20

names = ["ada", "grace", "alan"]

greeters = [lambda: "Hello, " + name for name in names]
print([g() for g in greeters])

['Hello, ada', 'Hello, grace', 'Hello, alan']

names = ["ada", "grace", "alan"]

greeters = [lambda: "Hello, " + name for name in names]
print([g() for g in greeters])

greeters = [lambda name=name: "Hello, " + name for name in names]
print([g() for g in greeters])

['Hello, alan', 'Hello, alan', 'Hello, alan']
['Hello, ada', 'Hello, grace', 'Hello, alan']

Summary

• A closure is a function plus the variables it uses from enclosing functions. It captures variables, not values.
• nonlocal lets an inner function rebind a captured variable. Mutating a captured object needs no declaration.
• Higher-order functions take or return functions. Fix lambdas-in-loops with a default argument, a factory or partial.
• Next: decorators.