Descriptors

Metaprogramming is code that works on code: classes that shape other classes, functions that read other functions. This module covers the tools Python gives you for it, starting with the one you have used most without seeing it. @property from methods and properties, every method call, and functools.cached_property all run through the same small protocol: descriptors. You need the data model lesson and decorators for this module; Annotations at runtime helps for the later lessons.

A property is an object in the class

When you write @property, the class gets an attribute that is a property object. That object has a __get__ method, and Python calls it when you read the attribute from an instance:

class Account:
    def __init__(self, balance):
        self._balance = balance

    @property
    def balance(self):
        return self._balance


attr = Account.__dict__["balance"]
print(type(attr).__name__)
print(hasattr(attr, "__get__"), hasattr(attr, "__set__"))
account = Account(50)
print(attr.__get__(account, Account))
print(account.balance)

property
True True
50
50

Calling attr.__get__(account, Account) by hand gives the same result as account.balance. That is the whole trick. Any object stored in a class that defines __get__, __set__ or __delete__ is a descriptor, and Python hands attribute access on instances over to it.

Writing a descriptor

__get__(self, obj, objtype) receives the descriptor itself, the instance the attribute was read from, and the class. When the attribute is read from the class rather than an instance, obj is None:

class Loud:
    def __get__(self, obj, objtype=None):
        print(f"__get__(obj={obj!r}, objtype={objtype.__name__})")
        return 42


class Thing:
    answer = Loud()

    def __repr__(self):
        return "Thing()"


print(Thing().answer)
print(Thing.answer)

__get__(obj=Thing(), objtype=Thing)
42
__get__(obj=None, objtype=Thing)
42

Most descriptors return self when obj is None, so that Thing.answer gives back the descriptor object for inspection. You will see that line in every descriptor below.

A validated field with __set_name__

A descriptor that also defines __set__ controls assignment. That makes it a reusable version of a property setter: write the check once and use it on any number of attributes.

The descriptor needs to know which attribute it is, both for the error message and for where to store the value. __set_name__(self, owner, name) tells it: Python calls it once, when the class body finishes, with the class and the attribute name.

class Positive:
    def __set_name__(self, owner, name):
        self.name = name
        self.private = "_" + name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return getattr(obj, self.private)

    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError(f"{self.name} must be > 0, got {value}")
        setattr(obj, self.private, value)

class Order:
    quantity = Positive()
    price = Positive()

    def __init__(self, quantity, price):
        self.quantity = quantity
        self.price = price


order = Order(3, 9.5)
print(order.quantity, order.price, vars(order))
try:
    order.price = -1
except ValueError as error:
    print("ValueError:", error)

3 9.5 {'_quantity': 3, '_price': 9.5}
ValueError: price must be > 0, got -1

vars(order) shows where the values live: in the instance, under _quantity and _price. The descriptor object is shared by every Order, so it must never keep per-instance data on self. One descriptor class now validates two fields, where properties would have needed two getters and two setters.

The lookup order

When you read obj.name, Python calls type(obj).__getattribute__(obj, "name"). The default version, object.__getattribute__, looks in three places, in this order:

1. A data descriptor on the class (one with __set__ or __delete__).
2. The instance’s own __dict__.
3. A non-data descriptor (only __get__) or a plain class attribute.

If all three fail with AttributeError, there is a fourth step: Python calls the class’s __getattr__, if it has one. That fallback is the subject of a later section.

So a data descriptor wins over the instance dictionary, and the instance dictionary wins over a non-data descriptor:

class Data:
    def __get__(self, obj, objtype=None):
        return "from the data descriptor"

    def __set__(self, obj, value):
        raise AttributeError("read-only")


class NonData:
    def __get__(self, obj, objtype=None):
        return "from the non-data descriptor"


class Box:
    a = Data()
    b = NonData()


box = Box()
box.__dict__["a"] = "from the instance"
box.__dict__["b"] = "from the instance"
print(box.a)
print(box.b)

from the data descriptor
from the instance

This order explains several things you already know. A property always runs, even if the instance dictionary has an entry with the same name, because property defines __set__ (it raises if there is no setter). And methods can be replaced per instance, because functions are non-data descriptors.

Methods are descriptors

A function stored in a class is a descriptor. Its __get__ returns a bound method: the function with the instance filled in as self. That is how greeter.hello() gets its self argument:

class Greeter:
    def hello(self):
        return f"hello from {type(self).__name__}"


func = Greeter.__dict__["hello"]
print(type(func).__name__, hasattr(func, "__get__"))
bound = func.__get__(Greeter(), Greeter)
print(type(bound).__name__, bound())
print(Greeter.hello is func)

function True
method hello from Greeter
True

Read from the class, obj is None, and the function returns itself. classmethod and staticmethod are descriptors too: theirs return a function bound to the class, or the plain function.

Caching with a non-data descriptor

Because the instance dictionary beats a non-data descriptor, a descriptor can compute a value once, store it in the instance under its own name, and never be called again for that instance:

class lazy:
    def __init__(self, func):
        self.func = func

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        value = self.func(obj)
        obj.__dict__[self.name] = value
        return value

class Report:
    def __init__(self, rows):
        self.rows = rows

    @lazy
    def total(self):
        print("computing total")
        return sum(self.rows)


report = Report([3, 4, 5])
print(report.total)
print(report.total)
print("total" in vars(report))

computing total
12
12
True

The second read found total in the instance dictionary and skipped the descriptor. The standard library ships this as functools.cached_property, so use that in real code. Since it stores into __dict__, it does not work on classes with __slots__ and no __dict__.

When lookup fails: __getattr__

__getattr__(self, name) runs only after normal lookup has failed. Attributes that exist are found as usual and never reach it. That makes it the tool for proxies that forward unknown names to another object:

class Recorder:
    def __init__(self, target):
        self._target = target
        self.calls = []

    def __getattr__(self, name):
        self.calls.append(name)
        return getattr(self._target, name)


words = Recorder(["b", "a"])
words.append("c")
words.sort()
print(words._target, words.calls)
try:
    words.missing
except AttributeError as error:
    print("AttributeError:", error)
try:
    len(words)
except TypeError as error:
    print("TypeError:", error)

['a', 'b', 'c'] ['append', 'sort']
AttributeError: 'list' object has no attribute 'missing'
TypeError: object of type 'Recorder' has no len()

append and sort went through __getattr__ and were recorded. _target and calls did not, because normal lookup found them. A name the list lacks still raises AttributeError, as it should. The last line shows the limit: len() looks up __len__ on the class directly, skipping both __getattribute__ and __getattr__, so a proxy must define the special methods it supports.

Every lookup: __getattribute__

Overriding __getattribute__ itself puts your code in front of every attribute read, including methods and attributes that exist:

class Traced:
    def __init__(self):
        self.x = 21

    def __getattribute__(self, name):
        print("looking up", name)
        return object.__getattribute__(self, name)

    def double(self):
        return self.x * 2


t = Traced()
print(t.double())
print(type(t).__name__)

looking up double
looking up x
42
Traced

Inside the method, any self.something would call __getattribute__ again and recurse forever, so it delegates to object.__getattribute__. type(t) does not look up an attribute, so it was not traced. Every attribute read now costs a Python function call. Use __getattr__ when you only need the missing names, which is almost always.

Module attributes: __getattr__ on a module

A module can define a function named __getattr__ too (PEP 562, Python 3.7). It runs when module.name is not found, which is the standard way to keep an old name working while warning that it is deprecated:

import warnings


def kilometres(miles):
    return miles * 1.609344


_RENAMED = {"to_km": "kilometres"}


def __getattr__(name):
    if name in _RENAMED:
        new = _RENAMED[name]
        warnings.warn(f"{name} is deprecated, use {new}",
                      DeprecationWarning, stacklevel=2)
        return globals()[new]
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

import warnings

import units

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    print(units.to_km(10))
print(caught[0].category.__name__, caught[0].message)
print(units.kilometres(1))
try:
    units.to_metres
except AttributeError as error:
    print("AttributeError:", error)

16.09344
DeprecationWarning to_km is deprecated, use kilometres
1.609344
AttributeError: module 'units' has no attribute 'to_metres'

kilometres exists, so it is found directly. to_km does not, so the module’s __getattr__ warns and returns the new function. from units import to_km takes the same path. Code inside units that uses a bare to_km is not covered: the hook only handles attribute access on the module object. The warning is recorded here so that it can be printed; normally Python writes it to standard error.

When not to use a descriptor

Write a descriptor when the same attribute logic repeats across fields or classes. For one attribute in one class, @property is clearer. The same goes for __getattr__: a proxy or a deprecation shim is a good use, but a class whose attributes appear from nowhere defeats autocompletion, dir() and type checkers. If you are validating many fields of a record, a dataclass with a __post_init__ check, or a validation library, is often less code than a descriptor and easier for colleagues to follow.

Common pitfalls

• Storing values on the descriptor. self.value = value in __set__ shares one value across every instance. Store it on obj.
• Forgetting obj is None. Without the check, Class.attr calls your getter with None and usually fails with a confusing AttributeError.
• Expecting a non-data descriptor to validate. Without __set__, assignment just writes to the instance dictionary and the descriptor is bypassed from then on.
• Recursing in __getattr__. If __getattr__ reads an attribute that is not set yet, such as self._target during unpickling or before __init__ assigns it, the read calls __getattr__ again. Guard against it, or raise AttributeError for names starting with _.
• Defining the descriptor after the class exists. __set_name__ only runs during class creation. Class.x = Positive() later needs a manual Class.x.__set_name__(Class, "x").

from typed_field import Typed


class Product:
    name = Typed(str)
    price = Typed(float)

    def __init__(self, name, price):
        self.name = name
        self.price = price


p = Product("pen", 1.5)
print(p.name, p.price)
try:
    p.price = "cheap"
except TypeError as error:
    print("TypeError:", error)
print(Product.price.kind.__name__)

pen 1.5
TypeError: price must be float, got str
float

class Typed:
    def __init__(self, kind):
        self.kind = kind

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj.__dict__[self.name]

    def __set__(self, obj, value):
        if not isinstance(value, self.kind):
            raise TypeError(f"{self.name} must be {self.kind.__name__}, "
                            f"got {type(value).__name__}")
        obj.__dict__[self.name] = value

class Shout:
    def __get__(self, obj, objtype=None):
        return "SHOUT"


class Word:
    text = Shout()

    @property
    def size(self):
        return 3


w = Word()
print(w.text)
w.text = "quiet"
print(w.text)
try:
    w.size = 10
except AttributeError as error:
    print("AttributeError:", error)
print(w.size)

SHOUT
quiet
AttributeError: property 'size' of 'Word' object has no setter
3

class NonNegative:
    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return self.value

    def __set__(self, obj, value):
        if value < 0:
            raise ValueError(f"{self.name} must be >= 0")
        self.value = value


class Account:
    balance = NonNegative()

    def __init__(self, balance):
        self.balance = balance


a = Account(10)
b = Account(99)
print(a.balance, b.balance)

99 99

class NonNegative:
    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj.__dict__[self.name]

    def __set__(self, obj, value):
        if value < 0:
            raise ValueError(f"{self.name} must be >= 0")
        obj.__dict__[self.name] = value


class Account:
    balance = NonNegative()

    def __init__(self, balance):
        self.balance = balance


a = Account(10)
b = Account(99)
print(a.balance, b.balance)

10 99

Summary

• obj.attr checks for a data descriptor on the class, then the instance __dict__, then a non-data descriptor or class attribute.
• A descriptor is a class attribute with __get__, and optionally __set__ or __delete__. __get__ receives obj=None when read from the class.
• __set_name__ tells a descriptor its attribute name when the class is created. Keep per-instance data on the instance, never on the descriptor.
• Properties, methods, classmethod, staticmethod and cached_property are all descriptors.
• __getattr__ runs only when normal lookup fails, on a class or (PEP 562) on a module. __getattribute__ runs for every read. Special methods used by built-ins such as len() bypass both.
• Descriptors and __set_name__ are how many tools read class fields; Annotations at runtime shows the other half, reading the annotations themselves, and Metaclasses shows the machinery that calls __set_name__.
• Next: customising classes, with hooks that run when a class is created.