A tour of the C API

Every lesson in this module has pointed at C: the reference counts of memory management, the Py_mod_gil slot of free-threading internals, the extension loaders of the import system. This lesson reads and writes a complete small extension, tally.c, built up over several blocks. You need to read C pointers and know what malloc is for; if you do not, the native extensions lesson covers ctypes, cffi and Cython, which spare you this level. You need a C compiler and the Python headers to build it, but you can follow the code without one.

PyObject

In C, every Python object is reached through a PyObject *. The object starts with a header: on the default build, its reference count and a pointer to its type; on the free-threaded build, the larger header from the previous lessons. Everything else depends on the type. A PyLongObject continues with digits, a PyListObject with an array of PyObject * items.

You never read the header fields directly. Use Py_TYPE(op), Py_INCREF(op), Py_DECREF(op) and Py_NewRef(op) (increment and return), which do the right thing on every build, including for immortal objects.

Owning references

The C API speaks of owning references, not objects. If you own a reference, you must eventually release it with Py_DECREF, or hand that duty to someone else. Each function’s documentation says which of three things happens:

• New reference: the function hands you ownership. Almost every function that returns an object does this, including all PyObject_*, PyNumber_*, PySequence_* and PyMapping_* calls.
• Borrowed reference: you may use the object, but you do not own it and must not release it. PyList_GetItem is the classic case. It is only safe while something else keeps the object alive, which on the free-threaded build another thread can break, so prefer PyList_GetItemRef and PyDict_GetItemRef, which return new references.
• Stolen reference: you pass an object and the function takes over your ownership. PyList_SetItem and PyTuple_SetItem do this.

Errors follow one convention: a function that fails sets an exception and returns NULL (for pointer results) or -1 (for int results). Your function must check every call, release what it owns, and return NULL itself, which makes Python raise the exception.

A function in C

count_true(iterable) counts the truthy items. Notice how each new reference is released on every path:

#include <Python.h>

/* tally.count_true(iterable) -> int: how many items are truthy. */
static PyObject *
count_true(PyObject *module, PyObject *iterable)
{
    PyObject *it = PyObject_GetIter(iterable);    /* new reference */
    if (it == NULL) {
        return NULL;                              /* exception is set */
    }
    long total = 0;
    PyObject *item;
    while ((item = PyIter_Next(it)) != NULL) {    /* new reference */
        int truth = PyObject_IsTrue(item);
        Py_DECREF(item);                          /* done with item */
        if (truth < 0) {
            Py_DECREF(it);
            return NULL;
        }
        total += truth;
    }

PyIter_Next returns NULL both at the end and on an error, so the end of the loop has to ask which one it was:

    Py_DECREF(it);
    if (PyErr_Occurred()) {                       /* iteration failed */
        return NULL;
    }
    return PyLong_FromLong(total);                /* new reference */
}

The returned int is a new reference, and ownership passes to the caller, the interpreter.

Arguments and error paths

scale(seq, factor=2.0) takes two arguments, the second optional and allowed as a keyword, and returns a new list. With METH_VARARGS | METH_KEYWORDS, the function receives a tuple and a dict, and PyArg_ParseTupleAndKeywords unpacks them: O stores an object, | starts the optional ones, d converts to a C double, and the name after : goes into error messages.

/* tally.scale(seq, factor=2.0) -> list of floats: each item * factor. */
static PyObject *
scale(PyObject *module, PyObject *args, PyObject *kwargs)
{
    static char *kwlist[] = {"seq", "factor", NULL};
    PyObject *seq;
    double factor = 2.0;                  /* default if not passed */
    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|d:scale", kwlist,
                                     &seq, &factor)) {
        return NULL;                      /* TypeError is set */
    }
    PyObject *it = PyObject_GetIter(seq);
    if (it == NULL) {
        return NULL;
    }
    PyObject *result = PyList_New(0);
    if (result == NULL) {
        Py_DECREF(it);
        return NULL;
    }

From here the function owns two references, it and result, so every failure must release both. C code does that with one error: label:

    PyObject *item;
    while ((item = PyIter_Next(it)) != NULL) {
        double x = PyFloat_AsDouble(item);
        Py_DECREF(item);
        if (x == -1.0 && PyErr_Occurred()) {
            goto error;                   /* not a number */
        }
        PyObject *value = PyFloat_FromDouble(x * factor);
        if (value == NULL || PyList_Append(result, value) < 0) {
            Py_XDECREF(value);
            goto error;
        }
        Py_DECREF(value);                 /* the list holds its own */
    }
    if (PyErr_Occurred()) {
        goto error;
    }
    Py_DECREF(it);
    return result;

error:                                    /* one exit for every failure */
    Py_DECREF(it);
    Py_DECREF(result);
    return NULL;
}

• PyFloat_AsDouble returns -1.0 both for a real -1.0 and on error, so only PyErr_Occurred() tells them apart.
• PyList_Append does not steal, so value is released after it, while PyList_SetItem would have taken it over.
• Py_XDECREF accepts NULL, which covers a failed PyFloat_FromDouble.

METH_FASTCALL passes the arguments as a C array, with no tuple to build. The parser for that array, PyArg_ParseArray, is new in 3.15 and not in the Limited API, so tally uses the tuple form, which also builds for abi3t below.

The module: slots and an export hook

A module is described by an array of PySlot entries, and the shared library exports one function, PyModExport_<name>, that returns it (PEP 793):

static PyMethodDef tally_methods[] = {
    {"count_true", count_true, METH_O, "Count the truthy items."},
    {"scale", (PyCFunction)(void (*)(void))scale,
     METH_VARARGS | METH_KEYWORDS, "Multiply every item by factor."},
    {NULL, NULL, 0, NULL}
};

PyABIInfo_VAR(abi_info);

static PySlot tally_slots[] = {
    PySlot_STATIC_DATA(Py_mod_abi, &abi_info),
    PySlot_STATIC_DATA(Py_mod_name, "tally"),
    PySlot_STATIC_DATA(Py_mod_doc, "Count things, in C."),
    PySlot_STATIC_DATA(Py_mod_methods, tally_methods),
    PySlot_DATA(Py_mod_gil, Py_MOD_GIL_NOT_USED),
    PySlot_END
};

PyMODEXPORT_FUNC PyModExport_tally(void);

PyMODEXPORT_FUNC
PyModExport_tally(void)
{
    return tally_slots;
}

• METH_O means the C function takes exactly one argument. METH_NOARGS, METH_VARARGS and METH_FASTCALL cover the other shapes. The table’s field is a plain PyCFunction, so scale, which takes a third argument, needs the cast.
• Py_mod_abi with PyABIInfo_VAR lets Python refuse an extension compiled for an incompatible ABI instead of crashing.
• Py_mod_gil is the free-threading promise from the GIL lesson. Neither function keeps shared state, so the promise holds.
• Everything is static except the hook, so nothing clashes with other extensions.

Before 3.15, the export was PyInit_<name>, returning a PyModuleDef. It still works, and is now soft deprecated: no new features, no plans for removal. If a library exports both, PyInit_<name> is ignored, so you can ship one source that supports older versions with PyInit_ and 3.15 with PyModExport_. The extension modules reference has both forms.

The hook name is derived from the module name, with a Punycode form for non-ASCII names:

def hook_name(name):
    try:
        suffix = b"_" + name.encode("ascii")
    except UnicodeEncodeError:
        suffix = b"U_" + name.encode("punycode").replace(b"-", b"_")
    return (b"PyModExport" + suffix).decode("ascii")


for module in ["spam", "tally", "café"]:
    print(f"{module:<6} {hook_name(module)}")

spam   PyModExport_spam
tally  PyModExport_tally
café   PyModExportU_caf_dma

Building and importing

Real projects build with meson-python, scikit-build-core or setuptools. For a one-file experiment on Linux, the compiler is enough:

$ gcc -shared -fPIC $(python3.15-config --includes) tally.c \
      -o tally$(python3.15-config --extension-suffix)
$ python3.15 -c "import tally; print(tally.count_true([0, 1, 'a', '', None, 3]))"
3
$ python3.15 -c "import tally; tally.count_true(5)"
Traceback (most recent call last):
  ...
TypeError: 'int' object is not iterable

The TypeError was set by PyObject_GetIter; count_true only passed the NULL on. scale behaves like a Python function, keyword argument and default included, and each failure arrives as an ordinary exception:

$ python3.15 -c "import tally; print(tally.scale([1, 2.5, True]))"
[2.0, 5.0, 2.0]
$ python3.15 -c "import tally; print(tally.scale(range(3), factor=0.5))"
[0.0, 0.5, 1.0]
$ python3.15 -c "import tally; tally.scale([1, 'two'])"
Traceback (most recent call last):
  ...
TypeError: must be real number, not str
$ python3.15 -c "import tally; tally.scale()"
Traceback (most recent call last):
  ...
TypeError: scale() missing required argument 'seq' (pos 1)

The first message comes from PyFloat_AsDouble, the second from the parser, which used the name after :.

Stable ABIs: abi3 and abi3t

A normal build, tally.cpython-315-x86_64-linux-gnu.so, works on 3.15 only. Defining a target ABI before including Python.h limits you to the Limited API, and in exchange the binary works on that version and every later one:

• Py_LIMITED_API targets abi3, the stable ABI for builds with a GIL (since 3.2).
• Py_TARGET_ABI3T targets abi3t, the stable ABI for free-threaded builds (PEP 803). Defining both gives one binary for both kinds of build, which the stable ABI documentation recommends naming with the abi3t tag.

Each takes a version in hex, 0x030f0000 for 3.15:

$ gcc -shared -fPIC $(python3.15-config --includes) \
      -DPy_LIMITED_API=0x030f0000 -DPy_TARGET_ABI3T=0x030f0000 \
      tally.c -o tally.abi3t.so
$ python3.15 -c "import tally; print(tally.count_true([1, 0, 2]))"
2
$ python3.15t -c "import sys, tally; print(tally.count_true([1, 0, 2]), sys._is_gil_enabled())"
2 False

The suffixes each interpreter accepts, in the order it tries them:

$ python3.15 -c "import importlib.machinery as m; print(m.EXTENSION_SUFFIXES)"
['.cpython-315-x86_64-linux-gnu.so', '.abi3-x86_64-linux-gnu.so', '.abi3.so', '.abi3t-x86_64-linux-gnu.so', '.abi3t.so', '.so']
$ python3.15t -c "import importlib.machinery as m; print(m.EXTENSION_SUFFIXES)"
['.cpython-315t-x86_64-linux-gnu.so', '.abi3t-x86_64-linux-gnu.so', '.abi3t.so', '.so']

The Limited API costs some speed: macros such as PyList_GET_ITEM are unavailable, and calls that would be inlined become function calls. You can ship both: a faster version-specific build alongside a stable-ABI one as the fallback.

When not to write C by hand

Write raw C API code to learn how CPython works, to maintain an existing extension, or for a small hot function with no dependencies. For new bindings to a C library, ctypes or cffi need no compiler at install time. For new compiled code, Cython or Rust with PyO3 manage reference counts for you. A single forgotten Py_DECREF is a leak; a single extra one is a crash, possibly much later.

Common pitfalls

Releasing a borrowed reference. Py_DECREF(PyList_GetItem(...)) frees an object the list still uses.

Missing a release on an error path. Every early return NULL must first release what you own.

Ignoring return values. PyObject_IsTrue returns -1 on error; treating that as “true” hides exceptions.

Declaring Py_MOD_GIL_NOT_USED for code with static state. That is a data race waiting for the free-threaded build.

$ python3.15 -c "import hello; print(hello.greet('Ada'))"
Hello, Ada!
$ python3.15 -c "import hello; hello.greet(3)"
Traceback (most recent call last):
  ...
TypeError: name must be a str
$ python3.15t -c "import sys, hello; print(hello.greet('Ada'), sys._is_gil_enabled())"
Hello, Ada! False

#include <Python.h>

static PyObject *
greet(PyObject *module, PyObject *name)
{
    if (!PyUnicode_Check(name)) {
        PyErr_SetString(PyExc_TypeError, "name must be a str");
        return NULL;
    }
    return PyUnicode_FromFormat("Hello, %U!", name);  /* new reference */
}

static PyMethodDef hello_methods[] = {
    {"greet", greet, METH_O, "Return a greeting."},
    {NULL, NULL, 0, NULL}
};

PyABIInfo_VAR(abi_info);

static PySlot hello_slots[] = {
    PySlot_STATIC_DATA(Py_mod_abi, &abi_info),
    PySlot_STATIC_DATA(Py_mod_name, "hello"),
    PySlot_STATIC_DATA(Py_mod_methods, hello_methods),
    PySlot_DATA(Py_mod_gil, Py_MOD_GIL_NOT_USED),
    PySlot_END
};

PyMODEXPORT_FUNC PyModExport_hello(void);

PyMODEXPORT_FUNC
PyModExport_hello(void)
{
    return hello_slots;
}

import sys, tally

token = object()
items = [token] * 1000
before = sys.getrefcount(token)
tally.count_true(items)
print(sys.getrefcount(token) - before)

    while ((item = PyIter_Next(it)) != NULL) {
        int truth = PyObject_IsTrue(item);
        if (truth < 0) {
            Py_DECREF(it);
            return NULL;
        }
        total += truth;
    }

    while ((item = PyIter_Next(it)) != NULL) {
        int truth = PyObject_IsTrue(item);
        Py_DECREF(item);
        if (truth < 0) {
            Py_DECREF(it);
            return NULL;
        }
        total += truth;
    }

Summary

• Every object is a PyObject * with a header you access through macros.
• Know, for every call, whether you get a new or a borrowed reference, and whether you pass one in to be stolen. Release what you own on every path.
• Failures set an exception and return NULL or -1; pass them on, releasing what you own, often through one error: label.
• PyArg_ParseTupleAndKeywords turns a tuple and a dict into C values, with optional and keyword arguments.
• Modules are PySlot arrays returned by a PyModExport_<name> hook; PyInit_ is soft deprecated.
• Py_LIMITED_API builds abi3; Py_TARGET_ABI3T builds abi3t, which loads on both kinds of build.
• This completes the CPython internals module.