Monitoring and remote debugging

Debuggers, profilers and coverage tools all need the interpreter to tell them what is happening. For most of Python’s history that meant sys.settrace: a callback on every line of every traced function, paid whether the tool still needed it or not. The tools in this lesson work one level below the inspect signatures and source of introspection and the AST: they see code objects as they run. This lesson covers the modern replacements: an event API with per-location switches, a safe way to attach to a process that is already running, and native stacks that system profilers can read.

sys.monitoring

sys.monitoring (PEP 669, 3.12+) is a namespace in sys, not a module you import. A tool:

1. claims one of the tool ids 0 to 5 with use_tool_id(id, name). Ids 0, 1, 2 and 5 have names by convention: DEBUGGER_ID, COVERAGE_ID, PROFILER_ID, OPTIMIZER_ID;
2. registers a callback per event with register_callback(id, event, func);
3. turns events on, either globally with set_events(id, events) or for one code object with set_local_events(id, code, events);
4. releases the id with free_tool_id(id).

Several tools can be active at once, each with its own id. They are completely independent: one tool cannot monitor another. A call counter:

import sys
from collections import Counter

mon = sys.monitoring
TOOL = mon.PROFILER_ID
calls = Counter()


def on_start(code, instruction_offset):
    calls[code.co_qualname] += 1


def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)


mon.use_tool_id(TOOL, "call-counter")
mon.register_callback(TOOL, mon.events.PY_START, on_start)
mon.set_events(TOOL, mon.events.PY_START)
try:
    fib(10)
finally:
    mon.set_events(TOOL, mon.events.NO_EVENTS)
    mon.free_tool_id(TOOL)
print(calls.most_common())

[('fib', 177)]

PY_START fires when a Python function starts. Other events include PY_RETURN, CALL, LINE, JUMP, RAISE, EXCEPTION_HANDLED, and, since 3.14, BRANCH_LEFT and BRANCH_RIGHT, which replace the deprecated BRANCH. Each event passes different arguments; LINE passes (code, line_number).

DISABLE: pay once per location

A callback can return sys.monitoring.DISABLE. For local events such as LINE, that switches the event off for that one location. The interpreter then runs the location at full speed. For coverage this is exactly right: once a line has run, it is covered, and you never need to hear about it again.

import sys

mon = sys.monitoring
TOOL = mon.COVERAGE_ID
hit = set()


def on_line(code, line_number):
    hit.add((code, line_number))
    return mon.DISABLE        # this line is covered: stop reporting it


def grade(score):
    if score >= 50:
        return "pass"
    return "fail"


mon.use_tool_id(TOOL, "tiny-coverage")
mon.register_callback(TOOL, mon.events.LINE, on_line)
mon.set_local_events(TOOL, grade.__code__, mon.events.LINE)
print(grade(70), grade(80))
mon.free_tool_id(TOOL)

Then compare the lines that ran with the lines that exist:

code = grade.__code__
body = {line for _s, _e, line in code.co_lines() if line is not None}
body.discard(code.co_firstlineno)
missed = sorted(n - code.co_firstlineno for n in body if (code, n) not in hit)
print("missed lines:", missed)

pass pass
missed lines: [3]

Line +3, return "fail", never ran. The second call to grade produced no callbacks at all. co_lines() from the bytecode lesson supplies the set of lines that could have run. sys.monitoring.restart_events() turns every disabled location back on, for tools that measure in rounds.

3.15+Requires version 3.15 or later The events that are not tied to one location, PY_THROW, PY_UNWIND, RAISE, EXCEPTION_HANDLED and RERAISE, can now also be set per code object, and returning DISABLE from their callbacks turns them off for that whole code object (sys.monitoring, “Other events”). An exception tracker can now watch one function and stop paying once it has seen what it needs.

Attaching a debugger to a running process

A server is stuck, and it was not started under a debugger. Since 3.14 (PEP 768), you can attach pdb to it by process id. Debugging with pdb covered the commands and a first attach. A session, then what happens underneath:

$ python3.15 -m pdb -p 48211
> server.py(8)handle_job()
-> time.sleep(0.2)
(Pdb) where
  server.py(13)<module>()
-> handle_job()
> server.py(8)handle_job()
-> time.sleep(0.2)
(Pdb) p jobs_done
6
(Pdb) quit

(Paths shortened to the file name.) The target must run the same Python version. The debugger stops the target at its next bytecode instruction, so a process blocked in a system call attaches once it returns or receives a signal. quit detaches and leaves the server running; only one debugger can be attached at a time.

The building block underneath is sys.remote_exec(pid, script): it asks another CPython process to run a Python file at its next safe point, much like a signal handler. pdb -p is built on the same interface. You can use it directly:

import subprocess
import sys
import tempfile
from pathlib import Path

WORKER = """
import time
running = True
print("ready", flush=True)
while running:
    time.sleep(0.01)
print("worker stopped cleanly", flush=True)
"""

worker = subprocess.Popen([sys.executable, "-c", WORKER],
                          stdout=subprocess.PIPE, text=True)
print(worker.stdout.readline().strip())       # wait until it runs
with tempfile.TemporaryDirectory() as tmp:
    script = Path(tmp, "stop_worker.py")
    script.write_text("import __main__\n__main__.running = False\n",
                      encoding="utf-8")
    sys.remote_exec(worker.pid, str(script))
    rest, _ = worker.communicate(timeout=10)   # file must outlive the run
print(rest.strip())
print("exit code:", worker.returncode)

ready
worker stopped cleanly
exit code: 0

remote_exec returns at once, with no way to learn when the script ran, so the file must stay in place until it has. An exception in the script, even SystemExit, is printed by the target and swallowed, so this cannot be used to kill a process.

Attaching to another process is privileged, so the operating system limits it. On Linux you can generally attach to your own processes, unless the distribution’s “Yama” ptrace restrictions or a container’s seccomp profile forbid it. macOS disables debugging other processes by default; the Python binary has to be given a debugging entitlement. A program can opt out entirely with -X disable-remote-debug (hyphens, not underscores) or PYTHON_DISABLE_REMOTE_DEBUG=1 (remote debugging HOWTO). Python ignores an -X name it does not recognise without any warning, so a misspelled option silently leaves attaching on. Check with sys.is_remote_debug_enabled(): python3.15 -X disable-remote-debug -c "import sys; print(sys.is_remote_debug_enabled())" prints False, and the same command without the option prints True.

Frame pointers for native profilers

sys.monitoring sees Python frames. System profilers such as perf and eBPF tools see native C frames, and to walk a native stack quickly they follow frame pointers: a register each function uses to link to its caller. Compilers usually omit them to gain a register, which leaves profilers guessing.

3.15+Requires version 3.15 or later CPython is now built with frame pointers by default on platforms that support them (PEP 831), using -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer. The flags are exported through sysconfig, so extension modules built with Python’s flags inherit them; one native library built without them can still break the chain for the whole process. You can check a build:

import sys
import sysconfig

flags = sysconfig.get_config_var("CFLAGS") or ""
print(sys.version.split()[0], sys.platform)
print("frame pointers:", "-fno-omit-frame-pointer" in flags)

$ python3.15 frame_pointers.py
3.15.0 linux
frame pointers: True

Some redistributors enabled them earlier: the 3.14.8 build that uv installs prints True as well. 3.15 makes it the upstream default; --without-frame-pointers opts out when building CPython.

Choosing a tool

For a tool you write, use sys.monitoring, and switch off every location you are done with. Use sys.settrace only to support Pythons older than 3.12. To inspect a live process, use pdb -p first and sys.remote_exec for one-off, scripted actions. For CPU time across Python and native code, use a sampling profiler (the profiling package in 3.15, or perf), not an event hook. You will also meet third-party tools such as py-spy, which samples a running process from outside; check each tool’s documentation for the Python versions it supports.

Common pitfalls

Forgetting to free the tool id. The next use_tool_id raises ValueError. Release it in finally.

Heavy callbacks. The callback runs inside the monitored program. Record and return; analyse later.

Global events when one function would do. set_local_events limits the cost to the code you care about.

Expecting remote_exec to report back. It cannot. Have the script write a file or log line you can check.

['left']
['left', 'right']

import sys

mon = sys.monitoring
TOOL = mon.COVERAGE_ID
EVENTS = mon.events.BRANCH_LEFT | mon.events.BRANCH_RIGHT
taken = set()


def recorder(direction):
    def callback(code, instruction_offset, destination_offset):
        taken.add((instruction_offset, direction))
        return mon.DISABLE
    return callback


def grade(score):
    if score >= 50:
        return "pass"
    return "fail"

mon.use_tool_id(TOOL, "branch-coverage")
mon.register_callback(TOOL, mon.events.BRANCH_LEFT, recorder("left"))
mon.register_callback(TOOL, mon.events.BRANCH_RIGHT, recorder("right"))
mon.set_local_events(TOOL, grade.__code__, EVENTS)
grade(70)
print(sorted(d for _off, d in taken))
grade(10)
print(sorted(d for _off, d in taken))
mon.free_tool_id(TOOL)

import sys

mon = sys.monitoring


def count_calls(func, *args):
    calls = 0

    def on_start(code, offset):
        nonlocal calls
        calls += 1

    mon.use_tool_id(mon.PROFILER_ID, "counter")
    mon.register_callback(mon.PROFILER_ID, mon.events.PY_START, on_start)
    mon.set_events(mon.PROFILER_ID, mon.events.PY_START)
    func(*args)
    mon.set_events(mon.PROFILER_ID, mon.events.NO_EVENTS)
    return calls

def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)


print(count_calls(fib, 5))
print(count_calls(fib, 6))

import sys

mon = sys.monitoring


def count_calls(func, *args):
    calls = 0

    def on_start(code, offset):
        nonlocal calls
        calls += 1

    mon.use_tool_id(mon.PROFILER_ID, "counter")
    try:
        mon.register_callback(mon.PROFILER_ID, mon.events.PY_START, on_start)
        mon.set_events(mon.PROFILER_ID, mon.events.PY_START)
        func(*args)
    finally:
        mon.free_tool_id(mon.PROFILER_ID)   # also clears events, callbacks
    return calls

def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)


print(count_calls(fib, 5), count_calls(fib, 6))

15 25

ready
pending jobs: 3

import subprocess
import sys
import tempfile
from pathlib import Path

WORKER = """
import time
jobs = ["resize", "upload", "notify"]
running = True
print("ready", flush=True)
while running:
    time.sleep(0.01)
"""

PEEK = """
import __main__
print("pending jobs:", len(__main__.jobs), flush=True)
__main__.running = False
"""

worker = subprocess.Popen([sys.executable, "-c", WORKER],
                          stdout=subprocess.PIPE, text=True)
print(worker.stdout.readline().strip())
with tempfile.TemporaryDirectory() as tmp:
    script = Path(tmp, "peek.py")
    script.write_text(PEEK, encoding="utf-8")
    sys.remote_exec(worker.pid, str(script))
    rest, _ = worker.communicate(timeout=10)
print(rest.strip())

Summary

• sys.monitoring (3.12+): claim a tool id, register callbacks, enable events globally or per code object, free the id.
• Return DISABLE to stop paying for a location you are done with.
• python -m pdb -p PID attaches to a live process; sys.remote_exec runs a file inside one. Both need OS permission.
• 3.15+Requires version 3.15 or later CPython builds with frame pointers by default, so native profilers can walk Python processes.
• Next: a tour of the C API, the other side of Py_INCREF.