Measuring performance

You already know the basic tools from Measuring speed with timeit and cProfile: python -m timeit, timeit.repeat, taking the min, and checking that a fast version still gives the right answer. This module is about doing that work well enough to base decisions on it. This lesson covers measurement. Profiling finds where the time goes, and the later lessons cover what to do about it. If you finished the advanced project, its job runner is a good program to practise on through this module.

A benchmark answers one narrow question: on this machine, with this interpreter and this input, which of these is faster, and by how much? Most bad performance decisions come from benchmarks that answered a different question from the one asked.

Pick the right clock

Python has several clocks, and they measure different things:

• time.perf_counter() is the highest-resolution clock available. It counts wall time, including sleeps and waits, and it is monotonic: it never jumps backwards. Use it for elapsed time. perf_counter_ns() gives integer nanoseconds.
• time.process_time() counts CPU time used by this process, in all its threads. Sleeping and waiting for I/O do not count.
• time.time() is the calendar clock. NTP or a user can adjust it, so it can jump. Never benchmark with it.

time.get_clock_info tells you each clock’s properties:

import time

for name in ["perf_counter", "process_time", "time"]:
    info = time.get_clock_info(name)
    print(f"{name:12} monotonic={info.monotonic} adjustable={info.adjustable}")

wall_start, cpu_start = time.perf_counter(), time.process_time()
time.sleep(0.2)                       # waiting, not computing
wall = time.perf_counter() - wall_start
cpu = time.process_time() - cpu_start

print("wall clock saw the sleep:", wall >= 0.19)
print("CPU clock ignored it:", cpu < 0.05)

perf_counter monotonic=True adjustable=False
process_time monotonic=True adjustable=False
time         monotonic=False adjustable=True
wall clock saw the sleep: True
CPU clock ignored it: True

When wall time and CPU time disagree a lot, the program is waiting. No amount of faster Python code will fix that. Look at I/O, locks, or the concurrency models instead.

A harness you can trust

timeit already does the hard parts: it disables garbage collection during the run, uses perf_counter, and runs the statement in a loop so the clock’s resolution does not matter. What it leaves to you is choosing the loop count and reporting more than one number. Here is a small harness that calibrates the loop count, repeats, and returns the best and the median:

import statistics
import sys
import timeit


def bench(stmt, setup="pass", repeat=5, target=0.02):
    """Return (best, median) seconds per execution of stmt."""
    timer = timeit.Timer(stmt, setup)
    number = 1
    while timer.timeit(number) < target:   # calibrate the loop count
        number *= 2
    runs = [t / number for t in timer.repeat(repeat, number)]
    return min(runs), statistics.median(runs)


setup = "words = ['spam'] * 10_000"
join_best, join_median = bench("''.join(words)", setup)
loop_best, _ = bench("s = ''\nfor w in words: s = w + s", setup)

ratio = loop_best / join_best
print("best <= median:", join_best <= join_median)
print("prepend loop slower than join:", ratio > 1)
print("at least 20x slower:", ratio > 20)
print(f"measured: {ratio:.0f}x slower", file=sys.stderr)  # varies

best <= median: True
prepend loop slower than join: True
at least 20x slower: True

The last line goes to standard error, so it is not part of the output above. It prints the number behind the verdict, which changes from run to run. On my machine (a 4-core x86-64 Linux VM running CPython 3.15.0, the machine for every “on my machine” in this module) it said between 110x and 130x. When a verdict prints False, that line tells you whether you are looking at a bug or at a machine that measured differently.

The best run is your estimate of the code’s real cost, since noise only ever adds time. The median tells you how noisy the machine was. If the median is more than about 10% above the best, something else was competing for the CPU, so close it and measure again. Timer.autorange() does the same calibration as the loop above, aiming for 0.2 seconds per run. 3.15+Requires version 3.15 or later You can change that target with autorange(target_time=...) or python -m timeit --target-time, and the command-line output is in colour by default (set NO_COLOR=1 to turn it off).

Notice what the script prints as its result: comparisons with a wide margin, not times. Raw numbers belong in your notes (here, standard error), along with the machine, the Python version and the input size. A claim like “at least 20x slower” survives a different laptop if the margin is wide compared with the noise you measured: 20x against a measured 110x leaves plenty of room, while 100x would not. “316 µs” survives nothing. If a verdict fails, run the script three times and judge by the best run before you suspect the code.

For results you will publish or act on across releases, use pyperf, the tool behind CPython’s own benchmark suite. It runs each benchmark in several fresh processes, discards warm-up runs, and reports the mean and standard deviation. python -m pyperf system tune reduces CPU frequency scaling noise on Linux, and pyperf compare_to old.json new.json says whether a difference is significant.

Growth rate versus constant factors

The fundamentals lesson on Big O describes how cost grows with input size. Measurement lets you check the growth rate directly: time the code at n and at 10 * n, and look at the ratio. A linear algorithm takes about 10 times longer, a constant-time one about the same, and a quadratic one about 100 times longer.

import sys
import timeit


def per_call(stmt, n, number):
    setup = f"data = list(range({n})); s = set(data); x = {n} - 1"
    return min(timeit.repeat(stmt, setup, number=number, repeat=5)) / number


for stmt, number in [("x in data", 200), ("x in s", 200_000)]:
    small = per_call(stmt, 10_000, number)  # cheap code needs more loops
    big = per_call(stmt, 100_000, number)
    growth = big / small               # 10x more data: how much slower?
    if growth > 5:
        shape = "grows with n (linear)"
    elif growth < 2.5:
        shape = "flat (constant)"
    else:
        shape = "unclear, measure again"
    print(f"{stmt:10} {shape}")
    print(f"measured: {growth:.1f}x", file=sys.stderr)  # varies

x in data  grows with n (linear)
x in s     flat (constant)

On my machine the measured growth was about 10x for the list and between 1.0x and 1.3x for the set, so the thresholds of 5 and 2.5 sit well clear of both.

Growth rate decides what happens as data gets bigger. Constant factors decide what happens at the size you actually have. At ten items a linear scan of a tuple can beat building a set, because building the set costs more than the few comparisons it saves. So measure at realistic sizes, and at a larger one too if your data will grow.

When not to measure, and when not to optimise

Most code does not need a benchmark. If a function runs once at start-up, or takes a millisecond inside a request that waits 200 ms for a database, a 2x speed-up changes nothing a user can notice. Measure when you have a concrete target, such as a request deadline, a batch window or a frame budget, and a profile that points at the code. Without a target you cannot know when to stop, and without a profile you are probably measuring the wrong thing. Prefer the clearer version unless a measurement shows the other one matters at your real input size.

Common pitfalls

Timing set-up along with the work. Put data creation in setup, or the benchmark measures list(range(...)) instead of your code.

Benchmarking something lazy. A generator expression, map or filter object does no work until consumed. Timing its creation measures nothing (exercise 2).

Comparing across machines or versions. CPython gets faster every release, often by changing which idioms are fast. A result from 3.10 may be wrong on 3.15.

Trusting one run. Use the best of several runs, and look at the spread.

Folklore. “+= on strings is quadratic” is false for CPython in most loops, but true for other implementations. Measure on the interpreter you ship.

import io
import sys
import timeit


def plus_equals(words):
    s = ""
    for w in words:
        s += w
    return s


def prepend(words):
    s = ""
    for w in reversed(words):
        s = w + s
    return s

def join(words):
    return "".join(words)


def string_io(words):
    buf = io.StringIO()
    for w in words:
        buf.write(w)
    return buf.getvalue()

def best(fn, words):
    return min(timeit.repeat(lambda: fn(words), number=1, repeat=5))


small = [f"{i:09d} " for i in range(1_000)]
big = [f"{i:09d} " for i in range(10_000)]
for fn in [plus_equals, prepend, join, string_io]:
    assert fn(big) == join(big)
    growth = best(fn, big) / best(fn, small)
    shape = "quadratic" if growth > 40 else "linear"
    print(f"{fn.__name__:12} {shape}")
    print(f"measured: {growth:.0f}x", file=sys.stderr)  # varies

plus_equals  linear
prepend      quadratic
join         linear
string_io    linear

import timeit

setup = "data = list(range(10_000))"
as_list = min(timeit.repeat("[x * x for x in data]", setup,
                            number=100, repeat=5))
as_gen = min(timeit.repeat("(x * x for x in data)", setup,
                           number=100, repeat=5))

print("generator is faster:", as_gen < as_list)
print("generator is 50x faster:", as_gen * 50 < as_list)

generator is faster: True
generator is 50x faster: True

import sys
import timeit

setup = "data = list(range(10_000))"
as_list = min(timeit.repeat("sum([x * x for x in data])", setup,
                            number=100, repeat=5))
as_gen = min(timeit.repeat("sum(x * x for x in data)", setup,
                           number=100, repeat=5))

ratio = as_gen / as_list
print("same ballpark (within 3x):", 1 / 3 < ratio < 3)
print(f"measured: generator / list = {ratio:.2f}", file=sys.stderr)

same ballpark (within 3x): True

import sys
import timeit


def compare(a, b, setup="pass", margin=1.25):
    """Say which statement is faster, or 'tie' within the margin."""
    ta = min(timeit.repeat(a, setup, number=500, repeat=5))
    tb = min(timeit.repeat(b, setup, number=500, repeat=5))
    apart = max(ta, tb) / min(ta, tb)
    print(f"measured: {apart:.1f}x apart", file=sys.stderr)  # varies
    if ta * margin < tb:
        return "a"
    if tb * margin < ta:
        return "b"
    return "tie"


setup = "data = list(range(5_000)); s = set(data)"
print(compare("4_999 in data", "4_999 in s", setup))
print(compare("sorted(data)", "sorted(data)", setup, margin=2))
print(compare("data[:]", "[x for x in data]", setup))

b
tie
a

Summary

• Use perf_counter for elapsed time and process_time for CPU time. Never benchmark with time.time().
• Calibrate the loop count, repeat, and report the best run plus the spread. 3.15+Requires version 3.15 or later --target-time and coloured output arrived in the timeit CLI.
• Report ratios and growth rates with a clear margin. Keep raw times in your notes, with the machine and version.
• Check that the code under test really does its work, and that it gives the right answer.
• Measure only when there is a target to meet. Next: Profiling, to find where the time goes.