Processes

Threads take turns on the default build, so they cannot speed up CPU-bound Python. Processes can. Each worker process runs its own interpreter with its own GIL, on its own core. The price is that processes share nothing: every argument and every result is copied between them.

A process pool

concurrent.futures.ProcessPoolExecutor has the same interface as the thread pool. This program finds the number below 100,000 with the longest Collatz chain, split into four ranges that run in parallel:

from concurrent.futures import ProcessPoolExecutor


def steps(n):
    count = 0
    while n != 1:
        n = n // 2 if n % 2 == 0 else 3 * n + 1
        count += 1
    return count


def longest_chain(start, stop):
    best = max(range(start, stop), key=steps)
    return best, steps(best)


if __name__ == "__main__":
    starts = [1, 25_001, 50_001, 75_001]
    stops = [s + 25_000 for s in starts]
    with ProcessPoolExecutor() as pool:
        for best, length in pool.map(longest_chain, starts, stops):
            print(f"{best:6} needs {length} steps")

 23529 needs 281 steps
 35655 needs 323 steps
 52527 needs 339 steps
 77031 needs 350 steps

Without max_workers, the pool starts one process per CPU: 3.13+Requires version 3.13 or later per CPU this process may use, os.process_cpu_count(), rather than every CPU in the machine, os.cpu_count(), as before 3.13. Each job is a big chunk of work, which matters: sending a job to another process costs far more than calling a function, so many tiny jobs can make a process pool slower than a plain loop. For long lists of small items, pass chunksize= to map so each message carries a batch.

The main guard

Every process example in this module puts its top-level code under if __name__ == "__main__":, as you saw in Modules and imports. Here it is required, not just good style.

A worker process does not start as a copy of the running parent. It starts a fresh interpreter, which imports your main module to find the function to call. In the worker, the module’s __name__ is not "__main__", so the guarded block does not run. Without the guard, each worker would run the pool-creating code again. Python refuses with a RuntimeError inside the worker, and the pool breaks instead of spawning processes forever. Put function and class definitions at module level, and everything that does work under the guard. On Windows and macOS, where the start method is always spawn by default, a script without the guard cannot use a process pool at all.

What can be sent: pickling

Arguments, the function itself and results travel between processes with pickle. Pickle sends a function by name: “the function double in module __main__”. The worker then looks that name up. So a function must be importable at module level:

import pickle
from concurrent.futures import ProcessPoolExecutor


def double(x):
    return x * 2


if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(list(pool.map(double, [1, 2, 3])))
        try:
            print(list(pool.map(lambda x: x * 2, [1, 2, 3])))
        except pickle.PicklingError:
            print("PicklingError: a lambda cannot be sent to a worker")

[2, 4, 6]
PicklingError: a lambda cannot be sent to a worker

A lambda has no name to look up, and neither has a function defined inside another function. Open files, locks, sockets and database connections cannot be pickled either. Pass the file name, and let the worker open the file.

Nothing is shared

Each worker has its own copy of every module-level variable. A change made there never reaches the parent:

from concurrent.futures import ProcessPoolExecutor

seen = []


def record(word):
    seen.append(word)  # appends to the worker's own copy
    return len(seen)


if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=1) as pool:
        counts = list(pool.map(record, ["a", "b", "c"]))
    print("worker saw:", counts)
    print("parent sees:", seen)

worker saw: [1, 2, 3]
parent sees: []

The single worker kept its own seen list across three calls. The parent’s list stayed empty. The rule for process code is: send inputs as arguments, return outputs as results. Then there is nothing to race on, and no lock is needed.

Start methods

How a worker process comes to life is called the start method:

• spawn starts a brand-new Python interpreter. It is safe and works everywhere, but it is the slowest.
• fork copies the parent process on Unix. It is fast, but copying a process that has threads running can deadlock.
• forkserver starts one clean server process early, and forks workers from it.

3.14+Requires version 3.14 or later On Linux and other Unix platforms except macOS, the default changed from fork to forkserver in Python 3.14. Windows and macOS keep spawn. You can check what your platform uses:

import multiprocessing

if __name__ == "__main__":
    print(multiprocessing.get_start_method())
    print(multiprocessing.get_all_start_methods())

python3.15 start_method.py

On Linux it prints:

forkserver
['forkserver', 'fork', 'spawn']

On macOS and Windows the first line is spawn. Code that worked by accident under fork, because the child inherited the parent’s variables, breaks under the other two. To get the same behaviour everywhere, ask for a start method explicitly with a context:

import multiprocessing
from concurrent.futures import ProcessPoolExecutor


def cube(x):
    return x**3


if __name__ == "__main__":
    spawn = multiprocessing.get_context("spawn")
    with ProcessPoolExecutor(max_workers=2, mp_context=spawn) as pool:
        print(list(pool.map(cube, range(6))))

[0, 1, 8, 27, 64, 125]

Use a context rather than multiprocessing.set_start_method(), which changes the default for the whole program.

When a worker dies

If a worker process crashes, for example in a C extension, the pool cannot finish its jobs. Every pending future then raises BrokenProcessPool. 3.15+Requires version 3.15 or later Python 3.15 attaches the dead worker’s process ID and exit code to the error, as its __cause__:

import os
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures.process import BrokenProcessPool


def crash(code):
    os._exit(code)  # the worker dies without any cleanup


if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=1) as pool:
        future = pool.submit(crash, 3)
        try:
            future.result()
        except BrokenProcessPool as error:
            print(type(error).__name__)
            print("exit code 3" in str(error.__cause__))

BrokenProcessPool
True

On 3.14 the second line prints False. The traceback names the process: Process 20998 terminated abruptly with exit code 3.

multiprocessing.Pool is the older interface to the same machinery. Prefer ProcessPoolExecutor in new code: it matches the thread and interpreter pools.

When not to use processes

Processes cost start-up time, memory for a whole interpreter each, and a pickle round trip for every argument and result. For waiting on I/O, threads or asyncio are cheaper. For jobs that take milliseconds, the overhead can exceed the work. When start-up cost or memory matters and your extension modules support them, interpreters give the same isolation inside one process. Keep processes for CPU-bound work in big chunks, and for code that must survive a crash in a C extension. Never unpickle data from an untrusted source, either: loading a pickle can run arbitrary code.

Common pitfalls

Forgetting the main guard. Workers import the main module. Without the guard, the pool-creating code runs again in every worker.

Sending many tiny jobs. Each job is pickled and sent through a pipe. Make jobs big, or use chunksize.

Relying on global state. A global set in the parent after start-up, or changed in a worker, is not shared. Pass it as an argument.

Passing objects that cannot be pickled. Lambdas, nested functions, open files, locks and connections. Pass plain data and names instead.

[      0,   50000): 5133
[  50000,  100000): 4459
[ 100000,  150000): 4256
[ 150000,  200000): 4136
total: 17984

from concurrent.futures import ProcessPoolExecutor


def count_primes(start, stop):
    """Sieve the numbers below stop, then count those from start on."""
    is_prime = [True] * stop
    is_prime[:2] = [False] * min(2, stop)
    for n in range(2, int(stop**0.5) + 1):
        if is_prime[n]:
            is_prime[n * n :: n] = [False] * len(range(n * n, stop, n))
    return sum(is_prime[start:])


if __name__ == "__main__":
    bounds = [(i * 50_000, (i + 1) * 50_000) for i in range(4)]
    with ProcessPoolExecutor(max_workers=4) as pool:
        counts = list(pool.map(count_primes, *zip(*bounds)))
    for (start, stop), count in zip(bounds, counts):
        print(f"[{start:>7}, {stop:>7}): {count}")
    print("total:", sum(counts))

[      0,   50000): 5133
[  50000,  100000): 4459
[ 100000,  150000): 4256
[ 150000,  200000): 4136
total: 17984

from concurrent.futures import ProcessPoolExecutor


def main():
    rate = 0.2

    def add_tax(price):
        return round(price * (1 + rate), 2)

    with ProcessPoolExecutor(max_workers=2) as pool:
        try:
            print(list(pool.map(add_tax, [10, 25])))
        except Exception as error:
            print(type(error).__name__)


if __name__ == "__main__":
    main()

PicklingError

[12.0, 30.0]

import functools
from concurrent.futures import ProcessPoolExecutor


def add_tax(price, rate):
    return round(price * (1 + rate), 2)


def main():
    with_tax = functools.partial(add_tax, rate=0.2)
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(list(pool.map(with_tax, [10, 25])))


if __name__ == "__main__":
    main()

[12.0, 30.0]

from concurrent.futures import ProcessPoolExecutor


def tag(order):
    order["status"] = "packed"
    return order


if __name__ == "__main__":
    order = {"id": 7, "status": "new"}
    with ProcessPoolExecutor(max_workers=1) as pool:
        result = pool.submit(tag, order).result()
    print(order)
    print(result)
    print(result == order, result is order)

{'id': 7, 'status': 'new'}
{'id': 7, 'status': 'packed'}
False False

Summary

• ProcessPoolExecutor runs functions in separate processes, so CPU-bound Python uses every core.
• Workers import your main module. Keep definitions at module level and the work under if __name__ == "__main__":.
• Functions, arguments and results are pickled. Lambdas, nested functions, files and locks cannot be sent.
• Nothing is shared: pass inputs as arguments and return outputs as results.
• 3.14+Requires version 3.14 or later The default start method is forkserver on Linux, and spawn on macOS and Windows. Use get_context() for an explicit choice.
• 3.15+Requires version 3.15 or later A crashed worker’s exit code appears in the BrokenProcessPool error’s cause.
• Next: asyncio fundamentals, for many concurrent waits in one thread.