Concurrency & Asynchronous Performance Questions

Confirm the lock is the limit before changing anything

A lock (mutex) lets one thread at a time run a critical section, the code that touches shared data. Contention is threads queueing for it. The signature: throughput stops rising (or falls) as you add threads, CPU stays well below the core count, and threads show as blocked. Confirm with evidence:

1. Thread dumps (jstack <pid> several times): many threads in the state BLOCKED (on object monitor) (the JVM's label for a thread waiting to enter a synchronized block) on the same lock address.
2. A lock profile: async-profiler (a sampling profiler for Java; its -e lock option profiles lock contention) reports the lock or monitor class and the nanoseconds spent entering it (async-profiler documentation), so it shows where threads wait; step 4 is what tells you whether the section itself is slow.
3. A scaling test: run 1, 2, 4, 8 threads and plot throughput. A flat or falling line while cores idle is the case described.
4. Read the critical section: is anything slow inside it (allocation, logging, serialisation, I/O, a call out)? Moving that out is the cheapest fix.

The ceiling follows Amdahl's law: if a fraction f of every operation happens under the one lock, no number of threads can beat 1/f times the single-thread rate. With N threads the bound is 1 / (f + (1 - f) / N): the serial part f cannot be shared, and the rest shrinks N-fold. Worked for f = 0.30 and N = 8: 1 / (0.30 + 0.70 / 8) = 1 / (0.30 + 0.0875) = 2.58, shown as 2.6x. (us is microseconds.) The script computes the bound for a hypothetical operation of 1.0 us with 0.30 us under the lock, then the effect of splitting the lock into stripes (a stripe is one of several independent pieces, each with its own lock; the script's simplification: it assumes keys spread evenly and divides f by the stripe count).

# Hypothetical inputs: each operation takes 1.0 us in total, of which 0.30 us is spent holding the cache lock.
total_us, locked_us = 1.0, 0.30
f = locked_us / total_us
print("serial fraction f = %.2f, so throughput can never exceed 1/f = %.1fx the single-thread rate" % (f, 1 / f))
for stripes in (1, 4, 16, 64):
    f_eff = f / stripes                      # keys spread evenly over independent locks
    print("stripes=%2d  effective f=%.4f  bound on speedup with 8 threads: %.1fx"
          % (stripes, f_eff, 1 / (f_eff + (1 - f_eff) / 8)))

Output:

serial fraction f = 0.30, so throughput can never exceed 1/f = 3.3x the single-thread rate
stripes= 1  effective f=0.3000  bound on speedup with 8 threads: 2.6x
stripes= 4  effective f=0.0750  bound on speedup with 8 threads: 5.2x
stripes=16  effective f=0.0187  bound on speedup with 8 threads: 7.1x
stripes=64  effective f=0.0047  bound on speedup with 8 threads: 7.7x

Four structures, measured

The bound is a ceiling for an ideal scheduler. The measurements below can sit under it, and with four or more cores the global lock even falls below one thread's rate, because Amdahl's formula ignores the cost of threads going to sleep and being woken when they fail to get the lock.

The program compares one global lock, a read-write lock (many readers or one writer), 64 lock stripes (the key picks one of 64 independent maps each with its own lock) and ConcurrentHashMap, under 90% writes on 100,000 keys, at 1, 2, 4 and 8 threads. Reading the code: Cache is the two-method interface all four share; xorshift is a cheap per-thread random-number generator, so the threads do not contend on a shared one; the CyclicBarrier makes every thread start together; sink is a result the loop keeps so the JIT (the JVM's run-time compiler, which can delete work whose result is unused) cannot remove the reads; the first pass is a warm-up so the JIT has compiled the hot code before the printed pass. CAS (compare-and-set) is a hardware instruction that changes a value only if it still holds what you expected, the basis of lock-free updates that ConcurrentHashMap uses in places.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;

public class CacheScaling {
    interface Cache { void put(int k, int v); Integer get(int k); }

    // 1. one lock around everything
    static class Global implements Cache {
        final Map<Integer,Integer> m = new HashMap<>();
        public synchronized void put(int k, int v) { m.put(k, v); }
        public synchronized Integer get(int k) { return m.get(k); }
    }
    // 2. read-write lock (writers still exclusive)
    static class RW implements Cache {
        final Map<Integer,Integer> m = new HashMap<>();
        final ReadWriteLock l = new ReentrantReadWriteLock();
        public void put(int k, int v) { l.writeLock().lock(); try { m.put(k, v); } finally { l.writeLock().unlock(); } }
        public Integer get(int k) { l.readLock().lock(); try { return m.get(k); } finally { l.readLock().unlock(); } }
    }
    // 3. lock striping: the key picks one of N independent maps, each with its own lock
    static class Striped implements Cache {
        final Map<Integer,Integer>[] maps; final Object[] locks;
        @SuppressWarnings("unchecked")
        Striped(int n) { maps = new Map[n]; locks = new Object[n];
            for (int i = 0; i < n; i++) { maps[i] = new HashMap<>(); locks[i] = new Object(); } }
        int s(int k) { return (k * 0x9E3779B9 >>> 16) % maps.length; }
        public void put(int k, int v) { int i = s(k); synchronized (locks[i]) { maps[i].put(k, v); } }
        public Integer get(int k) { int i = s(k); synchronized (locks[i]) { return maps[i].get(k); } }
    }
    // 4. the JDK's concurrent map (fine-grained internal locking and CAS)
    static class Chm implements Cache {
        final ConcurrentHashMap<Integer,Integer> m = new ConcurrentHashMap<>();
        public void put(int k, int v) { m.put(k, v); }
        public Integer get(int k) { return m.get(k); }
    }

    static double run(Cache c, int threads, int opsPerThread, int writePercent) throws Exception {
        Thread[] ts = new Thread[threads];
        CyclicBarrier start = new CyclicBarrier(threads + 1);
        for (int t = 0; t < threads; t++) {
            final int seed = t * 7919 + 1;
            ts[t] = new Thread(() -> {
                int x = seed; long sink = 0;
                try { start.await(); } catch (Exception e) { return; }
                for (int i = 0; i < opsPerThread; i++) {
                    x ^= x << 13; x ^= x >>> 17; x ^= x << 5;          // xorshift: cheap per-thread random
                    int key = (x >>> 1) % 100_000;
                    if ((x & 0xFF) * 100 / 256 < writePercent) c.put(key, i);
                    else { Integer v = c.get(key); if (v != null) sink += v; }
                }
                if (sink == 42) System.out.print("");
            });
            ts[t].start();
        }
        start.await();
        long t0 = System.nanoTime();
        for (Thread t : ts) t.join();
        double sec = (System.nanoTime() - t0) / 1e9;
        return threads * (double) opsPerThread / sec / 1e6;              // million ops per second
    }

    public static void main(String[] a) throws Exception {
        System.out.println("availableProcessors = " + Runtime.getRuntime().availableProcessors() + ", 90% writes, 100k keys");
        String[] names = {"one global lock", "read-write lock", "striped x64", "ConcurrentHashMap"};
        for (int pass = 0; pass < 2; pass++) {                           // pass 0 warms up the JIT and is not printed
            if (pass == 1) System.out.printf("%-18s %8s %8s %8s %8s   (million ops/s by thread count)%n", "", "1", "2", "4", "8");
            for (int n = 0; n < names.length; n++) {
                StringBuilder sb = new StringBuilder(String.format("%-18s", names[n]));
                for (int threads : new int[]{1, 2, 4, 8}) {
                    Cache c = switch (n) { case 0 -> new Global(); case 1 -> new RW(); case 2 -> new Striped(64); default -> new Chm(); };
                    double mops = run(c, threads, pass == 0 ? 300_000 : 4_000_000, 90);
                    sb.append(String.format(" %8.1f", mops));
                }
                if (pass == 1) System.out.println(sb);
            }
        }
    }
}

One sample run in an eclipse-temurin:21-jdk container with --cpus=8 (OpenJDK 21; million operations per second; a noisy shared host, so absolute figures and even single-thread values differ from run to run):

availableProcessors = 8, 90% writes, 100k keys
                          1        2        4        8   (million ops/s by thread count)
one global lock        39.0     17.9     19.8     25.7
read-write lock        38.2     22.3     34.6     27.2
striped x64            26.1     40.7     93.6     59.3
ConcurrentHashMap      36.3     66.8    107.9    124.1

The shape, across re-runs at 1, 2, 4 and 8 CPUs (absolute figures swing by tens of percent between runs on a shared host, and a CPU set (--cpuset-cpus) gave different orderings from a CPU quota (--cpus) on one and two CPUs, so read the ordering, not the digits): with four or more cores, or a two-CPU quota, the global lock at 2 threads runs at roughly half its one-thread rate or less and stays below that rate out to 8 threads (threads spend time parking and waking), and striping and the concurrent map beat it from 2 threads up, typically by about 2 to 5 times at 2 and 4 threads (under 2 times at 2 threads in an occasional run). The concurrent map was usually the fastest row at 4 and 8 threads on 4 and 8 CPUs, but its 8-thread figure stayed within roughly a third either way of its 4-thread figure, so more threads than that did not buy more throughput even on 8 cores. Three rows need reading rather than explaining away. First, the read-write lock is erratic rather than a steady copy of the global lock: at 2 threads it is close to the global lock (often within half of it, in either direction), and at 4 and 8 threads it usually sat between the global lock and striping, sometimes close to striping at 8 threads (on two cores it was occasionally level with striping at 4 threads), so it is not reliably far below the striped map; its read path serves only the 10% of operations that are reads, and writers still exclude everyone. Second, single-thread figures are the least stable numbers in the table: striped x64 at one thread (26.1) is below the global lock (39.0) in this sample and in nearly all re-runs on 4 and 8 CPUs, though on one or two CPUs the two were close and either could lead, and a plausible reason is the extra work each operation does (computing the stripe, then touching one of 64 separate maps) that a single uncontended lock avoids, so treat the gap as a tendency, not a law. Third, striped x64 at 8 threads (59.3) is below its 4-thread figure (93.6), and on 8 CPUs that dip repeated in re-runs rather than washing out as noise, while the concurrent map stayed roughly level between 4 and 8 threads; the cause was not isolated here, so measure how this simple striped design scales from 4 to 8 threads on your own hardware. With one core there is no parallelism to unlock, so the order is not reliable and differed between a one-core CPU set and a one-CPU quota: striping gave no consistent gain over the global lock (ahead in some runs, behind in others), the read-write lock was ahead of the global lock at 4 and 8 threads in every re-run, and the concurrent map usually led the global lock on a one-core CPU set but trailed it at 8 threads under a one-CPU quota (every row is noisy, and striping cannot create CPU). With two cores the result depended on how they were given: under a two-CPU quota striping and the concurrent map beat the global lock by about 2 to 6 times at 2 threads, while on a two-core CPU set the global lock was often close to them or ahead at 2 threads, and its 2-thread rate was sometimes above its one-thread rate.

Options and what each costs

Recommendation

Start with the lock profile; if nothing slow is inside the section, replace the map with a concurrent one (or stripes if you need custom eviction per shard), because it fixes the measured failure at the least code. Use batching for write-behind data that tolerates staleness. Keep a read-write lock only if your own scaling test shows a read-dominated load. Verify the fix with the same 1, 2, 4, 8 thread test and by checking the lock profile no longer shows that lock on top. What flips the choice: a hot key (shard cannot spread it, so shorten the section or replicate), a need for exact size or consistent snapshots (stripes plus a global operation, or accept a coarse lock for those operations), or operating-system kernel code that runs in interrupt context (an interrupt handler is code that runs when hardware signals the CPU), where the code must not sleep, so a sleeping lock such as a mutex is not allowed there; use a spinlock (a lock that busy-waits instead of sleeping, in its interrupt-safe form when process-context code shares the data) held for a very short section, or per-CPU data (one copy of the data per CPU, needing no lock), to bound worst-case latency.

Principle: size each stage by what it consumes, then let the slowest stage set the pace

A pipeline moves at the speed of its bottleneck, the stage with the lowest capacity. The five stages consume different things, so they get different pools:

Backpressure means a full downstream queue slows the upstream stage instead of letting memory grow. Use bounded queues between stages, so a slow stage blocks the sender, and one admission limit (a semaphore, a counter of free slots that an image must take before it enters) on total images in flight, which is also the memory limit: in-flight images x decoded size.

Computed sizing (hypothetical inputs)

Which lever matters most: the bottleneck stage's capacity first (here the accelerator, set mostly by batch size), then the in-flight limit (it sets both memory and queueing latency), then pool sizes for the other stages, then priority and the maximum-wait timer, which only reshape latency once throughput is right.

The inputs below (40 ms fetch wait, 8 ms decode, accelerator 30 ms per batch plus 6 ms per image, 6 ms post-processing, 4 cores, a target of 120 images per second) are made-up numbers for the exercise, replace them with measured per-stage times. The script computes concurrency, core needs and the effect of batch size. Its last two columns are two different pictures of the batcher: the fill wait if it sat idle waiting for a batch to fill, and how many images arrive at the target rate while one batch is already running on the accelerator.

# Hypothetical stage costs (stated inputs, not measurements) for the pipeline.
FETCH_WAIT_MS = 40        # network wait, no CPU
DECODE_CPU_MS = 8
DETECT_SETUP_MS = 30      # per-batch fixed cost on the accelerator
DETECT_PER_IMG_MS = 6
POST_CPU_MS = 6           # annotate + render
CORES = 4
TARGET = 120              # images per second wanted

def detect_capacity(batch):
    return 1000 / (DETECT_SETUP_MS / batch + DETECT_PER_IMG_MS)

print(f"target {TARGET} img/s on {CORES} cores")
print("fetch   : in flight = rate x time (Little) =", round(TARGET * FETCH_WAIT_MS / 1000, 1), "requests -> I/O workers >= 5, no CPU used")
print("decode  : cores needed =", round(TARGET * DECODE_CPU_MS / 1000, 2))
print("post    : cores needed =", round(TARGET * POST_CPU_MS / 1000, 2))
print("CPU stages together    :", round(TARGET * (DECODE_CPU_MS + POST_CPU_MS) / 1000, 2), "of", CORES, "cores")
print("batch  accel img/s  fill wait if the batcher idles  arrivals while one batch runs")
for b in (1, 2, 4, 8, 12, 13, 16, 32):
    fill = (b - 1) / TARGET * 1000
    run_ms = DETECT_SETUP_MS + DETECT_PER_IMG_MS * b
    print(f"{b:>5}  {detect_capacity(b):>9.1f}  {fill:>14.1f} ms  {TARGET * run_ms / 1000:>16.1f}")
print("smallest batch that reaches the target:", next(b for b in range(1, 100) if detect_capacity(b) >= TARGET))
for cores in (1, 2, 4):
    print(f"CPU-stage ceiling on {cores} core(s): {cores * 1000 / (DECODE_CPU_MS + POST_CPU_MS):.1f} img/s")
# Memory: hypothetical 1920 x 1080 RGB image decoded to 3 bytes per pixel, 48 images in flight.
img_mb = 1920 * 1080 * 3 / 1e6
print(f"decoded image {img_mb:.1f} MB; 48 in flight = {48 * img_mb:.0f} MB")

Run in a python:3.12-slim container (CPython 3.12), it prints:

target 120 img/s on 4 cores
fetch   : in flight = rate x time (Little) = 4.8 requests -> I/O workers >= 5, no CPU used
decode  : cores needed = 0.96
post    : cores needed = 0.72
CPU stages together    : 1.68 of 4 cores
batch  accel img/s  fill wait if the batcher idles  arrivals while one batch runs
    1       27.8             0.0 ms               4.3
    2       47.6             8.3 ms               5.0
    4       74.1            25.0 ms               6.5
    8      102.6            58.3 ms               9.4
   12      117.6            91.7 ms              12.2
   13      120.4           100.0 ms              13.0
   16      127.0           125.0 ms              15.1
   32      144.1           258.3 ms              26.6
smallest batch that reaches the target: 13
CPU-stage ceiling on 1 core(s): 71.4 img/s
CPU-stage ceiling on 2 core(s): 142.9 img/s
CPU-stage ceiling on 4 core(s): 285.7 img/s
decoded image 6.2 MB; 48 in flight = 299 MB

Reading it: the CPU stages need 1.68 of 4 cores at the target, so they are not the limit. The accelerator is: with batch 1 it tops out at 27.8 img/s, and the smallest batch that reaches the 120 target is 13 (120.4 img/s); 16 gives 127 img/s with headroom. That is the batching trade-off in numbers: a bigger batch spreads the fixed 30 ms over more images (higher throughput) but takes longer to run and to fill.

How big a batch actually forms depends on the load, and the two columns explain it. Under a steady 120 img/s the accelerator is busy running the previous batch for 30 + 6 x 13 = 108 ms, and in that time 120 x 0.108 = 13 images arrive and queue up, so the next batch fills instantly from the backlog without anyone waiting for it. A batch of 13 is therefore self-consistent at the target rate: arrivals while it runs equal its size. If a batch happened to be smaller than that (say 3), capacity at that size would be about 1000 / (30 / 3 + 6) = 62 img/s, below the arrival rate, so the backlog grows and the next batch, which takes whatever has queued up to the batch limit, is bigger, until batches settle where capacity matches arrivals. The idle-batcher "fill wait" column (100 ms at 13) only applies when the queue is empty; then the batcher needs a maximum wait timer (20 ms in the program) that sends a partial batch so that a lone image is not held back. At light traffic that partial batch is small and capacity per image is poor, which is acceptable because the load is light. On 1 core the CPU stages cap the pipeline at 71 img/s, below the target, so with fewer cores the bottleneck moves from the accelerator to the CPU.

Simulation: batching, bulk versus interactive, priority lane

The Go program wires the stages with bounded queues and an admission limit of 48 images. Fetch and the accelerator are time.Sleep (they use no CPU); decode and post-processing spin on the CPU for their stated time. The costs are the simulated ones above, not a real model. A bulk producer keeps the pipeline full; 5 interactive requests per second arrive on top. In the third setup, interactive items go through a high-priority lane at every stage and flush the accelerator batch as soon as they arrive.

package main

import (
	"fmt"
	"runtime"
	"sort"
	"sync"
	"time"
)

// Stage costs are SIMULATED: fetch and the accelerator sleep (they use no CPU), decode and
// post-processing spin on the CPU for the stated time so that cores really are the limit.
const (
	fetchMs  = 40
	decodeMs = 8
	setupMs  = 30 // accelerator: fixed cost per batch
	perImgMs = 6  // accelerator: cost per image in the batch
	postMs   = 6
	maxWait  = 20 * time.Millisecond // longest the batcher waits to fill a batch
	inflight = 48                    // total images allowed in the pipeline: the memory limit
)

type item struct {
	interactive bool
	start       time.Time
}

func spin(ms int) {
	end := time.Now().Add(time.Duration(ms) * time.Millisecond)
	for time.Now().Before(end) {
	}
}

// pq is a two-lane queue. A chan is a Go channel: a bounded pipe between goroutines (Go's lightweight threads).
// Receivers always take the high lane first.
type pq struct{ hi, lo chan item }

func newPQ() *pq { return &pq{make(chan item, 16), make(chan item, 16)} }
func (q *pq) send(it item, prio bool) {
	if prio && it.interactive {
		q.hi <- it
	} else {
		q.lo <- it
	}
}
func (q *pq) recv() item {
	// Step 1: non-blocking peek at the high lane. A select with a default case never waits.
	select {
	case it := <-q.hi:
		return it
	default:
	}
	// Step 2: high lane was empty, so block until either lane has an item. Using only this select would
	// let Go pick at random when both lanes are ready, so a waiting high item could lose to a low one.
	select {
	case it := <-q.hi:
		return it
	case it := <-q.lo:
		return it
	}
}

func run(name string, batchMax int, prio bool, cores int, dur time.Duration) {
	fetchQ, decodeQ, detectQ, postQ := newPQ(), newPQ(), newPQ(), newPQ()
	done := make(chan item, 256)
	sem := make(chan struct{}, inflight) // admission: bounds memory

	var wg sync.WaitGroup
	pool := func(n int, in *pq, work func(item) item, out *pq) {
		for i := 0; i < n; i++ {
			wg.Add(1)
			go func() {
				defer wg.Done()
				for {
					it, ok := recvOrStop(in)
					if !ok {
						return
					}
					it = work(it)
					if out != nil {
						out.send(it, prio)
					} else {
						done <- it
					}
				}
			}()
		}
	}
	pool(16, fetchQ, func(it item) item { time.Sleep(fetchMs * time.Millisecond); return it }, decodeQ)
	pool(cores, decodeQ, func(it item) item { spin(decodeMs); return it }, detectQ)
	// Batcher: one accelerator, one batch at a time. An interactive item flushes the batch at once.
	wg.Add(1)
	go func() {
		defer wg.Done()
		for {
			first, ok := recvOrStop(detectQ)
			if !ok {
				return
			}
			batch := []item{first}
			deadline := time.After(maxWait) // channel that delivers one value after maxWait: the batch's wait budget
			flush := prio && first.interactive
		// Fill loop: keep adding items until the batch is full, the wait budget is spent, or an
		// interactive item asks for an early flush (send the batch to the accelerator now).
		fill:
			for len(batch) < batchMax && !flush { // if a backlog is queued, items arrive instantly and the timer never matters
				select {
				case it := <-detectQ.hi:
					batch = append(batch, it)
					flush = prio
				case it := <-detectQ.lo:
					batch = append(batch, it)
				case <-deadline:
					break fill
				}
			}
			time.Sleep(time.Duration(setupMs+perImgMs*len(batch)) * time.Millisecond)
			for _, it := range batch {
				postQ.send(it, prio)
			}
		}
	}()
	pool(cores, postQ, func(it item) item { spin(postMs); return it }, nil)

	stop := make(chan struct{})
	// Bulk producer: always has work, limited only by the in-flight cap (backpressure).
	go func() {
		for {
			select {
			case <-stop:
				return
			case sem <- struct{}{}:
				fetchQ.send(item{false, time.Now()}, prio)
			}
		}
	}()
	// Interactive producer: 5 requests per second.
	go func() {
		t := time.NewTicker(200 * time.Millisecond)
		for {
			select {
			case <-stop:
				return
			case <-t.C:
				select {
				case sem <- struct{}{}:
					fetchQ.send(item{true, time.Now()}, prio)
				case <-stop:
					return
				}
			}
		}
	}()

	var lat []float64
	bulk := 0
	t0 := time.Now()
	for time.Since(t0) < dur {
		it := <-done
		<-sem
		if it.interactive {
			lat = append(lat, float64(time.Since(it.start).Microseconds())/1000)
		} else {
			bulk++
		}
	}
	close(stop)
	sort.Float64s(lat)
	fmt.Printf("%-34s bulk %5.1f img/s | interactive n=%2d  p50 %6.0f ms  p99/max %6.0f ms\n",
		name, float64(bulk)/dur.Seconds(), len(lat), lat[len(lat)/2], lat[len(lat)-1])
	// goroutines are left to the process exit; this program is a one-shot experiment.
}

func recvOrStop(q *pq) (item, bool) { return q.recv(), true }

func main() {
	fmt.Println("GOMAXPROCS =", runtime.GOMAXPROCS(0))
	cores := runtime.GOMAXPROCS(0)
	dur := 8 * time.Second
	run("FIFO, batch 1", 1, false, cores, dur)
	run("FIFO, batch 16", 16, false, cores, dur)
	run("priority lanes, batch 16", 16, true, cores, dur)
}

Read it as: chan is Go's bounded queue between goroutines (lightweight threads); select waits on several channels and takes whichever is ready; recv first peeks at the high lane with a non-blocking select (the default case means "do not wait") and only then blocks on both lanes, which is why select appears twice: a single select over both lanes would choose at random when both are ready, and the high lane could lose. In the batcher, time.After(maxWait) is the wait budget, the fill loop adds items until the batch is full, the budget is spent, or an interactive item asks to flush (send the batch to the accelerator at once), and time.Sleep stands for the accelerator's run time. The semaphore channel sem is the admission window: a slot is taken before an image enters and returned when it leaves, so at most 48 images exist at once.

Compiled and run in a golang:1.23 container with go run pipeline.go and --cpuset-cpus=0-3 (GOMAXPROCS is the number of CPUs Go runs goroutines on at once, here 4), one sample run (host-dependent; repeat runs on 4 and 8 CPUs gave nearly the same figures. On 2 CPUs the batch-16 rows were the same or lower, about 85 to 118 img/s with FIFO and about 80 to 95 with priority lanes, because decode, post-processing and the batcher then compete for the cores. On one CPU, --cpuset-cpus=0, the CPU stages cap the pipeline as the script predicts: batch-16 FIFO ran at about 55 to 73 img/s, close to the 71 img/s ceiling the script computed, with an interactive median of about 0.55 to 0.75 s, priority lanes no longer cost bulk throughput there, and batch 1 was unchanged because the accelerator is its limit):

GOMAXPROCS = 4
FIFO, batch 1                      bulk  20.4 img/s | interactive n=29  p50   2002 ms  p99/max   2044 ms
FIFO, batch 16                     bulk 113.4 img/s | interactive n=37  p50    409 ms  p99/max    425 ms
priority lanes, batch 16           bulk  96.1 img/s | interactive n=38  p50    215 ms  p99/max    346 ms

What the shape says:

1. Batch 1 is accelerator-bound: bulk about 20 to 23 img/s plus about 4 interactive per second (n=29 to 31 in 8 s against about 39 requests issued at 5 per second: the missing ones arrived in the last two seconds of the window and were still in the pipeline when the measurement stopped, since each takes about 2 s) is about 26 img/s once all 5 interactive requests per second are counted, a little under the 27.8 img/s ceiling the script computed because real sleeps run slightly long. Every image, interactive or not, waits behind the full admission window: with 48 images allowed in flight at about 25 completions per second, a new request waits about 48 / 25 = 1.9 seconds, which matches the 2002 ms (re-runs gave a median of about 1.75 to 2.1 s). (p99/max is the slowest 1 percent; with only about 30 samples it is simply the slowest request.) The memory cap is also the latency cap: bounding in-flight work bounds the queue wait, and a smaller cap gives lower latency at the same throughput, but only while the cap stays above what the bottleneck needs in flight. By Little's law that need is the throughput times the unloaded time through the pipeline. For batch 1 the pipeline sustains under 28 img/s through a path of about 90 ms, so only about 3 images need to be in flight, and shrinking the cap from 48 to 12 cut the interactive median to roughly 0.5 seconds with about the same bulk throughput. For batch 16 at the 120 target the unloaded path is about 40 + 8 + (30 + 6 x 16) + 6 = 180 ms, so roughly 120 x 0.18 = 22 images must be in flight at the very least. That is a floor, not a target: the batcher also needs the next batch already waiting while the current one runs, so a cap of 12 gave about half the bulk throughput of a cap of 48 (about 60 against about 115 img/s) and even a cap of 24, just above the floor, gave only about three quarters of it (about 88 img/s). Choose a cap well above the floor (48 came close to the 127 img/s the accelerator can do at batch 16), then trim it only as far as the latency and memory budgets require.
2. Batch 16 is about 5 to 6 times the bulk throughput (113 against 20 img/s in the sample) and interactive latency falls to about 400 ms, because the pipeline now drains the window much faster (Little's law again: 48 in flight at about 120 completions per second is about 0.4 s).
3. Priority lanes roughly halve the interactive median (to about 200 to 290 ms in re-runs on 4 and 8 CPUs, against about 380 to 415 ms) at a cost to bulk throughput (about 94 to 102 against 113 to 120 img/s on 4 and 8 CPUs; on one CPU there is no such cost, because the CPU stages are the limit and the batch size is not), because each interactive arrival flushes a smaller batch and the fixed 30 ms is spread over fewer images. Interactive latency does not fall to its unloaded value (about 40 + 8 + 36 + 6 = 90 ms of service), mainly because the accelerator cannot preempt (interrupt) a batch already running (and decode and post-processing may be busy spinning on a bulk item already in service): an interactive item waits for the batch in progress, up to 30 + 6 x 16 = 126 ms, plus its own 36 ms.

Decisions, stated

• Pool sizing per stage: fetch by Little's law (about 5 in flight here, round up with headroom), CPU stages about one worker per core, one batcher per accelerator.
• Batch size: the largest batch whose fill wait plus run time still fits the interactive latency budget; in the example, 8 gives 102.6 img/s (short of the 120 target) and 13 is the smallest that reaches it, so the target forces a batch of 13 or more (16 with headroom) unless a second accelerator is added. The maximum-wait timer protects latency when traffic is low.
• Backpressure: bounded queues between stages plus the in-flight limit; the producer blocks, or the API returns 429 when interactive admission is full, rather than growing a queue.
• Prioritising urgent requests: a separate high-priority lane at each queue and early batch flush, and reserve a slice of in-flight capacity for interactive work so bulk cannot fill the whole admission window. Because a running batch cannot be preempted, cap the bulk batch size, or give interactive work its own small accelerator slot, if the p99 matters more than bulk throughput.
• Memory limit: in-flight images x decoded size (299 MB for 48 images at 6.2 MB, hypothetical); decode only when the next stage can accept the image, and pass compressed bytes until then. Do not set the limit at or near the minimum of roughly the target rate x the unloaded time through the pipeline (about 22 images here): the batcher needs a whole batch queued behind the one it is running, and a limit of 24 still left bulk throughput about a quarter short.

What would change this: if the detector is CPU-based and not an accelerator, it joins the CPU budget (decode + detect + render share the cores) and the sizing becomes one cores-limited pool with priorities; if a stage is separately scalable (its own service), size and autoscale it independently.

Design decisions, stated before the code

A worker pool here is N goroutines (Go's lightweight threads) that read jobs from a channel; a buffered channel is the bounded queue. Memory is bounded because at most N jobs run and at most Q wait.

Go terms used below. A channel is a typed pipe between goroutines; a buffered channel has a fixed capacity, and sending on a full one waits. select waits on several channel operations at once and runs whichever is ready first; with a default branch it never waits, it takes default when nothing is ready. A context.Context carries a cancellation signal and optional deadline down the call chain; ctx.Done() is a channel that closes when the context is cancelled or its deadline passes. A sync.RWMutex is a lock that admits many readers at once or one writer alone. atomic.Int64 is a counter that several goroutines can update safely without a lock. A sync.WaitGroup counts running goroutines so a caller can wait until all are done. defer runs a call when the surrounding function returns.

1. What submit does when the queue is full. Offer two calls and make the policy explicit. TrySubmit never blocks on a full queue: it returns ErrQueueFull at once (the one exception is a pending shutdown, described below). This is the default for a request/response service, because the caller can turn it into an HTTP 503 immediately instead of piling up waiting goroutines (which would just move the unbounded growth from the queue to the callers). Submit(ctx, job) blocks for space but only until the caller's context ends, which suits a batch producer that should slow down (backpressure: pushing slowness back to the sender). Dropping the oldest job is a third policy, used only when stale work is worthless; it is not built in here because it would silently lose work.
2. Shutdown with a deadline. Shutdown(ctx) first stops intake (later submits get ErrClosed), closes the queue so workers exit after draining it, and waits for workers. If the context's deadline passes first, it cancels the context handed to running jobs and waits for them to return. Jobs must watch that context to stop promptly: Go cannot kill a goroutine from outside, so a job that ignores it keeps Shutdown waiting. In the code, a read lock guards submit and a write lock guards the close, so a send on a closed channel cannot happen. This matters because in Go, sending on a closed channel panics (crashes the goroutine with a run-time error). Many submitters can hold the read lock at once; Shutdown takes the write lock, which waits until no submitter is mid-send, then sets closed and closes the channel. Any submitter arriving afterwards sees closed and returns ErrClosed before it touches the channel. A consequence: a blocked Submit holds the read lock until its context ends, which can delay Shutdown by that long, so always give Submit a deadline. It also affects TrySubmit: Go's sync.RWMutex makes new readers wait once a writer is pending, so while Shutdown is waiting for a blocked Submit, a new TrySubmit stalls on the lock until that Submit gives up (for example, a TrySubmit that arrives 200 ms into a blocked Submit with a 400 ms deadline waits about the remaining 200 ms and then returns ErrClosed). The stall is bounded by the Submit deadline, which is another reason to keep it short.
3. What an operator sees. A Stats snapshot: queue length and capacity (a queue that stays near capacity means saturation), active workers against total workers (all workers busy with a non-empty queue is saturation), a counter of rejected submissions (the load-shedding rate), a counter of completed jobs, and mean time a job waited in the queue (the number that users feel). Export them as gauges and counters (for example through Prometheus, a monitoring system that scrapes metrics over HTTP, or expvar, a Go standard-library package that publishes variables as JSON over HTTP) and alert on sustained queue depth above zero, a rising rejection rate and queue wait approaching the latency budget.

The code

pool.go:

package main

import (
	"context"
	"errors"
	"sync"
	"sync/atomic"
	"time"
)

var (
	ErrQueueFull = errors.New("pool: queue full")
	ErrClosed    = errors.New("pool: closed")
)

// Job receives a context that is cancelled if shutdown runs past its deadline.
type Job func(ctx context.Context)

type Pool struct {
	jobs     chan Job
	workers  int
	active   atomic.Int64 // workers currently running a job
	rejected atomic.Int64 // submissions refused because the queue was full
	done     atomic.Int64
	waitNs   atomic.Int64 // total time finished jobs spent queued before a worker took them

	mu     sync.RWMutex // guards closed and the close of jobs
	closed bool

	wg     sync.WaitGroup
	runCtx context.Context
	cancel context.CancelFunc
}

func New(workers, queueSize int) *Pool {
	ctx, cancel := context.WithCancel(context.Background())
	p := &Pool{jobs: make(chan Job, queueSize), workers: workers, runCtx: ctx, cancel: cancel}
	p.wg.Add(workers)
	for i := 0; i < workers; i++ {
		go p.worker()
	}
	return p
}

func (p *Pool) worker() {
	defer p.wg.Done()
	for job := range p.jobs { // exits when jobs is closed and drained
		p.active.Add(1)
		job(p.runCtx)
		p.active.Add(-1)
		p.done.Add(1)
	}
}

// timed wraps a job so the pool can record how long it sat in the queue.
func (p *Pool) timed(j Job) Job {
	queued := time.Now()
	return func(ctx context.Context) {
		p.waitNs.Add(int64(time.Since(queued)))
		j(ctx)
	}
}

// TrySubmit never blocks: when the queue is full the caller learns immediately.
func (p *Pool) TrySubmit(j Job) error {
	p.mu.RLock()
	defer p.mu.RUnlock()
	if p.closed {
		return ErrClosed
	}
	select {
	case p.jobs <- p.timed(j):
		return nil
	default:
		p.rejected.Add(1)
		return ErrQueueFull
	}
}

// Submit waits for queue space, but only until ctx ends (bounded blocking).
func (p *Pool) Submit(ctx context.Context, j Job) error {
	p.mu.RLock()
	defer p.mu.RUnlock()
	if p.closed {
		return ErrClosed
	}
	select {
	case p.jobs <- p.timed(j):
		return nil
	case <-ctx.Done():
		p.rejected.Add(1)
		return ctx.Err()
	}
}

// Shutdown stops intake, lets queued and running jobs finish, and if ctx expires first
// cancels the context the jobs received and waits for them to return.
func (p *Pool) Shutdown(ctx context.Context) error {
	p.mu.Lock()
	if !p.closed {
		p.closed = true
		close(p.jobs)
	}
	p.mu.Unlock()

	finished := make(chan struct{})
	go func() { p.wg.Wait(); close(finished) }()
	select {
	case <-finished:
		return nil
	case <-ctx.Done():
		p.cancel()
		<-finished
		return ctx.Err()
	}
}

type Stats struct {
	QueueLen, QueueCap, Active, Workers int
	Rejected, Done                      int64
	MeanQueueWait                       time.Duration // average over finished jobs
}

func (p *Pool) Stats() Stats {
	var mean time.Duration
	if d := p.done.Load(); d > 0 {
		mean = time.Duration(p.waitNs.Load() / d)
	}
	return Stats{len(p.jobs), cap(p.jobs), int(p.active.Load()), p.workers, p.rejected.Load(), p.done.Load(), mean}
}

main.go drives it deterministically: two jobs occupy both workers, then the queue is filled and overflowed.

package main

import (
	"context"
	"fmt"
	"time"
)

func main() {
	p := New(2, 2)
	release := make(chan struct{})
	started := make(chan struct{}, 8)
	block := func(ctx context.Context) {
		started <- struct{}{}
		select {
		case <-release:
		case <-ctx.Done():
		}
	}

	// 1. Occupy both workers, then fill the queue, then overflow it.
	p.TrySubmit(block)
	p.TrySubmit(block)
	<-started
	<-started
	for i := 1; i <= 5; i++ {
		fmt.Printf("submit %d: %v\n", i, p.TrySubmit(func(ctx context.Context) {}))
	}
	fmt.Printf("stats: %+v\n", p.Stats())

	// 2. Bounded blocking submit gives up after 50 ms.
	ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
	fmt.Println("blocking submit:", p.Submit(ctx, func(context.Context) {}))
	cancel()

	// 3. Graceful drain: release the blocked jobs, everything queued still runs.
	close(release)
	sctx, scancel := context.WithTimeout(context.Background(), time.Second)
	fmt.Println("shutdown:", p.Shutdown(sctx))
	scancel()
	fmt.Printf("stats: %+v\n", p.Stats())
	fmt.Println("submit after shutdown:", p.TrySubmit(func(context.Context) {}))

	// 4. Deadline exceeded: a job that ignores the drain window is cancelled.
	q := New(1, 1)
	q.TrySubmit(func(ctx context.Context) { <-ctx.Done() })
	dctx, dcancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
	fmt.Println("shutdown with stuck job:", q.Shutdown(dctx))
	dcancel()
}

pool_test.go runs 16 producers against a 4-worker pool while it shuts down, and asserts that every accepted job ran:

package main

import (
	"context"
	"sync"
	"sync/atomic"
	"testing"
	"time"
)

// Many producers hammer a small pool while it shuts down. Run with -race.
func TestConcurrentSubmitAndShutdown(t *testing.T) {
	p := New(4, 8)
	var ran atomic.Int64
	var accepted atomic.Int64
	var wg sync.WaitGroup
	for g := 0; g < 16; g++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for i := 0; i < 2000; i++ {
				if p.TrySubmit(func(context.Context) { ran.Add(1) }) == nil {
					accepted.Add(1)
				}
			}
		}()
	}
	time.Sleep(5 * time.Millisecond)
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	if err := p.Shutdown(ctx); err != nil {
		t.Fatal(err)
	}
	wg.Wait()
	if ran.Load() != accepted.Load() {
		t.Fatalf("accepted %d but ran %d: a queued job was lost", accepted.Load(), ran.Load())
	}
}

Reading the code. In New, p.wg.Add(workers) and the loop start N worker goroutines; each worker ranges over the channel, so it picks up one job at a time and exits when the channel is closed and empty, and defer p.wg.Done() tells the WaitGroup it has finished. timed records the time a job is submitted and returns a wrapper that, the moment a worker starts the job, adds the elapsed queue time to waitNs; that is how queue wait is measured without touching the caller's job. Stats divides the total by finished jobs, so the mean is exact once the pool is idle and approximate while jobs are still running. The test fails if a worker drops an accepted job or if a submit races the close; it would not notice a Shutdown that returned before the workers finished, which the Done:4 printed right after shutdown: <nil> in the go run output is what shows. In TrySubmit, the select has a default branch, so if the buffered channel has room the send happens and the call returns nil, and if it is full the default branch runs at once, counts a rejection and returns ErrQueueFull: that one construct is the whole non-blocking policy. Submit has no default; its select waits for either queue space or ctx.Done(), whichever comes first. In main.go, the started channel lets the program wait until both workers are really running a job before it fills the queue, and release is a channel that blocked jobs wait on, so closing it frees them all.

Run in a golang:1.23 container (go1.23.12, linux/arm64) with a go.mod containing module workerpool and go 1.23:

gofmt -l . && go vet ./... && go run -race . && go test -race -count=20 ./...

gofmt -l lists files that are not in Go's standard format and go vet reports suspicious constructs such as misused locks or wrong format verbs; both print nothing when the code is clean. The -race flag builds with Go's race detector, which reports a data race (two goroutines touching the same memory at the same time, at least one of them writing, with no synchronisation between them) if one happens while the program runs. Output of go run -race . (the queue-wait figure is timing-dependent and differs on every run; the rest is deterministic):

submit 1: <nil>
submit 2: <nil>
submit 3: pool: queue full
submit 4: pool: queue full
submit 5: pool: queue full
stats: {QueueLen:2 QueueCap:2 Active:2 Workers:2 Rejected:3 Done:0 MeanQueueWait:0s}
blocking submit: context deadline exceeded
shutdown: <nil>
stats: {QueueLen:0 QueueCap:2 Active:0 Workers:2 Rejected:4 Done:4 MeanQueueWait:25.323822ms}
submit after shutdown: pool: closed
shutdown with stuck job: context deadline exceeded

and go test -race -count=20 ./... printed ok workerpool 1.247s, so the race detector found no data race across 20 repetitions.

What the output shows: with 2 workers busy and a queue of 2, the first two submits queue and the next three are rejected (Rejected:3); the 50 ms blocking submit gives up with the context error and counts as a fourth rejection; after the blocked jobs are released, Shutdown returns nil having run all four accepted jobs (Done:4); and the stuck job is cancelled when the 100 ms deadline passes, with Shutdown reporting the deadline error.

Sizing and the default

Size workers from the work: about the core count for CPU-bound jobs, more for jobs that wait (workers = cores x (1 + wait/compute)). Size the queue from a latency budget: queue length ~ allowed wait x service rate (for example 0.2 s x 200 jobs/s = 40). Start with TrySubmit and a 503 for service traffic; use Submit with a deadline for internal producers.

Recommendation and what would change it

Bounded queue, fail-fast submit, drain with a deadline, and the five numbers above on a dashboard. Choose blocking submit if producers are batch jobs that can safely wait; choose drop-oldest for telemetry-like data where only the newest matters. If jobs run long and must survive restarts, a pool in memory is the wrong tool: use a durable queue.

What the GIL does and does not prevent

The Global Interpreter Lock (GIL) is a lock inside the standard CPython interpreter. Only the thread holding it can execute Python bytecode (the interpreter's instructions), so in one process, two threads cannot run pure-Python code on two cores at the same time. A thread gives up the GIL while it waits on a blocking operation such as a socket read, time.sleep or file I/O, and many C extensions release it during long native work (the hashlib demo below does). It does not stop threads from overlapping waiting, and it does not make your own multi-step updates safe: counter += 1 on shared data can still interleave between threads, so you still need locks for correctness.

So the performance rule is:

• I/O-bound (mostly waiting on network or disk): threads or asyncio are fine, because the GIL is released while waiting.
• CPU-bound pure Python (parsing, number crunching in loops): threads do not speed it up; use processes, or move the hot loop into a C extension or library (NumPy, a compiled module) that releases the GIL.

Evidence

import hashlib, pickle, sys, time, sysconfig
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def py_cpu(n):                     # pure Python: holds the GIL while it runs
    x = 0
    for i in range(n):
        x += i * i
    return x

def sha_cpu(buf):                  # hashlib releases the GIL for large buffers
    h = hashlib.sha256()
    for _ in range(40):
        h.update(buf)
    return h.hexdigest()

def timed(label, pool_cls, workers, fn, args):
    t = time.perf_counter()
    with pool_cls(workers) as ex:
        list(ex.map(fn, args))
    print("  %-34s %5.2f s" % (label, time.perf_counter() - t))

if __name__ == "__main__":
    print(sys.version.split()[0], "GIL disabled build:", sysconfig.get_config_var("Py_GIL_DISABLED"))
    W = 4
    jobs = [3_000_000] * W
    print("pure-Python CPU work, %d jobs" % W)
    t = time.perf_counter(); [py_cpu(n) for n in jobs]
    print("  %-34s %5.2f s" % ("sequential", time.perf_counter() - t))
    timed("%d threads" % W, ThreadPoolExecutor, W, py_cpu, jobs)
    timed("%d processes" % W, ProcessPoolExecutor, W, py_cpu, jobs)

    buf = b"x" * (8 * 1024 * 1024)
    bufs = [buf] * W
    print("hashlib.sha256 on 8 MiB buffers (releases the GIL), %d jobs" % W)
    t = time.perf_counter(); [sha_cpu(b) for b in bufs]
    print("  %-34s %5.2f s" % ("sequential", time.perf_counter() - t))
    timed("%d threads" % W, ThreadPoolExecutor, W, sha_cpu, bufs)

    print("cost of going multi-process")
    t = time.perf_counter()
    with ProcessPoolExecutor(1) as ex:
        ex.submit(int, 1).result()
    print("  %-34s %5.2f s" % ("first task incl. worker start", time.perf_counter() - t))
    big = list(range(2_000_000))
    t = time.perf_counter(); blob = pickle.dumps(big); pickle.loads(blob)
    print("  pickle round trip of 2M ints (%d MB): %.2f s" % (len(blob) // 10**6, time.perf_counter() - t))
    t = time.perf_counter()
    with ProcessPoolExecutor(1) as ex:
        ex.submit(sum, big).result()
    print("  send 2M-int list to a worker, sum it: %.2f s" % (time.perf_counter() - t))

Sample run, docker run --rm --cpus=4 on python:3.12-slim (Python 3.12.15). Timings are from a shared, noisy host:

3.12.15 GIL disabled build: None
pure-Python CPU work, 4 jobs
  sequential                          0.34 s
  4 threads                           0.31 s
  4 processes                         0.12 s
hashlib.sha256 on 8 MiB buffers (releases the GIL), 4 jobs
  sequential                          0.45 s
  4 threads                           0.12 s
cost of going multi-process
  first task incl. worker start       0.00 s
  pickle round trip of 2M ints (9 MB): 0.06 s
  send 2M-int list to a worker, sum it: 0.12 s

The shape: for pure Python, four threads take about as long as running the jobs one after another, while four processes finish in well under half the time (less than the full 4x because starting processes and returning results costs something). For the hashlib work that releases the GIL, threads do speed up. On one CPU the threads took about the same time as sequential and the processes were no faster (slightly slower, from start-up cost), since one core is the limit; with more CPUs the process speedup on four jobs was real but varied from run to run on a shared host, because only four jobs exist to run in parallel and process start-up is a fixed cost. GIL disabled build: None means this is a standard build with the GIL.

When to use processes

Use processes when the work is CPU-bound pure Python and you want multi-core throughput, when you want isolation (a crash, a memory leak or a stuck C library kills one worker, not the service), or when the web server model is already one process per worker. Many Python web servers are pre-fork: a parent starts N worker processes up front and each handles requests on its own, which gives multi-core throughput with no GIL contention between workers. A common arrangement is N processes (about one per core) each running a small thread pool or an event loop for I/O overlap.

What going multi-process costs

1. Data crosses a process boundary by serialization. ProcessPoolExecutor uses multiprocessing, and per the docs only picklable objects can be sent and returned (no lambdas or local functions). Arguments and results are pickled (turned into bytes) and unpickled, which costs CPU time proportional to size: in the sample, a 2-million-int list (about 9 MB pickled) took 0.06 s to pickle and unpickle, and sending it to one worker and summing took 0.12 s, far more than the sum itself would take. If the job is cheaper than moving its data, processes lose. Send small arguments (an id, a file path) and let each worker load its own data, or use shared memory for large arrays.
2. Memory is multiplied. Each worker is a full interpreter with its own heap. A start method is how a new worker process comes into being: fork copies the running parent process; spawn starts a fresh interpreter that imports your code again; forkserver asks a small helper process, started early while it is still clean, to fork each new worker. With the fork start method (the default on Linux up to Python 3.13; since 3.14 the Linux default is forkserver, and code that needs fork must ask for it explicitly, as the script below does) the child shares the parent's memory pages with the parent until one side writes to them, a scheme called copy-on-write: the OS gives the writer its own private copy of just that page. Reading Python objects still writes to them, because every object carries a reference count (a counter of how many references point at it) and even len(s) temporarily bumps and drops that counter. So the sharing erodes. This shipped script measures it with PSS (proportional set size: each shared page is divided among the processes that share it, so two processes sharing a 100 MB region are charged 50 MB each). The script reads the Pss: line of Linux's /proc/self/smaps_rollup, which is in kB, hence the division by 1024 to get MB:

import multiprocessing as mp, os

def pss_mb():
    for line in open("/proc/self/smaps_rollup"):
        if line.startswith("Pss:"):
            return int(line.split()[1]) / 1024

DATA = [str(i) * 3 for i in range(1_500_000)]     # built before the fork, shared copy-on-write

def worker(touch, q):
    before = pss_mb()
    if touch:
        n = 0
        for s in DATA:                            # read-only use: len() of each object
            n += len(s)
    q.put((before, pss_mb()))

if __name__ == "__main__":
    mp.set_start_method("fork")
    print("parent PSS: %.0f MB" % pss_mb())
    for touch in (False, True):
        q = mp.Queue()
        p = mp.Process(target=worker, args=(touch, q)); p.start()
        b, a = q.get(); p.join()
        print("child that %s the shared list: PSS before %.0f MB, after %.0f MB" % ("reads" if touch else "ignores", b, a))

Sample run (same container image): parent PSS: 117 MB, child that ignores the shared list: PSS before 60 MB, after 60 MB, child that reads the shared list: PSS before 60 MB, after 106 MB. Why the numbers move: the parent's 117 MB is measured before any child exists, so every page is the parent's alone. Right after the fork, parent and child share nearly all pages, so each is charged about half of the shared total, which is why the child's "before" is 60 MB, roughly half of 117 (the parent's own charge falls to a similar figure at that point). When the child only ignores the list, nothing is written and the 60 MB stays. When it reads every string, the reference-count writes force the OS to copy the pages that hold the strings, those copies become private to the child and are charged to it in full, and its charge rises to 106 MB. The child that only read the list pulled most of it into private memory. Budget memory per worker accordingly, not as "parent plus a little".
3. Start-up and warm-up. In the sample, with fork, starting a worker took a few milliseconds. The spawn method (the default on macOS and Windows) starts a fresh interpreter and re-imports your modules, which the docs describe as slower than fork; and per the docs, forking a process that already has threads is problematic and Python 3.12 emits a DeprecationWarning for it. Warm caches, connection pools and loaded models are per worker, not shared, so open database connections per process and watch the total (workers x pool size) against the database's connection limit.
4. Coordination and state. Shared in-memory caches, counters and locks need an external store or explicit shared memory. Debugging and tracing span processes.
5. Oversubscription. More processes than real cores stops helping, as in any CPU-bound pool.

Free-threaded builds

The GIL rule above describes the default CPython build that production servers run today; the free-threaded build is the one place it changes.

PEP 779 describes the free-threaded (GIL-less) build as experimental in Python 3.13 and sets the criteria for it to become officially supported, but still optional, in 3.14, the release the PEP targets for that step. The default build keeps the GIL. Check at runtime with sys._is_gil_enabled() (documented in the free-threading how-to). Adopting it means your dependencies must support it; a pure-Python thread pool could then use several cores, but threads would then also expose more of your own races.

Recommendation

For a typical I/O-heavy service keep threads or asyncio; add a process-per-core layer (pre-fork) for CPU-bound work or to use all cores, offload the heavy compute to a process pool or a native library, and size workers by cores and memory, not by hope. If the work is a long CPU job per request, a job queue with worker processes avoids tying request latency to a process pool.

Decide on the work each request does, then on the failure you can live with

A thread per request model gives each in-flight request its own operating-system (OS) thread, an independent flow of execution the kernel schedules onto CPU cores. Code reads top to bottom and a blocking call just parks that thread. An event loop model runs many requests on one thread (per process), which switches between requests only when a handler waits (await or a callback); a handler that blocks stops them all. A third option is a lightweight thread scheduled by the language runtime: goroutines in Go, virtual threads in Java 21. They read like blocking code but cost far less than OS threads.

Recommendation for a new service whose handlers mostly wait on databases and downstream calls: use the runtime-scheduled lightweight option if the stack has it (Java 21 virtual threads, Go), otherwise an event loop with a small separate pool for the work that blocks or burns CPU. Choose plain thread-per-request only when concurrency is modest (hundreds, not tens of thousands) and the team's code or libraries are synchronous. What would flip it: heavy CPU per request (then cores are the limit and the model barely matters; add worker processes), a mandatory blocking library you cannot replace (thread per request or an offload pool), or a team that cannot hold the discipline "nothing blocks the loop".

Latency and throughput

The sections below run one comparison on each runtime that matters for the model: a Java program for in-flight capacity and the pinning caveat, and a Python program for memory per connection. The table after them covers what a blocked handler does in each model.

When requests mostly wait, throughput is limited by how many can be waiting at once. By Little's law (requests in flight = arrival rate x time in system), 10,000 requests/s each spending 100 ms blocked needs 1,000 requests in flight. A pool of 200 OS threads caps this at 200 / 0.1 s = 2,000 requests/s no matter how many cores exist. An event loop or lightweight threads have no such cap from the thread count; their cap is CPU and the downstream's capacity.

The program below runs 10,000 handlers that each block 100 ms, once on 200 platform threads and once with one virtual thread per task, then times 64 sleeps inside synchronized (a Java monitor lock: the keyword makes a block of code run by one thread at a time per lock object).

How to read the program: handler() is a stand-in request that sleeps 100 ms (blocked, no CPU). run submits the requests to an executor (a pool that runs submitted tasks) and a CountDownLatch, a counter that starts at the request count and releases await() when every task has called countDown(), so the timer stops when the last request finishes. newFixedThreadPool(200) is 200 OS threads; newVirtualThreadPerTaskExecutor() starts one virtual thread per task; try (...) closes the executor afterwards. In the last block each task sleeps while holding its own private lock object, so any slowdown comes from the runtime, not from tasks waiting for each other.

import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

public class ModelCompare {
    // A "handler" that spends 100 ms blocked on a downstream call, then a little CPU.
    static void handler() {
        try { Thread.sleep(100); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    static double run(ExecutorService ex, int requests) throws Exception {
        long t = System.nanoTime();
        CountDownLatch done = new CountDownLatch(requests);
        for (int i = 0; i < requests; i++) ex.submit(() -> { handler(); done.countDown(); });
        done.await();
        return (System.nanoTime() - t) / 1e9;
    }

    public static void main(String[] a) throws Exception {
        System.out.println("availableProcessors = " + Runtime.getRuntime().availableProcessors());
        int requests = 10_000;
        try (ExecutorService ex = Executors.newFixedThreadPool(200)) {
            double s = run(ex, requests);
            System.out.printf("200 platform threads : %d blocking requests in %.2f s (%.0f req/s)%n", requests, s, requests / s);
        }
        try (ExecutorService ex = Executors.newVirtualThreadPerTaskExecutor()) {
            double s = run(ex, requests);
            System.out.printf("virtual threads      : %d blocking requests in %.2f s (%.0f req/s)%n", requests, s, requests / s);
        }
        // Little's law: 10_000 requests of 100 ms on 200 threads can finish no faster than 10_000 * 0.1 / 200 s
        System.out.printf("lower bound for 200 threads: %.2f s%n", requests * 0.1 / 200);
        // Pinning: blocking inside synchronized.
        try (ExecutorService ex = Executors.newVirtualThreadPerTaskExecutor()) {
            long t = System.nanoTime();
            int n = 64;
            CountDownLatch done = new CountDownLatch(n);
            for (int i = 0; i < n; i++) ex.submit(() -> {
                // each task uses its own monitor so tasks do not serialize on one lock
                Object own = new Object();
                synchronized (own) { handler(); }
                done.countDown();
            });
            done.await();
            System.out.printf("64 x 100 ms sleeps inside synchronized: %.2f s%n", (System.nanoTime() - t) / 1e9);
        }
    }
}

Run in an eclipse-temurin:21-jdk container (OpenJDK 21.0.12) with --cpus=1, then --cpus=4, it printed (availableProcessors reflected the CPU limit, not the host; timings are from a shared host and vary):

--- cpus=1
availableProcessors = 1
200 platform threads : 10000 blocking requests in 5.28 s (1894 req/s)
virtual threads      : 10000 blocking requests in 0.13 s (76548 req/s)
lower bound for 200 threads: 5.00 s
64 x 100 ms sleeps inside synchronized: 6.47 s
--- cpus=4
availableProcessors = 4
200 platform threads : 10000 blocking requests in 5.16 s (1939 req/s)
virtual threads      : 10000 blocking requests in 0.14 s (69427 req/s)
lower bound for 200 threads: 5.00 s
64 x 100 ms sleeps inside synchronized: 1.67 s

The shape: the 200-thread pool sits at its arithmetic ceiling (about 5 s, whatever the CPU count) while virtual threads finish in about 0.14 s because waiting costs almost nothing. The last line shows the caveat. Virtual threads are run by a small set of ordinary OS threads called carriers; the number of carriers defaults to the number of processors. Normally a virtual thread that blocks is unmounted: taken off its carrier so the carrier can run another virtual thread. On JDK 21 a virtual thread that blocks inside synchronized is pinned: it cannot unmount, so it holds its carrier for the whole block. With as many carriers as processors, so 64 pinned sleeps of 100 ms take about 64 x 0.1 / carriers seconds (6.4 s at one carrier, 1.6 s at four, matching the run). A JEP is a JDK Enhancement Proposal, the document that specifies a Java feature. JEP 444 documents the two pinning cases in JDK 21 (inside synchronized, and in native or foreign calls); JEP 491 removed the synchronized case in Java 24, leaving pinning while a native frame (a call into non-Java code) is on the stack, such as class loading or class initialisation that blocks.

Memory per connection

Each OS thread reserves a stack and kernel bookkeeping; a task on an event loop is a small heap object. A second program holds 2,000 idle waiting "connections" as asyncio tasks and as OS threads and reads resident memory (RSS, the memory actually held in RAM by the process) from /proc/self/status:

import asyncio, threading, resource

def rss_mb():
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS"):
                return int(line.split()[1]) / 1024

def threads(n):
    stop = threading.Event()
    ts = [threading.Thread(target=stop.wait) for _ in range(n)]
    base = rss_mb()
    for t in ts: t.start()
    used = rss_mb() - base
    stop.set()
    for t in ts: t.join()
    return used

async def tasks(n):
    stop = asyncio.Event()
    base = rss_mb()
    ts = [asyncio.create_task(stop.wait()) for _ in range(n)]
    await asyncio.sleep(0)
    used = rss_mb() - base
    stop.set()
    await asyncio.gather(*ts)
    return used

n = 2000
asyncio.run(asyncio.sleep(0))   # warm up the interpreter
a = asyncio.run(tasks(n))
t = threads(n)
print("%d idle connections: asyncio tasks +%.1f MB RSS (%.1f KB each), OS threads +%.1f MB RSS (%.1f KB each)"
      % (n, a, a*1024/n, t, t*1024/n))

On Linux in python:3.12-slim, two consecutive runs printed:

2000 idle connections: asyncio tasks +0.8 MB RSS (0.4 KB each), OS threads +57.6 MB RSS (29.5 KB each)
2000 idle connections: asyncio tasks +1.8 MB RSS (0.9 KB each), OS threads +59.0 MB RSS (30.2 KB each)

The difference is about 30 to 70 times per idle connection here. RSS counts only touched pages; each thread also reserves virtual address space (the range of addresses the OS sets aside for the process, whether or not it is backed by RAM yet) for its stack, and the OS limits threads per process, so thread counts can hit those limits before RSS looks alarming. Virtual threads keep their stacks as garbage-collected heap objects, and JEP 444 says memory use and garbage-collector activity of thread-per-request and asynchronous code should be roughly similar.

What happens when a handler blocks

Go's runtime documentation says GOMAXPROCS (the setting for how many goroutines may run on CPUs at once) limits the number of CPUs executing simultaneously and that there is no limit on threads blocked in system calls. In Node.js, file system, DNS lookup (dns.lookup), zlib and some crypto calls run on libuv's worker pool, whose default size is 4 threads (libuv docs; UV_THREADPOOL_SIZE changes it), so a handful of slow file or crypto calls can queue all the rest.

How failures show up in production

• Thread per request: thread count and queue depth climb, latency rises, then rejections or out-of-memory (OOM) errors, when the process is killed or refuses to allocate. Visible in thread dumps (jstack is the JDK tool that prints every thread's stack and state).
• Event loop: all endpoints get slow together, CPU is high on one core and idle on others, and timers fire late. A single bad regular expression or a large JSON parse can do it.
• Lightweight threads: unbounded concurrency moves the limit to the downstream: connection pools and databases saturate. Cap concurrency with a semaphore (JEP 444 says not to pool virtual threads), because a limit on thread count is gone.

Whatever you choose, set a timeout on every outbound call, bound every queue, and expose in-flight count and loop lag so the first failure is a graph, not a page.