Concurrency & Asynchronous Performance Questions

Why everything times out while CPU is low

An async server runs many requests on a few threads. A request that is waiting holds almost no CPU, but it can hold a shared, finite resource: a database connection, an outbound HTTP connection, a thread-pool slot, a file descriptor (fd, the OS handle for an open file or socket), or a lock. When the dependency slows from 20 ms to seconds, each request holds its resource a hundred times longer or more (150 times at 3 s). By Little's law (requests in the system = arrival rate x time each spends in it), the number held rises in proportion to latency: 30 requests/s that held a resource for 0.02 s occupy 0.6 of it on average, and at 3 s they want 90. A pool of 20 is exhausted within a second, new requests queue for a connection, and endpoints that never touch the slow dependency fail too because they share the pool. Low CPU is the signature: the process is waiting, not working.

Which shared resources could be saturating

The usual explanations in this pattern are items 1 to 3: a shared pool that something holds across a slow call. Items 4 to 7 are worth ruling out but explain fewer incidents.

1. A database connection pool shared by all endpoints (above all if a request holds a connection while it calls something else).
2. An outbound HTTP client pool or per-host connection limit.
3. A thread pool used for blocking calls (all workers parked on the slow call).
4. File descriptors / sockets: open connections accumulate; the per-process limit on open files and sockets (ulimit -n shows it) or the kernel's ephemeral ports (the temporary local port numbers it hands out to outgoing connections, a finite range) run out. Errors read "too many open files".
5. A lock or a shared cache whose critical section includes a call to the slow dependency.
6. The event loop itself, if retries or a flood of timeouts keep it busy (usually shows as high CPU on one core, so it is not this case, but rule it out).
7. The database's own limit: all instances together can ask for more connections than the database allows. PostgreSQL's max_connections (the server-wide cap on simultaneous client connections) defaults to typically 100, with some slots reserved for superusers (PostgreSQL docs).

Reproducing the blast radius

A semaphore is a counter of free slots: acquire() takes one and waits when none are left, release() returns one. Here it models a connection pool. asyncio.wait_for(x, t) gives up on x after t seconds, and asyncio.shield lets the server's work continue after the caller has stopped waiting.

The program models one server instance: a shared pool of 20 database connections, three endpoints (a calls a dependency, b and c only use the database for 5 ms), 90 requests/s, a 1 s caller timeout. Endpoint a holds a connection while it waits for the dependency. Reading the code: Server.db takes a pooled connection, uses it for work_s seconds and returns it, recording how long it waited for the connection. handle for endpoint a either holds a connection while it waits on the dependency (hold_conn_during_call, the anti-pattern) or calls the dependency first and takes a connection only afterwards. call_dependency optionally adds a timeout and a bulkhead (a small separate semaphore, so a slow dependency can fill only its own compartment). one_request counts a request as failed if the caller's 1 s timeout passes. Four scenarios follow, then the sizing arithmetic. The server keeps working after a caller times out, as most real servers do.

import asyncio, time

POOL_SIZE = 20          # shared database connection pool of one server instance
ACQUIRE_TIMEOUT = 0.5   # how long a request waits for a pooled connection
CLIENT_TIMEOUT = 1.0    # the caller gives up after this
RATE = 90               # requests per second, spread evenly over endpoints a, b, c
DURATION = 5.0

class Server:
    def __init__(self, dep_latency, hold_conn_during_call, bulkhead, dep_timeout):
        self.pool = asyncio.Semaphore(POOL_SIZE)
        self.dep_latency = dep_latency
        self.hold = hold_conn_during_call
        self.dep_slots = asyncio.Semaphore(bulkhead) if bulkhead else None
        self.dep_timeout = dep_timeout
        self.db_wait = []                     # time spent waiting for a pooled connection

    async def db(self, work_s):
        t = time.perf_counter()
        await asyncio.wait_for(self.pool.acquire(), ACQUIRE_TIMEOUT)
        self.db_wait.append(time.perf_counter() - t)
        try:
            await asyncio.sleep(work_s)
        finally:
            self.pool.release()

    async def call_dependency(self):
        if self.dep_slots is not None:
            if self.dep_slots.locked():
                raise RuntimeError("bulkhead full: fail fast")
            await self.dep_slots.acquire()
        try:
            coro = asyncio.sleep(self.dep_latency)
            await (asyncio.wait_for(coro, self.dep_timeout) if self.dep_timeout else coro)
        finally:
            if self.dep_slots is not None:
                self.dep_slots.release()

    async def handle(self, endpoint):
        if endpoint != "a":
            await self.db(0.005)              # b and c only need the database
            return
        if self.hold:                         # anti-pattern: hold a connection while waiting on the dependency
            await asyncio.wait_for(self.pool.acquire(), ACQUIRE_TIMEOUT)
            try:
                await self.call_dependency()
            finally:
                self.pool.release()
        else:
            await self.call_dependency()
            await self.db(0.005)

async def one_request(server, endpoint, stats):
    t = time.perf_counter()
    task = asyncio.create_task(server.handle(endpoint))     # the server keeps working after the client leaves
    try:
        await asyncio.wait_for(asyncio.shield(task), CLIENT_TIMEOUT)
        stats[endpoint]["ok"] += 1
    except Exception:
        stats[endpoint]["fail"] += 1
        task.add_done_callback(lambda f: f.exception() if not f.cancelled() else None)

async def scenario(name, **kw):
    server = Server(**kw)
    stats = {e: {"ok": 0, "fail": 0} for e in "abc"}
    cpu0, t0 = time.process_time(), time.perf_counter()
    tasks = []
    for i in range(int(RATE * DURATION)):
        tasks.append(asyncio.create_task(one_request(server, "abc"[i % 3], stats)))
        await asyncio.sleep(1 / RATE)
    await asyncio.gather(*tasks)
    cpu = (time.process_time() - cpu0) / (time.perf_counter() - t0)
    waits = sorted(server.db_wait) or [0]
    print("%-46s" % name, " ".join("%s ok %3d%%" % (e, 100 * s["ok"] // (s["ok"] + s["fail"])) for e, s in stats.items()),
          " pool wait p95 %5.0f ms  cpu %2.0f%%" % (waits[int(.95 * (len(waits) - 1))] * 1000, 100 * cpu))

async def main():
    await scenario("dependency healthy (20 ms)",            dep_latency=0.02, hold_conn_during_call=True,  bulkhead=0, dep_timeout=0)
    await scenario("dependency slow (3 s), conn held",      dep_latency=3.0,  hold_conn_during_call=True,  bulkhead=0, dep_timeout=0)
    await scenario("slow, conn NOT held across the call",   dep_latency=3.0,  hold_conn_during_call=False, bulkhead=0, dep_timeout=0)
    await scenario("slow, held + timeout 0.3 s + bulkhead 5", dep_latency=3.0, hold_conn_during_call=True,  bulkhead=5, dep_timeout=0.3)

    # sizing arithmetic with stated, hypothetical inputs
    instances, per_instance_pool, max_connections = 12, POOL_SIZE, 100
    print("\nconnections if every instance fills its pool: %d x %d = %d (database max_connections %d)"
          % (instances, per_instance_pool, instances * per_instance_pool, max_connections))
    rate_a, normal_latency = RATE / 3, 0.02
    print("normal in-flight calls to the dependency (Little's law): %.0f x %.2f = %.1f" % (rate_a, normal_latency, rate_a * normal_latency))
    print("if the call takes 3 s, in-flight calls without a cap: %.0f x 3 = %.0f" % (rate_a, rate_a * 3))

asyncio.run(main())

Run in a python:3.12-slim container with --cpus=1, it printed (percentages for b and c in the second row vary widely from run to run, roughly from 10% to 35%, and a stays at 0%; the pool-wait p95 in that row is usually just under the 0.5 s acquire timeout but can read as low as 0 ms, because it is taken only over the acquisitions that succeeded; CPU stays in the low single digits throughout):

dependency healthy (20 ms)                     a ok 100% b ok 100% c ok 100%  pool wait p95     0 ms  cpu  1%
dependency slow (3 s), conn held               a ok   0% b ok  26% c ok  32%  pool wait p95   495 ms  cpu  2%
slow, conn NOT held across the call            a ok   0% b ok 100% c ok 100%  pool wait p95     0 ms  cpu  3%
slow, held + timeout 0.3 s + bulkhead 5        a ok   0% b ok 100% c ok 100%  pool wait p95     0 ms  cpu  1%

connections if every instance fills its pool: 12 x 20 = 240 (database max_connections 100)
normal in-flight calls to the dependency (Little's law): 30 x 0.02 = 0.6
if the call takes 3 s, in-flight calls without a cap: 30 x 3 = 90

Reading the second row: endpoint a takes 30 of the 90 requests per second and holds a connection for 3 s each, so about 30 x 3 = 90 connections are wanted against 20 available; the 20 are taken within about a second, everyone else queues for a connection until the 0.5 s acquire timeout, and b and c queue with them. The healthy endpoints lose most of their requests and the pool-wait p95 is usually just under the 0.5 s acquire timeout (it counts only waits that ended in a connection, so it can read lower when few late ones succeed): the time is going into waiting for the shared pool, not into the dependency, for these endpoints. In the last two rows a still fails (the dependency really is down) but b and c are untouched. The 12 instances x 20 connections figure uses hypothetical inputs.

How to tell which resource it is

Checks 1 to 3 usually settle it; take them in this order, each with what it tells you:

1. Is it the slow dependency? Compare latency of the endpoints that call it and those that do not. If unrelated endpoints fail too, a shared resource is the link.
2. Pool metrics: active, idle, waiting count and wait time (HikariCP, a common Java connection pool, exposes them as metrics; SQLAlchemy, a Python database library, prints them from pool.status(); pg_stat_activity is a PostgreSQL view with one row per database session, showing its state and last state change). Active at maximum with a growing wait queue means pool exhaustion, and the wait-time histogram shows how long requests queue.
3. What the held connections are doing: pg_stat_activity showing many sessions in state idle (or idle in transaction if the request opened a transaction and has not finished it) with old state_change times, while the pool metrics from check 2 show every connection borrowed, means requests hold connections while doing something else, such as an outbound call: the pattern in the second row. Read it together with check 2, because a connection that sits unborrowed in the pool also shows as idle with an old state_change, so idle sessions alone do not prove they are held.
4. Thread dumps (jstack, py-spy dump): many threads parked in the same frame (a socket read, a pool acquire) point at the saturated resource.
5. File descriptors and sockets: ls /proc/<pid>/fd | wc -l against ulimit -n, and ss -s (a summary of socket counts by state) for socket states. TIME_WAIT is the state a closed TCP connection lingers in for a short while, so a flood of short outbound connections piles them up and eats ephemeral ports. /proc/<pid>/fd lists a process's open descriptors, one entry each, so counting its entries counts them. A worked reading: 1,020 entries against a limit of 1,024 would make the descriptor limit the saturated resource; 60 entries would rule it out.
6. Event loop lag and CPU per core: rules the loop in or out.
7. Instance arithmetic: instances x pool size against the database's max_connections. If the sum is larger, the pools together can exhaust the database even when each looks fine.

Containing the blast radius next time

1. Do not hold a scarce resource across a call to something else. Take the connection, do the database work, release it, then call out (the third row).
2. Timeouts on every outbound call, shorter than the caller's timeout, so a slow dependency costs seconds of a slot rather than minutes.
3. Bulkheads: a separate, small, bounded pool or semaphore per dependency, sized from rate x normal latency with headroom (0.6 in flight normally, so 5 is generous) so a slow dependency can fill only its own compartment. Fail fast when it is full (the fourth row).
4. Circuit breaker (named after the electrical device): after repeated failures or timeouts, stop calling the dependency for a while and return a fallback or an error immediately.
5. Sized pools across the fleet: instances x pool size must fit under max_connections (12 x 20 = 240 over 100 here), or use a connection proxy.
6. Load shedding (refusing some work early so the rest stays healthy) and alerts on pool wait time and in-flight count, so the page comes before the timeouts do.

Recommendation: ship 1 and 2 first (cheap, remove the failure mode), then bulkheads per dependency; add a circuit breaker once you have measured how the dependency fails.

Confirm the lock is the limit before changing anything

A lock (mutex) lets one thread at a time run a critical section, the code that touches shared data. Contention is threads queueing for it. The signature: throughput stops rising (or falls) as you add threads, CPU stays well below the core count, and threads show as blocked. Confirm with evidence:

1. Thread dumps (jstack <pid> several times): many threads in the state BLOCKED (on object monitor) (the JVM's label for a thread waiting to enter a synchronized block) on the same lock address.
2. A lock profile: async-profiler (a sampling profiler for Java; its -e lock option profiles lock contention) reports the lock or monitor class and the nanoseconds spent entering it (async-profiler documentation), so it shows where threads wait; step 4 is what tells you whether the section itself is slow.
3. A scaling test: run 1, 2, 4, 8 threads and plot throughput. A flat or falling line while cores idle is the case described.
4. Read the critical section: is anything slow inside it (allocation, logging, serialisation, I/O, a call out)? Moving that out is the cheapest fix.

The ceiling follows Amdahl's law: if a fraction f of every operation happens under the one lock, no number of threads can beat 1/f times the single-thread rate. With N threads the bound is 1 / (f + (1 - f) / N): the serial part f cannot be shared, and the rest shrinks N-fold. Worked for f = 0.30 and N = 8: 1 / (0.30 + 0.70 / 8) = 1 / (0.30 + 0.0875) = 2.58, shown as 2.6x. (us is microseconds.) The script computes the bound for a hypothetical operation of 1.0 us with 0.30 us under the lock, then the effect of splitting the lock into stripes (a stripe is one of several independent pieces, each with its own lock; the script's simplification: it assumes keys spread evenly and divides f by the stripe count).

# Hypothetical inputs: each operation takes 1.0 us in total, of which 0.30 us is spent holding the cache lock.
total_us, locked_us = 1.0, 0.30
f = locked_us / total_us
print("serial fraction f = %.2f, so throughput can never exceed 1/f = %.1fx the single-thread rate" % (f, 1 / f))
for stripes in (1, 4, 16, 64):
    f_eff = f / stripes                      # keys spread evenly over independent locks
    print("stripes=%2d  effective f=%.4f  bound on speedup with 8 threads: %.1fx"
          % (stripes, f_eff, 1 / (f_eff + (1 - f_eff) / 8)))

Output:

serial fraction f = 0.30, so throughput can never exceed 1/f = 3.3x the single-thread rate
stripes= 1  effective f=0.3000  bound on speedup with 8 threads: 2.6x
stripes= 4  effective f=0.0750  bound on speedup with 8 threads: 5.2x
stripes=16  effective f=0.0187  bound on speedup with 8 threads: 7.1x
stripes=64  effective f=0.0047  bound on speedup with 8 threads: 7.7x

Four structures, measured

The bound is a ceiling for an ideal scheduler. The measurements below can sit under it, and with four or more cores the global lock even falls below one thread's rate, because Amdahl's formula ignores the cost of threads going to sleep and being woken when they fail to get the lock.

The program compares one global lock, a read-write lock (many readers or one writer), 64 lock stripes (the key picks one of 64 independent maps each with its own lock) and ConcurrentHashMap, under 90% writes on 100,000 keys, at 1, 2, 4 and 8 threads. Reading the code: Cache is the two-method interface all four share; xorshift is a cheap per-thread random-number generator, so the threads do not contend on a shared one; the CyclicBarrier makes every thread start together; sink is a result the loop keeps so the JIT (the JVM's run-time compiler, which can delete work whose result is unused) cannot remove the reads; the first pass is a warm-up so the JIT has compiled the hot code before the printed pass. CAS (compare-and-set) is a hardware instruction that changes a value only if it still holds what you expected, the basis of lock-free updates that ConcurrentHashMap uses in places.

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;

public class CacheScaling {
    interface Cache { void put(int k, int v); Integer get(int k); }

    // 1. one lock around everything
    static class Global implements Cache {
        final Map<Integer,Integer> m = new HashMap<>();
        public synchronized void put(int k, int v) { m.put(k, v); }
        public synchronized Integer get(int k) { return m.get(k); }
    }
    // 2. read-write lock (writers still exclusive)
    static class RW implements Cache {
        final Map<Integer,Integer> m = new HashMap<>();
        final ReadWriteLock l = new ReentrantReadWriteLock();
        public void put(int k, int v) { l.writeLock().lock(); try { m.put(k, v); } finally { l.writeLock().unlock(); } }
        public Integer get(int k) { l.readLock().lock(); try { return m.get(k); } finally { l.readLock().unlock(); } }
    }
    // 3. lock striping: the key picks one of N independent maps, each with its own lock
    static class Striped implements Cache {
        final Map<Integer,Integer>[] maps; final Object[] locks;
        @SuppressWarnings("unchecked")
        Striped(int n) { maps = new Map[n]; locks = new Object[n];
            for (int i = 0; i < n; i++) { maps[i] = new HashMap<>(); locks[i] = new Object(); } }
        int s(int k) { return (k * 0x9E3779B9 >>> 16) % maps.length; }
        public void put(int k, int v) { int i = s(k); synchronized (locks[i]) { maps[i].put(k, v); } }
        public Integer get(int k) { int i = s(k); synchronized (locks[i]) { return maps[i].get(k); } }
    }
    // 4. the JDK's concurrent map (fine-grained internal locking and CAS)
    static class Chm implements Cache {
        final ConcurrentHashMap<Integer,Integer> m = new ConcurrentHashMap<>();
        public void put(int k, int v) { m.put(k, v); }
        public Integer get(int k) { return m.get(k); }
    }

    static double run(Cache c, int threads, int opsPerThread, int writePercent) throws Exception {
        Thread[] ts = new Thread[threads];
        CyclicBarrier start = new CyclicBarrier(threads + 1);
        for (int t = 0; t < threads; t++) {
            final int seed = t * 7919 + 1;
            ts[t] = new Thread(() -> {
                int x = seed; long sink = 0;
                try { start.await(); } catch (Exception e) { return; }
                for (int i = 0; i < opsPerThread; i++) {
                    x ^= x << 13; x ^= x >>> 17; x ^= x << 5;          // xorshift: cheap per-thread random
                    int key = (x >>> 1) % 100_000;
                    if ((x & 0xFF) * 100 / 256 < writePercent) c.put(key, i);
                    else { Integer v = c.get(key); if (v != null) sink += v; }
                }
                if (sink == 42) System.out.print("");
            });
            ts[t].start();
        }
        start.await();
        long t0 = System.nanoTime();
        for (Thread t : ts) t.join();
        double sec = (System.nanoTime() - t0) / 1e9;
        return threads * (double) opsPerThread / sec / 1e6;              // million ops per second
    }

    public static void main(String[] a) throws Exception {
        System.out.println("availableProcessors = " + Runtime.getRuntime().availableProcessors() + ", 90% writes, 100k keys");
        String[] names = {"one global lock", "read-write lock", "striped x64", "ConcurrentHashMap"};
        for (int pass = 0; pass < 2; pass++) {                           // pass 0 warms up the JIT and is not printed
            if (pass == 1) System.out.printf("%-18s %8s %8s %8s %8s   (million ops/s by thread count)%n", "", "1", "2", "4", "8");
            for (int n = 0; n < names.length; n++) {
                StringBuilder sb = new StringBuilder(String.format("%-18s", names[n]));
                for (int threads : new int[]{1, 2, 4, 8}) {
                    Cache c = switch (n) { case 0 -> new Global(); case 1 -> new RW(); case 2 -> new Striped(64); default -> new Chm(); };
                    double mops = run(c, threads, pass == 0 ? 300_000 : 4_000_000, 90);
                    sb.append(String.format(" %8.1f", mops));
                }
                if (pass == 1) System.out.println(sb);
            }
        }
    }
}

One sample run in an eclipse-temurin:21-jdk container with --cpus=8 (OpenJDK 21; million operations per second; a noisy shared host, so absolute figures and even single-thread values differ from run to run):

availableProcessors = 8, 90% writes, 100k keys
                          1        2        4        8   (million ops/s by thread count)
one global lock        39.0     17.9     19.8     25.7
read-write lock        38.2     22.3     34.6     27.2
striped x64            26.1     40.7     93.6     59.3
ConcurrentHashMap      36.3     66.8    107.9    124.1

The shape, across re-runs at 1, 2, 4 and 8 CPUs (absolute figures swing by tens of percent between runs on a shared host, and a CPU set (--cpuset-cpus) gave different orderings from a CPU quota (--cpus) on one and two CPUs, so read the ordering, not the digits): with four or more cores, or a two-CPU quota, the global lock at 2 threads runs at roughly half its one-thread rate or less and stays below that rate out to 8 threads (threads spend time parking and waking), and striping and the concurrent map beat it from 2 threads up, typically by about 2 to 5 times at 2 and 4 threads (under 2 times at 2 threads in an occasional run). The concurrent map was usually the fastest row at 4 and 8 threads on 4 and 8 CPUs, but its 8-thread figure stayed within roughly a third either way of its 4-thread figure, so more threads than that did not buy more throughput even on 8 cores. Three rows need reading rather than explaining away. First, the read-write lock is erratic rather than a steady copy of the global lock: at 2 threads it is close to the global lock (often within half of it, in either direction), and at 4 and 8 threads it usually sat between the global lock and striping, sometimes close to striping at 8 threads (on two cores it was occasionally level with striping at 4 threads), so it is not reliably far below the striped map; its read path serves only the 10% of operations that are reads, and writers still exclude everyone. Second, single-thread figures are the least stable numbers in the table: striped x64 at one thread (26.1) is below the global lock (39.0) in this sample and in nearly all re-runs on 4 and 8 CPUs, though on one or two CPUs the two were close and either could lead, and a plausible reason is the extra work each operation does (computing the stripe, then touching one of 64 separate maps) that a single uncontended lock avoids, so treat the gap as a tendency, not a law. Third, striped x64 at 8 threads (59.3) is below its 4-thread figure (93.6), and on 8 CPUs that dip repeated in re-runs rather than washing out as noise, while the concurrent map stayed roughly level between 4 and 8 threads; the cause was not isolated here, so measure how this simple striped design scales from 4 to 8 threads on your own hardware. With one core there is no parallelism to unlock, so the order is not reliable and differed between a one-core CPU set and a one-CPU quota: striping gave no consistent gain over the global lock (ahead in some runs, behind in others), the read-write lock was ahead of the global lock at 4 and 8 threads in every re-run, and the concurrent map usually led the global lock on a one-core CPU set but trailed it at 8 threads under a one-CPU quota (every row is noisy, and striping cannot create CPU). With two cores the result depended on how they were given: under a two-CPU quota striping and the concurrent map beat the global lock by about 2 to 6 times at 2 threads, while on a two-core CPU set the global lock was often close to them or ahead at 2 threads, and its 2-thread rate was sometimes above its one-thread rate.

Options and what each costs

Recommendation

Start with the lock profile; if nothing slow is inside the section, replace the map with a concurrent one (or stripes if you need custom eviction per shard), because it fixes the measured failure at the least code. Use batching for write-behind data that tolerates staleness. Keep a read-write lock only if your own scaling test shows a read-dominated load. Verify the fix with the same 1, 2, 4, 8 thread test and by checking the lock profile no longer shows that lock on top. What flips the choice: a hot key (shard cannot spread it, so shorten the section or replicate), a need for exact size or consistent snapshots (stripes plus a global operation, or accept a coarse lock for those operations), or operating-system kernel code that runs in interrupt context (an interrupt handler is code that runs when hardware signals the CPU), where the code must not sleep, so a sleeping lock such as a mutex is not allowed there; use a spinlock (a lock that busy-waits instead of sleeping, in its interrupt-safe form when process-context code shares the data) held for a very short section, or per-CPU data (one copy of the data per CPU, needing no lock), to bound worst-case latency.

Principle: a latency spike from a lock is time spent waiting, so measure waiting, not CPU

A mutex (mutual-exclusion lock) lets one thread at a time run a critical section, the code between lock and unlock. When a thread cannot get the lock it sleeps in the kernel (on Linux, glibc's pthread mutex asks the kernel to park it with a futex system call, a "fast userspace mutex"), and it is off-CPU: it uses no CPU while it waits. That is why lock stalls hide from a normal CPU profile. A service can show 30 percent CPU and terrible p99 latency (the 99th-percentile latency: 1 request in 100 is slower than this) because the missing time is waiting, not computing. Two numbers must stay separate for every lock: hold time (how long the owner stays in the critical section) and wait time (how long the others queue). Wait grows roughly with hold time x the number of threads that want the lock, so the lock to fix is the one with the largest total wait, which is not necessarily the one held longest.

Ordered investigation in production

First terms, in plain words. A flame graph is a picture of sampled call stacks: each box is a function, its width is the share of samples that included it, and the boxes stacked above it are the functions it called, so wide boxes are where the time goes. Sampling means looking at what every thread is doing at a fixed rate (99 times a second here) instead of recording every event. eBPF is a Linux facility that runs small sandboxed programs inside the kernel, which is how tools can record every time a thread goes to sleep without changing the application. A user stack is the call chain in your own code; the kernel stack is the chain inside the operating system beneath it. Context switches are the kernel swapping which thread runs on a core: voluntary ones happen when a thread blocks, involuntary ones when it is forced off after using its time slice.

If you can run only one tool first, run step 1 (it is one command and changes nothing); step 3 is the one that names the lock.

1. Confirm it is waiting. Compare CPU use with latency, and look at context switches (pidstat -w -p <pid> 1: cswch/s is voluntary switches per second, nvcswch/s involuntary ones per second). High voluntary switches with unremarkable CPU suggests threads sleeping on locks or I/O.
2. Take an on-CPU flame graph first (perf record -F 99 -g -p <pid> -- sleep 30, then fold stacks and render with the FlameGraph scripts). Sampling at 99 Hz (99 times a second, an odd rate chosen so it does not line up with timers that fire at 100 Hz) is cheap and safe. It shows whether threads burn CPU inside the critical section (a long hold) or in spin loops.
3. Take an off-CPU profile to see where threads block. Brendan Gregg's off-CPU analysis page describes the method: record blocked time with stack traces, and render it as a flame graph. With offcputime, a tool from the BPF Compiler Collection (bcc, a set of eBPF-based tracing tools) that, according to its examples file, shows stack traces that were blocked, and the total duration they were blocked, the page's example attaches to one process with offcputime -p, taking the PID from pgrep -nx mysqld. Its documented flags include -p PID, -U (user stacks only), -f (folded output for flame graphs) and -m MIN_BLOCK_TIME (microseconds, to drop short sleeps). That page also warns that scheduler events can reach millions per second, and in its test on an 8-CPU system under a heavy MySQL load a 10 second eBPF trace cost 6 to 13 percent of throughput, for about 17 seconds counting the processing afterwards. So: attach to one process, set -m to skip brief blocks, run for tens of seconds, and watch the service's p99 while tracing. With -f each line is one stack, frames joined by semicolons from outermost to innermost, followed by the total blocked time in microseconds. An illustrative line (invented values, only the shape is real):

worker;handle_request;cache_put;pthread_mutex_lock;futex_wait 482311
worker;handle_request;read_socket;recv 120044
worker;idle_loop;queue_pop;pthread_cond_wait;futex_wait 9120330

Read it left to right: thread name, then the call chain, then how long that exact chain spent blocked. The first line is a lock wait inside cache_put (the frame just before pthread_mutex_lock names the call site, and futex_wait is the sleep itself). The second is waiting for network data and the third is an idle worker waiting for a job: both big, neither a lock problem. In a flame graph the same data appears as boxes whose width is blocked time, so the lock waits are the wide boxes under pthread_mutex_lock. Because off-CPU time also includes legitimate waits (I/O, idle worker threads waiting for work), read only the stacks that end in lock functions on request-handling threads.
4. Quantify per lock. Stacks give you where, not hold versus wait. Two options: (a) for kernel locks, perf lock contention (the man page lists the subcommands record, report, script, info and contention, an option -b/--use-bpf that collects the statistics with a BPF program, and a lock-type filter whose values are kernel lock types such as spinlock, rwlock and mutex; check your perf version before relying on it for user-space pthread mutexes); (b) for your own user-space locks, wrap lock and unlock in a thin instrumented wrapper (or a library's built-in contention metrics), recording wait and hold per call site, as in the program below. A wrapper is also the direct way to get hold time, which the sampling tools above do not report (a probe attached to lock and unlock, such as a uprobe, can also measure it, but it fires on every lock call).
5. Attribute blocked time to a call stack cheaply. Keep the stack out of the hot path: in the wrapper, only record a stack (for example backtrace() or a stack id) when the wait exceeds a threshold such as 1 ms, and sample 1 in N of those. That bounds overhead to the slow, interesting cases. A heap profile does not answer this question (it shows allocation, not waiting); CPU profiling answers it only for the holder, which is why the on-CPU and off-CPU views are read together: the off-CPU graph shows the waiters' stacks, and the on-CPU graph shows what the holder was doing.
6. Check the spike correlation. Overlay lock wait p99 per second on request p99 per second. If they move together, the lock is the cause; if not, keep looking (GC, I/O, noisy neighbour).

Instrumented mutex: wait versus hold, and the effect of sharding

The program below has 8 threads each doing 50,000 lock/unlock pairs. The critical section is a loop over a volatile counter (volatile forces the compiler to do every read and write, so the loop is not optimised away; it takes about one microsecond). The two now_ns() calls read clock_gettime(CLOCK_MONOTONIC), a clock that only moves forward, around the lock call (wait) and around the critical section (hold). Mode 1 uses one global lock; mode 2 uses 16 locks chosen by key, each followed by 64 spare bytes so that no two locks share a 64-byte cache line. A cache line is the block of memory (64 bytes on common CPUs) that cores move between their caches as one unit; if two locks share a line, threads using different locks still make the line bounce between cores, which is called false sharing, so each lock is padded to keep neighbours on separate lines.

// Instrumented mutex: records wait time (blocked in lock) and hold time (inside the critical section)
// per named call site, then compares one global lock with a sharded lock set.
#define _GNU_SOURCE
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <time.h>

// CLOCK_MONOTONIC only moves forward (immune to wall-clock changes), so differences are safe to subtract.
static inline uint64_t now_ns(void) {
    struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts);
    return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec;
}

#define NSHARD 16
#define NTHREADS 8
#define OPS 50000

// pad: 64 spare bytes keep neighbouring locks on different 64-byte cache lines, so two threads working on
// different shards do not slow each other by bouncing one line between cores (false sharing).
typedef struct { pthread_mutex_t m; char pad[64]; } shard_t;
static shard_t shards[NSHARD];
// volatile: the compiler must really perform every read and write, so the loop is not optimised away.
static volatile long counters[NSHARD * 8];

typedef struct { uint64_t wait_ns, hold_ns, calls; } site_t;

static void *worker(void *arg) {
    site_t *site = arg;                 // per-thread stats, merged later (no shared counters)
    unsigned seed = (unsigned)(uintptr_t)&site;
    extern int g_sharded;
    for (int i = 0; i < OPS; i++) {
        int key = rand_r(&seed);
        shard_t *s = &shards[g_sharded ? key % NSHARD : 0];
        uint64_t t0 = now_ns();                 // t0..t1 = waiting for the lock
        pthread_mutex_lock(&s->m);
        uint64_t t1 = now_ns();
        for (int k = 0; k < 1000; k++) counters[(key % NSHARD) * 8] += k;   // the critical section
        uint64_t t2 = now_ns();                 // t1..t2 = holding the lock
        pthread_mutex_unlock(&s->m);
        site->wait_ns += t1 - t0; site->hold_ns += t2 - t1; site->calls++;
    }
    return NULL;
}
int g_sharded;

static void run(const char *name, int sharded) {
    g_sharded = sharded;
    pthread_t th[NTHREADS]; site_t st[NTHREADS] = {0};
    uint64_t t0 = now_ns();
    for (int i = 0; i < NTHREADS; i++) pthread_create(&th[i], NULL, worker, &st[i]);
    for (int i = 0; i < NTHREADS; i++) pthread_join(th[i], NULL);
    double wall = (now_ns() - t0) / 1e6;
    uint64_t w = 0, h = 0, c = 0;
    for (int i = 0; i < NTHREADS; i++) { w += st[i].wait_ns; h += st[i].hold_ns; c += st[i].calls; }
    printf("%-9s wall %7.1f ms | per call: wait %7.0f ns, hold %5.0f ns | wait/hold %5.1f\n",
           name, wall, (double)w / c, (double)h / c, (double)w / h);
}

int main(void) {
    for (int i = 0; i < NSHARD; i++) pthread_mutex_init(&shards[i].m, NULL);
    run("1 lock", 0);
    run("16 shards", 1);
    return 0;
}

Compiled with gcc -O2 -Wall -Wextra -pthread lockprof.c in a gcc:14 container (GCC 14, aarch64; figures are illustrative of the shape) and run with --cpuset-cpus=0-7, one sample run printed (host timings vary run to run):

1 lock    wall   517.7 ms | per call: wait    7955 ns, hold  1203 ns | wait/hold   6.6
16 shards wall   194.8 ms | per call: wait    2025 ns, hold  1248 ns | wait/hold   1.6

The shape: with one lock the 8 threads run one at a time, so wall time is about 8 x 50,000 x 1.2 microseconds (480 ms), the same as a single thread would take, and each call waits several times longer than it holds. With 16 shards the wall time dropped by a factor of about 2 to 3 in re-runs (not 8x and not 16x: with 8 threads choosing among 16 locks at random, two threads often pick the same lock, and the instrumented wait/hold ratio stayed above 1). Pinned to one CPU (--cpuset-cpus=0) the same binary took about the same time in both modes, close to what a single thread needs, because with one CPU only one thread runs at a time, so sharding has nothing to speed up there: the single CPU, not the lock, limits throughput. The measured wait is still several times the hold in both modes on one CPU (about 5 to 7 times), because a thread's wait also includes the time other threads spend on the CPU, so a high wait/hold ratio alone does not prove lock contention; the drop in wall time when sharding is what shows the lock was the limit. With 2 or 4 CPUs the gain from sharding is in between (roughly 1.4 to 2.6 times in re-runs, nearer the low end with 2 CPUs, and an occasional run showed no gain at all). The measured wait includes two clock reads per call, so read its ratio and trend, not its absolute value.

Mitigations, in the order to try them

The first two rows are the usual first moves: they change the least and carry the least risk. Rows 3 to 5 are for the one or two locks the wait ranking points at.

Recommendation: do 1 and 2 first (cheapest, no semantic change), then 3 for the one or two locks the wait ranking names. Verify by repeating the lock-wait p99 overlay under the same load: the lock's total wait should fall and request p99 should follow. If request p99 does not move, the lock was not the cause.

Principle: size each stage by what it consumes, then let the slowest stage set the pace

A pipeline moves at the speed of its bottleneck, the stage with the lowest capacity. The five stages consume different things, so they get different pools:

Backpressure means a full downstream queue slows the upstream stage instead of letting memory grow. Use bounded queues between stages, so a slow stage blocks the sender, and one admission limit (a semaphore, a counter of free slots that an image must take before it enters) on total images in flight, which is also the memory limit: in-flight images x decoded size.

Computed sizing (hypothetical inputs)

Which lever matters most: the bottleneck stage's capacity first (here the accelerator, set mostly by batch size), then the in-flight limit (it sets both memory and queueing latency), then pool sizes for the other stages, then priority and the maximum-wait timer, which only reshape latency once throughput is right.

The inputs below (40 ms fetch wait, 8 ms decode, accelerator 30 ms per batch plus 6 ms per image, 6 ms post-processing, 4 cores, a target of 120 images per second) are made-up numbers for the exercise, replace them with measured per-stage times. The script computes concurrency, core needs and the effect of batch size. Its last two columns are two different pictures of the batcher: the fill wait if it sat idle waiting for a batch to fill, and how many images arrive at the target rate while one batch is already running on the accelerator.

# Hypothetical stage costs (stated inputs, not measurements) for the pipeline.
FETCH_WAIT_MS = 40        # network wait, no CPU
DECODE_CPU_MS = 8
DETECT_SETUP_MS = 30      # per-batch fixed cost on the accelerator
DETECT_PER_IMG_MS = 6
POST_CPU_MS = 6           # annotate + render
CORES = 4
TARGET = 120              # images per second wanted

def detect_capacity(batch):
    return 1000 / (DETECT_SETUP_MS / batch + DETECT_PER_IMG_MS)

print(f"target {TARGET} img/s on {CORES} cores")
print("fetch   : in flight = rate x time (Little) =", round(TARGET * FETCH_WAIT_MS / 1000, 1), "requests -> I/O workers >= 5, no CPU used")
print("decode  : cores needed =", round(TARGET * DECODE_CPU_MS / 1000, 2))
print("post    : cores needed =", round(TARGET * POST_CPU_MS / 1000, 2))
print("CPU stages together    :", round(TARGET * (DECODE_CPU_MS + POST_CPU_MS) / 1000, 2), "of", CORES, "cores")
print("batch  accel img/s  fill wait if the batcher idles  arrivals while one batch runs")
for b in (1, 2, 4, 8, 12, 13, 16, 32):
    fill = (b - 1) / TARGET * 1000
    run_ms = DETECT_SETUP_MS + DETECT_PER_IMG_MS * b
    print(f"{b:>5}  {detect_capacity(b):>9.1f}  {fill:>14.1f} ms  {TARGET * run_ms / 1000:>16.1f}")
print("smallest batch that reaches the target:", next(b for b in range(1, 100) if detect_capacity(b) >= TARGET))
for cores in (1, 2, 4):
    print(f"CPU-stage ceiling on {cores} core(s): {cores * 1000 / (DECODE_CPU_MS + POST_CPU_MS):.1f} img/s")
# Memory: hypothetical 1920 x 1080 RGB image decoded to 3 bytes per pixel, 48 images in flight.
img_mb = 1920 * 1080 * 3 / 1e6
print(f"decoded image {img_mb:.1f} MB; 48 in flight = {48 * img_mb:.0f} MB")

Run in a python:3.12-slim container (CPython 3.12), it prints:

target 120 img/s on 4 cores
fetch   : in flight = rate x time (Little) = 4.8 requests -> I/O workers >= 5, no CPU used
decode  : cores needed = 0.96
post    : cores needed = 0.72
CPU stages together    : 1.68 of 4 cores
batch  accel img/s  fill wait if the batcher idles  arrivals while one batch runs
    1       27.8             0.0 ms               4.3
    2       47.6             8.3 ms               5.0
    4       74.1            25.0 ms               6.5
    8      102.6            58.3 ms               9.4
   12      117.6            91.7 ms              12.2
   13      120.4           100.0 ms              13.0
   16      127.0           125.0 ms              15.1
   32      144.1           258.3 ms              26.6
smallest batch that reaches the target: 13
CPU-stage ceiling on 1 core(s): 71.4 img/s
CPU-stage ceiling on 2 core(s): 142.9 img/s
CPU-stage ceiling on 4 core(s): 285.7 img/s
decoded image 6.2 MB; 48 in flight = 299 MB

Reading it: the CPU stages need 1.68 of 4 cores at the target, so they are not the limit. The accelerator is: with batch 1 it tops out at 27.8 img/s, and the smallest batch that reaches the 120 target is 13 (120.4 img/s); 16 gives 127 img/s with headroom. That is the batching trade-off in numbers: a bigger batch spreads the fixed 30 ms over more images (higher throughput) but takes longer to run and to fill.

How big a batch actually forms depends on the load, and the two columns explain it. Under a steady 120 img/s the accelerator is busy running the previous batch for 30 + 6 x 13 = 108 ms, and in that time 120 x 0.108 = 13 images arrive and queue up, so the next batch fills instantly from the backlog without anyone waiting for it. A batch of 13 is therefore self-consistent at the target rate: arrivals while it runs equal its size. If a batch happened to be smaller than that (say 3), capacity at that size would be about 1000 / (30 / 3 + 6) = 62 img/s, below the arrival rate, so the backlog grows and the next batch, which takes whatever has queued up to the batch limit, is bigger, until batches settle where capacity matches arrivals. The idle-batcher "fill wait" column (100 ms at 13) only applies when the queue is empty; then the batcher needs a maximum wait timer (20 ms in the program) that sends a partial batch so that a lone image is not held back. At light traffic that partial batch is small and capacity per image is poor, which is acceptable because the load is light. On 1 core the CPU stages cap the pipeline at 71 img/s, below the target, so with fewer cores the bottleneck moves from the accelerator to the CPU.

Simulation: batching, bulk versus interactive, priority lane

The Go program wires the stages with bounded queues and an admission limit of 48 images. Fetch and the accelerator are time.Sleep (they use no CPU); decode and post-processing spin on the CPU for their stated time. The costs are the simulated ones above, not a real model. A bulk producer keeps the pipeline full; 5 interactive requests per second arrive on top. In the third setup, interactive items go through a high-priority lane at every stage and flush the accelerator batch as soon as they arrive.

package main

import (
	"fmt"
	"runtime"
	"sort"
	"sync"
	"time"
)

// Stage costs are SIMULATED: fetch and the accelerator sleep (they use no CPU), decode and
// post-processing spin on the CPU for the stated time so that cores really are the limit.
const (
	fetchMs  = 40
	decodeMs = 8
	setupMs  = 30 // accelerator: fixed cost per batch
	perImgMs = 6  // accelerator: cost per image in the batch
	postMs   = 6
	maxWait  = 20 * time.Millisecond // longest the batcher waits to fill a batch
	inflight = 48                    // total images allowed in the pipeline: the memory limit
)

type item struct {
	interactive bool
	start       time.Time
}

func spin(ms int) {
	end := time.Now().Add(time.Duration(ms) * time.Millisecond)
	for time.Now().Before(end) {
	}
}

// pq is a two-lane queue. A chan is a Go channel: a bounded pipe between goroutines (Go's lightweight threads).
// Receivers always take the high lane first.
type pq struct{ hi, lo chan item }

func newPQ() *pq { return &pq{make(chan item, 16), make(chan item, 16)} }
func (q *pq) send(it item, prio bool) {
	if prio && it.interactive {
		q.hi <- it
	} else {
		q.lo <- it
	}
}
func (q *pq) recv() item {
	// Step 1: non-blocking peek at the high lane. A select with a default case never waits.
	select {
	case it := <-q.hi:
		return it
	default:
	}
	// Step 2: high lane was empty, so block until either lane has an item. Using only this select would
	// let Go pick at random when both lanes are ready, so a waiting high item could lose to a low one.
	select {
	case it := <-q.hi:
		return it
	case it := <-q.lo:
		return it
	}
}

func run(name string, batchMax int, prio bool, cores int, dur time.Duration) {
	fetchQ, decodeQ, detectQ, postQ := newPQ(), newPQ(), newPQ(), newPQ()
	done := make(chan item, 256)
	sem := make(chan struct{}, inflight) // admission: bounds memory

	var wg sync.WaitGroup
	pool := func(n int, in *pq, work func(item) item, out *pq) {
		for i := 0; i < n; i++ {
			wg.Add(1)
			go func() {
				defer wg.Done()
				for {
					it, ok := recvOrStop(in)
					if !ok {
						return
					}
					it = work(it)
					if out != nil {
						out.send(it, prio)
					} else {
						done <- it
					}
				}
			}()
		}
	}
	pool(16, fetchQ, func(it item) item { time.Sleep(fetchMs * time.Millisecond); return it }, decodeQ)
	pool(cores, decodeQ, func(it item) item { spin(decodeMs); return it }, detectQ)
	// Batcher: one accelerator, one batch at a time. An interactive item flushes the batch at once.
	wg.Add(1)
	go func() {
		defer wg.Done()
		for {
			first, ok := recvOrStop(detectQ)
			if !ok {
				return
			}
			batch := []item{first}
			deadline := time.After(maxWait) // channel that delivers one value after maxWait: the batch's wait budget
			flush := prio && first.interactive
		// Fill loop: keep adding items until the batch is full, the wait budget is spent, or an
		// interactive item asks for an early flush (send the batch to the accelerator now).
		fill:
			for len(batch) < batchMax && !flush { // if a backlog is queued, items arrive instantly and the timer never matters
				select {
				case it := <-detectQ.hi:
					batch = append(batch, it)
					flush = prio
				case it := <-detectQ.lo:
					batch = append(batch, it)
				case <-deadline:
					break fill
				}
			}
			time.Sleep(time.Duration(setupMs+perImgMs*len(batch)) * time.Millisecond)
			for _, it := range batch {
				postQ.send(it, prio)
			}
		}
	}()
	pool(cores, postQ, func(it item) item { spin(postMs); return it }, nil)

	stop := make(chan struct{})
	// Bulk producer: always has work, limited only by the in-flight cap (backpressure).
	go func() {
		for {
			select {
			case <-stop:
				return
			case sem <- struct{}{}:
				fetchQ.send(item{false, time.Now()}, prio)
			}
		}
	}()
	// Interactive producer: 5 requests per second.
	go func() {
		t := time.NewTicker(200 * time.Millisecond)
		for {
			select {
			case <-stop:
				return
			case <-t.C:
				select {
				case sem <- struct{}{}:
					fetchQ.send(item{true, time.Now()}, prio)
				case <-stop:
					return
				}
			}
		}
	}()

	var lat []float64
	bulk := 0
	t0 := time.Now()
	for time.Since(t0) < dur {
		it := <-done
		<-sem
		if it.interactive {
			lat = append(lat, float64(time.Since(it.start).Microseconds())/1000)
		} else {
			bulk++
		}
	}
	close(stop)
	sort.Float64s(lat)
	fmt.Printf("%-34s bulk %5.1f img/s | interactive n=%2d  p50 %6.0f ms  p99/max %6.0f ms\n",
		name, float64(bulk)/dur.Seconds(), len(lat), lat[len(lat)/2], lat[len(lat)-1])
	// goroutines are left to the process exit; this program is a one-shot experiment.
}

func recvOrStop(q *pq) (item, bool) { return q.recv(), true }

func main() {
	fmt.Println("GOMAXPROCS =", runtime.GOMAXPROCS(0))
	cores := runtime.GOMAXPROCS(0)
	dur := 8 * time.Second
	run("FIFO, batch 1", 1, false, cores, dur)
	run("FIFO, batch 16", 16, false, cores, dur)
	run("priority lanes, batch 16", 16, true, cores, dur)
}

Read it as: chan is Go's bounded queue between goroutines (lightweight threads); select waits on several channels and takes whichever is ready; recv first peeks at the high lane with a non-blocking select (the default case means "do not wait") and only then blocks on both lanes, which is why select appears twice: a single select over both lanes would choose at random when both are ready, and the high lane could lose. In the batcher, time.After(maxWait) is the wait budget, the fill loop adds items until the batch is full, the budget is spent, or an interactive item asks to flush (send the batch to the accelerator at once), and time.Sleep stands for the accelerator's run time. The semaphore channel sem is the admission window: a slot is taken before an image enters and returned when it leaves, so at most 48 images exist at once.

Compiled and run in a golang:1.23 container with go run pipeline.go and --cpuset-cpus=0-3 (GOMAXPROCS is the number of CPUs Go runs goroutines on at once, here 4), one sample run (host-dependent; repeat runs on 4 and 8 CPUs gave nearly the same figures. On 2 CPUs the batch-16 rows were the same or lower, about 85 to 118 img/s with FIFO and about 80 to 95 with priority lanes, because decode, post-processing and the batcher then compete for the cores. On one CPU, --cpuset-cpus=0, the CPU stages cap the pipeline as the script predicts: batch-16 FIFO ran at about 55 to 73 img/s, close to the 71 img/s ceiling the script computed, with an interactive median of about 0.55 to 0.75 s, priority lanes no longer cost bulk throughput there, and batch 1 was unchanged because the accelerator is its limit):

GOMAXPROCS = 4
FIFO, batch 1                      bulk  20.4 img/s | interactive n=29  p50   2002 ms  p99/max   2044 ms
FIFO, batch 16                     bulk 113.4 img/s | interactive n=37  p50    409 ms  p99/max    425 ms
priority lanes, batch 16           bulk  96.1 img/s | interactive n=38  p50    215 ms  p99/max    346 ms

What the shape says:

1. Batch 1 is accelerator-bound: bulk about 20 to 23 img/s plus about 4 interactive per second (n=29 to 31 in 8 s against about 39 requests issued at 5 per second: the missing ones arrived in the last two seconds of the window and were still in the pipeline when the measurement stopped, since each takes about 2 s) is about 26 img/s once all 5 interactive requests per second are counted, a little under the 27.8 img/s ceiling the script computed because real sleeps run slightly long. Every image, interactive or not, waits behind the full admission window: with 48 images allowed in flight at about 25 completions per second, a new request waits about 48 / 25 = 1.9 seconds, which matches the 2002 ms (re-runs gave a median of about 1.75 to 2.1 s). (p99/max is the slowest 1 percent; with only about 30 samples it is simply the slowest request.) The memory cap is also the latency cap: bounding in-flight work bounds the queue wait, and a smaller cap gives lower latency at the same throughput, but only while the cap stays above what the bottleneck needs in flight. By Little's law that need is the throughput times the unloaded time through the pipeline. For batch 1 the pipeline sustains under 28 img/s through a path of about 90 ms, so only about 3 images need to be in flight, and shrinking the cap from 48 to 12 cut the interactive median to roughly 0.5 seconds with about the same bulk throughput. For batch 16 at the 120 target the unloaded path is about 40 + 8 + (30 + 6 x 16) + 6 = 180 ms, so roughly 120 x 0.18 = 22 images must be in flight at the very least. That is a floor, not a target: the batcher also needs the next batch already waiting while the current one runs, so a cap of 12 gave about half the bulk throughput of a cap of 48 (about 60 against about 115 img/s) and even a cap of 24, just above the floor, gave only about three quarters of it (about 88 img/s). Choose a cap well above the floor (48 came close to the 127 img/s the accelerator can do at batch 16), then trim it only as far as the latency and memory budgets require.
2. Batch 16 is about 5 to 6 times the bulk throughput (113 against 20 img/s in the sample) and interactive latency falls to about 400 ms, because the pipeline now drains the window much faster (Little's law again: 48 in flight at about 120 completions per second is about 0.4 s).
3. Priority lanes roughly halve the interactive median (to about 200 to 290 ms in re-runs on 4 and 8 CPUs, against about 380 to 415 ms) at a cost to bulk throughput (about 94 to 102 against 113 to 120 img/s on 4 and 8 CPUs; on one CPU there is no such cost, because the CPU stages are the limit and the batch size is not), because each interactive arrival flushes a smaller batch and the fixed 30 ms is spread over fewer images. Interactive latency does not fall to its unloaded value (about 40 + 8 + 36 + 6 = 90 ms of service), mainly because the accelerator cannot preempt (interrupt) a batch already running (and decode and post-processing may be busy spinning on a bulk item already in service): an interactive item waits for the batch in progress, up to 30 + 6 x 16 = 126 ms, plus its own 36 ms.

Decisions, stated

• Pool sizing per stage: fetch by Little's law (about 5 in flight here, round up with headroom), CPU stages about one worker per core, one batcher per accelerator.
• Batch size: the largest batch whose fill wait plus run time still fits the interactive latency budget; in the example, 8 gives 102.6 img/s (short of the 120 target) and 13 is the smallest that reaches it, so the target forces a batch of 13 or more (16 with headroom) unless a second accelerator is added. The maximum-wait timer protects latency when traffic is low.
• Backpressure: bounded queues between stages plus the in-flight limit; the producer blocks, or the API returns 429 when interactive admission is full, rather than growing a queue.
• Prioritising urgent requests: a separate high-priority lane at each queue and early batch flush, and reserve a slice of in-flight capacity for interactive work so bulk cannot fill the whole admission window. Because a running batch cannot be preempted, cap the bulk batch size, or give interactive work its own small accelerator slot, if the p99 matters more than bulk throughput.
• Memory limit: in-flight images x decoded size (299 MB for 48 images at 6.2 MB, hypothetical); decode only when the next stage can accept the image, and pass compressed bytes until then. Do not set the limit at or near the minimum of roughly the target rate x the unloaded time through the pipeline (about 22 images here): the batcher needs a whole batch queued behind the one it is running, and a limit of 24 still left bulk throughput about a quarter short.

What would change this: if the detector is CPU-based and not an accelerator, it joins the CPU budget (decode + detect + render share the cores) and the sizing becomes one cores-limited pool with priorities; if a stage is separately scalable (its own service), size and autoscale it independently.

Principle: one concurrency limit per resource, not one for "the job runner"

The two task kinds are limited by different resources, so they get separate pools (a bulkhead: a partition so that trouble in one cannot drain the other). Mixing them in one pool lets slow waiting tasks occupy the threads that CPU tasks need, or lets CPU tasks starve the waiting ones.

Initial settings (hypothetical machine: 8 cores)

• CPU-saturating tasks: concurrency about equal to the cores they can use. A task that saturates a core gains nothing from running beside another on the same core; extra concurrency only adds context switches and lengthens every task. Reserve some cores for everything else: if the waiting tasks need about 2 ms of CPU each and run 300 per second (hypothetical), that is 300 x 0.002 = 0.6 core, so reserve one core and give the CPU pool 7 workers. In Python's standard build these must be processes (the global interpreter lock, or GIL, stops threads from running pure-Python CPU work in parallel; the optional free-threaded build removes that limit); in Java, Go or similar, threads or goroutines (Go's lightweight threads) work.
• Waiting tasks: concurrency set by Little's law and capped by the downstream. Little's law says in-flight requests (calls that have been sent and not yet answered) = arrival rate x time per request. With a target of 300 calls/s (hypothetical) and 100 ms per call, in-flight is 300 x 0.1 = 30. That need not be a fixed 30: the remote service has its own capacity, and the limit that matters is the highest number it can serve without its latency rising. Start under that ceiling (for a measured ceiling of 40, say 30 to 32) and leave headroom for other callers of the same service.

These are first estimates. The wait-to-compute ratio shifts under load, so treat them as the starting point of a test.

How to know the settings are right

Run a stepped load test (raise the offered rate or the pool limit one step at a time, hold each step long enough to settle) and read three curves:

1. Throughput versus concurrency. It rises, then flattens at the knee: the concurrency level where adding more stops increasing throughput. In the simulation below the knee is at 40 concurrent calls: 30 in flight gives 300 calls/s, 40 gives 400, and 50 still gives 400 because the extra 10 only queue and push latency from 100 ms to 125 ms. The right setting sits at or just below the knee; past it, throughput is flat or falling.
2. Latency versus concurrency. Flat until the knee, then rising. For waiting tasks watch the downstream call latency itself against its baseline; for CPU tasks watch task duration against its single-task duration.
3. Queue behaviour. Time spent waiting in the pool's queue, and queue depth over time. If the queue is empty most of the time, the pool is large enough; if it grows during sustained load, capacity is the limit, not burst.

In production, alert on the same signals: CPU utilisation per pool (target below saturation so spikes have room), p95 or p99 task latency (the latency 95 or 99 percent of tasks beat) against the target, queue depth and time-in-queue, rejected or timed-out submissions, and the error rate (timeouts and 429 or 503 responses) from each downstream. Latency inflation with flat throughput is the sign that you passed the knee.

Adapting as load shifts

A static number is right only while the downstream and the mix stay the same. A downstream that slows down, gets a new deploy, or is shared with other callers moves its capacity. Use an adaptive limit with the same logic as TCP congestion control (the rule by which network connections speed up gradually and cut their sending rate sharply when packets are lost), called AIMD (additive increase, multiplicative decrease): raise the limit by one while latency is near its baseline, and cut it by a fraction (here 30%) when latency or errors show overload. It keeps probing for spare capacity and backs off quickly. A hand trace with the program's numbers (capacity 40, increase threshold 1.2 x 100 ms = 120 ms): the limit goes 10, 11, 12 and so on, one per tick, while latency is 100 ms. At a limit of 48 latency is 100 x 48 / 40 = 120 ms, still allowed, so it rises to 49; at 49 latency is 122.5 ms, over the threshold, so the next limit is 49 x 0.7 = 34.3, and the climb restarts from there.

The model below is a deterministic simulation, not a measurement of a real service: a remote service handles up to capacity concurrent calls at 100 ms, and beyond that latency grows in proportion. Capacity is 40 until tick 200 and then drops to 20.

# Deterministic model of a remote service. Up to `capacity` concurrent calls run at the
# base latency; beyond that, latency grows in proportion (queueing at the remote side).
BASE_MS = 100.0

def latency_ms(in_flight, capacity):
    return BASE_MS if in_flight <= capacity else BASE_MS * in_flight / capacity

def run(ticks=400, drop_at=200, cap_before=40, cap_after=20):
    limit, floor, ceiling = 4.0, 2.0, 200.0
    trace = []
    for t in range(ticks):
        capacity = cap_before if t < drop_at else cap_after
        lat = latency_ms(limit, capacity)
        throughput = min(limit, capacity) * 1000.0 / BASE_MS   # completions per second
        # AIMD: additive increase while latency is near baseline, multiplicative decrease when it inflates
        if lat <= 1.2 * BASE_MS:
            limit = min(ceiling, limit + 1.0)
        else:
            limit = max(floor, limit * 0.7)
        trace.append((t, capacity, round(limit, 1), round(lat), round(throughput)))
    return trace

if __name__ == "__main__":
    tr = run()
    print("tick capacity limit latency_ms calls_per_s")
    for row in tr[::25] + [tr[-1]]:
        print("%4d %8d %5.1f %10d %12d" % row)
    for name, seg in (("before drop", tr[100:200]), ("after drop", tr[300:400])):
        lim = [r[2] for r in seg]
        print(name, "limit range: %.1f to %.1f" % (min(lim), max(lim)))

In the output, tick is one time step, capacity is how many concurrent calls the remote service handles at full speed, limit is the adaptive cap on in-flight calls, latency_ms is the call latency at that limit, and calls_per_s is completions per second, min(limit, capacity) x 10 because each call takes 100 ms.

Output of that program (deterministic, so it reproduces exactly):

tick capacity limit latency_ms calls_per_s
   0       40   5.0        100           40
  25       40  30.0        100          290
  50       40  39.3        100          383
  75       40  48.8        120          400
 100       40  42.7        104          400
 125       40  36.7        100          357
 150       40  46.1        113          400
 175       40  40.1        100          391
 200       20  34.0        243          200
 225       20  22.8        109          200
 250       20  22.9        109          200
 275       20  22.9        109          200
 300       20  22.9        109          200
 325       20  22.9        109          200
 350       20  22.9        109          200
 375       20  22.9        109          200
 399       20  21.9        104          200
before drop limit range: 33.6 to 48.7
after drop limit range: 16.9 to 24.9

The limit column is the value after that tick's adjustment. The shape to read: the limit climbs from 5, hovers around the 40-call capacity in a sawtooth (it overshoots, sees latency rise, then backs off), and when capacity halves at tick 200 latency spikes to 243 ms for a moment and the limit falls to about 20 to 25 within a few ticks. The sampled rows cannot show the sawtooth after tick 200: the cycle there is about 8 ticks long, so printing every 25th tick lands near the same phase each time and the limit looks constant at 22.9. The limit range lines are the honest summary (16.9 to 24.9 after the drop, 33.6 to 48.7 before it), and printing every tick of the same run makes the cycle visible: the limit from tick 203 reads 17.4, 18.4, 19.4, 20.4, 21.4, 22.4, 23.4, 24.4, then 17.1 at tick 211, each rise one step and each cut a drop of 30%. The throughput column tops out at the downstream's capacity (400 calls/s then 200); the adaptive limit finds that ceiling without being told it. The sawtooth briefly pushes latency above baseline (up to about 120 ms in this table), which is the price of probing; a tighter latency threshold (a smaller multiple of the baseline than the 1.2 used here) reduces it, a smaller increase step trims it only slightly, and a gentler cut (a multiplier closer to 1 than 0.7) shrinks the swings in the limit at the cost of recovering more slowly from a real overload. In production add a floor (never below 2), a ceiling (never above what memory and connections allow) and smoothing so one slow call does not trigger a cut.

Protecting the latency targets

• Admission control and priorities. Admission control means refusing or deferring new work at the door when the system is full. If CPU tasks and waiting tasks share a latency target, queue them separately, bound each queue, and drop or defer low-priority work first when queues grow.
• Downstream ceiling. The sum of every caller's limit must stay below the downstream's capacity; if the downstream publishes rate limits or returns 429, treat that as an overload signal to back off, not as an error to retry immediately.
• Retries with exponential backoff (each wait doubles) and jitter (a random offset so clients do not retry in lockstep), and a retry budget (a cap on retries as a fraction of normal traffic), so retries do not add load to a service that is already slow.

Recommendation

Two pools: CPU concurrency at cores minus a reserve, waiting-task concurrency from Little's law capped below the downstream's measured ceiling and adapted with AIMD. Prove it with a stepped load test, and keep alerts on queue time and latency inflation. If the CPU pool must meet a tight latency target while the machine is shared, lower its limit rather than raise it: a CPU-bound task finishes faster when it has a core to itself.