Concurrency & Asynchronous Performance Questions

Decide on the work each request does, then on the failure you can live with

A thread per request model gives each in-flight request its own operating-system (OS) thread, an independent flow of execution the kernel schedules onto CPU cores. Code reads top to bottom and a blocking call just parks that thread. An event loop model runs many requests on one thread (per process), which switches between requests only when a handler waits (await or a callback); a handler that blocks stops them all. A third option is a lightweight thread scheduled by the language runtime: goroutines in Go, virtual threads in Java 21. They read like blocking code but cost far less than OS threads.

Recommendation for a new service whose handlers mostly wait on databases and downstream calls: use the runtime-scheduled lightweight option if the stack has it (Java 21 virtual threads, Go), otherwise an event loop with a small separate pool for the work that blocks or burns CPU. Choose plain thread-per-request only when concurrency is modest (hundreds, not tens of thousands) and the team's code or libraries are synchronous. What would flip it: heavy CPU per request (then cores are the limit and the model barely matters; add worker processes), a mandatory blocking library you cannot replace (thread per request or an offload pool), or a team that cannot hold the discipline "nothing blocks the loop".

Latency and throughput

The sections below run one comparison on each runtime that matters for the model: a Java program for in-flight capacity and the pinning caveat, and a Python program for memory per connection. The table after them covers what a blocked handler does in each model.

When requests mostly wait, throughput is limited by how many can be waiting at once. By Little's law (requests in flight = arrival rate x time in system), 10,000 requests/s each spending 100 ms blocked needs 1,000 requests in flight. A pool of 200 OS threads caps this at 200 / 0.1 s = 2,000 requests/s no matter how many cores exist. An event loop or lightweight threads have no such cap from the thread count; their cap is CPU and the downstream's capacity.

The program below runs 10,000 handlers that each block 100 ms, once on 200 platform threads and once with one virtual thread per task, then times 64 sleeps inside synchronized (a Java monitor lock: the keyword makes a block of code run by one thread at a time per lock object).

How to read the program: handler() is a stand-in request that sleeps 100 ms (blocked, no CPU). run submits the requests to an executor (a pool that runs submitted tasks) and a CountDownLatch, a counter that starts at the request count and releases await() when every task has called countDown(), so the timer stops when the last request finishes. newFixedThreadPool(200) is 200 OS threads; newVirtualThreadPerTaskExecutor() starts one virtual thread per task; try (...) closes the executor afterwards. In the last block each task sleeps while holding its own private lock object, so any slowdown comes from the runtime, not from tasks waiting for each other.

import java.util.concurrent.*;
import java.util.concurrent.atomic.*;

public class ModelCompare {
    // A "handler" that spends 100 ms blocked on a downstream call, then a little CPU.
    static void handler() {
        try { Thread.sleep(100); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    static double run(ExecutorService ex, int requests) throws Exception {
        long t = System.nanoTime();
        CountDownLatch done = new CountDownLatch(requests);
        for (int i = 0; i < requests; i++) ex.submit(() -> { handler(); done.countDown(); });
        done.await();
        return (System.nanoTime() - t) / 1e9;
    }

    public static void main(String[] a) throws Exception {
        System.out.println("availableProcessors = " + Runtime.getRuntime().availableProcessors());
        int requests = 10_000;
        try (ExecutorService ex = Executors.newFixedThreadPool(200)) {
            double s = run(ex, requests);
            System.out.printf("200 platform threads : %d blocking requests in %.2f s (%.0f req/s)%n", requests, s, requests / s);
        }
        try (ExecutorService ex = Executors.newVirtualThreadPerTaskExecutor()) {
            double s = run(ex, requests);
            System.out.printf("virtual threads      : %d blocking requests in %.2f s (%.0f req/s)%n", requests, s, requests / s);
        }
        // Little's law: 10_000 requests of 100 ms on 200 threads can finish no faster than 10_000 * 0.1 / 200 s
        System.out.printf("lower bound for 200 threads: %.2f s%n", requests * 0.1 / 200);
        // Pinning: blocking inside synchronized.
        try (ExecutorService ex = Executors.newVirtualThreadPerTaskExecutor()) {
            long t = System.nanoTime();
            int n = 64;
            CountDownLatch done = new CountDownLatch(n);
            for (int i = 0; i < n; i++) ex.submit(() -> {
                // each task uses its own monitor so tasks do not serialize on one lock
                Object own = new Object();
                synchronized (own) { handler(); }
                done.countDown();
            });
            done.await();
            System.out.printf("64 x 100 ms sleeps inside synchronized: %.2f s%n", (System.nanoTime() - t) / 1e9);
        }
    }
}

Run in an eclipse-temurin:21-jdk container (OpenJDK 21.0.12) with --cpus=1, then --cpus=4, it printed (availableProcessors reflected the CPU limit, not the host; timings are from a shared host and vary):

--- cpus=1
availableProcessors = 1
200 platform threads : 10000 blocking requests in 5.28 s (1894 req/s)
virtual threads      : 10000 blocking requests in 0.13 s (76548 req/s)
lower bound for 200 threads: 5.00 s
64 x 100 ms sleeps inside synchronized: 6.47 s
--- cpus=4
availableProcessors = 4
200 platform threads : 10000 blocking requests in 5.16 s (1939 req/s)
virtual threads      : 10000 blocking requests in 0.14 s (69427 req/s)
lower bound for 200 threads: 5.00 s
64 x 100 ms sleeps inside synchronized: 1.67 s

The shape: the 200-thread pool sits at its arithmetic ceiling (about 5 s, whatever the CPU count) while virtual threads finish in about 0.14 s because waiting costs almost nothing. The last line shows the caveat. Virtual threads are run by a small set of ordinary OS threads called carriers; the number of carriers defaults to the number of processors. Normally a virtual thread that blocks is unmounted: taken off its carrier so the carrier can run another virtual thread. On JDK 21 a virtual thread that blocks inside synchronized is pinned: it cannot unmount, so it holds its carrier for the whole block. With as many carriers as processors, so 64 pinned sleeps of 100 ms take about 64 x 0.1 / carriers seconds (6.4 s at one carrier, 1.6 s at four, matching the run). A JEP is a JDK Enhancement Proposal, the document that specifies a Java feature. JEP 444 documents the two pinning cases in JDK 21 (inside synchronized, and in native or foreign calls); JEP 491 removed the synchronized case in Java 24, leaving pinning while a native frame (a call into non-Java code) is on the stack, such as class loading or class initialisation that blocks.

Memory per connection

Each OS thread reserves a stack and kernel bookkeeping; a task on an event loop is a small heap object. A second program holds 2,000 idle waiting "connections" as asyncio tasks and as OS threads and reads resident memory (RSS, the memory actually held in RAM by the process) from /proc/self/status:

import asyncio, threading, resource

def rss_mb():
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS"):
                return int(line.split()[1]) / 1024

def threads(n):
    stop = threading.Event()
    ts = [threading.Thread(target=stop.wait) for _ in range(n)]
    base = rss_mb()
    for t in ts: t.start()
    used = rss_mb() - base
    stop.set()
    for t in ts: t.join()
    return used

async def tasks(n):
    stop = asyncio.Event()
    base = rss_mb()
    ts = [asyncio.create_task(stop.wait()) for _ in range(n)]
    await asyncio.sleep(0)
    used = rss_mb() - base
    stop.set()
    await asyncio.gather(*ts)
    return used

n = 2000
asyncio.run(asyncio.sleep(0))   # warm up the interpreter
a = asyncio.run(tasks(n))
t = threads(n)
print("%d idle connections: asyncio tasks +%.1f MB RSS (%.1f KB each), OS threads +%.1f MB RSS (%.1f KB each)"
      % (n, a, a*1024/n, t, t*1024/n))

On Linux in python:3.12-slim, two consecutive runs printed:

2000 idle connections: asyncio tasks +0.8 MB RSS (0.4 KB each), OS threads +57.6 MB RSS (29.5 KB each)
2000 idle connections: asyncio tasks +1.8 MB RSS (0.9 KB each), OS threads +59.0 MB RSS (30.2 KB each)

The difference is about 30 to 70 times per idle connection here. RSS counts only touched pages; each thread also reserves virtual address space (the range of addresses the OS sets aside for the process, whether or not it is backed by RAM yet) for its stack, and the OS limits threads per process, so thread counts can hit those limits before RSS looks alarming. Virtual threads keep their stacks as garbage-collected heap objects, and JEP 444 says memory use and garbage-collector activity of thread-per-request and asynchronous code should be roughly similar.

What happens when a handler blocks

Go's runtime documentation says GOMAXPROCS (the setting for how many goroutines may run on CPUs at once) limits the number of CPUs executing simultaneously and that there is no limit on threads blocked in system calls. In Node.js, file system, DNS lookup (dns.lookup), zlib and some crypto calls run on libuv's worker pool, whose default size is 4 threads (libuv docs; UV_THREADPOOL_SIZE changes it), so a handful of slow file or crypto calls can queue all the rest.

How failures show up in production

• Thread per request: thread count and queue depth climb, latency rises, then rejections or out-of-memory (OOM) errors, when the process is killed or refuses to allocate. Visible in thread dumps (jstack is the JDK tool that prints every thread's stack and state).
• Event loop: all endpoints get slow together, CPU is high on one core and idle on others, and timers fire late. A single bad regular expression or a large JSON parse can do it.
• Lightweight threads: unbounded concurrency moves the limit to the downstream: connection pools and databases saturate. Cap concurrency with a semaphore (JEP 444 says not to pool virtual threads), because a limit on thread count is gone.

Whatever you choose, set a timeout on every outbound call, bound every queue, and expose in-flight count and loop lag so the first failure is a graph, not a page.

The problem and the pattern

When a cached value is missing or expired and 100 requests ask for it at the same moment, a naive cache lets all 100 go to the backend: a cache stampede (also called thundering herd or dogpile). The backend sees 100 identical calls instead of one, which can overload it exactly when it is slow. The fix is single-flight (request coalescing): the first caller starts the fetch, every later caller for the same key attaches to that same in-progress fetch and waits for its result.

In asyncio the natural handle for an in-progress fetch is a Task. The design:

1. A dictionary inflight maps key to the one running task. The first caller creates it; later callers find it and await it.
2. Waiters await asyncio.shield(task). Cancelling a task that is awaiting a task cancels the inner task too; shield stops one impatient waiter (a client that disconnected, a request timeout) from cancelling the fetch that nine others are waiting on.
3. Cancelled waiters are counted. If the last waiter leaves, the shared fetch is cancelled so the backend stops working for nobody, and the key is removed from inflight at once so a newcomer cannot attach to a task that is about to die.
4. Failures propagate to every waiter (awaiting a failed task raises its exception in each). They are not cached: the next request starts a fresh fetch. A fetch timeout (asyncio.timeout) prevents one hung call from holding all waiters.
5. Expiry is a time-to-live (TTL) stored with each value. After it passes, the next request misses, and single-flight makes the refresh a single call even if many requests arrive at that instant.

The class and a test run

import asyncio, time

class SingleFlightCache:
    def __init__(self, fetch, ttl, fetch_timeout=5.0):
        self.fetch, self.ttl, self.fetch_timeout = fetch, ttl, fetch_timeout
        self.values = {}          # key -> (value, expires_at)
        self.inflight = {}        # key -> [task, number_of_waiters]

    async def get(self, key):
        hit = self.values.get(key)
        if hit and hit[1] > time.monotonic():
            return hit[0]
        entry = self.inflight.get(key)
        if entry is None:                                    # first caller starts the one shared fetch
            task = asyncio.create_task(self._load(key))
            entry = self.inflight[key] = [task, 0]
            task.add_done_callback(lambda t, k=key, e=entry: self._finish(k, e, t))
        task = entry[0]
        entry[1] += 1
        try:
            return await asyncio.shield(task)                # a waiter's cancellation cannot cancel the shared task
        except asyncio.CancelledError:
            entry[1] -= 1
            if entry[1] == 0 and not task.done():            # last waiter left: stop paying for a result nobody wants
                if self.inflight.get(key) is entry:
                    del self.inflight[key]                   # newcomers must not join a doomed task
                task.cancel()
            raise

    async def _load(self, key):
        async with asyncio.timeout(self.fetch_timeout):
            return await self.fetch(key)

    def _finish(self, key, entry, task):
        if self.inflight.get(key) is entry:
            del self.inflight[key]
        if task.cancelled():
            return
        if task.exception() is None:                         # only successes are cached; failures are retried by the next caller
            self.values[key] = (task.result(), time.monotonic() + self.ttl)

class Backend:
    def __init__(self, latency=0.1, fail_first=0):
        self.calls, self.cancelled, self.latency, self.fail_first = 0, 0, latency, fail_first
    async def __call__(self, key):
        self.calls += 1
        try:
            await asyncio.sleep(self.latency)
        except asyncio.CancelledError:
            self.cancelled += 1; raise
        if self.calls <= self.fail_first:
            raise ConnectionError("backend down")
        return "value-of-" + key

async def main():
    # 1. 100 concurrent requests for one key; and the same load with no coalescing
    b = Backend(); c = SingleFlightCache(b, ttl=60)
    res = await asyncio.gather(*[c.get("user:7") for _ in range(100)])
    print("coalesced: 100 requests -> %d backend call(s), %d distinct result(s)" % (b.calls, len(set(res))))
    b2 = Backend(); await asyncio.gather(*[b2("user:7") for _ in range(100)])
    print("no coalescing: 100 requests -> %d backend calls" % b2.calls)

    # 2. different keys are independent
    b = Backend(); c = SingleFlightCache(b, ttl=60)
    await asyncio.gather(*[c.get("k%d" % (i % 5)) for i in range(100)])
    print("5 distinct keys, 100 requests -> %d backend calls" % b.calls)

    # 3. a failing fetch is delivered to every waiter and is not cached
    b = Backend(fail_first=1); c = SingleFlightCache(b, ttl=60)
    res = await asyncio.gather(*[c.get("k") for _ in range(20)], return_exceptions=True)
    print("failure: %d of 20 waiters got ConnectionError, backend calls %d" % (sum(isinstance(r, ConnectionError) for r in res), b.calls))
    print("next request:", await c.get("k"), "| backend calls", b.calls)

    # 4. some waiters cancelled: the others still get the result from the same single fetch
    b = Backend(); c = SingleFlightCache(b, ttl=60)
    tasks = [asyncio.create_task(c.get("k")) for _ in range(10)]
    await asyncio.sleep(0.02)
    for t in tasks[:3]: t.cancel()
    res = await asyncio.gather(*tasks, return_exceptions=True)
    print("3 of 10 waiters cancelled: %d got the value, backend calls %d, fetch cancelled %d time(s)"
          % (sum(r == "value-of-k" for r in res), b.calls, b.cancelled))

    # 5. every waiter cancelled: the shared fetch is cancelled too, and the next caller starts a fresh one
    b = Backend(); c = SingleFlightCache(b, ttl=60)
    tasks = [asyncio.create_task(c.get("k")) for _ in range(5)]
    await asyncio.sleep(0.02)
    for t in tasks: t.cancel()
    await asyncio.gather(*tasks, return_exceptions=True)
    await asyncio.sleep(0)
    print("all waiters cancelled: fetch cancelled %d time(s), in flight %d; next get ->" % (b.cancelled, len(c.inflight)), await c.get("k"), "| backend calls", b.calls)

    # 5b. a newcomer arrives in the same instant the last waiter leaves
    b = Backend(); c = SingleFlightCache(b, ttl=60)
    tasks = [asyncio.create_task(c.get("k")) for _ in range(3)]
    await asyncio.sleep(0.02)
    for t in tasks: t.cancel()
    newcomer = asyncio.create_task(c.get("k"))
    await asyncio.gather(*tasks, return_exceptions=True)
    print("newcomer at the last cancellation:", await newcomer, "| backend calls", b.calls)

    # 6. expiry
    b = Backend(latency=0.01); c = SingleFlightCache(b, ttl=0.2)
    await c.get("k"); await c.get("k")
    print("within ttl: backend calls", b.calls)
    await asyncio.sleep(0.25)
    await asyncio.gather(*[c.get("k") for _ in range(50)])
    print("after ttl, 50 concurrent requests: backend calls", b.calls)

asyncio.run(main())

Run in a python:3.12-slim container (Python 3.12) limited to 1, 2, 4 and 8 CPUs with --cpuset-cpus, it printed the same lines each time (the outcomes are deterministic by design; the backend delay is 0.1 s and the waits in the tests are much shorter or longer than that):

coalesced: 100 requests -> 1 backend call(s), 1 distinct result(s)
no coalescing: 100 requests -> 100 backend calls
5 distinct keys, 100 requests -> 5 backend calls
failure: 20 of 20 waiters got ConnectionError, backend calls 1
next request: value-of-k | backend calls 2
3 of 10 waiters cancelled: 7 got the value, backend calls 1, fetch cancelled 0 time(s)
all waiters cancelled: fetch cancelled 1 time(s), in flight 0; next get -> value-of-k | backend calls 2
newcomer at the last cancellation: value-of-k | backend calls 2
within ttl: backend calls 1
after ttl, 50 concurrent requests: backend calls 2

Handling the three cases the question names

• Fetch failures: every waiter receives the same exception, which is the honest outcome (one backend call failed, 20 callers asked). Not caching the error means the very next caller retries, which is right for a blip but can hammer a backend that is genuinely down. Add a short negative cache (remember the failure for a second or two) or exponential backoff per key, and decide whether to serve a stale value instead (stale-if-error).
• Cancelled waiters: shown above. Three of ten cancelled waiters change nothing for the other seven, and when all five waiters left, the fetch itself was cancelled and the next caller started a clean one. The newcomer line covers the sharpest edge: a caller that arrives in the same instant the last waiter leaves must not attach to the fetch that is being cancelled. Deleting the inflight entry before task.cancel() is what prevents that; if the del is removed, that line fails with CancelledError instead of printing a value, which is the check that the harness can fail. Without shield, the first cancellation would have cancelled the shared fetch and failed the other nine with CancelledError, a classic bug in hand-written versions. Without the waiter count, abandoned fetches would run to completion for nobody.
• Expiry: the TTL check happens on read, and the single-flight refresh keeps an expiry from becoming a stampede. To avoid every key expiring together after a bulk load, add random jitter (for example 10%) to each TTL. To avoid callers waiting on the refresh at all, serve the old value while one background fetch refreshes it (stale-while-revalidate), at the cost of briefly serving older data.

Limits of this version

• It coalesces within one process. With 20 server instances a cold key can still reach the backend up to 20 times at once. If that is too many, add a per-key distributed lock or short-lived "fetching" marker in a shared cache, accepting its own failure modes (a crashed holder needs a lease timeout).
• The shared task runs on the event loop, so the fetch function must be non-blocking; a blocking driver call has to go through an executor first.
• Memory: values grows without bound as written. A production cache needs a size limit and eviction.

What the symptom means

In asyncio, all coroutines (functions declared with async def, which can pause and resume) share one thread, the event loop, and a coroutine only gives control back at an await (the point where it pauses so others can run). If one coroutine calls something that blocks the thread (a synchronous HTTP or database client, time.sleep, a blocking file or DNS call), nothing else on that loop runs until it returns. Low CPU together with a freeze points at blocking I/O or a lock, not a computation: the thread is parked in the operating system waiting, not working. (A CPU-bound loop would show one core at 100%; that is the other case and the checks below tell the two apart.)

Ordered checks

In a live outage start with checks 1 and 3: the loop-lag probe confirms the diagnosis in seconds and the stack dump names the call; debug mode and the audit are for finding the cause in code and preventing a repeat.

1. Measure loop lag. Run a probe task that sleeps 10 ms in a loop and records how late it wakes. On a healthy loop the lag is a few milliseconds; a blocking call shows up as a spike about as long as the call. Export the max and p99 (the value that 99 percent of samples are at or under) as a metric.
2. Turn on asyncio debug mode (in staging, or briefly in production; it adds overhead). Per the Python docs, enable it with PYTHONASYNCIODEBUG=1, python -X dev, asyncio.run(..., debug=True) or loop.set_debug(True). In debug mode asyncio logs any callback or task step that runs longer than loop.slow_callback_duration (100 ms by default) and names the coroutine. That names the handler, though not always the exact line.
3. Take a stack dump of the live process with py-spy dump --pid <pid> (py-spy is an external sampling profiler that reads another process's stack without modifying it; on Linux in a container it needs the ptrace capability, the permission to inspect another process's memory, for example --cap-add=SYS_PTRACE). If the main thread's top frame is in a driver, requests, socket or time.sleep call beneath an async def, that is the blocking call. py-spy record over a window gives a flame graph (a chart in which each bar is a function and its width is the share of samples spent in it, stacked by caller) if the freeze is intermittent.
4. Separate it from the other suspects. CPU-bound work in a handler: a core at 100% and the stack shows your own function. A slow downstream awaited properly: the loop lag stays low, because the loop is free. A connection pool or lock wait: lag stays low but latency rises; look at pool-wait metrics. An executor or thread pool starved: lag low, but to_thread calls queue.
5. Audit for the usual offenders: synchronous database drivers or ORM (object-relational mapper) calls, requests and other sync HTTP clients, synchronous Redis/Mongo clients, blocking DNS lookups, open() on slow disks, large json.loads or regex on big inputs, time.sleep, subprocess.run, calling .result() on a future from the loop thread, and third-party libraries that block internally.

A runnable reproduction

import asyncio, logging, os, sys, time

logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(name)s: %(message)s")

async def lag_probe(samples, interval=0.01):
    """Ask the loop to wake us every 10 ms; record how late it actually does."""
    while True:
        t = time.perf_counter()
        await asyncio.sleep(interval)
        samples.append((time.perf_counter() - t - interval) * 1000)

def blocking_query():
    time.sleep(0.3)                    # stands in for a driver call that blocks the thread

async def handler_bad():
    blocking_query()                   # blocks the loop thread: nothing else runs for 300 ms

async def handler_good():
    await asyncio.to_thread(blocking_query)   # runs in the default thread pool instead

async def main(handler):
    samples = []
    probe = asyncio.create_task(lag_probe(samples))
    await asyncio.sleep(0.05)
    await asyncio.gather(*(handler() for _ in range(3)))
    await asyncio.sleep(0.05)
    probe.cancel()
    samples.sort()
    print("%-12s worst loop lag %6.1f ms, median %.1f ms" % (handler.__name__, samples[-1], samples[len(samples)//2]))

if __name__ == "__main__":
    which = sys.argv[1]
    debug = len(sys.argv) > 2 and sys.argv[2] == "debug"
    asyncio.run(main(handler_bad if which == "bad" else handler_good), debug=debug)

Run python blocked_loop.py bad, python blocked_loop.py good and python blocked_loop.py bad debug. Sample run, docker run --rm --cpus=4 on python:3.12-slim (Python 3.12.15), one run on a noisy host:

handler_bad  worst loop lag  907.9 ms, median 2.4 ms
handler_good worst loop lag    4.7 ms, median 1.1 ms
WARNING asyncio: Executing <Task finished name='Task-3' coro=<handler_bad() done, defined at /w/blocked_loop.py:15> result=None created at /usr/local/lib/python3.12/asyncio/tasks.py:695> took 0.307 seconds
WARNING asyncio: Executing <Task finished name='Task-4' coro=<handler_bad() done, defined at /w/blocked_loop.py:15> result=None created at /usr/local/lib/python3.12/asyncio/tasks.py:695> took 0.300 seconds
WARNING asyncio: Executing <Task finished name='Task-5' coro=<handler_bad() done, defined at /w/blocked_loop.py:15> result=None created at /usr/local/lib/python3.12/asyncio/tasks.py:695> took 0.301 seconds
handler_bad  worst loop lag  900.1 ms, median 0.6 ms

Reading a warning: Executing <Task ... coro=<handler_bad() done, defined at /w/blocked_loop.py:15> ...> took 0.307 seconds names the coroutine (handler_bad), the file and line where it is defined, and how long that one step of it ran without yielding to the loop; each of the three handler tasks produced one, about 300 ms each.

The shape: the median lag is tiny (the loop is idle most of the time), the worst lag is about 900 ms because three 300 ms blocking calls ran back to back on the one thread, and the debug-mode warnings name handler_bad with its duration. With to_thread the loop lag stays in single-digit milliseconds. --cpus=1 gave the same result (worst lag about 895 ms for the blocked version, about 7 ms for the fixed one), because the blocked version is waiting, not computing.

To see the stack of a process stuck this way, this script blocks the loop in a function called blocking_query:

import asyncio, time

def blocking_query():
    time.sleep(0.5)

async def handler():
    while True:
        blocking_query()
        await asyncio.sleep(0)

asyncio.run(handler())

Run in the same image with pip install py-spy and --cap-add=SYS_PTRACE, py-spy dump --pid <pid> printed:

Process 12: python serve_blocked.py
Python v3.12.15 (/usr/local/bin/python3.12)

Thread 12 (idle): "MainThread"
    blocking_query (serve_blocked.py:4)
    handler (serve_blocked.py:8)
    _run (asyncio/events.py:88)
    _run_once (asyncio/base_events.py:1999)
    run_forever (asyncio/base_events.py:645)
    run_until_complete (asyncio/base_events.py:678)
    run (asyncio/runners.py:118)
    run (asyncio/runners.py:195)
    <module> (serve_blocked.py:11)

The thread is marked idle (not using CPU), the top frame is the blocking function, and the frames beneath it are the event loop's own machinery: that confirms a blocking call on the loop thread. How to read the frames: each line is function (file:line), newest call first. blocking_query is where the thread is stuck; handler is the coroutine that called it; _run executes one scheduled step of a task; _run_once is a single pass of the event loop, which picks up the ready callbacks and runs them; run_forever, run_until_complete and the two run frames are the start-up path that asyncio.run created. Seeing your own function directly above _run with a blocking call on top is the signature: the loop is inside one callback and cannot start its next pass.

The fixes

• Offload the blocking call: await asyncio.to_thread(fn, ...) (or run_in_executor with a dedicated, bounded executor so slow calls cannot starve the default pool). The default executor holds min(32, os.cpu_count() + 4) threads, so concurrency of offloaded waits is capped; and cancelling the awaiting task does not stop the thread.
• Use an async client for the hot paths (an async HTTP client, an async database driver). The loop then waits on the socket instead of a thread, and thousands of calls can be in flight.
• Bound concurrency toward the database. Whichever way you go, the connection pool is the ceiling: converting to async raises how many requests try to reach the database at once, so the pool size and a semaphore (a counter that lets at most N coroutines proceed while the rest wait) in front of it decide how many succeed, and the rest should wait cheaply or fail fast.
• Memory and CPU impact of the migration. Coroutines are much cheaper than threads, so memory per in-flight request falls; but all coroutine code still runs on one core, so CPU-heavy steps (large JSON, serialization, templating) now compete with every other request. Run one process per core, and move heavy computation to a process pool or worker.
• Keep it from coming back. Alert on loop lag, run debug mode in tests and staging so a blocking call in a new async def is logged, and in code review treat any synchronous library call inside async def as suspect.

Verify

Reproduce with the probe and a load test: before the fix loop lag spikes to the blocking duration under load; after it, lag p99 stays in milliseconds, throughput rises and the p99 request latency stops tracking the slowest call. Check that the offload pool or database pool did not become the new bottleneck (queue time and pool-wait metrics).

Step 1: confirm the event loop is the problem

The symptom (every request slow, only while one endpoint runs) points at a callback holding the single JavaScript thread. Confirm it with evidence, in this order:

1. Measure event-loop delay. Node's perf_hooks.monitorEventLoopDelay() returns a histogram (a table counting how many samples fell into each delay range, from which max and percentiles are read) of how late the loop's timer fires. The timer resolution is how often it is set to fire, here every resolution ms; if a 200 ms computation runs, the next sample is about 200 ms late. Export max and p99 (the value that 99 percent of samples are at or under; p50 is the median) as metrics and alert on them. The histogram records the whole interval between timer firings, so a healthy idle loop shows delays near the resolution itself (about 10 ms when the resolution is 10), not near zero; spikes line up with the slow endpoint.
2. Correlate. Plot event-loop delay against per-endpoint request counts or traces. If loop delay spikes only when the heavy endpoint is hit, and other endpoints slow down by roughly the heavy endpoint's computation time, the loop is the cause. CPU on the process near 100% of one core while the host has idle cores is a supporting sign.
3. Reproduce locally with a CPU profile: node --cpu-prof app.js writes a .cpuprofile file you can open in Chrome DevTools, and the heavy function shows as a long block of self time.
4. Rule out the alternatives. A slow database or saturated libuv pool also slows requests, but event-loop delay stays low in those cases because the loop is idle waiting; the delay histogram separates them.

A runnable reproduction and fix

This program starts an HTTP server with a /heavy route (about 200 ms of synchronous arithmetic on a 4-CPU run like the one below; longer on a slower or shared core) and a cheap /ping route. A client sends 200 cheap requests 10 ms apart while four heavy requests are in flight, and prints latency and loop delay. Mode inline runs the work in the handler; mode worker runs it in a worker_threads Worker.

const http = require('node:http');
const { monitorEventLoopDelay } = require('node:perf_hooks');
const { Worker, isMainThread, parentPort, workerData } = require('node:worker_threads');

function heavy() {            // roughly 200 ms of synchronous CPU work
  let x = 0;
  for (let i = 0; i < 6e7; i++) x = (x + i * 7) % 1000003;
  return x;
}

if (!isMainThread) {
  parentPort.postMessage(heavy());
  return;
}

const MODE = process.argv[2];  // 'inline' or 'worker'
const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();

const server = http.createServer((req, res) => {
  if (req.url === '/heavy') {
    if (MODE === 'inline') {
      res.end(String(heavy()));
    } else {
      const w = new Worker(__filename);
      w.once('message', (v) => res.end(String(v)));
    }
  } else {
    res.end('ok');
  }
});

function get(path) {
  return new Promise((resolve) => {
    const t = process.hrtime.bigint();
    http.get({ port: 3000, path, agent: false }, (r) => {
      r.resume();
      r.on('end', () => resolve(Number(process.hrtime.bigint() - t) / 1e6));
    });
  });
}

server.listen(3000, async () => {
  const lat = [];
  const heavyCalls = [];
  for (let i = 0; i < 200; i++) {                  // 200 cheap requests, 10 ms apart
    if (i % 50 === 10) heavyCalls.push(get('/heavy'));   // 4 slow requests spread over the run
    lat.push(await get('/ping'));
    await new Promise((r) => setTimeout(r, 10));
  }
  await Promise.all(heavyCalls);
  lat.sort((a, b) => a - b);
  const p = (q) => lat[Math.min(lat.length - 1, Math.floor(q * lat.length))].toFixed(1);
  console.log(`mode=${MODE}  cheap-request latency ms: p50=${p(0.5)} p99=${p(0.99)} max=${lat[lat.length - 1].toFixed(1)}`);
  console.log(`loop delay (monitorEventLoopDelay) max=${(h.max / 1e6).toFixed(0)} ms, p99=${(h.percentile(99) / 1e6).toFixed(0)} ms`);
  server.close();
  h.disable();
  process.exit(0);
});

Reading the client loop: each iteration sends one /ping and waits for it, sleeps 10 ms, and every 50th iteration (starting at 10) also fires a /heavy request without waiting for it, so four heavy requests overlap the pings at different moments; the latency of each /ping is what goes into lat. The /heavy handler in worker mode starts a new Worker per request only to keep the demo short; the recommendation below is a reusable pool, because a Worker has start-up cost.

Run with node lag.js inline and node lag.js worker. Sample run on node:22-slim (Node 22.23.3) with --cpuset-cpus=0-3, one run on a noisy host:

mode=inline  cheap-request latency ms: p50=0.4 p99=202.9 max=217.3
loop delay (monitorEventLoopDelay) max=222 ms, p99=192 ms
mode=worker  cheap-request latency ms: p50=0.5 p99=8.8 max=13.4
loop delay (monitorEventLoopDelay) max=19 ms, p99=15 ms

The shape: the median stays around a millisecond or less in both modes, but with the work inline the p99 jumps to roughly the length of the computation, because any request that lands during the 200 ms waits for it. With the work in a Worker the cheap requests stay far below the 200 ms stall, typically around 1 to 15 ms at p99 across re-runs, and occasionally higher on a busy host. With one CPU (--cpuset-cpus=0) the worker mode still protected the cheap requests in repeat runs (p99 roughly 2 to 16 ms in re-runs), though one core must then be shared between the loop and the worker, so it is the CPU limit that decides how much heavy work can run at once. The inline stall tracks how long the computation takes on the core it gets: it is single-threaded work, so adding CPUs does not shorten it, and on a slower or busier core the p99 and loop-delay maximum rise above 200 ms, so "roughly the length of the computation" means the length on that host. The roughly 10 ms floor in the loop-delay numbers is the timer resolution, not a stall: the minimum recorded delay is usually about 9 ms in both modes (an occasional run on a loaded host recorded less), so worker-mode loop-delay maxima of roughly 12 to 30 ms across re-runs (occasionally higher on a busy host) are that baseline plus the time the main thread spends creating each Worker and scheduling noise on a shared host.

Step 2: the options, and the cost of each

The first two rows are the cheapest to try and need no new infrastructure; the worker pool is the usual next step for a 200 ms computation; the process, queue and service rows apply when the work outgrows a single request or needs isolation.

Recommendation

For a computation that is 200 ms and fits in one request: first check whether the work can be made cheaper or cached; if not, put it behind a fixed-size pool of worker threads, with a bounded queue (sized at about the cores you can give it, leaving a core for the loop) and reject or return 503 when the queue is full. If results can arrive later, or the computation grows past a second or two, move to a job queue so the API tier is never at the mercy of heavy jobs. Pass a result handle (SharedArrayBuffer, or transfer the ArrayBuffer) instead of copying big payloads. Choose child processes or a separate service when you need fault isolation or different scaling; the extra serialization and operations cost then pays for itself.

Verify the fix

Rerun the same load with event-loop delay exported: p99 loop delay should fall back toward the timer resolution and the other endpoints' p99 should stop tracking the heavy endpoint's rate. Also load-test the heavy endpoint itself at 2x expected traffic to confirm the pool's queue and rejection behave: the failure you want is a fast 503 on the heavy route, not slow responses everywhere.

The problem

asyncio runs all coroutines on one thread, the event loop. A coroutine (a function declared with async def, which can pause and resume) only yields control at an await (the point where it pauses and lets other coroutines run). A blocking driver call (one that waits on a socket without await) freezes the whole loop for its duration, so every other request waits (this is head-of-line blocking). There are five ways out, each moving the wait somewhere else. Options 1, 2 and 4 are the usual choices; option 3 applies only to CPU-bound work and option 5 to blocking code that cannot be changed.

The options and what each costs

Evidence for the thread-pool options

Per the Python docs, run_in_executor(None, ...) uses the loop's default ThreadPoolExecutor, and that executor's default size is min(32, os.cpu_count() + 4). This script shows what that means:

import asyncio, os, time
from concurrent.futures import ThreadPoolExecutor

def db_call(ms):
    time.sleep(ms / 1000)              # a blocking driver call
    return ms

async def timed(coro, label):
    t = time.perf_counter()
    await coro
    print("  %-46s %5.0f ms" % (label, (time.perf_counter() - t) * 1000))

async def main():
    loop = asyncio.get_running_loop()
    print("os.cpu_count() =", os.cpu_count(), " default max_workers =", min(32, os.cpu_count() + 4))

    # 1. 100 concurrent 100 ms blocking calls on the default executor
    await timed(asyncio.gather(*(asyncio.to_thread(db_call, 100) for _ in range(100))),
                "100 x 100 ms on the default executor")

    # 2. A slow batch starves a fast request: 40 slow calls, then one 5 ms call
    async def slow_batch():
        await asyncio.gather(*(asyncio.to_thread(db_call, 500) for _ in range(40)))
    async def fast_call():
        await asyncio.sleep(0.05)
        await timed(asyncio.to_thread(db_call, 5), "5 ms call submitted behind the slow batch")
    await asyncio.gather(slow_batch(), fast_call())

    # 3. Bulkhead: slow work gets its own small executor, fast work keeps the default one
    slow_pool = ThreadPoolExecutor(8, thread_name_prefix="slow-db")
    async def slow_batch_isolated():
        await asyncio.gather(*(loop.run_in_executor(slow_pool, db_call, 500) for _ in range(40)))
    async def fast_call_isolated():
        await asyncio.sleep(0.05)
        await timed(asyncio.to_thread(db_call, 5), "5 ms call, slow work on its own pool")
    await asyncio.gather(slow_batch_isolated(), fast_call_isolated())
    slow_pool.shutdown()

    # 4. Cancelling the awaiting task does not stop the thread
    finished = []
    def slow_call():
        time.sleep(0.3)
        finished.append(True)
    task = asyncio.create_task(asyncio.to_thread(slow_call))
    await asyncio.sleep(0.05)
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("  task cancelled; thread finished yet?", bool(finished))
    await asyncio.sleep(0.4)
    print("  0.4 s later, thread finished:", bool(finished))

asyncio.run(main())

Sample run, docker run --rm --cpus=4 on python:3.12-slim (Python 3.12.15). Timings are from a noisy host and the host has 14 CPUs, so the default pool is 18 threads:

os.cpu_count() = 14  default max_workers = 18
  100 x 100 ms on the default executor             616 ms
  5 ms call submitted behind the slow batch        955 ms
  5 ms call, slow work on its own pool              10 ms
  task cancelled; thread finished yet? False
  0.4 s later, thread finished: True

What it shows:

• 100 blocking calls of 100 ms through 18 threads take about ceil(100/18) = 6 rounds, so roughly 600 ms; an async driver with a connection pool of 100 could overlap all of them in about 100 ms. That gap is the cost of the thread pool: concurrency of waiting is capped by the pool size.
• 40 slow (500 ms) calls fill the default pool and queue ahead of one fast 5 ms call; the fast call finished after about 955 ms because it waited behind the queued slow work. The arithmetic: 40 slow calls on 18 threads run as waves of 18, 18 and 4, 500 ms each; the fast call, submitted at 50 ms, queues behind all 40 and starts in the third wave at 1,000 ms, runs 5 ms, and so finishes at about 1,005 ms, which is 955 ms after it was submitted. This is pool exhaustion: the default executor is shared by everything that offloads, so one slow query class starves the rest. Giving the slow work its own 8-thread pool (the bulkhead) left the fast call at about 10 ms, at the cost of the slow batch taking longer (it is limited to 8 at a time).
• Cancelling the awaiting task (for example on a client timeout) did not stop the thread: thread finished yet? False right after the cancel, then True 0.4 s later. The call still occupies a thread and a database connection until it returns, so timeouts must also be set in the driver or database (a statement timeout is a database setting that aborts any query running longer than a given time), not only in asyncio.

At --cpus=1 the same script gave the same shape (606 ms, 957 ms, 6 ms) because the work is waiting, not computing; it would change only if the offloaded work were CPU-bound.

Sizing the offload pool

Use Little's law: concurrent calls = arrival rate x call duration. At 200 queries/s of 20 ms each (hypothetical), 200 x 0.020 = 4 calls in flight, so a pool of 8 gives twice the headroom needed. A thread pool larger than the database connection pool does not help: the extra threads would just wait for a connection. So size the executor to match the connection pool (here a database pool of 8 connections and an executor of 8 threads, which is also the 2x headroom above), and bound how many requests wait for it (an asyncio.Semaphore, a counter that lets at most N coroutines proceed and makes the rest wait, or a bounded queue in front), so overload shows up as fast failures or timeouts rather than an ever-growing backlog. Keep CPU-heavy work off these threads.

Recommendation

Right now: wrap the blocking calls in run_in_executor with a dedicated, bounded executor sized to the database pool, so the event loop stays responsive and one slow path cannot starve the rest; set statement timeouts in the database. That is the smallest change that keeps the service correct. Then move the hot paths to an async driver, which removes the thread-per-call ceiling (the 100-call example above) and the cancellation problem. A process pool is the wrong tool for database waits, and a separate service is justified only when the blocking code cannot be changed (a vendor library) or needs different scaling.

What would flip this: if the number of simultaneous in-flight calls is large (thousands) the thread pool's memory and context-switch cost makes the async driver necessary; if the blocked call is CPU-heavy (for example big result parsing) a process pool or native library is the right offload.