Latency Analysis & Optimization Questions

Defining latency and throughput

• Latency: the time a single request takes to complete, start to finish (e.g., "this endpoint responds in 120ms").
• Throughput: how much work the system handles per unit of time (e.g., "this API handles 5,000 requests per second").

They're related but not the same thing, and a change can move them in opposite directions. Batching is the clearest example: waiting to accumulate a bigger batch before processing raises the latency of any individual item (it has to wait for the batch to fill), but it can raise overall throughput since more items get processed per unit of overhead.

Design decisions that mainly improve latency

Caching, reducing the number of network round trips, keeping connections warm (connection pooling/keep-alive so you're not paying a fresh TCP/TLS handshake per request), running independent calls in parallel instead of one after another, shrinking payload size, placing content closer to users (CDN/edge), and hedged requests for latency-critical calls.

Design decisions that mainly improve throughput

Batching, horizontal scaling (more instances processing work concurrently), asynchronous/queue-based processing that decouples accepting work from finishing it, connection pooling sized to maximize concurrent in-flight work, and more efficient serialization that uses less CPU per request (raising the requests-per-second ceiling for the same hardware). Some changes, like removing an N+1 query pattern (issuing one query to fetch a list, then a separate query per item in that list, instead of one batched query), tend to help both at once.

Prioritizing when page loads are slow but CPU utilization is low

Low CPU utilization while requests are still slow is a strong signal the bottleneck isn't compute, it's waiting: on a slow downstream call or database, on the network (sequential round trips that could run in parallel, or an asset with no CDN in front of it), or on some form of queuing or contention (waiting for a connection-pool slot, a lock, or a downstream's own tail latency) that doesn't show up as CPU burn at all.

Given that, the priority should NOT be "add more CPU or more instances," since low CPU utilization tells you compute capacity isn't the constraint. Instead: trace the request end to end to see exactly where the wall-clock time goes, check whether dependent calls that could run in parallel are instead being made sequentially, check the downstream services' own reported latency rather than assuming the problem is local, and check for time spent waiting on a connection pool or queue rather than actually doing work.

Why throughput and latency can pull in opposite directions

Latency is how long one request takes; throughput is how many requests the system completes per second. Techniques that raise throughput often do so by making individual requests wait so they can be processed more efficiently in bulk, which raises the latency of any single request even as the system does more work per second overall.

Concrete examples where increasing throughput worsens latency

• Batching: a service that waits up to 20ms to collect a batch of 50 database writes before sending them together cuts per-write overhead and raises writes per second, but a single write that arrives alone is now stuck waiting up to 20ms for the batch to fill or time out, versus being sent in under a millisecond by itself.
• Queueing: adding a queue in front of a fixed-size worker pool smooths bursts and keeps throughput high right up to the pool's capacity, but every request now waits in line behind whatever is ahead of it; that wait grows sharply as utilization approaches 100%.
• A small connection pool: reusing a small set of database connections avoids the cost of opening a new connection per request, which is good for throughput, but once every connection is busy, new requests queue for a free one, adding wait time under load.
• Parallelism with shared resources: running more worker threads increases throughput on a multi-core box, but once threads start contending for the same lock, disk, or downstream connection, individual request latency rises from contention even as aggregate throughput keeps climbing.

The math behind the tail turning up

wait_time ≈ service_time × utilization / (1 − utilization)

For example, with a fixed service_time of 10ms: at 50% utilization, wait_time ≈ 10 × 0.5 / 0.5 = 10ms of queueing on top of the 10ms of actual work. At 90% utilization, wait_time ≈ 10 × 0.9 / 0.1 = 90ms, nine times the queueing delay for less than double the load. That's the hockey stick: going from 50% to 90% utilization only doubles the work rate but multiplies the queueing wait by 9x.

Utilization here is arrival rate divided by service rate. At low utilization, wait time is small; as utilization climbs past roughly 70-80%, wait time rises steeply (the "hockey stick" curve) because there is less and less slack left to absorb bursts. Pushing utilization toward 100% to maximize throughput is a latency trap.

Choosing parameters for a service with both interactive and background traffic

• Give interactive requests their own worker pool and connection pool, sized so utilization stays well under 70% even at peak, keeping their queueing delay small.
• Run background jobs on a separate pool, or a lower-priority queue, so a burst of batch work cannot steal capacity from interactive requests; if they must share infrastructure, use priority queues so interactive requests always jump the line.
• Batch and pool aggressively for the background path (larger batch windows, bigger connection pools, more parallelism), since individual job latency does not matter there, only overall throughput.
• Size batch windows for anything on the interactive path (5-10ms, say) as a small fraction of the latency budget, not a value tuned purely for throughput.

Picking SLO targets

Start from what actually affects the business and users, not an arbitrary round number: for a billing API, a slow or failed checkout call directly risks lost revenue and support load, so the SLO (Service Level Objective: the internal target you set for a metric like latency or availability) should track what customers actually feel. Anchor the target against your own baseline distribution first (don't set a target far tighter than what the system currently, realistically achieves without major investment), and set it off percentiles, not the average, since the average can look healthy while a real slice of checkout attempts are painfully slow.

Metrics to use

• Latency: p95 and p99 response time for the billing endpoints, being explicit about whether failed or timed-out requests are included in the latency population or tracked separately (excluding them silently is a common way an SLI (Service Level Indicator: the actual measured metric, e.g. p99 latency, that gets compared against the SLO) quietly stops meaning what people think it means).
• Availability: the percentage of requests that complete successfully (not 5xx, not timed out), often tracked as its own SLO alongside latency rather than folded into it.

Translating business impact into thresholds

Look at your own funnel or conversion data for where checkout abandonment starts climbing as latency increases, and set the latency ceiling with real headroom below that point, not right at the edge of where users start leaving. The same logic applies to availability: estimate what a percentage point of failed billing requests costs in lost transactions and support tickets, and use that to justify how tight (and how expensive) the target should be, since tighter targets generally cost more engineering effort and infrastructure to sustain.

Error budgets

The error budget is simply what's left over: 1 minus the SLO. A 99.9% availability SLO over a 30-day month leaves a 0.1% error budget:

30 days = 43,200 minutes
0.1% of 43,200 minutes = 43.2 minutes

That's roughly 43 minutes of full-equivalent downtime allowed per month, whether spent as one incident or many smaller partial-degradation periods that add up.

How the error budget should drive release cadence and incident response

• Release cadence: while budget remains healthy, ship at normal (or even accelerated) velocity. As the budget gets consumed, slow down: require canary rollouts (releasing a change to a small slice of traffic or users first, before rolling out to everyone), extra review, or freeze risky changes entirely until the budget recovers. This turns "reliability" from a vague value into a concrete, negotiable resource that product and engineering both watch.
• Incident response: alert on burn RATE (how fast the budget is being consumed) rather than only on absolute SLO breach, so a fast, severe burn pages immediately while a slow, minor burn becomes a lower-urgency ticket. Postmortems and remediation priority should also weigh directly against how much budget an incident consumed, giving the team a shared, quantitative way to decide whether to prioritize a reliability fix over the next feature.

Signals for CPU-bound

• CPU utilization (user + system time) is high or near 100% across cores, and the run queue (threads ready to run but waiting for a free core) is longer than the number of available cores.
• iowait is low: the CPU isn't idle waiting on disk or network, it's genuinely busy computing.
• Throughput improves when you add cores or optimize the hot code path, but not when you just add more concurrent connections.

Signals for I/O-bound

• iowait is elevated: the CPU is frequently idle while waiting on disk or network operations to complete.
• Disk or network queue depth is high, or syscall tracing shows a large share of time inside read/write/connect calls.
• Requests are slow even though CPU utilization looks unremarkable or even low.

Tools and metrics to check

• top / vmstat: the us (user), sy (system), and id (idle) columns give a quick CPU-vs-idle split; mpstat -P ALL breaks this down per core, useful when one core is saturated while others are idle (a concurrency/parallelism problem, not a raw compute problem).
• iostat -x: disk %util and average wait time per I/O operation.
• Network metrics: throughput, retransmits, connection resets, which signal network-level stalls rather than local disk I/O.
• strace -c: a summary of time spent per syscall, useful to see whether a process is dominated by I/O syscalls versus CPU-heavy work.
• Flamegraphs and other CPU profiling tools can confirm which specific code is consuming CPU once you already know you're CPU-bound; that deeper profiling workflow is its own topic, here the goal is just to establish CPU-bound vs I/O-bound first.

Actions for each diagnosis

• CPU-bound: optimize the hot code path (an algorithmic or serialization-format improvement usually beats just adding hardware), or scale out horizontally if the workload is legitimately compute-heavy and correctly provisioned; batching or deduplicating repeated work also reduces total CPU spent per unit of useful output.
• I/O-bound: increase concurrency so other work can proceed while one request waits on I/O (as long as the downstream isn't already saturated), add caching to avoid the I/O entirely for repeat requests, tune connection pooling and keep-alive to cut reconnect overhead, and check whether the I/O itself (disk, network, or a downstream dependency) is saturated, in which case the fix is capacity or dependency-side, not something you can code around locally.
• Mixed signals (moderate CPU and moderate iowait together) often point somewhere else entirely, like lock contention or garbage collection, and are worth investigating as their own category rather than forcing them into a CPU-vs-I/O split.

Plausible hidden causes

Instrumented spans only measure the time a library explicitly wrapped. Roughly 700ms is unaccounted for, which points at delay happening outside what any span is timing:

1. Queuing before the traced work even starts. Time spent in a listen/accept queue, a load balancer queue, or a thread-pool/executor queue before a handler begins running is invisible to a span that only starts its clock once the handler executes.
2. CPU scheduling delay (off-CPU time). A thread can be runnable but not actually running because the CPU is busy with something else, another tenant, garbage collection, or kernel work. A span reports wall-clock duration, but if the underlying instrumentation doesn't distinguish "doing work" from "waiting to be scheduled," a big gap between a span's reported CPU time and its wall-clock time is exactly this.
3. Blocking calls that happen outside the traced boundary. DNS resolution, a TLS handshake, or waiting to check out a connection from a pool typically happen before an RPC client library's span even starts, so they're silently absorbed into the gap.
4. Garbage collection or other stop-the-world pauses on any of the hosts involved, which stall the whole process regardless of which span happens to be "active" at that moment.
5. Retries not reflected proportionally. A span might report only the final, successful attempt's duration, hiding one or two earlier failed or slow attempts that also consumed wall-clock time.

Three experiments/probes to surface the missing delay

1. Instrument the queueing boundary explicitly. Add a timestamp at the earliest possible point (e.g., when the load balancer or reverse proxy accepts the connection, or when a request enters the process) and compare it against when the handler's own span actually starts. This directly measures accept-to-dispatch queuing delay that current spans skip entirely.
2. Kernel-level off-CPU analysis. Use strace -T -f on the process to see per-syscall wait time (a lot of time in futex calls means lock contention; time in connect/read means I/O waits), or an eBPF-based tool (eBPF lets you attach lightweight tracing/profiling programs to running kernel events without changing kernel code) (like an off-CPU profiler) to directly measure how long threads sat runnable but not scheduled on a core. Checking perf sched or the kernel's scheduler run-queue stats tells you whether the host itself is CPU-oversubscribed.
3. Instrument the client library's connection lifecycle. Turn on debug metrics or logging for "time spent acquiring a connection from the pool," and separately time DNS resolution and the TLS handshake. A cold client, or one whose pooled connections keep expiring, can pay these costs on every request without any of it showing up as a traced RPC span.

Running all three against the same time window narrows which bucket (pre-dispatch queuing, CPU scheduling, or connection-lifecycle overhead) actually accounts for most of the missing ~700ms, rather than guessing.