Latency Analysis & Optimization Questions

The three signals

• Metrics: numeric, aggregated time series (request count, error rate, p50/p95/p99 latency (the 50th/95th/99th percentile: the value below which that fraction of requests finish)) sampled or rolled up over time. Cheap to store for a long time and great for spotting "is something wrong, and since when," but they lose per-request detail, a latency spike shows up as a number, not a reason.
• Logs: discrete, timestamped text records, often one per request or per notable event, with whatever free-form detail the code chose to write. Good for "what exactly happened for this one thing," but expensive to store and search at high volume, and a single log line rarely shows how one request's time was split across services.
• Traces: a tree of timed spans that shows how one specific request propagated across services, with every hop's start and duration linked by a shared trace ID. Good for "which hop actually took the time," but only exists for the requests that were sampled/captured.

Example for intermittent latency spikes

A metrics dashboard shows p99 latency spiking at 14:32 for about three minutes, with no matching rise in the error rate, telling you it's a slow path, not an outage. Searching logs around that window turns up a burst of "slow query" warnings from the database client tied to one specific query pattern. Pulling a few traces from that same window shows most of each request's time concentrated in the span for that database call, matching what the logs already hinted at.

Correlating them

Use the metric spike to find the exact time window worth investigating (metrics are cheap to scan broadly). Use that window to filter logs for anomalies within it (logs narrow "when" down to "what kind of thing"). Then pull a handful of specific traces, ideally ones the tracing system already flags as high-latency, from that same window to see exactly which span dominates, and cross-reference back to the log line by the shared request or trace ID. Each signal narrows the search for the next one; jumping straight to reading raw logs or traces without the metric to point you at a time window usually means searching too much data with too little direction.

Framing

The goal is an observability strategy that lets engineers find and fix latency problems across many services written in different languages, owned by different teams, without missing the signal or drowning in cost.

Distributed tracing

Adopt a vendor-neutral tracing standard, OpenTelemetry (a common API and set of language libraries for emitting traces, metrics, and logs in a compatible format), so every service, regardless of language, produces spans that share the same trace ID and can be stitched into one end-to-end view of a request's path.

Sampling strategies: head vs tail

• Head-based sampling decides whether to trace a request at its very first hop, for example "trace 1% of requests." It's cheap and simple, but samples blind to outcome, meaning most sampled traces end up being the boring, fast, successful ones, exactly the requests that don't need debugging.
• Tail-based sampling defers the sampling decision until the request finishes, so it can specifically keep traces for slow or failed requests, the ones that actually matter for latency debugging, and discard fast, successful ones at a much higher rate. The trade-off is it needs to buffer every span for a request somewhere until the decision is made, which costs more infrastructure, a collector layer holding spans in memory, than head-based sampling.
• A common practical approach combines both: low-rate head-based sampling for baseline statistical visibility, plus tail-based sampling specifically for anything exceeding a latency threshold or returning an error, giving both the aggregate picture and guaranteed visibility into the cases that actually need debugging.

Span enrichment

Attach high-value, low-cardinality attributes to spans, such as service name, endpoint, status code, region, and tenant tier, so traces can be filtered and aggregated meaningfully. Avoid attaching genuinely high-cardinality values, raw user IDs or full request bodies, directly as span attributes; that belongs in logs correlated to the trace, not the trace's indexed metadata.

Correlating traces with metrics and logs

Propagate the trace ID into log lines, structured logging with a trace_id field, and expose it as an exemplar on relevant histogram metrics, many metrics backends support attaching a sample trace ID to a specific histogram bucket. That way, an engineer looking at a metric spike, "p99 jumped at 14:32," can click through to an actual example trace from that moment, and from that trace's logs, rather than manually correlating timestamps across three separate tools.

Storage, retention, and query performance

Full-fidelity trace storage at scale gets expensive fast. A common pattern is short full-fidelity retention, days, for interactive debugging, with a downsampled or aggregated form, such as span-duration histograms per service and endpoint computed and stored, not the raw spans, kept for longer-term trend analysis, weeks to months, since most trace-level debugging happens within days of an incident, not months later.

Balancing observability detail vs cost

• Every extra span, attribute, or sampled trace has a real cost: storage, ingestion, and query-time cost. The strategy should default to answering the specific latency questions the team actually asks, where does time go per hop, which requests are outliers, what changed before a regression, rather than maximal instrumentation for its own sake.
• Tail-based sampling is the single highest-leverage cost control here: it keeps overall trace volume low while still guaranteeing the slow and failed requests that actually drive latency investigations are never missed.

Trace and span

A trace is the full record of one request's journey through a distributed system, everything that happened to serve it end to end. A trace is made up of spans: each span is one unit of work, like a single RPC call or a single database query, with its own start time, end time, name, and metadata (tags/attributes).

Parent-child relationships

Spans form a tree (or more generally a DAG): a span is the child of whichever span invoked it. For example, an API gateway's span is the parent of the auth-service span it calls, which might itself be the parent of a database-query span. That tree structure is what makes a trace useful beyond "the total time was X": it shows whether work happened sequentially (one span starts only after its parent's other children finish) or in parallel (several child spans overlap in time), which is exactly what you need to reason about where wall-clock time actually went.

Why context propagation matters

Context propagation is passing the trace's identifying information (a trace ID, the current span ID, and the sampling decision) along with the request as it crosses process and network boundaries, typically as HTTP headers (like the W3C traceparent header) or gRPC metadata. Without it, each service only knows about its own local work; there's no way to stitch five separate services' telemetry back into one coherent trace of a single user request.

Common sampling strategies

• Head-based sampling: decide whether to keep a trace right at the start of the request (e.g., keep a random 1%). Cheap and simple, but you're gambling before you know whether the request turns out to be interesting.
• Tail-based sampling: buffer the full trace and decide whether to keep it only after it completes, so you can apply rules like "always keep traces with errors or above a latency threshold." Much better at catching rare, interesting events, but requires a collector that can buffer and hold spans before deciding.
• Adaptive/rate-limited sampling: target a roughly fixed volume of kept traces per second regardless of traffic swings, so sampling doesn't silently drop to near-zero during a traffic spike or flood a backend during a burst.

What tracing helps answer

Which specific hop a regression is coming from, why one particular slow request was slow (as opposed to just "the p99 is high"), whether time was spent sequentially or in parallel, and where a request spent idle/waiting time versus actually doing work.

Trade-offs of high sampling rates and high-cardinality data

Storage and ingestion cost scales roughly with how many traces (and how much detail per trace) you keep, so sampling everything at high fidelity gets expensive fast. High-cardinality attributes on spans (like a full URL with query parameters, or a raw user ID as a tag) multiply that cost further, since backends often index on tag values, and a tag with millions of distinct values blows up that index. The practical trade-off is: sample richly enough (often via tail-based sampling specifically for errors and slow traces) to reliably catch the rare tail events you actually care about, without paying full cost to store the enormous volume of unremarkable, fast, successful traces that make up the bulk of traffic.

Approach

Use the nearest-rank method: sort the samples ascending, then for each requested percentile compute its rank as ceil((percentile / 100) * n), clamp it into [1, n], and return the value at that rank (adjusting for 0-indexing).

import math
from typing import List, Dict


def compute_percentiles(latencies: List[float], percentiles: List[float]) -> Dict[float, float]:
    """Nearest-rank percentile computation.

    latencies: sample of latency values in ms.
    percentiles: percentile values in (0, 100], e.g. [50, 95, 99].
    Returns a dict mapping each requested percentile to its value.
    """
    if not latencies:
        raise ValueError("latencies must not be empty")

    data = sorted(latencies)
    n = len(data)
    result = {}
    for p in percentiles:
        if not (0 < p <= 100):
            raise ValueError(f"percentile {p} out of range (0, 100]")
        rank = math.ceil((p / 100) * n)
        rank = max(1, min(rank, n))
        result[p] = data[rank - 1]
    return result


if __name__ == "__main__":
    samples = [12, 8, 45, 22, 19, 995, 31, 27, 1050, 15,
               18, 33, 29, 21, 1200, 17, 24, 26, 20, 30]
    percentiles = compute_percentiles(samples, [50, 95, 99])
    for p, v in percentiles.items():
        print(f"p{int(p)}: {v}ms")

Output:

p50: 24ms
p95: 1050ms
p99: 1200ms

Key points

Sorting dominates the cost; once sorted, every requested percentile is an O(1) index lookup. Nearest-rank always returns an actual sample from the data (no interpolation), which is simple and deterministic but slightly coarse: two different samples one apart in rank can give noticeably different answers on a small dataset, which is exactly why the streaming samples question below matters more than it might look.

Complexity

Time: O(n log n) for the sort. Space: O(n) to hold the sorted copy (or O(1) extra if sorting in place, but you still need all n values in memory to sort them at all).

Modifying this for streaming data or datasets that don't fit in memory

The approach above requires every sample in memory at once to sort it, which breaks down once latencies arrive continuously or the dataset is too large to hold. Two standard fixes:

1. Approximate streaming quantile sketches. Instead of storing raw samples, maintain a compact summary structure that's updated incrementally as each new latency arrives, using memory bounded by the sketch's configured size rather than by the number of samples seen. A t-digest (a data structure that keeps finer-grained buckets, or "centroids," near the tails of the distribution where percentile precision matters most, and coarser ones near the middle) or an HDR (High Dynamic Range) Histogram are the two most common choices in production latency monitoring, and this is what tools like Prometheus histograms and most APM (Application Performance Monitoring: tools that track a running service's live latency/error metrics) percentile dashboards actually do under the hood, rather than sorting raw samples.
2. A two-pass approach for exact percentiles on data too large for one machine. First pass: build a rough histogram (e.g., in a distributed batch job) to find which bucket the target rank falls into. Second pass: only sort or scan within that much smaller bucket to get the exact value, avoiding a full sort of the entire dataset.

Edge cases

Empty input (the implementation above raises rather than returning a misleading value), a single-element input (every percentile returns that one value), an out-of-range percentile request like 0 or 150 (rejected explicitly rather than silently clamped), duplicate values (nearest-rank handles them with no special casing needed), and unsorted input (the function must not assume it's already sorted, since callers will pass raw latency samples in arrival order).

Plausible hidden causes

Instrumented spans only measure the time a library explicitly wrapped. Roughly 700ms is unaccounted for, which points at delay happening outside what any span is timing:

1. Queuing before the traced work even starts. Time spent in a listen/accept queue, a load balancer queue, or a thread-pool/executor queue before a handler begins running is invisible to a span that only starts its clock once the handler executes.
2. CPU scheduling delay (off-CPU time). A thread can be runnable but not actually running because the CPU is busy with something else, another tenant, garbage collection, or kernel work. A span reports wall-clock duration, but if the underlying instrumentation doesn't distinguish "doing work" from "waiting to be scheduled," a big gap between a span's reported CPU time and its wall-clock time is exactly this.
3. Blocking calls that happen outside the traced boundary. DNS resolution, a TLS handshake, or waiting to check out a connection from a pool typically happen before an RPC client library's span even starts, so they're silently absorbed into the gap.
4. Garbage collection or other stop-the-world pauses on any of the hosts involved, which stall the whole process regardless of which span happens to be "active" at that moment.
5. Retries not reflected proportionally. A span might report only the final, successful attempt's duration, hiding one or two earlier failed or slow attempts that also consumed wall-clock time.

Three experiments/probes to surface the missing delay

1. Instrument the queueing boundary explicitly. Add a timestamp at the earliest possible point (e.g., when the load balancer or reverse proxy accepts the connection, or when a request enters the process) and compare it against when the handler's own span actually starts. This directly measures accept-to-dispatch queuing delay that current spans skip entirely.
2. Kernel-level off-CPU analysis. Use strace -T -f on the process to see per-syscall wait time (a lot of time in futex calls means lock contention; time in connect/read means I/O waits), or an eBPF-based tool (eBPF lets you attach lightweight tracing/profiling programs to running kernel events without changing kernel code) (like an off-CPU profiler) to directly measure how long threads sat runnable but not scheduled on a core. Checking perf sched or the kernel's scheduler run-queue stats tells you whether the host itself is CPU-oversubscribed.
3. Instrument the client library's connection lifecycle. Turn on debug metrics or logging for "time spent acquiring a connection from the pool," and separately time DNS resolution and the TLS handshake. A cold client, or one whose pooled connections keep expiring, can pay these costs on every request without any of it showing up as a traced RPC span.

Running all three against the same time window narrows which bucket (pre-dispatch queuing, CPU scheduling, or connection-lifecycle overhead) actually accounts for most of the missing ~700ms, rather than guessing.