Latency Analysis & Optimization Questions

Approach

EWMA (exponentially weighted moving average) keeps a single running estimate and blends in each new observation using a weight alpha that decays exponentially for older data, so the tracker reacts to recent latency shifts without storing the full history. The stable update form is:

new_estimate = old_estimate + alpha * (value - old_estimate)

This is algebraically the same as (1-alpha)*old + alpha*value, but it avoids computing two separate products of a potentially large old_estimate and value, so it accumulates less floating-point error over millions of updates, and it naturally handles the very first call: instead of starting the internal state at 0 (which would bias every early estimate low during warm-up), we seed it with the first observation.

Since the class must accept update(value, timestamp), alpha is defined as a decay rate per unit time rather than a fixed per-call weight: after a gap of dt seconds, the fraction of the old estimate's weight still remaining is (1 - alpha) ** dt. On a regularly sampled stream this collapses to the textbook fixed-alpha EWMA; on a bursty stream (long gaps, then rapid bursts) it keeps the smoothing time-consistent instead of over- or under-weighting an observation purely because of how densely requests happened to arrive around it.

class EWMA:
    """Exponentially weighted moving average for a streaming metric (e.g. request latency).
    Decay is defined PER UNIT TIME (alpha = fraction of old weight lost per second) so that
    irregular sampling intervals don't silently change the effective smoothing.
    """

    def __init__(self, alpha: float):
        if not (0.0 < alpha < 1.0):
            raise ValueError("alpha must be in (0, 1)")
        self.alpha = alpha
        self._value = None
        self._last_ts = None

    def update(self, value: float, timestamp: float) -> None:
        if self._value is None:
            self._value = float(value)
        else:
            dt = max(timestamp - self._last_ts, 0.0)
            decay = (1.0 - self.alpha) ** dt if dt > 0 else 1.0
            effective_alpha = 1.0 - decay
            self._value += effective_alpha * (value - self._value)
        self._last_ts = timestamp

    def value(self) -> float:
        if self._value is None:
            raise ValueError("no observations yet")
        return self._value


if __name__ == "__main__":
    e = EWMA(alpha=0.3)
    stream = [(0, 20), (1, 21), (2, 19), (3, 500), (4, 20), (5, 21), (6, 480), (7, 20)]
    for ts, v in stream:
        e.update(v, ts)
        print(f"t={ts} obs={v:4d}ms ewma={e.value():7.2f}ms")

Output:

t=0 obs=  20ms ewma=  20.00ms
t=1 obs=  21ms ewma=  20.30ms
t=2 obs=  19ms ewma=  19.91ms
t=3 obs= 500ms ewma= 163.94ms
t=4 obs=  20ms ewma= 120.76ms
t=5 obs=  21ms ewma=  90.83ms
t=6 obs= 480ms ewma= 207.58ms
t=7 obs=  20ms ewma= 151.31ms

Key points

• Seed the state with the first observation, never with 0; a 0-start biases every early estimate low until it "warms up."
• Use the incremental correction form (old + alpha*(new - old)), not the two-term weighted-sum form; it is the numerically stable idiom used in real monitoring libraries.
• Deriving the effective per-call alpha from elapsed time (1 - (1-alpha)**dt) keeps the tracker correct on bursty, irregularly spaced streams, not just clean fixed-interval ticks.

Complexity

O(1) time and O(1) space per update: the class holds exactly two scalars (the running estimate and the last timestamp), no matter how many observations have streamed through it.

Edge cases

• alpha outside (0, 1) is rejected in the constructor: alpha near 1 means "mostly ignore history" and alpha near 0 means "barely react," so an out-of-range value is a caller bug worth failing loudly on.
• Calling value() before any update() raises instead of silently returning 0, which would be indistinguishable from a real 0ms latency.
• A zero or negative dt (an out-of-order or duplicate timestamp) is clamped to 0 so it can't invert the decay math.

Why EWMA cannot give you p95

EWMA collapses the entire incoming distribution into one number, a weighted mean, and that mean has no memory of the shape of the data that produced it. A percentile is a question about the whole distribution (what value do 95% of requests fall under), and a single scalar cannot answer that: two very different distributions, one tight around 475ms and one bimodal split evenly between 50ms and 900ms (mean (50+900)/2 = 475ms), can produce the exact same EWMA value.

The run above makes this concrete. The true p99 of that eight-point stream is 500ms (the actual spike), but the EWMA line never gets near it. It jumps toward the spike for one step and decays right back down toward the steady ~20ms band. A dashboard reading "EWMA latency: 150ms" during that stream would completely hide the fact that individual requests spiked to 500ms, 25 times the ~20ms steady-state baseline the EWMA had otherwise settled back toward, not something visible from the 150ms reading itself.

Practical alternative: use a streaming quantile sketch that tracks the shape of the distribution with bounded memory. The two common choices are a t-digest (adaptively allocates more, smaller buckets near the tails, where precision matters most for p95/p99, and coarser buckets in the dense middle) and an HDR Histogram (fixed relative-error buckets on a log scale, cheap to merge across hosts and common in latency-focused APM tooling). Both let you query any percentile on demand with a small, bounded memory footprint and can be merged across shards, which a raw EWMA fundamentally cannot do, because it is not a percentile estimator at all, it is a smoothed mean.

The formats

• JSON (JavaScript Object Notation): text-based, human-readable, self-describing (field names travel with every record), no schema required to parse.
• Protobuf (Protocol Buffers): binary, schema-defined via a .proto file with numbered fields, compact and fast to encode/decode since field names aren't repeated in every record.
• MessagePack: binary, with essentially the same flexible, schema-less data model as JSON (maps, arrays, strings, numbers) but binary-encoded, so it's a fast drop-in for JSON without requiring a schema.
• Avro: binary, schema-defined, but the schema travels with the data (or via a shared schema registry) rather than being baked into generated code, which makes it well suited to systems where the schema evolves independently of any one consumer, common in data-engineering pipelines.

Impact on latency and throughput

Binary formats generally cost less CPU to encode and decode than JSON's text parsing, and they produce smaller payloads since numbers and repeated field names aren't spelled out as text. Both effects matter directly: less CPU time is less time on the request's critical path, and a smaller payload means less time spent transferring it over the network, which compounds at high request rates or with large payloads. JSON's overhead is usually a non-issue for low-volume or human-facing APIs, where readability and the ability to curl and eyeball a response are worth more than the CPU/network savings.

Worked example

Take one small event record: {"user_id": 48213, "event": "click", "ts": 1699999999}. Encoded compactly:

JSON:        49 bytes  (spells out "user_id", "event", "ts" as literal text every time)
MessagePack: 32 bytes  (binary, same schema-less model as JSON)
Protobuf:    17 bytes  (binary, numbered field tags instead of field names, schema-defined)
Avro:        14 bytes  (binary, no per-field tags at all, relies entirely on the shared schema)

JSON is roughly 2.9x the size of Protobuf and 3.5x the size of Avro here, because it repeats the field names on every single record; the binary formats spend zero bytes on that, and Avro saves a further two bytes over Protobuf by dropping per-field tags too, relying entirely on the shared schema for field identity. On the CPU side, a quick decode-time comparison (200,000 iterations in Python) showed JSON's parser taking about 830ns per record versus MessagePack's about 210ns, roughly a 4x gap from skipping text tokenization; the exact multiplier varies by language and library, but the direction (binary formats decode faster) holds broadly.

When to change format

Switch when serialization or payload size shows up as a real, measured chunk of your critical path, visible as CPU time in a profiler or a trace span, or as a meaningful fraction of your bandwidth budget, not preemptively. Also consider switching when you need first-class schema evolution across many independently-deployed services (Protobuf and Avro both support adding or deprecating fields safely via field numbering or a schema registry), which JSON and MessagePack don't provide on their own.

Trade-offs

• CPU cost: binary formats are cheaper to parse; JSON's text parsing has real, measurable overhead at high volume.
• Payload size: binary formats are smaller (no repeated field names, compact number encoding); this maps directly to network transfer time.
• Schema evolution: Protobuf and Avro have built-in field versioning; JSON and MessagePack have no enforced contract, compatibility is a team convention, not a guarantee.
• Developer ergonomics: JSON is directly readable and debuggable with generic tools; Protobuf and Avro need generated code or a schema registry and are opaque on the wire; MessagePack sits in between, binary but with JSON's exact data model, so no schema to maintain but also no schema-based safety.

Definitions

• SLI (Service Level Indicator): the actual metric you measure, a number that describes how the system is behaving right now. It's the raw measurement, not a target.
• SLO (Service Level Objective): the internal target you set for an SLI, e.g. "99% of X should happen within Y." It's the goal, used to guide engineering priorities.
• SLA (Service Level Agreement): an external, often contractual commitment built on top of an SLO, usually with real consequences (credits, penalties) if missed. SLAs are typically set looser than the internal SLO so there's margin before a miss becomes a contractual breach.

Example for a streaming ingestion pipeline

• SLI: the percentage of ingested events whose time from arrival at the pipeline to being written into the data lake is 5 seconds or less, measured over a rolling window.

  text

  SLI = (# events with write_timestamp - arrival_timestamp <= 5s) / (total events) over the window
  

• SLO: 99% of events meet that 5-second delivery target, measured over a rolling 30-day window. The remaining 1% is the "error budget": how much lateness is tolerated before this counts as a miss.

How you'd measure and report it

Stamp each event (or each micro-batch, if events are processed in batches) with its arrival timestamp at ingestion, and record the timestamp when it's successfully written to the data lake. Compute the delta for every event, bucket those deltas into a histogram, and calculate what fraction fall under the 5-second threshold in each reporting window. This is typically implemented as a latency histogram metric (for example, a Prometheus histogram with a bucket boundary at 5 seconds) with a recording rule that continuously computes the rolling percentage, surfaced on a dashboard alongside the raw p50/p95/p99 delivery latency so you can see not just "are we meeting the SLO" but "how much margin do we have." Burn-rate alerting on top of that (how fast the 1% error budget is being consumed) lets you page before the SLO is actually breached rather than after.

Framing

At the 99.99th percentile (the worst 1-in-10,000 requests), a microservices architecture's tail is usually dominated by compounding rare events: one overloaded dependency letting requests queue up, a concurrency limit that lets too much work pile onto a service at once, or one slow dependency's slowness cascading into every service that calls it. Fixing this needs architectural changes at each of those three points, not just tuning one dial.

Queueing

Bound every queue: in-process work queues, connection pools, and any message queue between services. An unbounded queue means that once a downstream gets slow, requests pile up indefinitely, and by the time most of them are finally processed, the caller has already timed out anyway, so nothing was gained by queuing them that long, while the growing queue consumes memory that could have served fresh requests instead, making the situation worse. Pair bounded queues with load shedding: once a queue is full, reject new work immediately (a fast, cheap failure) rather than accepting it and letting it sit past the caller's own timeout (a slow, wasted failure that still costs resources).

Concurrency limits

Cap the number of in-flight requests a service will accept or process at once, either a static limit or an adaptive one that adjusts based on observed latency (conceptually similar to how TCP congestion control backs off under contention). Without a concurrency limit, added load doesn't fail fast, it just makes every in-flight request slower together, the queuing-theory "hockey stick" effect where latency grows sharply, not gradually, as utilization approaches saturation.

Cascading dependencies

• Timeouts and deadline propagation at every hop, so a slow downstream doesn't hold an upstream's thread or connection open indefinitely, and so the total time budget is enforced end to end instead of each hop independently allowing its own full timeout, which just sums up across the chain.
• Circuit breakers per dependency: once a dependency's failure or timeout rate crosses a threshold, stop calling it for a cooldown period (fail fast) instead of letting every caller individually discover it's unhealthy through a slow timeout.
• Bulkheading: isolate resource pools (thread pools, connection pools) per dependency, so one slow dependency exhausting its own pool can't also starve calls to a completely unrelated, healthy dependency, a common way one bad dependency ends up taking down an entire service.
• Hedged requests, used narrowly, for the specific hop found to be most responsible for the p99.99 spikes, where the call is cheap and idempotent; kept narrow because hedging adds real load system-wide and can worsen the very congestion it's meant to fix if applied too broadly.

Together these changes mean a single bad dependency degrades gracefully (a bounded, fast failure that the caller can handle) instead of cascading into a full outage that shows up as user-facing timeouts.

Signals for CPU-bound

• CPU utilization (user + system time) is high or near 100% across cores, and the run queue (threads ready to run but waiting for a free core) is longer than the number of available cores.
• iowait is low: the CPU isn't idle waiting on disk or network, it's genuinely busy computing.
• Throughput improves when you add cores or optimize the hot code path, but not when you just add more concurrent connections.

Signals for I/O-bound

• iowait is elevated: the CPU is frequently idle while waiting on disk or network operations to complete.
• Disk or network queue depth is high, or syscall tracing shows a large share of time inside read/write/connect calls.
• Requests are slow even though CPU utilization looks unremarkable or even low.

Tools and metrics to check

• top / vmstat: the us (user), sy (system), and id (idle) columns give a quick CPU-vs-idle split; mpstat -P ALL breaks this down per core, useful when one core is saturated while others are idle (a concurrency/parallelism problem, not a raw compute problem).
• iostat -x: disk %util and average wait time per I/O operation.
• Network metrics: throughput, retransmits, connection resets, which signal network-level stalls rather than local disk I/O.
• strace -c: a summary of time spent per syscall, useful to see whether a process is dominated by I/O syscalls versus CPU-heavy work.
• Flamegraphs and other CPU profiling tools can confirm which specific code is consuming CPU once you already know you're CPU-bound; that deeper profiling workflow is its own topic, here the goal is just to establish CPU-bound vs I/O-bound first.

Actions for each diagnosis

• CPU-bound: optimize the hot code path (an algorithmic or serialization-format improvement usually beats just adding hardware), or scale out horizontally if the workload is legitimately compute-heavy and correctly provisioned; batching or deduplicating repeated work also reduces total CPU spent per unit of useful output.
• I/O-bound: increase concurrency so other work can proceed while one request waits on I/O (as long as the downstream isn't already saturated), add caching to avoid the I/O entirely for repeat requests, tune connection pooling and keep-alive to cut reconnect overhead, and check whether the I/O itself (disk, network, or a downstream dependency) is saturated, in which case the fix is capacity or dependency-side, not something you can code around locally.
• Mixed signals (moderate CPU and moderate iowait together) often point somewhere else entirely, like lock contention or garbage collection, and are worth investigating as their own category rather than forcing them into a CPU-vs-I/O split.