Latency Analysis & Optimization Questions

Governance policy

• Every customer-facing service publishes a latency SLO (service level objective: a target for a measured metric, such as "p99 under 300ms," that the owning team is held to), tied to a documented service level indicator, SLI (the exact metric and measurement point used to judge it), reviewed on a regular cadence against actual user impact rather than set once and forgotten.
• SLOs are set collaboratively with the owning team, anchored to what actually matters to users or downstream consumers, not mandated top-down with no input, and revisited whenever the service's role changes.
• New features or dependencies expected to sit on a service's critical path include an SLO-impact estimate before merge, so risk is surfaced before it ships, not discovered afterward.

Measurement tooling

• One shared metrics and tracing stack across teams, not each team building its own, so SLO compliance is comparable and centrally dashboarded; per-team dashboards roll up into one company-wide view for leadership.
• Error budgets, the amount of unreliability an SLO allows before it is breached over a window (for example, a 99.9% SLO over 30 days allows roughly 0.1% of requests to miss target before the budget is exhausted), computed automatically from the SLO and the real measured SLI, visible to the team and stakeholders, not manually reported.

Enforcement mechanisms

• Release gates: a deployment pipeline check that blocks a release, or requires an explicit, logged override, if the service's error budget is already exhausted. This is the standard error-budget policy: once the budget is burned, non-essential feature releases pause until it recovers, and reliability work takes priority.
• Guardrails at review time: any design or code review touching a service's critical path includes, as a checklist item, whether the change could plausibly move p99, not just correctness review.
• A lightweight escalation path: shipping something known to breach an SLO requires an explicit, logged decision from the team's lead, never a silent bypass.

Using error budgets to prioritize work

• With budget remaining, feature velocity is unconstrained. As the budget nears exhaustion, reliability work, the fixes causing the burn, gets prioritized over new features automatically. This is the core mechanism that balances shipping speed and reliability without a human renegotiating it every sprint.
• Track burn rate, not just remaining budget: a team burning its 30-day budget in 2 days needs an immediate response, while one burning it steadily over 25 days can likely finish the quarter's roadmap first.

Handling teams that repeatedly miss their SLOs

• The first response is investigative, not punitive: is the SLO itself unrealistic for what the service does, or is this a real reliability gap? Renegotiate the SLO if it was set wrong.
• If it's a real gap, require a documented remediation plan with a timeline, reviewed by the governance body (often an SRE or platform team), and freeze non-essential feature work on that service until the plan is executed, mirroring the error-budget release gate but made explicit for a chronic offender.
• Track repeat offenders as an organizational signal: a service missing its SLO quarter after quarter often points to a staffing, architecture, or prioritization problem above the team's control, and that needs to be escalated up, not just enforced down onto the team.

Plausible hidden causes

Instrumented spans only measure the time a library explicitly wrapped. Roughly 700ms is unaccounted for, which points at delay happening outside what any span is timing:

1. Queuing before the traced work even starts. Time spent in a listen/accept queue, a load balancer queue, or a thread-pool/executor queue before a handler begins running is invisible to a span that only starts its clock once the handler executes.
2. CPU scheduling delay (off-CPU time). A thread can be runnable but not actually running because the CPU is busy with something else, another tenant, garbage collection, or kernel work. A span reports wall-clock duration, but if the underlying instrumentation doesn't distinguish "doing work" from "waiting to be scheduled," a big gap between a span's reported CPU time and its wall-clock time is exactly this.
3. Blocking calls that happen outside the traced boundary. DNS resolution, a TLS handshake, or waiting to check out a connection from a pool typically happen before an RPC client library's span even starts, so they're silently absorbed into the gap.
4. Garbage collection or other stop-the-world pauses on any of the hosts involved, which stall the whole process regardless of which span happens to be "active" at that moment.
5. Retries not reflected proportionally. A span might report only the final, successful attempt's duration, hiding one or two earlier failed or slow attempts that also consumed wall-clock time.

Three experiments/probes to surface the missing delay

1. Instrument the queueing boundary explicitly. Add a timestamp at the earliest possible point (e.g., when the load balancer or reverse proxy accepts the connection, or when a request enters the process) and compare it against when the handler's own span actually starts. This directly measures accept-to-dispatch queuing delay that current spans skip entirely.
2. Kernel-level off-CPU analysis. Use strace -T -f on the process to see per-syscall wait time (a lot of time in futex calls means lock contention; time in connect/read means I/O waits), or an eBPF-based tool (eBPF lets you attach lightweight tracing/profiling programs to running kernel events without changing kernel code) (like an off-CPU profiler) to directly measure how long threads sat runnable but not scheduled on a core. Checking perf sched or the kernel's scheduler run-queue stats tells you whether the host itself is CPU-oversubscribed.
3. Instrument the client library's connection lifecycle. Turn on debug metrics or logging for "time spent acquiring a connection from the pool," and separately time DNS resolution and the TLS handshake. A cold client, or one whose pooled connections keep expiring, can pay these costs on every request without any of it showing up as a traced RPC span.

Running all three against the same time window narrows which bucket (pre-dispatch queuing, CPU scheduling, or connection-lifecycle overhead) actually accounts for most of the missing ~700ms, rather than guessing.

Framing

The target is p99 (the 99th percentile: the latency value only the slowest 1% of requests exceed) read latency under 10ms for users in three major regions, 100M monthly users, and 100k RPS at peak. A single cross-region network round trip alone (order of 70 to 100ms between distant continents) is already 7 to 10 times the entire budget, so the core design principle is: a read must never cross a region to succeed. Everything below exists to make that true almost all the time.

Data partitioning

Partition by a key that ties data to a "home region," typically derived from the user's or entity's registration location (e.g., account created in APAC gets a home region of the nearest of the three). Within a region, shard further by a hash of the entity ID across the regional database cluster so no single shard becomes a hotspot. This keeps the vast majority of a user's reads and writes local to one region instead of needing a global lookup.

Replication strategy

Each region owns writes for its local users (active-active across regions, not a single global leader) and asynchronously replicates every write to the other two regions as read-only followers. This gives every region a nearly-complete, read-only copy of the global dataset with a bounded replication lag target (documented, not assumed, e.g., aim for under one second under normal load), so a regional outage or a user traveling can still be served a recent, if slightly stale, read from another region rather than failing outright.

Read path

Client to nearest edge/regional point of presence, then a local in-memory cache, then a local read replica on a cache miss. No hop ever leaves the region for a successful read. A cache hit should land in the low single-digit milliseconds; a cache miss falling through to the local replica adds the local database's own read latency, still well inside the 10ms budget because everything stayed in-region.

Write path

Writes route to the entity's home region, apply to that region's leader, then replicate asynchronously outward. A user traveling away from their home region still writes there (extra latency on writes is an accepted trade-off in exchange for keeping the far more frequent reads fast and local).

Cache hierarchy

• L1: per-instance in-process cache for the hottest keys, sub-millisecond, zero network hop.
• L2: a shared regional cache cluster (e.g., Redis), same-AZ (Availability Zone, an isolated data-center location within a cloud region) round trip, generally under 1ms.
• L3: the regional read replica, the fallback on a cache miss.

Each tier exists so a request that misses the fastest tier still has a good chance of resolving locally rather than falling all the way through to a cross-region hop.

DNS routing

Use latency-based or anycast (routing that sends a request to whichever server location is nearest, via one shared IP address advertised from multiple sites) routing so each client resolves to its nearest healthy region automatically, backed by active health checks so an unhealthy region is pulled out of rotation. Keep the DNS TTL (Time To Live: how long a client caches a DNS answer before re-checking it) low enough to fail over within roughly a minute of a regional problem, but not so low that it causes excessive re-resolution overhead under normal operation; this is itself a trade-off between failover speed and steady-state DNS load.

Consistency model

Strong, read-your-writes consistency WITHIN a region (a user's own writes are visible on their next read against the same region's leader or a synchronously-replicated in-region follower). Across regions, the model is eventual, bounded-staleness consistency: a follower in another region may lag behind the true leader by up to the replication-lag target. This is a deliberate trade-off, not an oversight: enforcing global strong consistency would require a cross-region consensus round trip on every read, which alone costs 10x the entire latency budget.

Measuring and enforcing regional SLOs (Service Level Objectives, the internally agreed latency targets)

Instrument server-measured latency per region as a full distribution (p50/p95/p99/p999), never an average, and never blended across regions, a fast region and a slow region averaged together hides the slow one. Run synthetic canary requests from within each region against each region's own endpoints on a fixed schedule so a regression is caught before it shows up in real user traffic. Treat the 10ms p99 target as an error budget: track how much of it is being consumed and alert on the burn rate, not just on a single breach, so a slow but not-yet-failing trend gets caught early.

Trade-off called out explicitly

Active-active multi-region increases operational complexity: each region must carry enough spare capacity to absorb a failed-over neighbor's traffic (not just its own 1/3 share of 100k RPS), and cross-region replication lag has to be monitored as its own signal, since a silent increase in lag degrades the "how stale can a failover read be" guarantee without ever showing up as a latency breach.

The formats

• JSON (JavaScript Object Notation): text-based, human-readable, self-describing (field names travel with every record), no schema required to parse.
• Protobuf (Protocol Buffers): binary, schema-defined via a .proto file with numbered fields, compact and fast to encode/decode since field names aren't repeated in every record.
• MessagePack: binary, with essentially the same flexible, schema-less data model as JSON (maps, arrays, strings, numbers) but binary-encoded, so it's a fast drop-in for JSON without requiring a schema.
• Avro: binary, schema-defined, but the schema travels with the data (or via a shared schema registry) rather than being baked into generated code, which makes it well suited to systems where the schema evolves independently of any one consumer, common in data-engineering pipelines.

Impact on latency and throughput

Binary formats generally cost less CPU to encode and decode than JSON's text parsing, and they produce smaller payloads since numbers and repeated field names aren't spelled out as text. Both effects matter directly: less CPU time is less time on the request's critical path, and a smaller payload means less time spent transferring it over the network, which compounds at high request rates or with large payloads. JSON's overhead is usually a non-issue for low-volume or human-facing APIs, where readability and the ability to curl and eyeball a response are worth more than the CPU/network savings.

Worked example

Take one small event record: {"user_id": 48213, "event": "click", "ts": 1699999999}. Encoded compactly:

JSON:        49 bytes  (spells out "user_id", "event", "ts" as literal text every time)
MessagePack: 32 bytes  (binary, same schema-less model as JSON)
Protobuf:    17 bytes  (binary, numbered field tags instead of field names, schema-defined)
Avro:        14 bytes  (binary, no per-field tags at all, relies entirely on the shared schema)

JSON is roughly 2.9x the size of Protobuf and 3.5x the size of Avro here, because it repeats the field names on every single record; the binary formats spend zero bytes on that, and Avro saves a further two bytes over Protobuf by dropping per-field tags too, relying entirely on the shared schema for field identity. On the CPU side, a quick decode-time comparison (200,000 iterations in Python) showed JSON's parser taking about 830ns per record versus MessagePack's about 210ns, roughly a 4x gap from skipping text tokenization; the exact multiplier varies by language and library, but the direction (binary formats decode faster) holds broadly.

When to change format

Switch when serialization or payload size shows up as a real, measured chunk of your critical path, visible as CPU time in a profiler or a trace span, or as a meaningful fraction of your bandwidth budget, not preemptively. Also consider switching when you need first-class schema evolution across many independently-deployed services (Protobuf and Avro both support adding or deprecating fields safely via field numbering or a schema registry), which JSON and MessagePack don't provide on their own.

Trade-offs

• CPU cost: binary formats are cheaper to parse; JSON's text parsing has real, measurable overhead at high volume.
• Payload size: binary formats are smaller (no repeated field names, compact number encoding); this maps directly to network transfer time.
• Schema evolution: Protobuf and Avro have built-in field versioning; JSON and MessagePack have no enforced contract, compatibility is a team convention, not a guarantee.
• Developer ergonomics: JSON is directly readable and debuggable with generic tools; Protobuf and Avro need generated code or a schema registry and are opaque on the wire; MessagePack sits in between, binary but with JSON's exact data model, so no schema to maintain but also no schema-based safety.

Diagnose the shape of the problem

0.1% of keys driving 80% of reads is an extreme power-law (Zipfian) skew. The important detail is where those keys physically live: if the cache uses simple consistent hashing (a scheme that maps each key to a point on a hash ring, so a given key always lands on the same shard node), a handful of hot keys can pin themselves to one or two of many shard nodes, so those specific nodes are overloaded even though the cluster-wide average utilization looks completely healthy. Confirm this by mapping the sampled per-key request counts to shard/node ownership before proposing a fix.

Plan

1. Local (near) cache in the client with a short TTL (time-to-live, how long a cached value is trusted before re-fetching), for example 200-500ms. This bounds how often the hot key is even asked of the shared cache: with a 300ms TTL, a shared near-cache tier needs to refresh a given key at most once every 300ms (about 3 refreshes/sec) no matter how many reads arrive for it, regardless of whether that demand is 1,000 or 100,000 reads/sec.
2. Key-level replication for identified hot keys: store the value under N suffixed keys (key:0 ... key:N-1) on N different nodes, and have clients pick a replica at random or round-robin. This spreads the same 80% of reads across N nodes instead of concentrating them on one.
3. Request coalescing (single-flight): when a hot key misses, ensure only one in-flight fetch to the backend happens per key per node; concurrent requests for that key wait on the one fetch in progress instead of each triggering a separate origin call. This specifically protects against a stampede when a hot key's TTL expires.
4. If the hot-key population is small and known in advance, consider a dedicated cache tier or extra replicas just for it, so a spike there cannot degrade the shared cluster.

Trade-offs

Local caching trades a bounded staleness window for load reduction. Key replication costs extra memory and write amplification (every write to a hot key must fan out to N replicas) in exchange for read scalability. Coalescing adds a small wait for the "unlucky" requests behind an in-flight fetch, but caps backend load during a burst.

What to monitor to confirm it worked

Per-key (or per-key-bucket) request rate and per-node CPU/network, to confirm the skew moved from concentrated to spread. The p99 latency specifically on the previously-hot keys, not just the cluster aggregate, since an aggregate p99 can look fine while one shard is still degraded. Cache hit ratio and the actual observed staleness vs. the TTL you set. If you replicated keys, the write-amplification rate those replicas now cost.