Latency Analysis & Optimization Questions

The formats

• JSON (JavaScript Object Notation): text-based, human-readable, self-describing (field names travel with every record), no schema required to parse.
• Protobuf (Protocol Buffers): binary, schema-defined via a .proto file with numbered fields, compact and fast to encode/decode since field names aren't repeated in every record.
• MessagePack: binary, with essentially the same flexible, schema-less data model as JSON (maps, arrays, strings, numbers) but binary-encoded, so it's a fast drop-in for JSON without requiring a schema.
• Avro: binary, schema-defined, but the schema travels with the data (or via a shared schema registry) rather than being baked into generated code, which makes it well suited to systems where the schema evolves independently of any one consumer, common in data-engineering pipelines.

Impact on latency and throughput

Binary formats generally cost less CPU to encode and decode than JSON's text parsing, and they produce smaller payloads since numbers and repeated field names aren't spelled out as text. Both effects matter directly: less CPU time is less time on the request's critical path, and a smaller payload means less time spent transferring it over the network, which compounds at high request rates or with large payloads. JSON's overhead is usually a non-issue for low-volume or human-facing APIs, where readability and the ability to curl and eyeball a response are worth more than the CPU/network savings.

Worked example

Take one small event record: {"user_id": 48213, "event": "click", "ts": 1699999999}. Encoded compactly:

JSON:        49 bytes  (spells out "user_id", "event", "ts" as literal text every time)
MessagePack: 32 bytes  (binary, same schema-less model as JSON)
Protobuf:    17 bytes  (binary, numbered field tags instead of field names, schema-defined)
Avro:        14 bytes  (binary, no per-field tags at all, relies entirely on the shared schema)

JSON is roughly 2.9x the size of Protobuf and 3.5x the size of Avro here, because it repeats the field names on every single record; the binary formats spend zero bytes on that, and Avro saves a further two bytes over Protobuf by dropping per-field tags too, relying entirely on the shared schema for field identity. On the CPU side, a quick decode-time comparison (200,000 iterations in Python) showed JSON's parser taking about 830ns per record versus MessagePack's about 210ns, roughly a 4x gap from skipping text tokenization; the exact multiplier varies by language and library, but the direction (binary formats decode faster) holds broadly.

When to change format

Switch when serialization or payload size shows up as a real, measured chunk of your critical path, visible as CPU time in a profiler or a trace span, or as a meaningful fraction of your bandwidth budget, not preemptively. Also consider switching when you need first-class schema evolution across many independently-deployed services (Protobuf and Avro both support adding or deprecating fields safely via field numbering or a schema registry), which JSON and MessagePack don't provide on their own.

Trade-offs

• CPU cost: binary formats are cheaper to parse; JSON's text parsing has real, measurable overhead at high volume.
• Payload size: binary formats are smaller (no repeated field names, compact number encoding); this maps directly to network transfer time.
• Schema evolution: Protobuf and Avro have built-in field versioning; JSON and MessagePack have no enforced contract, compatibility is a team convention, not a guarantee.
• Developer ergonomics: JSON is directly readable and debuggable with generic tools; Protobuf and Avro need generated code or a schema registry and are opaque on the wire; MessagePack sits in between, binary but with JSON's exact data model, so no schema to maintain but also no schema-based safety.

What a CDN is

A CDN (Content Delivery Network) is a geographically distributed network of caching servers, often called edge servers or points of presence (PoPs), positioned physically closer to end users than your origin server. Instead of every request traveling all the way back to wherever your origin server actually lives, an edge server nearby can answer directly from a cached copy.

How it reduces latency

For static or cacheable assets like images, JavaScript/CSS bundles, and video, the nearest edge server serves a cached copy instead of the request crossing potentially thousands of kilometers to the origin. That directly cuts network round-trip time: light in fiber travels at roughly two-thirds the speed of light in a vacuum, so a cross-continent round trip alone can cost tens to over a hundred milliseconds before any processing even happens. Shrinking "user to origin" down to "user to nearest PoP" removes most of that physical distance. As a side benefit, it also offloads the origin server, since it no longer has to serve every request for that asset itself.

Where a CDN can hurt performance or correctness

• Dynamic or personalized content accidentally made publicly cacheable. If a per-user response is missing the right cache-control header (for example, forgetting Cache-Control: private), the CDN may cache and serve one user's personalized data to a different user entirely, a correctness bug, not just a performance one.
• A TTL (Time To Live, how long a cached copy is considered valid) set too long for content that changes often. Users keep seeing stale data until the TTL expires or someone manually purges the cache.
• Slow or incomplete cache purging after a deploy. If the CDN cache isn't invalidated (or the purge takes time to propagate to every edge location), users can end up with a new HTML page referencing an asset version that the CDN is still serving the old copy of, causing broken or mismatched behavior.
• Cache-control misconfiguration in the other direction, marking genuinely cacheable content as no-store, which throws away a free latency win for no reason.
• Content that's inherently unique per request (like an authenticated API response with no meaningful cache hit rate) gains an extra network hop through the CDN for essentially zero benefit, and can end up slightly slower than calling the origin directly.

Instrumentation

Add both client-side real user monitoring (RUM, measuring what the user actually experiences, including network and rendering, not just server time) and a server-side span breakdown of the login call chain (credential check, database lookup, token issuance, any downstream calls). Tag every measurement with client platform, geography, and (once testing starts) experiment bucket, so a regression or improvement can be attributed to a segment rather than only seen in aggregate.

A/B testing

Hold back a control group on the current login path and ship each candidate optimization to a treatment group, measuring both latency (p50/p95/p99) and the actual conversion metric (login completion rate, or downstream signup/purchase completion) with statistical significance, run long enough to smooth out day-of-week and time-of-day effects rather than reading a few hours of data.

Prioritizing fixes

Use the trace breakdown to find which single hop contributes most to the p99, and start there: an uncached lookup or a serial call that could be parallelized is usually a quick win that produces fast, low-risk signal. Reserve architectural changes (moving the auth check closer to the edge, adding a dedicated cache tier, restructuring the call chain) for after the quick wins are exhausted, since they take longer to build and are riskier to roll back.

Rollout and rollback

Ship behind a feature flag with a staged rollout (for example 1% then 10% then 50% then 100% of traffic), with automatic rollback triggers tied to BOTH latency and conversion regressions, not errors alone, since a change can be error-free and still slow down or hurt conversion.

Quantifying business impact

It's well established across the industry that added latency correlates with reduced conversion, but the actual sensitivity (how many conversions are gained per millisecond improved) is specific to this product and this funnel, so it must be measured from your own A/B data, not assumed from a general rule of thumb. From the experiment data, bucket users by observed login latency (or by control vs. treatment) and compare conversion rate across buckets; the resulting curve is your own quantified relationship, which is what should drive whether the next optimization is worth the engineering cost.

Defining latency and throughput

• Latency: the time a single request takes to complete, start to finish (e.g., "this endpoint responds in 120ms").
• Throughput: how much work the system handles per unit of time (e.g., "this API handles 5,000 requests per second").

They're related but not the same thing, and a change can move them in opposite directions. Batching is the clearest example: waiting to accumulate a bigger batch before processing raises the latency of any individual item (it has to wait for the batch to fill), but it can raise overall throughput since more items get processed per unit of overhead.

Design decisions that mainly improve latency

Caching, reducing the number of network round trips, keeping connections warm (connection pooling/keep-alive so you're not paying a fresh TCP/TLS handshake per request), running independent calls in parallel instead of one after another, shrinking payload size, placing content closer to users (CDN/edge), and hedged requests for latency-critical calls.

Design decisions that mainly improve throughput

Batching, horizontal scaling (more instances processing work concurrently), asynchronous/queue-based processing that decouples accepting work from finishing it, connection pooling sized to maximize concurrent in-flight work, and more efficient serialization that uses less CPU per request (raising the requests-per-second ceiling for the same hardware). Some changes, like removing an N+1 query pattern (issuing one query to fetch a list, then a separate query per item in that list, instead of one batched query), tend to help both at once.

Prioritizing when page loads are slow but CPU utilization is low

Low CPU utilization while requests are still slow is a strong signal the bottleneck isn't compute, it's waiting: on a slow downstream call or database, on the network (sequential round trips that could run in parallel, or an asset with no CDN in front of it), or on some form of queuing or contention (waiting for a connection-pool slot, a lock, or a downstream's own tail latency) that doesn't show up as CPU burn at all.

Given that, the priority should NOT be "add more CPU or more instances," since low CPU utilization tells you compute capacity isn't the constraint. Instead: trace the request end to end to see exactly where the wall-clock time goes, check whether dependent calls that could run in parallel are instead being made sequentially, check the downstream services' own reported latency rather than assuming the problem is local, and check for time spent waiting on a connection pool or queue rather than actually doing work.

Reconstructing the picture

Summing the named leaf spans: A to B network 0.1s, B's internal CPU work 0.5s, B to C network 0.05s, B waiting on the database 0.25s, and C's own total 0.1s, comes to 1.0s. The trace's stated end-to-end total is 1.2s, so 0.2s (about 17% of the total) is unaccounted for by any labeled span. That gap matters as much as any single named span: it's most likely the return-trip network time (B's response back to A, or C's response back to B) that this trace simply never instrumented, a common real-world gap when only outbound calls get their own spans.

Where the largest labeled portion of time is spent

Among the named spans, B's internal CPU work at 0.5s is the single largest contributor, roughly 42% of the total 1.2s. B's own wait-on-database time (0.25s, about 21%) is the second largest. Together, B's own span (0.8s total: CPU plus network to C plus DB wait) accounts for two-thirds of the whole request, making B the dominant hop regardless of how the untracked 0.2s gap is eventually explained.

Initial remediation steps

1. Investigate B's CPU work first, since it's the largest known, already-instrumented cost: profile that 0.5s for expensive serialization, an inefficient algorithm, lock contention, or garbage-collection pauses, any of which would show up as CPU time in a span without more detail than "CPU work."
2. Look at the 0.25s DB wait next: check the query plan, whether an index is being used, and whether that time is queuing (waiting for a connection or for the database to be free) rather than the query itself running slowly, since those have different fixes.
3. Close the instrumentation gap: add explicit spans for the return-trip legs (B to A, C to B) so the next trace accounts for the full 1.2s. Without that, any fix to B's CPU or DB wait risks leaving an unexplained 0.2s in the next trace too, and it's impossible to tell from this trace alone whether that gap is normal network return time or a second hidden problem.
4. Check for clock skew across the A, B, and C hosts, since a trace assembled from independently-clocked spans can show artifacts (small negative or inflated gaps) that look like missing time but are actually measurement error, not real latency.