Performance Troubleshooting & Incident Response Questions

Direct answer: Client-side retries generating request storms is retry amplification: a slow or failing dependency causes a client to retry, which adds more load onto an already-struggling backend, making it slower still and causing even more timeouts and retries, a feedback loop that can cascade into a full outage. The short-term fix is to cap or slow retries; the long-term fix is to make the whole call chain backpressure-aware so a slowdown can't turn into a spiral in the first place.

Structured elaboration:

• Short-term mitigations:
  • Cap total retry volume with a retry budget (for example, retries may not exceed 10% of the original non-retry request volume). State the denominator explicitly when you write the policy down, because "10% of total traffic", "10% of original traffic" and "10% of successful traffic" are three different ceilings and the first one is self-referential, since retries are part of total traffic. A budget breaks the amplification loop immediately because it is a hard cap rather than a slowdown.
  • Shed load with a fast, explicit failure (a 503 response with a Retry-After header) instead of letting requests queue indefinitely, so clients get a clear back-off signal instead of timing out and retrying blindly.
  • Add a circuit breaker (a safeguard that stops sending requests to a struggling dependency once its failure rate crosses a threshold) in front of the struggling dependency: after a threshold of failures, stop sending it traffic for a cooldown window instead of continuing to hammer it.
• Long-term design changes:
  • Retry policy: exponential backoff with jitter (randomizing the delay so many clients don't retry in synchronized waves) plus a bounded retry budget per client and per fleet.
  • Protocol: favor idempotent operations (retrying one produces the same end result as running it once, so a client can safely retry without causing duplicate effects like a double charge), and propagate a deadline end to end through the call chain, so a downstream hop knows how much time is left and stops doing work the caller has already given up on.
  • Backpressure: admission control at the backend that rejects or sheds new work once its queue depth crosses a threshold, rather than accepting everything and degrading uniformly, paired with a client-side rate limiter that adapts to observed backend health.

Worked example: Baseline traffic is 1000 requests per second (rps), normally handled fine at p99 200ms. The backend degrades slightly (a downstream database hiccup), so 5% of requests, 50 rps, now exceed the client's 300ms timeout. With a naive "retry immediately up to 3 times" policy, those 50 rps of failures generate up to 50 times 3 = 150 extra rps of retries, pushing total load from 1000 rps to 1150 rps, a 15% increase exactly when capacity is already reduced. If the backend degrades further under that added load, say the failure rate doubles to 10% of everything it is asked to serve, do the accounting as a steady state rather than by stacking rounds, because the 150 rps of retries are already inside the 1150 and a second round computed on top of that total counts them twice. Let A be the total attempts per second the backend actually sees. Real user traffic is still 1000 rps and never changed, and every failed attempt spawns 3 retries, so A = 1000 + 3 x 0.10 x A, which rearranges to A = 1000 / (1 - 0.30) = about 1430 rps, roughly 43% above baseline from a user load that is completely flat. Writing it as a fixed point rather than as rounds is what makes it useful under pressure, because it tells you where the spiral stops being a spiral and becomes an outage: A = 1000 / (1 - 3p) has no finite solution once 3p reaches 1, so with a 3-retry policy any sustained failure rate at or above about 33% makes offered load diverge and the service cannot climb out on its own no matter how long you wait. That threshold, not the size of any one round, is what a retry budget exists to keep you away from. Capping retries at 10% of the original non-retry volume holds the worst case at 1100 rps regardless of how bad the failure rate gets, instead of letting it settle at 1430 rps or run away entirely.

Trade-offs and pitfalls: eliminating retries entirely trades away real resilience for amplification safety, since some transient errors genuinely should be retried; the fix is bounding and randomizing retries, not banning them. Backpressure done badly, a hard reject with no guidance, can look like an outage to callers that don't handle a 503 gracefully, so it needs to be paired with a documented client contract for what to do when throttled.

Instrument the request path so each cache layer's lookup is individually timed and tagged, then attribute a slow request to whichever layer's timing span accounts for most of the elapsed time, instead of guessing from the overall request latency alone.

Instrumentation

• Wrap each layer's lookup (in-process memory check, local disk cache check, remote object store fetch) in its own timed span using a distributed tracing library, tagged with the layer, hit or miss, and the size of data returned.
• Attach these spans to the request's trace ID so a single slow request can be reconstructed end to end: how much time was spent checking memory, then disk (if memory missed), then the remote store (if disk missed too).
• Emit continuous per-layer histograms of lookup latency and hit rate, not just averages, so you can see p50 and p99 (50th and 99th percentile) per layer independently rather than only for the whole request.

Experiment design to isolate the cause

• Correlate spike timing against each layer's own metrics: does a spike line up with a drop in in-process cache hit rate (memory eviction or a cold cache), a rise in disk read latency (contention with another process for disk IO), or a rise in remote object store latency (network or the store's own load)?
• Deliberately manipulate one layer at a time in a controlled test, for example force a cold in-process cache by restarting the process, or throttle disk IO artificially, and see which manipulation reproduces the request-level spike. That layer is the culprit.
• Segment spiking requests by shape: are they disproportionately for large objects (more likely to fall through to the slower remote store on a miss), or concentrated on a specific key range (a hot key or an uneven shard on one store)?

Sampling strategy to keep overhead acceptable

• Head-based sampling for the common case: fully instrument a small fixed slice of all requests, for example 1 in 100, low enough to add negligible overhead to normal traffic.
• Always-on capture for the interesting case: fully instrument any request whose total latency exceeds a threshold near your current p99 baseline, since those are exactly the requests you care about and they're rare enough that capturing all of them doesn't add meaningful load.
• Keep the cheap per-layer histograms running at 100% of traffic as counters (not full traces), since counters cost far less than spans and are your early-warning signal even without a slow request in front of you.

Worked example: with always-on capture for every request over 200ms, a captured request comes in at 215ms total, and the discipline that makes per-layer attribution worth anything is that the 215ms has to add up. It breaks down as 0.2ms in the in-process memory lookup (a miss), 4.8ms in the local disk cache lookup (also a miss), 180ms inside the remote object store span, and 30ms of non-cache work, deserializing the fetched object plus the request handling on either side of the cache chain. Always reconcile that sum against the total, because the leftover is itself diagnostic: 30ms of ordinary application work is unremarkable, but if the spans had summed to 120ms of a 215ms request you would be looking at 95ms of missing instrumentation, not an answer, and the honest next step would be to instrument the gap rather than blame the slowest span you happen to have. Here the remote store is about 84% of the request, so the attribution is unambiguous. Correlating with the remote store's own metrics shows those spikes line up with periods when its p99 read latency rose, matching a known noisy-neighbor pattern (another tenant or workload sharing that same store, competing for the same resources and slowing it down for everyone else) on that shared store. The fix direction becomes populating the disk cache layer more aggressively for that key range to reduce dependence on the remote store, rather than tuning the in-process cache, which was never the bottleneck.

Trade-offs and pitfalls: sampling too sparsely, say 1 in 10,000, risks missing an intermittent issue that only affects a small fraction of requests entirely, which is why always-on capture for slow requests specifically matters, not random sampling alone. Per-layer instrumentation also adds real overhead if implemented naively, for example synchronous logging on the hot path, so span emission itself should be asynchronous and buffered so the instrumentation doesn't become part of the problem it's measuring.

Prove causation by overlaying the exact timestamps of the read latency spikes against the backup job's schedule across several nights, then confirm it with a resource-contention signal specific to the backup, not just "it happens at night."

Proving causation

• Overlay the read-latency graph and the backup job's start and end times over at least 3 to 5 nights to confirm the pattern repeats and the timing genuinely lines up, not just once.
• During a backup window, capture disk IO metrics with iostat -x 1, looking at IO utilization percentage and average wait time per disk, to see whether the backup is saturating IO bandwidth at the same time reads slow down.
• Check the database's own wait-event view during the backup window (most relational databases expose one, for example a wait-events view showing what queries are blocked on) to see whether reads are waiting on IO specifically, or on a lock the backup process holds.
• Run a controlled test: if it's safe, pause or reschedule the backup for one night and confirm read latency stays flat. That single test is the strongest evidence you can produce.

Short-term mitigations

• Move the backup to a genuinely off-peak window if it currently overlaps real traffic, or throttle its IO rate (most backup tools support an IO throttle or nice-style priority flag) so it competes less aggressively for disk.
• Run the backup against a read replica instead of the primary if one exists, removing the IO contention with production reads entirely.
• Temporarily reduce backup scope, for example incremental instead of full, if a full backup is the immediate driver of the IO spike.

Long-term fixes

• Stand up a dedicated replica specifically for backups so backup IO is permanently isolated from serving traffic.
• Move to a backup method that avoids heavy sequential IO against the live data files, such as storage-level snapshots, if the storage layer supports point-in-time snapshots, instead of a logical dump.
• Provision separate or higher-tier IO capacity if the underlying disk is genuinely undersized for concurrent backup and serving load.

Worked example: latency graphs for 4 consecutive nights show p95 read latency (95th percentile: 95% of read requests are faster than this number, so it captures the slow tail an average would hide) rising from 8ms to 60ms exactly during the 1:00 to 1:45 AM backup window each night. iostat -x 1 captured during that window shows disk utilization at 98% versus a normal 30%, and the database's wait-event view shows read queries spending most of their time in an IO-wait state rather than actually executing. Moving the backup to run against a read replica instead of the primary removes the correlation entirely on the following night's test.

Trade-offs and pitfalls: throttling the backup's IO extends how long it takes to complete, which can push it into business hours if you don't leave headroom, so any throttle needs to be checked against total backup duration. And confirming correlation over a single night isn't enough, one night could coincide with an unrelated cause, like a batch job that happens to run at the same time.

Direct answer: Rising memory doesn't automatically mean a leak; it could be a cache warming up and plateauing, heap fragmentation, or a genuine leak where objects are retained past when they should be freed. Confirm which one it is by comparing object growth to traffic rather than watching total memory alone, and keep the service alive in the meantime without a full restart by draining and restarting instances one at a time if you're running more than one, or capping the specific growing structure if you're not.

Structured elaboration:

• Distinguish leak from legitimate growth: does memory keep climbing even during a low-traffic window (points to a leak, for example a background timer or job), or does it track request volume and then plateau (points to cache warm-up, not a leak, though the cache may still need a size cap)?
• Get object-level evidence without a blocking pause: prefer a live sampling profiler (RSS: resident set size, the actual physical memory a process is using, isn't precise enough on its own) over a full blocking heap dump. A full heap dump on a large, already-struggling heap can itself cause a multi-second pause, which risks dropping the in-flight work you're trying to protect. Take two snapshots several minutes apart under similar load and diff object counts by type: an object type that keeps growing well past what live request volume could ever require is the smoking gun for a leak.
• Keep it running safely without a full restart:
  • Set an alerting threshold well below the OOM (out-of-memory kill) point so you have runway before the process is killed for you.
  • If a specific structure is identified as the source and there's an admin endpoint or feature flag to bound or clear it, use that instead of restarting the whole process.
  • If the service runs multiple replicas, do a rolling restart, one instance at a time with connections drained first, rather than an all-at-once restart; that avoids dropping in-flight work or losing the warm cache for all traffic simultaneously, at the cost of a cold cache on just the one instance being cycled.
  • If it's a single instance with no way to drain traffic, the safest interim step is capping or disabling the specific growing structure through configuration while you keep collecting evidence, and plan a maintenance-window restart once you actually understand the cause.

Worked example: RSS grows from 1.2GB to 4.8GB over six hours while request volume stays roughly flat, so there is about 3.6GB of growth to account for. Two heap histograms taken 30 minutes apart under similar load show one object type, call it PendingRequestContext, growing from about 350,000 to about 383,000 instances, while the legitimate warm-cache object type stays flat around 50,000 entries. Now do the cross-check rather than stopping at "that number went up", because a growing object count on its own is a suspect, not a conviction: 33,000 new contexts in 30 minutes is about 1,100 a minute, which across the six hours of observed RSS growth comes to roughly 400,000 contexts, and 3.6GB spread over 400,000 objects is about 9KB retained each, the right order of magnitude for a request context holding a buffer and a parsed body. The rate, the standing count, and the RSS growth all agree, and that agreement is what turns this into a confirmed diagnosis. Had they disagreed, for example if the accumulated count could only account for a tenth of the 3.6GB, the honest conclusion would be that this object type is real but is not the whole story and a second source of growth is still unexplained. Live concurrency could never require 400,000 pending contexts at once, so something is holding references to completed requests. With the service running three replicas behind a load balancer, the team sets an alert threshold below the historical OOM point and performs a rolling restart, one instance at a time with connections drained, while continuing to collect heap snapshots on the not-yet-restarted replicas to pin down the exact reference leak, which means following the retained-reference chain from a sample of those contexts back to a garbage-collection root rather than just re-confirming the count, since the count already told you everything it can.

Trade-offs and pitfalls: taking a full blocking heap dump on a memory-pressured process can itself cause the pause you're trying to avoid, so lead with a sampling profiler. Jumping to "it's a leak" without comparing growth against traffic wastes investigation time on what might just be a cache that needs a size cap, not a bug. A rolling restart is a real, if partial, restart, and it's worth being honest that it's a mitigation with a real cost (a cold cache on the cycled instance), not a way around the constraint entirely.

Treat this as a search problem across two axes: time (which commit introduced the regression) and evidence (which specific benchmark run proves it), rather than reading through commit history by eye.

CI artifacts: every CI (continuous integration) run should already store its build's performance-relevant artifacts, benchmark output, binary size, profiling data, tagged with the commit SHA. That gives you historical performance data attached to every commit without having to reproduce it after the fact.

Benchmark suites: run the same standardized workload, not ad hoc timing, across the suspect commit range. Look at which specific benchmark regressed and by how much, since that narrows which subsystem actually changed behavior.

Git bisection: use git bisect to binary-search between a known-good and a known-bad commit, running the benchmark suite (or a fast subset of it) at each step and marking it good or bad based on whether the regression reproduces. This finds the exact offending commit in roughly log2(n) steps instead of reading every commit in the range by hand.

Canary deployments: once a suspect fix or revert is ready, roll it to a small percentage of production traffic first and compare its live metrics, specifically the same percentile that regressed (for example p95 or p99, the response time only the slowest 5% or 1% of requests exceed), against a control group still running the previous version. Production traffic patterns can differ from synthetic benchmarks, so this catches anything the benchmark suite missed.

Validation before full rollout: only promote past canary once its metrics match the control baseline over a long enough window to cover normal traffic variation, ideally at least one full peak cycle, and keep an automatic rollback trigger tied to that same percentile in case it regresses again at higher traffic than the canary saw.

Worked example: a nightly benchmark suite shows p95 latency (95th percentile) in the "search" benchmark jump from 45ms to 60ms between two nightly CI runs spanning 30 commits. Running git bisect using that benchmark as the pass/fail check narrows it to a single commit that changed a JSON serialization library's default from compact output to pretty-printing. Reverting just that one setting, not the whole commit, and re-running the benchmark confirms the fix locally. A canary at 5% of production traffic shows p95 back at 46ms against a 61ms control group inside the first hour. That is enough to confirm the fix moves the metric in the right direction, and not enough to promote on: one hour of whatever traffic happens to be flowing does not cover the peak cycle the validation gate above asks for, and a serialization-cost regression scales with payload volume, so the peak is exactly where it would be worst. The canary therefore holds at 5% through the next daily peak with the automatic rollback trigger armed on p95, and only then expands in steps, 5% to 25% to 100%, re-running the same control comparison at each step rather than treating the first green reading as the decision.

Trade-offs and pitfalls: bisection assumes a single clean step-change from one commit, which fails on regressions that build up gradually across many small commits, each individually below your detection threshold; in that case you need to look at the trend line rather than a single good/bad boundary. Also, a benchmark suite with high run-to-run variance produces false bisection verdicts, so confirm the benchmark itself is stable (repeat runs, a real statistical margin) before trusting its automated good/bad calls.