Performance Troubleshooting & Incident Response Questions

Direct answer: Treat this as a live incident, not a puzzle to solve calmly. In the first minutes, establish scope and correlate the timing against recent changes (deploy, config, traffic), decide on a fast reversible mitigation, and run deeper diagnosis in parallel rather than waiting for a full root cause before acting.

Structured elaboration:

• Minute 0 to 5, confirm and scope. p99 (the 99th percentile latency: 99% of requests are faster than this number) rising while p50 (the median latency: half of all requests are faster than this) stays flat means a growing minority of requests are getting badly hit, not a uniform slowdown. Check if this is one endpoint, one host, one region, or everything, and check the error rate alongside latency, not latency alone.
• Minute 5 to 10, correlate with change. Pull the deploy log, config/feature-flag change log, and traffic graph for the last one to two hours. A step change in p99 that starts exactly at a deploy or config push is your strongest early signal.
• Minute 10 to 20, decide on mitigation. If a change correlates, roll it back, since rollback is usually fast and reversible. If nothing correlates but a resource looks saturated (CPU, DB connections, thread pool queue), scale up or shed load. If a downstream dependency is clearly the cause, trip a circuit breaker (automatically stop sending requests to that dependency for a cooldown period once its failure rate crosses a threshold, so it can recover instead of being hit harder) or fail over.
• In parallel, preserve evidence. Pull traces for the slowest requests before you touch anything that would erase them (an instance getting killed or scaled down loses its state).
• Set an explicit rollback threshold before you decide, not during. For example: "if p99 hasn't dropped within 5 minutes of the rollback completing, or the error rate is still climbing, treat this candidate as ruled out and move to the next hypothesis." Writing this down ahead of time stops the team from waiting indefinitely on a fix that isn't working.

Worked example: Service-wide p99 climbs from 200ms to 1200ms starting 55 minutes ago. The deploy log shows a release went out 58 minutes ago. Splitting the latency metric by version, the 95% of traffic still on the old build has p50 around 45ms and p99 around 210ms, while the 5% already on the new build has p50 around 700ms, p80 around 1200ms and p99 around 3000ms. Sanity-check that the two halves can actually produce the number on the dashboard before you act on them, because a mixture's p99 can never be worse than the worst of its parts: if the canary's own p99 were 1150ms, no mix of it with a 210ms version could ever show a service-wide p99 of 1200ms, and a pair of numbers like that means one of the two metrics is mislabelled or scoped wrong. Here the arithmetic works out: the aggregate p99 is the point where 1% of all requests are slower, the canary supplies 5% of traffic, and 0.05 x 0.20 = 0.01, so the service-wide p99 lands where the canary's slowest fifth begins, which is the canary's p80 of about 1200ms. Everything else is held constant (same infrastructure, same traffic mix, only the version differs), so that is a clean causal signal and you roll the deploy back immediately rather than spending the next 20 minutes profiling. The same arithmetic is worth running in reverse before a rollout: a 5% canary that is merely twice as slow as the old build barely moves the service-wide p99 at all, which is why you alert on canary-versus-baseline percentiles rather than waiting for the aggregate number to move.

Trade-offs and pitfalls: Rolling back the first thing that correlates without checking whether it's plausible wastes the recovery window if you're wrong. Restarting or scaling an unhealthy instance before pulling traces or a profile from it throws away the evidence you need for the post-incident review. For that review, capture the exact timeline (when it started, when you noticed, when you mitigated), the before and after metrics, the change you rolled back or the resource you scaled, and, once found, the actual root cause, packaged as a short runbook (a step-by-step reference for handling this specific type of incident) so the next person facing the same symptom has a checklist instead of starting from zero.

Direct answer: Because the growth correlates with a specific API request rather than just elapsed time, you should be able to reproduce and attribute it deliberately instead of doing a broad, undirected heap analysis. Confirm the correlation with per-request profiling, minimize customer impact with a stopgap that targets that specific endpoint, and treat a restart as a temporary bridge, never the fix itself.

Structured elaboration:

• Confirm the correlation is real. Plot memory usage next to that endpoint's call volume. A true leak shows a step up that doesn't come back down even after traffic on that endpoint returns to baseline; a benign cache warm-up plateaus instead.
• Collect evidence with per-request attribution. If your profiler can scope sampling to a specific route or trace context, do that; otherwise, replay the suspect endpoint in isolation (staging, or a low-traffic canary) while taking heap snapshots before and after a fixed number of calls. Seeing object counts grow by roughly one instance (or a fixed multiple) per call is a strong per-call leak signature.
• Minimize customer impact while you investigate. If you can identify which endpoint or client is triggering it, rate-limit or feature-flag just that path rather than degrading the whole service. Add horizontal capacity as a stopgap to slow the per-instance accumulation rate. A proactive rolling-restart schedule (restarting before memory crosses a known-safe threshold) is an acceptable stopgap, but only if it's explicitly tracked as temporary with a follow-up fix, not left in place indefinitely.
• Decide restart versus fix: a restart alone resets the clock but doesn't touch the underlying cause, so it's acceptable only paired with continued investigation. The real fix is removing the actual leaking pattern, commonly an unbounded module-level cache or list, a per-request callback or subscription that's registered but never removed, or a full response body buffered in memory instead of streamed.

Worked example: Memory grows by roughly 2MB every time /reports/export is called. Profiling shows a global in-memory list, _export_cache, gaining exactly one entry per call, about 2MB matching a full result set for that report, with no eviction or TTL (time to live, an expiration after which an entry is automatically removed). Over a day of roughly 400 calls, that accounts for about 800MB of unrecoverable growth, matching what's observed. As an immediate mitigation, that endpoint is rate-limited to slow the growth rate while a fix ships that removes the unbounded caching (or bounds it with an eviction policy), followed by a normal rolling deploy of the fix rather than a blind restart, since a restart alone would only reset the clock, not the bug.

Trade-offs and pitfalls: attributing growth to the "obviously heaviest" endpoint without profiling can send the investigation down the wrong path if two endpoints share the same underlying code. A scheduled restart that starts as an explicit stopgap has a real tendency to quietly become permanent infrastructure that masks the bug and that nobody wants to touch later; treat it as a ticking clock, not a solution.

Direct answer: Check the database's own health metrics, the fraction of request time its calls take up in a trace, and your application's own connection-pool wait time. If all three point the same direction, temporarily reducing the load hitting the database, for example by lowering a rate limit at the gateway, is a safe, immediately reversible mitigation that needs no code change.

Structured elaboration, three lightweight checks:

1. Database-side metrics: look at the database's own dashboard for active connections, queries queued waiting for a connection slot, slow-query log entries, and host-level CPU/IO utilization. If the database's own connection pool is maxed and queries are queueing there, that's direct evidence.
2. Trace-based time attribution: pull a handful of slow-request traces and look at what fraction of total request time is the database span versus everything else. If the database span dominates the trace, it's the bottleneck by definition, no code change needed to observe this.
3. Application-side connection-pool wait time: check your app's own connection-pool metrics (checked-out connections, average wait time to acquire one). If the app is waiting to acquire a database connection before it even issues a query, that's evidence of database-side saturation (the database is at or near its capacity limit for that resource, so work queues instead of being served immediately) visible purely from metrics you already emit.

One safe immediate mitigation: temporarily lower the rate limit or traffic admission at the API gateway or load balancer for the affected endpoint. It's instantly reversible, requires no deploy, and directly relieves pressure on the database by reducing the query volume hitting it. If a healthy read replica already exists for read-only traffic, failing reads over to it is another safe option, but only for genuinely read-only paths, since routing writes there won't help and using it for read-after-write paths can surface replication lag as apparent data inconsistency.

Worked example: The database dashboard shows active connections pinned at the pool's configured maximum (100 of 100) with a query queue depth of 40, starting at the exact minute the API got slow. Traces on the slow endpoint show the database span accounts for about 85% of total request time. Application-side pool metrics show average wait-to-acquire-a-connection time jumped from about 2ms to about 180ms at the same moment. All three signals agree: database bound. The gateway's rate limit for that endpoint is dropped from 500 requests per second to 300 for 15 minutes to relieve the connection queue, then restored once the queue drains.

Trade-offs and pitfalls: shedding load has a real cost, some users get throttled or see errors, so it should be communicated as a deliberate, time-boxed mitigation, not applied silently. A read-replica failover only helps genuinely read-only traffic; routing write-dependent or read-after-write requests there won't fix the slowness and can introduce a separate correctness problem from replication lag.

Direct answer: Rising memory doesn't automatically mean a leak; it could be a cache warming up and plateauing, heap fragmentation, or a genuine leak where objects are retained past when they should be freed. Confirm which one it is by comparing object growth to traffic rather than watching total memory alone, and keep the service alive in the meantime without a full restart by draining and restarting instances one at a time if you're running more than one, or capping the specific growing structure if you're not.

Structured elaboration:

• Distinguish leak from legitimate growth: does memory keep climbing even during a low-traffic window (points to a leak, for example a background timer or job), or does it track request volume and then plateau (points to cache warm-up, not a leak, though the cache may still need a size cap)?
• Get object-level evidence without a blocking pause: prefer a live sampling profiler (RSS: resident set size, the actual physical memory a process is using, isn't precise enough on its own) over a full blocking heap dump. A full heap dump on a large, already-struggling heap can itself cause a multi-second pause, which risks dropping the in-flight work you're trying to protect. Take two snapshots several minutes apart under similar load and diff object counts by type: an object type that keeps growing well past what live request volume could ever require is the smoking gun for a leak.
• Keep it running safely without a full restart:
  • Set an alerting threshold well below the OOM (out-of-memory kill) point so you have runway before the process is killed for you.
  • If a specific structure is identified as the source and there's an admin endpoint or feature flag to bound or clear it, use that instead of restarting the whole process.
  • If the service runs multiple replicas, do a rolling restart, one instance at a time with connections drained first, rather than an all-at-once restart; that avoids dropping in-flight work or losing the warm cache for all traffic simultaneously, at the cost of a cold cache on just the one instance being cycled.
  • If it's a single instance with no way to drain traffic, the safest interim step is capping or disabling the specific growing structure through configuration while you keep collecting evidence, and plan a maintenance-window restart once you actually understand the cause.

Worked example: RSS grows from 1.2GB to 4.8GB over six hours while request volume stays roughly flat, so there is about 3.6GB of growth to account for. Two heap histograms taken 30 minutes apart under similar load show one object type, call it PendingRequestContext, growing from about 350,000 to about 383,000 instances, while the legitimate warm-cache object type stays flat around 50,000 entries. Now do the cross-check rather than stopping at "that number went up", because a growing object count on its own is a suspect, not a conviction: 33,000 new contexts in 30 minutes is about 1,100 a minute, which across the six hours of observed RSS growth comes to roughly 400,000 contexts, and 3.6GB spread over 400,000 objects is about 9KB retained each, the right order of magnitude for a request context holding a buffer and a parsed body. The rate, the standing count, and the RSS growth all agree, and that agreement is what turns this into a confirmed diagnosis. Had they disagreed, for example if the accumulated count could only account for a tenth of the 3.6GB, the honest conclusion would be that this object type is real but is not the whole story and a second source of growth is still unexplained. Live concurrency could never require 400,000 pending contexts at once, so something is holding references to completed requests. With the service running three replicas behind a load balancer, the team sets an alert threshold below the historical OOM point and performs a rolling restart, one instance at a time with connections drained, while continuing to collect heap snapshots on the not-yet-restarted replicas to pin down the exact reference leak, which means following the retained-reference chain from a sample of those contexts back to a garbage-collection root rather than just re-confirming the count, since the count already told you everything it can.

Trade-offs and pitfalls: taking a full blocking heap dump on a memory-pressured process can itself cause the pause you're trying to avoid, so lead with a sampling profiler. Jumping to "it's a leak" without comparing growth against traffic wastes investigation time on what might just be a cache that needs a size cap, not a bug. A rolling restart is a real, if partial, restart, and it's worth being honest that it's a mitigation with a real cost (a cold cache on the cycled instance), not a way around the constraint entirely.

Direct answer: Both requests exceeded their own timeout, but not by the same margin: the database call ran 20% over its timeout, while the cache call ran 180% over its. Since a cache lookup is normally fast, that disproportionate breach makes the cache call the more anomalous of the two, and the fact that both upstreams (the dependencies this service calls out to, here the database and the cache) breached their timeouts about a second apart is more consistent with a shared cause than two unrelated slow dependencies.

Structured elaboration:

• Do the arithmetic on how far over timeout each call actually ran. Database: 1200ms actual duration against a 1000ms timeout, a ratio of 1200/1000 = 1.2, so 20% over. Cache: 1400ms actual duration against a 500ms timeout, a ratio of 1400/500 = 2.8, so 180% over. The cache breach is proportionally far worse even though its raw duration (1400ms) is only slightly higher than the database's (1200ms).
• Ask why each duration exceeds its own timeout at all. This is the sharpest clue in the two lines and it is easy to walk past. A call configured with a 500ms timeout should have been abandoned at about 500ms, so a logged duration of 1400ms demands an explanation, and there are only three candidates: duration_ms covers a whole attempt sequence rather than one attempt (1400ms is close to three 500ms attempts back to back), the timeout is not enforced on the leg that matters (the classic version is a connect timeout configured with no read or total-call timeout, so a socket that connects fast and then stalls runs unbounded), or duration_ms is end-to-end request time and the upstream call is only one part of it. The database line discriminates between those readings: 1200ms against a 1000ms timeout is one attempt plus about 200ms of surrounding work, which is exactly what a correctly enforced single-attempt timeout looks like. The most economical story that fits both lines is therefore one attempt against the database and about three against the cache, and that materially changes the plan: a client retrying three times into a struggling cache is adding load to the thing that is already slow, so capping retries there is a mitigation available immediately, before anyone has found the root cause.
• Read WARN as "slow," not automatically "failed." Check whether these requests were retried and eventually succeeded, or resulted in a hard error, before assuming customer impact. That check and the one above are the same check: the retry count is what tells you whether 1400ms was one long stall or several bounded ones.
• Check volume before drawing conclusions. Two WARN lines could be background noise if they're two out of thousands of requests, or evidence of a systemic issue if they're a large fraction of traffic in the same window. Aggregate WARN counts by upstream over the last 15 to 30 minutes.
• Check whether it's the same host or different hosts. Both log lines coming from the same host points toward a host-level resource issue (network interface saturation, meaning the network card is at its maximum data-transfer capacity, or a noisy neighbor: another workload sharing that same host competing for the same CPU, memory, network, or disk capacity and starving yours); different hosts hitting both a database and a cache in the same window points toward a shared network path or shared infrastructure problem.
• Prioritize the cache investigation first, because its timeout breach is proportionally larger, because the retry reading above makes it the one call actively adding load to its own dependency, and because cache failures often cascade: if the cache is unhealthy, more requests fall through to the database, which would explain both WARN lines from a single root cause rather than two independent ones.

Trade-offs and pitfalls: it's easy to fixate on the larger raw duration number (1400ms) without computing the ratio against each call's own timeout, which is what actually tells you which one is more abnormal. It's also a mistake to treat two WARN log lines as proof of a systemic problem before checking whether they represent 0.1% or 50% of traffic in that window.