Performance Troubleshooting & Incident Response Questions

Direct answer: Because the growth correlates with a specific API request rather than just elapsed time, you should be able to reproduce and attribute it deliberately instead of doing a broad, undirected heap analysis. Confirm the correlation with per-request profiling, minimize customer impact with a stopgap that targets that specific endpoint, and treat a restart as a temporary bridge, never the fix itself.

Structured elaboration:

• Confirm the correlation is real. Plot memory usage next to that endpoint's call volume. A true leak shows a step up that doesn't come back down even after traffic on that endpoint returns to baseline; a benign cache warm-up plateaus instead.
• Collect evidence with per-request attribution. If your profiler can scope sampling to a specific route or trace context, do that; otherwise, replay the suspect endpoint in isolation (staging, or a low-traffic canary) while taking heap snapshots before and after a fixed number of calls. Seeing object counts grow by roughly one instance (or a fixed multiple) per call is a strong per-call leak signature.
• Minimize customer impact while you investigate. If you can identify which endpoint or client is triggering it, rate-limit or feature-flag just that path rather than degrading the whole service. Add horizontal capacity as a stopgap to slow the per-instance accumulation rate. A proactive rolling-restart schedule (restarting before memory crosses a known-safe threshold) is an acceptable stopgap, but only if it's explicitly tracked as temporary with a follow-up fix, not left in place indefinitely.
• Decide restart versus fix: a restart alone resets the clock but doesn't touch the underlying cause, so it's acceptable only paired with continued investigation. The real fix is removing the actual leaking pattern, commonly an unbounded module-level cache or list, a per-request callback or subscription that's registered but never removed, or a full response body buffered in memory instead of streamed.

Worked example: Memory grows by roughly 2MB every time /reports/export is called. Profiling shows a global in-memory list, _export_cache, gaining exactly one entry per call, about 2MB matching a full result set for that report, with no eviction or TTL (time to live, an expiration after which an entry is automatically removed). Over a day of roughly 400 calls, that accounts for about 800MB of unrecoverable growth, matching what's observed. As an immediate mitigation, that endpoint is rate-limited to slow the growth rate while a fix ships that removes the unbounded caching (or bounds it with an eviction policy), followed by a normal rolling deploy of the fix rather than a blind restart, since a restart alone would only reset the clock, not the bug.

Trade-offs and pitfalls: attributing growth to the "obviously heaviest" endpoint without profiling can send the investigation down the wrong path if two endpoints share the same underlying code. A scheduled restart that starts as an explicit stopgap has a real tendency to quietly become permanent infrastructure that masks the bug and that nobody wants to touch later; treat it as a ticking clock, not a solution.

Direct answer: Check the database's own health metrics, the fraction of request time its calls take up in a trace, and your application's own connection-pool wait time. If all three point the same direction, temporarily reducing the load hitting the database, for example by lowering a rate limit at the gateway, is a safe, immediately reversible mitigation that needs no code change.

Structured elaboration, three lightweight checks:

1. Database-side metrics: look at the database's own dashboard for active connections, queries queued waiting for a connection slot, slow-query log entries, and host-level CPU/IO utilization. If the database's own connection pool is maxed and queries are queueing there, that's direct evidence.
2. Trace-based time attribution: pull a handful of slow-request traces and look at what fraction of total request time is the database span versus everything else. If the database span dominates the trace, it's the bottleneck by definition, no code change needed to observe this.
3. Application-side connection-pool wait time: check your app's own connection-pool metrics (checked-out connections, average wait time to acquire one). If the app is waiting to acquire a database connection before it even issues a query, that's evidence of database-side saturation (the database is at or near its capacity limit for that resource, so work queues instead of being served immediately) visible purely from metrics you already emit.

One safe immediate mitigation: temporarily lower the rate limit or traffic admission at the API gateway or load balancer for the affected endpoint. It's instantly reversible, requires no deploy, and directly relieves pressure on the database by reducing the query volume hitting it. If a healthy read replica already exists for read-only traffic, failing reads over to it is another safe option, but only for genuinely read-only paths, since routing writes there won't help and using it for read-after-write paths can surface replication lag as apparent data inconsistency.

Worked example: The database dashboard shows active connections pinned at the pool's configured maximum (100 of 100) with a query queue depth of 40, starting at the exact minute the API got slow. Traces on the slow endpoint show the database span accounts for about 85% of total request time. Application-side pool metrics show average wait-to-acquire-a-connection time jumped from about 2ms to about 180ms at the same moment. All three signals agree: database bound. The gateway's rate limit for that endpoint is dropped from 500 requests per second to 300 for 15 minutes to relieve the connection queue, then restored once the queue drains.

Trade-offs and pitfalls: shedding load has a real cost, some users get throttled or see errors, so it should be communicated as a deliberate, time-boxed mitigation, not applied silently. A read-replica failover only helps genuinely read-only traffic; routing write-dependent or read-after-write requests there won't fix the slowness and can introduce a separate correctness problem from replication lag.

Direct answer: Detect an N+1 problem (a code pattern that issues one extra database query per row of an initial result set, instead of one batched query) by counting queries per request, either from an APM (application performance monitoring: a tool that traces where time goes inside a single request) trace showing many near-identical child spans under one request, or from a query-logging middleware; verify by correlating the query count with the size of the result set, since a true N+1 scales linearly with rows returned. The fix is to batch the follow-up lookups into a single query.

Structured elaboration:

• Detect: turn on query logging or APM tracing for the slow endpoint and look at one representative request. If you see the same query shape repeated dozens or hundreds of times in a single request trace, each differing only by an ID, that is the signature of an N+1.
• Verify: confirm the query count scales with the size of the collection being iterated (10 orders means roughly 10 extra queries, 500 orders means roughly 500), which rules out coincidence and confirms causation rather than just correlation.
• Fix: replace the per-row lookup with either an eager-load/join at the ORM level, or an explicit batched query (WHERE id IN (...)) that fetches every needed row in one round trip.
• Complexity estimate: the code change itself is usually small (a few lines swapping a loop-of-lookups for one batched call), so the engineering effort is low; the risk is behavioral, not size, since a batched IN-clause has to handle duplicate keys, missing keys, and very large ID lists (which may need chunking) correctly.
• Testing before deploy: unit-test that the batched fetch returns identical results to the old per-row approach for the same input, add a regression test asserting the query count stays flat (or grows sub-linearly) as the input size grows, and load-test the endpoint at production-like data volumes before shipping, since N+1 problems often only show up once table sizes exceed what a small staging dataset exercises.

Worked example (runnable):

query_count = 0

# Toy in-memory "database": 500 orders, each pointing at one of 50 customers.
orders = [{"id": i, "customer_id": i % 50} for i in range(500)]
customers_table = {cid: f"customer_{cid}" for cid in range(50)}

def get_customer(customer_id):
    # Simulates one round trip to the DB for a single customer row.
    global query_count
    query_count += 1
    return customers_table[customer_id]

def get_customers_batch(customer_ids):
    # Simulates one round trip that fetches many customer rows at once
    # (SELECT * FROM customers WHERE id IN (...)).
    global query_count
    query_count += 1
    return {cid: customers_table[cid] for cid in customer_ids}

# Both versions start from the same single "SELECT * FROM orders". Count it in
# BOTH, otherwise the comparison is not like for like: the "+1" in "N+1" IS
# this query, and counting it in one column and not the other flatters the fix.
ORDERS_QUERY = 1

# Naive (N+1) version
query_count = 0
result_n1 = []
for order in orders:
    customer_name = get_customer(order["customer_id"])  # one query per order
    result_n1.append((order["id"], customer_name))
total_n1_queries = ORDERS_QUERY + query_count
print(f"N+1 version:   {len(orders)} orders -> {ORDERS_QUERY} + {query_count} = {total_n1_queries} queries")

# Fixed (batched) version
query_count = 0
customer_ids = {o["customer_id"] for o in orders}
customer_map = get_customers_batch(customer_ids)  # one query for all needed customers
result_fixed = [(o["id"], customer_map[o["customer_id"]]) for o in orders]
total_fixed_queries = ORDERS_QUERY + query_count
print(f"Fixed version: {len(orders)} orders, {len(customer_ids)} distinct customers -> {ORDERS_QUERY} + {query_count} = {total_fixed_queries} queries")

assert result_n1 == result_fixed
print("Results identical:", result_n1 == result_fixed)

# The claim is that one scales with the collection and the other does not.
# Demonstrate it at several sizes instead of asserting it in prose.
for n_orders in (10, 100, 1000):
    sample = [{"id": i, "customer_id": i % 50} for i in range(n_orders)]
    query_count = 0
    for o in sample:
        get_customer(o["customer_id"])
    n1 = ORDERS_QUERY + query_count
    query_count = 0
    get_customers_batch({o["customer_id"] for o in sample})
    fixed = ORDERS_QUERY + query_count
    print(f"  {n_orders:>4} orders -> N+1: {n1:>4} queries, batched: {fixed} queries")

Output:

N+1 version:   500 orders -> 1 + 500 = 501 queries
Fixed version: 500 orders, 50 distinct customers -> 1 + 1 = 2 queries
Results identical: True
    10 orders -> N+1:   11 queries, batched: 2 queries
   100 orders -> N+1:  101 queries, batched: 2 queries
  1000 orders -> N+1: 1001 queries, batched: 2 queries

The naive version issues 1 + 500 = 501 queries (the one that fetches the orders, plus one per order, which is literally what the name "N+1" describes); the batched version issues 1 + 1 = 2 (one for the orders, one IN-clause fetch for the 50 distinct customers needed), while producing the identical output. The sweep is the part that matters as evidence: the naive count tracks the collection size exactly (11, 101, 1001) while the batched count stays at 2 regardless, and note that the batched count does not grow with the number of DISTINCT customers either, only the size of the IN list does. That constant-versus-linear shape, not the single 501-versus-2 figure, is what you assert in the regression test.

Trade-offs and pitfalls: a batched IN-clause can hit a parameter-count limit on very large ID lists, so real fixes usually chunk the batch (for example 500 IDs per query) rather than assuming unlimited batch size. Also verify the fix doesn't just move the N+1 one layer up: if the batched customer fetch itself triggers a further per-row lookup (for example customer addresses), you've moved the problem rather than solved it, so re-check query counts end to end after the change, not just at the layer you touched.

Instrument the request path so each cache layer's lookup is individually timed and tagged, then attribute a slow request to whichever layer's timing span accounts for most of the elapsed time, instead of guessing from the overall request latency alone.

Instrumentation

• Wrap each layer's lookup (in-process memory check, local disk cache check, remote object store fetch) in its own timed span using a distributed tracing library, tagged with the layer, hit or miss, and the size of data returned.
• Attach these spans to the request's trace ID so a single slow request can be reconstructed end to end: how much time was spent checking memory, then disk (if memory missed), then the remote store (if disk missed too).
• Emit continuous per-layer histograms of lookup latency and hit rate, not just averages, so you can see p50 and p99 (50th and 99th percentile) per layer independently rather than only for the whole request.

Experiment design to isolate the cause

• Correlate spike timing against each layer's own metrics: does a spike line up with a drop in in-process cache hit rate (memory eviction or a cold cache), a rise in disk read latency (contention with another process for disk IO), or a rise in remote object store latency (network or the store's own load)?
• Deliberately manipulate one layer at a time in a controlled test, for example force a cold in-process cache by restarting the process, or throttle disk IO artificially, and see which manipulation reproduces the request-level spike. That layer is the culprit.
• Segment spiking requests by shape: are they disproportionately for large objects (more likely to fall through to the slower remote store on a miss), or concentrated on a specific key range (a hot key or an uneven shard on one store)?

Sampling strategy to keep overhead acceptable

• Head-based sampling for the common case: fully instrument a small fixed slice of all requests, for example 1 in 100, low enough to add negligible overhead to normal traffic.
• Always-on capture for the interesting case: fully instrument any request whose total latency exceeds a threshold near your current p99 baseline, since those are exactly the requests you care about and they're rare enough that capturing all of them doesn't add meaningful load.
• Keep the cheap per-layer histograms running at 100% of traffic as counters (not full traces), since counters cost far less than spans and are your early-warning signal even without a slow request in front of you.

Worked example: with always-on capture for every request over 200ms, a captured request comes in at 215ms total, and the discipline that makes per-layer attribution worth anything is that the 215ms has to add up. It breaks down as 0.2ms in the in-process memory lookup (a miss), 4.8ms in the local disk cache lookup (also a miss), 180ms inside the remote object store span, and 30ms of non-cache work, deserializing the fetched object plus the request handling on either side of the cache chain. Always reconcile that sum against the total, because the leftover is itself diagnostic: 30ms of ordinary application work is unremarkable, but if the spans had summed to 120ms of a 215ms request you would be looking at 95ms of missing instrumentation, not an answer, and the honest next step would be to instrument the gap rather than blame the slowest span you happen to have. Here the remote store is about 84% of the request, so the attribution is unambiguous. Correlating with the remote store's own metrics shows those spikes line up with periods when its p99 read latency rose, matching a known noisy-neighbor pattern (another tenant or workload sharing that same store, competing for the same resources and slowing it down for everyone else) on that shared store. The fix direction becomes populating the disk cache layer more aggressively for that key range to reduce dependence on the remote store, rather than tuning the in-process cache, which was never the bottleneck.

Trade-offs and pitfalls: sampling too sparsely, say 1 in 10,000, risks missing an intermittent issue that only affects a small fraction of requests entirely, which is why always-on capture for slow requests specifically matters, not random sampling alone. Per-layer instrumentation also adds real overhead if implemented naively, for example synchronous logging on the hot path, so span emission itself should be asynchronous and buffered so the instrumentation doesn't become part of the problem it's measuring.

Direct answer: When p95 latency (the 95th percentile response time) jumps and then holds flat right after a deploy, the fastest lever is usually rollback, since it's reversible in minutes, while you diagnose in parallel. Don't let the search for root cause block the mitigation decision.

Structured elaboration:

• Telemetry first: compare the new deploy's dashboards against the prior version for CPU utilization, garbage collection (GC: a managed runtime periodically pausing to reclaim memory it no longer needs) pause frequency and duration, downstream dependency latency from traces, and network-level retry or error counts, ideally split by version if canary tagging exists.
• Trace comparison: pull a handful of the slowest request traces from before and after the deploy and look at the span waterfall. Is the extra time inside your own service's code, in a lock, or waiting on a downstream call?
• Experiments: fault-inject added latency into one candidate downstream dependency in staging and measure how many milliseconds of p95 you get per millisecond you inject. Be precise about what that buys you, because it is easy to overclaim: reproducing a p95 shift proves only that the call sits on the request's critical path and gives you a sensitivity coefficient. It does not show that this dependency is what changed in production, which is the actual hypothesis. The causal argument needs both halves multiplied together: the sensitivity from the injection, and the dependency's own measured latency delta in production across the deploy boundary, with the product accounting for most of the observed p95 move. If the dependency's latency did not move in production, a clean injection result is irrelevant no matter how well it reproduces, and if it did move but only by 5ms against a 120ms p95 jump, it is a contributor and not the cause. Separately, load-test the new version alone, without any real downstream dependencies in the loop, at matching request volume, to see if the code path itself (CPU or GC) explains the jump on its own.
• Categorize the eventual fix along four axes: code (an inefficient new code path, such as added serialization or computation), resources (an undersized CPU, memory, or thread-pool allocation for the new code path), network (a chattier call pattern with more round trips), or configuration (a changed timeout, connection-pool size, or feature-flag default).
• Rollback versus fix forward: default to rollback in the first response window since it's low-risk and reversible. Fix forward only when the regression traces to a single config value you can flip immediately (a feature flag), or when rollback itself carries real risk, for example if the deploy also included a data migration that isn't safely reversible.

Worked example: The deploy goes out at 10:00. p95 jumps from 300ms to 420ms (a 40% increase) immediately afterward and holds flat for the next hour rather than climbing further, which rules out a slow leak and points to a discrete step change caused by the deploy itself. Trace comparison shows CPU time spent inside your own process rose from about 40ms to about 150ms per request, with no material change in downstream call latency, and GC pause telemetry is unchanged. That combination points to code, not GC and not network. You fault-inject 100ms of extra latency into that downstream dependency in staging and staging's p95 moves by roughly the same 100ms, which establishes sensitivity of about 1ms of p95 per 1ms of dependency latency and nothing more. The production traces show that dependency's own latency unchanged across the deploy boundary, so there is no delta for that sensitivity to act on: 1.0 times roughly zero explains none of the 120ms. "The dependency coincidentally got slower" is ruled out by the production measurement, not by the injection, and the injection's real value here was proving the traces were not simply missing the call. With no config change involved and the regression clearly inside application code, you roll back rather than trying to patch the code path live under pressure.

Trade-offs and pitfalls: rollback isn't free either. If the deploy included a schema or data change, rolling back the code without also reversing the data change can leave the system in an inconsistent state, so check reversibility before defaulting to it. Chasing GC or network hypotheses before ruling them out with the telemetry you already have wastes time exactly when time is scarce, and trusting a single slow-trace sample without a second one to confirm the pattern risks over-fitting to noise.