Performance Troubleshooting & Incident Response Questions

Prove causation by overlaying the exact timestamps of the read latency spikes against the backup job's schedule across several nights, then confirm it with a resource-contention signal specific to the backup, not just "it happens at night."

Proving causation

• Overlay the read-latency graph and the backup job's start and end times over at least 3 to 5 nights to confirm the pattern repeats and the timing genuinely lines up, not just once.
• During a backup window, capture disk IO metrics with iostat -x 1, looking at IO utilization percentage and average wait time per disk, to see whether the backup is saturating IO bandwidth at the same time reads slow down.
• Check the database's own wait-event view during the backup window (most relational databases expose one, for example a wait-events view showing what queries are blocked on) to see whether reads are waiting on IO specifically, or on a lock the backup process holds.
• Run a controlled test: if it's safe, pause or reschedule the backup for one night and confirm read latency stays flat. That single test is the strongest evidence you can produce.

Short-term mitigations

• Move the backup to a genuinely off-peak window if it currently overlaps real traffic, or throttle its IO rate (most backup tools support an IO throttle or nice-style priority flag) so it competes less aggressively for disk.
• Run the backup against a read replica instead of the primary if one exists, removing the IO contention with production reads entirely.
• Temporarily reduce backup scope, for example incremental instead of full, if a full backup is the immediate driver of the IO spike.

Long-term fixes

• Stand up a dedicated replica specifically for backups so backup IO is permanently isolated from serving traffic.
• Move to a backup method that avoids heavy sequential IO against the live data files, such as storage-level snapshots, if the storage layer supports point-in-time snapshots, instead of a logical dump.
• Provision separate or higher-tier IO capacity if the underlying disk is genuinely undersized for concurrent backup and serving load.

Worked example: latency graphs for 4 consecutive nights show p95 read latency (95th percentile: 95% of read requests are faster than this number, so it captures the slow tail an average would hide) rising from 8ms to 60ms exactly during the 1:00 to 1:45 AM backup window each night. iostat -x 1 captured during that window shows disk utilization at 98% versus a normal 30%, and the database's wait-event view shows read queries spending most of their time in an IO-wait state rather than actually executing. Moving the backup to run against a read replica instead of the primary removes the correlation entirely on the following night's test.

Trade-offs and pitfalls: throttling the backup's IO extends how long it takes to complete, which can push it into business hours if you don't leave headroom, so any throttle needs to be checked against total backup duration. And confirming correlation over a single night isn't enough, one night could coincide with an unrelated cause, like a batch job that happens to run at the same time.

Direct answer: When p95 latency (the 95th percentile response time) jumps and then holds flat right after a deploy, the fastest lever is usually rollback, since it's reversible in minutes, while you diagnose in parallel. Don't let the search for root cause block the mitigation decision.

Structured elaboration:

• Telemetry first: compare the new deploy's dashboards against the prior version for CPU utilization, garbage collection (GC: a managed runtime periodically pausing to reclaim memory it no longer needs) pause frequency and duration, downstream dependency latency from traces, and network-level retry or error counts, ideally split by version if canary tagging exists.
• Trace comparison: pull a handful of the slowest request traces from before and after the deploy and look at the span waterfall. Is the extra time inside your own service's code, in a lock, or waiting on a downstream call?
• Experiments: fault-inject added latency into one candidate downstream dependency in staging and measure how many milliseconds of p95 you get per millisecond you inject. Be precise about what that buys you, because it is easy to overclaim: reproducing a p95 shift proves only that the call sits on the request's critical path and gives you a sensitivity coefficient. It does not show that this dependency is what changed in production, which is the actual hypothesis. The causal argument needs both halves multiplied together: the sensitivity from the injection, and the dependency's own measured latency delta in production across the deploy boundary, with the product accounting for most of the observed p95 move. If the dependency's latency did not move in production, a clean injection result is irrelevant no matter how well it reproduces, and if it did move but only by 5ms against a 120ms p95 jump, it is a contributor and not the cause. Separately, load-test the new version alone, without any real downstream dependencies in the loop, at matching request volume, to see if the code path itself (CPU or GC) explains the jump on its own.
• Categorize the eventual fix along four axes: code (an inefficient new code path, such as added serialization or computation), resources (an undersized CPU, memory, or thread-pool allocation for the new code path), network (a chattier call pattern with more round trips), or configuration (a changed timeout, connection-pool size, or feature-flag default).
• Rollback versus fix forward: default to rollback in the first response window since it's low-risk and reversible. Fix forward only when the regression traces to a single config value you can flip immediately (a feature flag), or when rollback itself carries real risk, for example if the deploy also included a data migration that isn't safely reversible.

Worked example: The deploy goes out at 10:00. p95 jumps from 300ms to 420ms (a 40% increase) immediately afterward and holds flat for the next hour rather than climbing further, which rules out a slow leak and points to a discrete step change caused by the deploy itself. Trace comparison shows CPU time spent inside your own process rose from about 40ms to about 150ms per request, with no material change in downstream call latency, and GC pause telemetry is unchanged. That combination points to code, not GC and not network. You fault-inject 100ms of extra latency into that downstream dependency in staging and staging's p95 moves by roughly the same 100ms, which establishes sensitivity of about 1ms of p95 per 1ms of dependency latency and nothing more. The production traces show that dependency's own latency unchanged across the deploy boundary, so there is no delta for that sensitivity to act on: 1.0 times roughly zero explains none of the 120ms. "The dependency coincidentally got slower" is ruled out by the production measurement, not by the injection, and the injection's real value here was proving the traces were not simply missing the call. With no config change involved and the regression clearly inside application code, you roll back rather than trying to patch the code path live under pressure.

Trade-offs and pitfalls: rollback isn't free either. If the deploy included a schema or data change, rolling back the code without also reversing the data change can leave the system in an inconsistent state, so check reversibility before defaulting to it. Chasing GC or network hypotheses before ruling them out with the telemetry you already have wastes time exactly when time is scarce, and trusting a single slow-trace sample without a second one to confirm the pattern risks over-fitting to noise.

Direct answer: A slow 10% creep over four weeks with SLOs (service level objectives: the target reliability or latency thresholds a team commits to) still technically met is the hardest kind of performance problem, because there's no sharp before-and-after to bisect against. The investigation has to reconstruct its own timeline from whatever telemetry exists, form falsifiable hypotheses, and rank them by how cheaply they can be tested, rather than searching for a single dramatic root cause.

Structured elaboration:

• Build a coarse timeline retroactively. Overlay the p99 trend (p99 latency: the value below which 99% of requests fall, so it captures the slowest 1% and moves before the average does) against everything that happened across the four weeks: deploys, feature-flag ramps, data-volume growth, traffic-mix shifts, dependency version bumps. A slow creep is often several small contributing changes, so look for where the SLOPE changed, not just deploy timestamps.
• Identify and backfill telemetry gaps. Pin down exactly what's missing (retention window too short, a missing per-endpoint tag, no historical resource utilization) and check whether raw logs, sampled traces, or infrastructure-level metrics (which often retain longer than aggregated dashboards) still exist for the earlier weeks even if the dashboard doesn't show them. Reprocessing raw data beats accepting "we don't know."
• Test hypotheses in cheapest-to-verify order, and don't let convenience bias the ranking:
  • GC (garbage collection) hypothesis: on a managed runtime, check whether GC pause frequency or duration trended up, often the result of a slowly growing heap footprint (an expanding cache, for example).
  • CPU hypothesis: check whether average per-instance CPU utilization crept up over the window even without traffic growing proportionally, which shows as more time in the run queue and disproportionately hits p99 before it visibly moves p50 (the median, i.e. the typical request).
  • Queueing hypothesis: check whether thread-pool queue depth or request concurrency rose gradually as traffic grew without matching capacity growth; queueing delay shows up first in the tail (p99) long before it's visible in the average, which is exactly why the SLO still looked fine.
• Add temporary headroom while you investigate. More capacity buys time without fixing anything, but it stops the trend from crossing the SLO before the investigation finishes, and add the missing telemetry now so this class of "we can't tell" doesn't recur.
• Close the loop once a fix is identified. Validate it in pre-production against the same traffic shape or a replay if possible, not just a smoke test, then roll it out gradually to production while watching the same p99 metric at each stage, so a regression is caught before full exposure rather than repeating this entire slow-burn cycle.

Worked example: Backfilled infrastructure metrics show average per-instance CPU utilization crept from 35% to 52% over the four weeks while request volume grew only about 8% in the same period, which rules out "it's simply proportional to more traffic." Thread-pool queue depth, previously near zero at p99, starts showing nonzero queueing specifically in the last ten days. Cross-referencing the deploy log shows a feature flag for a new response-enrichment step ramped from 5% to 100% of traffic across exactly that four-week window, gradually adding CPU cost per request. GC metrics stay flat the entire time, ruling that hypothesis out. The team validates by disabling the flag for a subset in a pre-production replay at matching load and confirms p99 drops back to baseline, then rolls the fix out gradually while watching p99 at each rollout stage.

Trade-offs and pitfalls: "the SLO is still technically met" is a trap, since a slow burn quietly spends error budget (the amount of unreliability a team is allowed to spend against its SLO before owing reliability work) and customer complaints often arrive before the SLO alarm does, so don't wait for a formal breach to act. Ranking hypotheses by which data happens to be available, rather than by actual plausibility, biases the investigation toward whichever metric was retained longest, not necessarily the real cause.

Direct answer: Check the database's own health metrics, the fraction of request time its calls take up in a trace, and your application's own connection-pool wait time. If all three point the same direction, temporarily reducing the load hitting the database, for example by lowering a rate limit at the gateway, is a safe, immediately reversible mitigation that needs no code change.

Structured elaboration, three lightweight checks:

1. Database-side metrics: look at the database's own dashboard for active connections, queries queued waiting for a connection slot, slow-query log entries, and host-level CPU/IO utilization. If the database's own connection pool is maxed and queries are queueing there, that's direct evidence.
2. Trace-based time attribution: pull a handful of slow-request traces and look at what fraction of total request time is the database span versus everything else. If the database span dominates the trace, it's the bottleneck by definition, no code change needed to observe this.
3. Application-side connection-pool wait time: check your app's own connection-pool metrics (checked-out connections, average wait time to acquire one). If the app is waiting to acquire a database connection before it even issues a query, that's evidence of database-side saturation (the database is at or near its capacity limit for that resource, so work queues instead of being served immediately) visible purely from metrics you already emit.

One safe immediate mitigation: temporarily lower the rate limit or traffic admission at the API gateway or load balancer for the affected endpoint. It's instantly reversible, requires no deploy, and directly relieves pressure on the database by reducing the query volume hitting it. If a healthy read replica already exists for read-only traffic, failing reads over to it is another safe option, but only for genuinely read-only paths, since routing writes there won't help and using it for read-after-write paths can surface replication lag as apparent data inconsistency.

Worked example: The database dashboard shows active connections pinned at the pool's configured maximum (100 of 100) with a query queue depth of 40, starting at the exact minute the API got slow. Traces on the slow endpoint show the database span accounts for about 85% of total request time. Application-side pool metrics show average wait-to-acquire-a-connection time jumped from about 2ms to about 180ms at the same moment. All three signals agree: database bound. The gateway's rate limit for that endpoint is dropped from 500 requests per second to 300 for 15 minutes to relieve the connection queue, then restored once the queue drains.

Trade-offs and pitfalls: shedding load has a real cost, some users get throttled or see errors, so it should be communicated as a deliberate, time-boxed mitigation, not applied silently. A read-replica failover only helps genuinely read-only traffic; routing write-dependent or read-after-write requests there won't fix the slowness and can introduce a separate correctness problem from replication lag.

Direct answer: Detect an N+1 problem (a code pattern that issues one extra database query per row of an initial result set, instead of one batched query) by counting queries per request, either from an APM (application performance monitoring: a tool that traces where time goes inside a single request) trace showing many near-identical child spans under one request, or from a query-logging middleware; verify by correlating the query count with the size of the result set, since a true N+1 scales linearly with rows returned. The fix is to batch the follow-up lookups into a single query.

Structured elaboration:

• Detect: turn on query logging or APM tracing for the slow endpoint and look at one representative request. If you see the same query shape repeated dozens or hundreds of times in a single request trace, each differing only by an ID, that is the signature of an N+1.
• Verify: confirm the query count scales with the size of the collection being iterated (10 orders means roughly 10 extra queries, 500 orders means roughly 500), which rules out coincidence and confirms causation rather than just correlation.
• Fix: replace the per-row lookup with either an eager-load/join at the ORM level, or an explicit batched query (WHERE id IN (...)) that fetches every needed row in one round trip.
• Complexity estimate: the code change itself is usually small (a few lines swapping a loop-of-lookups for one batched call), so the engineering effort is low; the risk is behavioral, not size, since a batched IN-clause has to handle duplicate keys, missing keys, and very large ID lists (which may need chunking) correctly.
• Testing before deploy: unit-test that the batched fetch returns identical results to the old per-row approach for the same input, add a regression test asserting the query count stays flat (or grows sub-linearly) as the input size grows, and load-test the endpoint at production-like data volumes before shipping, since N+1 problems often only show up once table sizes exceed what a small staging dataset exercises.

Worked example (runnable):

query_count = 0

# Toy in-memory "database": 500 orders, each pointing at one of 50 customers.
orders = [{"id": i, "customer_id": i % 50} for i in range(500)]
customers_table = {cid: f"customer_{cid}" for cid in range(50)}

def get_customer(customer_id):
    # Simulates one round trip to the DB for a single customer row.
    global query_count
    query_count += 1
    return customers_table[customer_id]

def get_customers_batch(customer_ids):
    # Simulates one round trip that fetches many customer rows at once
    # (SELECT * FROM customers WHERE id IN (...)).
    global query_count
    query_count += 1
    return {cid: customers_table[cid] for cid in customer_ids}

# Both versions start from the same single "SELECT * FROM orders". Count it in
# BOTH, otherwise the comparison is not like for like: the "+1" in "N+1" IS
# this query, and counting it in one column and not the other flatters the fix.
ORDERS_QUERY = 1

# Naive (N+1) version
query_count = 0
result_n1 = []
for order in orders:
    customer_name = get_customer(order["customer_id"])  # one query per order
    result_n1.append((order["id"], customer_name))
total_n1_queries = ORDERS_QUERY + query_count
print(f"N+1 version:   {len(orders)} orders -> {ORDERS_QUERY} + {query_count} = {total_n1_queries} queries")

# Fixed (batched) version
query_count = 0
customer_ids = {o["customer_id"] for o in orders}
customer_map = get_customers_batch(customer_ids)  # one query for all needed customers
result_fixed = [(o["id"], customer_map[o["customer_id"]]) for o in orders]
total_fixed_queries = ORDERS_QUERY + query_count
print(f"Fixed version: {len(orders)} orders, {len(customer_ids)} distinct customers -> {ORDERS_QUERY} + {query_count} = {total_fixed_queries} queries")

assert result_n1 == result_fixed
print("Results identical:", result_n1 == result_fixed)

# The claim is that one scales with the collection and the other does not.
# Demonstrate it at several sizes instead of asserting it in prose.
for n_orders in (10, 100, 1000):
    sample = [{"id": i, "customer_id": i % 50} for i in range(n_orders)]
    query_count = 0
    for o in sample:
        get_customer(o["customer_id"])
    n1 = ORDERS_QUERY + query_count
    query_count = 0
    get_customers_batch({o["customer_id"] for o in sample})
    fixed = ORDERS_QUERY + query_count
    print(f"  {n_orders:>4} orders -> N+1: {n1:>4} queries, batched: {fixed} queries")

Output:

N+1 version:   500 orders -> 1 + 500 = 501 queries
Fixed version: 500 orders, 50 distinct customers -> 1 + 1 = 2 queries
Results identical: True
    10 orders -> N+1:   11 queries, batched: 2 queries
   100 orders -> N+1:  101 queries, batched: 2 queries
  1000 orders -> N+1: 1001 queries, batched: 2 queries

The naive version issues 1 + 500 = 501 queries (the one that fetches the orders, plus one per order, which is literally what the name "N+1" describes); the batched version issues 1 + 1 = 2 (one for the orders, one IN-clause fetch for the 50 distinct customers needed), while producing the identical output. The sweep is the part that matters as evidence: the naive count tracks the collection size exactly (11, 101, 1001) while the batched count stays at 2 regardless, and note that the batched count does not grow with the number of DISTINCT customers either, only the size of the IN list does. That constant-versus-linear shape, not the single 501-versus-2 figure, is what you assert in the regression test.

Trade-offs and pitfalls: a batched IN-clause can hit a parameter-count limit on very large ID lists, so real fixes usually chunk the batch (for example 500 IDs per query) rather than assuming unlimited batch size. Also verify the fix doesn't just move the N+1 one layer up: if the batched customer fetch itself triggers a further per-row lookup (for example customer addresses), you've moved the problem rather than solved it, so re-check query counts end to end after the change, not just at the layer you touched.