Performance Troubleshooting & Incident Response Questions

An hourly-on-the-hour pattern is the strongest clue you're given: anything that fires on a fixed wall-clock schedule is almost always a cron job, a scheduled task, or a periodic batch or cache-refresh process, so the runbook should confirm that correlation before doing anything else.

Triage

1. Confirm the pattern across at least 3 to 4 occurrences (not just one), and record the exact spike duration and magnitude.
2. List anything scheduled to run at that time: crontab entries, scheduled Kubernetes CronJobs, or application-level scheduled work like cache warmers, report generation, log rotation, or metric aggregation.
3. During the next occurrence, capture a live process list sorted by CPU (a top/htop snapshot) or a CPU profile, to see exactly which process or thread is consuming the CPU.
4. Check whether the spike is fleet-wide or isolated to one host. Fleet-wide points to a synchronized schedule; a single host points to something host-local, like that box's own log rotation or backup agent.

Temporary mitigations (one line each)

• Stagger or jitter the scheduled job's start time across hosts if it's fleet-wide and synchronized, to spread the load instead of eliminating it. Be honest about what this buys: it lowers the fleet-wide concurrent peak and the instantaneous load on anything shared downstream, but it does not reduce any single host's own CPU during its run, so it only clears the alert if the alert is on an aggregate.
• Throttle the job's resource usage (a CPU cgroup limit, cgroup being the Linux kernel mechanism for capping how much CPU or memory a process can use, or nice/ionice priority) so it can't starve request-handling threads.
• Move the job off-peak temporarily if its current hourly time overlaps real user traffic.
• Scale out capacity briefly around the spike window as a stopgap while the real fix is investigated.

Prioritized long-term investigations

1. Identify and profile the exact job or process causing the spike, since nothing else can be fixed without knowing this first.
2. Determine why it's expensive: unnecessary full-table scans, an unbounded cache rebuild, or synchronous work that should be batched or made asynchronous.
3. Evaluate whether it needs to run hourly at all, or could become incremental, processing only new data instead of reprocessing everything each time.
4. Add monitoring on the job itself (its own duration and resource usage) so a regression in the job is caught before it re-manifests as a mystery CPU spike again.

Worked example: the hourly spikes correlate exactly with a cron-scheduled cache-warming script that reloads the entire product catalog into memory every hour on the hour, across all 20 application hosts simultaneously, each saturating a CPU core for about 30 seconds during deserialization. Before reaching for jitter, pin down which alert you are trying to clear, because jitter only moves one of them. Spreading 20 start times uniformly across a 0 to 300 second window drops the number of hosts warming at the same instant from 20 to roughly 20 x 30 / 300, about 2, so a fleet-aggregate CPU metric and the shared catalog store both stop taking a 20-way simultaneous hit. It does nothing for any individual host: that host still pegs the same core for the same 30 seconds, just at a different second, so a per-host CPU alert fires just as often and just as hard, now 20 times staggered instead of once together. If the alert is per-host, the immediate lever is the cgroup CPU cap or nice, which bounds what the warmer can take from request-handling threads at the cost of the warmer running longer. The longer-term fix is switching the cache warmer to an incremental update that reloads only changed catalog entries, which is the only one of these that reduces the total work rather than redistributing it.

Trade-offs and pitfalls: jittering hides the symptom without reducing the total work done, so it buys time but is not the fix. Don't assume "hourly" always means an actual crontab entry, some message-queue flush intervals or batching layers can also produce a wall-clock-aligned pattern without a literal cron job behind it.

Direct answer: With three signals moving together (CPU, GC pause time, and external API retries), the trap is treating them as three independent causes when they may really be A causing B causing C, or all three being downstream symptoms of a fourth thing, like a traffic burst. The plan has to establish which one moved first and then test the causal story directly, not just observe that they correlate.

Structured elaboration:

• Reconstruct a fine-grained timeline. Pull second-level, not minute-level, time series for CPU, GC pause events, and retry counts around a single spike, overlaid on the same axis. Whichever metric's rise leads the others, even by a few hundred milliseconds, is the more likely upstream cause, though leading correlation alone is still not proof.
• Check for a shared upstream cause. Look at request concurrency or queue depth in the same window. If it spikes before all three of the other metrics, none of CPU, GC, or retries is the root cause; they're co-symptoms of a traffic burst instead.
• Test each hypothesis independently, don't just rank them by correlation:
  • To test "a GC pause froze the app long enough to trip the external call's timeout," check whether retry timestamps cluster right after logged GC pause events, and whether the configured call timeout is short enough that a pause of that duration alone would trip it.
  • To test "the external API got slow first, which caused retries, which caused GC/CPU pressure," check the external API's own latency or status data during the spike window; if it was degraded before your CPU or GC metrics moved, that argues it's upstream.
  • To test "a third confounder like a traffic burst," check whether the spikes line up with a known pattern (a batch job, a caller's own retry storm, a marketing push) independent of CPU, GC, and the external API.
• Reproduce the leading hypothesis in isolation. If you suspect the external API, fault-inject several seconds of added latency into a shadow call to that same dependency in staging at similar concurrency, and see if you reproduce a comparable GC-pause and CPU signature. Reproducing the coupled symptom by injecting only one variable is strong confirmation.

Worked example: In one eight-second spike, retry timestamps for the external API start climbing about 1.5 seconds before GC pause time increases, and about 2 seconds before CPU jumps. Checking the external API's own status dashboard shows a brief documented degradation in that exact window. That ordering plus outside corroboration point to: the external API slowed down, your client retried aggressively, the resulting retry storm (more concurrent in-flight requests and objects) drove up allocation rate, and GC pressure and CPU followed as downstream effects. You confirm by fault-injecting three to four seconds of added latency on a shadow call to that dependency in staging at comparable concurrency and reproduce a similar GC-pause and CPU bump.

Trade-offs and pitfalls: leading-edge correlation in a single incident is suggestive, not conclusive; you want the same ordering to hold across multiple spikes before committing engineering effort to a fix. It's also easy to over-index on whichever metric is loudest (CPU usually pages first) instead of tracing it backward, and it's a mistake to assume all three symptoms must share one root cause; the isolation step protects against wrongly merging two coincidental problems into a single story. Once the retry storm is confirmed as the trigger, the fix is adding jittered exponential backoff and a retry budget (a cap on how much total traffic is allowed to be retries) or a circuit breaker (a safeguard that stops sending requests to a dependency once its failure rate crosses a threshold, so it gets a chance to recover instead of being retried into the ground) on that external call, so a downstream blip can't amplify into a retry storm again.

Build the case in layers: confirm the symptom shape is even consistent with contention, use a profiler to find WHICH specific lock is contended, confirm causally rather than by correlation alone, and only then weigh each fix's engineering cost against how much of the measured p99 latency (99th percentile) it would actually recover.

Methodology to pinpoint the contended lock

• First confirm the shape: plot p99 latency against concurrency (concurrent in-flight requests or active threads), not against raw request volume. Lock contention specifically gets worse as more threads compete for the same lock, while a pure capacity issue degrades more with total request rate regardless of concurrency shape.
• Use perf record with call-graph capture during a live high-concurrency window to build a CPU flame graph (a visualization where the width of each bar shows how much time was spent in that function, making hot code paths easy to spot), but be exact about what that profile can and cannot see, because this is where most lock investigations quietly go wrong. perf record samples on a CPU event, so it only samples threads that are currently running. A thread waiting for a lock is parked in the kernel's futex wait (futex: the low-level kernel primitive threads block on while waiting to acquire a lock), burning no cycles, so it contributes essentially nothing to a CPU flame graph no matter how many seconds of wall clock it spends there. The CPU profile is still worth taking, for two reasons: it rules out the competing hypothesis that you are simply CPU-bound in application code, and a typical adaptive mutex spins briefly before it parks, so heavy contention does leave a visible spin signature in the lock-acquire path even though the sleeping part is invisible.
• To see the wait time itself you need an off-CPU view, since that time is off-CPU by definition: perf record -e sched:sched_switch with call graphs, perf sched, or an off-CPU sampling tool, each of which attributes blocked wall-clock time to the stack that blocked rather than to whoever was running. The practical consequence is a trap worth stating out loud: a CPU profile that shows all of its samples in your busy worker while p99 sits in lock wait is not evidence that locks are fine, it is evidence you took the wrong kind of profile.
• Use a dedicated lock-profiling tool (perf lock on Linux, or a language-level equivalent) to get direct contention statistics per lock: which lock, how many threads waited on it, and total aggregate wait time, ranking locks by wait time rather than by which one looks suspicious in the code.
• Optionally run a thread sanitizer (TSan) against a non-production instrumented build to also rule out an actual concurrency bug, a subtle data race hiding behind what looks like ordinary lock usage.

Confirming causally, not just correlating

• Once a specific lock is identified as the top contender by wait time, deliberately vary concurrency in a controlled load test (never in production) while holding that lock's usage pattern fixed, and confirm p99 latency scales with contention on that specific lock, not merely with overall load.
• If possible, instrument that one lock to log per-acquisition wait time, and confirm its wait-time distribution's tail matches the shape of the request-level p99 tail you started with.

Deciding which fix is worth the engineering cost

• Options in rough order of increasing cost: shrink the critical section (do less work while holding the lock, for example moving logging or serialization outside the locked region), reduce lock granularity (split one lock protecting many independent pieces of state into several finer-grained locks), replace the lock with a lock-free data structure on the specific hot path, or restructure the code path to avoid needing shared mutable state there at all, the most invasive option.
• Weigh each option against how much of the MEASURED contention wait time it would actually eliminate (a fix removing only 10% of the wait time isn't worth a risky rewrite), the correctness risk it introduces (lock-free code is notoriously easy to get subtly wrong), and how urgent the problem is against your SLO (service-level objective: the target latency or reliability the team has committed to) or error budget (the acceptable amount of unreliability left before you must act).
• Prefer the cheapest fix that removes the measured bottleneck, and escalate to a more invasive fix only if the cheap fix's wait-time reduction, once re-measured, still leaves the tail above target.

Worked example: perf lock output during a high-concurrency load test shows one mutex protecting a shared metrics counter accounts for 70% of all measured lock wait time across the process, far more than any other lock. Plotting p99 latency against concurrent thread count shows a sharp knee exactly at the thread count where that mutex's wait time starts dominating, consistent with contention rather than a raw throughput limit. Shrinking the critical section, moving the metrics serialization work outside the lock and keeping only the counter increment inside it, reduces that mutex's measured wait time by roughly 90% in a repeat perf lock run, and a repeat load test shows p99 flattening out well past the previous knee point.

Trade-offs and pitfalls: jumping straight to a lock-free rewrite without first measuring which lock accounts for how much wait time risks spending significant engineering effort on a lock that was never the real bottleneck. A fix validated at the load test's concurrency level should also be re-validated at production's actual peak concurrency, since contention effects can be nonlinear, a fix sufficient at 2x normal load may not hold at 5x.

A thread dump is a snapshot of every thread's stack trace and state at one instant. You read it by grouping threads by state (RUNNABLE, BLOCKED, WAITING, TIMED_WAITING) and by what each is doing, and by comparing several dumps taken a few seconds apart to see whether the same threads are stuck in the same place.

CPU bound: many threads in RUNNABLE state with distinct, active-looking stacks (real computation, not a wait call), and the stacks are diverse rather than piled up at one line. Cross-check against actual process CPU usage: if it's near 100% and threads are RUNNABLE inside real application logic, that confirms CPU-bound work rather than a lock or IO problem.

IO bound: threads in RUNNABLE or WAITING state whose stack bottoms out in a native socket or file read/write call (for example a socket read method or a JDBC driver's read call) rather than application code, while overall CPU utilization stays comparatively low. Many threads look parked at the same kind of IO call but on different connections.

Thread contention (lock contention): many threads in BLOCKED state, and the dump explicitly names which thread holds the lock they're waiting on ("waiting to lock ... held by thread X"). The telltale pattern is a large number of threads BLOCKED at the exact same stack frame, all funneling toward one holder. If that holder is itself stuck (waiting on something slow, like a synchronous external call inside a synchronized block), that's the head of the queue and the real cause.

BLOCKED only covers synchronized, and assuming otherwise is the biggest trap in reading a dump for contention. The BLOCKED state and the "waiting to lock ... held by" line come from intrinsic monitors, which means synchronized blocks and methods and nothing else. Contention on anything in java.util.concurrent, a ReentrantLock, a ReadWriteLock, a Semaphore, a bounded queue's put, parks the waiting thread instead, so it shows up as WAITING or TIMED_WAITING with jdk.internal.misc.Unsafe.park at the top of the stack and a line reading parking to wait for <0x00000000d7a4b3c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync), and the dump does not name the owner. A service built on java.util.concurrent can be completely seized by lock contention and show zero BLOCKED threads, so "no BLOCKED threads, therefore no contention" is wrong and is how this gets missed. Two things recover the missing information: capture with jstack -l or jcmd <pid> Thread.print -l, which adds a "Locked ownable synchronizers" section naming which thread owns each lock so you can rebuild the holder relationship by hand; and read the park sites exactly the way you read a BLOCKED funnel, since many threads parked on the same synchronizer identity is the same signal wearing different clothes.

Patterns to spot

• Same stack, many threads, one state: a real bottleneck (hot lock, saturated pool, slow shared resource).
• Diverse stacks, all RUNNABLE: probably genuinely CPU-heavy work, not contention.
• An explicit "Found one Java-level deadlock" section, or two threads each BLOCKED waiting on a lock the other holds: a deadlock, not ordinary contention.
• Every thread in one pool busy or blocked at the same call site while work backs up behind it: that pool is exhausted, and the fix lives upstream of the pool size (a timeout on whatever those threads are all waiting for), because a bigger pool just buys a few more seconds before the same queue forms. Its mirror image is worth naming separately because the two are easy to confuse: if the request-handling pool's threads are instead parked idle on their own task queue while a smaller, separate pool is saturated, the request pool is not the bottleneck at all, the smaller one is, and enlarging the request pool changes nothing.

Worked example: A dump shows 40 of 50 request-handling threads BLOCKED at the same line, "waiting to lock <0x00000000d7a4b3c0> (a com.example.audit.AuditWriter)", and the single thread holding that lock is itself parked inside a synchronous call to a legacy audit-logging component. The BLOCKED state tells you immediately that the lock is a synchronized one; had the same code used a ReentrantLock, those 40 threads would have read as WAITING on a park and you would have needed jstack -l to find the owner. A second dump taken 5 seconds later shows the exact same holder stuck at the exact same line, while total process CPU usage measured over that window is under 20%. That combination, low CPU plus a stable BLOCKED funnel, points squarely at lock contention around a slow synchronous call, not a CPU problem.

Trade-offs and pitfalls: a single dump can mislead, since a thread happening to be RUNNABLE at the exact snapshot moment doesn't prove it's the bottleneck. Always take 3 to 5 dumps a few seconds apart and look for a pattern that repeats, not a single frame. Thread dumps also only show state at the moment captured, so brief contention that resolves in milliseconds may never appear in a dump; a continuous sampling profiler is the better tool for catching that kind of short-lived event.

Direct answer: Detect an N+1 problem (a code pattern that issues one extra database query per row of an initial result set, instead of one batched query) by counting queries per request, either from an APM (application performance monitoring: a tool that traces where time goes inside a single request) trace showing many near-identical child spans under one request, or from a query-logging middleware; verify by correlating the query count with the size of the result set, since a true N+1 scales linearly with rows returned. The fix is to batch the follow-up lookups into a single query.

Structured elaboration:

• Detect: turn on query logging or APM tracing for the slow endpoint and look at one representative request. If you see the same query shape repeated dozens or hundreds of times in a single request trace, each differing only by an ID, that is the signature of an N+1.
• Verify: confirm the query count scales with the size of the collection being iterated (10 orders means roughly 10 extra queries, 500 orders means roughly 500), which rules out coincidence and confirms causation rather than just correlation.
• Fix: replace the per-row lookup with either an eager-load/join at the ORM level, or an explicit batched query (WHERE id IN (...)) that fetches every needed row in one round trip.
• Complexity estimate: the code change itself is usually small (a few lines swapping a loop-of-lookups for one batched call), so the engineering effort is low; the risk is behavioral, not size, since a batched IN-clause has to handle duplicate keys, missing keys, and very large ID lists (which may need chunking) correctly.
• Testing before deploy: unit-test that the batched fetch returns identical results to the old per-row approach for the same input, add a regression test asserting the query count stays flat (or grows sub-linearly) as the input size grows, and load-test the endpoint at production-like data volumes before shipping, since N+1 problems often only show up once table sizes exceed what a small staging dataset exercises.

Worked example (runnable):

query_count = 0

# Toy in-memory "database": 500 orders, each pointing at one of 50 customers.
orders = [{"id": i, "customer_id": i % 50} for i in range(500)]
customers_table = {cid: f"customer_{cid}" for cid in range(50)}

def get_customer(customer_id):
    # Simulates one round trip to the DB for a single customer row.
    global query_count
    query_count += 1
    return customers_table[customer_id]

def get_customers_batch(customer_ids):
    # Simulates one round trip that fetches many customer rows at once
    # (SELECT * FROM customers WHERE id IN (...)).
    global query_count
    query_count += 1
    return {cid: customers_table[cid] for cid in customer_ids}

# Both versions start from the same single "SELECT * FROM orders". Count it in
# BOTH, otherwise the comparison is not like for like: the "+1" in "N+1" IS
# this query, and counting it in one column and not the other flatters the fix.
ORDERS_QUERY = 1

# Naive (N+1) version
query_count = 0
result_n1 = []
for order in orders:
    customer_name = get_customer(order["customer_id"])  # one query per order
    result_n1.append((order["id"], customer_name))
total_n1_queries = ORDERS_QUERY + query_count
print(f"N+1 version:   {len(orders)} orders -> {ORDERS_QUERY} + {query_count} = {total_n1_queries} queries")

# Fixed (batched) version
query_count = 0
customer_ids = {o["customer_id"] for o in orders}
customer_map = get_customers_batch(customer_ids)  # one query for all needed customers
result_fixed = [(o["id"], customer_map[o["customer_id"]]) for o in orders]
total_fixed_queries = ORDERS_QUERY + query_count
print(f"Fixed version: {len(orders)} orders, {len(customer_ids)} distinct customers -> {ORDERS_QUERY} + {query_count} = {total_fixed_queries} queries")

assert result_n1 == result_fixed
print("Results identical:", result_n1 == result_fixed)

# The claim is that one scales with the collection and the other does not.
# Demonstrate it at several sizes instead of asserting it in prose.
for n_orders in (10, 100, 1000):
    sample = [{"id": i, "customer_id": i % 50} for i in range(n_orders)]
    query_count = 0
    for o in sample:
        get_customer(o["customer_id"])
    n1 = ORDERS_QUERY + query_count
    query_count = 0
    get_customers_batch({o["customer_id"] for o in sample})
    fixed = ORDERS_QUERY + query_count
    print(f"  {n_orders:>4} orders -> N+1: {n1:>4} queries, batched: {fixed} queries")

Output:

N+1 version:   500 orders -> 1 + 500 = 501 queries
Fixed version: 500 orders, 50 distinct customers -> 1 + 1 = 2 queries
Results identical: True
    10 orders -> N+1:   11 queries, batched: 2 queries
   100 orders -> N+1:  101 queries, batched: 2 queries
  1000 orders -> N+1: 1001 queries, batched: 2 queries

The naive version issues 1 + 500 = 501 queries (the one that fetches the orders, plus one per order, which is literally what the name "N+1" describes); the batched version issues 1 + 1 = 2 (one for the orders, one IN-clause fetch for the 50 distinct customers needed), while producing the identical output. The sweep is the part that matters as evidence: the naive count tracks the collection size exactly (11, 101, 1001) while the batched count stays at 2 regardless, and note that the batched count does not grow with the number of DISTINCT customers either, only the size of the IN list does. That constant-versus-linear shape, not the single 501-versus-2 figure, is what you assert in the regression test.

Trade-offs and pitfalls: a batched IN-clause can hit a parameter-count limit on very large ID lists, so real fixes usually chunk the batch (for example 500 IDs per query) rather than assuming unlimited batch size. Also verify the fix doesn't just move the N+1 one layer up: if the batched customer fetch itself triggers a further per-row lookup (for example customer addresses), you've moved the problem rather than solved it, so re-check query counts end to end after the change, not just at the layer you touched.