Production Incident Diagnosis and Distributed Systems Troubleshooting Questions

Direct answer. Error rate up with CPU and memory normal means the cause is very likely upstream of the process (a bad deploy, a dependency, or the network) rather than the box being overloaded, so the first ten minutes are about narrowing that down, not restarting anything.

Structured elaboration.

1. Confirm the blast radius before doing anything else. Check whether the 500s/5xx are concentrated on one endpoint, one region, or one instance versus spread everywhere. A dashboard that breaks error rate down by endpoint and by host answers this in under a minute and tells you whether you are looking at a code problem (one endpoint), an infrastructure problem (one host or one AZ, short for availability zone: an isolated data-center location within a cloud region), or a systemic one (everywhere at once). In practice this is a couple of concrete commands or queries, not just staring at a dashboard: something like curl -s -o /dev/null -w "%{http_code} %{time_total}\n" https://<endpoint> run against a handful of endpoints to spot-check live, or, on Kubernetes, kubectl get pods -o wide and kubectl top pods to rule out a single bad pod or node before assuming the problem is code-wide.
2. Check what changed 3 minutes ago. Pull up the deploy log, the feature-flag change log, and any config/infra change log for the last 15 to 30 minutes. A timestamp that lines up with the start of the errors is the strongest single signal you have; it does not prove causation but it tells you where to look first. Concretely, that is often kubectl rollout history deployment/<service> or your CI/CD tool's own release log, plus git log --oneline -10 against the service's repo if you need to see what actually shipped.
3. Read the actual error, not just the rate. Open a handful of the failing requests in your logs or tracing tool. A stack trace or error message ('connection refused to payments-db', a 5xx from a specific downstream call, a validation exception) usually narrows the search from 'everything is broken' to one dependency or one code path within a minute or two. A quick kubectl logs -l app=<service> --since=10m | grep -iE "5[0-9]{2}|exception|error" (or the equivalent saved query in your logging tool) is usually the fastest way to pull that sample without waiting on a dashboard to refresh.
4. Check the obvious downstream suspects. Since CPU and memory are normal, the next most common causes are a downstream dependency (database, cache, third-party API) returning errors or timing out, a recent deploy or config change, or a network-level issue between services. Check the error rate and latency of each downstream call your service makes.
5. Decide mitigate-vs-investigate. If step 2 found a deploy or flag flip in the matching window, your fastest safe move is usually to roll it back or flip the flag off and confirm the error rate recovers, rather than to keep root-causing live. If nothing matches, apply the narrowest containment you can (for example, shedding load to the failing endpoint, or failing over the specific downstream that looks unhealthy) while you keep investigating.
6. Communicate to stakeholders as you go, not just once you have an answer. A short, factual update to your incident channel or status page within the first few minutes (what's affected, the rough blast radius from step 1, what you've ruled in or out, and what you're doing next) costs you well under a minute and pays for itself twice over: it lets anyone who joins mid-investigation avoid re-doing your first five minutes, and it keeps support and leadership from guessing in the absence of information. Keep it factual, not speculative ('investigating a checkout error spike, blast radius looks contained to one endpoint, next update in 10 minutes' rather than promising a fix time you don't actually know yet), and post a second update the moment you mitigate or escalate.

Worked example. Say the breakdown in step 1 shows the errors are 92% concentrated on POST /checkout, and the deploy log shows a release to the checkout service 4 minutes before the alert. That is strong enough correlation to justify an immediate rollback rather than a deeper investigation first: you roll back, watch the error rate for the next 2 to 3 minutes, and if it drops back toward 0.2% you have both fixed the immediate problem and confirmed the cause. If the rollback does NOT bring the rate down, you now know the deploy was a coincidence and move to checking checkout's downstream dependencies (payment gateway, inventory service) instead.

Trade-offs and pitfalls. The most common mistake in the first ten minutes is rolling back or restarting before confirming blast radius and correlation, which burns the ten minutes on an action that may not fix anything and makes the timeline confusing to reconstruct later. The second most common mistake is treating 'normal CPU and memory' as if it rules out the service itself: a service can be fully healthy on resource metrics while returning errors because of a bug in a specific code path, a bad config value, or a dependency it calls out to. Keep a running note of what you checked and what you found as you go; when you escalate or hand off, that note is the difference between a colleague joining productively and a colleague re-doing your first five minutes. A third common mistake is treating stakeholder communication as something to do only once you have a root cause; going quiet for the first ten minutes while you investigate leaves support and leadership guessing, and a two-line status update costs far less time than the confusion it prevents.

Direct answer. A 3x jump in p95 with errors rising across several services is a signal that something shared or upstream degraded, not that every service independently broke at once, so the investigation should start by finding what those services have in common rather than debugging them one at a time.

Structured elaboration.

1. Find the common dependency. Check whether the affected services share a downstream call: a database, a cache, an authentication service, a service mesh sidecar, or a shared piece of infrastructure like DNS or a load balancer. Distributed tracing is the fastest tool for this: pull a handful of slow traces from different affected services and see where the time is actually going. If every trace bottoms out in the same downstream span, you've found your candidate cause in minutes instead of hours. Logs from the affected services for the same window are worth pulling alongside the traces, specifically scoped to the shared downstream call: an explicit error, timeout, or connection-refused message in the logs directly confirms what the trace can only imply, and often narrows 'the auth service is slow' down to something as concrete as 'the auth service is throwing a connection-pool-exhausted exception,' which is a more actionable finding than latency alone.
2. Correlate with metrics. Once you have a candidate, check that dependency's own metrics (its error rate, its latency, its saturation) for the same time window. If its p95 or error rate jumped at the same moment your services' did, that's strong corroborating evidence.
3. Run targeted experiments to confirm, not just infer. If you suspect a specific downstream, isolate a request to that path alone (bypassing the rest of the flow if you can) and see whether it reproduces the slowdown. Replaying a sample of real traffic against a canary (a small slice of live production instances running the candidate change) or a shadow environment (a copy of the system that receives a duplicate stream of real traffic but whose responses are discarded, used purely to observe behavior safely) is another way to confirm the same hypothesis without risking more production traffic.
4. Mitigate while you confirm. Depending on what you find, options include failing over to a healthy replica, shedding non-critical load off the shared dependency, or temporarily degrading a feature that depends on the slow path (serving cached or stale data instead of failing).
5. Validate the proposed fix the same way you validated the hypothesis: with a canary, not a blanket rollout. Deploy the candidate fix (a scale-out, a rollback, a config change) to a small slice of the affected service's instances first, and compare that slice's error rate and latency against the untreated rest over the next several minutes before widening it. If the canary slice doesn't recover, the root-cause hypothesis was wrong or incomplete, and that's better discovered on a small slice of traffic than on all of it.

Worked example. Say traces from three of the affected services all show a 300 to 400ms span on a call to the shared authentication service, while every other span in those traces looks normal. You check the auth service's own dashboard and see its p99 latency also jumped 3x at the same timestamp, and its CPU is pegged. That converges on a specific, testable hypothesis (the auth service itself is overloaded) rather than three separate mysteries. A quick check of the auth service's own recent changes and traffic volume tells you whether it's a capacity problem (traffic grew faster than provisioned capacity) or a regression (a recent deploy made it slower per request) so you know whether the mitigation is scaling it out or rolling it back.

Trade-offs and pitfalls. The main trap is debugging each affected service in isolation, which multiplies your investigation time by the number of services and can lead different responders to different, contradictory hypotheses. The other trap is stopping at 'auth service is slow' without confirming WHY, since 'scale it out' and 'roll back the last deploy' are very different mitigations that fix different root causes; guessing wrong burns time you don't have during an active, multi-service incident.

Direct answer. In the first 15 minutes the goal is to stabilize user-facing traffic and preserve evidence, in that order; you are not trying to find the root cause yet, you are trying to stop the bleeding and make sure you (or whoever picks this up) can still find the root cause afterward.

Structured elaboration.

1. Confirm the deploy is actually correlated before assuming it's the cause. Pull up the deploy timeline and compare it against when the 500s started. If a deploy to this endpoint landed in the same window, that is your leading hypothesis, but check it, don't assume it: some deploys are unrelated and the timing is coincidence.
2. Look at the error itself. Open a sample of the failing requests in logs or your APM (application performance monitoring tool, e.g. Datadog or New Relic). A stack trace or specific error code will usually tell you whether this is a code bug in the new release (null pointer, unhandled exception, a broken dependency call) versus something environmental (a config value that didn't get set, a database migration that didn't finish).
3. Check dashboards for blast radius. Is this endpoint's error rate the only thing affected, or is latency also up, are other endpoints degrading too, is a specific region or instance pool worse than others? This tells you whether containment can be narrow (this endpoint only) or needs to be broader.
4. Mitigate, choosing among your real options. A full rollback is usually the fastest, safest action, since patching forward live under pressure is a common source of a SECOND incident. If a full rollback isn't immediately available (say, other changes have shipped on top of it), a canary rollback, reverting just the newest batch of instances back to the prior version while leaving the rest alone, narrows the blast radius while you confirm the fix works. Throttling the affected endpoint (accepting some added latency or rejecting a fraction of requests) is a third lever worth having ready if neither rollback option is immediately deployable, buying time without needing a code or deploy change at all.
5. Preserve evidence as you go. Before you roll back, capture a handful of the actual failing request logs, the relevant dashboard screenshots, and the exact timestamps. Once you roll back, the failure signal disappears and you lose your best evidence for the eventual root-cause writeup.

Worked example. You see the 500s are concentrated in one endpoint, a deploy to that service's code landed 6 minutes before the alert, and a sample of the failing requests shows a NullPointerException in a new code path. That is enough correlation plus evidence to roll back immediately: you trigger the rollback, watch the error rate over the next 2 to 3 minutes, and confirm it returns to baseline. You now have a clean signal (rollback fixed it) and preserved logs, which together make the eventual root-cause writeup straightforward: the new code path didn't null-check a field that's optional in a subset of real traffic.

6. Communicate status to stakeholders throughout, not just at resolution. Post a short update to your incident channel or status page as soon as you've confirmed the correlation in step 1: what's affected, the rough blast radius from step 3, and what mitigation you're about to try. Once you roll back, a second short update (what you did, whether it worked, and what's still open) closes the loop. This matters even inside a 15-minute window, because stakeholders (support, other engineers, sometimes leadership) making decisions off silence or guesswork is a common secondary problem during an incident.

Trade-offs and pitfalls. The instinct to 'fix it properly' under pressure, by patching the bug live rather than rolling back, is usually the wrong call: a patch written and shipped in the middle of an active incident has not been reviewed or tested the way the original deploy was, and a second bad deploy during an active incident is a genuinely common failure pattern. The other pitfall is rolling back so fast that you never capture the failing-request evidence, which leaves the eventual root-cause analysis guessing instead of grounded in what you actually saw. A third pitfall is treating stakeholder communication as a nice-to-have that happens after the fact: going quiet during an active regression, even a short one, tends to generate more anxious pings and duplicated investigation than a two-line status update would have cost.

Direct answer. Logs, metrics, and traces answer different questions: metrics tell you something is wrong and roughly how bad, traces tell you WHERE in a multi-service request the time or error actually happened, and logs tell you the specific WHY once you know where to look; for the given scenario, the efficient order is metrics first, then traces, then logs.

Structured elaboration.

1. Metrics. What they provide: aggregated, numerical signals over time (request rate, error rate, latency percentiles) that are cheap to query and great for detecting that something changed and roughly when. Example symptom best diagnosed with metrics: a gradual latency creep over days, which is hard to spot from individual traces or logs but obvious on a time-series graph. Limitation: metrics are aggregates, so they can't tell you WHICH specific request failed or why; a metric can tell you error rate is 5% but not which five requests out of a hundred, or what made those five different from the other ninety-five.
2. Traces. What they provide: the path of a single request across multiple services, showing where time was spent and where in the chain something failed, which metrics alone can't show for multi-service systems. Example symptom best diagnosed with traces: a specific endpoint is slow, and you need to know whether the time is in your own service, a downstream call, or a database query, the way a span breakdown showed in an earlier question here. Limitation: traces are sampled in most systems at any real scale, so a rare intermittent bug may simply not appear in your trace sample, and traces don't easily show you patterns ACROSS many requests, only individual ones.
3. Logs. What they provide: detailed, often free-text or structured information about exactly what a piece of code was doing at a specific moment, including values, error messages, and stack traces. Example symptom best diagnosed with logs: understanding the EXACT error message or exception behind a failure that a trace has already told you is happening in a specific service. Limitation: logs are voluminous and expensive to search broadly, so they're a poor starting point when you don't yet know where to look; searching all logs for 'something is slow' is far less efficient than searching the specific service's logs once a trace has pointed you there.
4. Applying the order to the given scenario. For 'API service A shows a sustained increase in 5xx rate and p95 latency for the last 10 minutes': start with metrics, since you already have them and they confirm scope, timing, and severity (is it every endpoint or one, is it every region or one, is 5xx rising with latency or independently). Then pull traces for a sample of the actual slow or failing requests during that window, since that narrows WHERE in the request path the problem lives. Only once traces point to a specific service or call would you go read that service's logs for the exact error or exception.

Trade-offs and pitfalls. Starting with logs when you don't yet know where to look is the most common inefficient order: broad log search across many services, with no scope yet established, tends to waste the early minutes of an incident on noise. The reverse mistake, staying in metrics dashboards indefinitely without ever pulling a trace or a log, can leave you with a confirmed symptom and no explanation, since metrics alone rarely tell you the specific mechanism behind a failure in a multi-service system.

Direct answer. The causal chain here is: service-a is calling service-b, which is calling redis, and redis connections are timing out, which stalls service-b, which stalls service-a; the error message ('connect timeout to redis') is the concrete evidence that redis connectivity, not application logic, is the proximate cause.

Structured elaboration.

1. Build the chain from the evidence given, not assumption. The trace shows gateway calling service-a (600ms), which calls service-b (580ms), which calls redis (560ms). Almost all of service-a's 600ms is inside its call to service-b, and almost all of service-b's 580ms is inside its call to redis. Combined with the log line 'ERR connect timeout to redis host:6379' repeated in service-b, the chain reads: redis is not responding to new connections in time, so every request through service-b (and therefore through service-a) pays that timeout cost, and 10% of them fail outright.
2. Distinguish plausible causes of a redis connect timeout. This symptom is consistent with several different underlying problems: redis itself is overloaded or down, the network path to redis is degraded, redis's connection limit has been reached so new connections queue or get refused, or service-b's own connection pool to redis is exhausted or misconfigured (for example too few connections for its current traffic, or connections not being released properly).
3. Use additional signals to narrow it down. Redis's own metrics (CPU, memory, connected-clients count, command latency) would show whether redis itself is unhealthy. Service-b's connection-pool metrics would show whether it's a client-side exhaustion problem rather than a server-side one. A redis-cli PING or a synthetic connection test from the same network path as service-b would show whether the network path itself is the issue, separate from whether redis is overloaded.
4. Immediate remediation. Options depend on what step 3 finds: if redis is overloaded, shedding load or failing over to a replica; if it's a client-side connection-pool exhaustion, increasing the pool size or adding a circuit breaker so service-b fails fast instead of piling up timeouts; if it's the network path, routing around the degraded path if there's an alternative.
5. Long-term fixes. Add a circuit breaker around the redis call so a redis outage degrades service-b gracefully (serving stale or default data, or failing fast) instead of cascading its own latency upstream into service-a and the gateway. Add explicit alerting on redis connection-timeout rate specifically, since it's a distinctive and actionable signal that this trace shows was already present before the wider error rate and latency alerts fired.

Worked example. If redis's own dashboard shows connected_clients pinned at its configured maximum throughout the incident window while CPU and memory on the redis host stay normal, that specifically points to connection exhaustion rather than redis being overloaded: either too many clients are opening connections without closing them (a leak somewhere, possibly in service-b or another caller), or redis's maxclients setting is simply too low for current legitimate traffic. Checking service-b's own connection-pool configuration would tell you which: if service-b's pool size looks reasonable and connections are being released properly, the fix is raising redis's maxclients; if service-b is opening far more connections than its configured pool size suggests it should, the fix is finding and closing the leak in its client code.

Trade-offs and pitfalls. The trap in this kind of RCA is stopping at 'redis connect timeout' and treating that as the root cause, when it's really a symptom with at least four different plausible underlying causes; each one implies a different fix, and applying the wrong one (for example scaling up redis when the real problem is a client-side connection leak) won't resolve the incident. It's also worth being explicit in the writeup about WHY the fix you propose follows from the evidence you gathered, not just from a guess, since that's what lets someone else validate or challenge the conclusion.