Production Incident Diagnosis and Distributed Systems Troubleshooting Questions

Direct answer. etcd is a distributed key-value store (used, for example, inside Kubernetes) where a cluster of nodes elects a single leader via a consensus protocol and stays in sync using periodic heartbeats between members. Leader-election flapping means the cluster can't settle on a stable leader, and the three usual suspects (network partitions, clock skew, resource exhaustion) each break a different assumption the election protocol depends on, so the diagnosis is about figuring out which assumption is actually being violated.

Structured elaboration.

1. Check for network partitions or flakiness between nodes first, since this is the most common cause: intermittent packet loss or latency spikes between cluster members can cause a leader to miss enough heartbeats that followers time out and call a new election, even though the leader itself is otherwise healthy. Cluster-internal network metrics (round-trip time and loss between specific node pairs) and the etcd cluster's own peer-communication logs are the first place to look.
2. Check clock skew across nodes. Election timeouts are time-based; if nodes' clocks have drifted apart meaningfully, their sense of 'how long since I heard from the leader' can disagree, triggering elections that a properly synchronized cluster wouldn't. NTP (Network Time Protocol) sync status and drift metrics on each node settle this quickly.
3. Check resource exhaustion on the current or candidate leader nodes. A leader that's CPU-starved, I/O-starved (etcd is sensitive to disk write latency specifically, since it writes to its log on every proposal), or memory-pressured can become too slow to send heartbeats within the expected interval, which followers interpret as a dead leader.
4. Correlate the timing of flapping events against each of these three candidate signals, rather than checking them in isolation; the one whose anomalies line up with the actual election timestamps is your answer.
5. Hardening, once you know the cause. Tuning election timeouts (raising them, within reason, to be more tolerant of transient blips) is a safe, non-disruptive first lever if the underlying cause is intermittent and hard to eliminate outright. Isolating resources (dedicated, unshared disks and CPU for the cluster's data directory, if resource contention is the cause) addresses exhaustion directly. Applying QoS or prioritization for cluster-internal traffic addresses network-flakiness causes when the flakiness comes from contention with other traffic on a shared link, not an actual outage.
6. Escalation. Tuning timeouts and checking metrics are non-disruptive and safe to do live. Rolling restarts of cluster members are more disruptive (a restart itself can trigger another election) and should be reserved for cases where you've confirmed a SPECIFIC node is unhealthy and restarting it is the actual fix, not a blind first response to flapping.

Worked example. Suppose peer round-trip-time metrics show the link between two specific nodes spiking to 300 to 500ms intermittently, well above the cluster's configured election timeout, while clock-drift metrics across all nodes stay under a few milliseconds and CPU/disk metrics look unremarkable. That converges on network flakiness between those two specific nodes as the cause, not clock skew or resource exhaustion. If those two nodes happen to be in different racks or availability zones sharing a link with other, unrelated traffic, the fix might be as simple as confirming whether that link is oversubscribed at the times flapping occurs, alongside a safe, immediate mitigation of modestly raising the election timeout to tolerate the observed 300 to 500ms spikes without triggering unnecessary elections.

Trade-offs and pitfalls. Raising the election timeout too far trades election flapping for slower failover when there's a GENUINE leader failure, since the cluster will now wait longer before noticing; the right value should be set based on the actual observed latency distribution between nodes, with margin, not an arbitrary large number to make the symptom go away. Restarting nodes as a first response, before you've identified which node (if any) is actually unhealthy, risks making things worse by triggering additional elections during the restart itself.

Direct answer. With limited resources and a live cascading failure, the decision framework should weigh speed of relief against risk of losing work, in that order of urgency, since a model that's already causing outages downstream needs the fastest safe reduction in its blast radius, with the fully correct long-term fix as a secondary concern for right now.

Structured elaboration.

1. Prioritized decision framework. First: does the cascading failure pose an active, worsening risk to other services (in which case speed dominates every other consideration)? If yes, immediate rollback is usually the safest default, since it's typically the fastest way to guarantee the excess calls stop, and it's a well-understood, low-risk operation compared to a live code patch. Second, if rollback isn't immediately available or would itself be risky (say, the previous model version is known to have its own separate issues), throttling the model's call volume to the upstream dependency directly addresses the cascading mechanism without needing to fully revert the model. Third, patching the serving code (adding a cache, a rate limiter, or a circuit breaker around the specific upstream call) is the most targeted fix but takes the longest to build and test safely, making it usually the wrong choice for stopping an ACTIVE cascade, better suited as the follow-up once the immediate risk is contained.
2. Short-term mitigations to list and weigh, given limited engineering resources specifically. Rollback: fast, low custom-engineering effort, but loses whatever improvement the new model version provided. Throttling: moderate effort (may require a config change or a simple gate, not a full redeploy), keeps the new model largely in place but caps its potential value while capped. Patching: highest effort, most precise, but slowest and riskiest to build correctly under time pressure with limited resources.
3. Estimate probable outcomes under uncertainty. Rollback: high confidence the cascade stops (since it removes the triggering behavior directly), moderate cost (losing the new model's benefit until a proper fix ships). Throttling: moderate confidence the cascade eases (depends on whether the throttled volume is actually low enough to stop overloading the downstream), lower cost (keeps most of the model's value). Patching under time pressure: lower confidence it actually works correctly on the first attempt (a rushed fix has real risk of its own bugs), and attempting it while resources are limited and the incident is active adds risk of a SECOND incident from an untested change.
4. Choose based on the specific uncertainty you're facing. If you're highly confident throttling to a specific, known-safe volume will stop the cascade (because you understand the mechanism precisely), it's often the best balance of speed and value preservation. If you're not confident in that mechanism, or the downstream impact is severe enough that any residual risk is unacceptable, rollback is the safer default despite losing the new model's value temporarily.

Worked example. Suppose the new model version calls an enrichment service roughly 3 times more per request than the previous version (due to a new feature requiring additional lookups), and that enrichment service's capacity was sized for the old model's call volume; if you can confidently throttle the new model's calls to that specific service back down to roughly the old rate (say, via a simple per-model rate limit at the enrichment service's gateway), that directly and precisely addresses the cascading mechanism without losing the new model's other improvements entirely. If, however, the exact multiplier or mechanism isn't clearly understood yet, and the enrichment service is critical to multiple OTHER dependent services beyond just this cascade, the safer call under that uncertainty is a full rollback: guaranteed to stop the excess calls, at the acknowledged cost of losing the new model's benefits until a proper, tested fix (like a smarter caching layer for the enrichment lookups) can ship without time pressure.

Trade-offs and pitfalls. The core judgment call is how confident you are in the PRECISE mechanism before choosing a mitigation more targeted than rollback; a wrong guess about the mechanism, acted on under limited resources and time pressure, risks not fixing the cascade at all while still consuming the scarce engineering time you have. It's worth being honest with stakeholders about this trade-off explicitly: 'we're choosing rollback because we're not yet confident enough in throttling to bet the ongoing cascade on it' is a defensible, transparent decision, more so than a confident-sounding guess that turns out wrong.

Direct answer. When p99 is bad but the median is fine, the problem is affecting a MINORITY of requests in a specific way, so averaging or median-based dashboards will actively hide it; the investigation has to specifically pull out the slow tail and ask what's different about those requests.

Structured elaboration.

1. Isolate the slow requests specifically, don't look at aggregate dashboards. Filter your tracing or logging to just the requests above, say, the 95th or 99th percentile latency threshold, and look for what they have in common: a specific endpoint, a specific customer or tenant, a specific payload size or shape, a specific instance or availability zone, or a specific time-of-day pattern.
2. Check for resource contention that only bites intermittently. A thread pool or connection pool that's usually adequate can occasionally saturate under a burst, causing a small fraction of requests to queue while most sail through; this produces exactly a good-median-bad-tail signature. Garbage-collection pauses have the same shape: most requests are unaffected, a small fraction land during a pause and get delayed.
3. Check for a specific slow dependency call that only some requests make. If a feature flag, a conditional code path, or a specific customer's data shape triggers an extra downstream call (a cache miss requiring a database lookup, for example) for only a subset of requests, the median (mostly fast, cache-hit requests) looks fine while the tail (the subset needing the extra call) is slow.
4. Reproduce under load, targeting the specific pattern found in step 1. Once you have a hypothesis (say, large payloads are slow), a load test that specifically varies that dimension can confirm it without waiting for it to reoccur naturally in production.
5. Validate the fix without risking more user impact. Roll a fix out to a small percentage of traffic first and specifically watch the SAME tail-latency metric you used to find the problem, not just the aggregate, since a fix could improve the median while barely touching the tail if you mis-diagnosed the cause.

Worked example. Say filtering to the slowest 1% of requests shows they're disproportionately concentrated on one specific API key, and further digging shows that customer's requests average 50 times larger payloads than typical traffic (large batch uploads versus the typical small requests). The service's payload-parsing step is O(n) in a way that's invisible at typical sizes but adds real milliseconds at 50x the size, and if that customer represents roughly 1 to 2% of total request volume, that lines up closely with a p99 (but not median) impact. The mitigation is either optimizing the parsing step's complexity or, as a faster stopgap, routing especially large payloads to a separate processing path so they don't compete for the same resources as typical small requests.

Trade-offs and pitfalls. The trap here is trusting a healthy-looking median or average as evidence nothing is wrong; those aggregates are exactly the metric that HIDES this class of problem by construction, since a small fraction of slow requests barely moves the median at all. It's also worth being skeptical of 'it's probably just noise' when the tail is consistently bad rather than randomly bad: consistent tail badness usually has a specific, findable cause, while genuinely random noise (network jitter, for example) tends to look different in the data.

Direct answer. With two independent systems down at once, the immediate priority is stopping further damage from EACH failure separately, since they likely require different mitigations, before attempting any cross-system reconciliation, which only makes sense once both systems are stable again.

Structured elaboration.

1. Immediate mitigations, one per system. For the message broker: pause or buffer producers if possible (rather than letting them fail loudly or silently drop messages) while the broker is unavailable, and identify a fallback path if one exists (a secondary broker, or a degraded synchronous mode for the most critical messages only). For the cache: fail over to a source-of-truth read path (the database directly) for critical reads, accepting higher latency and load on the database temporarily rather than serving nothing or serving from a broken cache.
2. Steps to restore each service, prioritized by which is more critical to core functionality and which is faster to restore; restoring them doesn't need to happen simultaneously, and restoring the more critical one first, even if the other stays degraded a bit longer, is a legitimate prioritization call.
3. Reconcile state between systems, once both are back. The message broker's outage likely means some messages were never delivered (or were queued and are about to be delivered late, out of their original order relative to other system state); the cache's outage means the database was serving all reads directly during the outage, so the cache, once restored, needs to be treated as fully stale and either invalidated entirely or warmed fresh rather than trusted to reflect current state.
4. Deduplicate side effects. Anything that happened during the hour needs auditing for duplicate processing: did the broker's recovery cause any messages to be redelivered that had already been processed via a fallback path during the outage, and did any process see stale cached data and take an action based on it that's now inconsistent with the database's actual current state.
5. Validation checks post-recovery. Confirm the broker's queue depth and consumer lag return to normal, confirm the cache's hit rate and served-data freshness look correct (not still serving anything stale from before the outage), and specifically audit for duplicate or inconsistent side effects from the dual-failure window before declaring the incident fully resolved, not just 'services are up'.

Worked example. Suppose during the hour-long outage, a fallback path processed a subset of critical messages synchronously (bypassing the broker entirely) to avoid losing them, and separately, reads fell back to the database directly once the cache was known to be down. Once the broker recovers, it may still have QUEUED versions of those same messages that were already handled via the synchronous fallback, which would cause duplicate processing unless each message carries an idempotency key that the consumer checks before acting; if that idempotency key exists and is checked, redelivery is safe and requires no special reconciliation. If it does NOT exist, an explicit reconciliation pass, cross-referencing what the fallback path already processed against what the broker is now redelivering, is required before letting the broker's backlog drain automatically, to avoid, for example, double-charging a customer or double-sending a notification.

Trade-offs and pitfalls. Restoring both systems and immediately resuming full, unrestricted normal operation without a reconciliation pass risks exactly the duplicate-processing and stale-data problems this scenario sets up; the temptation to declare victory the moment both systems report healthy again is real, especially after an hour of firefighting, but the validation and reconciliation step is what actually confirms the incident is over rather than just the immediate symptoms disappearing. It's also worth treating this dual-failure as a prompt to check whether the two systems have any DEPENDENCY on each other's health that wasn't obvious before (did the cache failure make the broker situation worse, or vice versa) since understanding that interaction matters for preventing a similar dual-failure in the future.

Direct answer. A 504 specifically means the gateway or load balancer gave up waiting on an upstream response, so the investigation should center on finding which hop in the chain is actually slow or unresponsive, not on the gateway itself, which is usually just the messenger.

Structured elaboration.

1. Confirm where the timeout is actually happening. Check the gateway or load balancer's own timeout configuration and logs: a 504 tells you the gateway gave up, but not which upstream it gave up on. If you have distributed tracing, pull a handful of the failing requests and look at which span never completes or takes far longer than normal.
2. Check load and concurrency on the microservice itself. Look at request rate, thread-pool or connection-pool saturation, and queue depth. A service that's fine at low traffic can start timing out purely from being overwhelmed, well before it would show as unhealthy on CPU or memory.
3. Check its own downstream calls. If the microservice calls a database, cache, or another service, check THOSE for elevated latency; the microservice may be blocked waiting on something further down the chain rather than being slow itself.
4. Review the code for the specific bugs that turn a slow dependency into a hung request. Three areas are worth checking directly: whether the client for each downstream call has an explicit timeout set at all (a client with no timeout can hang indefinitely, which is a different and worse bug than one with a merely generous timeout); whether the connection or thread-pool size is hardcoded or read from a config value that may not have been updated as traffic grew; and whether there's a retry-without-backoff loop anywhere in the request path that would amplify load on an already-slow dependency instead of failing fast.
5. Reproduce under load if you can. A quick load test against a staging or canary instance, mimicking the traffic pattern that triggers the 504s, can confirm a capacity-related hypothesis in minutes rather than guessing from production alone.
6. Mitigate. Immediate options include raising the gateway timeout slightly if the upstream is close to finishing anyway (a stopgap, not a fix), adding capacity if it's a load problem, or failing fast with a clear error instead of hanging if the root cause is a stuck downstream dependency.

Worked example. Traces show requests hanging in a call from the microservice to its database, with the query itself sometimes taking 8 to 10 seconds under load versus under 200ms normally. Checking the database's own connection pool shows it's fully saturated during the incident window. That converges on a specific, testable hypothesis: the service's connection pool to the database is too small for current traffic, requests queue up waiting for a connection, and eventually the gateway's shorter timeout fires before the query even starts. The fix is raising the pool size (or adding a circuit breaker so requests fail fast instead of queuing indefinitely) rather than anything at the gateway layer.

Trade-offs and pitfalls. A common mistake is treating the 504 itself as the problem and only tuning the gateway timeout, which can mask a real capacity or dependency issue and just delays the same failure. The other pitfall is not distinguishing 'slow' from 'stuck': a slow downstream call that eventually returns is a capacity problem you can often scale your way out of, but a stuck call that never returns (a connection that hangs instead of timing out) needs a timeout and circuit breaker at every hop, or one hung dependency can eventually exhaust every caller's resources.