Production Incident Diagnosis and Distributed Systems Troubleshooting Questions

Direct answer. A region-wide network partition means the on-call engineer is dealing with genuine uncertainty about which side of the partition owns truth, so the runbook has to prioritize NOT making the split worse before it prioritizes restoring full service.

Structured elaboration.

1. Immediate mitigation steps. Confirm the partition's scope first (is it truly the whole region, or a subset of services within it) using an independent monitoring path that doesn't itself depend on the partitioned region, since monitoring that routes through the affected region can give a false picture. If there's a designated failover region or a documented authoritative side for this partition scenario, follow that designation rather than deciding ad hoc under pressure; if there isn't, that's itself a gap this incident should surface.
2. Prioritized checks. Which services have region-local state that could diverge if both sides keep accepting writes (the split-brain risk); which services are stateless or read-only and can safely continue serving from either side without correctness risk; and whether any cross-region dependency (a shared queue, a shared database) is itself affected by the partition or still reachable from one or both sides.
3. Escalation paths. Who owns the decision to formally fail over the region (this is usually a decision above a single on-call engineer for anything with real data-correctness stakes), and what's the communication chain to notify affected teams and, if customer-facing impact is significant, support or communications teams.
4. Recovery validation steps once the partition heals. Before declaring the incident over, explicitly check for divergence: did both sides of the partition accept writes to the same data during the split, and if so, has that been reconciled (following the same logic as any split-brain reconciliation) before resuming normal, unrestricted operation. Confirm no data loss by comparing a checksum or count of critical data against what was expected, not just by observing that services report healthy again.

Worked example. A concrete instantiation: region A and region B lose connectivity between them for 12 minutes. The runbook's first check (independent, cross-region monitoring) confirms it's a true regional partition affecting all inter-region traffic, not a single service issue. Following the documented authoritative-side designation, region A continues accepting writes for the shared data store while region B is placed into a read-only, degraded mode for anything requiring cross-region coordination. Stateless read services in both regions continue serving local traffic normally throughout. Once connectivity restores, before lifting region B's read-only mode, a reconciliation check confirms no writes were accepted on B's side during the partition (since it was correctly held read-only) and no data loss occurred; if B HAD somehow accepted writes despite the read-only mode (a bug in the mode's enforcement, for example), that would be a second, more serious finding requiring the split-brain-style reconciliation before recovery is complete.

Trade-offs and pitfalls. The instinct to restore full service everywhere as fast as possible has to be weighed against the risk of both sides having independently accepted writes, since resuming full bidirectional operation before confirming no divergence occurred can permanently and silently corrupt data; a slightly slower, verified recovery is almost always the better trade for anything with real correctness stakes. It's also worth pre-deciding the authoritative-side designation and the stateless-vs-stateful service list BEFORE an incident, in the runbook itself, since deciding either under active pressure is slower and more error-prone than following a pre-made call.

Direct answer. This is a genuinely counter-intuitive result, since adding instances should spread load and reduce latency, so the debugging checklist has to specifically look for mechanisms where MORE instances create MORE overhead or contention rather than simply assuming the scaling itself is broken.

Structured elaboration.

1. Shared-resource contention. If the new instances all compete for the same downstream resource (a database, a cache, a shared connection pool with a fixed total size), adding instances doesn't add capacity to that shared resource, it just adds more competitors for the same fixed pie; check whether a shared downstream's own load or connection count grew proportionally to the new instance count, and whether ITS latency (not just this service's) got worse at the same time.
2. DNS or iptables-level issues. A load balancer or service-discovery mechanism that takes non-trivial time to register or fully propagate new instances can cause uneven traffic distribution during scale-up (some instances overloaded while others are still ramping up); at the OS level, an inefficient iptables ruleset (iptables is the Linux kernel's packet-filtering and routing system, commonly used under the hood to implement service load-balancing rules in container networking) can scale poorly with a growing number of backend targets, adding real per-packet overhead as the instance count grows.
3. Connection-pool exhaustion, from a different angle than shared-resource contention: if EACH instance opens its own pool of connections to a downstream, more instances can mean MORE total connections than the downstream can handle, even if each instance's own pool looks reasonably sized in isolation.
4. Leader-election thrashing, if this service participates in any kind of coordination or leader election: more instances competing for leadership, or more instances triggering more frequent rebalancing in whatever coordination mechanism is in use, can itself consume real resources and add latency, especially if the coordination overhead scales poorly with instance count.
5. Gather evidence for each candidate rather than guessing. For each of the above, there's a specific, checkable signal: shared-resource metrics correlated with instance count and latency; load-balancer target-registration timing and per-instance traffic distribution; per-instance versus aggregate connection counts against the shared downstream's own limits; and coordination-service metrics (rebalance frequency, election frequency) if relevant.
6. Remedial actions per cause. Shared-resource contention needs the shared resource itself scaled or partitioned, not just the calling service. Load-balancer or iptables issues need investigation of the specific networking layer causing the overhead, which may mean a different load-balancing algorithm or container-networking mode. Connection-pool exhaustion needs a TOTAL connection budget considered across all instances, not just per-instance. Leader-election thrashing needs either fewer participants in the coordination (a subset act as candidates, not every instance) or a coordination mechanism that scales better.

Worked example. Suppose per-instance connection-pool size is a fixed 20 connections to a shared database, and the database's own max-connections limit is 200; at 8 instances, that's 160 total possible connections, comfortably under the limit. Scaling to 15 instances pushes the theoretical maximum to 300, which exceeds the database's 200-connection limit; once actual usage approaches that ceiling, connections start queueing or getting rejected, and EVERY instance (not just the newest ones) experiences worse latency waiting for a connection slot, which is exactly the paradox described: more instances made the shared resource, not any individual instance, the bottleneck. The fix is either reducing per-instance pool size as instance count grows (keeping the total bounded) or increasing the database's connection limit and capacity to match, with the total connection budget explicitly tracked as a fleet-wide constraint rather than a per-instance setting nobody re-evaluates as the fleet grows.

Trade-offs and pitfalls. The core lesson worth internalizing here is that per-instance settings (pool sizes, timeouts, cache sizes) that look reasonable in isolation can become a fleet-wide problem purely from being multiplied across a growing instance count; any config that's 'per instance' should be evaluated against what happens at your MAXIMUM realistic instance count, not just today's count. It's also worth being skeptical of your own instinct here: 'add more instances' is such a common, usually-correct scaling response that it's easy to reach for it again as the FIX when it was actually the trigger.

Direct answer. etcd is a distributed key-value store (used, for example, inside Kubernetes) where a cluster of nodes elects a single leader via a consensus protocol and stays in sync using periodic heartbeats between members. Leader-election flapping means the cluster can't settle on a stable leader, and the three usual suspects (network partitions, clock skew, resource exhaustion) each break a different assumption the election protocol depends on, so the diagnosis is about figuring out which assumption is actually being violated.

Structured elaboration.

1. Check for network partitions or flakiness between nodes first, since this is the most common cause: intermittent packet loss or latency spikes between cluster members can cause a leader to miss enough heartbeats that followers time out and call a new election, even though the leader itself is otherwise healthy. Cluster-internal network metrics (round-trip time and loss between specific node pairs) and the etcd cluster's own peer-communication logs are the first place to look.
2. Check clock skew across nodes. Election timeouts are time-based; if nodes' clocks have drifted apart meaningfully, their sense of 'how long since I heard from the leader' can disagree, triggering elections that a properly synchronized cluster wouldn't. NTP (Network Time Protocol) sync status and drift metrics on each node settle this quickly.
3. Check resource exhaustion on the current or candidate leader nodes. A leader that's CPU-starved, I/O-starved (etcd is sensitive to disk write latency specifically, since it writes to its log on every proposal), or memory-pressured can become too slow to send heartbeats within the expected interval, which followers interpret as a dead leader.
4. Correlate the timing of flapping events against each of these three candidate signals, rather than checking them in isolation; the one whose anomalies line up with the actual election timestamps is your answer.
5. Hardening, once you know the cause. Tuning election timeouts (raising them, within reason, to be more tolerant of transient blips) is a safe, non-disruptive first lever if the underlying cause is intermittent and hard to eliminate outright. Isolating resources (dedicated, unshared disks and CPU for the cluster's data directory, if resource contention is the cause) addresses exhaustion directly. Applying QoS or prioritization for cluster-internal traffic addresses network-flakiness causes when the flakiness comes from contention with other traffic on a shared link, not an actual outage.
6. Escalation. Tuning timeouts and checking metrics are non-disruptive and safe to do live. Rolling restarts of cluster members are more disruptive (a restart itself can trigger another election) and should be reserved for cases where you've confirmed a SPECIFIC node is unhealthy and restarting it is the actual fix, not a blind first response to flapping.

Worked example. Suppose peer round-trip-time metrics show the link between two specific nodes spiking to 300 to 500ms intermittently, well above the cluster's configured election timeout, while clock-drift metrics across all nodes stay under a few milliseconds and CPU/disk metrics look unremarkable. That converges on network flakiness between those two specific nodes as the cause, not clock skew or resource exhaustion. If those two nodes happen to be in different racks or availability zones sharing a link with other, unrelated traffic, the fix might be as simple as confirming whether that link is oversubscribed at the times flapping occurs, alongside a safe, immediate mitigation of modestly raising the election timeout to tolerate the observed 300 to 500ms spikes without triggering unnecessary elections.

Trade-offs and pitfalls. Raising the election timeout too far trades election flapping for slower failover when there's a GENUINE leader failure, since the cluster will now wait longer before noticing; the right value should be set based on the actual observed latency distribution between nodes, with margin, not an arbitrary large number to make the symptom go away. Restarting nodes as a first response, before you've identified which node (if any) is actually unhealthy, risks making things worse by triggering additional elections during the restart itself.

Direct answer. Once split-brain has happened, the priority order is stop the bleeding, reconcile what already diverged, and only then fix the underlying mechanism that allowed two leaders to exist at once, in that order, because reconciling while divergence is still ongoing means reconciling against a moving target.

Structured elaboration.

1. Immediate containment: stop further divergence first. Identify which replica should be authoritative going forward (often, but not always, the one with the majority of recent, valid writes, or the one that still has quorum support) and fence off the other: stop it from accepting further writes, ideally by revoking its ability to do so at the network or application level, not just asking it nicely to stop.
2. Reconciliation plan for the writes that already diverged. For each conflicting write, decide programmatically where possible (a deterministic rule: last-writer-wins by a reliable timestamp, or a domain-specific merge rule if the data supports one, like a counter that can be summed rather than overwritten) and manually where the rule genuinely can't decide (two conflicting updates to the same field with no clear ordering and no valid merge). Automation should handle the common, decidable cases; manual review should be reserved for genuinely ambiguous ones, since forcing every conflict through manual review doesn't scale and forcing every conflict through blind automation risks silently discarding a valid, important write.
3. Long-term architecture changes to prevent recurrence. Quorum writes (requiring a majority of nodes to acknowledge a write before it's considered committed) make it much harder for two partitions to both believe they have authority, since only one side of a partition can typically achieve a majority. Fencing tokens (a monotonically increasing token issued to whichever node believes it's the leader, checked by anything the leader writes to, so a stale former leader's writes are rejected even if it doesn't yet know it's been deposed) directly prevent a demoted leader from causing damage even if it hasn't gotten the message. Stronger leader election reduces how often the cluster ends up in an ambiguous state to begin with: a protocol that requires a candidate to hold a stable quorum for a full timeout window before it's allowed to become leader, rather than flipping leadership on the first missed heartbeat, means a brief network hiccup or a few dropped packets doesn't cause two nodes to each independently conclude they're now in charge.
4. Validation after reconciliation. Run a data-integrity check across the previously diverged replicas to confirm they now agree; check for orphaned or duplicate side effects that might have resulted from writes made during the split (for example, a payment or notification triggered by BOTH sides during the divergence window); and confirm the split-brain condition genuinely can't recur under the same trigger (a repeat of whatever caused the two sides to both believe they had authority) before declaring the incident closed.

Worked example. Suppose during a 5-minute partition, region A's replica accepted 40 writes and region B's replica accepted 25 writes to overlapping keys, with clear, reliable timestamps on all of them. Applying a last-writer-wins rule programmatically resolves the roughly 55 non-conflicting writes (where the two regions touched different keys) automatically and immediately; of the remaining conflicts on the same keys, suppose 8 have a clear timestamp winner and can also be resolved automatically, while 2 involve writes so close in time (within the same few milliseconds, near the limits of clock synchronization) that timestamp ordering isn't trustworthy, and those 2 go to manual review. That's a concrete, auditable split: the vast majority resolved deterministically and quickly, with a small, genuinely ambiguous remainder handled by a human who can look at the actual business context (which write reflects what the customer actually intended).

Trade-offs and pitfalls. Automated last-writer-wins is fast but can silently discard a legitimate write if two updates happen close enough together that clock precision can't reliably order them, which is exactly why a fallback to manual review for near-simultaneous conflicts matters rather than trusting automation universally. Quorum writes trade some availability (a partition that leaves neither side with a majority means NEITHER side can accept writes, which is a deliberate, safer failure mode) for the strong guarantee that split-brain of this kind becomes structurally much harder to reach in the first place.

Direct answer. p99 latency has four common suspects (GC, or garbage collection, pauses; lock contention; network wait; downstream calls), and the fastest way to distinguish them is to collect data that isolates TIME SPENT rather than guess from the symptom alone, starting with whatever's cheapest to check and narrowing from there.

Structured elaboration.

1. Start with what you already have: request-level tracing. If spans show most of the request's time inside a single downstream call, that points at downstream calls directly and you can stop chasing GC or locks; if the time is spent inside your own service's code with no obvious downstream span accounting for it, that points at GC, lock contention, or CPU-bound work inside the process itself.
2. If it's inside the process, check GC metrics next, since they're usually the cheapest to pull: GC pause time and frequency, correlated against the same time window as the latency spike. A JVM (Java Virtual Machine) or similar managed runtime, such as the .NET CLR or the Go runtime, typically exposes this directly. If GC pauses spike in the same window as p99 latency, that's a strong lead.
3. If GC looks clean, check for lock contention. Thread-dump sampling (or a profiler's lock-contention view) during the bad window shows whether threads are spending time BLOCKED waiting for a lock rather than doing work. This is a common cause of tail latency specifically (not average latency) because it only bites when enough concurrent requests collide on the same lock at once.
4. Check network wait separately from downstream-call latency itself, since they're different things: network wait is time lost to the network layer (DNS resolution, TCP handshake, TLS negotiation, or packet loss causing retransmits) as distinct from the downstream service's own processing time. Span attributes that separate 'time to first byte' from 'connection setup time' help distinguish these.
5. Order of investigation, in practice: traces first (cheap, and often points straight at the answer), then GC (cheap to check, common cause), then lock contention (needs a profiler or thread dump, more involved), then network-level (needs packet-level tools, most involved), unless traces already pointed you at one of these directly.

Worked example. Say traces show no single downstream call accounts for the tail latency, but GC metrics show pause times jumping from a typical 10 to 15ms to occasional 400 to 600ms pauses during the bad window, and those pauses' frequency correlates with a recent increase in heap allocation rate (visible as a steeper slope on the allocated-bytes metric). That's consistent with a garbage collector doing more frequent full collections because allocation pressure increased, likely from a recent code change that allocates more per request than before. The fix path is either reducing allocation in the hot path or tuning the collector's heap sizing and generation thresholds (the size limits that decide when the collector promotes an object from the fast-collected 'young generation' to the less-frequently-collected 'old generation') to handle the new allocation rate without such long pauses, and you'd validate by checking that p99 latency and max GC pause time both drop after the change, in the same dashboard you used to spot the correlation.

Trade-offs and pitfalls. The temptation is to check the cheapest signal (metrics) and stop as soon as you see ANY anomaly, but a GC metric that looks slightly elevated might be a red herring if it doesn't actually correlate with the specific timestamps of the latency spikes; always confirm the timing lines up, not just that both look 'bad'. It's also easy to conflate lock contention and GC pauses since both show up as 'CPU-bound work stalling', but they need very different fixes (reducing critical-section size or lock granularity versus tuning collector behavior or reducing allocations), so profiler-level evidence matters more than a guess based on which one you've seen before.