Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer

Structure the program as a pyramid of increasing blast radius: start with single-instance, single-connection experiments in a canary slice of traffic, and only widen scope once the previous step stayed green. Every experiment, regardless of layer, needs the same three things stated up front: a falsifiable hypothesis (what should happen if the system is as resilient as you believe), a blast-radius control (the mechanism that caps how much traffic or infrastructure the experiment can touch), and a rollback criterion (the automated trigger that aborts the experiment before it becomes an incident).

Program structure by layer

The pyramid runs left to right within a layer, and layer to layer (service before database before network) because a service-level failure is the easiest to reason about and the cheapest to roll back; database and network faults touch more of the system at once and take longer to reverse cleanly.

Execution discipline

• Pre-flight: a written runbook, on-call and stakeholders notified, an automated abort mechanism wired to the rollback criteria (not a human watching a dashboard and deciding), and the experiment coded as a reproducible, version-controlled script rather than an ad hoc manual action.
• During: watch the rollback-criteria metrics in real time; the abort has to be automatic and fast, because by the time a human notices a metric crossing threshold and manually intervenes, the blast radius has often already grown past what the control was meant to cap.
• After: a lightweight postmortem regardless of outcome (a clean pass is still evidence worth recording), and only widen scope for the next run once the current one is unambiguously green, not "green with an asterisk."

Trade-offs & pitfalls

The single most common mistake is skipping straight to a production-wide experiment because a staging environment "doesn't reproduce the failure mode," which is often true but doesn't change the fact that the first production run of any new fault type belongs in the smallest blast radius you can construct, even if that means accepting a less realistic signal initially. A second pitfall is defining rollback criteria in terms of the fault itself (for example, "abort if packet loss exceeds 2 percent") instead of user-facing impact (error rate, latency, success rate); the fault is the input you're controlling, the rollback trigger has to watch the output, or you can hit exactly your intended fault level while still causing an unacceptable customer-facing outage. Not every fault type generalizes across domains the same way either: a GPU training job's most dangerous failure mode isn't a crashed worker (checkpointing handles that cheaply) but silent numerical divergence (the training job keeps running, but silently starts computing mathematically wrong updates to the model, with no crash or error to announce it), where the job keeps running and producing wrong gradients (a gradient is the per-step adjustment the training process makes to the model's internal numbers; a wrong one nudges the model in a bad direction instead of a good one) with no immediate error signal, so the safe blast-radius control there is different in kind from an HTTP service's traffic-percentage cap: it's about capping how long a divergence can run undetected before an automated metric check, watching a loss curve (a plot of the model's error over time, which should trend down) or a gradient norm (a single number summarizing how large the model's updates are; a sudden spike signals training has gone unstable), kills the job, not about capping how many requests are affected. A resilience program that only ever tests one fault type at a time also under-tests: real incidents are frequently two failures at once (a network blip during a deploy, a slow dependency during a traffic spike), so a mature program's later stages deliberately combine fault types once single-fault experiments across all three layers are consistently passing.

Direct answer: Single-leader replication routes all writes through one node and copies them out to followers, giving strong consistency on the leader but a failover gap if it dies. Multi-leader replication lets several nodes accept writes independently and merge them later, trading consistency for local write availability. Quorum-based replication has no fixed leader; reads and writes each require acknowledgment from a configurable subset of replicas, and the overlap between those subsets is what determines the consistency guarantee.

Structured elaboration

The quorum math that determines consistency: for $N$ replicas, a write quorum of $W$ nodes and a read quorum of $R$ nodes, the system guarantees a read overlaps with the most recent write whenever

This is a direct pigeonhole argument: if $W$ and $R$ are subsets of the same $N$-replica set and $|W| + |R| > N$, they cannot be disjoint (two disjoint subsets can sum to at most $N$ elements total), so they must share at least one replica, and that shared replica has both the latest write and is included in the read.

Worked example with $N=3$:

• $W=2, R=2$: $W+R=4 > 3$, so every read quorum is guaranteed to overlap every write quorum by at least one replica. This gives strong (read-your-writes) consistency, at the cost of needing acknowledgment from 2 of 3 replicas on both reads and writes.
• $W=1, R=1$: $W+R=2 \leq 3$, no overlap is guaranteed. A write can land on replica A while a read is served entirely from replica B, missing it. This is fast (single-replica round trip) but only eventually consistent.

Trade-offs & pitfalls

• Single-leader is the simplest to reason about and the default choice unless you have a specific reason not to use it; its main weakness is the failover window (detecting the leader is gone and safely promoting a replacement), not steady-state operation.
• Multi-leader avoids that failover gap for writes but pushes complexity into conflict resolution; it's the right choice specifically when you need low-latency local writes at multiple sites and can tolerate (or algorithmically resolve) concurrent edits, not as a general-purpose upgrade over single-leader.
• Quorum systems let you dial the $W$/$R$ trade-off per workload (e.g., $W=1$ for a write-heavy, tolerant-of-staleness workload; $W=N$ for a read-heavy workload that wants every read to be a single, fast, guaranteed-fresh replica read), but that tunability is also a footgun: teams often ship with $W+R \leq N$ by default (e.g., both set to 1 for speed) without realizing they've silently given up the consistency guarantee they assumed they had.
• A common wrong turn: treating "quorum-based" as automatically stronger than single-leader. With $W+R \leq N$ it is weaker, not stronger, than a single-leader system with synchronous replication to at least one follower.

Direct answer: I'd layer health signals from cheap-and-fast (process liveness) to expensive-and-meaningful (dependency and business-metric checks), require multiple consecutive failures before declaring a node unhealthy to avoid reacting to a single blip, and keep a human in the loop specifically for the step that's hardest to reverse: promoting a new primary for stateful services, where an incorrect automated failover can cause data loss or a split-brain (two nodes each believing they're the one true primary and accepting conflicting writes at the same time), versus something like removing an unhealthy instance from a load-balancer pool, which is cheap to reverse and safe to fully automate.

Structured elaboration

Why "N consecutive failures" instead of one: a single failed check can be a genuine blip (a GC pause, a brief network hiccup) rather than a real failure. If each check independently has some baseline flakiness probability $p$ (a transient failure unrelated to a real outage), then requiring $N$ consecutive failures before declaring unhealthy makes the false-trigger probability:

With, say, $p = 0.05$ (5% chance any single check fails transiently) and $N=1$ (react on the first failure), the false-trigger probability is just $p = 5\%$, meaning roughly 1 in 20 blips would incorrectly trigger action. Requiring $N=3$ consecutive failures drops that to:

a 400x reduction, at the cost of a real failure now taking 3 check intervals (here, up to 15 seconds at a 5s interval) longer to detect instead of one. That's the actual dial being turned: detection speed versus false-positive rate, and it should be set from an observed flakiness rate for your specific checks, not copied from another team's runbook.

Decision flow, including where automation stops and a human is required:

flowchart TD
    A[Health check runs] --> B{N consecutive<br/>failures?}
    B -->|No| A
    B -->|Yes| C{What kind of<br/>action?}
    C -->|Remove from LB pool| D[Fully automated:<br/>cheap, instantly reversible]
    C -->|Open circuit breaker| D
    C -->|Promote new primary<br/>for stateful service| E{Safety checks pass?<br/>replica caught up,<br/>quorum reachable}
    E -->|No| F[Page human,<br/>do not auto-promote]
    E -->|Yes, and blast radius<br/>is well-understood| G[Auto-promote,<br/>but page for review]
    E -->|Yes, but ambiguous<br/>e.g. partition, not clear failure| F

"Quorum reachable" in that safety check means enough replicas are online and able to vote that a new primary can be safely elected, a majority agreeing on who's in charge, without risking two nodes each believing they're the primary at once.

Trade-offs & pitfalls

• Fully automated failover is right when the action is cheap and reversible (removing an unhealthy node from rotation); it's risky when the action is expensive or irreversible (promoting a database replica, since promoting the wrong one, or promoting during a network partition rather than an actual failure, can cause split-brain or data loss). The line isn't "how critical is the service," it's "how reversible is this specific action."
• Requiring consecutive failures trades detection speed for false-positive protection; too aggressive a requirement (e.g., N=10) means a genuine failure runs uncaught far longer than the blast radius justifies.
• Checks that themselves depend on a shared resource (e.g., every health check queries the same central database) can produce correlated, simultaneous "failures" across an entire fleet when that shared resource degrades, which looks like a mass outage but is really a single point of failure in the monitoring path itself.
• A common wrong turn: only checking process liveness and assuming that's sufficient. A process can be alive, passing liveness checks, and still be completely unable to serve real traffic because its only database connection pool is exhausted; readiness and dependency checks catch what liveness checks structurally cannot.

Direct answer

Two separate failures compounded here: the failover wasn't gated on the old leader actually being provably stopped (a real split-brain window during leader election, a period where two nodes can each believe they're the current leader and both may accept writes at the same time), and even a clean failover doesn't save you if the sinks aren't idempotent, because at-least-once delivery will still double-write on any redelivery. The durable fix addresses both: close the leader-election race with fencing, and make sinks idempotent as defense in depth, not either one alone.

Two-track architecture fix

flowchart LR
  P[Producer] --> S[Partitioned stream]
  S --> OL[Old leader epoch 5]
  S --> NL[New leader epoch 6]
  OL -->|late write epoch 5| GATE{Fencing check}
  NL -->|write epoch 6| GATE
  GATE -->|epoch >= committed| COMMIT[Upsert by event_id]
  GATE -->|epoch stale| DROP[Discard write]

Fencing tokens close the leader-election race. Every leader term gets a monotonically increasing epoch number. The sink (or a shared coordination point in front of it) only accepts writes carrying an epoch at or above the last committed epoch:

Concretely: the old leader holds epoch 5, becomes unreachable during a network partition, but is still alive and later resumes sending writes at epoch 5. A new leader was elected at epoch 6 after the lease expired and starts committing. Once epoch 6 has committed anything, the sink rejects any further epoch-5 writes outright, so the old leader's late writes are discarded rather than double-applied. This requires the storage layer to support an atomic conditional write (compare-and-swap on epoch, or a native fencing primitive); without that, the check can itself race.

Idempotent sinks are the second, independent layer. Every event carries a stable, globally unique event_id assigned at ingestion. Sinks write with an upsert keyed on event_id (INSERT ... ON CONFLICT (event_id) DO NOTHING, or a warehouse MERGE), so even a legitimate at-least-once redelivery, with no leader-election bug involved at all, lands exactly once in the sink's state.

Worked example

Both mechanisms are demonstrated in the fencing example above: the epoch comparison (5 versus 6) is the concrete, reproducible logic that decides accept versus reject, and it's independent of the idempotency layer. If fencing alone failed for some reason (a coordination bug elsewhere), the event_id upsert would still catch the duplicate, because the old leader would be resending events that already have IDs the sink has seen. Layering the two means either one failing doesn't reintroduce the bug, only both failing together would.

Trade-offs and pitfalls

The pitfall that caused this exact incident is fixing only the sink you noticed the duplicates in. Idempotency has to be audited across the full fan-out, a warehouse MERGE keyed on event_id doesn't help if a downstream webhook or notification consumer isn't also deduplicated, and duplicate user-visible side effects (a second email, a duplicate charge) will still occur even after the "main" sink is fixed. Fencing tokens add real requirements on the storage layer: many systems don't support atomic conditional writes natively, which pushes you toward optimistic-concurrency primitives (row versioning, conditional writes) as a design constraint on the whole pipeline, not just the failover logic. Idempotent upserts also add write overhead (a uniqueness check or MERGE is more expensive than a blind insert), which is a fair trade for correctness here but should be sized against actual throughput, not assumed free.

Direct answer

Keep serving reads from the healthy read regions immediately and unconditionally, since they don't depend on the write region at all; reject writes that require strong consistency or exactly-once guarantees outright with a clear error rather than pretending to accept them; and for writes that tolerate eventual consistency, queue them locally and durably so nothing is lost, then replay the queue once the write region recovers. The one rule that should never be violated is silently accepting a write and losing it: every write path is either "accepted and durably queued for later replay" or "rejected immediately," never "accepted and quietly dropped."

Architecture

Writes accepted by the anchor leader flow out to every read region as a CDC stream (change data capture: a continuous feed of every row-level change, read off the leader's own write log and replayed onto each replica), which is what keeps the read replicas caught up during normal operation, shown as the CDC node in the diagram below.

flowchart TD
    CLIENT[Clients worldwide] --> RA[Region A read replica]
    CLIENT --> RB[Region B read replica]
    CLIENT --> RC[Region C read replica]
    RA --> FWD[Write forwarder]
    RB --> FWD
    RC --> FWD
    FWD --> LEADER[Anchor write leader, sharded by key range]
    LEADER --> CDC[CDC stream]
    CDC --> RA
    CDC --> RB
    CDC --> RC

When the write region (the anchor leader) goes down, the forwarder can no longer reach it. That's the trigger for the degradation workflow below; everything downstream of "detect the write region is unreachable" is about what each request type does next.

What stays available, what fails, what queues

Communicating the degraded state

• API-level: every write endpoint that's disabled returns a structured error body, not a bare 503, so client applications can distinguish "the write region is down" from a generic server error and render an appropriate message instead of a raw failure.
• UX-level: a persistent banner ("Changes are temporarily disabled; you can still view everything. We'll apply queued updates once service resumes.") rather than silent failures scattered across individual actions, and per-action status for anything queued ("Saved locally, will publish when service resumes") so users aren't left wondering whether their action actually did anything.
• Ops-level: a status page update and, for major customers, proactive notification with an ETA, since 2 hours of read-only mode is the kind of thing enterprise customers expect to hear about before they notice it themselves.

Reconciliation when the writer returns

Queued writes are replayed in the order they were queued, using the idempotency key each write was tagged with to detect and skip duplicates (a client that retried a queued write multiple times during the outage shouldn't apply multiple times on replay). Conflict handling depends on the data type: commutative operations (counters, like counts) merge deterministically with no ambiguity; simple last-write-wins fields use the write's original logical timestamp, not the replay time, so a user's genuinely earlier edit doesn't overwrite a genuinely later one just because it replayed second; and content edits with a real risk of conflicting concurrent changes (two people editing the same document) get surfaced to the user as a merge conflict rather than silently resolved, because a silent wrong resolution is worse than asking.

Trade-offs & pitfalls

The scope of what gets classified "critical" versus "queueable" is the actual design decision here, and it's easy to get wrong in both directions: classifying too much as critical means the read-only window feels far more restrictive than it needs to be, while classifying too much as queueable risks a replay-time conflict resolution mess for data where "eventually consistent" was never actually an acceptable property. A related pitfall is bounding the queue: an unbounded local write queue during a multi-hour outage can grow large enough that the replay itself becomes a second incident (a burst of stale updates hitting the recovered write region all at once), so the queue needs both a size cap (reject new non-critical writes past a threshold, same as critical ones) and a TTL, with the two-hour outage in this scenario sized against realistic write volume before committing to "queue everything non-critical" as the policy. The same read-only degradation shape applies whether the thing behind the write region is a recommendation service, an image-preview pipeline, a personalization engine, or an analytics dashboard. In every case the same triage question decides the design: does this write need to be right immediately, or can it be right eventually, and only the answer to that question, not the specific feature, determines whether it queues or rejects.