Fault Tolerance, High Availability, and Disaster Recovery Questions

Active-active runs all nodes or regions serving live traffic at the same time, so failover is mostly a routing problem: stop sending traffic to the unhealthy node. Active-passive keeps one side idle (or partially warmed) as a standby, so failover is a promotion problem: detect the primary is down, make the standby the new primary, then redirect traffic to it. That difference in what has to happen during failover is what drives everything else: recovery speed (RTO, recovery time objective: how long it takes to restore service after a failure), data-consistency risk (RPO, recovery point objective: how much data, measured in time, you could lose in a failure), and cost.

Comparing the two

Worked example: deriving RTO for each

Pin the same detection policy to both: a health check runs every 5 seconds, and 3 consecutive failures are required before the system acts (15 seconds to declare a node down; this avoids reacting to a single dropped probe).

Active-active RTO: once the node is declared down, the load balancer or GSLB removes it from rotation immediately (no promotion step). If we assume that rotation update takes about 5 seconds to propagate to all edge/LB nodes:

Active-passive RTO: the same 15 seconds to detect, plus a promotion step (electing the standby, replaying any un-applied log entries, opening it for writes, say 20 seconds for a warm standby with low replication lag) plus DNS or routing propagation (assume a low TTL of 30 seconds, and worst case the full TTL has to expire before every client picks up the change):

Under these pinned assumptions, active-active recovers about 3x faster, entirely because it skips the promotion step, and that gap only grows if the standby is cold rather than warm (add provisioning time) or if the routing layer is DNS with a high TTL instead of a fast health-checked LB.

Trade-offs and pitfalls

Active-active's speed advantage is real but not free: the moment two regions can both accept writes to the same record, you have a distributed-write problem, and skipping it (assuming replication will "just" reconcile) is the most common wrong turn. It needs either a conflict-resolution strategy (last-write-wins with vector clocks, CRDTs) or partitioned ownership (each region owns a disjoint key range) to avoid silently losing or corrupting data during a partition. Active-passive's failure mode is different: over-eager health checks or a flapping network link can trigger a premature promotion while the old primary is still technically reachable by some clients, producing two nodes that both believe they're primary (split-brain), which is why production active-passive systems add fencing (forcibly cutting off the old primary's access to shared storage) on top of the detection logic above, not just a timer. The same trade-off shows up outside a classic web/DB stack too: failing over an ML-serving fleet is closer to active-active in spirit (multiple replicas of the same model serving concurrently, so losing one just drops capacity) unless the model itself is being hot-swapped, in which case the promotion-style risk (serving from a half-loaded or stale model version) reappears.

Direct answer

Correctness has to win outright for balance mutations: shard accounts across a consensus group (Raft is the standard choice for its operational clarity) with a single elected leader per shard, and require every write to reach a majority quorum of replicas before it's acknowledged as durable. Availability pays for this in write latency, since a quorum commit has to wait for the slowest replica in the majority, not just the leader, and during a leader failure there's a real (bounded, but nonzero) window where that shard's writes are unavailable while a new leader is elected. The way to minimize the damage is to make that window as short as possible through fast leader election and leader leases, and to make the latency cost as small as possible through deliberate quorum placement, not by weakening the consistency guarantee itself.

Architecture

flowchart TD
    LB[Global load balancer] --> A1[Region A: 2 replicas, leader here]
    LB --> B1[Region B: 2 replicas]
    LB --> C1[Region C: 1 replica]
    A1 --> RAFT[Raft group, N=5, quorum=3]
    B1 --> RAFT
    C1 --> RAFT
    RAFT --> LEDGER[Append-only transaction log]

Accounts are partitioned by account ID using consistent hashing (a hashing scheme where each account maps to a position on a ring of shards, chosen so that adding or removing a shard only reshuffles a small fraction of accounts instead of nearly all of them) into shards; each shard is its own independent Raft group with its own leader, so write throughput scales by adding shards rather than being bottlenecked by one global leader. Client writes route to the shard's current leader; reads can be served from the leader for strict linearizability (every read reflects the most recent completed write, as if all operations happened one at a time in real-time order), or from a follower with a leader-issued lease for bounded-staleness reads that don't need to pay quorum latency.

Deriving the quorum

For a Raft group of $N$ replicas, a write commits once it's replicated to a majority:

For $N = 5$:

The number of simultaneous replica failures the group tolerates while still having a live quorum is:

Placement and latency

Place the 5 replicas as 2 in the leader's home region (Region A), 2 in a second region (Region B), and 1 in a third region (Region C). This placement matters for two separate reasons. First, region-level fault tolerance: losing Region A or Region B (2 replicas each) leaves exactly 3 replicas, which is exactly quorum, so the group survives the loss of any single region without losing write availability, precisely because no single region holds more than $F_{\text{tolerated}} = 2$ replicas. If any region instead held 3 replicas, losing that region alone would drop the group below quorum.

Second, latency: because Raft only needs a majority, not all 5, the leader can commit as soon as 2 of the 4 followers acknowledge (its own copy plus 2 acks makes 3, quorum). The fastest 2 acks available are the local Region-A follower (low latency, same region) and whichever of Region B or Region C responds first:

With an assumed same-region RTT of 2ms and cross-region RTTs of 70ms (A to B) and 90ms (A to C):

The commit is gated by the faster of the two remote regions, not the slower one, and not by all four followers, which is the concrete reason majority quorum beats "wait for everyone": a design that required all 5 replicas to ack would pay the 90ms worst-case RTT on every write instead of the 70ms best-of-two.

Leader failover

On leader loss, followers detect the missing heartbeat after an election timeout and hold a new election; the group is unavailable for writes to that shard for roughly the election-timeout-plus-one-round-trip window, which is why election timeouts are tuned as short as reliably possible without causing spurious elections from ordinary network jitter. A leader lease (the leader holds exclusive write authority for a short renewable period) lets followers serve bounded-staleness reads without contacting the leader at all, and automated leader transfer for planned maintenance (the outgoing leader hands off cleanly before stepping down) avoids paying the full election-timeout cost for maintenance events, which are predictable and shouldn't cost the same as an unplanned crash.

Cross-shard transfers

A single-shard write (both accounts in a transfer live in the same shard) is a normal Raft-committed entry, no special handling needed. A cross-shard transfer needs coordination between two independent Raft groups: a coordinator writes a prepare entry to both shards' logs, and only commits the transfer once both shards' local quorums have accepted the prepare; if either shard fails to prepare within a timeout, the coordinator writes a compensating abort entry to whichever shard did succeed, rather than leaving the transfer half-applied. This is deliberately simpler than a general two-phase-commit protocol across arbitrary participants, because both sides are already individually linearizable Raft logs, so the coordinator's job is narrowly "get both to agree" rather than solving distributed consensus from scratch.

Trade-offs & pitfalls

The quorum design directly trades latency for correctness: every write pays at least the RTT to the nearest remote region in the placement, which is unavoidable if you actually want the durability guarantee (a write acknowledged locally-only in Region A could be silently lost if Region A is the one that fails), and no client-side retry logic fixes a fundamentally under-replicated design. A common mistake is treating replica count as the only lever, when placement matters just as much: 5 replicas concentrated so that one region holds 3 gives you the same $N=5$, $Q=3$ numbers on paper, but a single region failure then drops the group below quorum, which defeats the entire point of geographic redundancy even though nothing about the Raft math itself is wrong. Idempotency and fencing tokens are non-negotiable additions on top of the consensus layer: a Raft term number rejects stale writes from a leader that's since been superseded, and a client-supplied idempotency key prevents a transaction from being double-applied if a client retries after a timeout that turned out to be a lost acknowledgment rather than a real failure, not a rejected write.

Direct answer

Model each component's availability as a probability of being up, then combine them with two rules: components in series (every one must work for the system to work) multiply, and components in parallel (redundant copies of the same thing, any one of which is enough) combine as one minus the product of their failure probabilities. Once you have that model, "where does an extra dollar of redundancy buy the most availability" becomes a concrete comparison: compute the marginal gain in availability from each candidate investment, divide by its cost, and fund whichever option has the highest availability-per-dollar, repeating as budget allows.

The two combination rules

Series availability is always less than or equal to the least available component, because every additional required link is one more way for the chain to break. Parallel availability is always greater than or equal to the most available individual replica, because failure now requires every redundant copy to fail simultaneously, which independence makes rapidly unlikely as replica count grows.

Modeling redundant replicas across regions

For $n$ identical replicas within a region (parallel, independent failures) and $r$ regions (also parallel: the system is up if any one region is up):

where $a$ is a single replica's availability. The system-wide failure probability collapses to $(1-a)^{nr}$: the whole deployment is down only if every one of the $n \times r$ independent replicas is simultaneously down.

Worked example: where does the next dollar go

Start with a single instance at $a = 0.99$ (two nines, a not-unreasonable single-VM baseline) and no redundancy ($n=1, r=1$):

Converting to expected downtime per year (using 525,600 minutes/year, $365 \times 24 \times 60$):

Going from no redundancy to a second replica in the same region already removes over 5,200 minutes of downtime per year. Going from one region to two regions on top of that removes only another 52.5 minutes. That's the diminishing-returns pattern the model predicts: each additional nine costs roughly the same effort but buys an order of magnitude less absolute downtime reduction than the nine before it, because the failure probability being multiplied away is already small.

Deciding where the next dollar goes when both axes already have redundancy

For a system already at $n$ replicas and $r$ regions, the marginal availability gain from adding one more replica (holding regions fixed) versus adding one more region (holding replicas fixed):

At $n=3$, $r=2$, $a=0.99$: $(1-a)^{nr} = (0.01)^6 = 10^{-12}$. The bracket terms are $1-(0.01)^2 = 0.9999$ for adding a replica, versus $1-(0.01)^3 = 0.999999$ for adding a region, so $\Delta A_{\text{region}}$ is about 0.01 percent larger than $\Delta A_{\text{replica}}$ here. The pattern behind that: the bracket term is largest for whichever axis currently has fewer units ($r=2 < n=3$ in this example), so all else equal, invest in whichever axis is thinner. Then divide each $\Delta A$ by its real cost: $\text{ROI} = \Delta A / \text{cost}$, and since a new replica in an existing region is almost always cheaper than standing up an entire new region (new infrastructure, new operational surface, cross-region egress), the raw availability numbers alone usually favor adding replicas once you already have two or more regions, and the case for a third region has to come from something the pure independence model doesn't capture: correlated failure risk.

Trade-offs & pitfalls

The entire model rests on an independence assumption that gets weaker as you add replicas within the same region and stays strongest across regions: two replicas in the same region share a power grid, a network fabric, and often a control plane, so a regional outage takes both of them down together in a way the $(1-a)^n$ formula doesn't account for, since that formula assumes each replica fails independently of the others. This is precisely why real systems keep investing in additional regions even when the pure math above says another same-region replica is cheaper per nine: the region-diversity investment is buying protection against a correlated-failure mode the independence model is structurally blind to, not buying a bigger $\Delta A$ number on paper. A second pitfall is applying the series-multiplication rule to a system that has hidden shared dependencies across "independent" series stages, for example a DNS provider or a certificate authority that every stage secretly relies on; multiplying stage availabilities together understates the real risk whenever a single upstream failure can take out multiple "independent" stages simultaneously, which is exactly the class of failure that showed up in the largest real-world multi-service outages. The practical fix for both pitfalls is the same: use the independence-based formula as the starting estimate and a genuine planning tool, but treat any two components sharing physical infrastructure, a network path, or an operational dependency as a single correlated unit for the purposes of the model, not as two separate multiplicands.

The four standard DR tiers form a spectrum from cheapest-and-slowest to most-expensive-and-fastest, and each one trades infrastructure spend for recovery speed (RTO, recovery time objective: how long restoring service takes) and data freshness (RPO, recovery point objective: how much data, measured in time, you could lose): backup-and-restore keeps only backups running, pilot light keeps a minimal always-on core, warm standby keeps a scaled-down full copy running, and active-active multi-site keeps a full copy running and serving live traffic.

Comparing the four tiers

The RTO/RPO ranges above are the typical shape of the trade-off, not a fixed number for any specific system: the exact figures depend on data volume, automation maturity, and how the replication is actually implemented within each tier.

Worked example: a budget-constrained startup

A mid-sized SaaS with a fixed infrastructure budget doesn't have to pick one tier for the whole system; the standard move is to mix tiers by criticality. Say the product has three logical components: authentication/billing (must never meaningfully go down, since it blocks every paying customer from doing anything), the core application (needs to come back reasonably fast but a short outage is tolerable), and internal admin tooling (only the ops team notices if it's down for a few hours).

A budget-conscious allocation: active-active for auth/billing (the one component where the cost premium is justified because its outage blocks revenue entirely, and it's usually small enough in infrastructure footprint that duplicating it fully is affordable), pilot light for the core application (keep the database replicated continuously so RPO stays low, but only spin up the app-server fleet in DR when actually needed, since that's the majority of the compute cost), and backup-and-restore for admin tooling (cheapest tier, acceptable because nobody customer-facing is blocked by it being down for hours). This gets the highest-blast-radius component the fastest recovery while keeping the overall DR bill proportional to what each component actually costs the business if it's down, instead of buying active-active everywhere by default.

Trade-offs and pitfalls

The most expensive mistake in this space isn't picking the "wrong" tier, it's picking a tier and never testing failover into it: a pilot-light setup that's never actually been promoted to full capacity under load is a theoretical RTO, not a real one, and the first real DR event is a bad time to discover the app layer doesn't actually scale up cleanly from zero. A related pitfall is under-provisioning a warm standby's capacity: "scaled down" often means it can absorb DR traffic at reduced performance, and teams sometimes forget to validate that the scaled-down size can actually handle 100% of production load once promoted, not just serve health checks. Finally, active-active's real cost isn't just the duplicate infrastructure line item, it's the ongoing engineering cost of keeping a multi-writer data model correct, which is easy to underestimate when comparing tiers purely on an RTO/RPO/dollar table.

Direct answer

A cascading failure is when one component's failure increases load or latency on the components that depend on it, and that increased load causes those components to fail too, propagating outward until a large part of the system is affected, even though only one component actually broke in the first place. The mechanism is almost always resource exhaustion: threads, connections, or memory tied up waiting on the failed component instead of being freed quickly.

Walkthrough: C fails, B overloads, A degrades

flowchart LR
  A[API Gateway] -->|rate limit and timeout| B[Order Service]
  B -->|bulkhead pool: payments| C[Payment Service]
  C -.fails.-> B
  B -->|circuit breaker opens| D[Fallback: queue order for async retry]
  A -->|circuit breaker opens| E[Fallback: 503 with Retry-After]
  B -->|isolated pool: other deps unaffected| F[Inventory Service]

1. C (Payment Service) fails, hanging instead of returning errors quickly, perhaps due to a downstream outage of its own.
2. B (Order Service) calls C without a tight timeout. Each call to C now blocks for far longer than normal, tying up a thread or connection from B's pool for the duration.
3. B's resource pool exhausts. As more requests arrive at B, more threads get stuck waiting on C, until B has no capacity left to serve any request, including ones that don't even touch C.
4. A (API Gateway) calls B, and B is now slow or unresponsive for everything, so A's calls to B start timing out or queueing too, degrading A's own capacity in turn.

Worked example: how fast does B's pool actually exhaust?

Little's Law relates the number of requests in flight to the arrival rate and the time each spends being processed:

Say B receives 500 requests per second, and under normal conditions each call to C takes 50ms:

25 concurrent requests is a light load on a typical connection pool. Now C hangs, and B's HTTP client has no explicit timeout of its own, falling back to a default of 30 seconds:

If B's thread pool has 200 threads, the time to exhaust it entirely is:

Under 400 milliseconds. That's how quickly a single hung dependency with no timeout turns into total unavailability for a service handling 500 requests per second: the pool never gets close to steady-state at the 30-second hang time, it simply fills with stuck requests almost instantly and stays full.

Protections at each layer

• At B, calling C: a tight, explicit timeout (measured in low hundreds of milliseconds, not the client library's 30-second default) so a hung call fails fast and frees the thread quickly; a circuit breaker that opens after a run of failures or timeouts, so B stops even attempting calls to C once it's clearly down, and falls back to queueing the order for later processing; a bulkhead, a dedicated connection pool just for calls to C, so exhaustion from C-related calls doesn't consume the threads B needs to serve requests that don't touch C at all (like inventory checks).
• At A, calling B: the same pattern one layer up, a timeout on calls to B, a circuit breaker that trips once B's error rate or latency crosses a threshold, and a fallback (a fast 503 with Retry-After rather than a hung request) so A's own capacity isn't consumed waiting on a B that's already struggling.

Trade-offs & pitfalls

Timeouts that are too aggressive cause false-positive failures under normal, brief latency variance; timeouts that are too loose don't prevent the cascade fast enough, as the Little's Law example shows. Bulkheads cost real resources (a dedicated pool per dependency uses more total connections or threads than one shared pool) in exchange for isolation, so they're worth applying to the dependencies most likely to fail or most likely to take down unrelated traffic if they do. The most common mistake is only protecting the first hop (B to C) and assuming that's sufficient; as the walkthrough shows, without protection at the A to B hop too, the failure still reaches A once B is degraded, just one layer later.