Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer

Stabilize in two stages: first debounce each instance's raw signal over time (require several consecutive consistent samples before trusting a state change), then aggregate the debounced per-instance states into one service-level decision using a quorum or percentage threshold, not "any single unhealthy instance flips the whole service." Stacking a temporal filter and a spatial one multiplies down the false-positive rate far more than either alone.

Two-stage design

stateDiagram-v2
  [*] --> Healthy
  Healthy --> Suspect: 1 bad sample
  Suspect --> Healthy: 1 good sample
  Suspect --> Unhealthy: k consecutive bad samples
  Unhealthy --> Recovering: 1 good sample
  Recovering --> Healthy: k consecutive good samples
  Recovering --> Unhealthy: 1 bad sample

Per instance, debouncing with a threshold of $k$ consecutive bad samples before committing to "unhealthy" (this is the state machine above) reduces the chance a single blip changes the reported state to:

where $p$ is the per-sample probability that a healthy instance reports a bad sample due to transient noise (a slow GC pause, a dropped probe packet). Recovering asymmetrically (fewer good samples needed to go back healthy than bad samples needed to go unhealthy, or the reverse, tuned by SLA) keeps the machine from oscillating.

Service-level aggregation then requires $m$ of $N$ debounced instance states to agree before changing the reported service health, using the binomial tail:

Worked example

Suppose monitoring shows each sample has a 10% chance of spuriously flagging bad ($p=0.1$, pinned for this example):

Debouncing with k=2 already drops the per-instance spurious-flip rate from 10% to 1%. Now aggregate across N=10 instances using that debounced 1% rate:

With no quorum (any one debounced instance flips the service, m=1), the service-level false-positive rate is still nearly 10% per decision window, because with ten independent instances the chance that at least one of them blips is much higher than any single instance's own rate. Requiring a majority (m=6 of 10) collapses that to about 2 in 10 billion. That's the concrete case for why per-instance signals should never directly drive service-level decisions: debounce alone isn't enough once you have more than a handful of instances, you need the quorum too.

Trade-offs and pitfalls

Setting the quorum too high (near N) trades false-positive suppression for real-outage blindness: if 90% of instances genuinely go down, requiring unanimous agreement before declaring the service unhealthy delays a true incident response. Size $m$ against the actual failure semantics you care about (a realistic simultaneous-failure scenario, like an AZ outage taking out a third of instances) rather than only against noise suppression. Also keep the debounce window and heartbeat interval in proportion: a long debounce window with a slow heartbeat interval adds real seconds to genuine-outage detection time, so the same knobs that suppress flapping also directly set your worst-case MTTD (mean time to detect: how long a genuine outage takes before it's recognized as unhealthy), and that trade-off needs to be explicit, not accidental.

Direct answer

Deploy a read replica in every user-facing region and route reads to the nearest one for latency, while anchoring writes to a single-writer-per-shard model: each account or entity has one home region that owns writes for it (sharded by key, not globally centralized), and a write originating from any other region gets forwarded to that entity's home region. Reads stay fast everywhere because they never leave the local region; writes pay a forwarding cost only when they originate somewhere other than the entity's home region, which for most workloads is the minority case.

Architecture

flowchart TD
    CLIENT[Clients worldwide] --> RA[Region A read replica]
    CLIENT --> RB[Region B read replica]
    CLIENT --> RC[Region C read replica]
    RA --> FWD[Write forwarder]
    RB --> FWD
    RC --> FWD
    FWD --> LEADER[Anchor write leader, sharded by key range]
    LEADER --> CDC[CDC stream]
    CDC --> RA
    CDC --> RB
    CDC --> RC

• Read routing: DNS-based or edge-proxy latency routing sends each client to its nearest region; that region serves reads from its local replica, optionally backed by a local edge cache for hot keys to cut load further.
• Write routing: a lightweight write-forwarder in each region inspects the target entity's shard key, determines which region owns it, and forwards the write there if it isn't local; the write commits in its home region and the forwarder returns the result (or an idempotency-tracked async acknowledgment) to the originating client.
• Replication: the write's home region streams committed changes via change-data-capture (CDC) to every read replica, asynchronously; replicas apply changes in the order the CDC stream delivers them, tracking a per-record last-writer timestamp so replay order and true causal order stay consistent.

Consistency model

This design is eventually consistent for reads: a read served from Region B immediately after a write committed in Region A's home shard may not reflect that write yet, bounded by CDC replication lag rather than by any hard guarantee. That's an explicit, necessary trade for the latency goal, since making every read wait for a cross-region round-trip to confirm it has the absolute latest value would defeat the entire point of routing reads to the nearest region. Where a specific read genuinely needs to see its own very-recent write (a user immediately viewing an item they just created), the standard fix is read-your-own-writes: route that specific read to the entity's home region (or to a replica known to have caught up past a specific CDC watermark) instead of the nearest replica, rather than weakening the consistency model for every read to satisfy the rare case.

Write conflicts are structurally rare by design, because each entity has exactly one home region and therefore exactly one writer at any time; there's no multi-master merge problem to solve because there's no multi-master. The forwarding hop is the cost of ruling that problem out entirely rather than solving it after the fact.

Worked example: tracing one write end to end

Pin a concrete case: user_id=482's home region is us-east (that's where its shard's writer lives), but the client happens to be connected to the nearest edge in eu-west. The eu-west write-forwarder inspects the shard key for user_id=482, sees it belongs to us-east, and forwards the write there; assume a cross-region round trip of about 90ms for that forward, so the write is accepted and committed in us-east roughly 95ms after the client sent it (90ms network plus a small local processing cost). From there, the CDC stream carries that committed change out to every read replica asynchronously; assume that hop adds about 300ms before eu-west's local replica has applied it (its own network hop plus normal stream batching, separate from the synchronous forward that carried the write there). So the total time from write-acceptance to eu-west's replica reflecting the change is about $95\text{ms} + 300\text{ms} = 395\text{ms}$, call it roughly 400ms. A read served from eu-west's local replica 1 second (1,000ms) after the write was accepted already reflects it, comfortably past the ~400ms it takes to land; a read served only 50ms after acceptance would not yet reflect it, since 50ms is well inside that ~400ms propagation window, which is exactly the read-your-own-writes gap the design accepts and works around by routing that specific kind of read to the home region instead.

Trade-offs & pitfalls

The biggest latency cost this design accepts is on writes that originate far from an entity's home region: a user in Region C writing to an entity whose home shard is in Region A pays a full cross-region round trip for that write, even though every other user's reads and most other writes stay fast. If write locality doesn't naturally match user geography (an entity created in one region gets written to mostly by users somewhere else over time), this cost compounds instead of amortizing away, which is worth checking against real traffic patterns before committing to a static shard-to-region mapping. Replication lag is the other pitfall: CDC-based replication is asynchronous by nature, so a region that falls behind (network partition, replica overload) serves increasingly stale reads without necessarily surfacing an error, which is why lag needs to be an actively monitored metric with alerting, not just an assumed-small property of the pipeline. Finally, resist the temptation to solve the "occasional write from any region" requirement with full multi-master writes accepted locally everywhere; that reintroduces exactly the conflict-resolution complexity (concurrent writes to the same entity from two regions, needing merge logic or last-write-wins with its own correctness risks) that single-writer-per-shard was specifically chosen to avoid, in exchange for a write-latency win that the stated 90-percent-read workload doesn't actually need.

CAP theorem says that during a network partition, a distributed system must choose between consistency (every read sees the latest acknowledged write) and availability (every request gets a response), because it can't guarantee both while the partitioned halves can't talk to each other. Partition tolerance itself isn't optional for any system that spans more than one node, so the real choice interview questions are testing is CP versus AP, and the right answer depends entirely on what happens if you get it wrong.

Financial ledger: choose CP

A ledger getting a stale or divergent balance is a correctness bug with real financial consequences (double-spend, incorrect balance shown, a transaction accepted twice), so consistency has to win. The concrete mechanism is a consensus protocol requiring a write quorum.

Deriving what "sacrifice availability" actually means, with a pinned example: take a 5-node replica set (a typical Raft cluster size) where a majority quorum is required to commit a write:

That cluster tolerates up to $N - \text{quorum} = 2$ node failures while still committing writes. Now suppose a network partition splits the 5 nodes into a 3-node side and a 2-node side. The 3-node side still has a majority (3 ≥ quorum of 3), so it keeps accepting writes and stays both consistent and available. The 2-node side does not have a majority (2 < 3), so by design it must refuse writes, becoming unavailable, specifically to prevent both sides from independently committing conflicting transactions. That refusal on the minority side, not a global shutdown, is the literal, computable meaning of "CP sacrifices availability during a partition": only the minority partition goes unavailable, and only for writes.

Analytics event aggregator: choose AP

An analytics pipeline getting an event a few seconds late, or briefly double-counted before deduplication catches up, is a rounding error on a dashboard, not a financial loss, so availability wins: every node keeps ingesting even when it can't see the others.

The mechanism looks different from the ledger: instead of a write quorum gatekeeping every write, use leaderless or partitioned ingestion (each partition or node accepts writes for its own shard independently, as in Kafka-style partitioned logs or a Dynamo-style leaderless store), so there's no majority to lose and no write path that can be blocked by a partition. The cost shifts from "unavailable during a partition" to "eventually reconciled after one": deduplication on ingest, idempotent consumers, and background reconciliation jobs to merge whatever diverged while the partition was open.

Comparing the two designs

Trade-offs and pitfalls

The common misreading of CAP is treating CP as "the whole system goes down during any partition," when the quorum math above shows it's specifically the minority side, and only for writes, reads can often still be served (possibly stale, depending on the read-consistency level chosen). The common mistake on the AP side is stopping at "it's eventually consistent" without actually building the reconciliation path: leaderless ingestion without idempotent consumers and deduplication just relocates the correctness problem downstream instead of solving it. And even a CP ledger still needs idempotency keys on the client side: a client that times out waiting for a quorum ack and retries the same transaction must not have it applied twice, which is a consistency concern CAP itself doesn't cover but that any real ledger design has to handle regardless of which side of CAP it lands on.

Direct answer

The 5-minute TTL isn't the actual bottleneck: DNS caching in the real world (ISP resolvers with minimum-TTL floors, browser caches, persistent keep-alive connections that never re-resolve at all) means real failover time doesn't track the advertised TTL cleanly, which is exactly why you're seeing 3 minutes instead of something close to 5. Getting under 30 seconds for most clients means building an explicit time budget (detect, update, propagate) that sums under 30s for compliant clients, and accepting that DNS alone can't guarantee it for the tail of clients whose resolvers or connections don't re-check in time.

Building the time budget

where $k$ is the number of consecutive failed health checks required before failing over (the detection threshold) and interval is the health-check period.

sequenceDiagram
  participant C as Client
  participant R as Resolver
  participant D as Authoritative DNS
  participant H as Health Monitor
  participant A as Origin A
  participant B as Origin B
  H->>A: probe every 5s
  A--xH: 2 consecutive failures (10s)
  H->>D: update record to B
  C->>R: resolve hostname
  R->>D: query (TTL expired)
  D-->>R: return B, TTL 10s
  R-->>C: B
  C->>B: connect

Worked example

Redesign inputs: health-check interval 5s, failure threshold k=2 (avoids single-blip flaps), API-driven record push under 1s, and TTL lowered from 300s to 10s.

That covers clients and resolvers that honor the lowered TTL, with about 9 seconds of margin. It does not cover the two categories that caused the original 3-minute number: resolvers that enforce a minimum TTL floor above what you set, and clients holding a persistent connection that has no reason to re-resolve DNS at all until it errors. For those, add a client-side backstop that's independent of TTL: short keep-alive and idle timeouts so connections periodically re-establish (and therefore re-resolve), and connect-level retry to a secondary IP on failure (a Happy-Eyeballs-style fallback) rather than trusting DNS to be the only failover signal.

Trade-offs and pitfalls

What you give up: a 10s TTL multiplies authoritative DNS query volume roughly 30x versus the 300s baseline, which is a real cost and load increase on your DNS infrastructure, and a false-positive failover (from setting k too low) now flips production traffic in as little as 5 to 10 seconds, so your health check needs to be more conservative about what counts as "down," not less. The pitfall that caused the original bug is assuming all clients and resolvers honor your TTL uniformly; they don't, and any redesign that only lowers the TTL without a client-side or network-level backstop will hit the same wall for the same tail of misbehaving resolvers, just with a lower number attached to it.

Direct answer

Blast radius is the scope of impact when a component fails: how many users, tenants, or dependent services are affected, and how severely, not just whether the failure happened at all. Reducing blast radius means designing so a single failure touches the smallest possible slice of the system, which makes outages smaller, easier to detect, and faster to recover from, even if it doesn't reduce how often failures happen at all.

Practices that reduce it, and what they cost

Worked example

Consider a service with 1,000 tenants sharing a single connection pool. If that pool exhausts, every tenant is affected. Now split that same total capacity into 10 isolated pools of 100 tenants each, so each pool serves 100 of the 1,000 tenants and only that pool's own tenants are affected if it exhausts:

Splitting the same total capacity into 10 pools of 100 tenants each means a single pool's exhaustion now affects only 100 of the 1,000 tenants, 10% of the blast radius of the shared-pool design, for the same total resources. The cost is operational: 10 pools to monitor and size instead of one, and if traffic isn't evenly distributed across tenants, some pools may be under-utilized while others are tight, which the shared pool didn't have to worry about.

Trade-offs & pitfalls

Reducing blast radius is generally a trade of operational complexity and some resource inefficiency for smaller, more contained failures; it doesn't reduce the underlying failure rate of any individual component. The common mistake is treating blast-radius reduction as free: partitioning by tenant, region, or dependency multiplies the number of things to monitor and can hide a systemic bug (one that affects every partition equally) behind what looks like ten separate, unrelated small incidents instead of one clearly systemic one.