Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer

DNS-based failover works by changing the authoritative record (A, AAAA, or CNAME) for a name once a health check detects the current target is down, so future lookups resolve to the healthy endpoint. The catch is caching: recursive resolvers, OS stub resolvers, and some client libraries hold the old answer and are not obligated to re-check before the TTL expires, so a client that resolved the name right before the failure can keep sending traffic to a dead endpoint for up to a full TTL, even though the authoritative record already changed.

Structured elaboration

sequenceDiagram
    participant Client
    participant Resolver
    participant AuthDNS as Authoritative DNS
    participant HealthCheck as Health Checker
    participant OriginA as Origin A (primary)
    participant OriginB as Origin B (standby)

    Client->>Resolver: Resolve api.example.com
    Resolver->>AuthDNS: Query (cache miss)
    AuthDNS-->>Resolver: A record = Origin A (TTL 300s)
    Resolver-->>Client: Origin A (cached 300s)
    Client->>OriginA: Request
    Note over OriginA: Origin A fails
    HealthCheck->>OriginA: Health probe fails twice (60s)
    HealthCheck->>AuthDNS: Update A record to Origin B
    Note over Client,Resolver: Cache still holds Origin A until TTL expires
    Client->>OriginA: Request still routed to dead origin
    Note over Resolver: TTL expires
    Client->>Resolver: Re-resolve api.example.com
    Resolver->>AuthDNS: Query (cache miss)
    AuthDNS-->>Resolver: A record = Origin B
    Client->>OriginB: Request now routed correctly

How TTL bounds the blast radius. The authoritative TTL is an upper bound, not the whole story. Worst case for a client is roughly detection time (how long health checks take to notice the endpoint died) plus record update propagation (usually seconds for managed DNS) plus up to a full TTL for a resolver that cached the record right before the failure. Best case is a client whose cache had already expired at the moment of failure, seeing the change almost immediately.

Reducing client-visible downtime

• Shorten TTL ahead of planned risk windows (deploys, maintenance) at the cost of more DNS query volume.
• Put a health-checked global load balancer or CDN in front of origins: it does fast, active health checking and reroutes without waiting on client-side DNS caches at all, since clients only ever resolve to the load balancer's stable address.
• Use anycast so the same IP is announced from multiple locations and failover happens at the routing layer, bypassing DNS caching entirely.

Worked example

Health checks poll every 30 seconds and require 2 consecutive failures before triggering failover, to avoid flapping on a single missed check: detection time = 60 seconds. A managed DNS provider's record update completes in about 5 seconds.

At TTL = 300 seconds, worst-case client-visible downtime (client cached the record 1 second before failure):

Best case, client's cache had already expired at the moment of failure:

If TTL is tuned down to 30 seconds ahead of a risky change:

That is a 3.8x improvement in worst-case client-visible downtime, but caches now expire 10 times more often ($300/30$), meaning roughly 10 times more queries hit the authoritative nameservers for the same client population, a real cost and load trade-off, not a free win.

Trade-offs & pitfalls

• Some resolvers and client libraries do not strictly honor TTL, a small minority cache longer or shorter than instructed; design for the worst case, not the documented one.
• Setting TTL near zero everywhere, permanently, trades a failover problem for a cost and latency problem, since every request now pays a fresh DNS lookup and some providers charge per query.
• DNS failover alone cannot verify the new target is actually healthy from the client's network path; combine it with active health checks at the load balancer or CDN layer rather than relying on DNS as the sole signal.
• A common mistake is failing over DNS but leaving the client's connection pool holding an open connection to the dead endpoint. DNS failover only affects new lookups, not already-established connections, so client-side connection timeouts and retry logic matter just as much as the TTL.

CAP theorem says that during a network partition, a distributed system must choose between consistency (every read sees the latest acknowledged write) and availability (every request gets a response), because it can't guarantee both while the partitioned halves can't talk to each other. Partition tolerance itself isn't optional for any system that spans more than one node, so the real choice interview questions are testing is CP versus AP, and the right answer depends entirely on what happens if you get it wrong.

Financial ledger: choose CP

A ledger getting a stale or divergent balance is a correctness bug with real financial consequences (double-spend, incorrect balance shown, a transaction accepted twice), so consistency has to win. The concrete mechanism is a consensus protocol requiring a write quorum.

Deriving what "sacrifice availability" actually means, with a pinned example: take a 5-node replica set (a typical Raft cluster size) where a majority quorum is required to commit a write:

That cluster tolerates up to $N - \text{quorum} = 2$ node failures while still committing writes. Now suppose a network partition splits the 5 nodes into a 3-node side and a 2-node side. The 3-node side still has a majority (3 ≥ quorum of 3), so it keeps accepting writes and stays both consistent and available. The 2-node side does not have a majority (2 < 3), so by design it must refuse writes, becoming unavailable, specifically to prevent both sides from independently committing conflicting transactions. That refusal on the minority side, not a global shutdown, is the literal, computable meaning of "CP sacrifices availability during a partition": only the minority partition goes unavailable, and only for writes.

Analytics event aggregator: choose AP

An analytics pipeline getting an event a few seconds late, or briefly double-counted before deduplication catches up, is a rounding error on a dashboard, not a financial loss, so availability wins: every node keeps ingesting even when it can't see the others.

The mechanism looks different from the ledger: instead of a write quorum gatekeeping every write, use leaderless or partitioned ingestion (each partition or node accepts writes for its own shard independently, as in Kafka-style partitioned logs or a Dynamo-style leaderless store), so there's no majority to lose and no write path that can be blocked by a partition. The cost shifts from "unavailable during a partition" to "eventually reconciled after one": deduplication on ingest, idempotent consumers, and background reconciliation jobs to merge whatever diverged while the partition was open.

Comparing the two designs

Trade-offs and pitfalls

The common misreading of CAP is treating CP as "the whole system goes down during any partition," when the quorum math above shows it's specifically the minority side, and only for writes, reads can often still be served (possibly stale, depending on the read-consistency level chosen). The common mistake on the AP side is stopping at "it's eventually consistent" without actually building the reconciliation path: leaderless ingestion without idempotent consumers and deduplication just relocates the correctness problem downstream instead of solving it. And even a CP ledger still needs idempotency keys on the client side: a client that times out waiting for a quorum ack and retries the same transaction must not have it applied twice, which is a consistency concern CAP itself doesn't cover but that any real ledger design has to handle regardless of which side of CAP it lands on.

Direct answer: Redundancy is having extra, interchangeable components standing by so one can take over if another fails; replication is actively keeping copies of state (data) synchronized across multiple nodes so the state itself survives a failure, not just the compute that serves it. They're often used together, but they solve different problems: redundancy alone is enough for a stateless service, because any interchangeable instance can serve any request; a stateful service needs replication too, because a fresh redundant instance with no data isn't actually a working replacement.

Structured elaboration

Stateless example: a set of identical app server instances behind a load balancer, handling API requests with no local state. If one instance dies, the load balancer routes around it and an autoscaler replaces it; the new instance needs no data transfer because there was never any instance-local state to lose. This is pure redundancy: extra interchangeable copies of the same stateless computation.

Stateful example: a primary-replica database. The primary accepts writes; replicas continuously receive a copy of the write stream (replication). If the primary fails, a replica is promoted to take over. Unlike the stateless case, simply having an extra database instance running (redundancy alone, no replication) would give you an empty database, not a working replacement, because there's no mechanism copying the actual data into it.

A failure mode redundancy alone doesn't protect against, for the stateful case: data loss or corruption on the primary itself. If the primary's disk corrupts a row, or a bad write silently corrupts application-level data, having a redundant (but not yet caught-up, or synchronously replicating that same bad write) standby doesn't help, because either the standby doesn't have the data yet (async lag) or it faithfully replicated the corruption along with everything else (synchronous replication of a logically bad write). Redundancy protects against a node dying; it does not protect against the data itself being wrong, that requires backups (a separate, point-in-time copy decoupled from live replication) and, for silent corruption specifically, checksums or application-level validation.

How this generalizes: a useful mental checklist for fault tolerance covers five distinct techniques, and redundancy and replication are only two of them: retries (recover from a transient failure by trying again), bulkheads (isolate one failure from spreading to unrelated resources), failover (the mechanism that switches traffic to a healthy replacement), redundancy (having that replacement exist at all), and replication (making sure the replacement actually has the state it needs). A strong answer names which of these a given design decision is actually addressing, since "redundancy" gets used loosely to mean all five in casual conversation.

Trade-offs & pitfalls

• Replication has a cost redundancy alone doesn't: network bandwidth, storage for extra copies, and a consistency model to reason about (synchronous replication costs write latency; asynchronous replication risks data loss on failover, the classic RPO trade-off).
• A common wrong turn: assuming "we have 3 replicas" automatically means "we're protected," without checking replication lag. A replica that's minutes behind at failover time silently loses however much data arrived in that window, unless the promotion logic explicitly accounts for lag and refuses to promote a too-far-behind replica.
• Redundancy for stateless services is comparatively cheap and low-risk to over-provision; replication for stateful services is not, since more replicas means more write-path coordination overhead (for synchronous replication) or more divergence risk (for asynchronous/multi-leader), so it isn't a "just add more" lever in the same way.

Direct answer

A disaster recovery runbook is only useful if a stressed engineer can execute it top to bottom without needing to look anything else up. That means naming an owner, stating the RTO and RPO it is designed to meet, listing ordered and specific actions (exact commands or console steps, not descriptions), and ending with concrete verification steps that prove the system is actually back, not just that the steps were followed.

Structured elaboration

Versioning and access. Store the runbook in source control with required review on changes, so every edit has an author and a diff. Test it in a real drill, not a tabletop discussion only, on a cadence tied to how critical the service is, quarterly for anything customer-facing. Keep it reachable when the primary systems it recovers are down: a runbook that lives only on an internal wiki hosted in the region that just failed is not a disaster recovery runbook.

Worked example

For a service with RTO = 30 minutes, a well-built runbook's step timings should sum to that budget, and the sum should be checked, not assumed:

That sum matching the stated RTO exactly is what makes the RTO a testable claim rather than a number pasted at the top of the document. If a quarterly drill shows execution consistently takes 20 minutes instead of 15, the runbook's RTO is wrong and needs to be corrected, not explained away.

Trade-offs & pitfalls

• A runbook with no owner drifts out of date the first time the architecture changes; ownership is not optional metadata.
• Testing via tabletop discussion only, never an actual drill, hides the gap between the steps sounding right and the steps working; most drift is caught only by execution.
• Over-specifying every command for a fast-moving system creates a maintenance burden that causes the runbook to be abandoned; balance specificity against how often the underlying commands change.
• Storing the only copy behind the same authentication system that depends on the region that just failed is a common, self-defeating mistake.

Direct answer: I'd layer health signals from cheap-and-fast (process liveness) to expensive-and-meaningful (dependency and business-metric checks), require multiple consecutive failures before declaring a node unhealthy to avoid reacting to a single blip, and keep a human in the loop specifically for the step that's hardest to reverse: promoting a new primary for stateful services, where an incorrect automated failover can cause data loss or a split-brain (two nodes each believing they're the one true primary and accepting conflicting writes at the same time), versus something like removing an unhealthy instance from a load-balancer pool, which is cheap to reverse and safe to fully automate.

Structured elaboration

Why "N consecutive failures" instead of one: a single failed check can be a genuine blip (a GC pause, a brief network hiccup) rather than a real failure. If each check independently has some baseline flakiness probability $p$ (a transient failure unrelated to a real outage), then requiring $N$ consecutive failures before declaring unhealthy makes the false-trigger probability:

With, say, $p = 0.05$ (5% chance any single check fails transiently) and $N=1$ (react on the first failure), the false-trigger probability is just $p = 5\%$, meaning roughly 1 in 20 blips would incorrectly trigger action. Requiring $N=3$ consecutive failures drops that to:

a 400x reduction, at the cost of a real failure now taking 3 check intervals (here, up to 15 seconds at a 5s interval) longer to detect instead of one. That's the actual dial being turned: detection speed versus false-positive rate, and it should be set from an observed flakiness rate for your specific checks, not copied from another team's runbook.

Decision flow, including where automation stops and a human is required:

flowchart TD
    A[Health check runs] --> B{N consecutive<br/>failures?}
    B -->|No| A
    B -->|Yes| C{What kind of<br/>action?}
    C -->|Remove from LB pool| D[Fully automated:<br/>cheap, instantly reversible]
    C -->|Open circuit breaker| D
    C -->|Promote new primary<br/>for stateful service| E{Safety checks pass?<br/>replica caught up,<br/>quorum reachable}
    E -->|No| F[Page human,<br/>do not auto-promote]
    E -->|Yes, and blast radius<br/>is well-understood| G[Auto-promote,<br/>but page for review]
    E -->|Yes, but ambiguous<br/>e.g. partition, not clear failure| F

"Quorum reachable" in that safety check means enough replicas are online and able to vote that a new primary can be safely elected, a majority agreeing on who's in charge, without risking two nodes each believing they're the primary at once.

Trade-offs & pitfalls

• Fully automated failover is right when the action is cheap and reversible (removing an unhealthy node from rotation); it's risky when the action is expensive or irreversible (promoting a database replica, since promoting the wrong one, or promoting during a network partition rather than an actual failure, can cause split-brain or data loss). The line isn't "how critical is the service," it's "how reversible is this specific action."
• Requiring consecutive failures trades detection speed for false-positive protection; too aggressive a requirement (e.g., N=10) means a genuine failure runs uncaught far longer than the blast radius justifies.
• Checks that themselves depend on a shared resource (e.g., every health check queries the same central database) can produce correlated, simultaneous "failures" across an entire fleet when that shared resource degrades, which looks like a mass outage but is really a single point of failure in the monitoring path itself.
• A common wrong turn: only checking process liveness and assuming that's sufficient. A process can be alive, passing liveness checks, and still be completely unable to serve real traffic because its only database connection pool is exhausted; readiness and dependency checks catch what liveness checks structurally cannot.