Fault Tolerance, High Availability, and Disaster Recovery Questions
Keeping a system serving despite failure, from code-level resilience to infrastructure-level recovery: circuit breakers, retries with backoff and jitter, timeouts, bulkheads, graceful degradation, and preventing cascading failures, alongside redundancy, failover (active-active versus active-passive), RPO and RTO objectives, backup and restore, and multi-region failover. Covers dependency-failure isolation, chaos engineering to validate resilience, failure-mode analysis, designing to nines of availability, cost-versus-availability tradeoffs, and recovery runbooks. Spans both the patterns that isolate partial failure and the disaster-recovery planning that restores a business-critical system after a major outage.
Using the CAP theorem, walk through the trade-off you'd make for a financial ledger service versus an analytics event aggregator. Which guarantee does each give up during a network partition, and why?
Sample Answer
CAP theorem says that during a network partition, a distributed system must choose between consistency (every read sees the latest acknowledged write) and availability (every request gets a response), because it can't guarantee both while the partitioned halves can't talk to each other. Partition tolerance itself isn't optional for any system that spans more than one node, so the real choice interview questions are testing is CP versus AP, and the right answer depends entirely on what happens if you get it wrong.
Financial ledger: choose CP
A ledger getting a stale or divergent balance is a correctness bug with real financial consequences (double-spend, incorrect balance shown, a transaction accepted twice), so consistency has to win. The concrete mechanism is a consensus protocol requiring a write quorum.
Deriving what "sacrifice availability" actually means, with a pinned example: take a 5-node replica set (a typical Raft cluster size) where a majority quorum is required to commit a write:
quorum=⌊2N⌋+1=⌊25⌋+1=3That cluster tolerates up to N−quorum=2 node failures while still committing writes. Now suppose a network partition splits the 5 nodes into a 3-node side and a 2-node side. The 3-node side still has a majority (3 ≥ quorum of 3), so it keeps accepting writes and stays both consistent and available. The 2-node side does not have a majority (2 < 3), so by design it must refuse writes, becoming unavailable, specifically to prevent both sides from independently committing conflicting transactions. That refusal on the minority side, not a global shutdown, is the literal, computable meaning of "CP sacrifices availability during a partition": only the minority partition goes unavailable, and only for writes.
Analytics event aggregator: choose AP
An analytics pipeline getting an event a few seconds late, or briefly double-counted before deduplication catches up, is a rounding error on a dashboard, not a financial loss, so availability wins: every node keeps ingesting even when it can't see the others.
The mechanism looks different from the ledger: instead of a write quorum gatekeeping every write, use leaderless or partitioned ingestion (each partition or node accepts writes for its own shard independently, as in Kafka-style partitioned logs or a Dynamo-style leaderless store), so there's no majority to lose and no write path that can be blocked by a partition. The cost shifts from "unavailable during a partition" to "eventually reconciled after one": deduplication on ingest, idempotent consumers, and background reconciliation jobs to merge whatever diverged while the partition was open.
Comparing the two designs
| Dimension | Ledger (CP) | Analytics aggregator (AP) |
|---|---|---|
| Write path | Quorum-gated (majority must ack) | Leaderless / partitioned, no quorum gate |
| During a partition | Minority side refuses writes | Both sides keep accepting writes independently |
| Consistency mechanism | Consensus (Raft/Paxos-style) | Eventual consistency + reconciliation |
| Failure mode if you pick the wrong side | Double-spend, incorrect balances | Stale dashboard, temporary undercounts |
| Client-side pattern needed | Idempotency keys so a client retry after a rejected write is safe | Deduplication keys so late/duplicate events don't double-count on reconciliation |
Trade-offs and pitfalls
The common misreading of CAP is treating CP as "the whole system goes down during any partition," when the quorum math above shows it's specifically the minority side, and only for writes, reads can often still be served (possibly stale, depending on the read-consistency level chosen). The common mistake on the AP side is stopping at "it's eventually consistent" without actually building the reconciliation path: leaderless ingestion without idempotent consumers and deduplication just relocates the correctness problem downstream instead of solving it. And even a CP ledger still needs idempotency keys on the client side: a client that times out waiting for a quorum ack and retries the same transaction must not have it applied twice, which is a consistency concern CAP itself doesn't cover but that any real ledger design has to handle regardless of which side of CAP it lands on.
Define cascading failure and walk through a realistic example: service C fails, B (which depends on C) gets overloaded, and A (which depends on B) starts degrading too. At each layer, what protection would you put in place to stop the cascade from propagating?
Sample Answer
Direct answer
A cascading failure is when one component's failure increases load or latency on the components that depend on it, and that increased load causes those components to fail too, propagating outward until a large part of the system is affected, even though only one component actually broke in the first place. The mechanism is almost always resource exhaustion: threads, connections, or memory tied up waiting on the failed component instead of being freed quickly.
Walkthrough: C fails, B overloads, A degrades
flowchart LR
A[API Gateway] -->|rate limit and timeout| B[Order Service]
B -->|bulkhead pool: payments| C[Payment Service]
C -.fails.-> B
B -->|circuit breaker opens| D[Fallback: queue order for async retry]
A -->|circuit breaker opens| E[Fallback: 503 with Retry-After]
B -->|isolated pool: other deps unaffected| F[Inventory Service]
- C (Payment Service) fails, hanging instead of returning errors quickly, perhaps due to a downstream outage of its own.
- B (Order Service) calls C without a tight timeout. Each call to C now blocks for far longer than normal, tying up a thread or connection from B's pool for the duration.
- B's resource pool exhausts. As more requests arrive at B, more threads get stuck waiting on C, until B has no capacity left to serve any request, including ones that don't even touch C.
- A (API Gateway) calls B, and B is now slow or unresponsive for everything, so A's calls to B start timing out or queueing too, degrading A's own capacity in turn.
Worked example: how fast does B's pool actually exhaust?
Little's Law relates the number of requests in flight to the arrival rate and the time each spends being processed:
L=λWSay B receives 500 requests per second, and under normal conditions each call to C takes 50ms:
Lnormal=500×0.05=25 concurrent in-flight requests25 concurrent requests is a light load on a typical connection pool. Now C hangs, and B's HTTP client has no explicit timeout of its own, falling back to a default of 30 seconds:
Lfailure=500×30=15,000 concurrent in-flight requests neededIf B's thread pool has 200 threads, the time to exhaust it entirely is:
texhaust=500200=0.4 sUnder 400 milliseconds. That's how quickly a single hung dependency with no timeout turns into total unavailability for a service handling 500 requests per second: the pool never gets close to steady-state at the 30-second hang time, it simply fills with stuck requests almost instantly and stays full.
Protections at each layer
- At B, calling C: a tight, explicit timeout (measured in low hundreds of milliseconds, not the client library's 30-second default) so a hung call fails fast and frees the thread quickly; a circuit breaker that opens after a run of failures or timeouts, so B stops even attempting calls to C once it's clearly down, and falls back to queueing the order for later processing; a bulkhead, a dedicated connection pool just for calls to C, so exhaustion from C-related calls doesn't consume the threads B needs to serve requests that don't touch C at all (like inventory checks).
- At A, calling B: the same pattern one layer up, a timeout on calls to B, a circuit breaker that trips once B's error rate or latency crosses a threshold, and a fallback (a fast 503 with
Retry-Afterrather than a hung request) so A's own capacity isn't consumed waiting on a B that's already struggling.
Trade-offs & pitfalls
Timeouts that are too aggressive cause false-positive failures under normal, brief latency variance; timeouts that are too loose don't prevent the cascade fast enough, as the Little's Law example shows. Bulkheads cost real resources (a dedicated pool per dependency uses more total connections or threads than one shared pool) in exchange for isolation, so they're worth applying to the dependencies most likely to fail or most likely to take down unrelated traffic if they do. The most common mistake is only protecting the first hop (B to C) and assuming that's sufficient; as the walkthrough shows, without protection at the A to B hop too, the failure still reaches A once B is degraded, just one layer later.
You're given a timeline from an outage: a database compaction job triggers a CPU spike, latency breaches its SLO, the autoscaler overreacts and adds pods, network queues fill up, a leader election kicks off, and the system ends in a multi-region failover. Walk through a structured root-cause analysis: the immediate cause, the contributing factors, and at least three long-term changes you'd prioritize to prevent a repeat.
Sample Answer
Direct answer
Structure the RCA in three layers: the immediate cause (the single technical event that started the chain), the contributing factors (the systemic gaps that let a local blip cascade into a multi-region event), and at least three long-term fixes, each targeted at a specific link in the chain rather than a generic "add more monitoring."
Working the timeline
flowchart TD
A[Compaction job starts] --> B[CPU/IO spike on primary]
B --> C[Latency breaches SLO]
C --> D[Autoscaler adds pods]
D --> E[Network queues fill]
E --> F[Leader election triggered]
F --> G[Multi-region failover]
Immediate cause: a database compaction job drove sustained CPU and I/O contention on the primary node, pushing request latency past its SLO.
Contributing factors, one per causal hop:
- No compaction scheduling or resource throttling, so a routine maintenance job was free to consume the same resources serving live traffic.
- The autoscaler reacted to a single metric (latency or CPU) without a stabilization window, and added pods that increased load on the already-contended storage and network layer rather than relieving it, a classic autoscaler pile-on.
- No backpressure or circuit-breaker between the newly added pods and the network fabric, so the extra load filled queues instead of being shed or rate-limited.
- Leader election was sensitive to transient latency spikes rather than sustained, confirmed unavailability, so it fired on a symptom instead of a real failure.
- The election's ambiguity (or its side effects) triggered a multi-region failover, an enormous blast-radius response to what was, at its root, a single-node maintenance-job problem.
Worked example: applying the framework to this timeline
Table mapping each causal link to a specific, targeted fix:
| Causal link | Long-term fix | What it prevents |
|---|---|---|
| Compaction ran unthrottled during live traffic | Schedule and resource-limit compaction (ionice/cgroups or a load-aware maintenance window) | Removes the trigger event entirely under normal operating conditions |
| Autoscaler piled on during contention | Multi-metric, stabilization-windowed scaling decisions (queue depth and error rate, not latency alone, with a cooldown) | Stops added capacity from making resource contention worse |
| No backpressure between pods and network fabric | Circuit breakers and bounded queues with load shedding | Converts a cascading pile-up into a contained, visible degradation |
| Leader election fired on transient latency | Require sustained, multi-signal confirmation (not a single latency spike) before triggering election | Prevents unnecessary elections from becoming unnecessary failovers |
Any three of these, prioritized by how early they intercept the chain (compaction throttling breaks the trigger, the autoscaler and backpressure fixes break the propagation, the leader-election fix breaks the final escalation), meet the "at least three" bar and target the actual mechanism rather than a symptom.
Trade-offs and pitfalls
The most common wrong turn is fixing only the most visible layer, throttling the compaction job, and declaring the postmortem closed. That removes this specific trigger but leaves the autoscaler pile-on and missing backpressure in place, so a different trigger (a slow query, a burst of legitimate traffic) would cascade through the exact same chain next time. Prioritize fixes that break the propagation path (autoscaler stabilization, backpressure) at least as highly as the one that removes this incident's specific trigger. Also resist the instinct to add more alerts as a "fix": better detection of the same broken chain isn't a structural fix, it just tells you faster that the cascade is happening again.
What does a disaster recovery runbook actually need to contain to be useful during a real region failure? Walk through the essential sections: owner, RTO/RPO, step-by-step actions, and verification.
Sample Answer
Direct answer
A disaster recovery runbook is only useful if a stressed engineer can execute it top to bottom without needing to look anything else up. That means naming an owner, stating the RTO and RPO it is designed to meet, listing ordered and specific actions (exact commands or console steps, not descriptions), and ending with concrete verification steps that prove the system is actually back, not just that the steps were followed.
Structured elaboration
| Section | What it contains | Why it's required |
|---|---|---|
| Title and scope | Which system or service, and which failure modes it covers | A runbook that does not say what it's for gets grabbed for the wrong incident |
| Owner and escalation contacts | Primary owner, backup owner, and how to reach them, not just a name | Someone must be accountable for keeping it accurate and reachable during the incident |
| RTO / RPO | The target time to recover and the acceptable data loss this runbook is designed to hit | Without a target, "did the runbook work" has no answer |
| Prerequisites | Required access, credentials, tickets, and any upstream dependency that must already be healthy | Discovering you're locked out mid-incident is the worst time to find out |
| Step-by-step actions | Numbered, specific commands or console actions, including a rollback for each risky step | Vague steps like "promote the standby" force the responder to improvise under pressure |
| Verification steps | Health checks, smoke tests, and the specific metrics that confirm recovery | "The steps finished" is not the same as "the system works" |
| Post-incident tasks | Root cause capture, stakeholder communication, runbook update | A runbook that isn't updated after every real use rots |
Versioning and access. Store the runbook in source control with required review on changes, so every edit has an author and a diff. Test it in a real drill, not a tabletop discussion only, on a cadence tied to how critical the service is, quarterly for anything customer-facing. Keep it reachable when the primary systems it recovers are down: a runbook that lives only on an internal wiki hosted in the region that just failed is not a disaster recovery runbook.
Worked example
For a service with RTO = 30 minutes, a well-built runbook's step timings should sum to that budget, and the sum should be checked, not assumed:
| Step | Budget |
|---|---|
| Detection and paging | 5 minutes |
| Triage and decision to fail over | 5 minutes |
| Execution (promote standby, update routing) | 15 minutes |
| Verification (smoke tests, dashboards green) | 5 minutes |
That sum matching the stated RTO exactly is what makes the RTO a testable claim rather than a number pasted at the top of the document. If a quarterly drill shows execution consistently takes 20 minutes instead of 15, the runbook's RTO is wrong and needs to be corrected, not explained away.
Trade-offs & pitfalls
- A runbook with no owner drifts out of date the first time the architecture changes; ownership is not optional metadata.
- Testing via tabletop discussion only, never an actual drill, hides the gap between the steps sounding right and the steps working; most drift is caught only by execution.
- Over-specifying every command for a fast-moving system creates a maintenance burden that causes the runbook to be abandoned; balance specificity against how often the underlying commands change.
- Storing the only copy behind the same authentication system that depends on the region that just failed is a common, self-defeating mistake.
Two regions running asynchronous replication get network-partitioned, and both keep accepting writes. When the partition heals, how do you detect the divergence and reconcile the conflicting writes?
Sample Answer
Direct answer
This is the multi-master conflict case: because replication was asynchronous and both regions kept accepting writes, there are now two divergent histories that both look locally valid. Detecting divergence means comparing state between regions, via hashes or version metadata, once connectivity returns, not waiting for a customer to report bad data. Reconciling means classifying each conflict by whether it's safely auto-mergeable or needs a compensating action; financial and other invariant-sensitive writes should never be silently overwritten.
Structured elaboration
Detecting divergence. Attach causal metadata to every write, a vector clock (an array with one counter per region, incremented only when that region processes a write, so comparing two vectors shows whether one write causally happened after the other, every counter at least as high, or whether the two happened independently) or at minimum a per-region monotonic sequence number plus timestamp, so that on reconnect two versions of the same key can be compared to tell, mathematically, whether one supersedes the other or whether they are genuinely concurrent, meaning both sides wrote independently during the partition.
sequenceDiagram
participant RegionA
participant RegionB
participant Reconciler
Note over RegionA,RegionB: Partition: both regions accept writes independently
RegionA->>RegionA: Write key X, vector clock [A:5,B:3]
RegionB->>RegionB: Write key X, vector clock [A:4,B:4]
Note over RegionA,RegionB: Partition heals
Reconciler->>RegionA: Fetch change log with vector clocks
Reconciler->>RegionB: Fetch change log with vector clocks
Reconciler->>Reconciler: Compare vector clocks, classify conflicts
Reconciler->>Reconciler: Auto-merge commutative writes
Reconciler->>Reconciler: Apply compensating transaction for invariant conflicts
Reconciler->>RegionA: Apply resolved state
Reconciler->>RegionB: Apply resolved state
Reconciling. Not every conflict resolves the same way:
| Conflict type | Resolution |
|---|---|
| Non-overlapping keys, touched on only one side | Trivial union merge, no real conflict |
| Commutative or CRDT-safe operations (counter increment, set-add). CRDT: conflict-free replicated data type, a data structure designed so concurrent updates always merge to the same result automatically, no coordination needed | Merge automatically, order doesn't matter |
| Same-field conflicting writes with no safe merge rule | Deterministic policy where business semantics allow it, such as latest-timestamp-wins, logged to a resolved-log for audit |
| Financial or invariant-sensitive conflicts | Never blind-overwrite; apply a compensating transaction, an explicit correcting entry linked to the original, so history stays auditable and reversible |
Worked example: detecting a real conflict with vector clocks
Two regions independently update the same key, "user:42.email," during the partition. Region A's write carries vector clock VA=[A:5,B:3], Region B's write carries VB=[A:4,B:4].
A vector clock V1 dominates V2, meaning V1 happened after V2 and can safely overwrite it, only if every component of V1 is greater than or equal to the corresponding component of V2, with at least one strictly greater. Checking:
VA[A]=5>VB[A]=4butVA[B]=3<VB[B]=4Neither vector dominates the other, A leads on its own component, B leads on its own, so this is a genuine concurrent conflict, not a case where one write simply supersedes the other, and it must go through the merge policy rather than being resolved by picking the higher-looking clock.
Worked example: scoping the reconciliation workload
Assume the partition lasted 10 minutes, each region accepted 150 writes/sec across a 5-million-key space:
n1=n2=150×600=90,000 writes per side E[colliding keys]=Nn1×n2=5,000,00090,000×90,000=5,000,0008,100,000,000=1,620Of 180,000 total writes across both sides, an estimated 1,620 keys, 0.9%, are true conflicts needing the merge policy; the rest merge by simple union. Sizing the reconciliation pipeline, and any manual-review queue, around this number, not around all writes during the partition, is what keeps the process fast.
Trade-offs & pitfalls
- Silently applying last-writer-wins to every conflict is the most common shortcut, and it's the wrong default for anything with a business invariant, balances, inventory counts: it loses data without a trace and without anyone knowing which write "won."
- Scanning the entire keyspace for divergence, instead of the range touched during the outage window, wastes time and delays recovery.
- A merge policy that isn't tested against the conflict-type classification before the incident means the policy is being designed live, under pressure, which is when mistakes compound.
- Compensating transactions must themselves be idempotent and auditable; a reconciliation "fix" that isn't traceable back to the original conflicting writes creates a second, harder-to-debug data quality problem.
Unlock Full Question Bank
Get access to all Fault Tolerance, High Availability, and Disaster Recovery interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.