Fault Tolerance, High Availability, and Disaster Recovery Questions
Keeping a system serving despite failure, from code-level resilience to infrastructure-level recovery: circuit breakers, retries with backoff and jitter, timeouts, bulkheads, graceful degradation, and preventing cascading failures, alongside redundancy, failover (active-active versus active-passive), RPO and RTO objectives, backup and restore, and multi-region failover. Covers dependency-failure isolation, chaos engineering to validate resilience, failure-mode analysis, designing to nines of availability, cost-versus-availability tradeoffs, and recovery runbooks. Spans both the patterns that isolate partial failure and the disaster-recovery planning that restores a business-critical system after a major outage.
What's the difference between N+1 and N+2 redundancy? For a service normally sized at 10 instances, walk through what each strategy actually buys you in failure tolerance, and when the extra cost of N+2 is worth it.
Sample Answer
Direct answer: N+1 means you provision one spare unit beyond what's needed to serve current load, so the system tolerates exactly one simultaneous failure with zero capacity loss. N+2 provisions two spares, tolerating two simultaneous failures (or one failure plus a second one arriving while the first is still being repaired). For a service sized at 10 instances, N+1 is 11 instances and N+2 is 12; the extra instance in N+2 is worth it when failures are likely to be correlated or when repair (MTTR) is slow enough that a second failure landing during the first one's recovery window is a real possibility, not a hypothetical.
Structured elaboration
- What "N" means: N is the number of units actually required to serve load at your target performance, not the number you happen to run. If 10 instances are needed to handle peak traffic at acceptable latency, N=10.
- N+1: one extra unit. Any single instance, host, rack, or power supply can fail and the system still serves at full capacity from the remaining N. It does not protect against a second, overlapping failure.
- N+2: two extra units. Protects against two simultaneous failures, which matters specifically during the repair window of the first failure (you're running on N+1 capacity while node 1 is being replaced; if node 2 fails during that window, N+1 would drop you below N, but N+2 still covers you).
- When N+2's extra cost is worth it: the decision comes down to how correlated failures are and how long repair takes, not just how critical the service is in the abstract.
Worked example: quantifying the risk N+2 removes
A capacity shortfall only happens when multiple instances are down at the same time, which means the model has to use the instantaneous probability that an instance is down at any given moment, not the probability that it fails at some point during the year (an annual failure probability answers a different question and silently ignores repair-window overlap). The right building block is the instantaneous-unavailability formula: at any random moment, the fraction of time a single instance has historically spent broken and being repaired is MTTR divided by the full working-plus-repair cycle, MTBF+MTTR, which is exactly the probability that instance happens to be down at an arbitrary moment in time:
q=MTBF+MTTRMTTRPin illustrative values: each instance has an MTBF of 8,760 hours (fails on average about once a year) and an MTTR of 4 hours (time to detect and replace or restart a failed instance). Then:
q=8760+44=87644≈0.000456(0.0456%)That's the probability any single instance is down (mid-repair) at a random moment.
For N+1 (11 total instances), capacity drops below the needed N=10 only if 2 or more instances are down simultaneously:
P(down≥2∣n=11,q)=1−(011)(1−q)11−(111)q(1−q)10 =1−0.994991−0.004998=0.0000114(0.00114%)For N+2 (12 total instances), capacity drops below 10 only if 3 or more are down simultaneously:
P(down≥3∣n=12,q)=1−k=0∑2(k12)qk(1−q)12−k =1−0.994537−0.005450−0.0000137=0.0000000209(0.0000021%)Both numbers are tiny snapshot probabilities; the more useful reading is as the expected fraction of the year the system spends in a shortfall state, converted into expected annual downtime minutes by taking that same fraction-of-time-in-shortfall and multiplying it by the number of minutes in a year, 525,600 (365 days x 24 hours x 60 minutes), the standard way a fraction-of-time becomes an annual downtime figure:
N+1: 0.0000114×525,600≈6.0 minutes/year N+2: 0.0000000209×525,600≈0.011 minutes/year(≈0.66 seconds/year)So under this repair-window-conditioned model, N+1 carries about 6 minutes/year of expected capacity-shortfall exposure, and N+2 cuts that to about 0.01 minutes/year, roughly a 548x reduction, not because any instance's individual failure rate changed, but because a shortfall now requires a second failure to land inside the narrow repair window of the first, and adding a spare pushes that bar from "2 simultaneous" to "3 simultaneous," a much rarer event once q is small. Whether that ~6-minute-a-year difference is worth one extra instance's cost is exactly the trade-off to walk through out loud: for a service where even a few minutes of capacity shortfall risks an SLA breach, cutting expected exposure by roughly two and a half orders of magnitude for one extra instance is usually cheap insurance; for an internal batch service, shortfall risk this small to begin with is very likely not worth the extra spend.
Common concrete instances of the same reasoning: UPS/power-supply sizing (N+1 power modules in a rack survive one PSU failure; N+2 covers one failed unit plus one more failing during the swap), and network device sizing (N+1 top-of-rack switches vs N+2 when switch firmware upgrades take units offline for extended maintenance windows, effectively acting like a "planned failure" that N+1 alone can't absorb if an unplanned one happens at the same time).
Trade-offs & pitfalls
- N+2 isn't "more reliable" in a vacuum, it's specifically insurance against overlapping failures; if your MTTR is minutes and failures are rare and independent, N+1 is usually sufficient and N+2 is paying for a scenario that almost never occurs.
- Fault-domain correlation matters more than the raw redundancy count: N+1 spread across a single rack doesn't protect against a rack-level power failure taking out several "independent" instances at once; redundancy has to be placed in genuinely independent failure domains (different racks, AZs, or power feeds) or the N+1/N+2 math above doesn't hold, because the independence assumption breaks.
- A common wrong turn: treating N+1 as "one extra instance total" when instances are correlated (e.g., all on the same physical host or the same AZ). The formula only protects capacity if the spare's failure mode is independent of the others.
- N+2 costs roughly 20% more standing capacity than N+1's 10% here; that recurring cost has to be justified against the downtime cost it avoids, not assumed.
What is chaos engineering, and why would a company deliberately break its own production systems on purpose? Walk through the basic methodology: how you'd define steady state, form a hypothesis, and run a safe first experiment.
Sample Answer
Chaos engineering is the practice of deliberately injecting failure into a system, in a controlled way, to find weaknesses before they find you during a real incident. The reasoning behind doing it on purpose: most production failures aren't hypothetical, dependencies do time out, nodes do crash, networks do partition, and the choice isn't between "failures happen" and "failures don't happen," it's between discovering how your system responds to them during a planned, low-stakes experiment or during an unplanned, high-stakes 3 a.m. page.
Methodology
1. Define steady state. Pick measurable indicators of normal health, request success rate, latency percentiles, throughput, that represent "the system is working" in terms an on-call engineer would actually check on a dashboard, not an abstract notion of "healthy."
2. Form a hypothesis. State, before running anything, what you expect to happen and why: "if we kill one instance of the recommendation service, overall page error rate will stay flat because the client has a fallback path." A real hypothesis is falsifiable; "let's see what happens" isn't chaos engineering, it's just causing an outage without a way to learn from it.
3. Design a safe first experiment. Choose the smallest fault that could test the hypothesis (kill one non-critical replica, not the whole fleet) and decide the blast radius up front: what fraction of traffic or users can be affected, and for how long.
4. Run it with an abort condition already defined. Before starting, decide the exact metric threshold that ends the experiment immediately (for example, page error rate exceeding a set ceiling), so the decision to stop isn't made under pressure in the moment.
5. Observe against the steady-state baseline. Watch the same metrics defined in step 1, not new ones invented mid-experiment, so the comparison is apples-to-apples.
6. Learn and iterate. If the hypothesis held, expand the blast radius gradually on future runs. If it didn't, that's the actual finding, fix the missing fallback or retry logic, and re-run the same experiment to confirm the fix works before calling it done.
Worked example: a first, safe experiment
Target: a non-critical "related items" widget on a product page, deliberately chosen because a broken hypothesis here degrades a widget, not checkout. Steady state: page load success rate and p95 latency, whatever their current normal values are for that page. Hypothesis: "terminating one replica of the related-items service will not change page load success rate or p95 latency, because the front end treats that service as optional with a client-side timeout and empty-state fallback." Experiment: kill one replica (not all of them) during a low-traffic window, with an abort condition of "page success rate drops below its normal range" defined before starting. Outcome either confirms the fallback works as designed, or reveals it doesn't, which is the actual value of running it: finding that out on a Tuesday afternoon experiment instead of during a real node failure at peak traffic.
Trade-offs and pitfalls
The most common misunderstanding is that chaos engineering means "randomly break things in production," when the entire method is built around the opposite instinct: a stated hypothesis, a bounded blast radius, and a predefined abort condition are what separate a chaos experiment from just causing an outage. A related pitfall is skipping the hypothesis step and injecting a fault "to see what happens": without a stated expectation, there's no way to say afterward whether the result was surprising or how bad it was relative to what should have happened. Teams also sometimes skip straight to production chaos before validating the tooling and abort mechanism in staging first, running the injection and rollback machinery against a stage environment is itself a smaller, safer experiment worth doing before trusting it against real traffic.
A downstream service you depend on starts responding slowly, and requests to it start backing up on your side, growing queues and increasing latency. Walk through your immediate mitigations and your longer-term architectural fix, and explain the trade-off each one introduces.
Sample Answer
Direct answer: The immediate priority is to stop the slowdown from consuming your own resources: set aggressive timeouts, open a circuit breaker so you stop calling the failing dependency, and isolate the connection/thread pool used for that call so it can't starve everything else. The longer-term fix is architectural: decouple the caller from the dependency's latency entirely, usually via an async queue or by making the call non-blocking, so a slow downstream degrades throughput instead of taking the whole service down with it.
Structured elaboration
Why this happens (the mechanism): by Little's Law, the number of requests in flight L equals arrival rate λ times the time each request spends in the system W: L=λW. If a downstream call's latency goes from 50ms to 500ms while your request rate stays at, say, 200 requests/second, the in-flight count grows from L=200×0.05=10 to L=200×0.5=100, a 10x increase, purely from the latency change with no change in incoming traffic. If your thread or connection pool was sized for ~10-20 concurrent in-flight requests to that dependency, it's now exhausted, and requests start queueing on your side, which is exactly the symptom described.
Immediate mitigations (minutes, not a redesign):
| Mitigation | What it does | Trade-off it introduces |
|---|---|---|
| Tight timeouts | Caps how long you'll wait, preventing unbounded queue growth | Cuts off requests that might have succeeded a moment later; needs to be shorter than your own SLA to the caller |
| Circuit breaker | Stops calling the dependency once error/latency crosses a threshold, failing fast instead of queueing | Can trip on transient blips if thresholds are too sensitive; denies service even to calls that might succeed |
| Bulkhead (isolated pool) | Gives this dependency its own thread/connection pool so its slowdown can't exhaust pools shared by healthy dependencies | Reduces pooled efficiency (can't borrow capacity across dependencies); requires knowing sizing up front |
| Load shedding / fast 503 | Rejects excess requests immediately when queue depth crosses a threshold, protecting the instances still healthy | Directly reduces availability for shed requests; needs to shed selectively, not randomly, if some requests matter more |
Longer-term architectural fix:
- Decouple via an async queue: put a durable queue between the caller and the slow dependency so the caller can return quickly (accept-and-acknowledge) and the dependency is drained at its own sustainable pace, rather than the caller blocking on it synchronously. Trade-off: the caller can no longer return a synchronous success/failure for that operation; the interaction model has to change to something the client and product can tolerate (a "pending" state, a webhook, a poll).
- Idempotent retries with backoff and jitter: if retries are needed, they must be capped, exponential, and jittered so a fleet of callers doesn't retry in lockstep and re-create the exact overload it's recovering from. Trade-off: added complexity, and retries must be provably idempotent on the downstream side or they risk duplicate side effects.
- Capacity planning against the tail, not the average: provision the dependency (or the pool sized to call it) based on observed p99 latency, not p50, since it's the tail that determines when queues start building. Trade-off: costs more standing capacity for headroom that's idle most of the time.
Applying this to concrete variants of the same pattern: the reasoning above is the same whether the slow dependency is a payment-validation service (immediate: circuit breaker + fast-fail with a clear "try again" to the user rather than a silent hang; long-term: async payment confirmation via webhook), a message-queue consumer falling behind (immediate: shed or dead-letter the oldest low-priority messages, bulkhead the consumer pool by message type; long-term: scale consumers horizontally and partition by priority), a retry storm from a flood of client-side 503s (immediate: the client-side backoff-with-jitter above is the direct fix; long-term: make the shedding threshold adaptive so it doesn't itself become the trigger for a thundering herd), or a synchronous order-processing pipeline backing up (immediate: bulkhead the slow stage's pool; long-term: convert that stage to the async-queue pattern above).
Trade-offs & pitfalls
- Every immediate mitigation above trades some availability or correctness for stability: timeouts drop requests that might have succeeded, circuit breakers deny service during their open window, load shedding sacrifices some requests to save the rest. The point isn't to avoid the trade-off, it's to make it deliberately and visibly rather than let an unbounded queue make it for you via an eventual crash.
- A common wrong turn: adding retries as the first response to a slowdown. Naive retries without backoff amplify load on an already-struggling dependency and can turn a partial slowdown into a full outage (a retry storm).
- Circuit breakers and bulkheads need to be tuned against real traffic and latency distributions; thresholds copied from a different service's runbook are a common source of either false trips (unnecessary unavailability) or no protection at all (thresholds too loose to matter).
What's the difference between fault tolerance, high availability, and resilience? Give a concrete example of each, and explain how they show up in operational metrics like MTTR and MTBF.
Sample Answer
Fault tolerance, high availability, and resilience are related but answer different questions. Fault tolerance means a failure is masked entirely: the system keeps producing correct output with no visible interruption. High availability (HA) means downtime is minimized and recovery is fast, but a brief, bounded interruption is expected and acceptable. Resilience is the broadest term: the system's ability to keep delivering acceptable service under any kind of stress, not just component failure, including load spikes, bad inputs, or a slow dependency, usually by degrading gracefully rather than failing outright.
How each shows up architecturally
| Property | What it guarantees | Typical mechanism | Concrete example |
|---|---|---|---|
| Fault tolerance | No visible interruption during a failure | Redundant, synchronized components that vote or replicate in lockstep (RAID, dual power supplies, Raft/Paxos-replicated state) | A 3-node Raft cluster loses one node; the other two still form a quorum (a majority of the cluster, here 2 of the original 3 nodes, enough to safely keep operating and elect a leader if needed) and serve every request with zero downtime |
| High availability | Short, bounded interruption, fast automated recovery | Health-checked redundancy plus automated failover (active-passive DB failover, load-balanced app tier) | A primary database instance crashes; a monitor detects it in seconds and promotes a replica, so the outage is measured in seconds to low minutes, not zero |
| Resilience | Acceptable service continues even when something can't be masked or failed over cleanly | Circuit breakers, timeouts, bulkheads, graceful degradation, autoscaling | A recommendation service starts timing out; the product page serves without recommendations instead of failing the whole page load |
Fault tolerance and HA are usually about infrastructure failing; resilience is about the system's response to any kind of stress, including ones where nothing has technically "failed" yet (a slow but technically-up dependency, for instance).
Effect on MTTR and MTBF
MTTR (mean time to repair/recover) and MTBF (mean time between failures) combine into steady-state availability:
A=MTBF+MTTRMTBFFault tolerance mainly extends effective MTBF from the user's point of view: individual component failures still happen at whatever rate they happen, but they don't count as user-visible failures because they're masked, so the failure interval a customer would notice grows. High availability mainly drives MTTR down: the goal isn't to prevent the primary from ever failing, it's to make detection and recovery fast and automatic. Resilience patterns move both numbers in the same direction from a different angle: a circuit breaker doesn't prevent a dependency from failing (MTBF of the dependency is unchanged) but it prevents that dependency's failure from becoming your incident at all, which is a third way to improve the user-visible number.
Worked example
Take a service with a component MTBF of 720 hours and an MTTR of 30 minutes (0.5 hours) once a failure is detected and recovered:
A=720+0.5720=720.5720≈0.99931(99.931%)That converts to annual downtime using 525,600 minutes per year (365 days × 24 hours × 60 minutes):
(1−0.99931)×525,600≈364.8 minutes/year≈6.08 hours/yearNow compare two improvements starting from that baseline, holding the other variable fixed:
- Cut MTTR to 5 minutes (better HA: faster automated failover) with MTBF unchanged at 720h: A=720/720.083≈0.999884, about 61.0 minutes/year of downtime, a ~6x reduction driven entirely by faster recovery.
- Double MTBF to 1440 hours (better fault tolerance: the failure that used to happen now gets masked half as often) with MTTR unchanged at 30 min: A=1440/1440.5≈0.999653, about 182.4 minutes/year, a 2x reduction.
Neither number is "the metrics." They're two independent levers on the same availability formula, and which one is cheaper to pull depends on the system: automating failover (MTTR) is often cheaper than adding redundant hardware paths everywhere (MTBF).
Trade-offs and pitfalls
The common mix-up in interviews is treating "high availability" as if it means "never goes down," which is what fault tolerance actually promises, and at a much higher engineering cost (consensus protocols, lockstep replication) than HA's health-check-and-failover pattern. A resilient system is not automatically fault-tolerant or highly available either: a service with excellent circuit breakers and graceful degradation for its dependencies can still have a single database with no HA story of its own. These three properties are complementary, not substitutes, and a system typically needs different amounts of each depending on the blast radius of the component: mask failures (fault tolerance) for the smallest, cheapest, most critical primitives; fail over fast (HA) for stateful tiers where full masking is expensive; and degrade gracefully (resilience) at the edges where "reduced functionality" beats "hard failure." The same reasoning applies outside a classic web-service stack too: a GPU training job gets fault tolerance from checkpointing plus redundant nodes (a crashed worker resumes from the last checkpoint instead of restarting the whole job), and a data pipeline gets resilience from feature-store (a system that serves precomputed inputs to a machine-learning model) fallback values or a stale-but-served cache when an upstream API is delayed rather than hard-failing the request.
Design a retry strategy with exponential backoff and jitter for calls to a downstream dependency that's struggling. Walk through why jitter matters, and how you'd make sure your retries don't make the dependency's problem worse when it starts recovering.
Sample Answer
Plain exponential backoff (double the delay after each failed attempt) reduces load on a struggling dependency over time, but it has a hidden flaw: if many clients failed at roughly the same moment (which is exactly what happens when the dependency itself goes down), they all compute the same delay sequence and retry in lockstep, so the "backoff" just delays the same synchronized spike instead of spreading it out. Jitter fixes that by randomizing the delay so clients that failed together don't retry together.
Jitter strategies compared
| Strategy | Delay formula | Behavior |
|---|---|---|
| No jitter | delay=base×2attempt | Deterministic; every client that failed together retries together, recreating the spike at each step |
| Full jitter | delay=random(0, base×2attempt) | Maximum spread; delay can be anywhere from 0 up to the cap, so retries are smeared thinly across the whole window |
| Equal jitter | delay=2cap+random(0, 2cap) | Keeps a guaranteed minimum delay (never retries immediately) while still spreading the upper half randomly |
| Decorrelated jitter | delay=random(base, previous delay×3) | Grows the delay based on the client's own previous delay rather than a fixed exponential schedule, avoiding a hard cap while still spreading load |
Worked example: how much jitter actually reduces the spike
Pin a concrete scenario: 1000 clients failed at the same moment, base delay = 1 second, and this is their 3rd retry attempt (attempt = 3), so the backoff cap is:
cap=1s×23=8 secondsWithout jitter: every one of the 1000 clients computes the identical 8-second delay and retries at exactly the same instant, a spike of 1000 concurrent requests hitting the dependency in one moment, right as it may just be starting to recover.
With full jitter, each client independently draws a delay uniformly from [0,8] seconds. Dividing that 8-second window into 100 ms buckets gives 8000/100=80 buckets, and under a uniform distribution the expected number of clients landing in any single bucket is:
801000=12.5 requests per 100ms bucketThat's a peak-to-average reduction factor of 1000/12.5=80× under this modeling assumption (uniform, independent draws), turning one instantaneous spike of 1000 into a smooth trickle of roughly 12-13 requests every 100 ms across the full 8-second window, which a recovering dependency can absorb where a single 1000-request spike would knock it back down.
Why retries shouldn't make recovery worse
Jitter alone doesn't prevent the retry storm from getting worse over time if attempts aren't capped: a client that keeps failing and keeps retrying at base×2attempt forever will eventually be sending requests at a cap so large it's functionally giving up, or, worse, if the cap is bounded, converges back to a steady drumbeat of load that never lets the dependency fully recover. The fix is a hard cap on both the maximum delay and the maximum number of attempts, plus honoring any explicit signal the server provides (a Retry-After header or a 429/503 status) as authoritative over the client's own backoff schedule, since the server is in the best position to know its own recovery state.
Trade-offs and pitfalls
Full jitter maximizes spread but means some unlucky clients draw a near-zero delay and retry almost immediately, which is fine in aggregate (that's still only ~12-13 requests per 100ms bucket in the example above) but means full jitter alone doesn't guarantee a minimum backoff for any individual client; equal jitter trades some of that spread for a guaranteed floor, useful when even a small number of near-instant retries is unacceptable. A pitfall specific to mobile or otherwise unreliable-network clients: retries are only safe to jitter and reattempt if the underlying operation is idempotent (repeating it produces the same end result as doing it once, so a duplicate attempt is harmless), a non-idempotent submit (a payment, an order) retried after a client-side timeout can double-execute if the server had actually processed the first attempt and just failed to deliver the response, so the fix belongs on the server (idempotency keys deduping identical requests) not just in the client's backoff logic, jitter reduces load, it does not make an unsafe retry safe.
Unlock Full Question Bank
Get access to all 16 Fault Tolerance, High Availability, and Disaster Recovery interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.