Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer: A full backup copies everything; an incremental backup copies only what changed since the last backup of any kind; a differential backup copies everything changed since the last full backup; and a snapshot captures a point-in-time, storage-level image (often copy-on-write) rather than a separate file copy. For a large transactional database, the common answer is a weekly full plus daily incrementals plus continuous transaction-log shipping, because that combination gives a low recovery point (minutes of data loss) without paying full-backup storage and I/O cost every day.

Structured elaboration

Restore chain length is the right way to reason about recovery complexity without making an unverifiable wall-clock claim: full and snapshot restores involve one artifact; differential restores always involve exactly two (full + latest differential, regardless of how many days have passed); incremental restores involve as many artifacts as days since the last full, so a chain 6 days deep means 7 total pieces (1 full + 6 incrementals) must apply cleanly, and a single corrupted link breaks the whole chain.

Worked example: 5 TB transactional database, weekly full + daily incrementals

Pin the assumption explicitly since this has to be derived, not asserted: assume 2% of the database's data changes per day (an illustrative rate; the real number should come from measuring actual write volume, but 2%/day is a reasonable planning figure for a moderately active OLTP (online transaction processing: a system handling many small, frequent reads and writes, like order or payment records, as opposed to bulk analytics queries) system).

• Day 0: full backup = 5 TB.
• Each daily incremental $\approx 0.02 \times 5\text{ TB} = 100\text{ GB}$ (treating the changed-data fraction as roughly constant day to day, a simplification for this estimate).
• By day 6 (just before the next weekly full), total incremental storage accumulated is $6 \times 100\text{ GB} = 600\text{ GB}$.
• Total storage footprint for that week's backup set: $5\text{ TB} + 0.6\text{ TB} = 5.6\text{ TB}$, a 12% overhead over the full alone ($5.6 / 5 = 1.12$).

Compare to a differential-only strategy at the same 2%/day rate: day 6's differential (changes since the day-0 full) would also be roughly $600\text{ GB}$ if changes were non-overlapping, but in practice differentials tend to be larger than the sum of same-period incrementals, because a row updated on day 2 and again on day 5 shows up in every day's differential from day 2 onward but only once across the incrementals. So the "incremental is smaller in total storage, differential is smaller to restore" trade-off holds here as expected.

Recovery point: with only daily backups, the recovery point objective (RPO) is bounded by the backup interval: worst case, you lose up to 24 hours of data (a failure right before the next scheduled backup). Adding continuous transaction-log shipping on top of the daily incrementals tightens the RPO to roughly the log-shipping interval (commonly seconds to a few minutes), independent of the backup schedule, which is why "backups alone" and "backups plus log shipping" are different RPO conversations for a transactional system.

Trade-offs & pitfalls

• Incrementals minimize storage and per-run I/O but maximize restore complexity (more pieces that must all be intact and applied in order); a single corrupted incremental in the chain can break every restore point after it.
• Differentials trade some storage growth (they get bigger every day until the next full) for a simpler, faster-to-verify two-piece restore.
• Snapshots are excellent for fast recovery when the underlying storage supports them cheaply, but for a live transactional database they require application-consistent quiescing (flushing buffers, pausing writes, or using the database's own snapshot-consistency mechanism) or the snapshot can capture a torn, inconsistent state.
• A common wrong turn: treating "we take backups" as equivalent to "we can restore." The only real validation is a periodic restore drill that rebuilds the database from the backup chain end to end; storage-level backup success says nothing about whether the restore path actually works.

The four standard DR tiers form a spectrum from cheapest-and-slowest to most-expensive-and-fastest, and each one trades infrastructure spend for recovery speed (RTO, recovery time objective: how long restoring service takes) and data freshness (RPO, recovery point objective: how much data, measured in time, you could lose): backup-and-restore keeps only backups running, pilot light keeps a minimal always-on core, warm standby keeps a scaled-down full copy running, and active-active multi-site keeps a full copy running and serving live traffic.

Comparing the four tiers

The RTO/RPO ranges above are the typical shape of the trade-off, not a fixed number for any specific system: the exact figures depend on data volume, automation maturity, and how the replication is actually implemented within each tier.

Worked example: a budget-constrained startup

A mid-sized SaaS with a fixed infrastructure budget doesn't have to pick one tier for the whole system; the standard move is to mix tiers by criticality. Say the product has three logical components: authentication/billing (must never meaningfully go down, since it blocks every paying customer from doing anything), the core application (needs to come back reasonably fast but a short outage is tolerable), and internal admin tooling (only the ops team notices if it's down for a few hours).

A budget-conscious allocation: active-active for auth/billing (the one component where the cost premium is justified because its outage blocks revenue entirely, and it's usually small enough in infrastructure footprint that duplicating it fully is affordable), pilot light for the core application (keep the database replicated continuously so RPO stays low, but only spin up the app-server fleet in DR when actually needed, since that's the majority of the compute cost), and backup-and-restore for admin tooling (cheapest tier, acceptable because nobody customer-facing is blocked by it being down for hours). This gets the highest-blast-radius component the fastest recovery while keeping the overall DR bill proportional to what each component actually costs the business if it's down, instead of buying active-active everywhere by default.

Trade-offs and pitfalls

The most expensive mistake in this space isn't picking the "wrong" tier, it's picking a tier and never testing failover into it: a pilot-light setup that's never actually been promoted to full capacity under load is a theoretical RTO, not a real one, and the first real DR event is a bad time to discover the app layer doesn't actually scale up cleanly from zero. A related pitfall is under-provisioning a warm standby's capacity: "scaled down" often means it can absorb DR traffic at reduced performance, and teams sometimes forget to validate that the scaled-down size can actually handle 100% of production load once promoted, not just serve health checks. Finally, active-active's real cost isn't just the duplicate infrastructure line item, it's the ongoing engineering cost of keeping a multi-writer data model correct, which is easy to underestimate when comparing tiers purely on an RTO/RPO/dollar table.

CAP theorem says that during a network partition, a distributed system must choose between consistency (every read sees the latest acknowledged write) and availability (every request gets a response), because it can't guarantee both while the partitioned halves can't talk to each other. Partition tolerance itself isn't optional for any system that spans more than one node, so the real choice interview questions are testing is CP versus AP, and the right answer depends entirely on what happens if you get it wrong.

Financial ledger: choose CP

A ledger getting a stale or divergent balance is a correctness bug with real financial consequences (double-spend, incorrect balance shown, a transaction accepted twice), so consistency has to win. The concrete mechanism is a consensus protocol requiring a write quorum.

Deriving what "sacrifice availability" actually means, with a pinned example: take a 5-node replica set (a typical Raft cluster size) where a majority quorum is required to commit a write:

That cluster tolerates up to $N - \text{quorum} = 2$ node failures while still committing writes. Now suppose a network partition splits the 5 nodes into a 3-node side and a 2-node side. The 3-node side still has a majority (3 ≥ quorum of 3), so it keeps accepting writes and stays both consistent and available. The 2-node side does not have a majority (2 < 3), so by design it must refuse writes, becoming unavailable, specifically to prevent both sides from independently committing conflicting transactions. That refusal on the minority side, not a global shutdown, is the literal, computable meaning of "CP sacrifices availability during a partition": only the minority partition goes unavailable, and only for writes.

Analytics event aggregator: choose AP

An analytics pipeline getting an event a few seconds late, or briefly double-counted before deduplication catches up, is a rounding error on a dashboard, not a financial loss, so availability wins: every node keeps ingesting even when it can't see the others.

The mechanism looks different from the ledger: instead of a write quorum gatekeeping every write, use leaderless or partitioned ingestion (each partition or node accepts writes for its own shard independently, as in Kafka-style partitioned logs or a Dynamo-style leaderless store), so there's no majority to lose and no write path that can be blocked by a partition. The cost shifts from "unavailable during a partition" to "eventually reconciled after one": deduplication on ingest, idempotent consumers, and background reconciliation jobs to merge whatever diverged while the partition was open.

Comparing the two designs

Trade-offs and pitfalls

The common misreading of CAP is treating CP as "the whole system goes down during any partition," when the quorum math above shows it's specifically the minority side, and only for writes, reads can often still be served (possibly stale, depending on the read-consistency level chosen). The common mistake on the AP side is stopping at "it's eventually consistent" without actually building the reconciliation path: leaderless ingestion without idempotent consumers and deduplication just relocates the correctness problem downstream instead of solving it. And even a CP ledger still needs idempotency keys on the client side: a client that times out waiting for a quorum ack and retries the same transaction must not have it applied twice, which is a consistency concern CAP itself doesn't cover but that any real ledger design has to handle regardless of which side of CAP it lands on.

Plain exponential backoff (double the delay after each failed attempt) reduces load on a struggling dependency over time, but it has a hidden flaw: if many clients failed at roughly the same moment (which is exactly what happens when the dependency itself goes down), they all compute the same delay sequence and retry in lockstep, so the "backoff" just delays the same synchronized spike instead of spreading it out. Jitter fixes that by randomizing the delay so clients that failed together don't retry together.

Jitter strategies compared

Worked example: how much jitter actually reduces the spike

Pin a concrete scenario: 1000 clients failed at the same moment, base delay = 1 second, and this is their 3rd retry attempt (attempt = 3), so the backoff cap is:

Without jitter: every one of the 1000 clients computes the identical 8-second delay and retries at exactly the same instant, a spike of 1000 concurrent requests hitting the dependency in one moment, right as it may just be starting to recover.

With full jitter, each client independently draws a delay uniformly from $[0, 8]$ seconds. Dividing that 8-second window into 100 ms buckets gives $8000 / 100 = 80$ buckets, and under a uniform distribution the expected number of clients landing in any single bucket is:

That's a peak-to-average reduction factor of $1000 / 12.5 = 80\times$ under this modeling assumption (uniform, independent draws), turning one instantaneous spike of 1000 into a smooth trickle of roughly 12-13 requests every 100 ms across the full 8-second window, which a recovering dependency can absorb where a single 1000-request spike would knock it back down.

Why retries shouldn't make recovery worse

Jitter alone doesn't prevent the retry storm from getting worse over time if attempts aren't capped: a client that keeps failing and keeps retrying at $\text{base} \times 2^{\text{attempt}}$ forever will eventually be sending requests at a cap so large it's functionally giving up, or, worse, if the cap is bounded, converges back to a steady drumbeat of load that never lets the dependency fully recover. The fix is a hard cap on both the maximum delay and the maximum number of attempts, plus honoring any explicit signal the server provides (a Retry-After header or a 429/503 status) as authoritative over the client's own backoff schedule, since the server is in the best position to know its own recovery state.

Trade-offs and pitfalls

Full jitter maximizes spread but means some unlucky clients draw a near-zero delay and retry almost immediately, which is fine in aggregate (that's still only ~12-13 requests per 100ms bucket in the example above) but means full jitter alone doesn't guarantee a minimum backoff for any individual client; equal jitter trades some of that spread for a guaranteed floor, useful when even a small number of near-instant retries is unacceptable. A pitfall specific to mobile or otherwise unreliable-network clients: retries are only safe to jitter and reattempt if the underlying operation is idempotent (repeating it produces the same end result as doing it once, so a duplicate attempt is harmless), a non-idempotent submit (a payment, an order) retried after a client-side timeout can double-execute if the server had actually processed the first attempt and just failed to deliver the response, so the fix belongs on the server (idempotency keys deduping identical requests) not just in the client's backoff logic, jitter reduces load, it does not make an unsafe retry safe.

Direct answer

Without timeouts, a slow dependency doesn't just make one caller slow, it holds resources (threads, connections, memory) on every service upstream of it for as long as it stays slow, and a request that would eventually fail anyway keeps consuming capacity that could have served a different request. Timeouts bound how long any hop is willing to wait, which is what turns "one dependency is unhealthy" into a locally-contained problem instead of a chain reaction. The design decision that actually matters is not picking one timeout number, it's propagating a single deadline down the call chain so every hop knows how much time is left in the overall budget, rather than each hop independently guessing its own timeout and stacking guesses on top of each other.

Why missing or misconfigured timeouts fail badly

If A calls B calls C calls D and none of them have timeouts, a slow response from D blocks C's thread, which blocks B's thread, which blocks A's thread, and the failure propagates upward even though only D was actually unhealthy: this is the classic thread-pool-exhaustion cascade. If timeouts exist but are misconfigured, the most common failure is each hop independently setting its own generous timeout (for example every service defaults to a flat 5 seconds) without accounting for the hops beneath it: A waits up to 5s for B, but B is itself waiting up to 5s for C, which is waiting up to 5s for D, so the true worst case for A's caller is up to 15 to 20 seconds, far past what A's own SLA promised. Deadlines have to be propagated, not independently re-derived at each hop.

Allocating a timeout budget across A → B → C → D

Start from the client-facing SLA and work down, reserving both a client-side buffer and each hop's own local processing time before deciding how much budget is left to hand to the next hop:

Each service reserves a small slice for its own non-downstream work (validation, serialization) before computing the deadline it hands to the next hop:

Sum check, confirming the allocation exactly accounts for the full budget with nothing double-counted or lost:

sequenceDiagram
    participant Client
    participant A as Service A
    participant B as Service B
    participant C as Service C
    participant D as Service D
    Client->>A: request, deadline now+1200ms
    A->>A: local processing 50ms
    A->>B: call, deadline now+1050ms
    B->>B: local processing 50ms
    B->>C: call, deadline now+1000ms
    C->>C: local processing 50ms
    C->>D: call, deadline now+950ms
    D->>D: work budget 900ms, margin 50ms
    D-->>C: response
    C-->>B: response
    B-->>A: response
    A-->>Client: response within 1200ms budget

The critical implementation detail is that this should be an absolute deadline (a wall-clock timestamp, "now + 1050ms" computed once by A) propagated unchanged through the chain, not a relative timeout re-applied at each hop. If each hop instead independently applied its own full timeout regardless of how much time upstream has already spent, a slow B could still consume its entire local timeout even after A's overall budget was nearly exhausted, defeating the whole point of the allocation.

Trade-offs & pitfalls

The margins reserved at each hop (50ms here) are a judgment call: too tight and normal jitter in local processing causes spurious timeouts even when nothing is actually failing; too generous and you're wasting budget that could have gone to the hop most likely to need it, typically the leaf service doing real work like a database query. A frequent mistake is retrying at every hop independently: if C retries a failed call to D once, and B also retries its call to C once, and A also retries its call to B once, a single slow D can trigger up to $2^3 = 8$ actual calls to D in the worst case, which both burns through the timeout budget faster and amplifies load on the exact dependency that's already struggling; retries should generally happen at one layer of the chain, not every layer. Deadline propagation gets meaningfully harder past four hops: with ten or more hops, per-hop margins compound into a large fixed tax on the total budget, and the more useful technique becomes hedging (firing a second, redundant request to a replica after some fraction of the expected latency has elapsed and taking whichever response comes back first) rather than allocating an ever-thinner slice of a fixed budget to each additional hop. Finally, propagated deadlines assume clocks are close enough to trust; in practice this means treating the deadline as relative-to-receipt at each hop (subtracting elapsed time since the request arrived) rather than trusting an absolute timestamp computed on a different machine's clock without accounting for skew.