Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer

Structure the program as a pyramid of increasing blast radius: start with single-instance, single-connection experiments in a canary slice of traffic, and only widen scope once the previous step stayed green. Every experiment, regardless of layer, needs the same three things stated up front: a falsifiable hypothesis (what should happen if the system is as resilient as you believe), a blast-radius control (the mechanism that caps how much traffic or infrastructure the experiment can touch), and a rollback criterion (the automated trigger that aborts the experiment before it becomes an incident).

Program structure by layer

The pyramid runs left to right within a layer, and layer to layer (service before database before network) because a service-level failure is the easiest to reason about and the cheapest to roll back; database and network faults touch more of the system at once and take longer to reverse cleanly.

Execution discipline

• Pre-flight: a written runbook, on-call and stakeholders notified, an automated abort mechanism wired to the rollback criteria (not a human watching a dashboard and deciding), and the experiment coded as a reproducible, version-controlled script rather than an ad hoc manual action.
• During: watch the rollback-criteria metrics in real time; the abort has to be automatic and fast, because by the time a human notices a metric crossing threshold and manually intervenes, the blast radius has often already grown past what the control was meant to cap.
• After: a lightweight postmortem regardless of outcome (a clean pass is still evidence worth recording), and only widen scope for the next run once the current one is unambiguously green, not "green with an asterisk."

Trade-offs & pitfalls

The single most common mistake is skipping straight to a production-wide experiment because a staging environment "doesn't reproduce the failure mode," which is often true but doesn't change the fact that the first production run of any new fault type belongs in the smallest blast radius you can construct, even if that means accepting a less realistic signal initially. A second pitfall is defining rollback criteria in terms of the fault itself (for example, "abort if packet loss exceeds 2 percent") instead of user-facing impact (error rate, latency, success rate); the fault is the input you're controlling, the rollback trigger has to watch the output, or you can hit exactly your intended fault level while still causing an unacceptable customer-facing outage. Not every fault type generalizes across domains the same way either: a GPU training job's most dangerous failure mode isn't a crashed worker (checkpointing handles that cheaply) but silent numerical divergence (the training job keeps running, but silently starts computing mathematically wrong updates to the model, with no crash or error to announce it), where the job keeps running and producing wrong gradients (a gradient is the per-step adjustment the training process makes to the model's internal numbers; a wrong one nudges the model in a bad direction instead of a good one) with no immediate error signal, so the safe blast-radius control there is different in kind from an HTTP service's traffic-percentage cap: it's about capping how long a divergence can run undetected before an automated metric check, watching a loss curve (a plot of the model's error over time, which should trend down) or a gradient norm (a single number summarizing how large the model's updates are; a sudden spike signals training has gone unstable), kills the job, not about capping how many requests are affected. A resilience program that only ever tests one fault type at a time also under-tests: real incidents are frequently two failures at once (a network blip during a deploy, a slow dependency during a traffic spike), so a mature program's later stages deliberately combine fault types once single-fault experiments across all three layers are consistently passing.

Direct answer

Blast radius is the scope of impact when a component fails: how many users, tenants, or dependent services are affected, and how severely, not just whether the failure happened at all. Reducing blast radius means designing so a single failure touches the smallest possible slice of the system, which makes outages smaller, easier to detect, and faster to recover from, even if it doesn't reduce how often failures happen at all.

Practices that reduce it, and what they cost

Worked example

Consider a service with 1,000 tenants sharing a single connection pool. If that pool exhausts, every tenant is affected. Now split that same total capacity into 10 isolated pools of 100 tenants each, so each pool serves 100 of the 1,000 tenants and only that pool's own tenants are affected if it exhausts:

Splitting the same total capacity into 10 pools of 100 tenants each means a single pool's exhaustion now affects only 100 of the 1,000 tenants, 10% of the blast radius of the shared-pool design, for the same total resources. The cost is operational: 10 pools to monitor and size instead of one, and if traffic isn't evenly distributed across tenants, some pools may be under-utilized while others are tight, which the shared pool didn't have to worry about.

Trade-offs & pitfalls

Reducing blast radius is generally a trade of operational complexity and some resource inefficiency for smaller, more contained failures; it doesn't reduce the underlying failure rate of any individual component. The common mistake is treating blast-radius reduction as free: partitioning by tenant, region, or dependency multiplies the number of things to monitor and can hide a systemic bug (one that affects every partition equally) behind what looks like ten separate, unrelated small incidents instead of one clearly systemic one.

Direct answer: Reads stay available almost everywhere, because they can be served from a nearby region's replica; writes to data classified as critical degrade to a queued, eventually-reconciled state during the outage rather than being lost or silently accepted twice; and in-flight writes are handled by never acknowledging a write as durable until it's replicated to enough independent regions to survive the loss of any one. The core design move is classifying every write path up front as either strongly consistent (accept the latency cost of cross-region coordination) or eventually consistent (accept a bounded staleness window), because trying to make everything strongly consistent across regions makes the system slow everywhere, and trying to make everything eventually consistent risks silent conflicts on data where that's unacceptable (e.g., a financial balance).

Structured elaboration

flowchart TD
    U[Client] --> GLB[Global anycast<br/>load balancer]
    GLB --> RA[Region A: App + local cache]
    GLB --> RB[Region B: App + local cache]
    GLB --> RC[Region C: App + local cache]
    RA --> DBA[Region A DB replica]
    RB --> DBB[Region B DB replica]
    RC --> DBC[Region C DB replica]
    DBA <--> DBB
    DBB <--> DBC
    DBA <--> DBC
    RA --> Q[Durable write queue,<br/>replicated across regions]
    RB --> Q
    RC --> Q
    Q --> REC[Reconciliation worker:<br/>applies queued writes<br/>in consistent order]

What stays available during a full region loss (say Region A goes down):

• Reads: clients in Region A get routed by the global load balancer to Region B or C, served from that region's replica. As long as replication lag was small before the outage, reads are only mildly stale, not unavailable.
• Non-critical writes (things like a "last viewed" timestamp or a UI preference): accepted locally in the surviving regions and replicated asynchronously; no coordination required, so no availability impact.

What degrades:

• Critical writes (anything requiring strong ordering or exactly-once semantics, like a financial transaction) that were routed to Region A: any request in flight to Region A at the moment of failure is lost from the client's perspective (it should be retried), but nothing already durably written is lost, because durability for critical writes is defined as "replicated to a quorum of regions," not "written to the local region." A write that only reached Region A and hadn't yet reached quorum was never acknowledged as successful to the client, so there's no false confirmation to reconcile.
• Any strongly-consistent read that specifically required Region A's replica (rare, and worth avoiding on the design's hot path) blocks or errors until failover routing completes.

Handling in-flight writes specifically: the design principle is that a write is only ever acknowledged as durable after being replicated to a quorum of regions, not after landing in one. For 3 regions, requiring a write quorum of 2 out of 3 means the write survives the loss of any single region (the surviving 2 regions still hold the write), and the client only receives "success" once that quorum is met. Writes that were in flight and had not yet reached quorum when Region A failed were never acknowledged; the client's request simply times out or errors, and the client-side retry (which must be idempotent, via an idempotency key) resubmits it to a surviving region. This means "handling in-flight writes" isn't a special recovery procedure, it's a direct consequence of never acknowledging a write before it's actually safe.

Alternate targets this design has to flex around: a variant that requires strong consistency wherever feasible pushes more of the write path onto the quorum-write pattern above even at the cost of latency (accepting that some write paths take the cross-region round trip on every request); a variant that targets sub-100ms p95 globally instead pushes as much as possible onto locally-served reads from nearby replicas and async-replicated writes, accepting a wider staleness/conflict window in exchange for speed, and reserving the expensive quorum-write path only for the specific subset of operations (like payment) where correctness cannot be compromised for latency.

Trade-offs & pitfalls

• Classifying every write path as strong or eventual up front is real design work; skipping it and defaulting everything to "eventually consistent for speed" is how systems end up with silent data conflicts on paths that actually needed strong guarantees (double-spending a discount code, double-fulfilling an order).
• Quorum writes across regions add real latency (a network round trip to another region, not just another host in the same datacenter), often tens of milliseconds at minimum depending on geography; this cost has to be paid specifically on the writes that need it, not applied uniformly.
• A common wrong turn: relying on DNS failover alone to redirect traffic away from a dead region. DNS TTLs and client/resolver caching mean propagation is unpredictable; anycast routing or a global load balancer with active health checks removes a dead region from rotation far faster and more reliably.
• Reconciliation workers that apply queued writes after a region recovers need a deterministic conflict-resolution rule (e.g., last-write-wins with a trustworthy clock, or a CRDT, conflict-free replicated data type, a data structure specifically designed so concurrent updates from different replicas always merge into the same result automatically with no manual reconciliation needed, or application-level merge) decided in advance; discovering the conflict-resolution policy during an actual incident is a common and costly mistake.

Direct answer: A full backup copies everything; an incremental backup copies only what changed since the last backup of any kind; a differential backup copies everything changed since the last full backup; and a snapshot captures a point-in-time, storage-level image (often copy-on-write) rather than a separate file copy. For a large transactional database, the common answer is a weekly full plus daily incrementals plus continuous transaction-log shipping, because that combination gives a low recovery point (minutes of data loss) without paying full-backup storage and I/O cost every day.

Structured elaboration

Restore chain length is the right way to reason about recovery complexity without making an unverifiable wall-clock claim: full and snapshot restores involve one artifact; differential restores always involve exactly two (full + latest differential, regardless of how many days have passed); incremental restores involve as many artifacts as days since the last full, so a chain 6 days deep means 7 total pieces (1 full + 6 incrementals) must apply cleanly, and a single corrupted link breaks the whole chain.

Worked example: 5 TB transactional database, weekly full + daily incrementals

Pin the assumption explicitly since this has to be derived, not asserted: assume 2% of the database's data changes per day (an illustrative rate; the real number should come from measuring actual write volume, but 2%/day is a reasonable planning figure for a moderately active OLTP (online transaction processing: a system handling many small, frequent reads and writes, like order or payment records, as opposed to bulk analytics queries) system).

• Day 0: full backup = 5 TB.
• Each daily incremental $\approx 0.02 \times 5\text{ TB} = 100\text{ GB}$ (treating the changed-data fraction as roughly constant day to day, a simplification for this estimate).
• By day 6 (just before the next weekly full), total incremental storage accumulated is $6 \times 100\text{ GB} = 600\text{ GB}$.
• Total storage footprint for that week's backup set: $5\text{ TB} + 0.6\text{ TB} = 5.6\text{ TB}$, a 12% overhead over the full alone ($5.6 / 5 = 1.12$).

Compare to a differential-only strategy at the same 2%/day rate: day 6's differential (changes since the day-0 full) would also be roughly $600\text{ GB}$ if changes were non-overlapping, but in practice differentials tend to be larger than the sum of same-period incrementals, because a row updated on day 2 and again on day 5 shows up in every day's differential from day 2 onward but only once across the incrementals. So the "incremental is smaller in total storage, differential is smaller to restore" trade-off holds here as expected.

Recovery point: with only daily backups, the recovery point objective (RPO) is bounded by the backup interval: worst case, you lose up to 24 hours of data (a failure right before the next scheduled backup). Adding continuous transaction-log shipping on top of the daily incrementals tightens the RPO to roughly the log-shipping interval (commonly seconds to a few minutes), independent of the backup schedule, which is why "backups alone" and "backups plus log shipping" are different RPO conversations for a transactional system.

Trade-offs & pitfalls

• Incrementals minimize storage and per-run I/O but maximize restore complexity (more pieces that must all be intact and applied in order); a single corrupted incremental in the chain can break every restore point after it.
• Differentials trade some storage growth (they get bigger every day until the next full) for a simpler, faster-to-verify two-piece restore.
• Snapshots are excellent for fast recovery when the underlying storage supports them cheaply, but for a live transactional database they require application-consistent quiescing (flushing buffers, pausing writes, or using the database's own snapshot-consistency mechanism) or the snapshot can capture a torn, inconsistent state.
• A common wrong turn: treating "we take backups" as equivalent to "we can restore." The only real validation is a periodic restore drill that rebuilds the database from the backup chain end to end; storage-level backup success says nothing about whether the restore path actually works.

Direct answer

Meeting a roughly 60-second reroute target for most clients means combining health-check-driven DNS failover (to redirect clients as they re-resolve) with Anycast routing at the network layer (to redirect clients immediately, independent of DNS caching behavior), because DNS alone can't guarantee a hard bound once client-side resolver caching is accounted for.

Architecture

flowchart TD
  Client -->|Anycast IP| Edge[Anycast Edge and CDN]
  Edge --> GSLB[GSLB Health-Aware DNS]
  GSLB -->|healthy| R1[Region 1]
  GSLB -->|healthy| R2[Region 2]
  GSLB -->|healthy| R3[Region 3]
  HC[Active Health Checkers] -->|probe every 5s| R1
  HC -->|probe every 5s| R2
  HC -->|probe every 5s| R3
  HC -->|update after 3 consecutive fails| GSLB
  R1 -.fails.-> HC

GSLB (Global Server Load Balancing, shown in the diagram) is DNS that returns different regional IPs depending on which regions are currently healthy, the mechanism the DNS/load-balancer layer below relies on.

Health checks: active probes (HTTP and TCP, from multiple external vantage points) hit each region every 5 seconds, requiring 3 consecutive failures before a region is marked unhealthy, to avoid flapping on a single transient blip.

DNS/load-balancer configuration: authoritative DNS TTL for the service record is set low, 30 seconds, so clients that honor TTLs re-resolve quickly after a region is marked unhealthy. Anycast IPs are advertised from all three regions simultaneously; when a region fails, its BGP (Border Gateway Protocol: the protocol that advertises which network paths lead to a given IP address) announcement is withdrawn, which reroutes traffic to a healthy region at the network layer almost immediately, without waiting on any client's DNS cache to expire at all.

Worked example: does this hit the 60-second target?

Detection time, the health checker's confirmation that a region is actually down:

For clients relying on DNS re-resolution, the worst case adds the full TTL window on top of detection, since a client could have just refreshed its cache right before the failure:

That leaves 15 seconds of margin against the 60-second target for clients that honor the TTL correctly, covering the large majority of traffic (public recursive resolvers like major DNS providers generally respect low TTLs closely). For the remaining slice of clients behind resolvers that cache more aggressively than the stated TTL (some enterprise resolvers and certain mobile carrier networks), the Anycast BGP withdrawal is what actually gets them under the target: it operates at the network layer and doesn't depend on DNS caching behavior at all, so those clients are rerouted within the same roughly 15-second detection window, not the 45-second DNS-bound one. Combining the two mechanisms is what makes hitting the target for "most clients" (rather than only the well-behaved subset) achievable.

Long-lived connections during cutover

Neither DNS re-resolution nor an Anycast BGP withdrawal preserves an existing TCP connection or WebSocket session that was already established to the failed region; a BGP route change mid-flow actually breaks those connections rather than gracefully migrating them, since the new route doesn't carry the old connection's state. Clients holding long-lived connections need their own reconnect logic (detect the drop, re-resolve or reconnect, resume from the new region) and any in-flight request that was interrupted needs to be safely retryable, which pushes the requirement for idempotent write handling on the server side, since a client that reconnects and retries an interrupted request must not have that retry double-process the original attempt.

Trade-offs & pitfalls

A lower DNS TTL improves worst-case failover time but increases query volume against the authoritative DNS servers and, for high-traffic services, real cost; 30 seconds is a reasonable middle ground rather than pushing to something extremely aggressive like 5 seconds. Anycast gives fast, DNS-independent failover but requires BGP-level control over IP announcements, which is a meaningfully bigger operational lift than DNS alone and isn't available on every cloud platform without specific networking products. The most common mistake in this kind of design is validating the 60-second target only against health-check and DNS timers on paper, without ever measuring how real clients across different resolver populations actually behave in practice, which is the only way to know the theoretical margin actually holds up.