Fault Tolerance, High Availability, and Disaster Recovery Questions

The design combines two layers that solve different failure modes: multi-AZ redundancy within a primary region to survive the common case (a host, rack, or single-AZ failure) with near-zero disruption, and a warm standby in a second region, kept current via continuous replication, to survive the rare but severe case (a full regional outage). Trying to hit 99.99% and a 5-minute RPO (recovery point objective: the maximum data, measured in time since the last durable copy, you can afford to lose) with only one of those layers doesn't work: multi-AZ alone doesn't survive a regional disaster, and cross-region alone (without in-region redundancy) means every AZ blip forces an unnecessary cross-region failover.

Architecture

flowchart TB
  subgraph Primary["Primary region (us-east-1)"]
    LB1[Load Balancer]
    AZ1[App tier - AZ1]
    AZ2[App tier - AZ2]
    AZ3[App tier - AZ3]
    DBP[(Primary DB\nsynchronous multi-AZ)]
    LB1 --> AZ1 & AZ2 & AZ3
    AZ1 & AZ2 & AZ3 --> DBP
  end
  subgraph DR["DR region (eu-west-1)"]
    LB2[Load Balancer - standby]
    AZS[App tier - warm, scaled down]
    DBS[(Standby DB replica)]
    LB2 --> AZS --> DBS
  end
  DBP -- "async replication\n(CDC / log shipping)" --> DBS
  DNS[DNS / Global routing] --> LB1
  DNS -. failover .-> LB2

Redundancy strategy. Inside the primary region, the app tier runs across three AZs behind a load-balanced, health-checked pool (an AZ failure just removes that AZ's capacity from rotation), and the database runs synchronous or semi-synchronous multi-AZ replication so an AZ failure promotes a same-region replica with no data loss. This layer alone handles the overwhelming majority of real infrastructure failures.

Database replication approach. In-region: synchronous, to keep in-region failover at RPO≈0. Cross-region to the DR site: asynchronous (via change-data-capture or log shipping), because synchronous cross-region commits would add tens of milliseconds of write latency for every transaction just to protect against an event, a full regional outage, that's far rarer than an AZ blip. The async lag is what has to stay under the 5-minute RPO budget, so it's monitored continuously with an alert threshold well below 5 minutes (for example, paging at 60 seconds of lag) so operators have room to react before the budget is actually at risk.

Failover mechanics. In-region (AZ failure): automatic, health-check-driven, sub-minute, no human involved. Cross-region (regional failure): health checks on the primary region trigger an automated runbook that promotes the DR replica to primary, scales the warm (already-running, just smaller) app tier in the DR region up to full capacity, and updates DNS/global routing to point at the DR region's load balancer. The DR app tier is warm, not cold, specifically so this promotion is a scale-up operation, not a from-scratch provision, which is what keeps the cross-region RTO (recovery time objective: how long restoring service takes after a failure) in the minutes range instead of the tens-of-minutes range a cold standby would need.

Worked example: why the SLA is bounded by the DR path, not the multi-AZ path

Pin illustrative per-AZ availability at 99.9% for a single AZ's app-tier capacity. With three independent AZs, the app tier only fails when all three fail simultaneously:

That's a downtime budget of about $525{,}600 \times 10^{-9} \approx 0.0005$ minutes/year from independent AZ failures alone, far inside the 99.99% target's 52.56-minute/year budget. The conclusion this points to: independent-component math says multi-AZ redundancy alone should comfortably clear 99.99%, so in practice the SLA is bounded by things that math doesn't model, correlated failures, a bad deploy, a full regional outage, human error, not by an AZ dying in isolation. That's precisely why the DR region exists: it's insurance against the failure modes the availability formula can't see, not an incremental nudge on an already-tiny independent-failure number.

Trade-offs and pitfalls

The most common wrong turn is treating the independent-AZ math above as if it were the whole SLA story and skipping the cross-region layer as "redundant redundancy"; it's protecting against a different, uncorrelated failure class entirely. A second pitfall is under-provisioning the warm standby: if "warm" means a token instance that can't actually absorb full production load once promoted, the design has a theoretical RTO that doesn't survive contact with a real regional failover, so the standby's scaled capacity needs to be load-tested, not just health-checked. A concrete way this shows up: if the actual mandate is something like "cut our current 4-hour cold-standby RTO down to 15 minutes on a fixed budget," the honest trade-off conversation is that a full active-active design gets there fastest but costs the most and adds multi-writer complexity, while a well-automated warm standby (this design) gets to a low-single-digit-minutes RTO for a fraction of the cost, which is usually the better fit unless the traffic is latency-sensitive and truly global. Finally, none of this replaces DR drills: an automated runbook that's never been exercised against a real regional cutover is the same "theoretical RTO" problem as an untested pilot-light tier, just at higher stakes.

Direct answer

A good game day has a tightly bounded scope, a named set of stakeholders who signed off before the experiment starts, a real-time comparison of the system's behavior against its SLIs (service level indicators: the specific numbers you track, like latency and error rate, that tell you whether the system is healthy) during the run, and a retrospective that turns findings into tracked action items, not just a summary email. The hard part is not running the experiment; it's building the recurring program and organizational trust that lets you run harder ones over time.

Scoping the experiment

Pick a single, realistic failure mode against a bounded slice of traffic: a specific dependency (cache, database replica, a downstream API), a specific service, and ideally a canary or staging slice of load rather than 100% of production on the first run. Define upfront what "done" looks like: which SLIs you'll watch, what the abort condition is, and who has authority to hit the kill switch.

Who to involve

• Service owners and on-call engineers for the system under test, since they know the failure modes and own the runbook being validated.
• A designated incident commander for the exercise itself, separate from whoever is executing the fault injection, so there's a clear decision-maker if things go sideways.
• Product or support stakeholders when the blast radius could touch real users, so they understand what "the recommendations service is intentionally broken for 20 minutes" means for anyone who notices.
• Observability or SRE tooling owners to make sure dashboards and alerting are actually wired up to catch what you're about to do, not just to catch organic incidents.

Measuring impact against SLIs

Capture a baseline of your SLIs (latency percentiles, error rate, saturation) before injecting the fault, then watch the same SLIs in real time during the run and compare against the SLOs (service level objectives: the target values you've committed to for those same indicators, e.g. 99.9% success rate). The goal is not "did it break" (you know it will) but "did it break within the bounds you predicted, and did the defenses (timeouts, circuit breakers, autoscaling) behave the way the runbook assumes they do."

Turning findings into a recurring program

A single successful game day proves one thing worked once. Standing up a recurring practice requires a roadmap: start with low-risk, staging-only experiments to build muscle memory and trust, then progressively widen scope (larger blast radius, real production traffic, less-scripted scenarios) as the team demonstrates it can run these safely. Getting buy-in usually means showing leadership a concrete finding from an early, low-risk drill (a specific gap the exercise surfaced) rather than asking for blanket permission to break production up front. Once a cadence is established (for example, monthly), track a maturity metric across runs, such as the fraction of prior findings that were actually remediated before the next drill, so the program itself is accountable.

Worked example

Consider a payments API game day: inject 200ms of added latency into its database replica for a scoped window, on a canary slice of traffic, with a monthly error budget of 43.2 minutes at a 99.9% SLO:

During the drill, the induced latency causes synchronous retries to queue up, and the service is measurably degraded (error rate above SLO) for 12 minutes before the circuit breaker trips and the fallback path kicks in. That single test consumed:

That is a legitimate, alarming finding on its own: a single scoped drill burning over a quarter of the monthly error budget means either the blast radius needs to be tightened further (smaller canary percentage) or the circuit breaker's failure threshold needs to trip faster. Either way it's a concrete, numeric input for the retrospective and the case for continued investment in the program, rather than a vague "went well."

Trade-offs & pitfalls

Widening scope too fast is the single biggest risk to the program's survival: one game day that causes a real customer-visible incident before the team has built confidence can kill the practice for a year. The opposite failure is scoping every drill so conservatively that it never surfaces anything new, which also erodes stakeholder buy-in because the exercise starts to look like theater. The retrospective is where most of the value is either captured or lost; findings that don't get a tracked owner and a re-test in the next cycle tend to silently repeat.

Synchronous replication waits for the replica (or a quorum of replicas) to acknowledge a write before telling the client the write succeeded, so it costs extra write latency in exchange for near-zero data loss (a near-zero RPO, recovery point objective: how much data, measured in time, you could lose in a failure). Asynchronous replication acknowledges the write as soon as it's durable on the primary and ships it to replicas afterward, so writes stay fast but a failover can lose whatever hadn't shipped yet.

Comparing the two

Worked example: latency and RPO, with pinned assumptions

Pin a local write (fsync to disk) at 2 ms, a round-trip time to a same-region, cross-AZ replica at 4 ms, and a round-trip time to a cross-region replica at 70 ms (all stated as inputs for this comparison, not measurements of any specific vendor).

Synchronous, cross-AZ:

That's 3x the async latency of 2 ms. Acceptable for most OLTP systems.

Synchronous, cross-region:

That's 36x the async latency, which is why synchronous replication across regions is rare in practice for user-facing writes; the pattern that actually ships is synchronous within a region (to survive an AZ failure with RPO≈0) and asynchronous across regions (to survive a regional disaster, accepting a small RPO).

Quorum framing (this is where "synchronous" gets more precise than "one replica acks"): with $N=3$ replicas requiring a write quorum of $W=2$ (majority), a write only needs to wait for the fastest $W-1=1$ of the 2 non-primary replicas to ack, not all of them, which caps the latency cost at the RTT to whichever replica answers first rather than the slowest one. That's the practical reason quorum-based sync replication (Raft, Paxos-style commit) is preferred over "wait for every replica": it keeps the durability guarantee while bounding the latency tail.

Asynchronous RPO: if replication lag under normal load is 2 seconds but backs up to 30 seconds under a write burst, a failover during that burst loses up to 30 seconds of acknowledged-to-the-client-but-not-yet-replicated writes, i.e. $\text{RPO} \approx \text{replication lag at failure time}$, not a fixed number, which is exactly why teams monitor lag continuously rather than relying on the steady-state figure.

Trade-offs and pitfalls

The pitfall in the synchronous column isn't just latency, it's availability: a strict "wait for every replica" policy means a single slow or unreachable replica can stall every write on the primary, which is why real systems use quorum semantics (wait for a majority, not all) instead. The pitfall on the async side is treating "eventually consistent" as "eventually correct": if the primary accepts writes during a partition and then loses a leader election, those writes can simply vanish, so any system using async replication for anything beyond caches or analytics needs a defined reconciliation or conflict-resolution story, not just "replication will catch up." A common wrong turn is picking one mode globally instead of matching it to the data: a payments write path and an analytics event stream in the same system usually deserve different replication modes, not the same one applied uniformly for simplicity.

Direct answer: Redundancy is having extra, interchangeable components standing by so one can take over if another fails; replication is actively keeping copies of state (data) synchronized across multiple nodes so the state itself survives a failure, not just the compute that serves it. They're often used together, but they solve different problems: redundancy alone is enough for a stateless service, because any interchangeable instance can serve any request; a stateful service needs replication too, because a fresh redundant instance with no data isn't actually a working replacement.

Structured elaboration

Stateless example: a set of identical app server instances behind a load balancer, handling API requests with no local state. If one instance dies, the load balancer routes around it and an autoscaler replaces it; the new instance needs no data transfer because there was never any instance-local state to lose. This is pure redundancy: extra interchangeable copies of the same stateless computation.

Stateful example: a primary-replica database. The primary accepts writes; replicas continuously receive a copy of the write stream (replication). If the primary fails, a replica is promoted to take over. Unlike the stateless case, simply having an extra database instance running (redundancy alone, no replication) would give you an empty database, not a working replacement, because there's no mechanism copying the actual data into it.

A failure mode redundancy alone doesn't protect against, for the stateful case: data loss or corruption on the primary itself. If the primary's disk corrupts a row, or a bad write silently corrupts application-level data, having a redundant (but not yet caught-up, or synchronously replicating that same bad write) standby doesn't help, because either the standby doesn't have the data yet (async lag) or it faithfully replicated the corruption along with everything else (synchronous replication of a logically bad write). Redundancy protects against a node dying; it does not protect against the data itself being wrong, that requires backups (a separate, point-in-time copy decoupled from live replication) and, for silent corruption specifically, checksums or application-level validation.

How this generalizes: a useful mental checklist for fault tolerance covers five distinct techniques, and redundancy and replication are only two of them: retries (recover from a transient failure by trying again), bulkheads (isolate one failure from spreading to unrelated resources), failover (the mechanism that switches traffic to a healthy replacement), redundancy (having that replacement exist at all), and replication (making sure the replacement actually has the state it needs). A strong answer names which of these a given design decision is actually addressing, since "redundancy" gets used loosely to mean all five in casual conversation.

Trade-offs & pitfalls

• Replication has a cost redundancy alone doesn't: network bandwidth, storage for extra copies, and a consistency model to reason about (synchronous replication costs write latency; asynchronous replication risks data loss on failover, the classic RPO trade-off).
• A common wrong turn: assuming "we have 3 replicas" automatically means "we're protected," without checking replication lag. A replica that's minutes behind at failover time silently loses however much data arrived in that window, unless the promotion logic explicitly accounts for lag and refuses to promote a too-far-behind replica.
• Redundancy for stateless services is comparatively cheap and low-risk to over-provision; replication for stateful services is not, since more replicas means more write-path coordination overhead (for synchronous replication) or more divergence risk (for asynchronous/multi-leader), so it isn't a "just add more" lever in the same way.

Chaos engineering is the practice of deliberately injecting failure into a system, in a controlled way, to find weaknesses before they find you during a real incident. The reasoning behind doing it on purpose: most production failures aren't hypothetical, dependencies do time out, nodes do crash, networks do partition, and the choice isn't between "failures happen" and "failures don't happen," it's between discovering how your system responds to them during a planned, low-stakes experiment or during an unplanned, high-stakes 3 a.m. page.

Methodology

1. Define steady state. Pick measurable indicators of normal health, request success rate, latency percentiles, throughput, that represent "the system is working" in terms an on-call engineer would actually check on a dashboard, not an abstract notion of "healthy."

2. Form a hypothesis. State, before running anything, what you expect to happen and why: "if we kill one instance of the recommendation service, overall page error rate will stay flat because the client has a fallback path." A real hypothesis is falsifiable; "let's see what happens" isn't chaos engineering, it's just causing an outage without a way to learn from it.

3. Design a safe first experiment. Choose the smallest fault that could test the hypothesis (kill one non-critical replica, not the whole fleet) and decide the blast radius up front: what fraction of traffic or users can be affected, and for how long.

4. Run it with an abort condition already defined. Before starting, decide the exact metric threshold that ends the experiment immediately (for example, page error rate exceeding a set ceiling), so the decision to stop isn't made under pressure in the moment.

5. Observe against the steady-state baseline. Watch the same metrics defined in step 1, not new ones invented mid-experiment, so the comparison is apples-to-apples.

6. Learn and iterate. If the hypothesis held, expand the blast radius gradually on future runs. If it didn't, that's the actual finding, fix the missing fallback or retry logic, and re-run the same experiment to confirm the fix works before calling it done.

Worked example: a first, safe experiment

Target: a non-critical "related items" widget on a product page, deliberately chosen because a broken hypothesis here degrades a widget, not checkout. Steady state: page load success rate and p95 latency, whatever their current normal values are for that page. Hypothesis: "terminating one replica of the related-items service will not change page load success rate or p95 latency, because the front end treats that service as optional with a client-side timeout and empty-state fallback." Experiment: kill one replica (not all of them) during a low-traffic window, with an abort condition of "page success rate drops below its normal range" defined before starting. Outcome either confirms the fallback works as designed, or reveals it doesn't, which is the actual value of running it: finding that out on a Tuesday afternoon experiment instead of during a real node failure at peak traffic.

Trade-offs and pitfalls

The most common misunderstanding is that chaos engineering means "randomly break things in production," when the entire method is built around the opposite instinct: a stated hypothesis, a bounded blast radius, and a predefined abort condition are what separate a chaos experiment from just causing an outage. A related pitfall is skipping the hypothesis step and injecting a fault "to see what happens": without a stated expectation, there's no way to say afterward whether the result was surprising or how bad it was relative to what should have happened. Teams also sometimes skip straight to production chaos before validating the tooling and abort mechanism in staging first, running the injection and rollback machinery against a stage environment is itself a smaller, safer experiment worth doing before trusting it against real traffic.