Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer

A disaster recovery runbook is only useful if a stressed engineer can execute it top to bottom without needing to look anything else up. That means naming an owner, stating the RTO and RPO it is designed to meet, listing ordered and specific actions (exact commands or console steps, not descriptions), and ending with concrete verification steps that prove the system is actually back, not just that the steps were followed.

Structured elaboration

Versioning and access. Store the runbook in source control with required review on changes, so every edit has an author and a diff. Test it in a real drill, not a tabletop discussion only, on a cadence tied to how critical the service is, quarterly for anything customer-facing. Keep it reachable when the primary systems it recovers are down: a runbook that lives only on an internal wiki hosted in the region that just failed is not a disaster recovery runbook.

Worked example

For a service with RTO = 30 minutes, a well-built runbook's step timings should sum to that budget, and the sum should be checked, not assumed:

That sum matching the stated RTO exactly is what makes the RTO a testable claim rather than a number pasted at the top of the document. If a quarterly drill shows execution consistently takes 20 minutes instead of 15, the runbook's RTO is wrong and needs to be corrected, not explained away.

Trade-offs & pitfalls

• A runbook with no owner drifts out of date the first time the architecture changes; ownership is not optional metadata.
• Testing via tabletop discussion only, never an actual drill, hides the gap between the steps sounding right and the steps working; most drift is caught only by execution.
• Over-specifying every command for a fast-moving system creates a maintenance burden that causes the runbook to be abandoned; balance specificity against how often the underlying commands change.
• Storing the only copy behind the same authentication system that depends on the region that just failed is a common, self-defeating mistake.

Direct answer

Availability is whether the service is up and responding right now, the percentage of time requests get a correct response. Reliability is whether the service does the correct thing every time over a longer horizon, even if that means taking longer or failing loudly rather than silently. A service can be highly available (always responds) while being unreliable (frequently returns wrong or incomplete results), and vice versa.

How they're measured differently

• Availability: uptime percentage, request success rate (successful responses over total requests), and latency, all measured in real time against a rolling window.
• Reliability: job or transaction success rate over time, data-loss incidents, mean time between failures, and correctness checks like reconciliation counts, none of which are visible from a single point-in-time health check.

Worked example: an HTTP API versus a background worker

An HTTP API's job is to respond fast and stay up, so availability is the priority metric. Suppose the API calls three dependencies in sequence to serve a request: an auth service at 99.95% availability, a database at 99.9%, and a cache at 99.99%. Because a single request needs all three to succeed, the composed availability is the product of the three:

That's under three nines even though every individual dependency is at or above three nines, because failures compound across a serial chain. In annual downtime terms:

compared to a single 99.9% dependency on its own:

Chaining three otherwise-strong dependencies serially costs over 300 extra minutes of downtime a year versus just one of them alone. This is why an API-focused architect pushes hard on redundancy at each hop. To see how strong that lever is even when the underlying component is weaker, consider a hypothetical, cheaper cache tier, deliberately worse than the 99.99%-rated cache used above, where each individual replica only hits 99% availability on its own: two independent, parallel replicas of that weaker cache layer already beat any single component in the chain, the strong 99.99% cache included:

A background worker processing a queue of jobs, by contrast, doesn't need to respond within milliseconds; what matters is that every job eventually completes correctly, with no silent data loss, which is a reliability property, not an availability one. If the worker is down for ten minutes and then resumes and correctly processes every job that queued up during that window, availability took a hit but reliability didn't; if the worker stays "up" the whole time but drops or duplicates 0.01% of jobs due to a bug, availability looks perfect while reliability has quietly failed.

Trade-offs & pitfalls

Optimizing for availability alone can mask reliability problems: a service that always responds quickly, even by returning stale or wrong data rather than waiting for a correct answer, looks perfect on an uptime dashboard while silently corrupting downstream state. The practical approach is deciding, per component, which property is actually load-bearing: user-facing APIs generally prioritize availability with graceful degradation for correctness-adjacent risk, while systems of record and background processing prioritize reliability, often accepting higher latency or even temporary unavailability rather than risk an incorrect or lost write.

Direct answer

Automate failover when the detector is high-precision, the failover action is reversible and idempotent, and the cost of a wrong automatic trigger is bounded and recoverable. Require a human when any of those breaks down, especially when a wrong trigger risks unrecoverable data divergence or an irreversible action. Expected-value math on detection accuracy alone favors automation more than intuition suggests, but it is reversibility and blast radius, not raw precision, that should gate the decision.

Structured elaboration

A hybrid middle ground. Mature systems rarely pick one point on the automate-versus-manual spectrum. They tier it: automated detection and containment (circuit breakers, traffic throttling) run automatically because those actions are cheap to reverse, while the highest-blast-radius action (full regional failover, promoting a new primary) goes through an automated-detect, human-approve gate with an escalation timeout if nobody responds.

flowchart TD
    A[Alert fires] --> B{Multi-signal, high-precision detector?}
    B -->|No| M[Manual: page human, human confirms before failover]
    B -->|Yes| C{Action reversible and idempotent?}
    C -->|No| H[Hybrid: auto-detect and auto-contain, human approves full failover]
    C -->|Yes| D{Wrong trigger risks split-brain or data loss?}
    D -->|High risk| H
    D -->|Low risk| E[Automate: auto-detect and auto-failover with fencing token and audit log]

A fencing token here is a number that increases with every failover action; if a stale, already-superseded actor (an old primary that thinks it's still in charge, for example) tries to act after a newer one has taken over, its writes carry an outdated token and get rejected, so a late-arriving action from a process that no longer should be acting can't silently corrupt state.

Framing it as expected value. For a given alert, the expected value of automatic failover is:

where $p$ is the detector's precision, the probability an alert reflects a real failure.

Worked example

Assume correct auto-failover cuts RTO from a 15-minute human-paged response to a 2-minute automatic one, a 13-minute improvement, against a downtime cost of $50k/hour:

A false trigger causes roughly 3 minutes of avoidable disruption (connection draining and reconnect storms) at the same rate:

At a detector precision of $p = 0.9$:

Solve for the breakeven precision where $EV = 0$:

Pure expected value favors automation down to a detector that is right only 19% of the time, far noisier than any detector actually deployed. That is the point: raw EV almost always says automate. The equation treats every false trigger as a bounded $2,500 cost, which is only true if the action is reversible. If a wrong trigger can cause split-brain or an irreversible data promotion, the real cost of that tail case is not in the equation at all, which is why reversibility, not precision, is the dominant factor in practice.

Trade-offs & pitfalls

• The most common wrong turn is optimizing for detector precision and stopping there; a 99%-precision detector triggering an irreversible action is still a bad automation candidate if the 1% case is catastrophic.
• Automating containment (throttle, circuit-break) before automating the full failover captures most of the RTO benefit with much lower blast radius; teams often skip straight to automating the whole failover and take on risk they did not need.
• An approval gate with no timeout just becomes a slower manual failover with extra steps; if a human stays in the loop, define an explicit escalation timeout.
• Chaos-testing the automated path before trusting it in production is not optional. An automation that has never been exercised against a real failure is a new, untested failure mode, not a safety net.

RTO (Recovery Time Objective) is how long you're allowed to be down: the maximum acceptable gap between an outage starting and service being restored. RPO (Recovery Point Objective) is how much data you're allowed to lose: the maximum acceptable gap, measured in time, between the last durably captured write and the moment of failure. The requirement "payments must be restored within 30 minutes with no more than 5 minutes of data loss" is literally RTO = 30 min and RPO = 5 min stated in plain language, and each number drives a different part of the design.

What each number drives

RPO = 5 minutes drives replication and backup frequency. A nightly or even hourly backup can't meet this: if the outage happens 4 hours after the last backup, you'd lose 4 hours of transactions, not 5 minutes. A 5-minute RPO effectively requires continuous replication (near-synchronous in-region, or streaming WAL (write-ahead log: a durable, ordered record of every change, written before it's considered applied) or CDC (change-data-capture: a stream of those same row-level changes read off that log) shipping to the DR site) with replication lag actively monitored and alarmed well below the 5-minute budget, plus point-in-time recovery for protection against logical corruption that replication alone would just copy.

RTO = 30 minutes drives standby readiness and failover automation. A cold-standby DR site that has to be provisioned from scratch after the fact will blow past 30 minutes just on infrastructure boot time. A 30-minute RTO points toward a warm standby (already running, sized down, kept current via the same replication that satisfies the RPO) with an automated failover runbook: health-check detection, automated promotion, and DNS/routing cutover, because a manual, human-paged process realistically eats 10-15 minutes just in detection and decision-making before any recovery action starts.

Worked example: what these numbers cost against an annual SLA

A useful way to make the 30-minute number concrete is to check it against annual downtime budgets at standard availability tiers, using $525{,}600$ minutes per year (365 $\times$ 24 $\times$ 60):

A single incident with a 30-minute RTO, if the service is held to a 99.99% SLA, consumes:

of the entire year's downtime budget in one event. That reframes "30 minutes sounds generous" into "this design can absorb roughly one such incident a year and still hit four nines," which is exactly the kind of number that should drive whether the DR design gets warm-standby automation now or gets revisited after the first real incident eats most of the annual budget.

Trade-offs and pitfalls

The most common mix-up is treating RTO and RPO as interchangeable "how bad was it" numbers instead of two independent design constraints: a system can have a great RTO (back up in 2 minutes) and a terrible RPO (lost the last hour of writes) if it fails over to a backup instead of a live replica, or the reverse (RPO≈0 via synchronous replication, but a slow, manual promotion process blows the RTO). Both have to be solved, and usually by different mechanisms: RPO is a replication/backup-cadence problem, RTO is an automation/standby-readiness problem. A second pitfall specific to payments: RPO=0 sounds like the obviously "safer" number to chase, but strict synchronous replication that blocks writes during a replica outage can turn a replication hiccup into an availability incident, trading a data-loss risk you might never hit for a downtime risk you're now taking on every day.

Direct answer

A good game day has a tightly bounded scope, a named set of stakeholders who signed off before the experiment starts, a real-time comparison of the system's behavior against its SLIs (service level indicators: the specific numbers you track, like latency and error rate, that tell you whether the system is healthy) during the run, and a retrospective that turns findings into tracked action items, not just a summary email. The hard part is not running the experiment; it's building the recurring program and organizational trust that lets you run harder ones over time.

Scoping the experiment

Pick a single, realistic failure mode against a bounded slice of traffic: a specific dependency (cache, database replica, a downstream API), a specific service, and ideally a canary or staging slice of load rather than 100% of production on the first run. Define upfront what "done" looks like: which SLIs you'll watch, what the abort condition is, and who has authority to hit the kill switch.

Who to involve

• Service owners and on-call engineers for the system under test, since they know the failure modes and own the runbook being validated.
• A designated incident commander for the exercise itself, separate from whoever is executing the fault injection, so there's a clear decision-maker if things go sideways.
• Product or support stakeholders when the blast radius could touch real users, so they understand what "the recommendations service is intentionally broken for 20 minutes" means for anyone who notices.
• Observability or SRE tooling owners to make sure dashboards and alerting are actually wired up to catch what you're about to do, not just to catch organic incidents.

Measuring impact against SLIs

Capture a baseline of your SLIs (latency percentiles, error rate, saturation) before injecting the fault, then watch the same SLIs in real time during the run and compare against the SLOs (service level objectives: the target values you've committed to for those same indicators, e.g. 99.9% success rate). The goal is not "did it break" (you know it will) but "did it break within the bounds you predicted, and did the defenses (timeouts, circuit breakers, autoscaling) behave the way the runbook assumes they do."

Turning findings into a recurring program

A single successful game day proves one thing worked once. Standing up a recurring practice requires a roadmap: start with low-risk, staging-only experiments to build muscle memory and trust, then progressively widen scope (larger blast radius, real production traffic, less-scripted scenarios) as the team demonstrates it can run these safely. Getting buy-in usually means showing leadership a concrete finding from an early, low-risk drill (a specific gap the exercise surfaced) rather than asking for blanket permission to break production up front. Once a cadence is established (for example, monthly), track a maturity metric across runs, such as the fraction of prior findings that were actually remediated before the next drill, so the program itself is accountable.

Worked example

Consider a payments API game day: inject 200ms of added latency into its database replica for a scoped window, on a canary slice of traffic, with a monthly error budget of 43.2 minutes at a 99.9% SLO:

During the drill, the induced latency causes synchronous retries to queue up, and the service is measurably degraded (error rate above SLO) for 12 minutes before the circuit breaker trips and the fallback path kicks in. That single test consumed:

That is a legitimate, alarming finding on its own: a single scoped drill burning over a quarter of the monthly error budget means either the blast radius needs to be tightened further (smaller canary percentage) or the circuit breaker's failure threshold needs to trip faster. Either way it's a concrete, numeric input for the retrospective and the case for continued investment in the program, rather than a vague "went well."

Trade-offs & pitfalls

Widening scope too fast is the single biggest risk to the program's survival: one game day that causes a real customer-visible incident before the team has built confidence can kill the practice for a year. The opposite failure is scoping every drill so conservatively that it never surfaces anything new, which also erodes stakeholder buy-in because the exercise starts to look like theater. The retrospective is where most of the value is either captured or lost; findings that don't get a tracked owner and a re-test in the next cycle tend to silently repeat.