Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer: I'd layer health signals from cheap-and-fast (process liveness) to expensive-and-meaningful (dependency and business-metric checks), require multiple consecutive failures before declaring a node unhealthy to avoid reacting to a single blip, and keep a human in the loop specifically for the step that's hardest to reverse: promoting a new primary for stateful services, where an incorrect automated failover can cause data loss or a split-brain (two nodes each believing they're the one true primary and accepting conflicting writes at the same time), versus something like removing an unhealthy instance from a load-balancer pool, which is cheap to reverse and safe to fully automate.

Structured elaboration

Why "N consecutive failures" instead of one: a single failed check can be a genuine blip (a GC pause, a brief network hiccup) rather than a real failure. If each check independently has some baseline flakiness probability $p$ (a transient failure unrelated to a real outage), then requiring $N$ consecutive failures before declaring unhealthy makes the false-trigger probability:

With, say, $p = 0.05$ (5% chance any single check fails transiently) and $N=1$ (react on the first failure), the false-trigger probability is just $p = 5\%$, meaning roughly 1 in 20 blips would incorrectly trigger action. Requiring $N=3$ consecutive failures drops that to:

a 400x reduction, at the cost of a real failure now taking 3 check intervals (here, up to 15 seconds at a 5s interval) longer to detect instead of one. That's the actual dial being turned: detection speed versus false-positive rate, and it should be set from an observed flakiness rate for your specific checks, not copied from another team's runbook.

Decision flow, including where automation stops and a human is required:

flowchart TD
    A[Health check runs] --> B{N consecutive<br/>failures?}
    B -->|No| A
    B -->|Yes| C{What kind of<br/>action?}
    C -->|Remove from LB pool| D[Fully automated:<br/>cheap, instantly reversible]
    C -->|Open circuit breaker| D
    C -->|Promote new primary<br/>for stateful service| E{Safety checks pass?<br/>replica caught up,<br/>quorum reachable}
    E -->|No| F[Page human,<br/>do not auto-promote]
    E -->|Yes, and blast radius<br/>is well-understood| G[Auto-promote,<br/>but page for review]
    E -->|Yes, but ambiguous<br/>e.g. partition, not clear failure| F

"Quorum reachable" in that safety check means enough replicas are online and able to vote that a new primary can be safely elected, a majority agreeing on who's in charge, without risking two nodes each believing they're the primary at once.

Trade-offs & pitfalls

• Fully automated failover is right when the action is cheap and reversible (removing an unhealthy node from rotation); it's risky when the action is expensive or irreversible (promoting a database replica, since promoting the wrong one, or promoting during a network partition rather than an actual failure, can cause split-brain or data loss). The line isn't "how critical is the service," it's "how reversible is this specific action."
• Requiring consecutive failures trades detection speed for false-positive protection; too aggressive a requirement (e.g., N=10) means a genuine failure runs uncaught far longer than the blast radius justifies.
• Checks that themselves depend on a shared resource (e.g., every health check queries the same central database) can produce correlated, simultaneous "failures" across an entire fleet when that shared resource degrades, which looks like a mass outage but is really a single point of failure in the monitoring path itself.
• A common wrong turn: only checking process liveness and assuming that's sufficient. A process can be alive, passing liveness checks, and still be completely unable to serve real traffic because its only database connection pool is exhausted; readiness and dependency checks catch what liveness checks structurally cannot.

Direct answer

Idempotency means performing the same operation multiple times has the exact same effect as performing it once. It matters for retries because network failures make it impossible for a client to reliably tell "the request failed" apart from "the request succeeded but the response was lost"; without idempotency, a client that retries after a timeout risks creating a duplicate side effect, like charging a customer twice for one order.

Making a payment-creation endpoint safe to retry

The standard mechanism is a client-generated idempotency key attached to the request:

1. The client generates a unique key (a UUID) once per logical operation, before the first attempt, and sends it on every retry of that same logical operation in a header such as Idempotency-Key.
2. The server does an atomic check-and-set against a persistent store keyed by that idempotency key: if the key is new, it proceeds with the charge; if the key already exists, it returns the previously stored result instead of processing the charge again.
3. The check-and-set has to be atomic (a single transactional operation, not a read followed by a separate write) so two near-simultaneous retries can't both see "key doesn't exist" and both proceed.
4. The stored result includes enough to reconstruct the original response (status, charge ID, amount) and a status field (in_progress, succeeded, failed) so a retry that arrives while the first attempt is still executing gets told to wait or gets the eventual result, rather than racing ahead.
5. Keys are kept with a bounded TTL (commonly 24 to 72 hours) since indefinite retention is unnecessary once a client has almost certainly given up retrying, and TTL bounds the storage cost of the idempotency table.

sequenceDiagram
  participant Client
  participant API as API Region A
  participant Store as Idempotency Store
  participant PG as Payment Gateway
  Client->>API: POST charges Idempotency-Key K1
  API->>Store: check-and-set K1 in_progress
  Store-->>API: new key proceed
  API->>PG: create charge
  PG-->>API: charge succeeded
  API->>Store: save result for K1
  API-->>Client: 200 OK response lost in transit
  Client->>API: retry POST charges Idempotency-Key K1
  API->>Store: check K1
  Store-->>API: found status succeeded
  API-->>Client: 200 OK cached result no new charge

Worked example

In the sequence above, the server actually completes the charge and writes the success result to the idempotency store, but the client's connection drops before the 200 response arrives, so from the client's point of view the request timed out. The client retries with the same key K1. The server's check-and-set finds K1 already marked succeeded, so it returns the stored response (the original charge ID and amount) directly and never calls the payment gateway again. Exactly one charge exists, regardless of how many times the client retries.

Harder extension: retries across a cross-region failover

The same duplicate-request risk gets worse if the retry lands on a different region than the original attempt. Say the first request goes to Region A, and before the response comes back, DNS or Anycast reroutes the client (as part of a regional failover) so the retry with the same idempotency key goes to Region B. If the idempotency store is region-local and not replicated, Region B has never heard of K1, sees it as a new key, and processes a second charge, exactly the failure the mechanism was supposed to prevent.

The fix is that the idempotency store itself has to be as available and as replicated as the failover design assumes the rest of the system is: either a globally consistent store (accepting the added write latency) or, more commonly for payments specifically, delegating idempotency to the payment gateway itself, which usually supports its own idempotency keys and is already a single global system of record regardless of which region initiated the call. Relying on the gateway's own dedupe as the backstop means even a fully region-local idempotency store failing open during a failover doesn't result in a real double charge.

Trade-offs & pitfalls

Idempotency keys add a write to the hot path (the check-and-set) and a storage system that has to be highly available, since if the idempotency store itself is down, you're forced to choose between blocking the write entirely or risking a duplicate. TTL choice is a real trade-off: too short and a legitimately slow client retry after the TTL expires creates a duplicate; too long and the storage grows unnecessarily and stale in-progress records from crashed requests linger. The most common mistake is only deduplicating the write itself while forgetting downstream side effects (an email receipt, a webhook fired to a third party) that happen inside the same logical operation and need to be gated by the same check, not fired unconditionally every time the handler runs.

Direct answer

Multi-AZ (spreading across availability zones within one region) gives faster failover and lower replication latency because the zones are close together on a fast, low-latency network, but it only protects against zone-level failures, not a regional outage. Multi-region gives protection against a much larger class of failures (an entire region going down) at the cost of higher replication latency, slower failover, and meaningfully more operational complexity.

Comparison

Worked example: failover time

For a multi-AZ setup, failover time is dominated by load-balancer health-check detection: a 10-second check interval with a 3-consecutive-failure threshold to avoid flapping, plus roughly 10 seconds of connection draining:

For a multi-region setup using DNS-based failover, detection can use a tighter 5-second health-check interval (since it's typically backed by dedicated external health checkers rather than the load balancer itself), still with a 3-consecutive-failure threshold, plus a DNS TTL of 30 seconds that a client's resolver may need to fully expire before it re-resolves to the healthy region:

These land in a similar ballpark for a well-tuned setup on paper, but the multi-region number is the optimistic case: it assumes every client's resolver honors the 30-second TTL exactly, which enterprise resolvers and some mobile carriers don't always do, so real-world multi-region failover tail latency for the slowest clients is meaningfully worse than the AZ case, even though the median case is comparable.

Trade-offs & pitfalls

Choose multi-AZ when the goal is protecting against common, smaller-scale infrastructure failures with minimal added latency and complexity; choose multi-region when the requirement is surviving a full regional outage or meeting geographic redundancy or compliance mandates, and accept that this means designing for a non-zero RPO and a longer failover tail. A frequent mistake is defaulting straight to multi-region for availability reasons alone without first exhausting what multi-AZ already buys, since multi-region adds real replication-latency and consistency costs that many systems don't actually need to pay to meet their real availability target.

Direct answer: Availability is the fraction of time a system is usable, and "N nines" is shorthand for how close that fraction is to 100%. Going from 99.9% to 99.99% to 99.999% shrinks allowed downtime by roughly 10x at each step, and each step also costs roughly an order of magnitude more in engineering and infrastructure, because you're eliminating an entire category of failure (single-host, then single-zone, then single-region) rather than just adding more of the same redundancy.

Structured elaboration

Allowed downtime per year is derived from the availability target directly:

where 8760 is the number of hours in a 365-day year (24 x 365), and per-month downtime uses 730 hours (8760 / 12):

Plugging in each target:

Each jump divides allowed downtime by exactly 10, because each availability target divides $(1-A)$ by 10.

What actually changes architecturally between tiers

• 99.9% -> 99.99%: eliminate single points of failure inside one facility. Multi-AZ deployment, N+1 redundancy (one extra standby unit beyond what's strictly needed to handle normal load, so a single failure doesn't drop capacity below what's required) on stateful components (load balancers, databases with a standby replica), automated health-check-driven failover, and a real on-call rotation with paging. Most of the gain here comes from removing manual recovery steps: a human restarting a service takes minutes and that alone can burn the entire four-nines monthly budget.
• 99.99% -> 99.999%: eliminate the facility (zone or region) itself as a single point of failure. Multi-region active-active or hot standby (a fully-running backup kept ready to take over instantly, unlike a cold standby that would first need to be started up and warmed), automated cross-region failover (not human-triggered), synchronous or tightly-bounded-lag replication for the data that must survive a region loss, and rigorous testing of the failover path itself (chaos drills: deliberately triggering the failover in a controlled test so a broken failover path is discovered on a Tuesday afternoon, not during a real outage), because at this tier the failover mechanism is now a bigger risk to availability than the failures it's protecting against.
• Beyond 99.999%, the limiting factor usually isn't infrastructure, it's deployment risk (bad releases) and dependency risk (a vendor or DNS provider you don't control), so the remaining budget goes to progressive rollouts, fast automated rollback, and reducing the number of hard external dependencies on the critical path.

Worked example: composing a dependency chain

A request that serially depends on a load balancer (99.99%), an app tier (99.95%), and a database (99.99%) has a combined availability equal to the product of the individual availabilities, because all three must be up simultaneously:

That's roughly 99.93%, worse than any single component, which is why a system built entirely from 99.99%-rated pieces chained together does not automatically deliver 99.99% end to end. Adding a redundant standby database (parallel, either one being up is sufficient) with independent 99.99% availability changes only that term:

so the pair is effectively always up, and the chain's availability is then bounded by the weakest remaining serial link (the app tier at 99.95%), not the database.

Trade-offs & pitfalls

• Availability composes multiplicatively across a serial chain and the weakest link dominates: chasing five nines on your database while your app tier sits at three nines is wasted spend.
• Each nine costs disproportionately more: 99.9% to 99.99% is mostly process and automation (cheap-ish); 99.99% to 99.999% usually means paying for a second region and the operational overhead of keeping it truly independent (expensive, and dangerous if the failover path itself is untested).
• Downtime budgets don't distinguish planned from unplanned; a team that spends its whole error budget on deploy-related outages hasn't actually built a more resilient system, just a riskier release process.
• A very common interview trap: treating "99.99% uptime" as a promise about any single request rather than a time-integrated average. A system can meet 99.99% for the year while having a full 50-minute outage in one bad afternoon.

Plain exponential backoff (double the delay after each failed attempt) reduces load on a struggling dependency over time, but it has a hidden flaw: if many clients failed at roughly the same moment (which is exactly what happens when the dependency itself goes down), they all compute the same delay sequence and retry in lockstep, so the "backoff" just delays the same synchronized spike instead of spreading it out. Jitter fixes that by randomizing the delay so clients that failed together don't retry together.

Jitter strategies compared

Worked example: how much jitter actually reduces the spike

Pin a concrete scenario: 1000 clients failed at the same moment, base delay = 1 second, and this is their 3rd retry attempt (attempt = 3), so the backoff cap is:

Without jitter: every one of the 1000 clients computes the identical 8-second delay and retries at exactly the same instant, a spike of 1000 concurrent requests hitting the dependency in one moment, right as it may just be starting to recover.

With full jitter, each client independently draws a delay uniformly from $[0, 8]$ seconds. Dividing that 8-second window into 100 ms buckets gives $8000 / 100 = 80$ buckets, and under a uniform distribution the expected number of clients landing in any single bucket is:

That's a peak-to-average reduction factor of $1000 / 12.5 = 80\times$ under this modeling assumption (uniform, independent draws), turning one instantaneous spike of 1000 into a smooth trickle of roughly 12-13 requests every 100 ms across the full 8-second window, which a recovering dependency can absorb where a single 1000-request spike would knock it back down.

Why retries shouldn't make recovery worse

Jitter alone doesn't prevent the retry storm from getting worse over time if attempts aren't capped: a client that keeps failing and keeps retrying at $\text{base} \times 2^{\text{attempt}}$ forever will eventually be sending requests at a cap so large it's functionally giving up, or, worse, if the cap is bounded, converges back to a steady drumbeat of load that never lets the dependency fully recover. The fix is a hard cap on both the maximum delay and the maximum number of attempts, plus honoring any explicit signal the server provides (a Retry-After header or a 429/503 status) as authoritative over the client's own backoff schedule, since the server is in the best position to know its own recovery state.

Trade-offs and pitfalls

Full jitter maximizes spread but means some unlucky clients draw a near-zero delay and retry almost immediately, which is fine in aggregate (that's still only ~12-13 requests per 100ms bucket in the example above) but means full jitter alone doesn't guarantee a minimum backoff for any individual client; equal jitter trades some of that spread for a guaranteed floor, useful when even a small number of near-instant retries is unacceptable. A pitfall specific to mobile or otherwise unreliable-network clients: retries are only safe to jitter and reattempt if the underlying operation is idempotent (repeating it produces the same end result as doing it once, so a duplicate attempt is harmless), a non-idempotent submit (a payment, an order) retried after a client-side timeout can double-execute if the server had actually processed the first attempt and just failed to deliver the response, so the fix belongs on the server (idempotency keys deduping identical requests) not just in the client's backoff logic, jitter reduces load, it does not make an unsafe retry safe.