Fault Tolerance, High Availability, and Disaster Recovery Questions

Approach

A circuit breaker is a state machine with three states and two triggered transitions plus one time-based transition:

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> OPEN: consecutive_failures >= threshold
    OPEN --> HALF_OPEN: cooldown elapsed
    HALF_OPEN --> CLOSED: probe succeeds
    HALF_OPEN --> OPEN: probe fails
    CLOSED --> CLOSED: call succeeds, reset counter

The implementation needs: a counter for consecutive failures, a timestamp for when the breaker opened, a lock so state transitions are atomic under concurrent callers, and a guard in the HALF_OPEN state that only lets exactly one trial call through at a time (otherwise every thread that arrives during HALF_OPEN would fire its own trial simultaneously, defeating the point of testing recovery with a single probe). Work happens outside the lock so a slow downstream call doesn't block every other thread from checking the breaker's state.

Code (Python)

import time
import threading
from enum import Enum


class CircuitOpenError(Exception):
    pass


class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=10.0):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds

        self._state = State.CLOSED
        self._consecutive_failures = 0
        self._opened_at = None
        self._half_open_probe_in_flight = False
        self._lock = threading.Lock()

    @property
    def state(self):
        with self._lock:
            self._maybe_recover()
            return self._state

    def _maybe_recover(self):
        # caller must hold self._lock
        if self._state == State.OPEN and self._opened_at is not None:
            if time.monotonic() - self._opened_at >= self.cooldown_seconds:
                self._state = State.HALF_OPEN
                self._half_open_probe_in_flight = False

    def call(self, fn, *args, **kwargs):
        with self._lock:
            self._maybe_recover()
            if self._state == State.OPEN:
                raise CircuitOpenError("circuit is open, call rejected")
            if self._state == State.HALF_OPEN:
                if self._half_open_probe_in_flight:
                    raise CircuitOpenError("half-open probe already in flight")
                self._half_open_probe_in_flight = True

        try:
            result = fn(*args, **kwargs)
        except Exception:
            with self._lock:
                if self._state == State.HALF_OPEN:
                    self._trip()
                else:
                    self._consecutive_failures += 1
                    if self._consecutive_failures >= self.failure_threshold:
                        self._trip()
            raise
        else:
            with self._lock:
                self._consecutive_failures = 0
                self._state = State.CLOSED
                self._half_open_probe_in_flight = False
            return result

    def _trip(self):
        # caller must hold self._lock
        self._state = State.OPEN
        self._opened_at = time.monotonic()
        self._half_open_probe_in_flight = False

Verified against pinned assertions (deterministic, no wall-clock claims beyond a fixed 0.05s cooldown used only to make the test itself run fast):

def run_test():
    calls = {"n": 0}

    def flaky(should_fail):
        calls["n"] += 1
        if should_fail:
            raise RuntimeError("boom")
        return "ok"

    cb = CircuitBreaker(failure_threshold=3, cooldown_seconds=0.05)

    # 3 consecutive failures trips OPEN
    for _ in range(3):
        try:
            cb.call(flaky, True)
        except RuntimeError:
            pass
    assert cb.state == State.OPEN

    # calls are rejected without invoking fn while OPEN
    calls_before = calls["n"]
    try:
        cb.call(flaky, False)
        assert False, "expected CircuitOpenError"
    except CircuitOpenError:
        pass
    assert calls["n"] == calls_before

    # after cooldown, HALF_OPEN allows exactly one trial; success closes
    time.sleep(0.06)
    assert cb.state == State.HALF_OPEN
    assert cb.call(flaky, False) == "ok"
    assert cb.state == State.CLOSED

    # a failing trial in HALF_OPEN re-opens immediately
    for _ in range(3):
        try:
            cb.call(flaky, True)
        except RuntimeError:
            pass
    time.sleep(0.06)
    assert cb.state == State.HALF_OPEN
    try:
        cb.call(flaky, True)
    except RuntimeError:
        pass
    assert cb.state == State.OPEN

    print("all assertions passed")


run_test()  # prints: all assertions passed

Key points

• State transitions and counter updates all happen inside self._lock; the wrapped function fn executes outside the lock so one slow call can't block every other caller from even checking the breaker's state.
• _half_open_probe_in_flight is what limits HALF_OPEN to a single trial call; without it, every thread that arrives during the cooldown window would fire its own simultaneous trial against a possibly-still-struggling dependency.
• A failure while HALF_OPEN re-opens immediately, it does not require re-accumulating failure_threshold failures again, because a single failed trial is already sufficient evidence the dependency hasn't recovered.
• time.monotonic() is used instead of time.time() for the cooldown comparison, since wall-clock time can jump (NTP adjustment, manual clock change) but monotonic time cannot go backwards.

Complexity

Every operation (call, state, _maybe_recover, _trip) is $O(1)$ time and the breaker holds $O(1)$ state (a state enum, a counter, a timestamp, one boolean), independent of call volume or history length.

Edge cases

• Concurrent trial races in HALF_OPEN: handled by _half_open_probe_in_flight; a second thread arriving while a trial is in flight gets rejected immediately rather than launching a second simultaneous probe.
• Long-running trial call: the single-probe guard limits it to one concurrent trial, but a probe that hangs for a long time (rather than failing fast) delays recovery detection; a production version would typically pair the breaker with a timeout on the wrapped call itself so a hung probe can't block recovery indefinitely.
• Distributed instances: this implementation is single-process, in-memory state; if the same logical breaker needs to be shared across multiple service instances (so instance A tripping also stops instance B from hammering the same dependency), the state has to move to a shared store (Redis, or a sidecar) with the same atomicity requirement, which changes the performance and failure-mode profile (the shared store becoming a dependency of its own).
• Bulkhead-limited half-open probing: some implementations allow a small fixed number of concurrent HALF_OPEN trials (say 2 to 3) rather than exactly 1, trading a slightly larger blast radius during recovery testing for faster convergence back to CLOSED when the dependency genuinely has recovered; that's a one-line change to compare _half_open_probe_in_flight against a max count instead of a boolean.
• Success/failure classification: this implementation treats every exception as a breaker-relevant failure; a production breaker usually needs to distinguish exceptions that indicate the dependency is unhealthy (timeouts, connection errors, 5xx) from exceptions that are just normal application-level outcomes (a 404, a validation error) that shouldn't count against the breaker at all.

Direct answer: Availability is the fraction of time a system is usable, and "N nines" is shorthand for how close that fraction is to 100%. Going from 99.9% to 99.99% to 99.999% shrinks allowed downtime by roughly 10x at each step, and each step also costs roughly an order of magnitude more in engineering and infrastructure, because you're eliminating an entire category of failure (single-host, then single-zone, then single-region) rather than just adding more of the same redundancy.

Structured elaboration

Allowed downtime per year is derived from the availability target directly:

where 8760 is the number of hours in a 365-day year (24 x 365), and per-month downtime uses 730 hours (8760 / 12):

Plugging in each target:

Each jump divides allowed downtime by exactly 10, because each availability target divides $(1-A)$ by 10.

What actually changes architecturally between tiers

• 99.9% -> 99.99%: eliminate single points of failure inside one facility. Multi-AZ deployment, N+1 redundancy (one extra standby unit beyond what's strictly needed to handle normal load, so a single failure doesn't drop capacity below what's required) on stateful components (load balancers, databases with a standby replica), automated health-check-driven failover, and a real on-call rotation with paging. Most of the gain here comes from removing manual recovery steps: a human restarting a service takes minutes and that alone can burn the entire four-nines monthly budget.
• 99.99% -> 99.999%: eliminate the facility (zone or region) itself as a single point of failure. Multi-region active-active or hot standby (a fully-running backup kept ready to take over instantly, unlike a cold standby that would first need to be started up and warmed), automated cross-region failover (not human-triggered), synchronous or tightly-bounded-lag replication for the data that must survive a region loss, and rigorous testing of the failover path itself (chaos drills: deliberately triggering the failover in a controlled test so a broken failover path is discovered on a Tuesday afternoon, not during a real outage), because at this tier the failover mechanism is now a bigger risk to availability than the failures it's protecting against.
• Beyond 99.999%, the limiting factor usually isn't infrastructure, it's deployment risk (bad releases) and dependency risk (a vendor or DNS provider you don't control), so the remaining budget goes to progressive rollouts, fast automated rollback, and reducing the number of hard external dependencies on the critical path.

Worked example: composing a dependency chain

A request that serially depends on a load balancer (99.99%), an app tier (99.95%), and a database (99.99%) has a combined availability equal to the product of the individual availabilities, because all three must be up simultaneously:

That's roughly 99.93%, worse than any single component, which is why a system built entirely from 99.99%-rated pieces chained together does not automatically deliver 99.99% end to end. Adding a redundant standby database (parallel, either one being up is sufficient) with independent 99.99% availability changes only that term:

so the pair is effectively always up, and the chain's availability is then bounded by the weakest remaining serial link (the app tier at 99.95%), not the database.

Trade-offs & pitfalls

• Availability composes multiplicatively across a serial chain and the weakest link dominates: chasing five nines on your database while your app tier sits at three nines is wasted spend.
• Each nine costs disproportionately more: 99.9% to 99.99% is mostly process and automation (cheap-ish); 99.99% to 99.999% usually means paying for a second region and the operational overhead of keeping it truly independent (expensive, and dangerous if the failover path itself is untested).
• Downtime budgets don't distinguish planned from unplanned; a team that spends its whole error budget on deploy-related outages hasn't actually built a more resilient system, just a riskier release process.
• A very common interview trap: treating "99.99% uptime" as a promise about any single request rather than a time-integrated average. A system can meet 99.99% for the year while having a full 50-minute outage in one bad afternoon.

RTO (Recovery Time Objective) is how long you're allowed to be down: the maximum acceptable gap between an outage starting and service being restored. RPO (Recovery Point Objective) is how much data you're allowed to lose: the maximum acceptable gap, measured in time, between the last durably captured write and the moment of failure. The requirement "payments must be restored within 30 minutes with no more than 5 minutes of data loss" is literally RTO = 30 min and RPO = 5 min stated in plain language, and each number drives a different part of the design.

What each number drives

RPO = 5 minutes drives replication and backup frequency. A nightly or even hourly backup can't meet this: if the outage happens 4 hours after the last backup, you'd lose 4 hours of transactions, not 5 minutes. A 5-minute RPO effectively requires continuous replication (near-synchronous in-region, or streaming WAL (write-ahead log: a durable, ordered record of every change, written before it's considered applied) or CDC (change-data-capture: a stream of those same row-level changes read off that log) shipping to the DR site) with replication lag actively monitored and alarmed well below the 5-minute budget, plus point-in-time recovery for protection against logical corruption that replication alone would just copy.

RTO = 30 minutes drives standby readiness and failover automation. A cold-standby DR site that has to be provisioned from scratch after the fact will blow past 30 minutes just on infrastructure boot time. A 30-minute RTO points toward a warm standby (already running, sized down, kept current via the same replication that satisfies the RPO) with an automated failover runbook: health-check detection, automated promotion, and DNS/routing cutover, because a manual, human-paged process realistically eats 10-15 minutes just in detection and decision-making before any recovery action starts.

Worked example: what these numbers cost against an annual SLA

A useful way to make the 30-minute number concrete is to check it against annual downtime budgets at standard availability tiers, using $525{,}600$ minutes per year (365 $\times$ 24 $\times$ 60):

A single incident with a 30-minute RTO, if the service is held to a 99.99% SLA, consumes:

of the entire year's downtime budget in one event. That reframes "30 minutes sounds generous" into "this design can absorb roughly one such incident a year and still hit four nines," which is exactly the kind of number that should drive whether the DR design gets warm-standby automation now or gets revisited after the first real incident eats most of the annual budget.

Trade-offs and pitfalls

The most common mix-up is treating RTO and RPO as interchangeable "how bad was it" numbers instead of two independent design constraints: a system can have a great RTO (back up in 2 minutes) and a terrible RPO (lost the last hour of writes) if it fails over to a backup instead of a live replica, or the reverse (RPO≈0 via synchronous replication, but a slow, manual promotion process blows the RTO). Both have to be solved, and usually by different mechanisms: RPO is a replication/backup-cadence problem, RTO is an automation/standby-readiness problem. A second pitfall specific to payments: RPO=0 sounds like the obviously "safer" number to chase, but strict synchronous replication that blocks writes during a replica outage can turn a replication hiccup into an availability incident, trading a data-loss risk you might never hit for a downtime risk you're now taking on every day.

Direct answer

Model each component's availability as a probability of being up, then combine them with two rules: components in series (every one must work for the system to work) multiply, and components in parallel (redundant copies of the same thing, any one of which is enough) combine as one minus the product of their failure probabilities. Once you have that model, "where does an extra dollar of redundancy buy the most availability" becomes a concrete comparison: compute the marginal gain in availability from each candidate investment, divide by its cost, and fund whichever option has the highest availability-per-dollar, repeating as budget allows.

The two combination rules

Series availability is always less than or equal to the least available component, because every additional required link is one more way for the chain to break. Parallel availability is always greater than or equal to the most available individual replica, because failure now requires every redundant copy to fail simultaneously, which independence makes rapidly unlikely as replica count grows.

Modeling redundant replicas across regions

For $n$ identical replicas within a region (parallel, independent failures) and $r$ regions (also parallel: the system is up if any one region is up):

where $a$ is a single replica's availability. The system-wide failure probability collapses to $(1-a)^{nr}$: the whole deployment is down only if every one of the $n \times r$ independent replicas is simultaneously down.

Worked example: where does the next dollar go

Start with a single instance at $a = 0.99$ (two nines, a not-unreasonable single-VM baseline) and no redundancy ($n=1, r=1$):

Converting to expected downtime per year (using 525,600 minutes/year, $365 \times 24 \times 60$):

Going from no redundancy to a second replica in the same region already removes over 5,200 minutes of downtime per year. Going from one region to two regions on top of that removes only another 52.5 minutes. That's the diminishing-returns pattern the model predicts: each additional nine costs roughly the same effort but buys an order of magnitude less absolute downtime reduction than the nine before it, because the failure probability being multiplied away is already small.

Deciding where the next dollar goes when both axes already have redundancy

For a system already at $n$ replicas and $r$ regions, the marginal availability gain from adding one more replica (holding regions fixed) versus adding one more region (holding replicas fixed):

At $n=3$, $r=2$, $a=0.99$: $(1-a)^{nr} = (0.01)^6 = 10^{-12}$. The bracket terms are $1-(0.01)^2 = 0.9999$ for adding a replica, versus $1-(0.01)^3 = 0.999999$ for adding a region, so $\Delta A_{\text{region}}$ is about 0.01 percent larger than $\Delta A_{\text{replica}}$ here. The pattern behind that: the bracket term is largest for whichever axis currently has fewer units ($r=2 < n=3$ in this example), so all else equal, invest in whichever axis is thinner. Then divide each $\Delta A$ by its real cost: $\text{ROI} = \Delta A / \text{cost}$, and since a new replica in an existing region is almost always cheaper than standing up an entire new region (new infrastructure, new operational surface, cross-region egress), the raw availability numbers alone usually favor adding replicas once you already have two or more regions, and the case for a third region has to come from something the pure independence model doesn't capture: correlated failure risk.

Trade-offs & pitfalls

The entire model rests on an independence assumption that gets weaker as you add replicas within the same region and stays strongest across regions: two replicas in the same region share a power grid, a network fabric, and often a control plane, so a regional outage takes both of them down together in a way the $(1-a)^n$ formula doesn't account for, since that formula assumes each replica fails independently of the others. This is precisely why real systems keep investing in additional regions even when the pure math above says another same-region replica is cheaper per nine: the region-diversity investment is buying protection against a correlated-failure mode the independence model is structurally blind to, not buying a bigger $\Delta A$ number on paper. A second pitfall is applying the series-multiplication rule to a system that has hidden shared dependencies across "independent" series stages, for example a DNS provider or a certificate authority that every stage secretly relies on; multiplying stage availabilities together understates the real risk whenever a single upstream failure can take out multiple "independent" stages simultaneously, which is exactly the class of failure that showed up in the largest real-world multi-service outages. The practical fix for both pitfalls is the same: use the independence-based formula as the starting estimate and a genuine planning tool, but treat any two components sharing physical infrastructure, a network path, or an operational dependency as a single correlated unit for the purposes of the model, not as two separate multiplicands.

Direct answer

This is the multi-master conflict case: because replication was asynchronous and both regions kept accepting writes, there are now two divergent histories that both look locally valid. Detecting divergence means comparing state between regions, via hashes or version metadata, once connectivity returns, not waiting for a customer to report bad data. Reconciling means classifying each conflict by whether it's safely auto-mergeable or needs a compensating action; financial and other invariant-sensitive writes should never be silently overwritten.

Structured elaboration

Detecting divergence. Attach causal metadata to every write, a vector clock (an array with one counter per region, incremented only when that region processes a write, so comparing two vectors shows whether one write causally happened after the other, every counter at least as high, or whether the two happened independently) or at minimum a per-region monotonic sequence number plus timestamp, so that on reconnect two versions of the same key can be compared to tell, mathematically, whether one supersedes the other or whether they are genuinely concurrent, meaning both sides wrote independently during the partition.

sequenceDiagram
    participant RegionA
    participant RegionB
    participant Reconciler

    Note over RegionA,RegionB: Partition: both regions accept writes independently
    RegionA->>RegionA: Write key X, vector clock [A:5,B:3]
    RegionB->>RegionB: Write key X, vector clock [A:4,B:4]
    Note over RegionA,RegionB: Partition heals
    Reconciler->>RegionA: Fetch change log with vector clocks
    Reconciler->>RegionB: Fetch change log with vector clocks
    Reconciler->>Reconciler: Compare vector clocks, classify conflicts
    Reconciler->>Reconciler: Auto-merge commutative writes
    Reconciler->>Reconciler: Apply compensating transaction for invariant conflicts
    Reconciler->>RegionA: Apply resolved state
    Reconciler->>RegionB: Apply resolved state

Reconciling. Not every conflict resolves the same way:

Worked example: detecting a real conflict with vector clocks

Two regions independently update the same key, "user:42.email," during the partition. Region A's write carries vector clock $V_A = [A{:}5, B{:}3]$, Region B's write carries $V_B = [A{:}4, B{:}4]$.

A vector clock $V_1$ dominates $V_2$, meaning $V_1$ happened after $V_2$ and can safely overwrite it, only if every component of $V_1$ is greater than or equal to the corresponding component of $V_2$, with at least one strictly greater. Checking:

Neither vector dominates the other, A leads on its own component, B leads on its own, so this is a genuine concurrent conflict, not a case where one write simply supersedes the other, and it must go through the merge policy rather than being resolved by picking the higher-looking clock.

Worked example: scoping the reconciliation workload

Assume the partition lasted 10 minutes, each region accepted 150 writes/sec across a 5-million-key space:

Of 180,000 total writes across both sides, an estimated 1,620 keys, 0.9%, are true conflicts needing the merge policy; the rest merge by simple union. Sizing the reconciliation pipeline, and any manual-review queue, around this number, not around all writes during the partition, is what keeps the process fast.

Trade-offs & pitfalls

• Silently applying last-writer-wins to every conflict is the most common shortcut, and it's the wrong default for anything with a business invariant, balances, inventory counts: it loses data without a trace and without anyone knowing which write "won."
• Scanning the entire keyspace for divergence, instead of the range touched during the outage window, wastes time and delays recovery.
• A merge policy that isn't tested against the conflict-type classification before the incident means the policy is being designed live, under pressure, which is when mistakes compound.
• Compensating transactions must themselves be idempotent and auditable; a reconciliation "fix" that isn't traceable back to the original conflicting writes creates a second, harder-to-debug data quality problem.