Fault Tolerance, High Availability, and Disaster Recovery Questions

The circuit breaker pattern stops calling a failing dependency once it's clearly unhealthy, so callers fail fast instead of piling up waiting on a dependency that isn't going to answer, and the dependency gets breathing room to recover instead of being hit with an ever-growing retry storm on top of whatever's already wrong with it.

The three states

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failure threshold crossed
  Open --> HalfOpen: cooldown elapses
  HalfOpen --> Closed: probe requests succeed
  HalfOpen --> Open: probe requests fail

Choosing the threshold and window for a real dependency

Base the threshold on the dependency's own historical baseline, not a round number picked by feel: if a dependency's normal error rate is 1-2%, a threshold like "error rate exceeds 50% over a 1-minute window" is a real signal of degradation, not noise. Combine multiple signals rather than trusting one: an error-rate threshold alone can be fooled by a burst of retriable timeouts, so pairing it with a consecutive-failure count and a latency percentile (for example, p99 exceeding a set ceiling) catches degradation that shows up as slowness before it shows up as outright errors.

Worked example: why the half-open probe count matters

Say the breaker opens, waits out its cooldown, and moves to half-open, sending 5 probe requests before deciding whether to close. If the dependency is still genuinely degraded, with a true underlying failure rate of $p = 0.3$ (30% of calls failing), the probability that all 5 probes happen to succeed by chance despite that is:

That's not a rare fluke, it's roughly a 1-in-6 chance of prematurely closing the breaker on a dependency that's still 30% broken, which then immediately re-floods it with full traffic and likely reopens the breaker on the very next window. This is the concrete argument for either using more probes (the same calculation with 10 probes drops the false-close probability to $0.7^{10} \approx 0.028$, about 2.8%) or ramping traffic gradually after a half-open success instead of jumping straight from 5 probes to 100% traffic.

Trade-offs and pitfalls

Setting the threshold too sensitive (a low error-rate bar or a short window) causes flapping: the breaker opens on transient noise, degrades the user experience with unnecessary fallbacks, and can itself become a source of alerts nobody trusts. Setting it too lax delays protection long enough for the caller's own retries and connection-pool exhaustion to cascade into a second incident on top of the first. The half-open probe-count math above is the same trade-off in miniature: too few probes risk a premature, false-positive close; too many probes delay recovery and keep failing extra requests during the test window. In practice this is tuned with production data and game-day testing rather than picked once and left alone, and the same three-state logic applies regardless of what's on the other side of the call, an AI inference endpoint that starts throwing GPU-OOM errors under load trips the same breaker, on the same threshold logic, as a slow downstream REST dependency; only the specific error signal being watched changes.

Direct answer

A cascading failure is when one component's failure increases load or latency on the components that depend on it, and that increased load causes those components to fail too, propagating outward until a large part of the system is affected, even though only one component actually broke in the first place. The mechanism is almost always resource exhaustion: threads, connections, or memory tied up waiting on the failed component instead of being freed quickly.

Walkthrough: C fails, B overloads, A degrades

flowchart LR
  A[API Gateway] -->|rate limit and timeout| B[Order Service]
  B -->|bulkhead pool: payments| C[Payment Service]
  C -.fails.-> B
  B -->|circuit breaker opens| D[Fallback: queue order for async retry]
  A -->|circuit breaker opens| E[Fallback: 503 with Retry-After]
  B -->|isolated pool: other deps unaffected| F[Inventory Service]

1. C (Payment Service) fails, hanging instead of returning errors quickly, perhaps due to a downstream outage of its own.
2. B (Order Service) calls C without a tight timeout. Each call to C now blocks for far longer than normal, tying up a thread or connection from B's pool for the duration.
3. B's resource pool exhausts. As more requests arrive at B, more threads get stuck waiting on C, until B has no capacity left to serve any request, including ones that don't even touch C.
4. A (API Gateway) calls B, and B is now slow or unresponsive for everything, so A's calls to B start timing out or queueing too, degrading A's own capacity in turn.

Worked example: how fast does B's pool actually exhaust?

Little's Law relates the number of requests in flight to the arrival rate and the time each spends being processed:

Say B receives 500 requests per second, and under normal conditions each call to C takes 50ms:

25 concurrent requests is a light load on a typical connection pool. Now C hangs, and B's HTTP client has no explicit timeout of its own, falling back to a default of 30 seconds:

If B's thread pool has 200 threads, the time to exhaust it entirely is:

Under 400 milliseconds. That's how quickly a single hung dependency with no timeout turns into total unavailability for a service handling 500 requests per second: the pool never gets close to steady-state at the 30-second hang time, it simply fills with stuck requests almost instantly and stays full.

Protections at each layer

• At B, calling C: a tight, explicit timeout (measured in low hundreds of milliseconds, not the client library's 30-second default) so a hung call fails fast and frees the thread quickly; a circuit breaker that opens after a run of failures or timeouts, so B stops even attempting calls to C once it's clearly down, and falls back to queueing the order for later processing; a bulkhead, a dedicated connection pool just for calls to C, so exhaustion from C-related calls doesn't consume the threads B needs to serve requests that don't touch C at all (like inventory checks).
• At A, calling B: the same pattern one layer up, a timeout on calls to B, a circuit breaker that trips once B's error rate or latency crosses a threshold, and a fallback (a fast 503 with Retry-After rather than a hung request) so A's own capacity isn't consumed waiting on a B that's already struggling.

Trade-offs & pitfalls

Timeouts that are too aggressive cause false-positive failures under normal, brief latency variance; timeouts that are too loose don't prevent the cascade fast enough, as the Little's Law example shows. Bulkheads cost real resources (a dedicated pool per dependency uses more total connections or threads than one shared pool) in exchange for isolation, so they're worth applying to the dependencies most likely to fail or most likely to take down unrelated traffic if they do. The most common mistake is only protecting the first hop (B to C) and assuming that's sufficient; as the walkthrough shows, without protection at the A to B hop too, the failure still reaches A once B is degraded, just one layer later.

The four standard DR tiers form a spectrum from cheapest-and-slowest to most-expensive-and-fastest, and each one trades infrastructure spend for recovery speed (RTO, recovery time objective: how long restoring service takes) and data freshness (RPO, recovery point objective: how much data, measured in time, you could lose): backup-and-restore keeps only backups running, pilot light keeps a minimal always-on core, warm standby keeps a scaled-down full copy running, and active-active multi-site keeps a full copy running and serving live traffic.

Comparing the four tiers

The RTO/RPO ranges above are the typical shape of the trade-off, not a fixed number for any specific system: the exact figures depend on data volume, automation maturity, and how the replication is actually implemented within each tier.

Worked example: a budget-constrained startup

A mid-sized SaaS with a fixed infrastructure budget doesn't have to pick one tier for the whole system; the standard move is to mix tiers by criticality. Say the product has three logical components: authentication/billing (must never meaningfully go down, since it blocks every paying customer from doing anything), the core application (needs to come back reasonably fast but a short outage is tolerable), and internal admin tooling (only the ops team notices if it's down for a few hours).

A budget-conscious allocation: active-active for auth/billing (the one component where the cost premium is justified because its outage blocks revenue entirely, and it's usually small enough in infrastructure footprint that duplicating it fully is affordable), pilot light for the core application (keep the database replicated continuously so RPO stays low, but only spin up the app-server fleet in DR when actually needed, since that's the majority of the compute cost), and backup-and-restore for admin tooling (cheapest tier, acceptable because nobody customer-facing is blocked by it being down for hours). This gets the highest-blast-radius component the fastest recovery while keeping the overall DR bill proportional to what each component actually costs the business if it's down, instead of buying active-active everywhere by default.

Trade-offs and pitfalls

The most expensive mistake in this space isn't picking the "wrong" tier, it's picking a tier and never testing failover into it: a pilot-light setup that's never actually been promoted to full capacity under load is a theoretical RTO, not a real one, and the first real DR event is a bad time to discover the app layer doesn't actually scale up cleanly from zero. A related pitfall is under-provisioning a warm standby's capacity: "scaled down" often means it can absorb DR traffic at reduced performance, and teams sometimes forget to validate that the scaled-down size can actually handle 100% of production load once promoted, not just serve health checks. Finally, active-active's real cost isn't just the duplicate infrastructure line item, it's the ongoing engineering cost of keeping a multi-writer data model correct, which is easy to underestimate when comparing tiers purely on an RTO/RPO/dollar table.

Direct answer

Availability is whether the service is up and responding right now, the percentage of time requests get a correct response. Reliability is whether the service does the correct thing every time over a longer horizon, even if that means taking longer or failing loudly rather than silently. A service can be highly available (always responds) while being unreliable (frequently returns wrong or incomplete results), and vice versa.

How they're measured differently

• Availability: uptime percentage, request success rate (successful responses over total requests), and latency, all measured in real time against a rolling window.
• Reliability: job or transaction success rate over time, data-loss incidents, mean time between failures, and correctness checks like reconciliation counts, none of which are visible from a single point-in-time health check.

Worked example: an HTTP API versus a background worker

An HTTP API's job is to respond fast and stay up, so availability is the priority metric. Suppose the API calls three dependencies in sequence to serve a request: an auth service at 99.95% availability, a database at 99.9%, and a cache at 99.99%. Because a single request needs all three to succeed, the composed availability is the product of the three:

That's under three nines even though every individual dependency is at or above three nines, because failures compound across a serial chain. In annual downtime terms:

compared to a single 99.9% dependency on its own:

Chaining three otherwise-strong dependencies serially costs over 300 extra minutes of downtime a year versus just one of them alone. This is why an API-focused architect pushes hard on redundancy at each hop. To see how strong that lever is even when the underlying component is weaker, consider a hypothetical, cheaper cache tier, deliberately worse than the 99.99%-rated cache used above, where each individual replica only hits 99% availability on its own: two independent, parallel replicas of that weaker cache layer already beat any single component in the chain, the strong 99.99% cache included:

A background worker processing a queue of jobs, by contrast, doesn't need to respond within milliseconds; what matters is that every job eventually completes correctly, with no silent data loss, which is a reliability property, not an availability one. If the worker is down for ten minutes and then resumes and correctly processes every job that queued up during that window, availability took a hit but reliability didn't; if the worker stays "up" the whole time but drops or duplicates 0.01% of jobs due to a bug, availability looks perfect while reliability has quietly failed.

Trade-offs & pitfalls

Optimizing for availability alone can mask reliability problems: a service that always responds quickly, even by returning stale or wrong data rather than waiting for a correct answer, looks perfect on an uptime dashboard while silently corrupting downstream state. The practical approach is deciding, per component, which property is actually load-bearing: user-facing APIs generally prioritize availability with graceful degradation for correctness-adjacent risk, while systems of record and background processing prioritize reliability, often accepting higher latency or even temporary unavailability rather than risk an incorrect or lost write.

Direct answer

Automate failover when the detector is high-precision, the failover action is reversible and idempotent, and the cost of a wrong automatic trigger is bounded and recoverable. Require a human when any of those breaks down, especially when a wrong trigger risks unrecoverable data divergence or an irreversible action. Expected-value math on detection accuracy alone favors automation more than intuition suggests, but it is reversibility and blast radius, not raw precision, that should gate the decision.

Structured elaboration

A hybrid middle ground. Mature systems rarely pick one point on the automate-versus-manual spectrum. They tier it: automated detection and containment (circuit breakers, traffic throttling) run automatically because those actions are cheap to reverse, while the highest-blast-radius action (full regional failover, promoting a new primary) goes through an automated-detect, human-approve gate with an escalation timeout if nobody responds.

flowchart TD
    A[Alert fires] --> B{Multi-signal, high-precision detector?}
    B -->|No| M[Manual: page human, human confirms before failover]
    B -->|Yes| C{Action reversible and idempotent?}
    C -->|No| H[Hybrid: auto-detect and auto-contain, human approves full failover]
    C -->|Yes| D{Wrong trigger risks split-brain or data loss?}
    D -->|High risk| H
    D -->|Low risk| E[Automate: auto-detect and auto-failover with fencing token and audit log]

A fencing token here is a number that increases with every failover action; if a stale, already-superseded actor (an old primary that thinks it's still in charge, for example) tries to act after a newer one has taken over, its writes carry an outdated token and get rejected, so a late-arriving action from a process that no longer should be acting can't silently corrupt state.

Framing it as expected value. For a given alert, the expected value of automatic failover is:

where $p$ is the detector's precision, the probability an alert reflects a real failure.

Worked example

Assume correct auto-failover cuts RTO from a 15-minute human-paged response to a 2-minute automatic one, a 13-minute improvement, against a downtime cost of $50k/hour:

A false trigger causes roughly 3 minutes of avoidable disruption (connection draining and reconnect storms) at the same rate:

At a detector precision of $p = 0.9$:

Solve for the breakeven precision where $EV = 0$:

Pure expected value favors automation down to a detector that is right only 19% of the time, far noisier than any detector actually deployed. That is the point: raw EV almost always says automate. The equation treats every false trigger as a bounded $2,500 cost, which is only true if the action is reversible. If a wrong trigger can cause split-brain or an irreversible data promotion, the real cost of that tail case is not in the equation at all, which is why reversibility, not precision, is the dominant factor in practice.

Trade-offs & pitfalls

• The most common wrong turn is optimizing for detector precision and stopping there; a 99%-precision detector triggering an irreversible action is still a bad automation candidate if the 1% case is catastrophic.
• Automating containment (throttle, circuit-break) before automating the full failover captures most of the RTO benefit with much lower blast radius; teams often skip straight to automating the whole failover and take on risk they did not need.
• An approval gate with no timeout just becomes a slower manual failover with extra steps; if a human stays in the loop, define an explicit escalation timeout.
• Chaos-testing the automated path before trusting it in production is not optional. An automation that has never been exercised against a real failure is a new, untested failure mode, not a safety net.