Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer

The 5-minute TTL isn't the actual bottleneck: DNS caching in the real world (ISP resolvers with minimum-TTL floors, browser caches, persistent keep-alive connections that never re-resolve at all) means real failover time doesn't track the advertised TTL cleanly, which is exactly why you're seeing 3 minutes instead of something close to 5. Getting under 30 seconds for most clients means building an explicit time budget (detect, update, propagate) that sums under 30s for compliant clients, and accepting that DNS alone can't guarantee it for the tail of clients whose resolvers or connections don't re-check in time.

Building the time budget

where $k$ is the number of consecutive failed health checks required before failing over (the detection threshold) and interval is the health-check period.

sequenceDiagram
  participant C as Client
  participant R as Resolver
  participant D as Authoritative DNS
  participant H as Health Monitor
  participant A as Origin A
  participant B as Origin B
  H->>A: probe every 5s
  A--xH: 2 consecutive failures (10s)
  H->>D: update record to B
  C->>R: resolve hostname
  R->>D: query (TTL expired)
  D-->>R: return B, TTL 10s
  R-->>C: B
  C->>B: connect

Worked example

Redesign inputs: health-check interval 5s, failure threshold k=2 (avoids single-blip flaps), API-driven record push under 1s, and TTL lowered from 300s to 10s.

That covers clients and resolvers that honor the lowered TTL, with about 9 seconds of margin. It does not cover the two categories that caused the original 3-minute number: resolvers that enforce a minimum TTL floor above what you set, and clients holding a persistent connection that has no reason to re-resolve DNS at all until it errors. For those, add a client-side backstop that's independent of TTL: short keep-alive and idle timeouts so connections periodically re-establish (and therefore re-resolve), and connect-level retry to a secondary IP on failure (a Happy-Eyeballs-style fallback) rather than trusting DNS to be the only failover signal.

Trade-offs and pitfalls

What you give up: a 10s TTL multiplies authoritative DNS query volume roughly 30x versus the 300s baseline, which is a real cost and load increase on your DNS infrastructure, and a false-positive failover (from setting k too low) now flips production traffic in as little as 5 to 10 seconds, so your health check needs to be more conservative about what counts as "down," not less. The pitfall that caused the original bug is assuming all clients and resolvers honor your TTL uniformly; they don't, and any redesign that only lowers the TTL without a client-side or network-level backstop will hit the same wall for the same tail of misbehaving resolvers, just with a lower number attached to it.

Approach

A circuit breaker is a state machine with three states and two triggered transitions plus one time-based transition:

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> OPEN: consecutive_failures >= threshold
    OPEN --> HALF_OPEN: cooldown elapsed
    HALF_OPEN --> CLOSED: probe succeeds
    HALF_OPEN --> OPEN: probe fails
    CLOSED --> CLOSED: call succeeds, reset counter

The implementation needs: a counter for consecutive failures, a timestamp for when the breaker opened, a lock so state transitions are atomic under concurrent callers, and a guard in the HALF_OPEN state that only lets exactly one trial call through at a time (otherwise every thread that arrives during HALF_OPEN would fire its own trial simultaneously, defeating the point of testing recovery with a single probe). Work happens outside the lock so a slow downstream call doesn't block every other thread from checking the breaker's state.

Code (Python)

import time
import threading
from enum import Enum


class CircuitOpenError(Exception):
    pass


class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=10.0):
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds

        self._state = State.CLOSED
        self._consecutive_failures = 0
        self._opened_at = None
        self._half_open_probe_in_flight = False
        self._lock = threading.Lock()

    @property
    def state(self):
        with self._lock:
            self._maybe_recover()
            return self._state

    def _maybe_recover(self):
        # caller must hold self._lock
        if self._state == State.OPEN and self._opened_at is not None:
            if time.monotonic() - self._opened_at >= self.cooldown_seconds:
                self._state = State.HALF_OPEN
                self._half_open_probe_in_flight = False

    def call(self, fn, *args, **kwargs):
        with self._lock:
            self._maybe_recover()
            if self._state == State.OPEN:
                raise CircuitOpenError("circuit is open, call rejected")
            if self._state == State.HALF_OPEN:
                if self._half_open_probe_in_flight:
                    raise CircuitOpenError("half-open probe already in flight")
                self._half_open_probe_in_flight = True

        try:
            result = fn(*args, **kwargs)
        except Exception:
            with self._lock:
                if self._state == State.HALF_OPEN:
                    self._trip()
                else:
                    self._consecutive_failures += 1
                    if self._consecutive_failures >= self.failure_threshold:
                        self._trip()
            raise
        else:
            with self._lock:
                self._consecutive_failures = 0
                self._state = State.CLOSED
                self._half_open_probe_in_flight = False
            return result

    def _trip(self):
        # caller must hold self._lock
        self._state = State.OPEN
        self._opened_at = time.monotonic()
        self._half_open_probe_in_flight = False

Verified against pinned assertions (deterministic, no wall-clock claims beyond a fixed 0.05s cooldown used only to make the test itself run fast):

def run_test():
    calls = {"n": 0}

    def flaky(should_fail):
        calls["n"] += 1
        if should_fail:
            raise RuntimeError("boom")
        return "ok"

    cb = CircuitBreaker(failure_threshold=3, cooldown_seconds=0.05)

    # 3 consecutive failures trips OPEN
    for _ in range(3):
        try:
            cb.call(flaky, True)
        except RuntimeError:
            pass
    assert cb.state == State.OPEN

    # calls are rejected without invoking fn while OPEN
    calls_before = calls["n"]
    try:
        cb.call(flaky, False)
        assert False, "expected CircuitOpenError"
    except CircuitOpenError:
        pass
    assert calls["n"] == calls_before

    # after cooldown, HALF_OPEN allows exactly one trial; success closes
    time.sleep(0.06)
    assert cb.state == State.HALF_OPEN
    assert cb.call(flaky, False) == "ok"
    assert cb.state == State.CLOSED

    # a failing trial in HALF_OPEN re-opens immediately
    for _ in range(3):
        try:
            cb.call(flaky, True)
        except RuntimeError:
            pass
    time.sleep(0.06)
    assert cb.state == State.HALF_OPEN
    try:
        cb.call(flaky, True)
    except RuntimeError:
        pass
    assert cb.state == State.OPEN

    print("all assertions passed")


run_test()  # prints: all assertions passed

Key points

• State transitions and counter updates all happen inside self._lock; the wrapped function fn executes outside the lock so one slow call can't block every other caller from even checking the breaker's state.
• _half_open_probe_in_flight is what limits HALF_OPEN to a single trial call; without it, every thread that arrives during the cooldown window would fire its own simultaneous trial against a possibly-still-struggling dependency.
• A failure while HALF_OPEN re-opens immediately, it does not require re-accumulating failure_threshold failures again, because a single failed trial is already sufficient evidence the dependency hasn't recovered.
• time.monotonic() is used instead of time.time() for the cooldown comparison, since wall-clock time can jump (NTP adjustment, manual clock change) but monotonic time cannot go backwards.

Complexity

Every operation (call, state, _maybe_recover, _trip) is $O(1)$ time and the breaker holds $O(1)$ state (a state enum, a counter, a timestamp, one boolean), independent of call volume or history length.

Edge cases

• Concurrent trial races in HALF_OPEN: handled by _half_open_probe_in_flight; a second thread arriving while a trial is in flight gets rejected immediately rather than launching a second simultaneous probe.
• Long-running trial call: the single-probe guard limits it to one concurrent trial, but a probe that hangs for a long time (rather than failing fast) delays recovery detection; a production version would typically pair the breaker with a timeout on the wrapped call itself so a hung probe can't block recovery indefinitely.
• Distributed instances: this implementation is single-process, in-memory state; if the same logical breaker needs to be shared across multiple service instances (so instance A tripping also stops instance B from hammering the same dependency), the state has to move to a shared store (Redis, or a sidecar) with the same atomicity requirement, which changes the performance and failure-mode profile (the shared store becoming a dependency of its own).
• Bulkhead-limited half-open probing: some implementations allow a small fixed number of concurrent HALF_OPEN trials (say 2 to 3) rather than exactly 1, trading a slightly larger blast radius during recovery testing for faster convergence back to CLOSED when the dependency genuinely has recovered; that's a one-line change to compare _half_open_probe_in_flight against a max count instead of a boolean.
• Success/failure classification: this implementation treats every exception as a breaker-relevant failure; a production breaker usually needs to distinguish exceptions that indicate the dependency is unhealthy (timeouts, connection errors, 5xx) from exceptions that are just normal application-level outcomes (a 404, a validation error) that shouldn't count against the breaker at all.

Direct answer

The bulkhead pattern partitions a system's resources (thread pools, connection pools, CPU, or entire nodes) into isolated compartments, named after a ship's watertight bulkheads, so that one failing dependency or one noisy tenant can only exhaust the resources in its own compartment, not the resources every other caller depends on. Without bulkheads, a single slow or misbehaving dependency can consume every available thread or connection in a shared pool, and a completely healthy code path fails simply because it couldn't get a thread to run on.

Where to draw the isolation boundary

A concrete example: an API gateway calls three downstream services, an inventory service, a recommendations service, and a payments service, all through one shared thread pool. If recommendations starts responding slowly, every thread in the shared pool eventually ends up blocked waiting on recommendations calls, and inventory and payment requests start timing out too, even though nothing is wrong with either of them. The fix is a dedicated, bounded thread pool (or connection pool) per downstream dependency: recommendations gets its own pool of, say, 10 threads, so a recommendations outage can stall at most those 10 threads and its own queue, while inventory and payments keep running normally on their own separate pools.

The boundary should sit wherever one caller's failure or slowness shouldn't be able to spill onto another caller's request. Common places to draw it:

• Per-downstream-dependency, as in the example above: each external service or database gets its own pool so a slow one can't starve calls to a fast one.
• Per-tenant, in a multi-tenant system: each tenant (or tenant tier) gets a capped share of connections or CPU so one noisy or abusive tenant can't degrade service for everyone else on shared infrastructure.
• Per-criticality-tier: payment and auth paths get reserved capacity separate from lower-priority paths like analytics or notifications, so a spike in low-priority traffic can't crowd out the paths that actually matter.

Trade-offs & pitfalls

Bulkheads trade utilization for isolation: reserved capacity that a compartment isn't currently using sits idle rather than being available to a busier compartment, so a poorly sized bulkhead can cause localized throttling even while the system as a whole has spare capacity. Sizing is the actual hard part in practice, not the pattern itself: too small and a legitimate burst of normal traffic gets rejected by its own bulkhead; too large and the isolation becomes theoretical, because if every pool is sized close to the shared pool's original total, a single compartment can still consume enough of the machine's real resources (CPU, memory, file descriptors) to degrade its neighbors even though the pool counters look fine. Bulkheads are also a different tool from a circuit breaker and the two are frequently confused: a bulkhead limits how much of a shared resource one dependency can consume (a capacity boundary), while a circuit breaker stops sending requests to a dependency once it's clearly failing (a decision to stop calling at all); they're complementary, since the bulkhead caps the damage while the circuit breaker is deciding whether to keep trying, and production systems typically use both on the same dependency together. The same reasoning extends beyond web request threads: an ML-serving platform running GPU inference for multiple models on shared hardware applies the identical idea by pinning each model (or tenant) to a dedicated slice of GPU memory and compute, so one model that starts issuing runaway-batch-size requests can't starve GPU capacity away from every other model sharing that hardware.

Active-active runs all nodes or regions serving live traffic at the same time, so failover is mostly a routing problem: stop sending traffic to the unhealthy node. Active-passive keeps one side idle (or partially warmed) as a standby, so failover is a promotion problem: detect the primary is down, make the standby the new primary, then redirect traffic to it. That difference in what has to happen during failover is what drives everything else: recovery speed (RTO, recovery time objective: how long it takes to restore service after a failure), data-consistency risk (RPO, recovery point objective: how much data, measured in time, you could lose in a failure), and cost.

Comparing the two

Worked example: deriving RTO for each

Pin the same detection policy to both: a health check runs every 5 seconds, and 3 consecutive failures are required before the system acts (15 seconds to declare a node down; this avoids reacting to a single dropped probe).

Active-active RTO: once the node is declared down, the load balancer or GSLB removes it from rotation immediately (no promotion step). If we assume that rotation update takes about 5 seconds to propagate to all edge/LB nodes:

Active-passive RTO: the same 15 seconds to detect, plus a promotion step (electing the standby, replaying any un-applied log entries, opening it for writes, say 20 seconds for a warm standby with low replication lag) plus DNS or routing propagation (assume a low TTL of 30 seconds, and worst case the full TTL has to expire before every client picks up the change):

Under these pinned assumptions, active-active recovers about 3x faster, entirely because it skips the promotion step, and that gap only grows if the standby is cold rather than warm (add provisioning time) or if the routing layer is DNS with a high TTL instead of a fast health-checked LB.

Trade-offs and pitfalls

Active-active's speed advantage is real but not free: the moment two regions can both accept writes to the same record, you have a distributed-write problem, and skipping it (assuming replication will "just" reconcile) is the most common wrong turn. It needs either a conflict-resolution strategy (last-write-wins with vector clocks, CRDTs) or partitioned ownership (each region owns a disjoint key range) to avoid silently losing or corrupting data during a partition. Active-passive's failure mode is different: over-eager health checks or a flapping network link can trigger a premature promotion while the old primary is still technically reachable by some clients, producing two nodes that both believe they're primary (split-brain), which is why production active-passive systems add fencing (forcibly cutting off the old primary's access to shared storage) on top of the detection logic above, not just a timer. The same trade-off shows up outside a classic web/DB stack too: failing over an ML-serving fleet is closer to active-active in spirit (multiple replicas of the same model serving concurrently, so losing one just drops capacity) unless the model itself is being hot-swapped, in which case the promotion-style risk (serving from a half-loaded or stale model version) reappears.

Direct answer: Graceful degradation means a system keeps serving its core value under partial failure by deliberately shedding non-essential features, instead of failing completely because one dependency is unhealthy. It matters because most real outages are partial, not total, and a system that can't distinguish "checkout is down" from "product recommendations are down" ends up treating both the same way: total outage, when only one of them actually deserved it.

Structured elaboration

The core discipline is ranking features by how essential they are to the user's actual goal, then deciding in advance what happens to each tier when its supporting dependency fails:

How you decide what's "core": ask whether the feature is on the path the user came for. For a checkout service, that's the cart-to-payment path; product recommendations, reviews, and "recently viewed" are enrichment around that path, valuable but not why the user is there. For a search service, returning some relevant results is core; typo-correction, personalized re-ranking, and query autocomplete are enrichment that can be dropped without breaking the user's ability to search.

Detecting when to degrade: this has to be automatic, not something a human decides mid-incident. Health checks and latency/error-rate thresholds on each dependency feed a circuit breaker; when the breaker for the recommendations service opens, the front end (or an API gateway) simply omits that section rather than waiting on a call that's failing. The degraded state should be visible in monitoring (a "degraded mode" flag, not silence) so the team knows it's active and can address root cause.

Trade-offs & pitfalls

• Degrading too aggressively removes revenue-generating features (recommendations often drive real conversion) for failures that didn't actually require it; the tiering has to be based on actual dependency health, not a blanket "anything non-core gets cut."
• Degrading too conservatively (waiting too long, or requiring a human to flip a switch) means the cascading failure the degradation was supposed to prevent happens anyway, because by the time a human reacts, the core path is already backed up.
• Static thresholds don't generalize across traffic levels; a latency threshold tuned for average traffic can either never trigger during a real incident at peak load, or trigger too eagerly during a routine traffic spike that isn't actually a failure.
• Testing degraded paths is easy to skip because they're rarely exercised in normal operation; without deliberately forcing dependencies to fail in staging (or via chaos testing in production), the first real test of the degraded path is during an actual incident, which is the worst time to discover it's broken.
• The same tiering logic applies outside typical web services: an ML-serving system facing a slow or unavailable model can fall back to a cached prior response, swap to a smaller/cheaper model that's faster but less accurate, or return a safe default decision, the exact same "protect the core interaction, shed the enrichment" reasoning, just with "model quality" instead of "page richness" as the thing being traded off.