Fault Tolerance, High Availability, and Disaster Recovery Questions

Plain exponential backoff (double the delay after each failed attempt) reduces load on a struggling dependency over time, but it has a hidden flaw: if many clients failed at roughly the same moment (which is exactly what happens when the dependency itself goes down), they all compute the same delay sequence and retry in lockstep, so the "backoff" just delays the same synchronized spike instead of spreading it out. Jitter fixes that by randomizing the delay so clients that failed together don't retry together.

Jitter strategies compared

Worked example: how much jitter actually reduces the spike

Pin a concrete scenario: 1000 clients failed at the same moment, base delay = 1 second, and this is their 3rd retry attempt (attempt = 3), so the backoff cap is:

$$\text{cap} = 1\text{s} \times 2^3 = 8\text{ seconds}$$

Without jitter: every one of the 1000 clients computes the identical 8-second delay and retries at exactly the same instant, a spike of 1000 concurrent requests hitting the dependency in one moment, right as it may just be starting to recover.

With full jitter, each client independently draws a delay uniformly from $[0, 8]$ seconds. Dividing that 8-second window into 100 ms buckets gives $8000 / 100 = 80$ buckets, and under a uniform distribution the expected number of clients landing in any single bucket is:

$$\frac{1000}{80} = 12.5 \text{ requests per 100ms bucket}$$

That's a peak-to-average reduction factor of $1000 / 12.5 = 80\times$ under this modeling assumption (uniform, independent draws), turning one instantaneous spike of 1000 into a smooth trickle of roughly 12-13 requests every 100 ms across the full 8-second window, which a recovering dependency can absorb where a single 1000-request spike would knock it back down.

Why retries shouldn't make recovery worse

Jitter alone doesn't prevent the retry storm from getting worse over time if attempts aren't capped: a client that keeps failing and keeps retrying at $\text{base} \times 2^{\text{attempt}}$ forever will eventually be sending requests at a cap so large it's functionally giving up, or, worse, if the cap is bounded, converges back to a steady drumbeat of load that never lets the dependency fully recover. The fix is a hard cap on both the maximum delay and the maximum number of attempts, plus honoring any explicit signal the server provides (a Retry-After header or a 429/503 status) as authoritative over the client's own backoff schedule, since the server is in the best position to know its own recovery state.

Trade-offs and pitfalls

Full jitter maximizes spread but means some unlucky clients draw a near-zero delay and retry almost immediately, which is fine in aggregate (that's still only ~12-13 requests per 100ms bucket in the example above) but means full jitter alone doesn't guarantee a minimum backoff for any individual client; equal jitter trades some of that spread for a guaranteed floor, useful when even a small number of near-instant retries is unacceptable. A pitfall specific to mobile or otherwise unreliable-network clients: retries are only safe to jitter and reattempt if the underlying operation is idempotent (repeating it produces the same end result as doing it once, so a duplicate attempt is harmless), a non-idempotent submit (a payment, an order) retried after a client-side timeout can double-execute if the server had actually processed the first attempt and just failed to deliver the response, so the fix belongs on the server (idempotency keys deduping identical requests) not just in the client's backoff logic, jitter reduces load, it does not make an unsafe retry safe.

RTO (Recovery Time Objective) is how long you're allowed to be down: the maximum acceptable gap between an outage starting and service being restored. RPO (Recovery Point Objective) is how much data you're allowed to lose: the maximum acceptable gap, measured in time, between the last durably captured write and the moment of failure. The requirement "payments must be restored within 30 minutes with no more than 5 minutes of data loss" is literally RTO = 30 min and RPO = 5 min stated in plain language, and each number drives a different part of the design.

What each number drives

RPO = 5 minutes drives replication and backup frequency. A nightly or even hourly backup can't meet this: if the outage happens 4 hours after the last backup, you'd lose 4 hours of transactions, not 5 minutes. A 5-minute RPO effectively requires continuous replication (near-synchronous in-region, or streaming WAL (write-ahead log: a durable, ordered record of every change, written before it's considered applied) or CDC (change-data-capture: a stream of those same row-level changes read off that log) shipping to the DR site) with replication lag actively monitored and alarmed well below the 5-minute budget, plus point-in-time recovery for protection against logical corruption that replication alone would just copy.

RTO = 30 minutes drives standby readiness and failover automation. A cold-standby DR site that has to be provisioned from scratch after the fact will blow past 30 minutes just on infrastructure boot time. A 30-minute RTO points toward a warm standby (already running, sized down, kept current via the same replication that satisfies the RPO) with an automated failover runbook: health-check detection, automated promotion, and DNS/routing cutover, because a manual, human-paged process realistically eats 10-15 minutes just in detection and decision-making before any recovery action starts.

Worked example: what these numbers cost against an annual SLA

A useful way to make the 30-minute number concrete is to check it against annual downtime budgets at standard availability tiers, using $525{,}600$ minutes per year (365 $\times$ 24 $\times$ 60):

A single incident with a 30-minute RTO, if the service is held to a 99.99% SLA, consumes:

$$\frac{30}{52.56} \approx 0.571 \;\; (57.1\%)$$

of the entire year's downtime budget in one event. That reframes "30 minutes sounds generous" into "this design can absorb roughly one such incident a year and still hit four nines," which is exactly the kind of number that should drive whether the DR design gets warm-standby automation now or gets revisited after the first real incident eats most of the annual budget.

Trade-offs and pitfalls

The most common mix-up is treating RTO and RPO as interchangeable "how bad was it" numbers instead of two independent design constraints: a system can have a great RTO (back up in 2 minutes) and a terrible RPO (lost the last hour of writes) if it fails over to a backup instead of a live replica, or the reverse (RPO≈0 via synchronous replication, but a slow, manual promotion process blows the RTO). Both have to be solved, and usually by different mechanisms: RPO is a replication/backup-cadence problem, RTO is an automation/standby-readiness problem. A second pitfall specific to payments: RPO=0 sounds like the obviously "safer" number to chase, but strict synchronous replication that blocks writes during a replica outage can turn a replication hiccup into an availability incident, trading a data-loss risk you might never hit for a downtime risk you're now taking on every day.

Direct answer

Availability is whether the service is up and responding right now, the percentage of time requests get a correct response. Reliability is whether the service does the correct thing every time over a longer horizon, even if that means taking longer or failing loudly rather than silently. A service can be highly available (always responds) while being unreliable (frequently returns wrong or incomplete results), and vice versa.

How they're measured differently

• Availability: uptime percentage, request success rate (successful responses over total requests), and latency, all measured in real time against a rolling window.
• Reliability: job or transaction success rate over time, data-loss incidents, mean time between failures, and correctness checks like reconciliation counts, none of which are visible from a single point-in-time health check.

Worked example: an HTTP API versus a background worker

An HTTP API's job is to respond fast and stay up, so availability is the priority metric. Suppose the API calls three dependencies in sequence to serve a request: an auth service at 99.95% availability, a database at 99.9%, and a cache at 99.99%. Because a single request needs all three to succeed, the composed availability is the product of the three:

$$A_{serial} = 0.9995 \times 0.999 \times 0.9999 \approx 0.99840$$

That's under three nines even though every individual dependency is at or above three nines, because failures compound across a serial chain. In annual downtime terms:

$$\text{downtime}_{serial} = (1 - 0.99840) \times 525{,}600 \approx 840.6 \text{ min/yr}$$

compared to a single 99.9% dependency on its own:

$$\text{downtime}_{single} = (1 - 0.999) \times 525{,}600 \approx 525.6 \text{ min/yr}$$

Chaining three otherwise-strong dependencies serially costs over 300 extra minutes of downtime a year versus just one of them alone. This is why an API-focused architect pushes hard on redundancy at each hop. To see how strong that lever is even when the underlying component is weaker, consider a hypothetical, cheaper cache tier, deliberately worse than the 99.99%-rated cache used above, where each individual replica only hits 99% availability on its own: two independent, parallel replicas of that weaker cache layer already beat any single component in the chain, the strong 99.99% cache included:

$$A_{parallel} = 1 - (1-0.99)^2 = 0.9999$$

A background worker processing a queue of jobs, by contrast, doesn't need to respond within milliseconds; what matters is that every job eventually completes correctly, with no silent data loss, which is a reliability property, not an availability one. If the worker is down for ten minutes and then resumes and correctly processes every job that queued up during that window, availability took a hit but reliability didn't; if the worker stays "up" the whole time but drops or duplicates 0.01% of jobs due to a bug, availability looks perfect while reliability has quietly failed.

Trade-offs & pitfalls

Optimizing for availability alone can mask reliability problems: a service that always responds quickly, even by returning stale or wrong data rather than waiting for a correct answer, looks perfect on an uptime dashboard while silently corrupting downstream state. The practical approach is deciding, per component, which property is actually load-bearing: user-facing APIs generally prioritize availability with graceful degradation for correctness-adjacent risk, while systems of record and background processing prioritize reliability, often accepting higher latency or even temporary unavailability rather than risk an incorrect or lost write.

Direct answer: Graceful degradation means a system keeps serving its core value under partial failure by deliberately shedding non-essential features, instead of failing completely because one dependency is unhealthy. It matters because most real outages are partial, not total, and a system that can't distinguish "checkout is down" from "product recommendations are down" ends up treating both the same way: total outage, when only one of them actually deserved it.

Structured elaboration

The core discipline is ranking features by how essential they are to the user's actual goal, then deciding in advance what happens to each tier when its supporting dependency fails:

How you decide what's "core": ask whether the feature is on the path the user came for. For a checkout service, that's the cart-to-payment path; product recommendations, reviews, and "recently viewed" are enrichment around that path, valuable but not why the user is there. For a search service, returning some relevant results is core; typo-correction, personalized re-ranking, and query autocomplete are enrichment that can be dropped without breaking the user's ability to search.

Detecting when to degrade: this has to be automatic, not something a human decides mid-incident. Health checks and latency/error-rate thresholds on each dependency feed a circuit breaker; when the breaker for the recommendations service opens, the front end (or an API gateway) simply omits that section rather than waiting on a call that's failing. The degraded state should be visible in monitoring (a "degraded mode" flag, not silence) so the team knows it's active and can address root cause.

Trade-offs & pitfalls

• Degrading too aggressively removes revenue-generating features (recommendations often drive real conversion) for failures that didn't actually require it; the tiering has to be based on actual dependency health, not a blanket "anything non-core gets cut."
• Degrading too conservatively (waiting too long, or requiring a human to flip a switch) means the cascading failure the degradation was supposed to prevent happens anyway, because by the time a human reacts, the core path is already backed up.
• Static thresholds don't generalize across traffic levels; a latency threshold tuned for average traffic can either never trigger during a real incident at peak load, or trigger too eagerly during a routine traffic spike that isn't actually a failure.
• Testing degraded paths is easy to skip because they're rarely exercised in normal operation; without deliberately forcing dependencies to fail in staging (or via chaos testing in production), the first real test of the degraded path is during an actual incident, which is the worst time to discover it's broken.
• The same tiering logic applies outside typical web services: an ML-serving system facing a slow or unavailable model can fall back to a cached prior response, swap to a smaller/cheaper model that's faster but less accurate, or return a safe default decision, the exact same "protect the core interaction, shed the enrichment" reasoning, just with "model quality" instead of "page richness" as the thing being traded off.

Direct answer

The bulkhead pattern partitions a system's resources (thread pools, connection pools, CPU, or entire nodes) into isolated compartments, named after a ship's watertight bulkheads, so that one failing dependency or one noisy tenant can only exhaust the resources in its own compartment, not the resources every other caller depends on. Without bulkheads, a single slow or misbehaving dependency can consume every available thread or connection in a shared pool, and a completely healthy code path fails simply because it couldn't get a thread to run on.

Where to draw the isolation boundary

A concrete example: an API gateway calls three downstream services, an inventory service, a recommendations service, and a payments service, all through one shared thread pool. If recommendations starts responding slowly, every thread in the shared pool eventually ends up blocked waiting on recommendations calls, and inventory and payment requests start timing out too, even though nothing is wrong with either of them. The fix is a dedicated, bounded thread pool (or connection pool) per downstream dependency: recommendations gets its own pool of, say, 10 threads, so a recommendations outage can stall at most those 10 threads and its own queue, while inventory and payments keep running normally on their own separate pools.

The boundary should sit wherever one caller's failure or slowness shouldn't be able to spill onto another caller's request. Common places to draw it:

• Per-downstream-dependency, as in the example above: each external service or database gets its own pool so a slow one can't starve calls to a fast one.
• Per-tenant, in a multi-tenant system: each tenant (or tenant tier) gets a capped share of connections or CPU so one noisy or abusive tenant can't degrade service for everyone else on shared infrastructure.
• Per-criticality-tier: payment and auth paths get reserved capacity separate from lower-priority paths like analytics or notifications, so a spike in low-priority traffic can't crowd out the paths that actually matter.

Trade-offs & pitfalls

Bulkheads trade utilization for isolation: reserved capacity that a compartment isn't currently using sits idle rather than being available to a busier compartment, so a poorly sized bulkhead can cause localized throttling even while the system as a whole has spare capacity. Sizing is the actual hard part in practice, not the pattern itself: too small and a legitimate burst of normal traffic gets rejected by its own bulkhead; too large and the isolation becomes theoretical, because if every pool is sized close to the shared pool's original total, a single compartment can still consume enough of the machine's real resources (CPU, memory, file descriptors) to degrade its neighbors even though the pool counters look fine. Bulkheads are also a different tool from a circuit breaker and the two are frequently confused: a bulkhead limits how much of a shared resource one dependency can consume (a capacity boundary), while a circuit breaker stops sending requests to a dependency once it's clearly failing (a decision to stop calling at all); they're complementary, since the bulkhead caps the damage while the circuit breaker is deciding whether to keep trying, and production systems typically use both on the same dependency together. The same reasoning extends beyond web request threads: an ML-serving platform running GPU inference for multiple models on shared hardware applies the identical idea by pinning each model (or tenant) to a dedicated slice of GPU memory and compute, so one model that starts issuing runaway-batch-size requests can't starve GPU capacity away from every other model sharing that hardware.