Fault Tolerance, High Availability, and Disaster Recovery Questions

Direct answer: Not every service deserves the same redundancy budget, because redundancy costs money and operational complexity that only pays off if the service's failure actually hurts the business. I'd tier services by blast radius (does it touch revenue, does it touch customer trust, is there a regulatory SLA) rather than by how technically interesting the service is, and assign each tier a concrete SLO (service level objective: the target reliability number the team commits to hitting, e.g. 99.9% uptime), redundancy model, and error budget derived from that SLO, not an arbitrary one.

Structured elaboration: a three-tier model

The error budget column isn't asserted, it's derived from the SLO the same way allowed downtime is derived from an availability target: $\text{budget}_{\text{month}} = (1-A) \times 730 \times 60$ minutes. Notice the tiers are roughly 10x apart from each other in budget (4.38 to 43.8 to 438 minutes), which mirrors the 10x-per-nine relationship and makes the tiers easy to reason about and communicate.

How I'd assign a service to a tier:

1. Revenue/legal impact: does an outage stop money from moving, or breach a contractual SLA? If yes, floor is Critical regardless of traffic volume.
2. Blast radius to other teams: is this a shared dependency other services call synchronously? A "low traffic" internal auth service can still deserve Critical tier because everything downstream inherits its outage.
3. User-facing vs. internal: internal tools default one tier lower than an equivalent customer-facing service, unless (2) applies.
4. Recoverability: if a noncritical batch job fails, does it self-heal on the next run with no lasting damage? If yes, it can sit at 99% even if "important" sounding, because the cost of a missed run is low.

Worked example applying the model: a checkout service (Critical, 99.99%, active-active) calls a recommendations service (Important, 99.9%) for a "customers also bought" widget. If recommendations goes down, checkout's tier assignment only holds if checkout is built to degrade gracefully when that dependency fails (skip the widget, keep processing the order), rather than blocking checkout on it. This is the practical reason tiering isn't just about the service itself, it forces you to also classify every dependency edge as "must be up for me to meet my SLO" or "nice to have," which is often the more useful output of the exercise than the tier label itself.

Trade-offs & pitfalls

• Over-tiering (calling too many services Critical) defeats the purpose: if everything is active-active multi-region, you've spent the budget without actually differentiating risk, and you've made the org slower to ship everywhere instead of faster where it doesn't matter.
• Tiers need a review cadence. A service that started as an internal tool and quietly became load-bearing for a customer-facing flow needs to be re-tiered; without a periodic review, tier assignments rot and stop reflecting actual blast radius.
• Political pressure to over-tier is common (every team believes their service is critical); the fix is grounding tier assignment in the revenue/legal/blast-radius questions above with a named approver, not a self-declared label.
• A common wrong turn: tiering the service but not its dependencies. A Critical-tier service that synchronously calls a Noncritical-tier dependency on its hot path has effectively downgraded itself to the dependency's tier, no matter what its own SLO document says.

Direct answer

Frame the cut through the 99.99% error budget, not a blanket percentage. 99.99% allows about 52.6 minutes of downtime a year; anything that doesn't touch the paths that consume that budget is safe to cut aggressively, and anything that does must be modeled against it explicitly before you touch it. In practice that means chasing waste and inefficiency hard (usually most of the 30%), and treating redundancy, failover paths, and DR test cadence as protected unless you can show the specific budget impact is acceptable.

A three-bucket framework for the savings

That number is the currency every cut gets priced in.

Chase bucket 1 and 2 first and aggressively (they rarely conflict with availability); only reach into bucket 3 if 1 and 2 don't get you to 30%, and price every bucket-3 cut against the budget above.

Worked example

Suppose an audit of monthly compute spend of $300,000 finds 15% idle or overprovisioned capacity, plus another 10% recoverable through rightsizing and reserved commitments on stateless, replaceable capacity:

That's 25 of the 30 points from buckets 1 and 2 alone, with essentially zero availability risk. The remaining 5 points has to come from bucket 3, for example dropping a redundant standby from 3 independent paths to 2 on some service. Whether that's safe depends entirely on whether that service is in the 99.99% critical scope. Reusing the same series/parallel redundancy model above (an N-way redundant path's unavailability is the product of each independent unit's own unavailability, since all N units have to fail at the same time for the whole path to be down: $U_N = (1-a)^N$), for a building block at $a=0.99$:

Dropping that one path from N=3 to N=2 raises its expected downtime contribution from about half a minute a year (0.5256 min/yr) to 52.56 minutes a year, a 100x jump that would consume the entire annual budget for the 99.99% target on that one path alone. That's the concrete argument for why redundancy counts on in-scope critical services are protected regardless of the cost target: the math shows the cut doesn't save what it looks like it saves once you price the risk.

Trade-offs and pitfalls

The classic failure is applying a flat 30% cut across the board instead of segmenting critical from non-critical, which either misses easy wins in non-critical systems or, worse, quietly erodes redundancy on a critical path because nobody explicitly modeled the budget impact. Watch for Goodhart's-law style gaming too: cutting observability or alerting spend looks free on the invoice but raises mean time to detect, which inflates the effective downtime against the same budget without showing up as an "availability" line item until an incident hits. What to protect no matter what: replica or quorum counts below the tested minimum on in-scope services, cross-region failover paths, backup and DR test cadence, and on-call staffing, because all four either directly hold the redundancy math above or determine how fast you can react when it fails.

Direct answer: Redundancy is having extra, interchangeable components standing by so one can take over if another fails; replication is actively keeping copies of state (data) synchronized across multiple nodes so the state itself survives a failure, not just the compute that serves it. They're often used together, but they solve different problems: redundancy alone is enough for a stateless service, because any interchangeable instance can serve any request; a stateful service needs replication too, because a fresh redundant instance with no data isn't actually a working replacement.

Structured elaboration

Stateless example: a set of identical app server instances behind a load balancer, handling API requests with no local state. If one instance dies, the load balancer routes around it and an autoscaler replaces it; the new instance needs no data transfer because there was never any instance-local state to lose. This is pure redundancy: extra interchangeable copies of the same stateless computation.

Stateful example: a primary-replica database. The primary accepts writes; replicas continuously receive a copy of the write stream (replication). If the primary fails, a replica is promoted to take over. Unlike the stateless case, simply having an extra database instance running (redundancy alone, no replication) would give you an empty database, not a working replacement, because there's no mechanism copying the actual data into it.

A failure mode redundancy alone doesn't protect against, for the stateful case: data loss or corruption on the primary itself. If the primary's disk corrupts a row, or a bad write silently corrupts application-level data, having a redundant (but not yet caught-up, or synchronously replicating that same bad write) standby doesn't help, because either the standby doesn't have the data yet (async lag) or it faithfully replicated the corruption along with everything else (synchronous replication of a logically bad write). Redundancy protects against a node dying; it does not protect against the data itself being wrong, that requires backups (a separate, point-in-time copy decoupled from live replication) and, for silent corruption specifically, checksums or application-level validation.

How this generalizes: a useful mental checklist for fault tolerance covers five distinct techniques, and redundancy and replication are only two of them: retries (recover from a transient failure by trying again), bulkheads (isolate one failure from spreading to unrelated resources), failover (the mechanism that switches traffic to a healthy replacement), redundancy (having that replacement exist at all), and replication (making sure the replacement actually has the state it needs). A strong answer names which of these a given design decision is actually addressing, since "redundancy" gets used loosely to mean all five in casual conversation.

Trade-offs & pitfalls

• Replication has a cost redundancy alone doesn't: network bandwidth, storage for extra copies, and a consistency model to reason about (synchronous replication costs write latency; asynchronous replication risks data loss on failover, the classic RPO trade-off).
• A common wrong turn: assuming "we have 3 replicas" automatically means "we're protected," without checking replication lag. A replica that's minutes behind at failover time silently loses however much data arrived in that window, unless the promotion logic explicitly accounts for lag and refuses to promote a too-far-behind replica.
• Redundancy for stateless services is comparatively cheap and low-risk to over-provision; replication for stateful services is not, since more replicas means more write-path coordination overhead (for synchronous replication) or more divergence risk (for asynchronous/multi-leader), so it isn't a "just add more" lever in the same way.

Direct answer

Isolate at three layers so no single control point being bypassed breaks the guarantee: admission control at the edge (reject or queue over-quota requests before they consume shared resources), hard resource enforcement at the OS/scheduler level (so a tenant that gets past admission still can't starve others), and fairness-aware autoscaling and billing (scale for real aggregate demand, allocate new capacity by weighted share rather than first-come-first-served, and meter overage so legitimate extra usage is paid for instead of stolen from neighbors).

Architecture and enforcement

flowchart LR
  T[Tenant request] --> GW[Admission gateway: token bucket]
  GW -->|within quota| ENQ[Per-tenant queue]
  GW -->|over quota| REJ[429 / shed]
  ENQ --> SCHED[Scheduler: cgroup CPU/IO caps]
  SCHED --> POOL[Shared compute pool]
  POOL --> AS[Autoscaler: per-tenant saturation signal]
  AS --> POOL
  POOL --> MET[Usage metering]
  MET --> BILL[Billing: quota + overage]

Base quota per tenant is a weighted fair share of total capacity $C$:

• Admission layer: per-tenant token bucket enforces request-rate and concurrency caps before work is queued; this stops a noisy tenant from ever occupying shared queue depth.
• Enforcement layer: the OS and scheduler cap what a tenant's workload can actually consume, CPU time, memory, and network bandwidth, even if a request slips past admission control, defense in depth rather than trusting one gate alone. (Advanced implementation detail: on Linux this is typically cgroups for CPU bandwidth limits, OOM-score tuning to control which processes get killed first under memory pressure, and network policers such as tc/ingress qdiscs to cap per-tenant bandwidth.)
• Autoscaling: the trigger must look at per-tenant saturation, not just cluster-wide CPU. If a single tenant is the one hammering the cluster, scaling out on aggregate cluster metrics effectively rewards that tenant with new capacity paid for by everyone, which defeats the isolation goal economically even if it holds technically.
• Billing: usage beyond the base quota is metered as overage rather than throttled outright for tenants on a plan that allows bursting; quota tier is the plan's contract, burst is a paid escape valve, not a loophole.

Worked example

Cluster capacity $C=1000$ vCPU across four tenants with weights A=3, B=1, C=1, D=1 (their plan tiers):

If tenant D is idle, its unused 166.67 vCPU can be reclaimed and redistributed proportionally among the active tenants (weights 3, 1, 1 summing to 5), rather than sitting wasted:

A' + B' + C' + 0 = 1000, the full capacity, allocated fairly by weight instead of by who asked first. If D comes back online, the reclaim has to be preemptible (graceful, with a grace period) so D isn't hard-killed the moment it wakes up; it gets its guaranteed floor back and the borrowers scale down.

Trade-offs and pitfalls

Hard caps with no reclaim protect worst-case isolation perfectly but waste real capacity whenever a tenant is idle, which is most of the time for most tenants. Soft caps with reclaim use capacity efficiently but add real scheduler complexity, and the failure mode to avoid is yanking borrowed capacity abruptly instead of draining it, which turns "fairness" into a second source of noisy-neighbor-style disruption for the tenant who briefly borrowed it. The single most common wrong turn in this design is coupling the autoscaler's scale-out trigger directly to raw cluster-wide metrics: it looks like a capacity problem and "solves" it by adding nodes, but if the actual cause is one tenant blowing past its quota, autoscaling just subsidizes the noisy tenant at everyone else's expense until billing catches up, if it ever does.

Direct answer

Prevent the herd on the client side by spreading reconnection attempts out in time instead of letting every client retry at once, and on the server side by capping how many new connections get admitted per second and prioritizing the requests that matter most when demand exceeds that cap. Neither side alone is sufficient: perfect client-side jitter still fails if enough clients exist that even a "spread out" retry burst exceeds server capacity, and server-side admission control alone still means every client is hammering the door at once, just getting turned away instead of served, which is its own load problem.

Client-side: spreading the reconnect burst

• Exponential backoff with full jitter: on failure, wait min(maxBackoff, base * 2^attempt) * random(0, 1), not a fixed or even a deterministic exponential delay. Without the random multiplier, every client computes the identical backoff schedule and they all retry in sync anyway, which defeats the purpose.
• Staged reconnection windows: assign each client a window bucket derived from a stable hash of its client ID (hash(clientID) % N), and have it wait for its assigned window before attempting the first reconnect after a failover event. This smooths the very first wave of reconnects, which is usually the largest spike, before backoff-driven jitter takes over for any retries after that.
• Local rate limiting: a client-side token bucket capping how many new-connection attempts a single client (or client SDK instance, for a service-to-service caller) makes per second, so a client that's aggressively retrying in a loop due to a bug doesn't contribute an outsized share of the storm on its own.

Server-side: admission control and prioritization

• Global admission control: a token bucket (lets a client or window burst up to the bucket's size as long as its average rate stays within budget, so a legitimate short spike isn't punished) or leaky bucket (smooths every burst down to a strictly constant output rate, simpler to reason about but with no headroom for a legitimate spike) at the entry point representing sustainable connection or request throughput; once the bucket is exhausted, new connections get a 503 with a Retry-After header and, ideally, a suggested backoff window, rather than being accepted and then failing downstream.
• Priority tiers: not all reconnecting clients matter equally in the first seconds after a failover. Reserve a minimum percentage of capacity for high-priority traffic (auth, payments) ahead of admitting lower-priority traffic, using weighted fair queueing (each tier gets a guaranteed share of whatever processing capacity remains, in proportion to its assigned weight, so a lower-priority tier still gets served, just less of it, instead of being frozen out entirely while a higher tier is busy) within each tier so no single tier starves the others once its reservation is met.
• Ramp-up for admitted clients: even an admitted client shouldn't be allowed to immediately issue a full burst of requests; a short local rate limit on the newly-established connection avoids the "got in the door, then overwhelmed the backend anyway" failure mode.

Worked example: tracing the numbers

Say a failover just happened and 10,000 clients need to reconnect. On the client side, each client hashes its own ID into one of $N=20$ staged reconnection windows via hash(clientID) % 20; since the hash spreads client IDs roughly evenly, that works out to about $10{,}000 / 20 = 500$ clients assigned to each window. If windows open 500ms apart, the last window (bucket 19) doesn't open until $19 \times 500\text{ms} = 9.5\text{s}$ after the failover, so the whole staged rollout finishes within about 10 seconds, a lot smoother than every client hitting the new primary in the same instant. Within a single window, exponential backoff with full jitter still spreads that window's 500 clients across roughly the first 200ms of it rather than firing in the same millisecond, which works out to a peak arrival rate of about $500 / 0.2\text{s} = 2{,}500$ connections/sec from that one window's worth of clients. On the server side, suppose the newly-promoted primary can sustainably admit 3,000 connections/sec while still serving already-connected traffic; the 2,500/sec peak from one window fits comfortably under that 3,000/sec admission cap, so well-behaved (jittered) clients rarely get turned away with a 503 at all. The admission control's real job is the tail: naive or third-party clients that skip jitter and retry immediately in a loop, or the rare case where two windows' retries overlap after their own backoff, both of which the 3,000/sec cap catches and pushes back on with Retry-After rather than letting the primary fall over. (These figures are illustrative, chosen to be internally consistent, not measured from a real system; the actual $N$, window spacing, and admission cap for a given service come from load-testing its real client population and its real sustainable throughput.)

Trade-offs & pitfalls

Staged reconnection windows trade recovery speed for smoothness: a larger number of buckets $N$ spreads load more evenly but also means the last bucket doesn't even attempt to reconnect until later, so full traffic isn't restored until that window closes; picking $N$ is a direct trade between "how smooth" and "how fast fully recovered." Centralized server-side admission control gives the cleanest global view of capacity but becomes a coordination point of its own at very high scale, so large deployments typically shard the token bucket per backend shard or use an approximate, eventually-consistent counter rather than a single strictly-consistent global counter, accepting slightly imprecise enforcement in exchange for avoiding a new bottleneck. The single most common mistake is only solving this for the well-behaved client population and forgetting that naive or third-party clients that don't implement jitter still exist; the server-side admission control and clear Retry-After semantics are what protect the system against those clients, since you can't force every caller to implement backoff correctly. It's also worth noticing that a reconnection storm is really one instance of a more general pattern: the same admission-control-plus-jitter combination is the fix whether the trigger is a failover causing mass reconnects, a large cache's keys all expiring at the same instant and stampeding the origin, or a burst of failed calls each independently retrying and amplifying load on an already-struggling dependency. The prevention mechanism is the same in all three cases even though the trigger event is different.

Validating the fix

Load-test the specific failure mode, not just steady-state traffic: simulate a failover event, then fire a synthetic burst of N simultaneous reconnects with the client-side jitter logic enabled, and confirm the server's accepted-connections-per-second curve stays under the admission cap rather than spiking. Compare against the same test with jitter disabled to confirm the fix is actually doing something (a test that passes whether or not the fix is present isn't validating anything). Chaos-test the combined system periodically (trigger a real failover in staging and observe the reconnect curve end to end) rather than only unit-testing the backoff math in isolation, since the interaction between client jitter and server admission control is exactly the kind of behavior that's easy to get individually correct and collectively wrong.