Netflix Senior Site Reliability Engineer Interview Preparation Guide

Direct answer

In the meeting, redirect from "who" to "what" without dismissing the executives' urgency. Acknowledge that wanting accountability is a fair instinct after real impact, commit to a concrete, dated process for finding cause, and decline to name individuals in that room, because a name given under that kind of pressure becomes a verdict, not a finding.

The move: separate the two questions being asked at once

1. Acknowledge the pressure before redirecting it. Dismissing urgency ("let's not point fingers") reads as evasive right after real customer impact. Naming it as legitimate first earns you room to redirect.
2. Separate the two questions actually in the room: "are we stable right now" and "whose fault is this." Answer the first live, and say explicitly that the second is being deliberately parked, not avoided.
3. Commit to something concrete and dated, not a vague "we'll look into it." A named review with a defined output and a date is a commitment people can hold you to; a soft promise is not.
4. State the ground rule for that review out loud, in the room, while it's still tense. Contributing factors and process gaps, not individuals, is what will come back. Saying this while the executives are present makes it a commitment they witnessed, not a private policy you can quietly walk back.
5. Follow through in substance. The review has to actually surface system and process gaps rather than a person, or the promise made in that meeting becomes evidence of bad faith the next time something breaks.

Worked example

After a major outage, executives in the room want to know who's responsible, with the implication that someone should be let go. You acknowledge that wanting accountability after this kind of impact is completely reasonable, then separate the two questions: you answer "are we stable" directly with current mitigation status, and you name the second question, responsibility, as something you're deliberately not answering live because it needs evidence, not a reaction under pressure. You commit to a rapid review with a firm date and a defined output: a timeline, contributing factors, and remediation owners. You state the ground rule in the room: the review looks at system and process gaps, not individuals. The follow-up review actually produces that, and the remediation backlog with owners and dates is what goes back to the executives, not a name.

Trade-offs and pitfalls

Protecting "blameless" as a phrase in the room while privately identifying a scapegoat afterward is worse for trust than never promising blamelessness at all, because it's discovered eventually and reads as deliberate deception. Being so procedural and passive in the room that it sounds like dodging accountability altogether undercuts the point, you do need to commit to something concrete live, not just "we'll circle back." When there genuinely was individual negligence rather than a process gap, blameless doesn't mean no consequence, it means the review finds that out through evidence gathered afterward, not through a name given under live pressure in a tense meeting.

Direct answer

An SLI is the measured signal (e.g. the percentage of requests that were fast and successful), an SLO is the internal target for that signal over a time window (e.g. 99.9% of requests succeed under 300ms over a rolling 30 days), and an SLA is the external, often contractual promise built on top of an SLO, usually with margin, and real consequences (service credits, penalties) if it's breached. In short: the SLI measures, the SLO targets, the SLA promises with a business wrapper attached.

Defining each one concretely, for an e-commerce checkout service

Explained for a non-technical stakeholder: the SLO is the bar the engineering team holds itself to internally so problems get caught and fixed before they become customer-visible; the SLA is the, usually more forgiving, bar the business is willing to be held to externally, on paper, with money attached if it's missed. The gap between the two is deliberate headroom, not sloppiness.

Worked example: computing the error budget

The error budget is the amount of allowed failure baked into the SLO. It's what lets a team ship changes at all instead of freezing forever.

For the 99.9% SLO above:

Over the 30-day measurement window (30 days equals 2,592,000 seconds):

So this service is allowed about 43 minutes of budget-consuming failure (however that failure is defined: downtime, over-300ms responses, and so on) across a 30-day window before the SLO itself is breached. If checkout handles, say, 1,000,000 requests over that same window, the equivalent request-based budget is:

Both framings, time-based and request-based, describe the same budget; which one is more useful depends on whether the SLI is availability-style (was the service up) or ratio-style (what fraction of requests succeeded).

Trade-offs and pitfalls

• Confusing SLO and SLA in conversation causes real problems: teams sometimes design their alerting and release-gating around the SLA (the looser, contractual number) instead of the SLO (the tighter, internal number), which means by the time anyone notices, the team is already close to breaching the external promise with no margin left to react.
• An SLO with no error-budget policy attached is just a number on a dashboard; the value comes from what happens when the budget is nearly exhausted (freeze risky launches, redirect engineering time to reliability work), not from the target itself.
• Picking an SLI that doesn't reflect real user experience (e.g. "server process is running" instead of "requests succeed within an acceptable time") gives a green dashboard while users are still unhappy. The SLI has to be as close to what the user actually experiences as the team can measure.

Direct answer
Chaos testing fits into release validation as an OPTIONAL, carefully-scoped extra layer specifically for high-stakes releases: deliberately injecting a controlled failure (a dependency timeout, a node loss) during the canary phase to confirm the new version degrades gracefully under real adversity, not just under the ordinary traffic conditions a normal canary window happens to see.

Structured elaboration

• What faults to inject, and why during canary specifically: a simulated downstream-dependency timeout or error, a simulated node/pod loss, or injected network latency, run against the CANARY specifically (not the full production fleet) so any bad outcome from the chaos experiment itself is contained to the same small blast radius the canary was already limiting exposure to.
• Why canary is the right phase for this: a normal canary window validates behavior under whatever REAL conditions happen to occur during that window, which might not include a dependency hiccup at all; deliberately injecting one during canary tests a failure mode you might not otherwise observe before the release reaches full production traffic, catching a regression in error-handling or fallback logic specifically, not just in the happy path.
• Safety controls: a strict, pre-defined blast-radius limit on the experiment (only the canary's already-small traffic slice, never full production), an explicit kill switch to abort the chaos experiment immediately if it's causing more disruption than intended, and running it only during a window with adequate on-call attention, never as a fully unattended, unmonitored background process on a live release candidate.
• Feeding results back into rollout gates: if the canary handles the injected fault gracefully (a proper fallback, no cascading failure, error rates staying within an acceptable degraded-but-controlled band), that's positive evidence feeding into the normal promote/rollback decision alongside the standard metrics; if it reveals a genuine gap (the new version doesn't handle the injected failure as gracefully as the current stable version does), that's grounds to halt promotion and fix the gap, treated as seriously as a metric-based canary failure would be.

Worked example
A canary at 5% traffic has a chaos experiment injecting a simulated 2-second timeout on calls to a specific downstream dependency for a bounded 5-minute window. The canary's fallback logic (serving a cached, slightly stale response rather than erroring) engages correctly, and error rate stays within the acceptable band throughout, positive evidence added to the normal canary-analysis decision. On a different release, the same experiment reveals the canary's NEW error-handling path doesn't actually engage the fallback at all under this specific failure and instead returns a hard error to users, correctly halting that release's promotion despite the release having looked otherwise clean under normal traffic conditions.

Trade-offs and pitfalls
Chaos experiments add real operational overhead (someone needs to design, run, and monitor the experiment, not just watch a dashboard passively) and a genuine (if bounded) risk of their own, since deliberately injecting a failure is inherently a controlled but real risk; the right calibration is reserving this for releases where the failure mode being tested is actually plausible and consequential enough to justify the overhead, not running it as a blanket requirement on every routine release regardless of risk.

Situation: I'm incident commander for a production outage. Engineers propose an untested rollback that may restore service quickly but risks losing recent customer writes.

Decision process:

• Rapidly triage and gather facts (3–5 minute window): scope of impact, affected services, number of users, SLOs breached, estimated time-to-fix if we don't rollback.
• Convene a short decision group: primary on-call engineer, service owner (dev), DBA/storage lead, product/ops rep (for customer impact), and a QA/automation engineer if available.
• Frame the decision: Restore availability now (with potential data loss) vs preserve data and take longer to recover.

Key questions to ask about backups and integrity:

• What is the precise window of at-risk writes? (timestamps, sequence IDs)
• Do we have durable backups or write-ahead logs (WAL) that can replay or reconcile those writes?
• Are there consistent snapshots? Can we restore to a point-in-time (PITR) that includes recent writes?
• If we rollback code/config, which writes will be lost vs partially applied (idempotency)?
• What verification steps confirm data consistency after rollback? How long do they take?
• What are the legal/financial/customer impacts of losing that data (billing, transactions)?

How I weigh time-to-recovery vs data loss:

• Quantify both sides: estimate MTR (mean time to recover) with a safe fix versus rollback time and expected data loss magnitude (number of users, dollar impact, regulatory risk).
• Prioritize preserving correctness for high-integrity domains (payments, orders). For low-risk telemetry or caches, favor fast rollback.
• Consider SLOs and error budget: if continued degradation violates critical SLOs and causes major customer impact, faster recovery may be prioritized—but not at the expense of irreversible critical data loss.
• Use a decision matrix: Impact severity × recoverability × confidence in rollback. Favor rollback only when data loss is small/acceptable or can be recovered/reconciled reliably.

Temporary mitigations if we choose not to rollback:

• Implement feature flags or traffic routing to divert new writes away from the risky subsystem (read-only mode, degrade gracefully).
• Apply targeted throttling, circuit breakers, or rate limits to reduce new writes and buy time.
• Patch or hotfix to mitigate the root cause without full rollback (config change, DB index adjustment).
• Put a quick validated mitigation in front (API layer validation, queueing writes to durable buffer for later replay).
• Communicate: notify stakeholders and customers with expected timelines and mitigation steps; open a follow-up plan for reconciliation.

If we do rollback:

• Require a short checklist: test rollback in a staging copy, snapshot current state (PITR/WAL), freeze writes or put system in limited mode during rollback, run integrity checks post-rollback, and prepare replay plan for any lost writes.
• Ensure clear owner for reconciliation and customer remediation.

Result orientation:

• Make a timeboxed decision, execute with safety controls (snapshots, monitoring, canary), and document rationale for post-incident review.

Direct answer

Design the readiness review as a two-stage gate with a fixed artifact checklist, not a meeting where whoever shows up decides. A soft gate 24 to 72 hours before launch catches problems while there's still time to fix them; a final gate at T-minus-1-hour is a quick go/no-go confirmation, not a re-review. Every required artifact lives on the launch ticket, and the rollout itself is gated automatically by live metrics, not just by the humans who approved it beforehand.

Structured elaboration

1. Participants. Product manager (scope and business risk), engineering lead (implementation readiness), site reliability engineer or SRE lead (reliability, capacity, deployment plan), a performance/QA engineer (load test sign-off), security/compliance, a release manager (owns the gating decision), an on-call representative, and whoever owns the relevant metrics.
2. Required artifacts, all attached to the review ticket before the soft gate: a load-test report (workload model, p50/p95/p99 latency, the 50th/95th/99th-percentile response times, under expected load plus a burst multiplier); a capacity plan with autoscaling headroom; SLO (service-level objective) projections showing expected error-budget burn during rollout; tested rollback runbooks; a monitoring and alerting checklist; and a security/dependency sign-off.
3. Decision criteria. All must hold to proceed: load test stays under the latency threshold at the target burst multiplier; projected error-budget burn stays within an acceptable window for the first 24 to 72 hours; rollback has actually been dry-run, not just documented; on-call is staffed; no open critical security items.
4. Gating mechanics. Soft gate: sign-off from PM, engineering lead, and SRE based on the artifacts; any failing item gets a remediation owner and a re-check date, it does not get waived. Final gate: release manager, SRE, and on-call confirm nothing has changed and the canary plan is ready.
5. Rollout. Phased canary (for example 1% then 10% then 50% then 100% of traffic) with automated metric gates between each stage: if error rate, latency, or error-budget burn crosses a defined threshold during a stage, the rollout pauses and rolls back automatically rather than waiting for a human to notice.
6. Scaling the process. Enforce the artifact checklist in continuous integration and continuous delivery (CI/CD: the pipeline that builds, tests, and deploys changes) so a launch literally cannot reach the review ticket without its artifacts attached, which is what lets this scale to many launches without the review board becoming a bottleneck.

Worked example

A team plans to launch a new checkout flow to 500,000 daily users. The load test targets 3x expected peak: if peak is 200 requests per second (RPS), the test must hold p99 latency under the agreed threshold at 600 RPS. Suppose the SLO is 99.9% success and the SLA committed to customers is 99.5% monthly, giving a monthly error budget for the SLA of
$(1 - 0.995) \times 30 \times 24 \times 60 = 216 \text{ minutes/month of allowed degraded time}.$
The canary plan ramps 1% then 10% then 50% then 100%, each stage held for 30 minutes with an automated gate. If the 1% stage (roughly 5,000 users at 500,000 total) shows a projected full-rollout error-budget burn rate that would exhaust the 216-minute monthly budget in under 10 days at that error rate, the gate should hold or roll back automatically, because that burn rate is unsustainable for a whole month even though 1% of traffic alone caused only a small absolute number of errors.

Trade-offs and pitfalls

A strict gate slows time to market; that's the intended trade, not a flaw, but it only stays worth it if the gate is actually catching real risk rather than becoming ceremony. The most common failure mode is a rollback plan that was written but never dry-run: it looks complete on the checklist and fails the first time it's actually needed. Canary stages held too briefly won't surface slow-burning issues (memory growth, queue backlog) that only show up after sustained load, so stage duration matters as much as stage size.

Direct answer

The mechanism is the same whether or not you are formally accountable: name the gap in terms stakeholders already care about, build the smallest working proof that closes it, and let the proof, not the pitch, do the persuading.

Structured elaboration

• Spot the gap from recurring pain, not from what looks technically interesting. Teams complaining about the same unreliable output repeatedly is a stronger signal than an architecture you personally find suboptimal.
• Get explicit agreement on what "fixed" means before building anything. A concrete reliability or freshness target that the current state visibly fails makes success falsifiable rather than a matter of opinion later.
• Build a lightweight, working version scoped to reproduce the existing output, not a rewrite. It should be directly comparable to what exists today so stakeholders can check the improvement themselves instead of taking your word for it.
• Demo it to the people who will actually depend on it, not just to your manager. Their objections at that stage are cheap to fix; objections after rollout are not.
• Instrument it before cutover. Monitoring and comparison tests give you, and them, a way to catch regressions instead of relying on someone noticing a bad number days later.

Worked example

An ingestion pipeline is a set of unowned, ad hoc scripts, and downstream teams complain about late, inconsistent reports. You do not own the pipeline, so you bring the affected data-consuming teams into a short session and agree on the criteria that matter to them (a fixed refresh window, no missed runs) rather than the architecture you would personally prefer. You build a small parallel pipeline that reproduces the existing reports on a fixed schedule, with automated tests comparing its output against the current one row by row, and demo it against real data rather than a slide deck. Once it visibly matches or beats the current reports on the criteria the teams themselves picked, you propose a phased cutover with monitoring, and the pattern becomes a template other teams reuse rather than something you have to keep re-selling. The honest expectation is a real cutover period with a few reconciliation mismatches to chase down, not a clean instant swap; proving the direction is right and executing a flawless migration are two different jobs.

Trade-offs and pitfalls

Tailoring the pitch matters: the same proposal has to land differently with an executive who cares about risk and cost and an engineer who cares about whether the new system is actually easier to operate day to day, and a demo built for only one of those audiences stalls with the other. The common failure is skipping the agreement step and building the thing you think is right first; even a technically superior replacement gets resisted if the team was not part of defining what "better" means. The other is treating the pilot's success as permission to skip instrumentation on the real cutover, which is exactly when regressions are most likely and hardest to notice.

The four standard DR tiers form a spectrum from cheapest-and-slowest to most-expensive-and-fastest, and each one trades infrastructure spend for recovery speed (RTO, recovery time objective: how long restoring service takes) and data freshness (RPO, recovery point objective: how much data, measured in time, you could lose): backup-and-restore keeps only backups running, pilot light keeps a minimal always-on core, warm standby keeps a scaled-down full copy running, and active-active multi-site keeps a full copy running and serving live traffic.

Comparing the four tiers

The RTO/RPO ranges above are the typical shape of the trade-off, not a fixed number for any specific system: the exact figures depend on data volume, automation maturity, and how the replication is actually implemented within each tier.

Worked example: a budget-constrained startup

A mid-sized SaaS with a fixed infrastructure budget doesn't have to pick one tier for the whole system; the standard move is to mix tiers by criticality. Say the product has three logical components: authentication/billing (must never meaningfully go down, since it blocks every paying customer from doing anything), the core application (needs to come back reasonably fast but a short outage is tolerable), and internal admin tooling (only the ops team notices if it's down for a few hours).

A budget-conscious allocation: active-active for auth/billing (the one component where the cost premium is justified because its outage blocks revenue entirely, and it's usually small enough in infrastructure footprint that duplicating it fully is affordable), pilot light for the core application (keep the database replicated continuously so RPO stays low, but only spin up the app-server fleet in DR when actually needed, since that's the majority of the compute cost), and backup-and-restore for admin tooling (cheapest tier, acceptable because nobody customer-facing is blocked by it being down for hours). This gets the highest-blast-radius component the fastest recovery while keeping the overall DR bill proportional to what each component actually costs the business if it's down, instead of buying active-active everywhere by default.

Trade-offs and pitfalls

The most expensive mistake in this space isn't picking the "wrong" tier, it's picking a tier and never testing failover into it: a pilot-light setup that's never actually been promoted to full capacity under load is a theoretical RTO, not a real one, and the first real DR event is a bad time to discover the app layer doesn't actually scale up cleanly from zero. A related pitfall is under-provisioning a warm standby's capacity: "scaled down" often means it can absorb DR traffic at reduced performance, and teams sometimes forget to validate that the scaled-down size can actually handle 100% of production load once promoted, not just serve health checks. Finally, active-active's real cost isn't just the duplicate infrastructure line item, it's the ongoing engineering cost of keeping a multi-writer data model correct, which is easy to underestimate when comparing tiers purely on an RTO/RPO/dollar table.

Direct answer

Split retention into hot (raw, fast storage, days), warm (compressed, still full resolution, weeks), cold (downsampled and heavily sampled, up to a year), and archive (compliance-only raw extracts, years, cheapest storage). Alerting stays correct because every alert either reads from the hot/warm tiers where it needs full resolution, or is redesigned to work on the statistical properties (percentiles, error counts) that downsampling preserves. Compliance stays correct because the raw bytes a regulator might ask for are pulled into a separate, signed archive path that bypasses the operational downsampling entirely.

Tiering policy

Downsampling rule for metrics: never discard raw points, only aggregate. Roll up 10s scrapes into 1-minute buckets storing count, sum, min, max, and pre-computed percentile buckets (histogram, not just an average), so alerting on p95/p99 still works post-downsample. Averaging away the histogram is the single most common mistake here: an average of averages silently breaks p99-based alerts.

Downsampling rule for logs: keep 100% of error/warn-level events forever within the cold window; sample info/debug logs at a low, deterministic rate (hash on trace ID) so a given request's logs are either fully kept or fully dropped, never split. That preserves the ability to reconstruct a whole request's log trail if it was selected.

Keeping alerting correct across tiers

• Alert rules that fire on absolute thresholds (error count, latency p99) are pinned to run against hot/warm data only, since that is where fidelity is guaranteed.
• Longer-window alerts (weekly seasonality, slow burn SLO budget) run against cold-tier rollups, but only because the rollup step preserved histograms rather than plain averages, so the alert's percentile math still holds.
• Backtest every downsampling change: replay the last 90 days of raw data through the proposed rollup and confirm the alert would have fired at the same times it fired against raw data, before shipping the rollup config.

Keeping compliance correct across tiers

• Compliance retention is a separate write path, not a side effect of the operational tiers. At ingest, a policy tags records matching compliance scope (PII, regulated data classes, specific tenants); those records get a signed, immutable copy written to the archive tier at full fidelity, independent of what the operational hot/warm/cold pipeline does to the rest of the stream.
• The operational tiers are free to downsample or expire on their own schedule because the compliance obligation is already satisfied by the separate archive copy.
• Archive retrieval is on-demand (async restore), so day-to-day cost stays low, but an auditor's request for raw data from 3 years ago is answerable.

Worked example

Assume a service ingests $G = 500$ GB/day of combined logs and metrics, split 30% metrics / 70% logs, and the storage budget is a hot-tier unit price of $X = \$23\text{/TB-month}$ (a concrete stand-in for the given $X). Illustrative relative unit prices for the other tiers, typical of SSD-backed vs. compressed-object vs. archival storage:

Tier volumes (hot: 7 days raw; warm: next 23 days at 3:1 compression; cold: next 335 days, metrics down 10:1 and logs sampled to 1%; archive: a 2%-of-ingest compliance subset, retained 7 years = 2555 days, compressed 4:1):

Monthly cost:

Compare that to the naive policy of keeping everything raw, at hot-tier price, for the full 7-year compliance window:

Tiering plus targeted downsampling cuts the steady-state storage bill by roughly 99.5% relative to keeping everything at hot-tier fidelity for the retention window, while still satisfying a 7-year compliance obligation and keeping alerting on full-fidelity data for the first month.

flowchart LR
  Ingest[Raw Ingest] --> Hot[Hot 0-7d full res]
  Hot --> Warm[Warm 8-30d compressed 3x]
  Warm --> Cold[Cold 31-365d downsampled 10x, logs 1%]
  Cold --> Archive[Archive 365d+ compliance subset, 4x compressed]
  Hot --> AlertEval[Alert Evaluation]
  Warm --> AlertEval
  Cold --> AlertEval
  Archive --> Restore[On-demand Restore]
  Restore --> AlertEval
  Ingest --> Vault[Signed Compliance Vault]
  Vault --> Archive

Trade-offs and pitfalls

• Downsampling metrics to plain averages instead of histograms is the classic mistake: it looks fine until a p99-based SLO alert goes silent because the percentile can no longer be reconstructed from the rollup.
• Deterministic (hash-based) log sampling beats random sampling because it keeps a request's full log trail together instead of showing half a trace.
• Treating compliance retention as "whatever the operational tiers happen to still have" instead of a separate write path is a common design flaw: an operational cost-cutting change (shortening the cold window) can silently create a compliance gap unless the archive copy is decoupled from it.
• Archive restore latency (minutes to hours, depending on backend) is a real trade-off against cost; if legal holds require faster turnaround, that changes the archive tier choice and its price multiplier.
• Budget monitoring has to be proactive: alert on forecasted spend crossing the budget before the bill arrives, not after, since tier transitions are usually async and lag actual ingest growth.

Direct answer

I'd run the business impact analysis (BIA), the exercise that quantifies what an outage costs each business function and sets its recovery targets, as a structured set of stakeholder interviews mapped against an inventory of the application portfolio, not a survey blast. For each application I'd talk to the business process owner (what breaks and how fast the harm grows), a downstream consumer of that application (what breaks for them), and a technical lead (what this application actually depends on, so the impact can be traced through the chain). The finished output is a BIA register: one row per application with its criticality tier, tolerance window, business-side recovery time objective (RTO, the target time to get a function working again) and recovery point objective (RPO, how much recent data the function can afford to lose, measured in time), and its upstream and downstream dependencies, signed off by the business owner.

Structured elaboration

1. Scope and inventory. Build the list of in-scope applications from the business side (which processes exist) rather than the IT asset list, so nothing gets excluded just because it lacks a system owner in the configuration management database (CMDB), the IT system of record for tracked assets.

2. Stakeholder interviews. Two distinct conversations per application, deliberately kept separate so the technical lead can't quietly set the business tolerance:

• With the business/process owner: "If this is unavailable for 1 hour, 4 hours, 24 hours, 3 days, what actually happens? Who notices, what's the financial or legal consequence, is there a seasonal or peak period where this gets worse, and is there a manual workaround?"
• With a downstream consuming team: "What do you depend on this for, and does your own tolerance window get eaten by the time this stays down?" This surfaces cascading impact that the primary owner often doesn't see.
• With the technical/ops lead: "What does this depend on upstream (data, other systems, third parties, specific teams) and who would need to be involved to bring it back?" This is a dependency map, not a design conversation; the goal is knowing what's connected, not deciding how to recover it.

3. Scoring and tiering. Combine a quantitative score (revenue or cost impact per elapsed-time band, contractual penalties) with a qualitative one (regulatory exposure, reputational harm, customer trust) using a shared rubric applied the same way across every interview, specifically to counter the fact that every owner tends to describe their own system as critical.

4. Output. A BIA register document: application name, business owner, criticality tier, maximum tolerable period of disruption (MTPD), target RTO/RPO, upstream and downstream dependencies, and the date of last review. This becomes the input to recovery sequencing and to the exercise program's scope.

Worked example

Three systems from a recent BIA at a mid-size company, showing why the interview-based approach catches things a pure system-criticality guess would miss:

The HR and analytics examples are the reason the downstream-consumer interview is a separate step: the primary owner of both systems undersold their own criticality, and only the consuming team surfaced the real constraint.

Trade-offs and pitfalls

Interview-based BIAs are vulnerable to self-serving bias (every owner wants to be Tier 1 to guarantee resources), which is why a shared, applied-consistently rubric matters more than any individual interview. They're also vulnerable to missing cascading dependencies if you only interview the primary owner and skip the downstream-consumer conversation, as the HR and analytics examples show. A separate trap specific to a technical audience: it's tempting to let the technical-lead interview drift into "how would we actually achieve that RTO," which turns a business-impact exercise into a premature architecture discussion. Keep that conversation to what the system depends on, and leave the how-to-recover-it decision for the technical team afterward, informed by the tier the BIA assigned.

Direct answer

The fix has to work at both layers: the load balancer should bring a recovered instance back into rotation gradually (slow-start / weight ramp) rather than at full weight immediately, and clients should retry with jittered exponential backoff rather than reconnecting in lockstep the moment they see the instance healthy again. Neither alone is sufficient: LB-side ramping protects against a coordinated flood, but internal callers that bypass client backoff will still hit the instance hard once its weight is nonzero, so admission control (a hard per-instance connection/request cap) is the backstop that holds regardless of what clients do.

Load balancer mitigations

• Slow-start / weighted ramp-up: on transition to healthy, start the instance's LB weight near zero and increase it on a schedule (linear or exponential) instead of jumping to full share immediately.
• Health-check hysteresis: require several consecutive successful probes, not one, before even starting the ramp, so a flapping instance doesn't repeatedly trigger fresh thundering-herd events.
• Hard admission caps: independent of weight, enforce a per-instance connection/request ceiling at the LB so the ramp schedule isn't the only thing standing between the instance and overload.
• Sticky-routing caution: if session affinity is in use, avoid re-establishing it fully during ramp-up, since affinity can concentrate a disproportionate share of long-lived clients onto a still-warming instance.

Application-level mitigations

• Jittered exponential backoff on clients: spread reconnect attempts in time instead of retrying immediately or on a fixed interval that resynchronizes across clients.
• Admission queueing with a bound: the instance itself accepts up to a limit and returns 429/503 with Retry-After beyond that, rather than accepting everything and falling over.
• Circuit breakers upstream of the instance: if error/latency crosses a threshold during ramp-up, upstream callers back off automatically instead of continuing to hammer it.

Worked example

Suppose the recovered instance's established safe steady-state capacity is $C = 5{,}000$ RPS, and the pent-up demand specifically targeting it (clients that were failing over away from it and are now retrying) is $D = 50{,}000$ RPS, 10x its safe capacity. The LB ramps its weight starting at $w_0 = 1\%$, doubling every 10 seconds:

By t=40s the instance is already running at its full safe capacity even though the LB weight is still far below 100%; from that point on, the hard admission cap (not the weight ramp) is what's actually protecting it, and the ramp continuing upward just determines when the instance starts absorbing a "fair" share once the herd's backlog (D) has drained through the rest of the pool. This is the concrete reason both mechanisms are needed: the ramp controls the early window when D is far above C, and the hard cap controls everything after, once weight alone would otherwise overshoot.

Trade-offs and pitfalls

• Slow-start delays how quickly the pool's overall capacity recovers, which matters if the rest of the pool is itself under strain; the ramp duration is a real trade-off between protecting the recovering instance and relieving the rest of the fleet faster.
• Implementing only client-side backoff misses internal service-to-service callers that don't go through the same retry library; implementing only LB-side ramping misses the fact that a fixed weight schedule can still be overwhelmed if pent-up demand is large enough relative to the ramp rate, which is exactly why the hard admission cap has to exist independent of the ramp.
• A common failure in postmortems is tuning the ramp curve without ever computing whether the ramp rate can plausibly outrun realistic pent-up demand (as in the worked example above); a ramp that's too slow relative to D just delays the herd, and one that's too fast relative to C doesn't protect the instance at all.