Netflix Senior Site Reliability Engineer Interview Preparation Guide

Direct answer

In the meeting, redirect from "who" to "what" without dismissing the executives' urgency. Acknowledge that wanting accountability is a fair instinct after real impact, commit to a concrete, dated process for finding cause, and decline to name individuals in that room, because a name given under that kind of pressure becomes a verdict, not a finding.

The move: separate the two questions being asked at once

1. Acknowledge the pressure before redirecting it. Dismissing urgency ("let's not point fingers") reads as evasive right after real customer impact. Naming it as legitimate first earns you room to redirect.
2. Separate the two questions actually in the room: "are we stable right now" and "whose fault is this." Answer the first live, and say explicitly that the second is being deliberately parked, not avoided.
3. Commit to something concrete and dated, not a vague "we'll look into it." A named review with a defined output and a date is a commitment people can hold you to; a soft promise is not.
4. State the ground rule for that review out loud, in the room, while it's still tense. Contributing factors and process gaps, not individuals, is what will come back. Saying this while the executives are present makes it a commitment they witnessed, not a private policy you can quietly walk back.
5. Follow through in substance. The review has to actually surface system and process gaps rather than a person, or the promise made in that meeting becomes evidence of bad faith the next time something breaks.

Worked example

After a major outage, executives in the room want to know who's responsible, with the implication that someone should be let go. You acknowledge that wanting accountability after this kind of impact is completely reasonable, then separate the two questions: you answer "are we stable" directly with current mitigation status, and you name the second question, responsibility, as something you're deliberately not answering live because it needs evidence, not a reaction under pressure. You commit to a rapid review with a firm date and a defined output: a timeline, contributing factors, and remediation owners. You state the ground rule in the room: the review looks at system and process gaps, not individuals. The follow-up review actually produces that, and the remediation backlog with owners and dates is what goes back to the executives, not a name.

Trade-offs and pitfalls

Protecting "blameless" as a phrase in the room while privately identifying a scapegoat afterward is worse for trust than never promising blamelessness at all, because it's discovered eventually and reads as deliberate deception. Being so procedural and passive in the room that it sounds like dodging accountability altogether undercuts the point, you do need to commit to something concrete live, not just "we'll circle back." When there genuinely was individual negligence rather than a process gap, blameless doesn't mean no consequence, it means the review finds that out through evidence gathered afterward, not through a name given under live pressure in a tense meeting.

Direct answer

An SLI is the measured signal (e.g. the percentage of requests that were fast and successful), an SLO is the internal target for that signal over a time window (e.g. 99.9% of requests succeed under 300ms over a rolling 30 days), and an SLA is the external, often contractual promise built on top of an SLO, usually with margin, and real consequences (service credits, penalties) if it's breached. In short: the SLI measures, the SLO targets, the SLA promises with a business wrapper attached.

Defining each one concretely, for an e-commerce checkout service

Explained for a non-technical stakeholder: the SLO is the bar the engineering team holds itself to internally so problems get caught and fixed before they become customer-visible; the SLA is the, usually more forgiving, bar the business is willing to be held to externally, on paper, with money attached if it's missed. The gap between the two is deliberate headroom, not sloppiness.

Worked example: computing the error budget

The error budget is the amount of allowed failure baked into the SLO. It's what lets a team ship changes at all instead of freezing forever.

$$\text{error budget} = (1 - \text{SLO target})$$

For the 99.9% SLO above:

$$\text{error budget} = 1 - 0.999 = 0.001 = 0.1\%$$

Over the 30-day measurement window (30 days equals 2,592,000 seconds):

$$0.001 \times 2{,}592{,}000\text{s} = 2{,}592\text{s} \approx 43.2\text{ minutes}$$

So this service is allowed about 43 minutes of budget-consuming failure (however that failure is defined: downtime, over-300ms responses, and so on) across a 30-day window before the SLO itself is breached. If checkout handles, say, 1,000,000 requests over that same window, the equivalent request-based budget is:

$$0.001 \times 1{,}000{,}000 = 1{,}000 \text{ requests allowed to violate the SLI}$$

Both framings, time-based and request-based, describe the same budget; which one is more useful depends on whether the SLI is availability-style (was the service up) or ratio-style (what fraction of requests succeeded).

Trade-offs and pitfalls

• Confusing SLO and SLA in conversation causes real problems: teams sometimes design their alerting and release-gating around the SLA (the looser, contractual number) instead of the SLO (the tighter, internal number), which means by the time anyone notices, the team is already close to breaching the external promise with no margin left to react.
• An SLO with no error-budget policy attached is just a number on a dashboard; the value comes from what happens when the budget is nearly exhausted (freeze risky launches, redirect engineering time to reliability work), not from the target itself.
• Picking an SLI that doesn't reflect real user experience (e.g. "server process is running" instead of "requests succeed within an acceptable time") gives a green dashboard while users are still unhappy. The SLI has to be as close to what the user actually experiences as the team can measure.

Direct answer
Chaos testing fits into release validation as an OPTIONAL, carefully-scoped extra layer specifically for high-stakes releases: deliberately injecting a controlled failure (a dependency timeout, a node loss) during the canary phase to confirm the new version degrades gracefully under real adversity, not just under the ordinary traffic conditions a normal canary window happens to see.

Structured elaboration

• What faults to inject, and why during canary specifically: a simulated downstream-dependency timeout or error, a simulated node/pod loss, or injected network latency, run against the CANARY specifically (not the full production fleet) so any bad outcome from the chaos experiment itself is contained to the same small blast radius the canary was already limiting exposure to.
• Why canary is the right phase for this: a normal canary window validates behavior under whatever REAL conditions happen to occur during that window, which might not include a dependency hiccup at all; deliberately injecting one during canary tests a failure mode you might not otherwise observe before the release reaches full production traffic, catching a regression in error-handling or fallback logic specifically, not just in the happy path.
• Safety controls: a strict, pre-defined blast-radius limit on the experiment (only the canary's already-small traffic slice, never full production), an explicit kill switch to abort the chaos experiment immediately if it's causing more disruption than intended, and running it only during a window with adequate on-call attention, never as a fully unattended, unmonitored background process on a live release candidate.
• Feeding results back into rollout gates: if the canary handles the injected fault gracefully (a proper fallback, no cascading failure, error rates staying within an acceptable degraded-but-controlled band), that's positive evidence feeding into the normal promote/rollback decision alongside the standard metrics; if it reveals a genuine gap (the new version doesn't handle the injected failure as gracefully as the current stable version does), that's grounds to halt promotion and fix the gap, treated as seriously as a metric-based canary failure would be.

Worked example
A canary at 5% traffic has a chaos experiment injecting a simulated 2-second timeout on calls to a specific downstream dependency for a bounded 5-minute window. The canary's fallback logic (serving a cached, slightly stale response rather than erroring) engages correctly, and error rate stays within the acceptable band throughout, positive evidence added to the normal canary-analysis decision. On a different release, the same experiment reveals the canary's NEW error-handling path doesn't actually engage the fallback at all under this specific failure and instead returns a hard error to users, correctly halting that release's promotion despite the release having looked otherwise clean under normal traffic conditions.

Trade-offs and pitfalls
Chaos experiments add real operational overhead (someone needs to design, run, and monitor the experiment, not just watch a dashboard passively) and a genuine (if bounded) risk of their own, since deliberately injecting a failure is inherently a controlled but real risk; the right calibration is reserving this for releases where the failure mode being tested is actually plausible and consequential enough to justify the overhead, not running it as a blanket requirement on every routine release regardless of risk.

Decision matrix (score 1–5, higher is better). We weight SLA-critical uptime more (SLA weight shown).

Criteria (weight) — Hotfix (5-min downtime) — Feature-flag mitigation (no downtime)

• User impact (functional correctness) (w=1): Hotfix 5 (restores full functionality) — FF 3 (degraded behavior)
• Availability / downtime (w=3): Hotfix 1 (5-min outage) — FF 5 (no outage)
• Duration to implement (w=1): Hotfix 2 (deploy + verification ~10–20 min total ops) — FF 4 (flip flag fast)
• Rollbackability / safety (w=1): Hotfix 3 (can rollback but with downtime risk) — FF 5 (toggle off instantly)
• Regulatory / compliance constraints (w=2): Hotfix 4 (audit trail via planned maintenance) — FF 4 (must ensure degraded path meets regs)
• Contractual SLAs / penalties (w=3): Hotfix 1 (breaches uptime SLA) — FF 5 (keeps SLA)
• Monitoring & verification (w=1): Hotfix 3 (post-deploy checks needed) — FF 4 (easy to monitor flag behavior)

Weighted score calculation (sum(weight * score)):
Hotfix = 15 + 31 + 12 + 13 + 24 + 31 + 13 = 5 +3 +2 +3 +8 +3 +3 = 27
Feature-flag = 13 + 35 + 14 + 15 + 24 + 35 + 14 = 3 +15 +4 +5 +8 +15 +4 = 54

Decision: For a large enterprise customer with strict uptime SLAs, enable the feature-flag mitigation immediately.

Rationale:

• The customer’s contractual SLA and uptime penalties (highest weights) make any downtime unacceptable; feature-flag preserves availability and is reversible instantly.
• Risk is managed: degraded functionality is preferable to SLA breach. Ensure degraded path is safe and compliant before enabling.
  Action plan:

1. Validate mitigation in staging and smoke tests; run quick canary in prod if possible.
2. Enable flag in safe subset, monitor key metrics and errors (alerts for user-facing errors).
3. Communicate to customer and support (status page, expected user impact).
4. Schedule a fully tested hotfix for the earliest maintenance window (or when error budget allows) with rollback runbook and automated verification.
5. Postmortem and update runbooks to avoid recurrence.

Direct answer

Design the readiness review as a two-stage gate with a fixed artifact checklist, not a meeting where whoever shows up decides. A soft gate 24 to 72 hours before launch catches problems while there's still time to fix them; a final gate at T-minus-1-hour is a quick go/no-go confirmation, not a re-review. Every required artifact lives on the launch ticket, and the rollout itself is gated automatically by live metrics, not just by the humans who approved it beforehand.

Structured elaboration

1. Participants. Product manager (scope and business risk), engineering lead (implementation readiness), site reliability engineer or SRE lead (reliability, capacity, deployment plan), a performance/QA engineer (load test sign-off), security/compliance, a release manager (owns the gating decision), an on-call representative, and whoever owns the relevant metrics.
2. Required artifacts, all attached to the review ticket before the soft gate: a load-test report (workload model, p50/p95/p99 latency, the 50th/95th/99th-percentile response times, under expected load plus a burst multiplier); a capacity plan with autoscaling headroom; SLO (service-level objective) projections showing expected error-budget burn during rollout; tested rollback runbooks; a monitoring and alerting checklist; and a security/dependency sign-off.
3. Decision criteria. All must hold to proceed: load test stays under the latency threshold at the target burst multiplier; projected error-budget burn stays within an acceptable window for the first 24 to 72 hours; rollback has actually been dry-run, not just documented; on-call is staffed; no open critical security items.
4. Gating mechanics. Soft gate: sign-off from PM, engineering lead, and SRE based on the artifacts; any failing item gets a remediation owner and a re-check date, it does not get waived. Final gate: release manager, SRE, and on-call confirm nothing has changed and the canary plan is ready.
5. Rollout. Phased canary (for example 1% then 10% then 50% then 100% of traffic) with automated metric gates between each stage: if error rate, latency, or error-budget burn crosses a defined threshold during a stage, the rollout pauses and rolls back automatically rather than waiting for a human to notice.
6. Scaling the process. Enforce the artifact checklist in continuous integration and continuous delivery (CI/CD: the pipeline that builds, tests, and deploys changes) so a launch literally cannot reach the review ticket without its artifacts attached, which is what lets this scale to many launches without the review board becoming a bottleneck.

Worked example

A team plans to launch a new checkout flow to 500,000 daily users. The load test targets 3x expected peak: if peak is 200 requests per second (RPS), the test must hold p99 latency under the agreed threshold at 600 RPS. Suppose the SLO is 99.9% success and the SLA committed to customers is 99.5% monthly, giving a monthly error budget for the SLA of
$(1 - 0.995) \times 30 \times 24 \times 60 = 216 \text{ minutes/month of allowed degraded time}.$
The canary plan ramps 1% then 10% then 50% then 100%, each stage held for 30 minutes with an automated gate. If the 1% stage (roughly 5,000 users at 500,000 total) shows a projected full-rollout error-budget burn rate that would exhaust the 216-minute monthly budget in under 10 days at that error rate, the gate should hold or roll back automatically, because that burn rate is unsustainable for a whole month even though 1% of traffic alone caused only a small absolute number of errors.

Trade-offs and pitfalls

A strict gate slows time to market; that's the intended trade, not a flaw, but it only stays worth it if the gate is actually catching real risk rather than becoming ceremony. The most common failure mode is a rollback plan that was written but never dry-run: it looks complete on the checklist and fails the first time it's actually needed. Canary stages held too briefly won't surface slow-burning issues (memory growth, queue backlog) that only show up after sustained load, so stage duration matters as much as stage size.

Direct answer

The mechanism is the same whether or not you are formally accountable: name the gap in terms stakeholders already care about, build the smallest working proof that closes it, and let the proof, not the pitch, do the persuading.

Structured elaboration

• Spot the gap from recurring pain, not from what looks technically interesting. Teams complaining about the same unreliable output repeatedly is a stronger signal than an architecture you personally find suboptimal.
• Get explicit agreement on what "fixed" means before building anything. A concrete reliability or freshness target that the current state visibly fails makes success falsifiable rather than a matter of opinion later.
• Build a lightweight, working version scoped to reproduce the existing output, not a rewrite. It should be directly comparable to what exists today so stakeholders can check the improvement themselves instead of taking your word for it.
• Demo it to the people who will actually depend on it, not just to your manager. Their objections at that stage are cheap to fix; objections after rollout are not.
• Instrument it before cutover. Monitoring and comparison tests give you, and them, a way to catch regressions instead of relying on someone noticing a bad number days later.

Worked example

An ingestion pipeline is a set of unowned, ad hoc scripts, and downstream teams complain about late, inconsistent reports. You do not own the pipeline, so you bring the affected data-consuming teams into a short session and agree on the criteria that matter to them (a fixed refresh window, no missed runs) rather than the architecture you would personally prefer. You build a small parallel pipeline that reproduces the existing reports on a fixed schedule, with automated tests comparing its output against the current one row by row, and demo it against real data rather than a slide deck. Once it visibly matches or beats the current reports on the criteria the teams themselves picked, you propose a phased cutover with monitoring, and the pattern becomes a template other teams reuse rather than something you have to keep re-selling. The honest expectation is a real cutover period with a few reconciliation mismatches to chase down, not a clean instant swap; proving the direction is right and executing a flawless migration are two different jobs.

Trade-offs and pitfalls

Tailoring the pitch matters: the same proposal has to land differently with an executive who cares about risk and cost and an engineer who cares about whether the new system is actually easier to operate day to day, and a demo built for only one of those audiences stalls with the other. The common failure is skipping the agreement step and building the thing you think is right first; even a technically superior replacement gets resisted if the team was not part of defining what "better" means. The other is treating the pilot's success as permission to skip instrumentation on the real cutover, which is exactly when regressions are most likely and hardest to notice.

Direct answer

Keep the system active-active for everything except the operations that must be strongly consistent, and handle those with one of three narrower mechanisms: give the specific entity a single owning region, run a lightweight consensus round only for that transaction, or accept the write optimistically everywhere and reconcile afterward. The right choice depends on how often the strongly-consistent operation happens and how expensive a wrong answer is.

Three alternatives

Worked example

A transfer between two accounts whose home regions differ (say, one in us-east, one in eu-west) is the interesting case. With single-leader-per-entity alone, this transfer needs the two home regions to cooperate, which reintroduces the exact cross-region coordination problem you were trying to avoid for the rest of the system, so most designs pair it with per-transaction consensus: only THIS transfer pays a synchronous round trip to lock and debit one account and credit the other, while a same-region transfer (the common case) never leaves its home region.

Trade-offs and pitfalls

Optimistic reconciliation is tempting because it adds zero write-path latency, but for money it should never be the sole line of defense: detecting an overdraft after the fact means the funds were briefly, genuinely double-spent, and the fix (a compensating reversal, possibly with a fee or manual dispute) is an incident, not a routine operation. The senior answer scopes strong consistency as narrowly as possible (only the transfer operation, not the whole account record) rather than making an entire dataset active-passive out of caution, which would give up the latency benefits active-active was chosen for in the first place.

I would first separate disagreement on the outcome from disagreement on the method. Then I would bring the executives into a short decision session with a one-page brief: the business goal, the options, the trade-offs, and the decision needed. I would make decision rights explicit using RACI, which means Responsible, Accountable, Consulted, and Informed. That way, everyone knows who recommends, who decides, and who simply needs to stay informed.

For example, if one sponsor wants speed, another wants cost savings, and a third wants risk reduction, I would ask which metric is the tie-breaker if they conflict. I would propose a shared scorecard with 2 or 3 measures, such as revenue impact, operational risk, and delivery date, then ask the accountable executive to make the final call in writing.

While the debate is happening, I would keep teams executing on work that is not dependent on the unresolved choice, pause only the parts that could be wasted, and communicate a clear interim plan. The goal is to prevent thrash, protect momentum, and get everyone back to one set of success criteria.

For example, on a customer-onboarding automation initiative, three executives disagreed about halfway through: the VP of Engineering wanted to prioritize system reliability given a recent outage, the VP of Finance wanted to prioritize cost savings from reduced manual onboarding labor, and the VP of Risk wanted to prioritize compliance controls given a pending audit. In the decision session, the RACI mapping made the VP of Product the accountable decision-maker, with all three VPs as consulted. The proposed scorecard used three metrics: onboarding error rate (tied to reliability), manual labor hours saved per month (tied to cost), and number of unresolved audit findings (tied to risk). The accountable VP decided that the audit-findings metric was the tie-breaker for this quarter, since the audit deadline was fixed and immovable, while the reliability and cost metrics would be weighted equally starting the following quarter. While that decision was being finalized in writing, the teams kept building the shared onboarding data pipeline, which every option needed regardless of the outcome, and paused only the specific reporting dashboard whose design depended on which metric ultimately won.

The four standard DR tiers form a spectrum from cheapest-and-slowest to most-expensive-and-fastest, and each one trades infrastructure spend for recovery speed (RTO, recovery time objective: how long restoring service takes) and data freshness (RPO, recovery point objective: how much data, measured in time, you could lose): backup-and-restore keeps only backups running, pilot light keeps a minimal always-on core, warm standby keeps a scaled-down full copy running, and active-active multi-site keeps a full copy running and serving live traffic.

Comparing the four tiers

The RTO/RPO ranges above are the typical shape of the trade-off, not a fixed number for any specific system: the exact figures depend on data volume, automation maturity, and how the replication is actually implemented within each tier.

Worked example: a budget-constrained startup

A mid-sized SaaS with a fixed infrastructure budget doesn't have to pick one tier for the whole system; the standard move is to mix tiers by criticality. Say the product has three logical components: authentication/billing (must never meaningfully go down, since it blocks every paying customer from doing anything), the core application (needs to come back reasonably fast but a short outage is tolerable), and internal admin tooling (only the ops team notices if it's down for a few hours).

A budget-conscious allocation: active-active for auth/billing (the one component where the cost premium is justified because its outage blocks revenue entirely, and it's usually small enough in infrastructure footprint that duplicating it fully is affordable), pilot light for the core application (keep the database replicated continuously so RPO stays low, but only spin up the app-server fleet in DR when actually needed, since that's the majority of the compute cost), and backup-and-restore for admin tooling (cheapest tier, acceptable because nobody customer-facing is blocked by it being down for hours). This gets the highest-blast-radius component the fastest recovery while keeping the overall DR bill proportional to what each component actually costs the business if it's down, instead of buying active-active everywhere by default.

Trade-offs and pitfalls

The most expensive mistake in this space isn't picking the "wrong" tier, it's picking a tier and never testing failover into it: a pilot-light setup that's never actually been promoted to full capacity under load is a theoretical RTO, not a real one, and the first real DR event is a bad time to discover the app layer doesn't actually scale up cleanly from zero. A related pitfall is under-provisioning a warm standby's capacity: "scaled down" often means it can absorb DR traffic at reduced performance, and teams sometimes forget to validate that the scaled-down size can actually handle 100% of production load once promoted, not just serve health checks. Finally, active-active's real cost isn't just the duplicate infrastructure line item, it's the ongoing engineering cost of keeping a multi-writer data model correct, which is easy to underestimate when comparing tiers purely on an RTO/RPO/dollar table.

Direct answer

The fix has to work at both layers: the load balancer should bring a recovered instance back into rotation gradually (slow-start / weight ramp) rather than at full weight immediately, and clients should retry with jittered exponential backoff rather than reconnecting in lockstep the moment they see the instance healthy again. Neither alone is sufficient: LB-side ramping protects against a coordinated flood, but internal callers that bypass client backoff will still hit the instance hard once its weight is nonzero, so admission control (a hard per-instance connection/request cap) is the backstop that holds regardless of what clients do.

Load balancer mitigations

• Slow-start / weighted ramp-up: on transition to healthy, start the instance's LB weight near zero and increase it on a schedule (linear or exponential) instead of jumping to full share immediately.
• Health-check hysteresis: require several consecutive successful probes, not one, before even starting the ramp, so a flapping instance doesn't repeatedly trigger fresh thundering-herd events.
• Hard admission caps: independent of weight, enforce a per-instance connection/request ceiling at the LB so the ramp schedule isn't the only thing standing between the instance and overload.
• Sticky-routing caution: if session affinity is in use, avoid re-establishing it fully during ramp-up, since affinity can concentrate a disproportionate share of long-lived clients onto a still-warming instance.

Application-level mitigations

• Jittered exponential backoff on clients: spread reconnect attempts in time instead of retrying immediately or on a fixed interval that resynchronizes across clients.
• Admission queueing with a bound: the instance itself accepts up to a limit and returns 429/503 with Retry-After beyond that, rather than accepting everything and falling over.
• Circuit breakers upstream of the instance: if error/latency crosses a threshold during ramp-up, upstream callers back off automatically instead of continuing to hammer it.

Worked example

Suppose the recovered instance's established safe steady-state capacity is $C = 5{,}000$ RPS, and the pent-up demand specifically targeting it (clients that were failing over away from it and are now retrying) is $D = 50{,}000$ RPS, 10x its safe capacity. The LB ramps its weight starting at $w_0 = 1\%$, doubling every 10 seconds:

$$w(t) = \min\!\left(1,\; w_0 \cdot 2^{t/10}\right), \qquad admitted(t) = \min\big(w(t)\cdot D,\; C\big)$$

By t=40s the instance is already running at its full safe capacity even though the LB weight is still far below 100%; from that point on, the hard admission cap (not the weight ramp) is what's actually protecting it, and the ramp continuing upward just determines when the instance starts absorbing a "fair" share once the herd's backlog (D) has drained through the rest of the pool. This is the concrete reason both mechanisms are needed: the ramp controls the early window when D is far above C, and the hard cap controls everything after, once weight alone would otherwise overshoot.

Trade-offs and pitfalls

• Slow-start delays how quickly the pool's overall capacity recovers, which matters if the rest of the pool is itself under strain; the ramp duration is a real trade-off between protecting the recovering instance and relieving the rest of the fleet faster.
• Implementing only client-side backoff misses internal service-to-service callers that don't go through the same retry library; implementing only LB-side ramping misses the fact that a fixed weight schedule can still be overwhelmed if pent-up demand is large enough relative to the ramp rate, which is exactly why the hard admission cap has to exist independent of the ramp.
• A common failure in postmortems is tuning the ramp curve without ever computing whether the ramp rate can plausibly outrun realistic pent-up demand (as in the worked example above); a ramp that's too slow relative to D just delays the herd, and one that's too fast relative to C doesn't protect the instance at all.