Netflix Staff Site Reliability Engineer Interview Preparation Guide

Start by defining the two models briefly: Centralized Incident Command (CIC) uses a small, trained incident command center (IC) that coordinates detection, triage, decisions and communications. Decentralized Empowered-Responder (DER) gives service teams or on-call engineers authority to act independently, with lightweight coordination.

Centralized Incident Command

• Advantages: consistent decision-making, single source of truth for status/communications, easier cross-service prioritization, preserves exec-level escalation path. Good for high-severity outages affecting many services.
• Risks: bottlenecked decisions, slower local remediation, single-point-of-failure if IC is overloaded or understaffed, reduced domain-context in decisions.
• Scaling: scale by staffing rotating IC shifts, runbooks, playbooks, and tooling (war-rooms, status pages). Beyond certain incident volume/variety, IC becomes overwhelmed.
• Governance: strict roles (commander, scribe, communications), playbooks, simulated drills, SLA for IC response times, auditing and runbook maintenance.

Decentralized Empowered-Responder

• Advantages: fast local remediation, leverages domain expertise, encourages ownership and rapid iteration, reduces coordination overhead for localized issues.
• Risks: inconsistent communications, conflicting mitigations across teams, difficulty prioritizing cross-cutting incidents, harder to present unified postmortems.
• Scaling: requires reliable observability, automated guardrails (circuit breakers, feature flags), standardized alerting taxonomy, and runbook templates to keep coherence as teams grow.
• Governance: clear boundaries of authority, escalation criteria, mandatory post-incident reporting, shared incident metadata standards, and periodic cross-team drills.

When to adopt which (criteria)

• Prefer CIC when incidents are high blast-radius, regulatory-sensitive, cross-domain, or require executive coordination; or when org maturity for decentralized ops is low.
• Prefer DER when services are well-isolated, teams are experienced, ownership culture exists, and fast local fixes reduce customer impact.
• Hybrid: most large enterprises should adopt hybrid: empower responders for low-to-medium severity and local issues while IC activates for multi-service or high-impact incidents. Implement trigger criteria (severity thresholds, affected-service-count, legal/PR risk) that automatically flip into centralized mode.

Operational recommendations

• Define clear severity-to-governance mapping and automated escalation rules.
• Invest in standardized tooling: incident templates, shared dashboards, chatops, and a central incident metadata store.
• Regularly exercise both modes (game days) and refine decision thresholds.
• Measure: mean time to acknowledge, time to mitigate, communication lag, and postmortem quality to tune the balance.

Direct answer

When legal or compliance flags a possible regulatory problem on something about to ship, that flag is new information, not an attack on the project. The first move is to separate the specific risk from the whole feature: find out exactly what triggers the concern, then look for a way to ship everything outside that blast radius (the specific data, users, or markets the flagged concern actually touches) while the risky piece gets handled properly. Treating the flag as either a full block to fight or a formality to route around are both weak answers; the senior move is to make the freeze as small as the actual risk.

Structured elaboration

1. Turn the flag into a scoped, written finding

Ask for the specific clause or regulation, the specific data flow or behavior it applies to, and which markets or user segments are affected. A flag that sounds like 'this violates a regulation' often narrows down to 'this one data field, in these two markets.' Until that scoping happens, nobody can reason about mitigation, they can only argue about the abstract freeze.

2. Sort what's actually blocked from what's just slow

Once scoped, most flags fall into three buckets: genuinely unsafe to ship anywhere (rare, but real, treat it as a hard stop); unsafe in specific markets or for specific data (the common case, often scoped out with a flag or market-level rule); or unsafe as currently designed but fixable with a smaller change than a full freeze (needs a scoped rework, not a blanket delay).

3. Bring a mitigation, not just a constraint

Offer a concrete option: disable the flagged behavior for the affected markets, gate it behind a feature flag (a toggle that turns a piece of functionality on or off without a new deployment), or ship a version that omits the specific data flow while the rest proceeds. This turns the conversation from 'can we go or not' into 'does this mitigation satisfy the concern,' which moves much faster.

4. Get joint, written sign-off before proceeding

Both the business owner and compliance need to agree in writing on what shipped, what did not, the remaining risk, and who owns closing it. This protects everyone if the interpretation is questioned later and prevents the same argument from recurring next release.

5. If a real freeze can't be avoided, negotiate the timeline explicitly

Sometimes there is no safe scoped path and the freeze has to hold for the affected piece. Here the negotiation shifts to: what's the minimum change needed to clear the concern, who is assigned to it, and can the review be fast-tracked with a dedicated reviewer instead of sitting in a general queue. A freeze with a committed, shrinking timeline is a very different conversation from an open-ended one.

Worked example

A team is about to ship a feature that logs a new field for product analytics, and legal flags that collecting that field may violate a data-protection rule in one region. Scoping the flag shows the issue is narrow: one field, one region. Instead of freezing the whole release, the team ships everywhere else immediately, and for the flagged region ships the same feature with that one field's collection disabled behind a config switch. Legal signs off on the scoped version in writing. The team opens a follow-up item, with an owner and a target date, to redesign how that field is collected (for example, aggregating it instead of storing it per user), so the region isn't stuck without the feature indefinitely.

Trade-offs and pitfalls

• Treating every compliance flag as either a full block or a nuisance to route around is the most common mistake here; both extremes erode trust with the compliance function over time.
• Scoped mitigations (flags, market gating, field exclusions) are good short-term tools but can quietly become permanent if nobody owns the follow-up fix. The sign-off should name an owner and a date, not just describe a workaround.
• Escalating past compliance to force a ship date, without addressing the underlying concern, tends to resurface later as a bigger problem: a real violation or a regulator inquiry. Speed gained by skipping the process rarely survives contact with the risk it was protecting against.
• The strongest signal of seniority isn't how fast the team got to yes, it's whether the final decision is something both sides would still defend the same way months later.

Direct answer

Lead with the baseline metric, the specific change you made, the resulting metric with enough of the underlying numbers shown that the improvement is checkable, and the trade-off you knowingly accepted, in that order. The trade-off is not optional detail: naming it, and what you did to monitor it, is what separates a senior answer from a number without context.

Structured elaboration

The four-part shape:

1. Baseline: what was the metric before, and how was it measured?
2. Change: the specific decision, not a list of everything you tried.
3. Result: the new metric, with enough of the underlying numbers shown that the improvement is checkable, not just asserted.
4. Trade-off and monitoring: what got worse or riskier as a direct consequence, and what you put in place to catch it if it went too far.

Common metric families by domain (pick the one that matches your role; the story shape is identical):

Worked example

"A service's average response time was too high under peak load. Baseline: 40% of requests hit a warm cache (5ms), the other 60% missed and hit the database (200ms). Baseline average latency: (40% × 5ms) + (60% × 200ms) = 2ms + 120ms = 122ms. The change: I raised the cache TTL from 30 seconds to 10 minutes, which pushed the effective hit rate to 85%, at the cost of serving data up to 10 minutes stale instead of 30 seconds stale. New average latency: (85% × 5ms) + (15% × 200ms) = 4.25ms + 30ms = 34.25ms. That's a drop from 122ms to 34.25ms, a (122 minus 34.25) divided by 122, roughly 72% reduction. The trade-off: any field that changed within that 10 minute window could be served stale. I mitigated it by adding explicit cache invalidation on writes for the two fields that actually mattered for correctness, account balance and permission level, and left everything else on the longer TTL, plus a staleness alert if invalidation events started failing silently."

Trade-offs and pitfalls

• Never present the "after" number without the baseline; an improvement with no starting point is unfalsifiable and interviewers know it.
• Don't hide the trade-off; claiming a change had zero downside reads as either dishonest or shallow. Every real optimization costs something.
• Match your monitoring to the specific failure mode you introduced; generic "we added logging" is weaker than "we alerted specifically on the thing that could go wrong because of this change."
• Round, checkable numbers you can defend beat impressively precise ones you can't reconstruct if asked.

SLI vs SLO vs SLA (concise):

• SLI (Service Level Indicator): a measured metric representing user experience (e.g., p99 latency, successful request rate).
• SLO (Service Level Objective): an internal target for an SLI over a window (e.g., 99.9% of requests under 200ms).
• SLA (Service Level Agreement): a contractual commitment to customers, often with financial penalties if missed; typically built from SLOs but legally binding.

Concrete SLIs for a globally distributed web API:

1. Latency SLI: p95 end-to-end request latency measured at edge (client-to-edge-to-backend) per region.
   • SLO: p95 < 150 ms per region, measured over 30 days, with 99.9% compliance.
2. Availability SLI: successful request ratio (HTTP 2xx/3xx) measured at the API gateway.
   • SLO: 99.99% successful requests over 30 days (≈4.38 minutes downtime/month).

Operational actions when error budget is exhausted (organization-wide):

• Immediately throttle non-essential work: halt launches, feature rollouts, experiments, and large migrations until budget is restored.
• Shift to reliability-focused work: prioritize incidence remediation, root-cause fixes, capacity increases, rollback risky changes.
• Increase monitoring & alerting sensitivity and run targeted game-days to reproduce/fix.
• Require change freezes and stricter change approvals (e.g., canary percentage limits, staged rollouts).
• Communicate to stakeholders and adjust customer expectations; if SLA risk exists, notify legal/PM and prepare mitigation/compensation plans.
• After recovery, run a blameless postmortem and update SLOs, runbooks, and automation to prevent recurrence.

Direct Answer

Split ingestion by region so no single path crosses a WAN in the hot loop: collectors batch and buffer locally, hand off to a partitioned durable log keyed by series identity, and a stream layer deduplicates and rolls up before anything touches long-term storage. The three levers that make one million points per second tractable are partition count (parallelism), batch size (write amplification), and pre-aggregation (what actually needs to survive at full resolution).

Structured Elaboration

Pipeline topology

flowchart LR
  subgraph REGION["Per-Region Tier (x3)"]
    APP[Service Instances] --> AGENT[Collector Agent]
    AGENT --> WAL[("Local WAL Buffer")]
    WAL --> KAFKA[["Kafka: 32 partitions by tenant+series key"]]
  end
  KAFKA --> SP["Stream Processor: dedup + rollup"]
  SP --> TSDB[("Hot TSDB")]
  SP --> OBJ[("Object Storage: rollups")]

Collector and agent placement
Run a lightweight collection tier per region (behind a regional load balancer, autoscaled in Kubernetes) so no metric point leaves its region before being durably buffered. Cross-region replication happens downstream, at the storage layer, never in the write-critical path, so a WAN blip in one region does not add latency to the other two.

Buffering and batching
Each collector holds a local disk-backed buffer (a WAL, RocksDB-backed queue, or equivalent) so a restart or a downstream stall does not drop in-flight data. Batch by size, not purely by time: a size trigger keeps latency proportional to actual load instead of always waiting out a fixed window.

Message broker and partitioning keys
Use a partitioned durable log (Kafka or equivalent) per region. Partition key: hash of tenant_id + metric_name. This keeps every point for a given series in the same partition, which is what makes per-series ordering and local windowed aggregation possible downstream, at the cost of potential hot partitions for very high-cardinality single tenants.

Deduplication
Give every point an idempotency key (source_id + monotonic sequence number). The stream processor keeps a rolling window of seen keys; anything already seen in that window is a duplicate produced by a retry, not new data.

Backpressure handling
The collector never blocks its callers. When the local buffer approaches capacity, it sheds lowest-priority series first (a stated priority tier, not silent random drop) and raises an explicit metric so the shedding is visible, not a silent gap in a dashboard three weeks later.

Fault tolerance
Replicate the log (replication factor 3) across brokers in multiple availability zones. Collectors are stateless except for the local buffer, so a lost collector instance loses only unflushed buffer content, not history. Stream processors checkpoint offsets so a crash resumes from the last committed point, not from zero.

Pre-aggregation and rollups
Do windowed rollups (count, sum, min, max, and a percentile sketch) in the stream layer before the write to the hot store. Keep raw resolution for a short window (operators debugging an active incident need seconds-level data); roll everything older than that window up to coarser resolution, since almost no dashboard or alert needs one-second granularity on data from an hour ago.

Worked Example

Assume 1,000,000 points/sec split across 3 regions and an average encoded point size of 150 bytes (timestamp, value, and label set after protobuf encoding, a stated design input, not a benchmark). Regional steady load is then:

Plan for regional skew: if one region fails, its traffic can fail over to the nearest healthy region, so size for up to 1.5x the average, 500,000 pts/s = 75 MB/s peak.

Partition count. Choose a conservative per-partition write budget of 10 MB/s (accounts for replication-factor-3 fsync overhead on commodity brokers, a stated assumption, not a vendor benchmark):

Round up to 32 partitions per regional topic: this gives 4x headroom over the peak-throughput floor of 8, so future traffic growth or a temporary partition hot-spot does not require an emergency repartition, and it splits evenly across, say, 8 stream-processor instances at 4 partitions each.

Batch fill time. At the peak-derived floor of 8 partitions, each carries roughly 50 MB/s / 8 = 6.25 MB/s. A 1 MB size-triggered batch fills in:

That is well under a reasonable 500 ms linger cap, so the size trigger (not the timeout) governs flush cadence under normal load, and the timeout only matters for low-traffic partitions.

Deduplication memory. With a 2-minute (120 s) dedup window at the regional steady rate, the number of distinct keys the stream processor must track at once is:

Sizing a Bloom filter (a compact structure that answers "have I possibly seen this key before," with a small, tunable false-positive rate but no false negatives) for these keys at a 0.1% false-positive rate ($p = 0.001$):

A roughly 72 MB Bloom filter per region is cheap enough to keep fully in memory and gives a bounded, known false-positive rate for the dedup layer, versus an exact hash-set which would need far more memory to track 40 million live keys.

Local buffer disk sizing. To survive a 10-minute (600 s) broker outage at the regional steady rate without dropping data:

Provisioning roughly 30 GB of local disk per regional collector tier is the concrete number that backs the "collectors survive a broker outage" claim, not just an assertion that buffering exists.

Trade-offs and Pitfalls

A common alternative is writing directly from collectors to the time-series store, skipping the durable log entirely. That removes a hop and its operational cost, but couples ingest availability directly to storage availability: any storage hiccup now blocks collectors instead of just delaying a downstream consumer. The durable log is worth its cost specifically because it decouples those failure domains.

Pre-aggregation trades ingest-time compute for downstream storage and query cost: computing rollups at 1,000,000 points/sec needs real CPU budget in the stream layer, and if the rollup logic has a bug, it is much harder to recompute correct history than if raw data were simply sitting untouched in cheap storage. Keep raw data for a bounded window specifically so a bad rollup is recoverable.

The partition key of tenant_id + metric_name preserves per-series locality but can create a hot partition if one tenant emits a disproportionate share of traffic; watch for this and consider adding a bucket suffix to the key for known outlier tenants rather than repartitioning the whole topic reactively.

I’d start by aligning everyone on the outcome, not the function-specific asks.

Step 1: Clarify the shared goal
I’d bring product, design, data, and engineering into one working session and define the business outcome, success metrics, and deadline constraints.

Step 2: Map dependencies and capacity
I’d list the critical dependencies, identify who owns each one, and make capacity visible by function. That exposes where the real bottlenecks are.

Step 3: Sequence the plan
I’d build the plan around the critical path: what must happen first, what can run in parallel, and what can be deferred. If capacity is tight, I’d use a simple trade-off framework: highest business value, lowest risk, and strongest dependency unlocks first.

Step 4: Create commitment
I’d confirm decision rights, document what each team is committing to, and define checkpoints where we can re-plan if assumptions change.

The goal is not to make everyone equally happy; it’s to make the trade-offs explicit so each group can commit to a plan they helped shape.

Worked example
Say the initiative is a checkout redesign that needs a payments-data migration (data team), a new UI (product design and frontend), and an updated fraud-detection model (data science). In the working session, the shared goal turns out to be reducing checkout abandonment by a set amount before the next major sales event, which becomes the deadline constraint. Mapping dependencies shows the new UI can't ship until the data migration completes, and the fraud model needs at least two weeks of production traffic on the new UI before it can be retrained safely, so the data migration is the critical-path item. Applying the trade-off framework, the data migration (highest dependency-unlock value) is sequenced first, the UI ships second, and the fraud-model update is explicitly deferred to just after the sales event rather than rushed; each team commits to that sequence in writing, with a checkpoint two weeks before launch to re-plan if the migration slips.

Situation: I notice many low-severity incidents happening repeatedly over the month—each is small but frequent. The goal is to decide whether to apply quick fixes or invest in an architectural change.

1. Measure cumulative business impact

• Quantify frequency and scope: incidents per day/week, affected requests/users, percent of traffic.
• Convert to business metrics: lost revenue = affected_requests * conversion_rate * avg_order_value; SLA penalty risk = incidents × penalty_per_violation; operational cost = oncall_hours_per_incident × hourly_rate.
• Customer experience: % of sessions affected, churn signal (support tickets, NPS drops).
• Reliability metrics: additional error rate Δ, total minutes degraded, total availability impact = sum(degraded_minutes)/total_minutes.

Example quick formula:
CumulativeCost = Σ_i (user_impact_i * value_per_user + ops_time_i * cost_per_hour + sla_penalty_i)

2. Analytics to run

• Time-series trend analysis (Prometheus/Grafana) to show incident frequency and correlation with deploys/traffic.
• Error budget burn-rate and MTTR/MTTA trends.
• Root-cause clustering: group incidents by error signature, host, region, code path (use ELK or Sentry).
• Cohort analysis: which customers/regions see most impact.
• Cost-per-root-cause: combine frequency and cost to get annualized cost for each cluster.
• Heatmap of incidents by time-of-day and service/component.

3. Decision process: band-aid vs architectural change

• Prioritize by cumulative cost and recurrence: rank clusters by AnnualizedCost.
• Define thresholds: e.g., if a cluster’s annualized cost > cost_to_build_long_term_fix OR burn-rate causes > X% of error budget monthly, choose deeper fix.
• Consider risk/time trade-offs:
  • Short-term (band-aid) when: low cumulative cost, imminent release deadlines, fix reduces customer-visible impact immediately, buy time for proper design.
  • Long-term (architectural) when: high recurrence, large annualized cost, repeated firefighting consumes >Y% of team time, or fixes brittle areas that impede scaling.
• Estimate ROI / payback period: PaybackMonths = ImplementationCost / MonthlySavings.
• Stakeholder alignment: present analytics, ROI, and risk to product and engineering managers; get budget and timeline.
• Iterate: implement band-aid with observability improvements and a scheduled project for the architectural fix; track whether band-aid reduces cost or only postpones.

Result/Example: After clustering, I found one error signature caused 60% of incidents and ~$12k/mo in ops and lost conversions. A quick config fix cut incidents 30% (immediate benefit). A planned refactor estimated at $60k would pay back in 5 months — approved and scheduled. Meanwhile I added a focused alert with runbook to reduce MTTR.

This approach ensures decisions are data-driven, balance short-term availability needs with long-term technical health, and quantify business value for prioritization.

Direct answer

Baking an image means pre-installing everything a server needs (OS packages, hardening, the app itself) into a reusable image with a tool like Packer, instead of configuring the server after it boots. Build the pipeline around one principle: nothing reaches production as an image that has not been baked, tested, and scanned the same way every time, and the fleet gets updated by replacing instances behind health checks and connection draining rather than patching them in place. The design has two paths through the same pipeline: the normal path (bake, test, promote through environments, canary, full rollout) and a fast path for urgent security patches that skips environment promotion but never skips the tests or the scan.

Structured elaboration

Image build and test

• CI triggers a Packer build on a base-image or application change: provision the base OS, apply hardening (CIS-style benchmarks, i.e. standardized security-configuration checklists), install the app artifact, and pull secrets via short-lived tokens rather than baking them in.
• Baked-in automated tests run as part of the same pipeline, not as a separate manual step: unit and config-validation tests during the bake, then a post-bake stage that launches the image in an isolated environment and runs integration and smoke tests against it.
• A vulnerability scan (for example Trivy or Grype against the baked image) runs in that same post-bake stage. This is a hard gate, not advisory: an image with a scan finding above the agreed severity threshold does not get published.
• On pass, the image is registered in the artifact registry tagged with its git SHA, build ID, SBOM, and the CVE baseline it passed against, so any later question of "what is actually running" and "was it scanned against what we knew at the time" has an answer.

Promotion across environments

Promotion is a pipeline gate, not a person clicking approve in a console: dev, then staging with regression tests, then a canary slice of production, each gated on the previous stage's tests and monitoring staying green.

Swapping the fleet over

• The fleet sits behind an ASG (or equivalent instance group) and a load balancer. The launch template points at the new image; an ASG instance refresh (or an equivalent rolling-replace controller) walks the fleet in batches.
• Per instance: deregister from the target group first, which starts connection draining; wait for in-flight requests to finish or the drain timeout to hit; only then terminate it. The replacement instance must pass its health check before the load balancer sends it any traffic.
• A minimum-healthy-percentage setting (for example 90%) caps how much capacity can be replacing at once, so a bad new image degrades a fraction of the fleet rather than all of it while it is still being watched.
• For workloads that carry state (a service with long-lived connections, or one with session affinity), connection draining alone is not enough: the drain window also has to respect existing session affinity, and if any part of the workload is stateful in the sense of holding data (not just connections), that has to coordinate with the data layer's own replication or failover process rather than treating the instance as freely swappable the moment its health check fails.

Fast-tracking an urgent security patch

The fast path changes how far the image travels before real traffic sees it, not whether it is tested:

• Skip the full dev-then-staging promotion chain; go straight from bake to a canary slice of production.
• Keep the bake-time tests and the vulnerability scan as hard gates; an urgent patch that has not been scanned is exactly the failure mode a patch process exists to prevent.
• Shorten, but do not remove, the canary observation window, and have the rollback path pre-verified rather than improvised, since this path is exercised under time pressure.
• Immediately backfill afterward: once the emergency patch has gone through the fast path, run it (or its base) through the normal dev and staging pipeline the following day, so the fast-tracked version does not become a permanent exception living outside the standard promotion history.

Worked example

flowchart TD
  A[Source or base image change] --> B[CI triggers Packer bake]
  B --> C[Bake time tests: hardening, vuln scan, smoke tests]
  C --> D[Publish image with SBOM and CVE tags to registry]
  D --> E[Promote through dev then staging]
  E --> F[Canary: weighted traffic on new AMI]
  F --> G{Health checks and SLOs pass?}
  G -- Yes --> H[Full fleet rollout via ASG instance refresh]
  G -- No --> I[Roll back to prior AMI, tag new image as bad]
  B -.urgent security patch.-> J[Fast path: skip dev and staging, bake plus scan only]
  J --> F

Concretely: a CVE lands in the base OS image. CI triggers a Packer bake immediately (the dashed path above). The bake produces a new image; the same automated tests and the same vulnerability scan run against it as any normal build, just without waiting for a scheduled promotion window. It goes straight to a canary slice of the fleet, monitored against the same health checks and error-rate thresholds as any other rollout, then to the full fleet via instance refresh. The following day, the same image is run through the normal dev and staging environments to confirm nothing outside the emergency scope regressed.

Trade-offs & pitfalls

• Baking images takes longer than patching in place, and that trade-off is deliberate: reproducibility and a clean rollback (revert the launch template to the previous image ID) are worth the extra build minutes.
• The most dangerous version of a "fast path" is one that quietly also skips testing or scanning under time pressure; the fast path should only ever shorten promotion, never verification.
• For stateful workloads, connection draining and health checks are necessary but not sufficient; assuming they are enough to make image replacement safe for anything holding data is a common design mistake.
• Rolling back an in-flight instance refresh needs to be a rehearsed, one-command action (point the launch template back at the previous image ID), not something improvised the first time it is needed.

Direct answer
My default is to build a temporary mitigation myself rather than wait, unless the vendor's ETA is both short and credible, because customer experience shouldn't depend on a timeline I don't control and can't verify. The key criteria are how reliable the vendor's own status communication has historically been, how cheap and low-risk a mitigation is to build, and whether building a mitigation might itself mask a vendor problem that needs to stay escalated.

Structured elaboration

• Assess the vendor's ETA credibility. Vendors are often optimistic about their own timelines; check their status page's track record, whether they've given a specific, committed time or a vague 'we're looking into it,' and weigh that against your own tolerance for continued impact.
• Assess mitigation cost and risk. A cheap, well-understood mitigation (a circuit breaker to fail fast instead of hanging, a cached fallback response, a degraded-but-functional mode) is usually worth building even for a short outage; a mitigation that requires new, untested code under time pressure carries its own risk and might not be worth it for a vendor issue expected to resolve in minutes.
• Don't let your mitigation hide a problem that still needs escalating. If you build a good enough fallback, make sure someone is still tracking and escalating the underlying vendor issue; a well-built mitigation can quietly make an important problem invisible to leadership or to the vendor relationship owner.
• Verify the vendor issue is what you think it is, not just assumed: check the vendor's own status page or reach out directly rather than purely inferring from your own symptoms, since 'looks like a vendor is down' and 'confirmed the vendor is down' warrant different confidence levels.

Worked example
A third-party payment processor starts intermittently failing, causing checkout errors for a subset of users. The vendor's status page shows 'investigating' with no ETA, and their historical incidents have often run longer than initially communicated. The team decides not to wait: they add a circuit breaker so failing calls fail fast rather than hanging and degrading the whole checkout flow, and enable a secondary, lower-priority payment path they'd already built for exactly this kind of situation. They keep the primary vendor issue actively tracked and continue monitoring the vendor's status page, so once the vendor recovers they can cleanly revert to the primary path, rather than letting the fallback silently become permanent.

Trade-offs and pitfalls
Building a mitigation under pressure risks shipping throwaway code that never gets properly cleaned up and becomes unplanned permanent technical debt; it's worth explicitly flagging a mitigation as temporary and following up after the incident. Waiting on a vendor whose communicated ETA turns out to be optimistic (vendors very often are) means your customers experience longer impact than necessary, purely because you trusted someone else's timeline you had no way to verify. The same underlying judgment applies whether the failing dependency is a payment processor, a cloud provider's specific service, or a network transit provider throttling traffic to a region: the calibration is always about ETA credibility versus mitigation cost, not about the specific vendor.

Situation: A critical downstream service (payments) is degraded and our dependent microservices are burning SLO error budget fast. If unchecked, upstream services will cascade and the product-level error budget will be exhausted, risking customer impact and regulatory exposure.

Immediate actions I’d take (first 0–30 minutes)

• Triage & scope: confirm SLIs (latency/error rate) and identify which services’ requests are failing vs timing out using dashboards (Prometheus/Grafana, tracing).
• Short-term protective controls:
  • Apply circuit breakers and bulkheads (Envoy/Istio) to prevent retries from amplifying load.
  • Throttle non-essential traffic (batch jobs, analytics, low-tier customers) using rate-limiting rules.
  • Switch failing calls to a degraded path / fallback cached responses where safe.
• Trigger an incident bridge (PagerDuty) and notify stakeholders (product, engineering, infra, legal, comms).

Coordinating SLO relaxation, rollbacks, throttles (30–120 minutes)

• Convene decision authority (SRE lead + product + service owners). Present current burn rate, projected budget exhaustion time, and impact matrix (who loses what function).
• Use a predefined error-budget policy: if projected burn > X% in Y hours, allow temporary SLO relaxation or traffic reduction for lower-priority tiers. Example: relax non-customer facing read-only SLOs from 99.95% to 99.5% for up to 24 hours approved by SRE lead + product manager.
• If a recent deploy likely caused regression, trigger automated safe rollback via CI/CD with canary abort and quick rollback playbook. If rollback risky, apply feature-flag cut or throttles instead.
• Apply customer-tier throttles: preserve premium customer traffic while shedding best-effort workloads.

Guardrails to prevent cascading burn (longer-term/proactive)

• Implement automated circuit-breaker and retry-budgeting per-service; retries consume retry budget tracked by metrics to avoid amplification.
• Enforce bulkheads: isolate resource pools per downstream dependency to prevent global resource exhaustion.
• Implement adaptive throttling with backpressure signals (HTTP 429 propagation, gRPC headers) and central policy engine (Kong/Istio).
• Per-customer quotas and priority queues to protect SLAs for high-value users.
• Automate detection + automated mitigation: runbooks encoded in tools (Playbook-as-code) that, upon patterns (error-rate spike + downstream latency), automatically enact safe throttles and notify humans.
• Ensure SLO governance: preapproved escalation matrix for SLO relaxation, maximum duration, rollback criteria, and mandatory post-incident review.

Customer trust & communication

• For customer-visible degradation, coordinate comms: clear status page updates, targeted notifications to impacted customers, estimated ETA, and mitigations taken. Preserve transparency and compensation policy triggers for premium customers.

Post-incident

• Run blameless postmortem with data: root cause, why protections failed/weren’t in place, and concrete remediation (e.g., add caching, increase capacity, refactor dependency calls).
• Update SLOs/SLIs if they no longer reflect customer expectations and add automation to prevent repeat (tests, chaos experiments).
• Measure effectiveness: track reduced time-to-mitigate, reduction in error-budget burn in future incidents, and customer satisfaction.

Why this approach works

• Mixes immediate automated protections to stop amplification with governance to avoid unsafe permanent SLO changes.
• Protects highest-value customers while maintaining transparency.
• Encodes decisions into policies and automation to reduce manual coordination latency in future incidents.