Error Budget Management and Release Velocity Questions
Understanding error budgets—how much downtime is acceptable given SLO commitments—and using error budgets to balance reliability and feature velocity. At senior level, understand how to calculate error budgets, communicate them to product teams, and make principled decisions about feature releases versus stability work. If you have error budget remaining, you can push features. If you're running low, you need to focus on stability. Design processes for error budget-aware release planning.
MediumBehavioral
33 practiced
Behavioral: Tell me about a time when you recommended pausing or throttling releases because the error budget was low or being consumed too fast. Use the STAR method and describe how you communicated the decision, handled pushback, and what the outcome and learnings were.
Sample Answer
Situation: Our payments service had a 99.95% availability SLO (≈22 minutes allowed downtime/month). During a week with a big feature rollout, our monitoring showed the error budget was being consumed at a 4x burn rate — we’d use the remaining month’s budget in ~3 days. Incidents and customer-visible 5xxs were increasing in canaries and a few prod pods.Task: As the on-call SRE, I had to decide whether to pause/throttle releases to protect the SLO, communicate that decision to stakeholders, and keep momentum while minimizing customer impact.Action:- I triaged metrics (error rate by deployment, request path, region) and confirmed the primary regressions were tied to the new release’s rate-limited path.- I paged the release owner, engineering manager, and product manager and presented a concise risk assessment: current burn rate, projected budget exhaustion time, and expected customer impact if we continued.- I recommended pausing further rollouts and throttling the release ramp (rollback to 20% canary and enable rate limits), citing the SLO risk and mitigation plan.- I documented the recommendation in Slack and our incident channel, opened an incident ticket, and asked for a 30-minute decision window.- When engineers pushed back (concern about missed deadline / momentum), I offered a compromise: pause global ramp but keep a limited canary with enhanced logging and circuit breakers to gather more data while preventing large-scale blast radius.- I coordinated the throttling change, validated reduced error rate, and ran quick A/B experiments to confirm fixes before resuming the rollout.Result: Pausing the rollout and throttling reduced the burn rate from 4x to below 1x within an hour, preserving ~90% of the monthly error budget. We identified and fixed two regressions, resumed rollout gradually with automated throttling rules, and avoided an SLO breach. Stakeholders accepted the trade-off because I provided data, a clear mitigation path, and a short timeline.Learnings:- Automate error-budget alarms with explicit remediation playbooks so decisions don’t rely solely on on-call judgment.- Include release-throttle knobs and guardrails in CI/CD pipelines.- Communicate quickly with data and a short, actionable plan — that builds trust and reduces pushback.
HardTechnical
31 practiced
Design an organizational error budget governance model for a company with 40 product teams. Include: how SLOs are negotiated, central vs team-level oversight, how to handle legal SLAs that require financial penalties, reporting cadence, and escalation paths when systemic budgets are at risk.
Sample Answer
Situation: At a company with 40 product teams, reliability consistency was uneven and business stakeholders wanted a clear, enforceable approach that aligned engineering autonomy with company risk.Model summary (high level):- Federated governance: central SRE reliability office (RRO) sets standards, tooling, and mediates; product teams own SLOs and error budgets for their services.- Negotiation follows a standardized cadence and template; legal SLAs are treated as a separate “contractual tier” with stricter controls and financial-risk mitigation.How SLOs are negotiated:- Teams propose SLOs using a standard template (service description, user-facing KPI, measurement window, target, error budget burn policy, mitigation plan).- RRO reviews for consistency with company-wide risk appetite, regulatory exposure, and dependencies. Negotiation includes product PM, engineering lead, security/compliance, and a BizOps rep.- Outcome recorded in a living SLO registry with versioning and owner contacts.Central vs team-level oversight:- Team-level: set/measure SLOs, runbooks, immediate mitigations, error-budget-driven feature gating.- Central RRO: provides observability stack, SLO review board, quarterly audits, cross-service impact analysis, and systemic reliability initiatives.- RRO enforces minimum baseline SLOs (e.g., availability for core platform services) that teams cannot set below without exec approval.Handling legal SLAs with financial penalties:- Any SLA with monetary penalties is escalated to a Contract SLA tier. RRO + Legal + Finance must approve SLO measurement method and dispute resolution clauses.- Mitigation: require guardrail SLOs, additional redundancy, dedicated on-call rotations, and escrowed financial reserves for high-risk contracts.- Introduce pre-deployment risk review and mandatory runbook & playbook validations for services tied to SLAs.Reporting cadence and tooling:- Daily: team dashboards with rolling error-budget burn alerts integrated into PagerDuty/Slack.- Weekly: RRO automated roll-up showing teams with >50% burn rate.- Monthly: SLO health report to engineering leadership and BizOps (trendlines, top contributors to failures).- Quarterly: SLO review board meets to approve changes, retire stale SLOs, and prioritize systemic investments.Escalation paths when systemic budgets are at risk:1. Team remediation: if burn > 50% in 7 days, team triggers a temporary freeze on non-critical releases and executes mitigation runbook.2. RRO intervention: if multiple teams or a core service triggers (>50% burn across >3 teams or >80% on core infra), RRO convenes incident bridge, assigns cross-team task force, and pauses dependent deployments.3. Executive escalation: persistent systemic risk (sustained high burn >30 days or contractual SLA exposure) escalates to CTO/Head of Ops; triggers cross-functional war room, customer communications plan, and contractual review for compensation exposure.Example outcome:- After implementing this model, we reduced cross-service major incidents by 40% in six months, standardized SLO definitions across teams, and eliminated unexpected SLA penalties by catching contract risk pre-deployment.Key principles:- Clear ownership, minimum central guardrails, standardized negotiation, visible metrics, and pre-approved escalations balance autonomy with company risk.
HardTechnical
34 practiced
For a distributed system with several dependent services (A -> B -> C), propose formulas and a practical process to attribute error budget consumption to the correct team when an end-to-end SLO is violated. Consider cascading failures and partial degradations.
Sample Answer
Approach summary: treat end-to-end SLO violations as a measurable loss (error budget consumed) and split that loss among contributing services using a combination of causal weighting (latency/error contribution) and responsibility heuristics (ownership of root cause). Use metrics and traces to compute per-service contribution, then apply governance rules for ambiguous cases.Formulas:- E_total = minutes_of_SLO_violation (or ratio × window)- For each request i during violation window, compute contribution score S_j(i) for service j: S_j(i) = w_error * I_error_j(i) + w_latency * normalized_latency_j(i) where I_error_j=1 if service returned error; normalized_latency_j = (latency_j - p50_j)/(p99_j - p50_j), clipped [0,1]. Choose w_error (e.g.,0.7), w_latency (0.3).- Per-service share: Share_j = (sum_i S_j(i)) / (sum_k sum_i S_k(i)) -> E_j = Share_j * E_total- Incorporate cascading multiplier: if trace shows upstream failure caused retries/fallbacks downstream, apply multiplier M_cascade to the originating service (e.g., +20–50%) and reduce downstream shares proportionally.Practical process:1. Instrumentation: ensure distributed tracing with unique request IDs, per-service latency/error logs, and circuit-breaker/fallback indicators.2. Detection: when E_total detected, extract affected traces and aggregate per-request S_j(i).3. Attribution run: compute shares, apply cascading multiplier when traces show error propagation, and reconcile with owner-reported root-cause if post-incident analysis identifies specific faulty change (override threshold, e.g., if RCA identifies >60% caused by one service, assign 100%).4. Governance: publish provisional attribution within 48 hours; allow teams to dispute with evidence (logs/traces/bugfix PRs) within 5 business days; finalize and deduct error budget.5. Automation/reporting: daily job computes provisional E_j, stores lineage, and surfaces dashboards and alerting to teams approaching budgets.Edge cases & safeguards:- Partial degradations: normalized_latency captures partial degradations; use user-impact weighting (e.g., some errors are non-user-facing).- Sparse traces: fallback to aggregate metrics (error rates × impact scores) with conservative attribution to owning service.- Incentive alignment: cap per-incident charge to avoid punishing dependent teams unfairly; encourage joint postmortems.Why this works: combines trace-level causality with simple, explainable math, handles both hard failures and slow degradations, provides human review to catch instrumentation gaps, and balances automatic attribution with governance to prevent blame.
HardSystem Design
34 practiced
Design a cross-region release strategy that uses per-region error budgets and a global error budget for a globally distributed service. Include traffic routing, failover behavior, coordination for multi-region feature releases, and rollback policies that respect both local and global budgets.
Sample Answer
Requirements:- Per-region SLOs and error budgets (e.g., 99.95% per-region; monthly error budget)- Global SLO and global error budget (aggregate behavior)- Low-latency failover, predictable rollback, multi-region feature coordination, automationHigh-level design:- Global traffic controller (GTC): DNS + Anycast + cloud global load balancer for geo-steering.- Regional frontends (R1..Rn) behind regional load balancers and region-local control plane.- Control plane services: Error Budget Manager (EBM), Feature Flag Service, Orchestrator.- Observability: region-metrics pipeline → central aggregator (metrics, traces, logs).Traffic routing & progressive rollout:- Default: geo-routing to nearest healthy region.- For releases, use progressive delivery: 1. Canary in origin region (1% traffic) with feature flag on. 2. Gradual fan-out across regions per policy: staged rollout (e.g., 1%→10%→50%→100%) with automated waits and checks.- Use traffic-splitting via service mesh (Envoy/ISTIO) or layer-7 LB with weighted routing.Error budgets & failover behavior:- EBM continuously computes consumed error budget per-region and global (weighted by traffic).- Local failure: if region exceeds its local error budget threshold (e.g., 50% consumed in a window) Orchestrator: - Throttle new canary increments in that region. - Shift region-specific non-critical traffic to other regions per capacity and latency constraints. - Trigger local rollback of feature flag in that region.- Global breach: if aggregated consumption exceeds global threshold, Orchestrator: - Pause all rollouts globally. - Initiate rollback for the release across all regions. - Activate cross-region load rebalancing: increase weights to healthier regions with available capacity and within latency SLOs.Coordination for multi-region releases:- Release policy defines ordering (e.g., seed regions, high-traffic regions last), required checks per stage (error rate, p95 latency, request success).- Feature Flag Service supports per-region overrides and gradual ramp.- Release Orchestrator enforces gates: must pass region-level metrics and global budget headroom before next stage.- Pre-declare maintenance windows and capacity headroom policies to avoid overloading neighboring regions during failover.Rollback policies:- Two-tier rollback: 1. Local automatic rollback: triggered when region’s short-window error rate exceeds threshold or rapid error budget burn; rollback only in that region and divert its traffic. 2. Global rollback: triggered when global error budget breach or multiple regions auto-rollback within short period; rollback everywhere.- Safe rollback steps: flip flag → drain new connections → scale down new code pods → route traffic back progressively → run automated verification tests.- Post-rollback: create incident, run blameless postmortem, update release policy.Automation & runbooks:- Automate gates in CI/CD pipeline tied into EBM metrics and Feature Flag API.- Runbooks for manual override, emergency global pause, and capacity escalation.- Alerts: tiered (warning when 25% budget used, action at 50%, critical at 80%); pages include suggested actions.Trade-offs:- Aggressive failover improves availability but can increase latency and cost due to cross-region traffic.- Per-region rollbacks reduce blast radius but complicate feature consistency; use idempotent features and data compatibility guarantees.- Strong automation reduces MTTR but requires robust testing of orchestration itself.This strategy balances local autonomy (per-region budgets and rollbacks) with global safety (global budget and coordinated global rollback), enabling safe progressive delivery at global scale.
MediumSystem Design
25 practiced
Design a canary deployment strategy for a service receiving 1,000,000 requests per minute, using error budgets to protect production. Include traffic percentages for each stage, monitoring windows, fail/rollback conditions based on SLI, and how to limit blast radius while preserving release velocity.
Sample Answer
Requirements & constraints:- Traffic: 1,000,000 RPM (~16.7k RPS)- Use error budget / SLO-driven rollouts, limit blast radius, preserve velocity (automated short canaries + progressive ramp).Canary strategy (stages & traffic):1. Dark/Smoke (0% user traffic): Deploy to canary nodes; run synthetic smoke tests, internal QA traffic for 5–10 minutes.2. Small canary (0.5% traffic ≈ 5k RPM / ~83 RPS) — 10 minutes observation.3. Medium canary (2% ≈ 20k RPM / ~333 RPS) — 15 minutes observation.4. Large canary (10% ≈ 100k RPM / ~1.67k RPS) — 20 minutes observation.5. Gradual ramp to 50% over automated steps (25%, 50%) with 15–30 minute windows; final 100% after manual/automated approval if healthy.Monitoring windows & metrics:- Short windows for canaries: rolling 5m and 15m windows; compute SLIs on both to detect spikes and sustained regressions.- Primary SLIs: error rate (5xx + client-impacting errors), p99 latency, request success ratio, saturation (CPU/memory/queue depth), user-impacting business metrics (checkout success, transactions).- Baseline: compare canary SLI to stable baseline using statistical test (KL/divergence or two-sample t-test) and relative thresholds.Fail / rollback conditions (SLO-driven):- Immediate abort if: - Absolute error rate > 1% (or configured threshold) in 1–5m window OR - p99 latency degrades > 2x baseline in 5m window OR - Any alert on resource exhaustion (CPU >90%, OOMs).- Slow-fail (pause/limit ramp) if: - Relative error increase > 50% vs baseline sustained for 15m OR - Business KPI drop > 2% in 15m.- Automated rollback if any immediate abort; automated pause + manual review for slow-fail.Error budget integration:- If service has remaining error budget > 50%: allow automated progressive ramps as above.- If error budget <= 50%: require throttled canaries (max 2% until budget replenished) and manual approval for >2%.- If error budget exhausted: freeze releases to >0% user traffic; only emergency fixes allowed.Limiting blast radius while preserving velocity:- Start very small (0.5%) to catch high-impact defects quickly with tiny user exposure.- Route by user segments: route canary to internal users, non-revenue users, or low-risk geo regions first.- Use feature flags + traffic steering (Istio/Traffic Manager) to decouple deploy from enablement.- Isolate canary instances (separate node pools, lower autoscale limits) to avoid broad resource contention.- Automate gating in CI/CD: run smoke + integration checks, auto-promote when SLI tests pass, with human-in-the-loop only at higher traffic thresholds or low error budget.- Maintain fast rollback automation, post-deploy canary postmortems, and learnings to shrink future canary windows.Trade-offs:- Short windows detect fast failures but may miss slow regressions — mitigated by layered windows (5m + 15m) and progressive ramps.- Conservative thresholds reduce risk but slow releases; error budget policy balances safety vs velocity.
Unlock Full Question Bank
Get access to hundreds of Error Budget Management and Release Velocity interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.