SLIs, SLOs, SLAs, and Error Budgets Questions
Defining and operating reliability targets. Covers choosing service level indicators, setting service level objectives and agreements, computing and spending error budgets, and using them to drive engineering decisions. Includes negotiating reliability targets with stakeholders.
How would you present monitoring and alerting design rationale differently to SREs versus product managers? Include the SLIs, SLOs you would propose, example alert thresholds, and the essential contents of a runbook you'd attach.
Sample Answer
The same underlying monitoring and alerting design needs a genuinely different presentation for each audience, because SREs, product managers, and executives are each trying to answer a different question with the same underlying data, and a re-architecture change is a good concrete example of when all three need to hear about it simultaneously but differently.
Structured elaboration
To SREs: full technical detail, the actual SLIs and alert thresholds, the runbook itself (health checks, mitigation commands, rollback criteria, escalation contacts), since they're the ones who'll actually operate the new architecture and need the complete operational picture. To product managers: what changes about user-facing behavior or risk during the transition, what the new SLO commitments are in plain terms, and what decisions (if any) need their input (e.g. a temporary feature freeze during the migration window). To executives (the third audience tier): a much shorter framing focused on risk and business impact, e.g. "this re-architecture reduces our long-term outage risk but carries a defined, time-boxed elevated-risk window during migration, here's what we're doing to manage that."
Worked example
For a re-architecture that changes failure modes (say, moving from a single-region to a multi-region deployment): the SRE-facing document proposes concrete SLIs/SLOs, for instance an availability SLO of 99.95% measured over a rolling 30-day window (up from the single-region architecture's 99.9%, since the redundant design should sustain a tighter target) and a p99 latency SLO of under 250ms accounting for the added cross-region replication overhead. Example alert thresholds: page on a 4x burn-rate sustained for 15 minutes (fast, high-confidence incident) or a 1.5x burn rate sustained for 6 hours (slow degradation worth a ticket). The attached runbook's essential contents: the failover health-check commands to confirm which region is currently primary, the manual failover trigger command and its expected completion time, explicit rollback criteria (revert to single-region operation if failover doesn't complete within 10 minutes), and the on-call escalation contact chain specific to the migration window. The product-manager-facing summary states the practical implication ("during the two-week migration window, deploy velocity for this service will be reduced as we validate the new setup; user-facing reliability should be unaffected or improved") with a clear ask (approval to slow feature releases for that window). The executive-facing note is a few sentences: the business risk being reduced, the temporary managed risk during transition, and a single sentence on cost, with an offer to go deeper if wanted.
Trade-offs and pitfalls
Sending the full SRE-level technical document to executives (or the reverse, a one-line executive summary to the SRE team who actually need the runbook) is a common and costly mismatch; each audience needs a genuinely DIFFERENT document, not the same document with a different cover page. It's also worth deciding in advance who owns writing and maintaining each of the three versions as the re-architecture plan evolves, since letting the technical document update in real time while the executive summary goes stale (or vice versa) leaves at least one audience working from outdated information exactly when a migration is underway and things are most likely to change.
What is an 'error budget policy'? Give two concrete, real-world examples of policies (one automated and one manual) describing thresholds, actors, and actions. Explain why each example helps balance velocity with reliability.
Sample Answer
An error budget policy is the pre-agreed set of rules for what actually happens once budget consumption crosses specific thresholds, turning "we have an error budget" from an abstract concept into concrete, predictable action.
Structured elaboration
A good policy names the threshold, who is notified or empowered to act, and exactly what they do or are authorized to do; it needs both an AUTOMATED tier (fast, mechanical, no human judgment required) and a MANUAL tier (requires a person to weigh context, because the automated response alone isn't sufficient at that severity). Automated example: for a 99.9%-SLO service, if more than 50% of the WEEKLY error budget allocation is consumed, the CI/CD pipeline automatically restricts merges to hotfix-only, no human approval needed to trigger this, though an override exists for genuine emergencies. Manual example: if the error budget is fully exhausted for an entire QUARTER (a sustained, structural reliability problem rather than a one-off incident), engineering leadership and the product owner hold a joint review to reallocate the next quarter's roadmap toward reliability work, since the situation has moved from a tactical release-freeze to a strategic resourcing decision.
Worked example
Applying this to a data-engineering ingestion team illustrates the same mechanism in a different domain: an ingestion pipeline with a 99.9% freshness SLO uses its error budget the same way a request-serving API would, to decide the trade-off between shipping new pipeline features and investing in reliability fixes; when the pipeline's budget burns past the weekly threshold, new feature work pauses automatically (via the same CI gate pattern) in favor of fixing the freshness regression first, and if the budget stays exhausted across a full quarter, the team's roadmap gets the same leadership-level reallocation review as any other service.
Trade-offs and pitfalls
A policy that only defines automated actions without a manual, strategic tier will keep re-triggering the same tactical response (freeze, unfreeze, freeze again) without ever addressing a structural reliability problem; conversely a policy with only manual review and no automated tier is too slow to prevent damage during an active incident. The two tiers are solving different problems (fast tactical response vs slow structural correction) and a mature policy needs both, explicitly distinguished by which threshold triggers which.
Describe the key success metrics and SLOs you defined for a production data service you built. Include quantitative thresholds (throughput, p95/p99 latency, data freshness, error rate), how you chose the thresholds based on business needs, and how you validated those SLOs during launch and operation.
Sample Answer
Choosing SLOs for a data service you actually built starts from the specific business need the service exists to serve, then works down to the quantitative thresholds, not the other way around.
Structured elaboration
Success metrics worth defining: throughput (requests or records served per unit time, confirming the service can sustain expected load), p95/p99 latency (tail experience, since a data service's consumers often run bursty batch queries where the tail matters as much as, or more than, the median), data freshness (how current the served data is relative to its source), and error rate (proportion of requests failing outright). Thresholds should be chosen against what the CONSUMING use case actually requires: a dashboard refreshed hourly can tolerate looser freshness than a real-time alerting pipeline consuming the same underlying data service.
Worked example
For a production data service feeding both an hourly-refreshed executive dashboard and a near-real-time fraud-detection pipeline: p95 latency < 200ms (chosen based on what the fraud pipeline's own SLA requires, the tighter of the two consumers), freshness < 5 minutes (again driven by the tighter consumer, even though the dashboard alone would have tolerated much looser freshness), error rate < 0.1%. Validation during launch: a soft-launch period against a subset of real consumer traffic, checking that the chosen thresholds are both ACHIEVABLE (the service can actually sustain them under real load) and SUFFICIENT (the tightest consumer's actual needs are genuinely met, confirmed by that consumer's own downstream metrics staying healthy).
Trade-offs and pitfalls
Setting one blended threshold to satisfy the AVERAGE of all consumers' needs, rather than the TIGHTEST consumer's actual requirement, is a common mistake: it can look reasonable on paper while quietly under-serving the one consumer (often the most business-critical one) whose needs were actually the tightest. It's also worth revisiting these thresholds as new consumers are added over the service's life, since a threshold validated against the original set of consumers may become insufficient once a new, more demanding consumer starts depending on the same service without anyone re-checking whether the existing SLO still covers their needs.
Implement a Python function that consumes a time-series dataset of per-minute 'success_count' and 'total_count' and computes SLO compliance for a 30-day rolling window. Input: list of (timestamp, success_count, total_count). Output: percentage of rolling windows that meet an SLO of 99.9% success. Describe assumptions and how you handle missing minutes and bad data.
Sample Answer
Rolling SLO compliance asks, for every day in a series, whether the trailing N-day window ending that day met the target, which is a different (and more informative) question than a single all-time compliance percentage.
Structured elaboration
For each day with data, the function looks back over the trailing window, sums successes and totals across all days that fall in that window, and checks whether the resulting ratio meets the SLO target; the final output is the percentage of such windows (not days, and not raw minutes) that were compliant, which is what a team actually wants to see plotted over time to spot a slow-building degradation.
Worked example (executed; python3, 40 synthetic days: 30 good days then 10 degraded days)
from datetime import datetime, timedelta
from collections import defaultdict
from typing import List, Tuple
def rolling_slo_compliance(records: List[Tuple[datetime, int, int]],
window_days: int = 30,
slo_target: float = 0.999) -> float:
by_day = defaultdict(lambda: [0, 0]) # day -> [success, total]
for ts, succ, tot in records:
if succ is None or tot is None:
continue # bad/missing minutes excluded from both numerator and denominator
day = ts.date()
by_day[day][0] += succ
by_day[day][1] += tot
days_sorted = sorted(by_day.keys())
if not days_sorted:
return 0.0
compliant = 0
evaluated = 0
for anchor in days_sorted:
window_start = anchor - timedelta(days=window_days - 1)
succ_sum, tot_sum = 0, 0
for d in days_sorted:
if window_start <= d <= anchor:
succ_sum += by_day[d][0]
tot_sum += by_day[d][1]
if tot_sum == 0:
continue
evaluated += 1
if succ_sum / tot_sum >= slo_target:
compliant += 1
return (compliant / evaluated * 100.0) if evaluated else 0.0
Using 40 synthetic days where the first 30 days ran at 1000/1000 (100% success) and the final 10 days dropped to 500/1000 (50% success), the function correctly computed 75.0% of evaluated 30-day windows as compliant: the anchors on days 0-29 (30 anchors) still had entirely clean data inside their trailing windows and passed the 99.9% target, while the anchors on days 30-39 (10 anchors) each had at least one degraded day inside their trailing window and correctly failed, giving 30 of 40 total windows compliant (30 / 40 = 75.0%), exactly matching hand-computed expectations.
Trade-offs and pitfalls
This implementation excludes missing minutes from both the numerator and denominator of a window, which is a DIFFERENT convention from the conservative all-missing-counts-as-bad approach used for a real-time alerting SLI; that divergence is deliberate and must be stated explicitly, since silently applying the alerting convention here would make routine data gaps look like manufactured SLO violations in a report that's supposed to reflect actual customer-facing reliability. Early anchors near the start of the series (e.g. day 0, whose trailing 30-day window only has 1 day of real data, since no earlier days exist yet) are technically evaluated against an incomplete window; a stricter implementation might require a minimum number of in-window days before counting an anchor as evaluated at all.
Implement a Python function compute_error_budget_consumption(slo_percent: float, window_minutes: int, good_minute_flags: List[int]) that returns the percentage of the error budget consumed in the window. good_minute_flags is a list with 1 for a 'good' minute and 0 for a 'bad' minute. Assume the list length equals window_minutes. Show examples and complexity.
Sample Answer
The function needs to translate a minute-by-minute good/bad record into a single "percent of the allowed error budget already spent" number, which is the quantity teams actually watch (not the raw bad-minute count).
Structured elaboration
Given an SLO percentage, the allowed BAD fraction of the window is 1−slo/100, and multiplying that by the window length gives the allowed bad minutes. Consumption is then the ratio of actual bad minutes to allowed bad minutes, expressed as a percentage: a result over 100% means the budget for that window has already been overspent, which is a normal and important state to represent (not an error condition). Complexity: the function makes a single pass over good_minute_flags to sum it (sum() is O(window_minutes) time), plus O(1) arithmetic afterward, so overall time complexity is O(window_minutes) and space complexity is O(1) beyond the input list itself (no auxiliary data structures are allocated).
Worked example (executed; python3, verified against 5 cases including an edge case)
from typing import List
def compute_error_budget_consumption(slo_percent: float, window_minutes: int,
good_minute_flags: List[int]) -> float:
if len(good_minute_flags) != window_minutes:
raise ValueError("good_minute_flags length must equal window_minutes")
allowed_bad_fraction = 1 - slo_percent / 100.0
allowed_bad_minutes = allowed_bad_fraction * window_minutes
actual_bad_minutes = window_minutes - sum(good_minute_flags)
if allowed_bad_minutes == 0:
return 100.0 if actual_bad_minutes > 0 else 0.0
return (actual_bad_minutes / allowed_bad_minutes) * 100.0
Verified results: a 99.9% SLO over 1440 minutes (1 day) allows 0.001×1440=1.44 bad minutes; with exactly 1 bad minute observed, consumption is 1/1.44×100≈69.44% (confirmed by running the code). With 720 of 1440 minutes bad, consumption comes out to 50,000% (720/1.44*100), correctly signalling a catastrophic overspend rather than clamping silently to 100%. A perfectly clean window returns exactly 0.0%, and a length mismatch between the flag list and the stated window raises a ValueError rather than silently truncating.
Trade-offs and pitfalls
Clamping the output at 100% is tempting for a dashboard, but it destroys the information a responder needs to judge HOW badly the budget was blown, which matters for deciding release-freeze severity; better to let it exceed 100% and clamp only at the presentation layer if needed. A real production version also needs to handle partial-credit minutes (a minute that was only degraded, not fully down) rather than the binary 1/0 flag used here, and needs to decide how to treat missing minutes (this implementation implicitly requires every minute to be explicitly flagged, which forces the caller to make a conscious choice for gaps rather than silently defaulting them to good).
Unlock Full Question Bank
Get access to all SLIs, SLOs, SLAs, and Error Budgets interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.