SLIs, SLOs, SLAs, and Error Budgets Questions
Defining and operating reliability targets. Covers choosing service level indicators, setting service level objectives and agreements, computing and spending error budgets, and using them to drive engineering decisions. Includes negotiating reliability targets with stakeholders.
Describe the key success metrics and SLOs you defined for a production data service you built. Include quantitative thresholds (throughput, p95/p99 latency, data freshness, error rate), how you chose the thresholds based on business needs, and how you validated those SLOs during launch and operation.
Sample Answer
Choosing SLOs for a data service you actually built starts from the specific business need the service exists to serve, then works down to the quantitative thresholds, not the other way around.
Structured elaboration
Success metrics worth defining: throughput (requests or records served per unit time, confirming the service can sustain expected load), p95/p99 latency (tail experience, since a data service's consumers often run bursty batch queries where the tail matters as much as, or more than, the median), data freshness (how current the served data is relative to its source), and error rate (proportion of requests failing outright). Thresholds should be chosen against what the CONSUMING use case actually requires: a dashboard refreshed hourly can tolerate looser freshness than a real-time alerting pipeline consuming the same underlying data service.
Worked example
For a production data service feeding both an hourly-refreshed executive dashboard and a near-real-time fraud-detection pipeline: p95 latency < 200ms (chosen based on what the fraud pipeline's own SLA requires, the tighter of the two consumers), freshness < 5 minutes (again driven by the tighter consumer, even though the dashboard alone would have tolerated much looser freshness), error rate < 0.1%. Validation during launch: a soft-launch period against a subset of real consumer traffic, checking that the chosen thresholds are both ACHIEVABLE (the service can actually sustain them under real load) and SUFFICIENT (the tightest consumer's actual needs are genuinely met, confirmed by that consumer's own downstream metrics staying healthy).
Trade-offs and pitfalls
Setting one blended threshold to satisfy the AVERAGE of all consumers' needs, rather than the TIGHTEST consumer's actual requirement, is a common mistake: it can look reasonable on paper while quietly under-serving the one consumer (often the most business-critical one) whose needs were actually the tightest. It's also worth revisiting these thresholds as new consumers are added over the service's life, since a threshold validated against the original set of consumers may become insufficient once a new, more demanding consumer starts depending on the same service without anyone re-checking whether the existing SLO still covers their needs.
Design an enterprise reliability governance model that standardizes SLO ownership, error budget policies, reporting cadence, escalation paths, and incentives across multiple product lines. Include roles (product owner, SRE, engineering manager), required dashboards, and how you would enforce and audit compliance without slowing delivery.
Sample Answer
Enterprise-scale reliability governance needs a genuinely tiered structure, since the same rigid policy applied uniformly across 50 microservices of wildly different criticality either over-constrains the unimportant ones or under-protects the critical ones, and every layer of that structure needs an explicit, named owner and a real audit mechanism.
Structured elaboration
Roles: product owner (owns the business-criticality tier assignment and reporting to leadership), SRE (owns technical SLO measurement and the enforcement mechanism), engineering manager (owns the team's actual remediation work and roadmap trade-offs when budgets burn). Required dashboards: a standard, company-wide reliability dashboard template (current SLO status, burn rate, and tier per service) that every product line adopts, so an executive or auditor can compare services across lines without each team inventing its own bespoke reporting format. Standardized process elements: an explicit SLO-proposal APPROVAL workflow (who can propose a new SLO or change an existing one, and who signs off), a defined escalation and remediation-window procedure for repeated breaches (not just a one-off incident response, but what happens after the SECOND or THIRD consecutive miss), and tiered budget ALLOCATION across services of varying criticality: a concrete 3-tier example assigns critical services a strict policy triggering action at 50% budget consumed, important services at 75%, and best-effort services only at full exhaustion, with corresponding automated or manual actions at each threshold per tier. For a large-scale system (e.g. 50 microservices), budgets need explicit shared-vs-component allocation logic (does a cascading failure across several services draw from one shared pool or each service's own budget) and periodic audits to confirm the allocation still reflects actual current criticality, not a stale assessment from when the service first launched.
Worked example
For a 40-team organization with legal SLAs carrying real financial penalties: the governance model requires any SLO the model designates as SLA-relevant to go through a stricter approval path involving legal, not just engineering sign-off, with a monthly reporting cadence to a cross-functional steering committee (not just the platform team), and an escalation path where two consecutive missed SLOs trigger a mandatory remediation-window review with a defined completion deadline, escalating to executive visibility if the remediation window itself is also missed. Framed as an SLO-driven-development adoption initiative, the model explicitly ties reliability targets back to architecture decisions: a service repeatedly missing its tier's SLO despite remediation attempts should trigger an architecture review, not just another remediation cycle, since repeated tactical fixes failing to hold the target is itself evidence of a structural, not tactical, problem.
Trade-offs and pitfalls
The audit-without-slowing-delivery balance is the hardest part of this design: an overly heavy-handed compliance process (mandatory sign-offs for every minor SLO adjustment) will be quietly worked around by teams under delivery pressure, while an overly light one produces exactly the governance theater problem seen elsewhere, where the framework exists on paper but has no real teeth; the right calibration usually reserves the heaviest process (legal sign-off, executive escalation) for genuinely high-stakes changes (legally-binding SLAs, critical-tier services) while keeping best-effort-tier governance lightweight, so the total compliance burden scales with actual risk rather than being uniform across every tier.
Explain what an error budget is and how it is calculated. Give a numeric example: if the SLO is 99.9% availability over a 30-day window, show how much downtime is allowed and how you'd compute burn rate over a 6-hour window.
Sample Answer
An error budget is simply the inverse of your SLO, expressed as an amount of acceptable "badness" you're allowed to spend over a given window, and burn rate measures how quickly you're spending it.
Structured elaboration
If the SLO is 99.9% availability, the error budget is 1−0.999=0.001, or 0.1% of the window. Over a 30-day window, that translates to an absolute downtime allowance you can compute directly, and burn rate over any shorter sub-window (like 6 hours) compares the OBSERVED bad-fraction in that sub-window against the same 0.1% allowance, scaled to the sub-window's length.
Worked example (computed and verified)
30-day window in minutes: 30×24×60=43,200 minutes. Allowed downtime at 99.9% SLO: 0.001×43,200=43.2 minutes over the full 30 days. For a 6-hour (360-minute) window, the PROPORTIONAL allowance within that window is 0.001×360=0.36 minutes (about 21.6 seconds); if the observed downtime in that 6-hour window is, say, 2 minutes, the burn rate is 2/0.36≈5.56x, meaning the service is burning budget more than five times faster than sustainable for that window length.
Trade-offs and pitfalls
The same underlying mechanism applies well beyond a single request-serving API: a data-engineering ingestion team can use the identical trade-off framing (velocity vs reliability) to decide, for example, whether to ship a new ingestion feature or spend the sprint fixing a freshness regression, using their own SLO (e.g. 99.9% of partitions land on time) and the same 1-minus-SLO error-budget math. The most common mistake when presenting this is quoting the 30-day allowance (43.2 minutes) as if it applied unchanged to every sub-window; the allowance must always be scaled to the window actually being measured, or the resulting burn-rate comparison is meaningless.
Telemetry gaps occur in multiple regions due to a sidecar outage, causing missing SLI samples for 6 hours. Describe robust strategies to compute meaningful SLOs during and after the outage, including imputation, confidence intervals, synthetic tests, and preventing incorrect burn attribution.
Sample Answer
A multi-region telemetry gap means you genuinely don't know what happened during those six hours, and the honest response is to make that uncertainty explicit in the SLO calculation rather than silently filling the gap with an assumption that happens to be convenient.
Structured elaboration
During the outage: rather than excluding the gap entirely (which would understate risk if the gap coincided with an actual problem) or assuming it was fully healthy (optimistic and unverifiable) or fully bad (pessimistic and potentially inaccurate in the other direction), the more defensible approach reports the SLI for that period with an explicit CONFIDENCE INTERVAL reflecting the genuine uncertainty, alongside whatever independent CORROBORATING signal is available (synthetic checks that continued running through the gap, since a synthetic check doesn't depend on the same sidecar pipeline that failed, giving at least a partial, independent read on actual health during the gap). After the outage: any backfilled or imputed data for that window should be clearly FLAGGED as such in historical records, never silently blended in as if it were an equally-trustworthy real sample, so a future analyst looking back at this period understands its provenance.
Worked example
For burn-rate calculation specifically, the 6-hour gap should NOT be attributed a definite burn value at all; instead, the burn-rate calculation for that window is reported as "insufficient confidence, treat as N/A" rather than silently defaulting to zero (which would understate a real problem that happened to occur during the gap) or defaulting to the pre-outage baseline (equally an unverified assumption). If the synthetic checks that continued running through the outage showed healthy results, that's reported as corroborating (not conclusive) evidence, explicitly labeled with its lower confidence relative to full production telemetry.
Trade-offs and pitfalls
This same discipline generalizes well beyond the specific sidecar-outage scenario to any noisy or partial telemetry situation: whenever windows must be imputed or excluded, the choice of imputation method should be checked for FAIRNESS, specifically whether it systematically biases the reported SLI in either direction (excluding gaps entirely tends to make the reported number look BETTER than a fuller accounting would, since a real problem hiding in an excluded gap simply vanishes from the calculation), and the choice of aggregation semantics (sum vs average, how partial windows are weighted) should be documented explicitly since different reasonable choices can produce meaningfully different reported numbers from the exact same underlying gap. The single most damaging mistake here would be quietly attributing burn to the WRONG window because of a timestamp or timezone handling bug introduced while patching around the gap, which is worth explicitly testing for when building any gap-handling logic.
Draft an error budget borrowing and repayment policy between teams, including allowed borrow amounts, repayment time windows, visibility requirements, and penalties or incentives. Provide examples of compliant and non-compliant borrowing scenarios.
Sample Answer
Error-budget borrowing lets a team spend AHEAD of its normal allocation for a specific, justified reason (a planned risky migration, a time-sensitive launch), as long as it's repaid, and the policy's whole job is making sure "borrowing" doesn't quietly become "just not tracking budget properly."
Structured elaboration
Allowed borrow amounts: capped at a modest fraction of the team's NEXT period's allocation (e.g. up to 20% of next month's budget can be borrowed against this month), preventing a team from borrowing so much that they're effectively operating with zero real margin for an extended stretch. Repayment time windows: a defined, short horizon (e.g. borrowed budget must be repaid, meaning the team operates with EXTRA caution and reduced risk-taking, within the following month), not an open-ended IOU that can persist indefinitely. Visibility requirements: every borrow event is logged in the same shared, auditable registry used for other governance actions, visible to any team or stakeholder who might reasonably want to know a team is currently operating on borrowed budget. Penalties or incentives: a team that fails to repay within the agreed window loses borrowing privileges for a defined cooldown period (e.g. two subsequent months), a real but proportionate consequence.
Worked example
Compliant scenario: Team A wants to ship a planned, moderately-risky database migration next week that they've assessed will likely consume more budget than their current month's remaining allocation supports; they formally request to borrow 15% of next month's budget, documented with the specific justification and expected repayment plan, approved via the standard lightweight sign-off, and visible to other teams sharing any affected infrastructure. Non-compliant scenario: Team B has been quietly running over budget for three consecutive months without ever formally requesting to borrow, simply treating each month's overage as if it will "sort itself out," with no visibility to anyone else and no explicit repayment plan; this is exactly the undisciplined pattern the formal borrowing policy exists to replace, since an UNTRACKED overage gives none of the planning benefit a formally-documented borrow arrangement provides.
Trade-offs and pitfalls
Without a real, felt consequence for failing to repay, "borrowing" degrades into a polite fiction that lets a team perpetually operate over budget while calling it something more palatable; the cooldown-period consequence needs to be genuinely enforced, not waived informally when a team has a good excuse each time. It's also worth deciding whether TWO teams should be allowed to simultaneously borrow against a SHARED downstream dependency's stability, since two individually-approved, modest borrow requests could combine to create a much larger combined risk than either looks like in isolation; a genuinely lightweight policy still needs at least a basic check for this kind of compounding risk across simultaneous borrow requests.
Unlock Full Question Bank
Get access to all SLIs, SLOs, SLAs, and Error Budgets interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.