SLIs, SLOs, SLAs, and Error Budgets Questions
Defining and operating reliability targets. Covers choosing service level indicators, setting service level objectives and agreements, computing and spending error budgets, and using them to drive engineering decisions. Includes negotiating reliability targets with stakeholders.
Write a SQL query (or outline it) to compute the 30-day sliding-window error budget burn rate at hourly granularity given a table requests(timestamp, status_code, service). The output should be a time series of burn_rate per hour over the past 90 days. Explain how you handle aggregation and performance considerations.
Sample Answer
A 90-day, hourly-granularity burn-rate series needs an efficient way to compute a 30-day TRAILING sum at every hour, which is a classic windowed-aggregation problem, not a per-row calculation.
Structured elaboration
Rather than recomputing a 30-day sum from scratch for every output hour (an O(n * window) approach), a window function computes each trailing sum in a single pass. First pre-aggregate raw requests into hourly buckets (service, hour_bucket, total, bad), converting each bucket's timestamp into an integer hour-index (hour_num) so the frame can use a numeric RANGE (dates directly are not portable across engines for RANGE frames). Then a SUM(...) OVER (PARTITION BY service ORDER BY hour_num RANGE BETWEEN 719 PRECEDING AND CURRENT ROW) (719 = 30 days x 24 hours - 1) gives the true trailing 30-day total and bad count at every hour in a single pass; the burn rate at each hour is the ratio of the trailing bad-to-total requests, normalized by the SLO's allowed error fraction.
Worked example (executed; sqlite3, hourly aggregation and 30-day trailing window confirmed against a small synthetic dataset)
WITH hourly AS (
SELECT service,
strftime('%Y-%m-%d %H:00:00', timestamp) AS hour_bucket,
CAST(strftime('%s', strftime('%Y-%m-%d %H:00:00', timestamp)) / 3600 AS INTEGER) AS hour_num,
COUNT(*) AS total,
SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) AS bad
FROM requests
WHERE timestamp >= datetime('now', '-90 days')
GROUP BY service, hour_bucket
),
trailing AS (
SELECT service, hour_bucket, hour_num, total, bad,
SUM(total) OVER (PARTITION BY service ORDER BY hour_num
RANGE BETWEEN 719 PRECEDING AND CURRENT ROW) AS total_30d,
SUM(bad) OVER (PARTITION BY service ORDER BY hour_num
RANGE BETWEEN 719 PRECEDING AND CURRENT ROW) AS bad_30d
FROM hourly
)
SELECT service, hour_bucket, total_30d, bad_30d,
ROUND(bad_30d * 1.0 / total_30d, 4) AS observed_error_rate_30d,
ROUND((bad_30d * 1.0 / total_30d) / 0.001, 2) AS burn_rate_30d_vs_999_slo
FROM trailing
ORDER BY hour_bucket;
Against a small synthetic requests table (three consecutive hourly buckets with own error counts of 1/3, 0/2, and 2/3 respectively), the trailing 30-day totals accumulate correctly across hours: hour 1 (3 total, 1 bad, own rate 33.33%, no prior history so trailing = own = 333.33x); hour 2 (cumulative 5 total, 1 bad, trailing rate 20% -> 200.00x, which is LOWER than hour 2's own instantaneous 0% would suggest in isolation only because it is blended with the still-elevated hour 1 inside the trailing window); hour 3 (cumulative 8 total, 3 bad, trailing rate 37.5% -> 375.00x). These trailing figures were confirmed by running the query and differ meaningfully from a naive per-hour-only calculation, which is exactly the bug in a non-trailing implementation: a single bad hour would appear to "clear" the alert the moment the next hour is healthy, when in reality 30 days of accumulated bad requests are still being carried forward.
Trade-offs and pitfalls
SQLite's window RANGE frame only supports numeric offsets, not native date/time intervals, which is why the hour timestamp is first cast to an integer hour-index; PostgreSQL supports RANGE BETWEEN INTERVAL '30 days' PRECEDING AND CURRENT ROW directly on a timestamp column, which is the more natural production choice if available. At 90 days of hourly data per service this is a modest row count for the window function, but if this pattern is applied at per-minute granularity across many services, a windowed aggregate over raw (rather than pre-aggregated hourly) rows becomes expensive fast; pre-aggregating to hourly rollups (as done here) before applying the window is the standard mitigation, and the window itself should be computed on the much smaller hourly table, never re-scanning raw per-request rows for every output hour.
Draft an error budget borrowing and repayment policy between teams, including allowed borrow amounts, repayment time windows, visibility requirements, and penalties or incentives. Provide examples of compliant and non-compliant borrowing scenarios.
Sample Answer
Error-budget borrowing lets a team spend AHEAD of its normal allocation for a specific, justified reason (a planned risky migration, a time-sensitive launch), as long as it's repaid, and the policy's whole job is making sure "borrowing" doesn't quietly become "just not tracking budget properly."
Structured elaboration
Allowed borrow amounts: capped at a modest fraction of the team's NEXT period's allocation (e.g. up to 20% of next month's budget can be borrowed against this month), preventing a team from borrowing so much that they're effectively operating with zero real margin for an extended stretch. Repayment time windows: a defined, short horizon (e.g. borrowed budget must be repaid, meaning the team operates with EXTRA caution and reduced risk-taking, within the following month), not an open-ended IOU that can persist indefinitely. Visibility requirements: every borrow event is logged in the same shared, auditable registry used for other governance actions, visible to any team or stakeholder who might reasonably want to know a team is currently operating on borrowed budget. Penalties or incentives: a team that fails to repay within the agreed window loses borrowing privileges for a defined cooldown period (e.g. two subsequent months), a real but proportionate consequence.
Worked example
Compliant scenario: Team A wants to ship a planned, moderately-risky database migration next week that they've assessed will likely consume more budget than their current month's remaining allocation supports; they formally request to borrow 15% of next month's budget, documented with the specific justification and expected repayment plan, approved via the standard lightweight sign-off, and visible to other teams sharing any affected infrastructure. Non-compliant scenario: Team B has been quietly running over budget for three consecutive months without ever formally requesting to borrow, simply treating each month's overage as if it will "sort itself out," with no visibility to anyone else and no explicit repayment plan; this is exactly the undisciplined pattern the formal borrowing policy exists to replace, since an UNTRACKED overage gives none of the planning benefit a formally-documented borrow arrangement provides.
Trade-offs and pitfalls
Without a real, felt consequence for failing to repay, "borrowing" degrades into a polite fiction that lets a team perpetually operate over budget while calling it something more palatable; the cooldown-period consequence needs to be genuinely enforced, not waived informally when a team has a good excuse each time. It's also worth deciding whether TWO teams should be allowed to simultaneously borrow against a SHARED downstream dependency's stability, since two individually-approved, modest borrow requests could combine to create a much larger combined risk than either looks like in isolation; a genuinely lightweight policy still needs at least a basic check for this kind of compounding risk across simultaneous borrow requests.
List the key factors you should consider when choosing SLO targets for latency and availability for a user-facing service. Provide a concise checklist that includes customer impact, business criticality, historical telemetry, error budget considerations, and cost/operational constraints. Give one short rule-of-thumb example for a latency SLO (e.g., choosing p95 target relative to current percentiles).
Sample Answer
Picking an SLO target well means grounding it in what users actually need and can afford, not in whatever number is easiest to hit or sounds impressively high.
Structured elaboration
A practical checklist: customer impact (how does a small change in this metric actually affect users; a payment API's latency matters differently than an internal analytics dashboard's); business criticality (does this service sit on a revenue-critical path or is it best-effort); historical telemetry (what has the service ACTUALLY achieved recently; setting a target far above or below observed reality produces either an unachievable aspiration or a meaningless rubber-stamp); error budget considerations (does the chosen target leave a workable amount of budget for normal operational risk like deploys and experiments, or does it leave none); cost/operational constraints (each additional nine of reliability costs real engineering effort and infrastructure spend, and that cost should be weighed against the actual user-facing benefit of tightening further).
Worked example
For a latency SLO, a reasonable rule of thumb is to set the target relative to CURRENTLY OBSERVED percentiles rather than an arbitrary round number: if the service's current p95 latency sits around 180ms with healthy variance, a target of "p95 < 250ms" gives real operating headroom above normal variation, whereas setting the target at "p95 < 190ms" (barely above the current observed value) leaves almost no room for ordinary day-to-day fluctuation and will generate constant, low-value alerting.
Trade-offs and pitfalls
The most common mistake is picking a target that sounds good in a slide deck (five nines) without checking whether the cost of achieving it is remotely justified by the actual user or business impact of the last increment of reliability; for many internal or best-effort services, 99% is genuinely fine and chasing 99.99% would be a poor use of engineering time. The other common mistake runs the other way: setting a target so loose (based on already-degraded historical telemetry) that it locks in mediocre reliability as the permanent baseline instead of using the target-setting exercise as an opportunity to actually improve.
Design SLOs for a background batch job pipeline (daily ETL ingest) that is allowed to run for several hours. Which SLIs would you choose (e.g., completion rate, freshness, correctness), how do you measure them, and how do you reflect them as SLOs given inherently asynchronous behavior?
Sample Answer
A batch ETL job has no request-response latency signal to lean on, so its SLIs need to be built around completion, freshness, and correctness instead, each measured once per run rather than continuously.
Structured elaboration
Completion rate: did the job finish successfully at all, measured per scheduled run (e.g. daily), as a simple success/failure signal rolled up over a window (e.g. 29 of 30 daily runs completed = 96.7% completion rate over 30 days). Freshness: how long after the expected completion time did the data actually become available, since a job that "completes" 6 hours late is a real problem even though it technically succeeded; measured as the delta between actual and expected completion timestamp per run. Correctness: did the completed run produce data passing validation checks (row-count sanity, referential integrity, reconciliation against a source total), since a job can complete on schedule and still write wrong data.
Worked example
An SLO for a daily ETL job might state: 99% completion rate over 30 days (allowing roughly 1 failed run per month), 95% of runs complete within 2 hours of the expected daily window close (freshness), and 99.9% of completed runs pass validation checks (correctness). Reflecting these as SLOs given the job's inherently asynchronous, once-a-day cadence means the "window" for the SLO is naturally the number of RUNS, not a continuous time-based rolling window the way a request-serving API's SLO would be; a 30-day SLO here really means "30 daily runs," and the statistical confidence behind any percentage is much lower with only 30 discrete samples a month than with millions of continuous requests, which is worth being explicit about when presenting the number.
Trade-offs and pitfalls
Treating a batch job's SLO like a continuous-traffic API's SLO (e.g. trying to alert on a "burn rate" mid-run before the run has even finished) doesn't map cleanly, since there's no meaningful partial signal until the run either completes or times out; the alerting strategy here is closer to "did today's run complete on time and pass validation" as a per-run check, with the SLO itself being a rollup of many such per-run checks over the month. With only 30 samples per month, a single bad run moves the SLI by over 3 percentage points, so this kind of SLO needs a genuinely different interpretation discipline than a high-volume request-based SLI: one bad night is a much bigger statistical event here than one bad minute is for a continuously-serving API.
Given a service with an availability SLO of 99.95% over a 30-day window, design an error categorization and automated mitigation strategy that uses the error budget to trigger automated actions (e.g., rate-limiting non-critical features, halting non-essential background jobs, rolling back recent deploy). Explain thresholds, processes to avoid unsafe automated actions, and how to surface these mitigations to stakeholders.
Sample Answer
Turning error budget into automated mitigation requires deciding, IN ADVANCE, which categories of error count against the budget at all, and building enough safety margin into each automated action that a false trigger doesn't cause more harm than the problem it's meant to prevent.
Structured elaboration
Error categorization: define categories (e.g. customer-caused 4xx, genuine backend 5xx, timeout-ambiguous, third-party-dependency-caused) with an explicit, documented decision for which count against the SLO; this classification scheme needs its own APPROVAL process (not a single engineer's judgment call) and a defined review cadence, since misclassifying a genuine failure as "customer error" to avoid budget impact is exactly the gaming pattern discussed elsewhere, and the scheme itself needs updating as new failure modes are discovered over time. Automated mitigation tiers, keyed to burn-rate thresholds: moderate burn triggers rate-limiting of non-critical, discretionary features (protecting core functionality by shedding non-essential load); severe or fast burn triggers halting non-essential background jobs (freeing capacity for user-facing work); very severe or sustained burn triggers an automated rollback of the most recent deploy, the most drastic and highest-confidence-required action.
Worked example
At 99.95% SLO over 30 days: moderate burn (2x sustained over 6 hours) automatically disables a non-critical background recommendation-refresh job; severe burn (6x sustained over 1 hour) automatically rate-limits a discretionary bulk-export feature; critical burn (14x sustained over 15 minutes) triggers an automatic rollback of the most recent deploy IF one occurred within the last 2 hours (correlating the burn spike with a plausible causal deploy event before rolling back, rather than rolling back blindly whenever severe burn is detected with no clear causal link). Every automated action is logged and surfaced to stakeholders via a real-time status update (not buried in an internal log), since a mitigation that silently degrades a feature without any visible communication risks confusing both users and internal stakeholders about what's currently happening and why.
Trade-offs and pitfalls
The riskiest automated action is a deploy rollback triggered without correlating the burn signal to an actual recent deploy; blindly rolling back "the most recent deploy" whenever severe burn is detected, even when the true cause was unrelated (a third-party outage, a traffic spike), needlessly reverts unrelated, healthy work and can itself introduce new risk. Every automated tier needs a documented manual override path, and periodically REVIEWING whether each mitigation tier has actually fired appropriately (not too often, not missing real events) is itself part of the process, since a mitigation policy set once and never revisited tends to drift out of calibration as the service's traffic and architecture change.
Unlock Full Question Bank
Get access to all SLIs, SLOs, SLAs, and Error Budgets interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.