SLIs, SLOs, SLAs, and Error Budgets Questions
Defining and operating reliability targets. Covers choosing service level indicators, setting service level objectives and agreements, computing and spending error budgets, and using them to drive engineering decisions. Includes negotiating reliability targets with stakeholders.
Given a Prometheus histogram metric http_request_duration_seconds_bucket{job='api',route='/checkout',le='0.2'}, write PromQL expressions to compute (a) the fraction of requests under 200ms over the last 5 minutes, and (b) an approximation of p95 latency over the last 1 hour. Explain limitations of your queries.
Sample Answer
Prometheus histograms let you compute both a fraction-under-threshold metric and a full percentile from the same underlying bucket counters, using two different query patterns.
Structured elaboration
The fraction-under-200ms is simply the rate of the le="0.2" cumulative bucket divided by the rate of the le="+Inf" (total) bucket, since Prometheus histogram buckets are cumulative by construction. An approximate p95, by contrast, uses histogram_quantile, which linearly interpolates within whichever bucket the target quantile falls into; its accuracy is bounded by how many bucket boundaries exist near that quantile; a coarse bucket layout with few boundaries near p95 gives a much rougher estimate than one with several closely-spaced boundaries there.
Worked example (executed; promtool unit test against a real Prometheus binary)
sum(rate(http_request_duration_seconds_bucket{job="api",route="/checkout",le="0.2"}[5m]))
/
sum(rate(http_request_duration_seconds_bucket{job="api",route="/checkout",le="+Inf"}[5m]))
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket{job="api",route="/checkout"}[1h])) by (le)
)
Run against synthetic cumulative bucket counters (80/95/99/100 cumulative observations at le=0.1/0.2/0.5/+Inf respectively), the fraction-under-200ms query correctly returned 0.95 (95 of 100 total observations fall at or under 200ms), and histogram_quantile returned approximately 0.2 seconds, correctly landing right at the 0.2-second bucket boundary where the 95th percentile actually sits in this synthetic distribution.
Trade-offs and pitfalls
histogram_quantile's accuracy degrades badly if your bucket boundaries are sparse near the quantile you care about; if most of your buckets are clustered below 100ms but your real p95 lives up around 800ms, the interpolation between two far-apart boundaries can be wildly wrong, so bucket boundaries should be chosen with the SLO's target percentile in mind, not just convenient round numbers. Both queries here also aggregate ACROSS all instances of the job via the outer sum(); if you need per-instance visibility (e.g. to find one bad pod dragging down the aggregate), you must keep the instance label in the by () clause instead of summing it away.
Your company must produce auditable SLA reports to customers and regulators. Design the data retention, immutability, and reporting pipeline so that SLA measurements are tamper-evident and reproducible. Include backup, timezone handling, and legal considerations.
Sample Answer
Auditable SLA reporting needs the underlying data pipeline treated with the same rigor as a financial reporting system: immutable, reproducible, and legally defensible, since a regulator or a disputing customer will eventually ask exactly how a number was produced.
Structured elaboration
Data retention and immutability: raw measurement data (not just the computed monthly percentage) should be retained for the full period any SLA dispute could reasonably be raised (often driven by contractual or regulatory requirements, commonly several years), stored in an APPEND-ONLY or otherwise tamper-evident form (e.g. write-once storage, or a cryptographic hash chain over sequential report batches) so a later modification to historical data would be detectable rather than silently possible. Reporting pipeline: the computation from raw data to the published SLA percentage needs to be fully REPRODUCIBLE, meaning the exact query/computation logic used for a given historical report is itself versioned and retained, so re-running the same computation against the same retained raw data later produces an identical result, not a different one due to a since-changed query.
Worked example
Timezone handling: every raw timestamp needs to be stored with an explicit, unambiguous timezone (UTC is the standard choice) rather than a local time that could be ambiguous around a daylight-saving transition; a report boundary defined as "midnight local time" needs the LOCAL timezone convention documented explicitly and applied consistently, since inconsistent timezone handling is a classic, embarrassing source of an SLA dispute where two parties' independently-computed numbers disagree purely due to a boundary-handling bug, not a real measurement discrepancy. Backup: the raw retained data itself needs backup with the same immutability guarantee as the primary copy, since a backup that CAN be silently altered defeats the tamper-evidence of the primary.
Trade-offs and pitfalls
Legal considerations often drive the retention period and evidentiary format more than pure engineering preference; the specific requirements (how long to retain, what counts as sufficiently tamper-evident, what documentation a regulator would accept) should be confirmed with legal/compliance BEFORE the pipeline is built, not retrofitted after a dispute reveals the existing pipeline doesn't meet the actual evidentiary bar required. It's also worth being explicit that "reproducible" means genuinely re-running the ORIGINAL computation logic against the ORIGINAL raw data, not just re-generating a similar-looking report from today's (possibly since-updated) query logic against the same historical data, which could silently produce a different number even with unchanged raw inputs.
Design an automated chaos engineering program that validates SLO resilience without burning through excessive error budget. Explain experiment selection, blast-radius controls, scheduling relative to error budget, and safety checks that abort experiments when risk thresholds are exceeded.
Sample Answer
A chaos program that itself burns significant error budget defeats its own purpose; the design needs experiment SELECTION, blast-radius controls, and SCHEDULING all explicitly aware of the current budget position, not run on a fixed calendar regardless of reliability state.
Structured elaboration
Experiment selection: prioritize experiments validating the failure modes MOST likely to actually occur and MOST likely to be poorly handled if they do (informed by past incident data, not just whatever's easiest to simulate), since a chaos program's value is proportional to how representative its experiments are of real risk, not how many experiments it runs. Blast-radius controls: start with the smallest possible scope (a single instance, a tiny percentage of traffic) and expand only after validating safety at each smaller scope, with a hard cap on how much of the SYSTEM's actual capacity or traffic any single experiment can touch. Scheduling relative to error budget: experiments should be SUSPENDED automatically when the target service's error budget is already low (running a deliberate failure injection against a service that's already struggling to meet its SLO compounds risk exactly when it's least affordable), resuming only once the budget position has recovered to a healthy level. Safety checks: automated abort criteria (a real-time SLI monitor watching the experiment's blast radius, aborting immediately if observed impact exceeds a pre-defined threshold, faster than a human could react) are mandatory, not optional, for any experiment run against production.
Worked example
Before scheduling a new chaos experiment against a service, the system checks that service's current error-budget remaining percentage; if it's below a defined floor (e.g. 30%), the experiment is automatically deferred and rescheduled for after the budget position recovers, logged as a deferred (not skipped) experiment so the program's coverage tracking still reflects that it's owed. During an active experiment, a real-time SLI monitor watches for the target's error rate or latency crossing an abort threshold; if it does, the experiment auto-aborts and rolls back the injected failure condition immediately, well before the service's own SLO-alerting would have fired, since the chaos program's OWN safety mechanism needs to be faster and more conservative than the production alerting it's testing.
Trade-offs and pitfalls
A chaos program that runs on a fixed schedule regardless of current reliability state risks actively causing (not just testing for) an SLO breach during an already-fragile period, which is a real, avoidable own-goal; budget-aware scheduling is the core discipline that distinguishes a mature chaos program from a naive "randomly break things periodically" one. It's also worth deferring, not silently skipping, an experiment that gets deferred due to low budget; if deferred experiments are just dropped rather than tracked and rescheduled, the program's actual coverage silently degrades over time in exactly the periods (services under the most reliability stress) where validating resilience arguably matters most.
Draft an error budget borrowing and repayment policy between teams, including allowed borrow amounts, repayment time windows, visibility requirements, and penalties or incentives. Provide examples of compliant and non-compliant borrowing scenarios.
Sample Answer
Error-budget borrowing lets a team spend AHEAD of its normal allocation for a specific, justified reason (a planned risky migration, a time-sensitive launch), as long as it's repaid, and the policy's whole job is making sure "borrowing" doesn't quietly become "just not tracking budget properly."
Structured elaboration
Allowed borrow amounts: capped at a modest fraction of the team's NEXT period's allocation (e.g. up to 20% of next month's budget can be borrowed against this month), preventing a team from borrowing so much that they're effectively operating with zero real margin for an extended stretch. Repayment time windows: a defined, short horizon (e.g. borrowed budget must be repaid, meaning the team operates with EXTRA caution and reduced risk-taking, within the following month), not an open-ended IOU that can persist indefinitely. Visibility requirements: every borrow event is logged in the same shared, auditable registry used for other governance actions, visible to any team or stakeholder who might reasonably want to know a team is currently operating on borrowed budget. Penalties or incentives: a team that fails to repay within the agreed window loses borrowing privileges for a defined cooldown period (e.g. two subsequent months), a real but proportionate consequence.
Worked example
Compliant scenario: Team A wants to ship a planned, moderately-risky database migration next week that they've assessed will likely consume more budget than their current month's remaining allocation supports; they formally request to borrow 15% of next month's budget, documented with the specific justification and expected repayment plan, approved via the standard lightweight sign-off, and visible to other teams sharing any affected infrastructure. Non-compliant scenario: Team B has been quietly running over budget for three consecutive months without ever formally requesting to borrow, simply treating each month's overage as if it will "sort itself out," with no visibility to anyone else and no explicit repayment plan; this is exactly the undisciplined pattern the formal borrowing policy exists to replace, since an UNTRACKED overage gives none of the planning benefit a formally-documented borrow arrangement provides.
Trade-offs and pitfalls
Without a real, felt consequence for failing to repay, "borrowing" degrades into a polite fiction that lets a team perpetually operate over budget while calling it something more palatable; the cooldown-period consequence needs to be genuinely enforced, not waived informally when a team has a good excuse each time. It's also worth deciding whether TWO teams should be allowed to simultaneously borrow against a SHARED downstream dependency's stability, since two individually-approved, modest borrow requests could combine to create a much larger combined risk than either looks like in isolation; a genuinely lightweight policy still needs at least a basic check for this kind of compounding risk across simultaneous borrow requests.
For a distributed database with read replicas and eventual consistency, propose SLIs and SLOs that reflect client-visible correctness and freshness. Explain measurement strategies that account for replication lag and how to alert on serious regressions.
Sample Answer
A distributed database with read replicas needs SLIs that go beyond plain availability, since the interesting failure mode here is a client getting a technically-successful but STALE or inconsistent read, which a naive uptime check would never catch.
Structured elaboration
Candidate SLIs: (1) read success rate - proportion of read requests that return without error, the standard baseline; (2) replication lag - time delta between a write committing on the primary and becoming visible on a replica, measured continuously per replica; (3) staleness-at-read - a more direct, client-visible measure: for reads actually served, what fraction were served within an acceptable staleness bound (e.g. under 1 second lag) versus served from a replica that was further behind. The third SLI is the one that most directly reflects "client-visible correctness and freshness," since replication lag alone is an internal signal that doesn't tell you how many actual client reads were affected by it.
Worked example
An SLO might target: 99.9% read availability, and separately, 99.5% of reads served with less than 1 second of replication lag, measured over a 30-day rolling window. If a specific replica falls behind by 30 seconds during a load spike, the staleness-at-read SLI (not just the replication-lag metric) reveals what fraction of ACTUAL client reads during that window were affected, versus reads that happened to be routed to a healthier replica; alerting is set on a serious REGRESSION in that fraction (e.g. staleness-at-read dropping below 98% over a 15-minute window), not on lag crossing an absolute threshold in isolation, since a brief lag spike with no reads actually routed to the lagging replica during that window has no client impact at all.
Trade-offs and pitfalls
Measuring replication lag alone and alerting on it directly is a common but incomplete approach: it conflates an internal system-health signal with actual user impact, and can either over-alert (lag spikes briefly with no client ever affected) or under-alert (if lag is measured as an average across replicas, one severely-lagging replica can hide inside a healthy average while still serving stale data to whichever unlucky clients get routed to it). Attributing which replica actually served each read (so staleness-at-read can be computed per-request, not just estimated from an aggregate lag number) requires the read path to tag each response with which replica served it, which is easy to skip in early instrumentation and expensive to retrofit later.
Unlock Full Question Bank
Get access to all SLIs, SLOs, SLAs, and Error Budgets interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.