SLIs, SLOs, SLAs, and Error Budgets Questions
Defining and operating reliability targets. Covers choosing service level indicators, setting service level objectives and agreements, computing and spending error budgets, and using them to drive engineering decisions. Includes negotiating reliability targets with stakeholders.
Describe the key success metrics and SLOs you defined for a production data service you built. Include quantitative thresholds (throughput, p95/p99 latency, data freshness, error rate), how you chose the thresholds based on business needs, and how you validated those SLOs during launch and operation.
Sample Answer
Choosing SLOs for a data service you actually built starts from the specific business need the service exists to serve, then works down to the quantitative thresholds, not the other way around.
Structured elaboration
Success metrics worth defining: throughput (requests or records served per unit time, confirming the service can sustain expected load), p95/p99 latency (tail experience, since a data service's consumers often run bursty batch queries where the tail matters as much as, or more than, the median), data freshness (how current the served data is relative to its source), and error rate (proportion of requests failing outright). Thresholds should be chosen against what the CONSUMING use case actually requires: a dashboard refreshed hourly can tolerate looser freshness than a real-time alerting pipeline consuming the same underlying data service.
Worked example
For a production data service feeding both an hourly-refreshed executive dashboard and a near-real-time fraud-detection pipeline: p95 latency < 200ms (chosen based on what the fraud pipeline's own SLA requires, the tighter of the two consumers), freshness < 5 minutes (again driven by the tighter consumer, even though the dashboard alone would have tolerated much looser freshness), error rate < 0.1%. Validation during launch: a soft-launch period against a subset of real consumer traffic, checking that the chosen thresholds are both ACHIEVABLE (the service can actually sustain them under real load) and SUFFICIENT (the tightest consumer's actual needs are genuinely met, confirmed by that consumer's own downstream metrics staying healthy).
Trade-offs and pitfalls
Setting one blended threshold to satisfy the AVERAGE of all consumers' needs, rather than the TIGHTEST consumer's actual requirement, is a common mistake: it can look reasonable on paper while quietly under-serving the one consumer (often the most business-critical one) whose needs were actually the tightest. It's also worth revisiting these thresholds as new consumers are added over the service's life, since a threshold validated against the original set of consumers may become insufficient once a new, more demanding consumer starts depending on the same service without anyone re-checking whether the existing SLO still covers their needs.
Design an enterprise reliability governance model that standardizes SLO ownership, error budget policies, reporting cadence, escalation paths, and incentives across multiple product lines. Include roles (product owner, SRE, engineering manager), required dashboards, and how you would enforce and audit compliance without slowing delivery.
Sample Answer
Enterprise-scale reliability governance needs a genuinely tiered structure, since the same rigid policy applied uniformly across 50 microservices of wildly different criticality either over-constrains the unimportant ones or under-protects the critical ones, and every layer of that structure needs an explicit, named owner and a real audit mechanism.
Structured elaboration
Roles: product owner (owns the business-criticality tier assignment and reporting to leadership), SRE (owns technical SLO measurement and the enforcement mechanism), engineering manager (owns the team's actual remediation work and roadmap trade-offs when budgets burn). Required dashboards: a standard, company-wide reliability dashboard template (current SLO status, burn rate, and tier per service) that every product line adopts, so an executive or auditor can compare services across lines without each team inventing its own bespoke reporting format. Standardized process elements: an explicit SLO-proposal APPROVAL workflow (who can propose a new SLO or change an existing one, and who signs off), a defined escalation and remediation-window procedure for repeated breaches (not just a one-off incident response, but what happens after the SECOND or THIRD consecutive miss), and tiered budget ALLOCATION across services of varying criticality: a concrete 3-tier example assigns critical services a strict policy triggering action at 50% budget consumed, important services at 75%, and best-effort services only at full exhaustion, with corresponding automated or manual actions at each threshold per tier. For a large-scale system (e.g. 50 microservices), budgets need explicit shared-vs-component allocation logic (does a cascading failure across several services draw from one shared pool or each service's own budget) and periodic audits to confirm the allocation still reflects actual current criticality, not a stale assessment from when the service first launched.
Worked example
For a 40-team organization with legal SLAs carrying real financial penalties: the governance model requires any SLO the model designates as SLA-relevant to go through a stricter approval path involving legal, not just engineering sign-off, with a monthly reporting cadence to a cross-functional steering committee (not just the platform team), and an escalation path where two consecutive missed SLOs trigger a mandatory remediation-window review with a defined completion deadline, escalating to executive visibility if the remediation window itself is also missed. Framed as an SLO-driven-development adoption initiative, the model explicitly ties reliability targets back to architecture decisions: a service repeatedly missing its tier's SLO despite remediation attempts should trigger an architecture review, not just another remediation cycle, since repeated tactical fixes failing to hold the target is itself evidence of a structural, not tactical, problem.
Trade-offs and pitfalls
The audit-without-slowing-delivery balance is the hardest part of this design: an overly heavy-handed compliance process (mandatory sign-offs for every minor SLO adjustment) will be quietly worked around by teams under delivery pressure, while an overly light one produces exactly the governance theater problem seen elsewhere, where the framework exists on paper but has no real teeth; the right calibration usually reserves the heaviest process (legal sign-off, executive escalation) for genuinely high-stakes changes (legally-binding SLAs, critical-tier services) while keeping best-effort-tier governance lightweight, so the total compliance burden scales with actual risk rather than being uniform across every tier.
Design a continuous reliability testing program that validates SLOs before and after releases. Include SLO/SLA validation tests, chaos engineering experiments, synthetic transactions, degraded-network tests, automated rollback triggers, and monitoring/alerting to ensure operational readiness.
Sample Answer
A continuous reliability testing program validates SLOs at multiple points in the release lifecycle, layering the classic test pyramid underneath SLO-specific validation, since a fast unit test catches a different class of problem than a slow chaos experiment, and both matter.
Structured elaboration
Foundational test-pyramid layer: unit tests (fast, isolated logic checks, the base of the pyramid, run on every commit), contract tests (confirming service interfaces match what consumers expect, catching integration breakage before it reaches a shared environment), integration tests (confirming components work together correctly, run less frequently than unit tests but still pre-merge), and system tests (full end-to-end validation in a realistic environment, the smallest and slowest layer). SLO/SLA validation tests specifically layer on top: pre-release, a synthetic-transaction suite exercising the exact user journeys the SLO cares about, confirming the release candidate meets latency/success-rate expectations before it ever reaches real traffic; post-release, continued synthetic transactions plus real production SLI monitoring, confirming the SLO actually holds once real traffic hits the new version.
Worked example
Chaos engineering experiments and degraded-network tests specifically probe RESILIENCE (not just correctness): does the system still meet its SLO when a dependency is artificially slowed or a network partition is injected, validating that the SLO's target was set with real margin for imperfect conditions, not just for a happy-path scenario. Automated rollback triggers tie this all together operationally: if post-release synthetic transactions or live SLI monitoring detect a regression, the same automated release-gating mechanism used for ordinary deploys triggers a rollback, with monitoring/alerting confirming the rollback itself actually restored healthy SLO status rather than assuming it worked. Flakiness prevention is a genuinely distinct, easy-to-neglect sub-discipline here: a test suite (at any pyramid layer) with a meaningful flaky-failure rate erodes trust in the whole program, since engineers start ignoring "test failures" that are actually just flakiness, exactly the same erosion-of-trust problem alert fatigue causes for monitoring; flaky tests need to be tracked, quarantined, and fixed with the same rigor as a genuine production incident, not tolerated as background noise.
Trade-offs and pitfalls
The test pyramid's SHAPE matters as much as having each layer present at all: an inverted pyramid (heavy reliance on slow, expensive system/chaos tests with few fast unit tests underneath) makes the whole program slow and expensive to run frequently, which works against the "continuous" part of continuous reliability testing; keeping the pyramid properly shaped (many fast unit tests, progressively fewer and slower tests at each higher layer) is what makes frequent, continuous validation actually sustainable. It's also worth being explicit that chaos experiments and degraded-network tests specifically need the SAME error-budget-aware scheduling discussed elsewhere (not run against an already-fragile production service), since this program's own testing activity shouldn't itself become a source of the very SLO breaches it exists to prevent.
Write an algorithm or Python function that computes a rolling error budget burn rate which weights incidents by severity. Inputs: list of incidents with start, end, severity (1-5), total_window_seconds, and SLO_allowed_error_seconds. Output: weighted_burn_rate. Explain your weighting scheme, complexity, and edge cases.
Sample Answer
Weighting incident duration by severity reflects that a short but severe outage can cost more in user trust and support load than a long but minor degradation of the same raw duration.
Structured elaboration
The weighted burn rate sums each incident's duration multiplied by its severity, then divides by the SLO's allowed error-seconds for the window, giving a rate expressed relative to what a "sustainable, unweighted" consumption would look like; a result above 1 means the service is burning budget faster (once severity-weighted) than the SLO's allowance supports. Complexity: the function makes a single pass over the incidents list, doing O(1) work per incident, so time complexity is O(n) where n is the number of incidents; space complexity is O(1) beyond the input list (only a running accumulator is kept, no per-incident storage is allocated).
Worked example (executed; python3, verified against a hand-computed case)
from typing import List, Dict
def weighted_burn_rate(incidents: List[Dict], total_window_seconds: float,
slo_allowed_error_seconds: float) -> float:
weighted_seconds = 0.0
for inc in incidents:
duration = max(0.0, inc['end'] - inc['start'])
weighted_seconds += duration * inc['severity']
if slo_allowed_error_seconds <= 0:
raise ValueError("slo_allowed_error_seconds must be > 0")
return weighted_seconds / slo_allowed_error_seconds
For a 30-day window at a 99.9% SLO, allowed error-seconds is 0.001×30×24×3600=2592 seconds. Two incidents, a 10-minute severity-1 blip (600s) and a 5-minute severity-5 outage (300s), give a severity-weighted total of 600×1+300×5=2100 seconds, so the weighted burn rate is 2100/2592≈0.81, confirmed by running the code; a severity-0 incident (e.g. a data point that turned out not to be a real incident) correctly contributes zero.
Trade-offs and pitfalls
The severity weights themselves are a policy choice, not a measured quantity, and teams disagree about the right multiplier (linear 1-5 versus exponential); whatever scheme is chosen should be documented and applied consistently, because otherwise this number becomes impossible to compare month over month. This function also treats all severity-5 seconds as equally bad regardless of WHEN they occurred (peak traffic vs 3am), which is a real simplification; a more sophisticated version would additionally weight by concurrent request volume, at the cost of needing traffic data alongside the incident list.
Implement (or outline) a service that rate-limits approval of feature releases based on global and per-service error budget constraints. Inputs: service_id, requested_approvals_per_hour; outputs: allowed_approvals and suggested_cooldown. Describe data model, algorithms, and how to handle burst approvals and manual overrides.
Sample Answer
This is a rate-limiter, but the thing being limited is release APPROVALS rather than requests, and the limit itself should shrink as either the global or the service-level error budget burns down.
Structured elaboration
Data model: a per-service record tracks {service_id, hard_cap_per_hour, budget_remaining_pct, rolling_approval_count, window_start_timestamp}, refreshed from the same error-budget system that computes SLO burn; a single global record tracks the analogous {global_hard_cap_per_hour, global_budget_remaining_pct, rolling_global_approval_count} shared across all services; and every approval decision (including overrides) is written to an append-only approval-log table {service_id, timestamp, requested, allowed, cooldown, manual_override, justification} so the audit trail exists independent of the current-state records, which only hold the latest rolling counts. Two caps apply simultaneously: a hard per-service cap (protects that service from too-frequent changes) and a hard global cap (protects shared deploy infrastructure and blast radius across all services); the function must always honor the tighter of the two. On top of the hard caps, each cap is scaled down as its corresponding budget burns: above 50% remaining budget the full hard cap applies, and below 50% it ramps linearly to zero at 0% remaining, so approvals get progressively harder to obtain exactly when the service or the whole fleet is already unhealthy. A manual override exists for genuine emergencies (a critical security patch), but it must still respect the hard caps (never grant more than the absolute ceiling) so a panicked override can't itself become the cause of an outage.
Worked example (executed; python3, verified against 5 scenarios)
import math
def approve_release(service_id, requested_approvals_per_hour, global_budget_remaining_pct,
service_budget_remaining_pct, global_cap_per_hour=20,
service_cap_per_hour=5, manual_override=False):
def budget_scaled_cap(hard_cap, remaining_pct):
if remaining_pct >= 50:
return hard_cap
return math.floor(hard_cap * (remaining_pct / 50.0))
if manual_override:
return min(requested_approvals_per_hour, global_cap_per_hour, service_cap_per_hour), 0
g_cap = budget_scaled_cap(global_cap_per_hour, global_budget_remaining_pct)
s_cap = budget_scaled_cap(service_cap_per_hour, service_budget_remaining_pct)
allowed = max(0, min(requested_approvals_per_hour, g_cap, s_cap))
binding = min(g_cap, s_cap)
if binding <= 0:
cooldown = 60
elif requested_approvals_per_hour > binding:
cooldown = math.ceil(60 / max(binding, 1))
else:
cooldown = 0
return allowed, cooldown
Confirmed behavior: a request for 3 approvals/hour with both budgets healthy passes through unchanged (allowed=3, cooldown=0). A burst request of 10/hour against a healthy 5/hour service cap is capped at 5 with a 12-minute suggested cooldown between bursts. When the service budget alone drops to 10% remaining, the effective service cap shrinks to 1/hour even though the request and the global budget are unchanged, correctly localizing the throttle to the unhealthy service. A manual override at 5%/5% budget still caps at 5 (the hard service cap), not the requested 10, so the override cannot exceed the absolute ceiling.
Trade-offs and pitfalls
A pure per-hour cap can be gamed by bursting right at the top of every hour boundary; a production version should use a sliding window (or a token-bucket refill) rather than a fixed hourly reset. The linear budget-scaling ramp is a simplification; teams often prefer a steeper (e.g. quadratic) ramp near zero so the last sliver of budget is protected much more aggressively than the middle of the range. Logging every override with its justification is not optional: a bypass mechanism with no audit trail is exactly the kind of control that gets silently abused under deadline pressure.
Unlock Full Question Bank
Get access to all SLIs, SLOs, SLAs, and Error Budgets interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.