SLIs, SLOs, SLAs Definition and Implementation Questions
Understanding Service Level Indicators (SLIs - what you measure), Service Level Objectives (SLOs - targets you set), and Service Level Agreements (SLAs - commitments to customers). At senior level, design SLOs that align with business requirements and user expectations. Choose meaningful SLIs like availability, latency, error rate. Understand how SLOs drive reliability decisions, allocation of engineering effort, and error budgets. Design monitoring to track SLI achievement. Address multi-tiered SLOs for different service tiers or customer segments.
HardTechnical
67 practiced
Medium/hard coding: Given an events table:Design a query or streaming approach to compute daily successful payment rate accounting for out-of-order arrivals. Explain windowing, deduplication, and state you maintain.
sql
events(txn_id text, event_time timestamptz, event_type text)
-- event_type in ('initiated','processed','confirmed')Sample Answer
Approach (summary)- We want daily successful payment rate = confirmed / initiated per calendar day (by event_time) while handling out-of-order arrivals and duplicates. Use event-time windows with watermarks, per-transaction state to deduplicate & track final state, and emit daily aggregates once window lateness is tolerated.Streaming design (Apache Flink-style pseudocode)- Maintain keyed state by txn_id: lastSeenEventType, lastSeenEventTime- Maintain per-day aggregates (initiated_set, confirmed_set) or counters plus dedup sets (or bloom filters for memory).- Use event-time tumbling window (1 day, aligned to UTC) with watermark = currentEventTime - maxAllowedLateness (e.g. 2 hours).- On event: - Lookup txn state; if event_time older than stored max for txn, ignore (or reconcile if needed). - If new event_type changes transaction status from not-confirmed -> confirmed, update per-day counters: decrement prior day's initiated if moving days, increment confirmed for event_date. - Update txn state (type, event_time, event_date).- On window close (watermark passes window_end + allowed_lateness) emit successful_rate = confirmed_count / initiated_count.Example SQL (batch / materialized view / CDC approach)- If you can run periodic batch/reconciliation (Postgres + CTE):Key concepts / trade-offs- Windowing: use event-time tumbling windows; watermarks handle out-of-order. Choose allowed lateness based on business SLA.- Deduplication/state: keep the latest event per txn_id (in streaming, keyed state); use TTL on state to bound memory (e.g., keep txn state for window_end + allowed_lateness + retention).- Scalability: replace exact sets with probabilistic structures (Bloom/HyperLogLog) if cardinality huge but accept false positives/estimates.- Correctness vs latency: larger allowed lateness increases correctness but delays SLI; pick value to balance.- Alerting: compute SLI over sliding daily windows and trigger if rate below threshold; surface delayed corrections in metrics dashboards and postmortems.
sql
WITH latest AS (
SELECT DISTINCT ON (txn_id)
txn_id,
event_type,
event_time,
(event_time AT TIME ZONE 'UTC')::date AS event_date
FROM events
ORDER BY txn_id, event_time DESC -- latest event per txn
),
per_day AS (
SELECT
event_date,
COUNT(*) FILTER (WHERE event_type = 'initiated') AS initiated,
COUNT(*) FILTER (WHERE event_type = 'confirmed') AS confirmed
FROM latest
GROUP BY event_date
)
SELECT event_date, confirmed, initiated,
CASE WHEN initiated=0 THEN NULL ELSE confirmed::decimal/initiated END AS success_rate
FROM per_day
ORDER BY event_date;MediumTechnical
57 practiced
Coding (Python): Implement a function that, given a list of timestamped request outcomes [(timestamp_iso, success_bool)], computes the error budget burn rate over a sliding 1-hour window sampled every 5 minutes. Keep time complexity O(n). Describe assumptions and edge handling (e.g., sparse data).
Sample Answer
Approach:- Parse and sort input events by timestamp.- Slide a 1-hour window and sample every 5 minutes. Maintain a deque of events inside the window and running counts of total and errors; update by popping old events and pushing new events as sample time advances.- Burn rate = (error_rate_in_window) / slo_error_rate where error_rate_in_window = errors/total. Expose parameter slo_target (e.g., 0.001 for 99.9% SLO).- If no requests in window, return None (or 0) — configurable.Key points:- O(n + m) time where n = events, m = samples; in practice sliding updates are O(1) per event/sample so O(n).- Space O(n) worst-case for deque within a 1-hour window.Assumptions & edge handling:- Input timestamps may be sparse; when a window has zero requests returns None (configurable to 0).- Events with identical timestamps included.- slo_error_rate must be >0.Alternatives:- Bucket-based histogram (fixed-size per-minute counts) for very high event rates to reduce memory.
python
from collections import deque
from datetime import datetime, timedelta
from typing import List, Tuple, Optional
def compute_burn_rate(events: List[Tuple[str, bool]],
slo_error_rate: float = 0.001,
sample_interval_min: int = 5,
window_min: int = 60) -> List[Tuple[str, Optional[float]]]:
"""
events: list of (ISOtimestamp, success_bool). success_bool=True => success.
Returns list of (ISO sample time, burn_rate or None if no data).
"""
if not events:
return []
# parse and sort
parsed = [(datetime.fromisoformat(t), not ok) for t, ok in events] # store is_error
parsed.sort(key=lambda x: x[0])
start = parsed[0][0]
end = parsed[-1][0]
sample_interval = timedelta(minutes=sample_interval_min)
window = timedelta(minutes=window_min)
q = deque() # (ts, is_error)
total = 0
errors = 0
res = []
i = 0 # index into parsed events
sample_time = start.replace(second=0, microsecond=0)
# ensure first sample covers first event
while sample_time + window < start:
sample_time += sample_interval
while sample_time <= end:
window_start = sample_time
window_end = sample_time + window
# add events up to window_end
while i < len(parsed) and parsed[i][0] < window_end:
ts, is_err = parsed[i]
q.append((ts, is_err))
total += 1
errors += 1 if is_err else 0
i += 1
# remove events before window_start
while q and q[0][0] < window_start:
_, is_err = q.popleft()
total -= 1
errors -= 1 if is_err else 0
if total == 0:
burn = None
else:
err_rate = errors / total
burn = err_rate / slo_error_rate
res.append((window_start.isoformat(), burn))
sample_time += sample_interval
return resEasyTechnical
48 practiced
Provide an example error budget policy for an API with an SLO of 99.9% availability over a 28-day window. Include actions at three thresholds (e.g., caution, enforce, stop releases), responsibilities, and escalation path. Be specific about time windows and stakeholders.
Sample Answer
Context: SLO = 99.9% availability over a 28-day window → allowed downtime = 0.1% of 28 days = 28*24*60*0.001 = 40.32 minutes (error budget = 40.32 min of unavailability or equivalent error-rate budget).Policy (thresholds measured as cumulative error-budget consumed in the rolling 28-day window and reported hourly):1) CAUTION — 30% consumed (~12.1 minutes)- Trigger: Monitoring alerts when consumption ≥30% for any 3 consecutive hourly reports.- Actions: - SRE on-call investigates root causes within 1 hour; increase sampling/alert fidelity (trace, latency, dependency health). - Notify Service Owner and Tech Lead via Slack + incident channel. - Require weekly reliability checkpoint: include reliability ticket(s) in next sprint planning.- Responsibilities: SRE on-call executes investigation; Service Owner prioritizes fixes; Engineering team schedules follow-up.- Escalation: If unresolved degradation >24 hours, escalate to Engineering Manager.2) ENFORCE — 70% consumed (~28.2 minutes)- Trigger: Consumption ≥70% sustained for 2 hours or spikes causing projected >100% before end of 28-day window.- Actions: - Open a Reliability Incident (RI) within 30 minutes; run focused mitigation (traffic shaping, rate-limits, temporary feature flags). - Pause non-critical changes and deployments to the API until root cause is identified and mitigations validated. - Daily status reports to Product Manager and Platform Reliability Lead.- Responsibilities: SRE lead coordinates mitigations and RCA; Release Manager halts scheduled releases; Engineering teams implement fixes.- Escalation: If not contained within 12 hours, escalate to Head of Engineering and Product Director for resourcing decisions.3) STOP RELEASES — 100% consumed (~40.32 minutes)- Trigger: Consumption ≥100% or predicted exhaustion in <48 hours.- Actions: - Immediate deployment freeze for the service and dependent services. Revert recent changes if they correlate with errors. - All hands on containment: SREs + engineers operate under an incident command (IC) within 15 minutes. - Initiate full RCA and create timeline of mitigating actions; communicate customer impact and ETA to stakeholders and support.- Responsibilities: Incident Commander (SRE lead) runs the response; Release Manager enforces freeze; Service Owner coordinates postmortem remediation.- Escalation: Notify CTO and Customer Success if customer-impacting; schedule executive update within 4 hours.Operational details:- Measurement: hourly rolling 28-day window; dashboard + automated alerts (PagerDuty for on-call; Slack channel for broader notifications).- Communication: predefined incident templates, weekly reliability reports, and a mandatory blameless postmortem if >30% budget consumed or any STOP action taken.- Success criteria: restore projected consumption below 50% within 48 hours for ENFORCE; complete RCA and remediation plan within 5 working days for STOP RELEASES.
MediumSystem Design
93 practiced
Design a monitoring strategy so that failures in internal dependencies (datastore, auth service) can be mapped and correlated to the user-facing service SLOs. Explain instrumentation, dashboards, and how to surface root-cause vs symptom.
Sample Answer
Requirements & goals:- Map failures in internal dependencies (datastore, auth) to user-facing SLOs (latency/availability).- Fast RCA: distinguish root-cause (dependency outage) from symptom (SLO breach).- Low alert noise, actionable runbooks.Instrumentation- Metrics: instrument each service with three Golden Signals — latency (p50/p95/p99), error rate, saturation (CPU, DB connections). Export via Prometheus-compatible metrics with labels: service, endpoint, backend_dependency, region, shard.- Tracing: use OpenTelemetry to propagate trace context across services. Add span attributes: dependency.name, dependency.type, call.success, peer.address, db.statement (sanitized).- Logs: structured JSON logs with trace_id, span_id, user_id, request_id, error_code, dependency_outcome.- Synthetic checks: end-to-end availability tests and synthetic dependency probes (e.g., read/write to datastore, auth token validation) from multiple regions.Dashboards & Mapping- Top-level SLO dashboard: SLO burn rate, error budget, user-impacting request volume. Clicking SLO violation drills into service view.- Service view: Golden Signals per endpoint, per region. Include an annotated dependency graph showing percentage of requests calling each dependency and their contribution to failures.- Dependency health panel: aggregate dependency error rates and latencies with heatmap by time and region.- Traces panel: top slowest/error traces; filter by dependency.name to see spike correlation.- Correlation widgets: use trace sampling to compute "fraction of SLO-failing requests that include dependency X errors" (e.g., 70% of 500ms+ requests had DB 5xx in trace).Alerting & Triage- Two alert tiers: 1. Dependency severity alerts (on dependency error rate, latency, saturation) routed to owning team — alert only when correlated to increased user errors or high burn rate. 2. SLO breach alerts (user-facing) trigger incident channel and immediately include correlated dependency signals and top traces.- Automated enrichment: when SLO alert fires, attach top 10 traces, top failing dependency, synthetic probe status.Surface Root-cause vs Symptom- Correlate temporally: require that dependency error/latency spike precedes/increases fraction of failing user requests.- Use trace-based causal scoring: for each failing user request, mark if dependency span failed or was slow beyond threshold; compute % contributing.- If >X% (e.g., 60%) of failing requests include the dependency failure within causal window, mark dependency as probable root-cause; otherwise treat as symptom.- Confirm with dependency health (synthetic probes, metrics) and owner confirmation.Runbooks & Playbooks- Predefined runbooks: steps to mitigate (circuit-break, increase replicas, failover read-only, roll back release, rotate auth keys), how to reduce user impact (rate-limit, cache responses).- Post-incident: add mapping of dependency->SLO impact, tighten instrumentation where gaps found.Trade-offs & notes- Sampling: high-volume tracing needs adaptive sampling focused on errors/slowness.- Label cardinality: avoid high-cardinality labels on metrics; push details to logs/traces.- Start with high-level dashboards, iterate on dependency correlation thresholds from historical incidents.This strategy gives fast, evidence-based RCA: metrics for signal, traces for causality, logs for detail, dashboards for exploration, and automated correlation to separate root-cause from downstream symptoms.
HardSystem Design
58 practiced
Hard: Design a capacity planning and cost model to ensure SLO compliance over the next 12 months given projected traffic growth of 4x. Explain how you'd model headroom, redundancy, auto-scaling policies, cost trade-offs, and how SLO targets influence these choices.
Sample Answer
Requirements & constraints:- Maintain defined SLOs (e.g., 99.95% availability, p95 lat <200ms) while traffic grows 4x over 12 months.- Minimize cost subject to risk (error budget), support rolling upgrades, tolerate AZ failures.High-level approach:1) Forecast demand curve (monthly): multiply baseline qps by 4x, add seasonality and 95th-percentile spikes. Produce P50/P95/P99 traffic scenarios.2) Capacity model per tier (frontend, app, DB): convert qps → resource units using performance profiles (benchmarks): e.g., one app pod handles 500 qps at p95 <200ms under typical load.3) Headroom & redundancy: - Headroom = capacity * (1 + H), H derived from target error budget. For 99.95% SLO, use H=40–50% (higher for stateful DBs), lower for stateless tier with fast autoscale. - Redundancy: N+1 per AZ; design to survive full AZ loss (serve at degraded SLO via traffic shedding or degraded features).4) Autoscaling policies: - Horizontal Pod Autoscaler on CPU/RPS with predictive/step scaling: scale-up aggressively on p95 traffic rise; scale-down conservatively with cooldowns. - For stateful services, use pre-warmed standby replicas and scheduled scaling based on forecast. - Include warm pools and surge capacity for deploys/rollbacks.5) Cost model: - Compute expected monthly cost = sum(resource_units_needed * unit_cost) across scenarios. Model on-demand vs reserved/savings plans: buy reservations for base load (e.g., P50) and rely on on-demand for p95/p99 spikes. - Include cost of redundancy, buffer, and amortized CI/CD/warmpools.6) SLO influence & trade-offs: - Tighter SLOs → larger headroom, more reserved capacity, higher cost. Use error budget to accept occasional degraded performance and reduce cost by leaning on autoscale and traffic shaping. - For low-latency frontends, prefer over-provision + reservations. For batch/non-critical, use more aggressive scale-to-zero and spot instances.7) Monitoring & operationalization: - Continuous validation: run chaos tests, autoscaler tuning experiments, and compare actual to forecast monthly. Alert when capacity utilization > target thresholds. - Recompute headroom monthly with updated traffic and remaining error budget.Resulting deliverable:- A spreadsheet/model with inputs (baseline qps, growth, perf per unit, cost/unit, H), outputs (nodes/pods per month, cost bands), suggested reservation strategy, autoscaler rules, and runbook for exceeding capacity or using error budget.
Unlock Full Question Bank
Get access to hundreds of SLIs, SLOs, SLAs Definition and Implementation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.