Site Reliability Engineering Fundamentals Questions

Covers foundational site reliability engineering concepts that interviewers expect all candidates to understand. Topics include Service Level Objectives and Service Level Indicators and how they relate to availability targets and measurable system health, the notion of error budgets and trade offs between velocity and reliability, incident management including detection, escalation, on call rotations, and blameless postmortems, the importance of monitoring and observability for alerting and root cause analysis, basic deployment and rollback strategies, and an automation mindset to reduce toil. Candidates should be able to explain these ideas at a conceptual level, discuss how they influence decision making, and reference common practices used to improve reliability.

HardSystem Design

59 practiced

You must forecast capacity for the next 12 months for a service with seasonality and occasional spikes. Describe the statistical and engineering approaches you would use (trend analysis, percentiles, headroom, autoscaling policy), how to validate forecasts, and how to translate forecasts into procurement or cloud cost planning.

Sample Answer

Requirements & constraints:- Forecast 12 months capacity for a service with clear weekly/monthly seasonality + occasional spikes.- Objectives: meet SLOs (latency/error targets), minimize cost, provide procurement lead times (e.g., 90 days for reserved capacity) and cloud budgeting.- Inputs: historical telemetry (QPS, p95/p99 latency, CPU/mem, request mix), business events calendar, marketing/feature roadmap, anomaly/incident history.

High-level approach:1. Data preparation- Clean and aggregate telemetry to multiple resolutions (1m, 5m, hourly, daily).- Annotate events (deploys, promotions, outages) and flag maintenance windows.- Decompose by traffic class (baseline, batch, spikes) and critical endpoints.

2. Statistical forecasting- Baseline + seasonality: use additive models (Prophet) or SARIMAX/ETS for monthly+weekly patterns.- Trend modeling: piecewise linear trend with changepoint detection (to capture growth shifts).- Spikes: model tail behavior separately — fit a peaks-over-threshold (POT) using generalized Pareto or use extreme value theory to estimate p99.9/p99.99.- Scenario percentiles: produce forecasted percentile curves (median, p75, p90, p95, p99, p99.9) rather than single mean.

3. Engineering (capacity translation)- Convert request-level forecasts to resource need using resource-per-request profiles from performance tests or microbenchmarks (e.g., 95th-percentile CPU/latency per request).- Apply headroom policy: required_capacity = forecasted_percentile * (1 + safety_margin) + planned_batch_capacity. Choose percentile per SLO and risk appetite (e.g., p95 for cost-sensitive, p99.9 for critical paths).- Autoscaling policy: combine reactive (CPU/RPS-based HPA) with predictive scaling: schedule scale-outs ahead of expected demand using forecasted spikes and warm-up times. Use cooldowns and rate-limited scale changes to avoid oscillation.- Reserve vs on-demand: map predictable baseline to reserved/committed capacity; map volatile spikes to autoscaling (on-demand) or spot with fallbacks.

4. Validation & feedback- Backtest: run historical backtesting with sliding windows; compute coverage (how often actual <= forecast percentile), RMSE, MAPE, and calibration for tail percentiles.- Shadow/prediction deployment: run predictive autoscaling in “observe-only” for 4–8 weeks and compare actions to real system.- Post-incident analysis: after spikes/incidents, update models and annotate causes; retrain cadence (weekly for recent trends, monthly full retrain).- Alerts on model drift: monitor residuals, increase alarm if error exceeds threshold.

5. Procurement & cost planning- Produce capacity plan by month with resource types (VM families, container node counts, storage IOPS) and associated percentiles.- For 12-month budgeting: propose tiered purchases — purchase committed discounts for baseline (e.g., 60–80% of median), short-term commitments for expected peaks during known events, keep buffer for on-demand/autoscaling for tail.- Include cost scenarios: base-case (median), high-case (p95), extreme (p99.9) with probabilities and expected cost delta.- Translate into procurement artifacts: recommended commit quantities, purchase timing (lead time), and contingency fund for on-demand overage.

Trade-offs & risks:- Heavier tail protection increases cost; use SLO-driven percentile selection.- Predictive scaling reduces cold-starts but risks overprovisioning if forecasts are wrong — mitigate with short warm-ups and graceful degradation.- Reserved capacity reduces unit cost but reduces flexibility; use mixed strategy.

Deliverables:- Forecast dashboard (percentiles + prediction intervals), validation metrics, autoscaling rules (documented), and monthly procurement plan with cost scenarios and recommended commits.

HardTechnical

94 practiced

Explain the causes of metrics cardinality explosion and list practical mitigation strategies (label design, cardinality limits, histograms/aggregations, sampling). Provide an example of a harmful label pattern and a corrected version.

Sample Answer

Causes:- High-cardinality label values (user IDs, request IDs, timestamps, UUIDs) attached to metrics so each unique value creates a separate time series.- Unbounded dimensions from routing, query params, or dynamic tags (e.g., ?session=).- Tagging at wrong frequency (per-request instead of per-aggregation window).- Instrumentation that mirrors entire objects (JSON blobs) into labels.- Multi-dimensional joins in dashboards that expand combinations exponentially.

Practical mitigations:- Label design: only include dimensions you will meaningfully filter/aggregate on. Keep label values low-cardinality and well-defined (env, region, endpoint-group).- Cardinality limits: enforce quotas in ingestion pipeline or use Prometheus relabel_rules to drop/replace labels (rejecting or truncating values).- Use histograms/summaries for distributions (latency): they record buckets or quantiles instead of separate series per value.- Pre-aggregate: record counters/gauges at higher aggregation (per-service, per-endpoint-class) rather than per-user.- Sampling: sample high-volume events (trace/special metrics) and annotate sample rate so you can scale counts.- Tag normalization: map dynamic values to stable buckets (e.g., status code groups, endpoint patterns).- Alerting rules: base alerts on aggregated series, not raw high-cardinality labels.- Monitoring tooling: use remote write adapters that drop noisy labels and expose cardinality metrics.

Harmful pattern (bad):http_request_duration_seconds{user_id="user-12345", request_id="a1b2c3", path="/api/v1/item/98765"}

Why bad: user_id, request_id, and full resource id make series explode.

Corrected pattern (good):# record latency per endpoint class and statushttp_request_duration_seconds_bucket{endpoint="/api/v1/item/:id", status_group="2xx"} and record user-specific counters separately at lower cadence or via sampled telemetry.

Concretely, use relabeling to replace dynamic IDs with placeholders, use histograms for latency, and enforce ingestion limits to prevent runaway costs.

HardTechnical

53 practiced

Design a tabletop incident-response exercise for a cross-functional team. Include objectives, roles to involve, sequence of injects, success criteria, evaluation rubric, and how you'd run a blameless after-action. Explain how often to run these exercises and how to incorporate learnings into runbooks.

Sample Answer

Objectives:- Validate cross-team coordination, decision-making, and tooling for high-severity incidents (P1/P0).- Exercise detection, escalation, mitigation, communication, and rollback/playbook execution.- Identify gaps in runbooks, on-call handoffs, and monitoring/alert fidelity.

Roles to involve:- Incident Commander (SRE lead)- Service Owner / Engineering representative- On-call Engineers (SRE + App)- Product Manager- QA/Release Engineer- Security Lead (if breach scenario)- Communications/PR or Customer Ops- Observability/Monitoring owner

Sequence of injects (45–60 min tabletop):1. T0 — Alert: Pager triggers for elevated error rate and latency for key endpoint.2. T+10 — New inject: Increase in error budget burn; customers reporting timeouts.3. T+20 — Complication: Rollout metadata shows recent deploy; deploy owner unavailable.4. T+30 — Complication: Simulated downstream cache outage amplifies errors.5. T+40 — Business inject: Executive asks for ETA; external vendor asks for log access.6. T+50 — Resolution: Restore traffic by rollback / circuit-breaker; post-fix monitoring shows recovery.

For each inject participants state: observations, actions, decisions, communications, and owners.

Success criteria:- Incident commander declared and TTR (time-to-meaningful-mitigation) under target (e.g., 30 min).- Clear role assignment and single source of truth (incident doc/Slack channel).- Appropriate mitigations executed (rollback, traffic-shift, feature-flag) and validated.- Stakeholder communications sent within SLA (e.g., initial notif <30 min, updates every 60 min).- No tooling or critical runbook gaps preventing mitigation.

Evaluation rubric (scores 1–5 across categories):- Detection & Triage (1=missed alerts, 5=quick, correct diagnosis)- Decision-making (1=confused, 5=clear, timely choices)- Runbook usage (1=no useful runbook, 5=followed & effective)- Coordination & Communication (1=chaotic, 5=single source, timely updates)- Technical Mitigation (1=ineffective, 5=correct fix, validated)- Learning Capture (1=no actions captured, 5=clear action items with owners)

Facilitated blameless after-action (30–45 min):- Start with safety framing: focus on systems/processes, not people.- Walk the timeline; ask what happened, why, and what stopped faster resolution.- Use “What went well”, “What didn’t”, “Action items” format.- Assign owners, target dates, and severity for each action; limit to 5 prioritized items.- Publish a concise AAR document and add action items to tracked backlog (e.g., sprint or reliability board).

Frequency and cadence:- Run full cross-functional tabletop quarterly for high-impact services, with lighter focused scenarios monthly for critical on-call skills.- After major incidents run an immediate focused tabletop for the specific failure mode within 2–4 weeks.

Incorporating learnings into runbooks:- For each action item, update the runbook section: detection thresholds, step-by-step mitigations, command snippets, rollback steps, contact matrix, and verification checks.- Version the runbook, note last-tested date, and tag which tabletop validated it.- Add automated tests where possible (synthetic alerts, chaos tests) and schedule runbook rehearsals on-call (game days).- Track maturity metrics: runbook coverage %, runbook-tested-last-90-days, and mean time to mitigation improvements.

This design balances technical validation, team coordination, and continuous improvement while keeping accountability to actions and measurable outcomes.

MediumTechnical

66 practiced

Design an error budget policy that describes: how to measure burn rate, what concrete actions are taken at different burn thresholds (e.g., stop launches, require remediation sprint), who makes the call, and how you integrate this with the release process.

Sample Answer

Overview: Define burn rate as how quickly an SLO’s error budget is being consumed and tie automated and human governance to concrete thresholds so releases are gated when reliability is at risk.

How to measure burn rate- SLIs → SLO (e.g., successful requests ≥ 99.95% per 30 days). Define error budget = 1 - SLO.- Rolling windows: compute error budget consumed in multiple windows (1h, 24h, 30d).- Burn rate formula: burn_rate = (error_budget_consumed_over_window) / (window_length_fraction_of_SLO_period). Example: if 0.5% of budget used in 1 day and budget for 30 days is 0.05%/day equivalent, burn_rate = observed_rate / expected_rate.- Automated dashboard + alerting (PagerDuty/Slack) for burn_rate per service.

Thresholds & concrete actions- Green (burn_rate ≤ 1): normal releases and canaries proceed.- Yellow (1 < burn_rate ≤ 3): restrict high-risk releases; require post-merge canary for 24h; SRE notified; on-call to triage trends.- Orange (3 < burn_rate ≤ 8): stop non-essential launches, pause all big feature toggles or experiments, require an SRE-approved mitigation plan and limited scoped release with 1% traffic canary only.- Red (burn_rate > 8 or projected budget exhaustion in <24h): immediate halt to all launches and experiments; rollback recent deployments likely responsible; require remediation sprint or hotfix within 48-72h; declare incident if customer impact high.- Each threshold triggers automated actions where possible (CI gates, rollout controllers) and human escalation otherwise.

Decision authority- Automated enforcement for initial actions (CI/CD block, canary limits).- For Yellow/Orange: SRE on-call + Release Manager + Feature Owner must agree to proceed; decision logged in runbook.- For Red: SRE Manager + Engineering Manager + Product Manager make final call to stop launches and declare remediation sprint. If contention, Product takes customer-priority view but SRE has veto for site reliability.

Integration with release process- CI/CD gates: pre-release check reads current burn_rate and fails pipeline when above thresholds for the change type (e.g., major vs patch).- Canary & progressive rollout by default; automated rollback if canary SLI violations exceed thresholds.- Release checklist includes SLO health sign-off; feature flags required for risky features.- Change approval workflow: automated tests + SLO health snapshot attached to PR; Release Manager dashboard shows burn_rate and required approvers.- Post-release: require monitoring baseline for 24–72h; if threshold moves up, trigger remediation workflow and postmortem.

Governance & feedback loop- Quarterly review of SLOs, thresholds, and policy efficacy; incorporate runbook learnings into automation.- KPI tracking: number of blocked releases, time-to-remediation, frequency of red-level events.This policy balances automated fast responses with human judgment, keeps releases moving when safe, and forces remediation when reliability is endangered.

MediumTechnical

57 practiced

Draft a concise post-incident review template that includes timeline, impact, root cause analysis, contributing factors, actions, owner, and verification steps. Explain which parts must be completed within 24 hours and which can be deferred, and why.

Sample Answer

Post‑Incident Review Template (concise)

1) Title & Incident ID2) Severity & Start/End timestamps3) Summary (1–2 sentences)4) Timeline (chronological bullet list of key events)5) Impact (systems, customers, metrics, % users affected, business impact)6) Immediate mitigation & rollback actions (what was done to stop/mitigate)7) Root Cause (short hypothesis + evidence)8) Contributing factors (human, process, tools, monitoring gaps)9) Actions (short-term and long-term) — each with owner, due date, priority10) Verification steps (how we’ll confirm fix works/monitoring to add)11) Communication log (internal/external messages and timestamps)12) Postmortem owner and review meeting date13) Lessons learned

Which parts to complete within 24 hours (required):- Title, Incident ID, Severity & timestamps: for record-keeping and compliance.- Summary: quick executive-friendly explanation for stakeholders.- Initial Timeline (key events up to mitigation): preserves facts while fresh.- Impact (initial measured impact): informs leadership and customers.- Immediate mitigation & rollback actions: documents what stopped the outage.- Postmortem owner and communication log: assigns responsibility and tracks notices.Why: rapid, accurate snapshots are essential for customer comms, legal/compliance needs, and to avoid losing ephemeral details.

Which parts can be deferred (24–72 hours or longer):- Full Root Cause Analysis & Contributing Factors: requires log collection, deeper forensic analysis, interviews.- Complete list of long-term actions, precise due dates, and detailed verification plans: need engineering investigation and planning.- Detailed verification scripts/tests and monitoring dashboards: built and validated after fixes are proposed.Why: deeper analysis benefits from complete data, reproducible tests, and stakeholder consultation; rushing can produce incorrect conclusions. Aim to finalize RCA and action plan within 3–7 days, plus verification after fixes are implemented. Make the 24‑hour deliverable clear and blameless; iterate the postmortem as new evidence appears.

Unlock Full Question Bank

Get access to hundreds of Site Reliability Engineering Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.