Covers the design and operation of incident response programs and the creation and maintenance of actionable runbooks and playbooks for production systems. Candidates should be able to explain the incident lifecycle from detection and classification through investigation, escalation, remediation, and post incident analysis. Topics include severity definitions and assessment, escalation procedures, team roles and responsibilities, communication protocols during incidents, on call rotations, alert triage, and coordination across teams during outages. Also includes designing automated remediation steps where appropriate, integrating runbooks with monitoring and alerting systems, maintaining playbooks for common failure modes such as malware, data exfiltration, denial of service, and account compromise, and conducting blameless post incident reviews and continuous improvement. Candidates should be able to discuss metrics for measuring response effectiveness such as mean time to detect, mean time to repair, and response success rate, and describe approaches to improve those metrics over time.
HardTechnical
70 practiced
You are given false-positive logs for a critical alert that triggers 80% of the time without real impact. How would you build a detection/classification approach to reduce false positives while maintaining acceptable sensitivity? Discuss trade-offs between sensitivity and specificity, test datasets, thresholds, and operational validation steps.
Sample Answer
Situation & goal: We have a critical alert that fires 80% false positives. Goal is to reduce false positives substantially while keeping sensitivity high enough to meet SLOs / acceptable MTTD.Approach overview:1. Data & labeling- Collect historical alert logs, context (metrics, traces, deployment events, maintenance windows), and post-incident labels (true/false). If labels are missing, run a short manual adjudication or use on-call annotations to bootstrap.- Split into train/validation/test by time (preserve temporal order) to avoid leakage.2. Feature engineering- Add features: metric windows (mean, slope, variance), related service metrics, request rates, error types, host metadata, recent deploys/rollbacks, incident suppression flags, seasonality (hour/day), dedup counts.- Derive event-level features: alert duration, previous alert history, correlated alerts across hosts.3. Model / detection design- Start with deterministic rules to eliminate obvious false positives (maintenance windows, known noise signatures).- Build a lightweight classifier (e.g., gradient-boosted trees or logistic regression) to predict P(true incident). Ensemble rule + model reduces risk of black-box failures.- Use cost-sensitive learning or class weights because positives are rare.4. Thresholds & trade-offs- Produce ROC and precision-recall curves on validation set. Prefer optimizing for high recall at acceptable precision if missing incidents is costly; otherwise tune for higher precision.- Use a cost matrix: quantify business cost of missed incident vs. cost of false alert (on-call fatigue). Choose threshold minimizing expected cost.- Consider two-stage alerts: if model P > high_threshold => page; if between low and high => create a low-priority ticket or notify a dashboard only.5. Testing & validation- Backtest on holdout period; report metrics: recall, precision, F1, false positive rate, alert volume reduction, time-to-detect simulated incidents.- Run offline simulation that injects known incidents to measure sensitivity.6. Operational rollout- Canary the classifier for a subset of services or teams; run in “audit” mode where original alert still pages but model logs decisions.- After safe validation, enable suppression for a small percentage of pages (or route to non-paging channel) and monitor missed-detection signals for a few weeks.- Implement automated fallback: if model service unhealthy, revert to original alerting rules.7. Feedback loop & monitoring- Add instrumentation: model confidence, distribution drift, feature importance, and alert reduction dashboards.- Require on-call feedback annotation for any suppressed-but-actually-incident cases; retrain periodically and after major infra changes.Trade-offs & reasoning:- Sensitivity vs specificity: increasing threshold reduces false positives but risks missed incidents. Use explicit cost-based thresholding and two-stage paging to balance human costs vs risk.- Complexity vs reliability: prefer simple, explainable models plus rules initially; avoid opaque systems that on-call won't trust.- Operational risk: canary + audit mode + fallback mitigate production risk.Expected outcomes:- Significant reduction in paged alerts (target e.g., 50–90%) while maintaining recall above SLO-derived minimum. Continuous monitoring and on-call feedback will keep performance acceptable over time.
MediumTechnical
80 practiced
Propose a set of KPIs and dashboard widgets to measure incident response effectiveness. Include at least: MTTD, MTTR, incidents per week by severity, mean time to acknowledge, pager burn rate, and action-item closure rate. For each KPI describe the data source (logs, alerts, incident tickets), how you'd compute it (brief formula or query), and an appropriate alerting threshold.
Sample Answer
Below is a proposed KPI set and dashboard widget suite to measure incident response effectiveness for an SRE team. For each KPI I list data source, computation (brief), recommended visualization widget, and an alerting threshold rationale.1) Mean Time to Detect (MTTD)- Data source: alerting system timestamps / monitoring logs (first alert or anomalous log event time) vs. incident ticket creation time.- Compute: average(incident.ticket_created_at - alert.first_detected_at) over window.- Widget: rolling 7‑ and 30‑day line + histogram.- Alert threshold: > SLA target (e.g., > 5m for critical services) or +200% of baseline.2) Mean Time to Restore/Resolve (MTTR)- Data source: incident tickets (resolved_at), service health events.- Compute: average(incident.resolved_at - incident.opened_at).- Widget: stacked bar by severity + trend line.- Alert threshold: > SLO target (e.g., > 1h for P0) or rising 2x week-over-week.3) Incidents per week by severity- Data source: incident tickets with severity labels.- Compute: count grouped by week and severity.- Widget: stacked column chart with drilldown.- Alert threshold: weekly P0/P1 count > X (e.g., P0 > 1 or P1 > 3) or > 50% increase vs. rolling 4‑week median.4) Mean Time to Acknowledge (MTTA)- Data source: alerting system (alert fired) and on‑call acknowledgement timestamps (pager/incident ticket).- Compute: average(ack_timestamp - alert_timestamp).- Widget: box-and-whisker by shift/on-call.- Alert threshold: > 1 minute for P0 (pager escalation workflow), > 5m for P1.5) Pager Burn Rate- Data source: paging system logs (pages sent per engineer / on-call schedule).- Compute: pages_per_person_per_week / target_pages (or pages normalized by on-call hours).- Widget: heatmap per engineer + trend.- Alert threshold: any engineer > 2x expected pages/week or team burn rate > capacity (predicts burnout).6) Action-item Closure Rate (post-incident)- Data source: postmortem/issue tracker tasks linked to incident.- Compute: closed_tasks / total_tasks within target window (e.g., 30 days).- Widget: progress bar per incident + SLA compliance %- Alert threshold: closure rate < 80% at 30 days.7) Incident Reopen Rate / Recurrence- Data source: incident tickets (reopen flags or repeat incidents tied to same root cause).- Compute: incidents_marked_reopen / total_incidents.- Widget: KPI tile + list of recurring services.- Alert threshold: > 5% or any repeat P0 within 30 days.8) Post-Incident RCA Quality Score (qualitative but measurable)- Data source: postmortem repository, checklist automation (completeness flags).- Compute: percent of postmortems meeting checklist (timeline, root cause, actions).- Widget: gauge + sample links.- Alert threshold: < 90% completeness.Implementation notes:- Normalize timezones; use consistent incident lifecycle events (detect → open → ack → resolve).- Provide filters (service, team, severity) and exportable CSVs for SLO reviews.- Alerting should be for sustained violations (e.g., 3 consecutive windows) to avoid noise.
MediumTechnical
67 practiced
Design a tabletop incident simulation (game day) for a major outage affecting payment processing. Specify objectives, participants (roles), timeline of injects, success criteria, required observability/data, and post-exercise actions. Explain how to run the exercise with minimal production risk.
Sample Answer
Objectives:- Validate people/process for a major payment-processing outage (detection, communication, mitigation, recovery).- Verify runbooks, failover procedures, SLO/error-budget decisions, and stakeholder comms.- Identify tooling/observability gaps and actionable improvements.Participants / Roles:- Incident Commander (IC) — SRE (leads exercise)- Payment Service On-call Engineer — domain expert- Database Lead — DBA- API/Gateway Engineer — ingress control- Security/Compliance Rep — PCI considerations- Product/PM — customer/business priorities- Customer Support Lead — external comms- Communications Lead — public/partner messaging- Observers/Note-takers — two SREs for timelines/action itemsFormat & how to minimize risk:- Tabletop (discussion-driven) only; no changes to production.- Use a realistic staging environment or replay logs on a sandbox for any interactive checks.- Pre-brief everyone on scope, safety rules (no production ops), and “pause” condition.- Time-boxed injects; IC controls pacing and can pause when follow-ups require deeper research.Timeline of injects (90–120 minutes total)- 0–10m: Intro, objectives, roles, safety rules.- 10–25m (Inject 1): Monitoring alerts show increased payment failures (5xx) and queue backlog. Task: Triage root cause hypotheses, assign owner.- 25–45m (Inject 2): Downstream DB replication lag and slow writes detected. Task: Decide mitigation (route to secondary, enable degraded mode), assess SLO impact.- 45–60m (Inject 3): Third-party payment gateway returns 502 for a subset of merchants. Task: Toggle feature flags, apply circuit breaker, communicate to partners.- 60–75m (Inject 4): Regulatory/compliance question: should we keep retrying sensitive card operations? Task: consult Security/Compliance, adjust retry policy.- 75–90m (Inject 5): Simulated customer complaints spike; exec requests ETA and rollback plan. Task: craft public comms and prioritized recovery plan.- 90–120m: Wrap-up, capture action items, vote on success criteria.Success Criteria (measurable)- Detection: Alert surfaces within target MTTR threshold (e.g., 5 min).- Roles: IC assigned within 10 min; owners for each domain identified.- Mitigation: Team applies at least one safe mitigation (degraded mode/routing) within 30–45 min in the tabletop.- Communication: Clear internal and external messages drafted within 60 min.- Learnings: ≥3 concrete, owner-assigned action items with deadlines.Required observability / data to present- Payment 5xx rate and latency trends, per-merchant and per-region.- Queue/backlog metrics (size, processing rate).- DB metrics: replication lag, write latency, connection pool saturation.- API gateway error rates and downstream status codes.- Third-party gateway health and SLA telemetry.- SLO/error-budget dashboard and burn rate.- Recent deployment history and feature-flag state.- Customer support ticket volume and top complaint themes.Post-exercise actions- 48-hour AAR: Distribute notes, timelines, decisions, and action items; classify severity of gaps.- Create prioritized tracker (Jira) with owners and SLAs for fixes (runbook updates, monitoring coverage, automation).- Update runbooks and playbooks with gaps discovered; add clear decision points and communication templates.- Schedule follow-up tabletop or live drill to validate implemented fixes within 4–8 weeks.- Share a short exec summary for stakeholders and include updated SLO impact assessment.Why this works- Tabletop avoids production risk while exercising communication and decision flows.- Time-boxed injects emulate pressure without requiring unsafe operations.- Observable metrics and realistic injects ensure technical and non-technical stakeholders engage.- Concrete success criteria and tracked remediation turn the exercise into measurable reliability improvements.
EasyTechnical
58 practiced
Describe the full incident lifecycle for a production service from detection through post-incident analysis. In your answer, explicitly cover: (a) how incidents are detected (monitoring, alerts, customer reports, synthetic checks), (b) how incidents are classified and triaged, (c) investigation and evidence-gathering steps, (d) escalation rules and role handoffs, (e) remediation and rollback execution, (f) recovery validation, and (g) post-incident review and follow-up actions. Give concrete examples of artifacts produced at each stage (logs, timelines, runbook executions, postmortems).
Sample Answer
1) Detection- Sources: monitoring/metrics (Prometheus alerts), log-based alerts (ELK), synthetic checks (uptime probes), customer reports (Zendesk). - Artifact: alert ticket in PagerDuty with alert context, Grafana dashboard links, initial alert email.2) Classification & Triage- Determine severity (P1/P2) using SLOs and impact (users, revenue). Assign incident commander (IC) and timeline.- Artifact: incident ticket with severity, affected services, initial runbook reference.3) Investigation & Evidence Gathering- Actions: collect logs, traces (Jaeger), metrics, recent deploys, feature flags; reproduce via canary; capture heap/cpu profiles.- Artifacts: supporting logs, flamegraphs, trace spans, "what changed" diff from CI/CD.4) Escalation & Handoffs- Rules: time-to-escalate (e.g., 15 min no progress) → rota escalation (on-call → SRE lead → engineering owner).- Artifact: escalation log in incident channel (Slack), handoff checklist.5) Remediation & Rollback- Execute safe remediation: config change, scaling, traffic-shift, or rollback via automated pipelines.- Artifact: runbook execution log, deployment rollback PR and CI record.6) Recovery Validation- Verify via synthetics, user journeys, error rate back to baseline, SLOs restored.- Artifact: post-recovery metrics snapshot, incident closure checklist.7) Post-Incident Review & Follow-up- Produce postmortem: timeline, root cause, action items, owners, deadlines, blameless analysis.- Artifacts: written postmortem, tracked JIRA follow-ups, updated runbooks/playbooks, monitoring rule refinements.
EasyTechnical
75 practiced
Explain the concept of an error budget. How does an SRE team use an error budget to prioritize work, decide when to roll back a risky change, and trigger operational constraints? Provide an example: given an SLO of 99.9% monthly availability, calculate the monthly error budget in minutes and explain what a 10x burn rate means.
Sample Answer
An error budget is the allowable amount of unreliability you accept while still meeting an SLO. It’s computed as 1 − SLO and expressed over the SLO period (e.g., minutes of downtime per month). Error budgets shift conversations from “zero downtime” to risk-managed trade-offs between feature velocity and reliability.How SREs use an error budget- Prioritize work: If budget is healthy, teams can focus on shipping features. If it’s burning quickly, priority shifts to reliability work (bug fixes, capacity, testing).- Decide rollbacks: A sudden spike in error-budget burn after a deployment is a strong signal to roll back or disable the risky change to stop further consumption.- Trigger operational constraints: Defined thresholds (e.g., burn rate rules) automatically enforce limits: freeze launches, require additional QA, increase monitoring, call in on-call responders, or escalate to incident review.Example calculation- Monthly minutes (approx): 30 days × 24 hours × 60 minutes = 43,200 minutes.- SLO 99.9% → allowed downtime = 0.1% = 0.001 × 43,200 = 43.2 minutes/month. That’s the monthly error budget.What a 10× burn rate means- Burn rate = (observed error per unit time) / (allowed error per unit time).- 10× burn rate means you are consuming error budget ten times faster than the nominal pace. Practically: if you used 10 minutes of downtime in the first 3 days, you’re on track to exhaust the 43.2-minute budget in ~13 days, not the full month (because 10× faster).- Operational response: a 10× sustained burn typically triggers immediate mitigation: roll back the recent change, block new releases, increase paging/war-room focus, and run a postmortem to prevent recurrence.This framework makes reliability decisions objective, automatable, and aligned with business risk.
Unlock Full Question Bank
Get access to hundreds of Incident Response and Runbook Design interview questions and detailed answers.
