Incident Severity Classification and Escalation Questions

Monitoring and observability tooling should feed incident-severity decisions by surfacing SLI breaches, trend context, and correlated signals — not just noisy single metrics. As an SRE I expect Prometheus/Grafana/Datadog to provide: clear SLI panels, alert rules mapped to severity, and event windows for correlation.

Key SLIs and charts for severity decisions

• Latency percentiles: p95 and p99 for user-facing RPCs or HTTP endpoints (charted as heatmaps + trend lines). p99 spikes -> higher severity.
• Error rates: 5xx rate (absolute and relative to traffic) and grouped by endpoint/service. Sustained >SLO breach or sudden jump => higher severity.
• Successful transaction ratio: end-to-end success (business-critical flows). Drops here are highest priority.
• Saturation metrics: CPU, memory, queue length, DB connections — indicate capacity incidents.
• Throughput and traffic shifts: request/sec and RPS per endpoint to distinguish overload vs regression.
• External dependencies: third-party latency/error rates.

Correlated symptoms and event windows

• Present a unified timeline: overlay SLIs, infrastructure metrics, deployments, config changes, and alerts for a configurable window (e.g., -15m to +60m).
• Show alerts grouped by service and severity with causality hints (e.g., deployment at T0, p99 rise at T+2m).
• Use rolling windows and anomaly detection to avoid reacting to single-point blips; require sustained breach (e.g., >5 minutes or >=X requests) for higher-severity escalation.
• Annotate dashboards with recent deploys, autoscaling events, and incident notes to avoid misclassification.

Practical rules to avoid misclassification

• Tie alerting to SLO-derived thresholds (warning vs critical) and require multiple correlated signals for escalation.
• Suppress alerts during known maintenance/deploy windows or mark them as informational.
• Use service-level composite alerts (business transaction failure + elevated 5xx + p99 increase) to trigger P1; single non-sustained metric -> P3.
• Provide playbook links and quick triage checklist in the alert to speed accurate severity decision.

Situation: Multiple services across teams are breaching SLOs during a major incident. As the senior SRE on-call, I need a fast, defensible way to decide what to triage first and how to coordinate fixes.

Framework (decision flow):

1. Rapid intake (first 10–15 min)

• Gather SLO status, error budgets, alert counts, and basic telemetry for each service.
• Triage into buckets: critical (user-visible outage), degraded (partial functionality), non-user-facing.

2. Scoring (0–10) for each service — combine three axes:

• Customer/Business Impact (weight 40%): revenue risk, SLA penalties, number of users affected, regulatory exposure. Example: global checkout outage = 10.
• Blast Radius (weight 30%): number of downstream systems, cascading risk, cross-region spread. Example: auth service used by many = 9.
• Error Budget State (weight 30%): remaining error budget %; exhausted or negative budgets raise priority. Example: budget at -50% = 10.

Compute weighted score = 0.4Impact + 0.3Blast + 0.3*ErrorBudgetUrgency. Triage highest scores first.

3. Quick qualifiers & run actions

• If customer-visible outage → immediate mitigation (rollback/rate-limit/traffic-shift).
• If high blast radius → isolate (circuit-breakers, feature flags).
• If error budget exhausted but low customer impact → schedule sustained mitigation and communication, but deprioritize against outages.

4. Cross-team coordination

• Appoint an incident commander (IC) and owners per service for top N items.
• IC runs prioritization scoreboard updated every 5–10 minutes.
• Use a shared war room (Slack + incident board) and a single source of truth (dashboard).
• Predefined escalation: exec notification threshold tied to score and time-to-resolution.

5. Communication & governance

• Communicate customer-facing status early if impact > threshold.
• Apply temporary guardrails (traffic caps, fail-open/off) agreed in runbooks.
• After containment, conduct blameless postmortem prioritized by impact and error-budget burn rate; feed findings into SLO tuning and runbook improvements.

Why this works:

• Balances business impact, systemic risk, and the constrained resource of error budgets.
• Provides quantitative, repeatable prioritization under pressure.
• Ensures focused cross-team action and clear ownership, minimizing firefighting and cascading failures.

Example quick decision: Checkout service (impact 10, blast 8, error-budget 9) vs internal reporting (impact 3, blast 2, error-budget 10) → checkout handled first; reporting scheduled after containment.

Situation: On my SRE team we schedule quarterly "game days" to validate incident processes and ensure our severity definitions map to real-world impact.

Task: Run a simulated multi-service degradation during peak traffic to test whether on-call teams correctly declare severity, follow runbooks, and restore service within SLOs.

Action:

• Scenario: At simulated 11:30am peak, the API gateway latency increases 5x and payment service returns 50% errors due to a downstream DB connection pool exhaustion; background jobs back up and dashboards show increased error budget burn.
• Pre-game: notify observers, instrument time-stamped injects, and ensure safe rollback kill-switch.
• Run: Let on-call follow normal alerting; observers record decisions and timing; we avoid hints unless safety requires.
• Metrics captured:
  • Time-to-detect (first alert to acknowledgement)
  • Time-to-declare (acknowledgement to severity declaration)
  • Time-to-mitigation (declaration to first effective mitigation)
  • Judgement accuracy (declared severity vs. postmortem impact score)
  • Communication metrics (time to notify stakeholders, number of status updates)
  • SLO/error-budget impact simulated vs. real
• Post-game: conduct blameless postmortem with quantitative data and recordings.

Result / Lessons & incorporation:

• If teams under-declared severity (long time-to-declare or low judgement accuracy), we tighten thresholds: e.g., map API latency >2000ms with payment error rate >20% during peak to Severity 1. Update runbooks to add explicit triage checklist: check dependent service error rates, SLO burn rate, and quick rollback steps.
• If false-positives occur, we add additional signal validation (cross-check metrics, reduce alert sensitivity) and improve alert aggregation rules.
• We convert learnings into:
  • Concrete severity decision matrix (signals → severity)
  • Runbook edits with play-by-play mitigation + checklists
  • Training sessions for new on-call rotations
  • Automated runbooks (playbooks) to reduce human judgment delays

This process ensures severity definitions are evidence-based, measurable, and continuously improved via practiced, safe drills.

Approach: evaluate severity rules in priority order (Sev1 first), handle missing or bad fields robustly by treating absent numeric fields as 0 and non-boolean recent_deploy ignored for classification. Return 'sev1'/'sev2'/'sev3'.

def classify_alert(alert: dict) -> str:
    """
    Classify alert into 'sev1', 'sev2', or 'sev3' per rules:
    - Sev1 if customers_affected > 10000 OR error_rate_pct >= 5
    - Sev2 if error_rate_pct >= 1 OR latency_p95_ms > 2000
    - Else Sev3

    Missing numeric fields are treated as 0 for safe classification.
    """
    # helper to coerce numeric fields safely
    def num(key):
        v = alert.get(key, 0)
        try:
            return float(v)
        except (TypeError, ValueError):
            return 0.0

    customers = num("customers_affected")
    error_pct = num("error_rate_pct")
    latency = num("latency_p95_ms")

    # Priority: Sev1 first
    if customers > 10000 or error_pct >= 5.0:
        return "sev1"
    if error_pct >= 1.0 or latency > 2000.0:
        return "sev2"
    return "sev3"

Unit tests (pytest):

import pytest

def test_sev1_by_customers():
    assert classify_alert({"customers_affected": 10001, "error_rate_pct":0}) == "sev1"

def test_sev1_by_error_exact_threshold():
    assert classify_alert({"customers_affected":0, "error_rate_pct":5.0}) == "sev1"

def test_sev2_by_error_exact():
    assert classify_alert({"customers_affected":0, "error_rate_pct":1.0}) == "sev2"

def test_sev2_by_latency_exact_boundary():
    assert classify_alert({"latency_p95_ms":2000}) == "sev3"  # >2000 required
    assert classify_alert({"latency_p95_ms":2001}) == "sev2"

def test_sev3_normal():
    assert classify_alert({"customers_affected":10, "error_rate_pct":0.5, "latency_p95_ms":100}) == "sev3"

def test_missing_fields():
    # missing keys treated as zeros -> sev3
    assert classify_alert({}) == "sev3"

def test_bad_types():
    # non-numeric values are coerced to 0
    assert classify_alert({"customers_affected":"notanumber", "error_rate_pct":"5%"}) == "sev3"
    # but valid numeric strings work
    assert classify_alert({"error_rate_pct":"5"}) == "sev1"

Key points:

• Explicit priority ensures Sev1 takes precedence.
• Safe coercion prevents exceptions on malformed alerts.
• Tests cover exact thresholds, missing fields, type edge cases.

Learning objectives:

• Correctly classify incidents (sev0–sev3) using SLOs, customer impact, blast radius
• Triage, escalate, and communicate with stakeholders
• Execute runbooks, mitigation, rollback, and post-incident reviews
• Write clear runbooks and update runbook library
• Demonstrate independent on-call readiness with SLA adherence

Two-week schedule (Mon–Fri ×2):
Week 1
Day 1: Orientation—SRE principles, SLOs/SLIs, incident taxonomy. Exercise: map alerts→severity for 10 sample alerts.
Day 2: Monitoring & alerting logic. Exercise: tune alert thresholds in sandbox; reduce noise.
Day 3: Triage tooling & communication templates. Exercise: practice pager response, Slack/bridge etiquette.
Day 4: Simulated Incident A (Medium / Sev2): partial latency spike for a single region. Tasks: triage, apply mitigation, page owner, document timeline. Debrief + blameless mini-postmortem.
Day 5: Runbook-writing workshop. Assignment: draft runbook for Day4 incident.

Week 2
Day 6: Automation & safe rollback patterns. Exercise: run rollback in staging.
Day 7: Escalation and cross-team coordination. Exercise: coordinate with mock dev/product teams.
Day 8: Observability deep-dive (logs/traces). Exercise: root-cause hunting challenge.
Day 9: Simulated Incident B (Critical / Sev1): data-loss or global outage. Full war room, multiple handoffs, escalate to incident commander, mitigation, hotfix. Debrief + full postmortem template.
Day 10: Assessment day: written & practical exam, final runbook submission, readiness review.

Assessments & criteria:

• Practical: complete triage and mitigation within target time (Sev2: 45–60 min; Sev1: 15–30 min to initial mitigation)
• Communication: timely status updates (first update <10 min for Sev1, <30 min for Sev2) and clear incident timelines
• Runbook quality: steps are correct, safely automatable, includes rollbacks, verification, paging matrix, and owner
• Postmortem: identifies root cause, contributing factors, and specific action items with owners and deadlines

Runbook-writing assignments:

• Draft for Week1 incident (Day5): include prechecks, mitigation steps, verification, rollback, runbook owner
• Finalize with automation hooks after Day9 simulation; submit on Day10

Certification & readiness:

• Pass written quiz (≥80%) on taxonomy, SLOs, and escalation rules
• Pass practical simulation (demonstrated by checklist) including: triage, mitigation, communication, and handoff
• Runbook accepted by SME reviewer (>=3 reviewers) and merged into library
• Mentor sign-off: two shadowed independent on-call shifts with mentor backing, then one solo monitored shift with 0 critical errors

Governance:

• Quarterly recertification via tabletop or live sim
• Metrics tracked: MTTR, number of misclassified incidents, runbook usage rate; remediation plans for failures.