Incident Severity Classification and Escalation Questions

Situation: I need to design a 24/7 on-call for a 6-person ML engineering team that preserves reliability, learning, and avoids burnout.

Rotation length:

• 7-day primary on-call shifts (1 week) with a 48-hour follow-on “light duty” buffer after intense incidents; for fairness, rotate weekly so each engineer covers ~8–9 weeks/year.
• Secondary (backup) assigned for each week to reduce pager fatigue and provide immediate escalation.

Handoff process:

• Daily handoff doc template (status, ongoing incidents, mitigations, key metrics, deploys planned).
• 30–60 minute overlap at shift start for verbal handoff and a short async checklist in Slack/PagerDuty.
• Maintain runbooks and runbook playlists in a central repo; each on-call must update runbooks after incidents.

Escalation path to SRE:

• Tiered flow: Primary on-call → Secondary on-call → SRE (pager rule after N minutes or if production infra fault) → Engineering manager/incident commander.
• Use PagerDuty with escalation policies, Slack incident channel, and clear severity definitions in runbooks.

Incentives:

• On-call stipend or overtime pay per week + compensatory time off (1 day per high-severity incident).
• Quarterly recognition and learning credit (conference/training) for heavy on-call contributors.

Planned vacations / swaps:

• Self-service swap system in PagerDuty with manager approval and mandatory shadowing for new takeovers at least one shift before coverage starts.
• Require coverage confirmations 2 weeks before vacation.

Takeover during incidents:

• Define roles: responder (hands-on), communicator (customer/stakeholder updates), commander (coordinates and escalates).
• If handover mid-incident, incoming person shadows for 30 minutes and commander approves transfer.
• Post-incident: blameless postmortem within 3 business days, action items assigned and tracked.

Knowledge sharing & burnout mitigation:

• Pair rotations: every few months pair a less-experienced engineer as secondary with a senior primary.
• Monthly runbook review, incident war-room drills, limit consecutive on-call weeks to 2 for anyone in special situations.
• Monitor pager metrics and survey team wellbeing quarterly to adjust cadence.

Being "on‑call" for an ML model/service means you’re responsible for responding to production incidents and ensuring the model remains reliable, accurate, and safe outside business hours. It’s less about building models and more about operating them: monitoring, diagnosing, mitigating, and communicating until normal service is restored.

Common responsibilities

• Alert triage: evaluate alerts (latency, error rate, prediction distribution drift, data pipeline failures, resource exhaustion). Determine severity and scope.
• Mitigation: apply short‑term fixes (traffic throttling, feature freezes, input validation, scaling replicas, restarting pods/services).
• Rollback: revert to a known good model or previous container/image if mitigation fails or model behaves unpredictably.
• Root cause & remediation: gather logs, metrics, explainability traces; implement permanent fixes or task tickets.
• Customer communication: post initial acknowledgement, status updates, and incident summary for stakeholders and downstream teams.
• Postmortem: document timeline, impact, root cause, corrective actions.

Expected response times (typical)

• P0/Critical (system down, bad outputs affecting customers): ack within 5–15 minutes, remediate ASAP.
• P1/High (degraded performance or strong drift): ack within 15–30 minutes, action within an hour.
• P2/Low (nonblocking anomalies): ack within a few hours, scheduled fix.

Minimal runbook for a newly owned model (must-haves)

• Service overview: purpose, SLA, owners, on‑call rotation, escalation path, business impact tiers.
• Health signals & dashboards: key metrics (latency, error rate, throughput, input feature distributions, PSI/KS, model confidence, AUC/regression metrics if applicable), where to view them.
• Alert matrix: alert names, thresholds, severity, and likely causes.
• Quick mitigation steps: run commands, scripts, or UI actions for common faults (scale, restart, revert model artifact ID, block bad inputs).
• Rollback procedure: exact artifact IDs, storage paths, deployment commands, validation checklist.
• Data pipeline checks: how to validate upstream data, re-run preprocessing, or replay jobs.
• Logging & debug pointers: locations, log queries, sample queries for hypothesis.
• Communication templates: initial ack, incident updates, customer-facing message, postmortem template.
• Safety checks: model guardrails, feature gating, kill switches.
• Links: runbook repo, playbooks, dashboards, runbook owner contact info.

These elements let you respond quickly, reduce blast radius, and hand incidents off cleanly while you work toward permanent fixes.

P1 — Critical (Safety, data loss, legal/regulatory breach)
Example: Model generates unsafe/biased outputs in production causing customer harm, or a retrain job accidentally deletes customer records.
On-call responsibilities:

• Immediately acknowledge alert and take mitigation (disable model, roll back to previous safe version, enable circuit-breaker).
• Preserve logs and inputs for postmortem.
  Escalation contacts: ML Lead, Security/Trust & Safety, Product Legal, SRE.
  Communication expectations:
• 15-minute initial ack with incident channel; hourly public updates until mitigated.
• Post-incident report within 24–72 hours.
  Time-to-resolution goal: 1–4 hours to safe state; full root-cause and remediation 72 hours.

P2 — High (Major performance loss, data pipeline failures without safety implications)
Example: Production accuracy falls 10–20% causing user-visible degradation or feature store lag blocking predictions.
On-call responsibilities:

• Triage: revert recent model/config changes, retrain using backup data, restart pipelines.
• Monitor rollback impact and document actions.
  Escalation contacts: Model Owner, Data Engineering, SRE.
  Communication expectations:
• 30-minute ack; updates every 2 hours to stakeholders; incident channel open until resolved.
  Time-to-resolution goal: 4–24 hours to restore acceptable performance; full RCA within 3–7 days.

P3 — Low (Minor drift, non-critical latency/memory increases, monitoring alerts)
Example: Small metric drift (<5%), increased inference latency under SLA but not user-breaking.
On-call responsibilities:

• Diagnose, add data to drift logs, schedule retrain/experiment.
• Tune thresholds, create follow-up tickets.
  Escalation contacts: Model Owner, ML Platform team (if recurring).
  Communication expectations:
• Acknowledge within business day; summary in weekly status; no immediate stakeholder page unless worsens.
  Time-to-resolution goal: 1–7 days for mitigation; planned fix within 2–4 weeks.

Notes: always attach reproducible artifacts (logs, model versions, data snapshots), follow blameless postmortem practices, and update runbooks after each incident.