P1 — Critical (Safety, data loss, legal/regulatory breach)
Example: Model generates unsafe/biased outputs in production causing customer harm, or a retrain job accidentally deletes customer records.
On-call responsibilities:
- Immediately acknowledge alert and take mitigation (disable model, roll back to previous safe version, enable circuit-breaker).
- Preserve logs and inputs for postmortem.
Escalation contacts: ML Lead, Security/Trust & Safety, Product Legal, SRE.
Communication expectations:
- 15-minute initial ack with incident channel; hourly public updates until mitigated.
- Post-incident report within 24–72 hours.
Time-to-resolution goal: 1–4 hours to safe state; full root-cause and remediation 72 hours.
P2 — High (Major performance loss, data pipeline failures without safety implications)
Example: Production accuracy falls 10–20% causing user-visible degradation or feature store lag blocking predictions.
On-call responsibilities:
- Triage: revert recent model/config changes, retrain using backup data, restart pipelines.
- Monitor rollback impact and document actions.
Escalation contacts: Model Owner, Data Engineering, SRE.
Communication expectations:
- 30-minute ack; updates every 2 hours to stakeholders; incident channel open until resolved.
Time-to-resolution goal: 4–24 hours to restore acceptable performance; full RCA within 3–7 days.
P3 — Low (Minor drift, non-critical latency/memory increases, monitoring alerts)
Example: Small metric drift (<5%), increased inference latency under SLA but not user-breaking.
On-call responsibilities:
- Diagnose, add data to drift logs, schedule retrain/experiment.
- Tune thresholds, create follow-up tickets.
Escalation contacts: Model Owner, ML Platform team (if recurring).
Communication expectations:
- Acknowledge within business day; summary in weekly status; no immediate stakeholder page unless worsens.
Time-to-resolution goal: 1–7 days for mitigation; planned fix within 2–4 weeks.
Notes: always attach reproducible artifacts (logs, model versions, data snapshots), follow blameless postmortem practices, and update runbooks after each incident.
