Clarify goal and constraints: limited SRE capacity (assume 5 engineers available), need to minimize customer revenue/SL impact, operational risk, and cascading/systemic failure while keeping experiments moving.
Prioritization matrix (criteria, weight):
- Revenue impact (40%)
- Number of customers affected (20%)
- Regulatory/compliance risk (10%)
- Risk of cascading failures (20%)
- Time-to-recover difficulty (10%)
Score each incident 1–5 (5 = worst) and compute weighted score:
- P2 model-quality degradation (small set of VIP customers)
- Revenue impact: 4 (VIPs high ARPU) → 4*0.40 = 1.6
- Customers affected: 2 → 0.4
- Regulatory risk: 1 → 0.1
- Cascading risk: 3 → 0.6
- TTR difficulty: 3 → 0.3
Total = 3.0
- P1 outage internal dashboards (ops)
- Revenue impact: 2 → 0.8
- Customers affected: 1 → 0.2
- Regulatory risk: 1 → 0.1
- Cascading risk: 4 (blocks ops’ ability to react) → 0.8
- TTR difficulty: 4 → 0.4
Total = 2.3
- P3 training backlog delaying experiments
- Revenue impact: 1 → 0.4
- Customers affected: 1 → 0.2
- Regulatory risk: 1 → 0.1
- Cascading risk: 2 → 0.4
- TTR difficulty: 2 → 0.2
Total = 1.3
Ranking by score (higher = higher priority):
- P2 model-quality degradation (3.0)
- P1 internal dashboards outage (2.3)
- P3 training backlog (1.3)
Justification:
- VIP customers: model degradation risks churn and high immediate revenue loss and reputational damage; fixing model / rollback mitigates customer SLA breaches.
- Dashboards: while P1, impact is internal; high cascading risk because ops visibility is reduced — must restore quickly but can be staffed after immediate customer-facing fixes.
- Training backlog: low immediate business impact; can be deferred or run on spot capacity.
Resource allocation (5 engineers):
- 3 SREs + 1 AI engineer focused on P2: diagnosis (input drift, feature pipeline, recent model changes), quick mitigation (roll back to prior model version, deploy feature flags), communicate with VIP account managers.
- 1 SRE on P1: restore dashboards (switch to secondary metrics, recreate critical alerts), enable runbooks for ops, escalate if unable to restore in X hours.
- 0.5 SRE + 0.5 AI engineer time on P3: triage, prioritize critical experiments, shift low-priority jobs to off-peak or cheaper GPU pools.
Mitigations & communication:
- Immediate: RACI, incident channels, ETA updates every 30–60min to stakeholders (VIPs, ops, product).
- Short-term: implement model canary rollback, add synthetic tests for model-quality to avoid recurrence.
- Medium-term: automate dashboard failover and prioritize training orchestration improvements.
Metrics to close incident: customer SLA metrics for VIPs restored, dashboard alerting and mean-time-to-detect improved, backlog reduced below threshold. This quantifiable, weighted approach balances revenue risk and systemic stability while making pragmatic trade-offs.
