Situation & goals (one line)
Post-release triage must quickly protect customers, determine reproducibility, and allocate limited engineering capacity to the highest-impact fixes.
Triage roles
- Triage Lead (rotating EM/tech lead): owns decisioning, SLA tracking, stakeholder comms.
- On-call Engineer (engineer): validates reproducibility, provides technical assessment.
- Product Manager / Support Rep: provides customer context, business priority.
- QA/Telemetry Owner: reproduces in lab, supplies logs/metrics.
- Engineering Manager (you): approves capacity shifts, schedules follow-up work.
Severity levels & definitions
- P0 (Showstopper): complete outage or data loss for many customers; no workaround.
- P1 (High): major feature broken for key customers; partial workaround possible.
- P2 (Medium): single-customer or non-blocking bug with workaround.
- P3 (Low): cosmetic or minor UX issue.
SLAs for triage response
- Acknowledge (initial response): P0 = 15 min, P1 = 1 hour, P2 = 8 hours, P3 = 48 hours.
- Repro/initial analysis: P0 = 1 hour, P1 = 8 hours, P2 = 3 business days, P3 = 2 sprints.
- Decision (fix now / schedule / won't-fix): same windows as reproducibility.
Balancing reproducibility & capacity
- Use a scoring matrix: Impact (customer count / revenue) x Severity x Reproducibility cost (estimated engineer-hours). Prioritize high score with low repro cost.
- Quick repro-first rule: if unreproducible in 1 hour, collect telemetry and deprioritize to scheduled bug backlog unless business-critical.
- Dedicated fast-lane: reserve one small rotation (1–2 engineers or a bug sprint) for P0/P1 fixes each week to avoid context-switch overload.
Tools & tracking
- Issue tracker (Jira) with custom fields: severity, repro status, customer impact, estimated effort, SLA timestamps.
- Alerting/ops (PagerDuty) for P0/P1 + Slack incident channel template.
- Telemetry & replay (Datadog, Sentry, logs, tracing) linked in ticket.
- Confluence runbook + postmortem templates.
Escalation to incident response
Escalate immediately when one or more apply:
- Wide outage affecting many customers or service-level objective breach.
- Confirmed customer data corruption/loss or regulatory impact.
- Security breach or compromise.
- No workable mitigation and business-critical customer impacted.
When escalated: open incident, notify execs, allocate incident commander, and follow incident lifecycle (contain, mitigate, root cause, postmortem).
Metrics & feedback
- Track SLA compliance, mean time to acknowledge, mean time to repair, repro-to-fix ratio, and backlog age. Review weekly in leadership meeting and adjust capacity or runbooks.
This process gives fast protection for customers, enforces reproducibility discipline to avoid wasted effort, and formalizes capacity trade-offs so engineering focus aligns with business impact.
