Situation framing (one-liner)
I lead the cross-functional response as TPM: coordinate engineering triage, own timelines and stakeholder comms, and align short-term fixes with measurable prevention and team support.
1) Immediate technical triage (0–2 hours)
- Assemble responder pod (SRE lead, service owner, on-call, TPM) and open incident channel.
- Capture timeline, impact scope (customers, API endpoints, regions), error rates, latency, and SLA metrics.
- Run quick hypothesis checklist: recent deploys, config changes, DB/storage, third-party failures, autoscaling, rate-limits.
- Execute parallel tasks: revert suspect deploys, roll back config, scale capacity, toggle feature flags, enable circuit breakers.
- Preserve evidence: collect logs, spans, metrics, and configuration snapshots.
2) Customer & executive communication timeline + templates
- T+30m: Acknowledge + high-level impact (public customers).
- T+60m: Interim update with scope and mitigation in progress.
- T+4hrs: Detailed update + ETA for next update.
- Post-incident (24–48hrs): Root cause outline and remediation plan.
Customer template (30m):
"We are aware of an incident affecting [service/APIs], impacting [customers/regions]. Our engineers are investigating. We will provide an update by [T+60m]. Contact [support link]/[account rep]."
Executive template (30m):
"Incident affecting [service]; user impact: [# customers/% traffic/errors]. On-call and SRE engaged. Initial mitigation: [action]. Next update: [T+60m]. I own stakeholder comms."
3) Short-term mitigations (first 24 hours)
- Roll back risky deploys or toggle flags.
- Scale up capacity, apply emergency throttles or rate-limits for non-critical traffic.
- Route traffic to healthy regions/instances.
- Provide customer workarounds (e.g., retry logic, alternate endpoints).
4) Measurement to prevent recurrence
- Define success metrics: error rate < X, p95 latency < Y, no SLA breaches for 30 days.
- Add alerting thresholds tied to business impact and automated runbooks.
- Postmortem deliverables: timeline, root cause, corrective actions, owners, ETA.
- Track remediation in a public-facing SLA improvement plan with quarterly checkpoints.
5) Preserve engineering morale
- Shield engineers from customer pressure; TPM/Eng Lead handle comms.
- Publicly credit team for quick actions; focus on facts not blame.
- Limit on-call rotations; authorize OT compensation/time-off.
- Host a blameless postmortem, capture learnings, and commit to concrete tooling/process investments.
I would run this plan, iterate as we learn, and ensure customers and execs have clear, honest timelines while protecting the team and delivering measurable fixes.
