Situation: A key customer reports a broken critical feature and threatens to churn.
Immediate actions (0–2 hours)
- Triage: Convene a 15–30 minute incident huddle (PM, Eng lead, SRE, CS rep). Capture: affected scope, impact, reproduction steps, user/workaround, logs.
- Customer acknowledgement (within 1 hour): CS/PM send an initial transparent message.
Customer message — initial
Hello [Customer Name],
Thank you — we’ve received your report about [feature]. We’re treating this as high priority. Our engineering team is triaging now; I’ll update you within 2 hours with status and expected next steps. We appreciate your patience and will prioritize a fix.
— [Your Name, PM]
Short-term engineering path (2–24 hours)
- Scoped bug vs. regression vs. outage. If rollback possible, prepare Canary rollback (2–6 hours). If hotfix needed, assign 1-2 engineers for patch + automated tests + quick QA.
- Define success criteria (repro steps pass, smoke tests green).
- Daily cadence: 3 checkpoints (2h, 8h, 24h).
Customer message — 8/24 hour update
Hello [Customer],
Update: Root cause identified as [brief]. We are implementing [rollback/hotfix] and expect deployment by [ETA]. Interim workaround: [if any]. We’ll notify immediately when fixed and validate with you.
— [Name]
Internal stakeholder alignment (immediate)
- Sales: Share impact, account risk, renewal timeline, and recommended goodwill (discount/credits/demo). Template: concise impact + ask.
- CS: Lead customer comms, coordinate validation, schedule follow-up.
- Legal: If contract SLAs or credit obligations triggered, notify legal to review and pre-clear suggested remediation.
- Leadership: Brief execs if account is strategic.
Internal update template (to Sales/CS/Legal)
Subject: Incident: [Customer] — [Feature] — Impact & Ask
Summary: Impact, root cause (if known), ETA for fix, validation plan, recommended customer remediation (e.g., 10% credit / extended trial), sales/CS asks (e.g., pause renewal conversation, assigned CS rep for follow-up), legal: review SLA exposure.
Validation & close (24–72 hours)
- Deploy fix to staging → run regression suite → deploy to prod during low-risk window or emergency release. Coordinate CS to validate with customer.
- Post-mortem within 72 hours: timeline, root cause, remediation, verification of preventive actions, and customer-facing apology + remediation offer.
Customer message — resolution
Hello [Customer],
We’ve deployed a fix for [feature] at [time]. Our testing and your validation show [result]. We apologize for the disruption. As a gesture, we’re offering [credit/discount/feature extension]. Please let me know a convenient time to review and confirm you’re fully operational.
— [Name]
Metrics and follow-up
- Track time-to-detect, time-to-fix, customer satisfaction (CSAT), and renewal outcome.
- Implement guardrails (monitoring, tests, rollout policies) and communicate completed improvements to the customer in 1–2 weeks to rebuild trust.
Why this works
- Fast, transparent communication calms customers.
- Clear internal asks avoid finger-pointing and accelerate fixes.
- Offering remediation and demonstrating follow-through reduces churn risk.