Incident Communication and Stakeholder Management Questions

Direct answer

Stop the error, quantify who was hit and by how much, then tell every affected customer proactively and specifically, before they discover it. Refund automatically, add a goodwill gesture (a credit beyond the refund, given to acknowledge the inconvenience), reach out personally to those who churned, and keep a visible weekly rhythm of updates until you can show what controls now prevent a repeat.

Sequence over the following weeks

• Day 0: fix and freeze the pricing logic, and confirm the exact scope (which customers, which period, over- or under-charged).
• Day 1: a public acknowledgement plus a personal email to each affected customer with their own amounts. Finance approves the remedy; legal reviews wording.
• Days 2 to 7: refunds issued automatically (no claim form), a goodwill credit, support briefed with a single script and a dedicated queue. Personal outreach to customers who cancelled, with an easy way back.
• Weeks 2 to 4: a short public write-up of what happened and which controls were added (pricing changes now need review and a reconciliation check before billing runs; a reconciliation check compares what the system billed against what the price list says it should have billed, and flags any mismatch). Then report results: refunds completed, complaints trending down.

Remediation and compensation principles

• Automatic, not by request. Making people ask compounds the harm.
• Exact numbers per customer, so the message is credible.
• Undercharged customers: normally do not claw back (ask them to pay the shortfall after the fact); decide with finance and legal, and say so plainly.
• Goodwill should be proportionate and simple, not a maze of tiers.

Rebuilding trust

Trust returns through repeated proof, not a single message. Track refund completion rate (refunds paid divided by refunds owed, for example 1,140 of 1,200 is 95%), complaint volume, and churn (customers who cancel) among affected customers versus a baseline of similar unaffected ones, meaning customers of the same plan and age who were not billed wrongly. If affected customers cancel at 4% a month and the baseline is 1%, the error is still costing you. Report those honestly to leadership.

Worked example (illustrative numbers)

A bug billed 1,200 customers $49 instead of $39 for one month.

• Overcharge per customer = $49 minus $39 = $10.
• Total refund = 1,200 times $10 = $12,000.
• Goodwill credit of $15 each = 1,200 times $15 = $18,000.
• Total cost of remedy = $12,000 plus $18,000 = $30,000.

Now compare it with lifetime value (LTV, the total revenue a customer brings before leaving). Suppose a customer pays $39 a month and stays 24 months on average: LTV = $39 times 24 = $936. If just 5% of the 1,200 churn because of the error, that is 60 customers, and 60 times $936 = $56,160 of lost revenue. The $30,000 remedy is smaller than that loss, which is the case you make to finance.

Trade-offs and pitfalls

• Compensating generously risks setting a precedent; compensating stingily amplifies public complaints. Tie the amount to actual harm plus a modest gesture.
• Vague apologies ("some customers may have been affected") read as evasion; specific ones read as ownership.
• Announcing a fix without a control (review or reconciliation) invites the same story next quarter.

Direct answer

Make one leader decide the trade-off fast, then give each group a specific ask in its own terms, and tell the customer before they find out. I would recommend prioritising the fix: name a small swarm team (a few engineers pulled off other work to focus on this one problem), slip a specific roadmap item by a stated amount, and give the customer a plan with an update every 12 hours.

Structured elaboration

Hours 0-4: fact base and decision

• What is the SLA (service-level agreement, the contractual performance promise), how far will we miss, what is the cost of the miss (credits and relationship), what is the fix cost.
• Take a one-page trade-off to the engineering director, who decides.

Tailor the message

Customer contact: proactive, before the SLA is missed, with facts, mitigation and a named contact.

Worked example (illustrative)

Suppose the SLA credit is 10% of a $40,000 monthly fee: 0.10 x 40,000 = $4,000. The account's annual value is 12 x 40,000 = $480,000. The credit is small; the risk is losing the renewal, which is 120 times the credit (480,000 / 4,000). Roadmap slip: one feature delayed by 3 days.

Pricing the fix (the cost comparison, with assumed figures). Two engineers for 48 hours at an assumed loaded cost of $100 per hour each: 2 x 48 x 100 = $9,600. The 3-day slip of a feature assumed to be worth $1,500 per day: 3 x 1,500 = $4,500. Total cost of fixing: 9,600 + 4,500 = $14,100.

Pricing the risk of not fixing. The $4,000 credit is certain. Add an assumed 10% chance the customer walks at renewal: 0.10 x 480,000 = $48,000. Expected exposure: 4,000 + 48,000 = $52,000.

Decision: $52,000 of exposure against $14,100 of fix cost is roughly 3.7 times (52,000 / 14,100), so prioritising the fix wins. The "120 times" figure is the upper bound if the renewal is lost outright; the 10% assumption is what turns it into a defensible expected number. If the churn chance were only 1%, exposure would be 4,000 + 4,800 = $8,800, below the fix cost, and the answer would flip. That sensitivity is what the director needs to see.

Message to the customer: "Our current fix path means we may not meet the response-time target by Thursday. We are assigning two senior engineers and will update you at 9am and 9pm daily. Your account manager is your single contact."

Trade-offs and pitfalls

• Framing it as "engineering is behind" makes teams defensive; frame it as a cost comparison.
• Do not overpromise a date to calm the customer.
• Avoid pulling everyone; a small swarm beats a crowd.
• What would flip it: if the fix cost exceeds the credit and the relationship value, or the miss is unavoidable, communicate honestly and offer a mitigation instead.

Direct answer

Treat customer reports as evidence until proven otherwise. Tell customers plainly that we believe them and are investigating, ask for specific data that lets us find the failing requests, and give a next-update time. Internally, open a low-severity incident with one named owner, so the investigation has a channel and a clock. Never tell a customer "our monitoring shows no problem".

What to ask customers for

• Exact timestamps with time zone (three or four examples, not "sometimes").
• Request or correlation IDs (an identifier attached to each request so we can trace it through logs), or the full error text and HTTP status code.
• Account or tenant ID (a tenant is one customer's account in a system shared by many customers), region, and which endpoint or screen.
• Client details: browser or SDK version, mobile or web, network (office, VPN, mobile data).
• How often (1 in 10 requests? only large uploads?) and whether it began after any change on their side.
• Whether a retry succeeds, since that points to flaky dependencies rather than a hard failure.

Why dashboards can be clean while customers are right

1. Averages and 5-minute rollups (metrics squashed into one number per 5-minute bucket) hide short spikes. Check P99 (the latency that 99 in 100 requests beat). Example: in one 5-minute window, 9,800 requests take 100 ms and 200 take 10,000 ms. The average is (9,800 x 100 + 200 x 10,000) / 10,000 = 298 ms, which looks healthy, but sorting the requests from fastest to slowest, the 9,900th (the P99) is 10,000 ms, and 1 in 50 requests waited 10 seconds.
2. Metrics may cover the wrong layer: healthy servers behind a load balancer (the component that spreads traffic across servers), while the CDN (content delivery network) edge, the outer servers customers reach first, or the client side fails.
3. A small slice (one tenant, one region, one request size) vanishes inside a global rate. Cut by tenant and region.
4. Requests that die before reaching our code (DNS, which looks up a site's address; TLS, which encrypts the connection; the edge) never appear in application metrics.
5. Sampling (recording only a fraction of events) or log gaps may drop exactly the failures.

Internal communication

• One incident channel, one investigator, one person handling customer contact so support does not improvise.
• Post a hypothesis list ("edge, one shard, third-party dependency", where a shard is one slice of a split database) and mark each confirmed or ruled out.
• Give support an approved holding line and tell them what to collect, so every ticket arrives with the fields above.
• Set escalation triggers in advance: raise severity if three independent customers give matching evidence, or if a paying tier is affected.

Worked example (illustrative)

Customer message: "Thanks for flagging this. We can't see a fault on our side yet, but that doesn't mean you're wrong, and we are investigating. Could you send two or three request IDs with times (including time zone) where you saw an error, plus the error text? We will update you by 16:00 UTC even if we have nothing new."

Internal note: "Low-severity investigation opened. Owner: A. Kim. Three reports, all EU tenants, all file uploads. Checking edge logs and P99 for uploads first. Support: please collect request IDs, do not promise a fix."

Pitfalls

• Saying the fault is on the customer side before evidence exists.
• Asking for logs the customer cannot produce. Ask for things they can see.
• Silence: a "still investigating, next update at X" is worth sending.

Direct answer

Customers contracted with you, so you own the outage to them even if a vendor caused it. Tell them plainly what is affected, what you are doing and when the next update comes, mentioning the supplier only where it helps and is public. Executives need exposure and options. Handle the vendor through formal escalation and evidence, and check whether your commitments to customers exceed what the vendor promised you.

What customers hear

• Impact, status, workaround if any, next update time.
• Ownership: "a component provided by a supplier failed, and we are responsible for restoring your service."
• Do not use the vendor as an excuse. Name them only if their status page (the public web page where a provider posts its outage updates) is already public and it aids understanding.

What executives hear

• Customer exposure and the SLA (service-level agreement) position.
• Options: wait for the vendor, switch to a fallback (a backup supplier or backup path), or degrade gracefully (keep the core service running with the affected feature switched off), each with cost and risk.
• Legal and finance implications of credits and vendor claims.

Handling the vendor

1. Escalate at the highest severity through the named support and account contacts; get their incident reference (the ticket number that lets both sides point to the same case).
2. Send evidence: timestamps, error samples, affected regions.
3. Ask for their status cadence, root cause report and their own credits.
4. Keep it professional and private; public fights rarely speed recovery.
5. Log every downtime minute, since credit claims usually need records.

Contractual commitments: check for a gap

Your promise to customers and the vendor's promise to you are different numbers. Example (illustrative):

• Vendor SLA: 99.5% monthly. Allowed downtime = 43,200 minutes times 0.005 = 216 minutes.
• Your SLA to customers: 99.9%. Allowed downtime = 43,200 times 0.001 = 43.2 minutes.
• Outage: 180 minutes.

The vendor has not breached (180 is under 216, so no vendor credit), but you have breached your customers' SLA (180 is far over 43.2). You absorb the difference: your allowance was 43.2 minutes, so 180 minus 43.2 = 136.8 minutes are unprotected, and you owe your customers credits for that breach with no vendor credit to offset it. If your contract credits 10% of the monthly fee for this band (illustrative) and a customer pays $5,000, that is $500 each, so 100 such customers cost $50,000. Legal should check whether your terms have a carve-out (a clause excluding failures caused by third-party suppliers from your SLA; many do not) before anyone promises or denies credits.

Longer term

Record the finding in a vendor risk review (a written assessment of how dependent you are on a supplier and what happens if it fails): a fallback or second supplier for critical paths, contract renegotiation to narrow the gap, and better alerting on the vendor's status.

Trade-offs and pitfalls

• Blaming the vendor publicly may be accurate but breaks trust with customers who want accountability from you.
• Waiting silently for the vendor leaves customers guessing.
• Assuming your SLA has a third-party carve-out without reading it is a costly assumption.

Direct answer

A good public post-incident summary tells customers, in plain words: what went wrong, who was affected and how, what we did to fix it, and what we are changing so it does not recur. For a day of wrong results the priority is impact honesty: say exactly what was wrong, for whom and what they should check or redo. Keep it plain by leading with effects, and stay transparent by stating the unknowns and owning the cause without blame or spin.

Draft (illustrative, placeholders in brackets)

Title: Incorrect results in [feature], [date] 08:00 to [date+1] 08:00 UTC

Summary
For about 24 hours, [feature] showed incorrect [figures] for some customers.
The problem is fixed and results are now correct.

What happened
A change we released on [date] altered how [figures] were calculated.
It passed our tests but behaved differently on real data. We noticed when
customers reported mismatches and rolled the change back.

Who was affected
Roughly [X percent] of accounts that ran [feature] during the window. If
you did, your affected reports are listed in [link]. Results outside that
window were not affected.

What you may need to do
If you used those [figures] for decisions or exports, please re-run them.
We have already corrected stored results and will email affected accounts.

What we are changing
- Tests using production-like data before release.
- An automatic check that flags results differing from the prior day.
- A faster path from a customer report to our on-call team.

We are sorry. Questions: [contact].

Filled-in version of the same summary (illustrative facts)

Title: Incorrect totals in Usage Reports, 3 March 08:00 to 4 March 08:00 UTC

Summary
For about 24 hours, Usage Reports showed totals that were too low for some
customers. The problem is fixed and totals are now correct.

Who was affected
About 1 in 10 accounts that ran a report in that window. Affected reports
are listed at status.example.com/incident-0303.

What you may need to do
If you used those totals in a billing review or export, please re-run them.

Why it is written this way

• Title and first line carry the whole message: impact, period and status.
• Impact section is concrete: who, what, when, how to find out if you were affected. Wrong-results incidents differ from outages because customers may have acted on bad data, so the "what you may need to do" section is essential.
• Cause in one plain paragraph: name the type of cause (a change that behaved differently on real data), skip internal names.
• Prevention items are specific and checkable, not "we will be more careful".
• Transparency without exposure: state what we know and what we are still checking ("we are reviewing whether any exports were sent to third parties"). Do not include customer names, security-sensitive detail or speculation.

Worked example of plain wording

Three internal terms, in plain words: a regression is a change that breaks something that used to work; an aggregation job is a scheduled program that adds up raw records into totals; a stale cache hit means the system reused an old saved answer instead of recalculating.

Internal: "A regression in the aggregation job's rounding logic produced a stale cache hit." Public: "A change to how we add up your figures gave wrong totals for some accounts." Same fact, one less barrier.

Trade-offs and pitfalls

• Vagueness ("some issues") looks evasive; over-precision that you cannot back up (an exact affected-customer number you have not verified) is worse. Use verified numbers or clear ranges.
• Passive blame-shifting ("a vendor problem") loses trust unless it is true and you still own the customer outcome.
• Have legal and support review before publishing, and align the summary with what support has already told customers.
• Publish soon after resolution; a late summary looks like hiding.