Incident Communication and Stakeholder Management Questions

Direct answer

Run it in two stages: first finish an internal blameless review so I know the facts and my own contribution, then hold a structured customer review with a pre-read, a shared timeline, an honest discussion of the architecture's part, and dated commitments. Keep the SLA (service-level agreement, the availability promise in the contract) credit and contract discussion separate, owned by account management and legal, so the technical review stays about learning.

Structured elaboration

Prepare

• Internal postmortem first (a written review of what happened and why; blameless means we examine decisions and systems, not people), so I never learn a fact in front of the customer.
• Get the SLA (service-level agreement, the availability promise in the contract) math right. Example: a 99.9% monthly target on a 30-day month allows 43,200 minutes x 0.001 = 43.2 minutes down. If the outage was 95 minutes, availability = (43,200 - 95) / 43,200 = 43,105 / 43,200 = 99.78%, a breach.
• Pre-read sent 24 hours ahead: timeline, impact, causes, actions.

Agenda (60 minutes)

1. Impact in their terms (10 min).
2. Timeline, agreed by all (10 min).
3. Contributing factors, including our architecture (15 min).
4. What we change, dated and owned (15 min).
5. What we need from them, questions (10 min).

Handling "your architecture is to blame"
Do not dispute it. Separate the trigger from the design weakness that let it become an outage: "The trigger was X. Our design lacked Y, so it turned into an outage. That part is ours." Then show what we change.

Worked example (illustrative)

Opening line: "Thank you for making time. I will not defend the outage. My aim is for you to leave with a clear account and commitments you can hold us to." When the customer says "your architecture is the problem", answer: "Partly, yes: a single queue was a shared failure point, and we did not test its failure. Here is the change and the date."

Trade-offs and pitfalls

• Arguing contract terms in the technical meeting turns learning into a dispute. Park credits with the account owner.
• Do not promise "no recurrence"; commit to controls and a follow-up review in 30 days.
• Do not blame a vendor unless the evidence is shared and settled.
• What would change the approach: if the customer's counsel attends, legal joins my preparation and sets the wording.

Direct answer

Tell people early, tell them what is true, own the decision, give them a plan with a date for the next update, and then be visibly reliable on that date. Trust survives when the other side learns bad news from you first, sees you understand their impact, and can predict when they will hear from you next.

The approach, in order

1. Tell them first, fast. Give a short heads-up before they discover it. Silence reads as concealment.
2. State facts in three buckets: what we know, what we do not know yet, what we are doing to find out.
3. Describe the impact in their terms (their launch date, their customers), not our internals.
4. Own the decision. Say what we decided, why it seemed right at the time, and what we would change. No blame on a vendor or a colleague.
5. Offer a concrete mitigation and a revised date, with confidence stated honestly ("firm" or "best estimate").
6. Commit to the next update time and keep it even when there is nothing new.
7. Follow through, then close the loop with what changed so it does not repeat.

Different audiences, same facts. Clients get impact, remedy and dates. Internal stakeholders (sales, support, leadership) additionally get scope, risks and what to say and not say, so everyone gives the client one consistent story.

Worked example (illustrative)

We chose to migrate a client's data pipeline to a new platform, and a compatibility problem pushes delivery from the 10th to the 24th.

"Heads-up before your team plans around the 10th: delivery will move to the 24th. We chose to migrate to the new platform for reliability, and we underestimated a compatibility problem in your data format. That estimate was ours. Your reporting launch is affected; to protect it we can deliver the old-platform report on the 10th as a stopgap. I will update you Thursday at 3pm either way."

Then send the internal note to account and support teams with the same dates and a line on what not to promise.

Trade-offs and pitfalls

• Over-promising a new date to sound reassuring is the most common trust-killer. A later, firm date beats an early, missed one.
• Long apologies and technical detail crowd out the plan. One sentence of ownership is enough.
• Do not let different stakeholders hear different versions.
• What would change the approach: if contracts or safety are involved, involve legal before the wording goes out.

Direct answer

An executive one-pager answers three questions in the first ten seconds: how bad was it, is it over, and what do I need to do. Lead with the bottom line, keep every point in business terms (customers, money, trust), and leave off anything an engineer would need but an executive would not act on.

The six points, in order

1. Bottom line. One sentence: what broke, for whom, and whether it is fixed now. Example: "Checkout was down for 65 minutes on Tuesday afternoon (14:02 to 15:07) and cleanup finished after 98 minutes; it is fully restored and we have no evidence of lost orders. Use the customer-facing duration (until mitigation) in the headline, and keep total incident length for the timeline."
2. Customer and business impact. Who was affected and how much: share of customers, orders or revenue at risk, support tickets, any SLA (service-level agreement, the contractual uptime promise) exposure. Use ranges labelled as estimates if the final number is not confirmed.
3. Timeline in four timestamps. Started, detected, mitigated (customers no longer hurt), fully resolved. The gap between "started" and "detected" is the number executives care about, because it shows how long customers suffered before we knew.
4. Cause in one plain sentence. "A configuration change to the payment service was released without a check that would have caught it." No component names an executive cannot picture.
5. What we are changing. The top three corrective actions, each with a named owner and a date. This is what turns an apology into a plan.
6. Decisions or asks, plus residual risk. What you need from leadership (budget, a priority call, approval to contact customers) and what could still go wrong. If you need nothing, say so explicitly.

What to leave off

• A minute-by-minute log, chat transcripts, dashboards and tool names (they belong in the postmortem, the blameless written review of an incident).
• Names of individuals who made the change. Blame teaches people to hide problems.
• Unverified speculation, and a long list of every possible improvement (three owned actions beat twenty).
• Technical debate about alternatives considered during the fix.

Worked example (illustrative numbers)

Timestamps: started 14:02, detected 14:19, mitigated 15:07, resolved 15:40.

• Time to detect = 14:19 minus 14:02 = 17 minutes.
• Customer-facing duration until mitigation = 15:07 minus 14:02 = 65 minutes.
• Total incident length = 15:40 minus 14:02 = 98 minutes.

Point 3 on the page then reads: "We took 17 minutes to notice; customers were affected for 65 minutes; cleanup finished after 98." An action in point 5 follows directly: "Add an alert on failed payments (owner: SRE lead, due in two weeks) so detection drops from minutes to seconds."

Trade-offs and pitfalls

• Completeness versus readability: if it does not fit on one page, the extra belongs in an appendix link, not in smaller font.
• Overconfident causes: if the cause is not confirmed, write "current best understanding" and the date the confirmed version follows.
• Burying the ask: a page whose ask sits at the bottom of a wall of history gets skimmed and nothing is decided.

Direct answer

Call the customer yourself, quickly, with senior backing, before they hear it any other way. Own the recommendation, apologise for the impact, share only verified facts, and offer concrete help to recover data and make the customer whole. Internally, run it as one coordinated response with a single owner, so the customer hears one consistent voice.

What you tell them and when

• Within hours of confirmation: a phone or video call from a senior person (an architect lead or executive sponsor), then a short written follow-up. Not a ticket reply, not email first.
• On the call: what we know (their environment, what was lost, time window), what we do not yet know (why exactly), that the architecture we recommended played a part (once confirmed), and when they will hear next.
• Sample opening of the call: "Thank you for making time. I am calling personally because we recommended the architecture you used, and it now looks like it contributed to the data you lost. I am sorry. Here is what we have confirmed, what we have not, and what we are doing today. You will hear from me again by 6 pm."
• Sample written follow-up: "Following our call: confirmed so far, your storage in the affected time window was deleted; not yet confirmed, the exact cause. Your named contact is [name]. Recovery engineers are assigned. Next update by 9 am tomorrow."
• Do not speculate about the cause, blame their configuration, or promise compensation before finance and legal agree.

What you offer

1. Recovery help first: dedicated engineers, attempts to restore from backups or logs, and help verifying what came back.
2. Independent architecture review at our cost, to fix the design for them.
3. Commercial remedy (money-based compensation): credits or fee relief, approved by finance and legal within the contract, framed as a gesture, not as an admission of liability (legal responsibility for the loss).
4. A cadence: a named contact, updates on a fixed rhythm, and a written analysis after the facts are confirmed.

Coordinating inside the company

Also check every other customer who followed the same recommendation, and send them a proactive advisory (a warning sent before they report any problem). That protects them and shows the first customer you are fixing the cause, not just their case.

Worked example (illustrative timeline)

A guide recommended replicating storage across zones (separate data-centre locations) for durability. Replication means every change is automatically copied to the other locations, and that includes deletions, so a bad script erased the data in every zone at once. The customer also had no point-in-time backups (saved snapshots of the data at earlier moments that you can restore from), so there was no clean earlier copy to return to.

• Hour 0 to 2: facts confirmed, owner and sponsor named, legal engaged.
• Hour 2 to 4: sponsor calls the customer, apologises, offers recovery engineers.
• Day 1: written summary, restoration under way, next update time set.
• Day 3: recovery status and the architecture review scheduled.
• Week 1: commercial remedy agreed, guidance corrected, advisory sent to similar customers.

Trade-offs and pitfalls

• Full ownership versus legal exposure: apologise for impact and own the recommendation, but let legal shape statements about liability.
• Generous offers made too early can be withdrawn or contradicted; agree first, then say.
• Fixing only this customer while the same guidance sits in the docs is the costliest mistake.

Direct answer

Sequence it inside-out but fast, aiming for the first hour, and no later than about two hours for the public statement: get verified facts from engineering, align legal and executives on one position, brief sales and support so nobody freelances, reach the customer's leadership privately, and only then, or at the same moment, publish a short public statement. The statement acknowledges impact and commits to updates. It does not assign blame, speculate, or litigate (argue the dispute point by point in public as if in court).

Sequence

1. Engineering (minute 0). One incident owner (the single person accountable for the response and its facts) and one fact sheet (a one-page internal record of confirmed facts). Example sheet: "Confirmed: API errors 14:02 to 14:40 UTC, all regions. Not confirmed: whether the customer's config contributed. Evidence saved: deploy log, error graphs, change history. Owner: A. Kim. Updated 14:45." It also lists what we do not know and what evidence is preserved (logs, timelines, change history).
2. Legal and executives (minutes 0 to 30). Agree the position, who speaks publicly, and what can and cannot be said. Legal reviews the wording, but does not get to make it unreadable.
3. Sales, account and support (before the statement). A short internal brief and one approved script, so customers hear one story and nobody freelances (improvises their own public answer) or tweets a private opinion. Example script line: "We are aware of the disruption and are working with the customer. Please direct questions to [contact]; do not discuss cause."
4. The customer's leadership (privately, before or with the statement). A senior call. They should never learn our position from social media.
5. Public statement (within about the first hour or two). Then updates on a promised schedule.

What the public statement does

• Acknowledges the impact and shows we take it seriously.
• States only verified facts, and that we are investigating jointly.
• Names when the next update comes.
• Offers a direct channel to the customer, so the disagreement moves off the timeline.

What it does not do

• Blame the customer, even if their configuration contributed. Correct the record later, privately or in a joint postmortem (a written, blameless review of what happened, written together with the customer) with their consent.
• Speculate on cause or promise compensation not yet approved.
• Reveal the customer's data or setup, or mention legal action.
• Argue point by point. Public arguments are rarely won.

Worked example (illustrative statement)

"We know [Customer]'s users were affected by an outage on Tuesday, and we are sorry for the disruption. We are working with their team to establish exactly what happened and will share what we confirm by 6 pm today. We are in direct contact with their leadership."

It acknowledges, commits to a time, and leaves out the cause and any blame.

Trade-offs and pitfalls

• Silence lets the customer's version stand, but a wrong or defensive statement is worse than a late one. Speed matters, facts matter more.
• Legal caution versus honesty: push for plain language, since evasive statements read as guilt.
• If the fault really is the customer's, a factual correction comes later, jointly, not as a rebuttal.