Incident Communication and Stakeholder Management Questions

Direct answer

Separate the two conversations from the fix. Executives get a steady, honest status and a clear view of exposure. Customers get an acknowledgement, a named contact, and a rhythm of updates. Credit or compensation (money or service time given back for missing the promised uptime) is discussed through the contract process (the steps the customer agreement lays out for claiming it), after the facts are measured, and never improvised mid-incident by the TPM (Technical Product Manager).

Executive conversation

• One update template: state, impact, next step, next update time. Send it on a fixed cadence so they stop calling engineers.
• Flag exposure: which customers have SLA (service-level agreement, the contractual uptime promise) commitments, and the rough credit liability (the total credits you may owe), labelled as an estimate.
• Ask for decisions only they can make: engineer priorities, approval of customer messages, and who speaks to the largest accounts.

Customer conversation

• A senior, named contact per high-profile account, one call early, then written updates on the promised schedule.
• Say what is known, what is not, and when you will speak next. Do not guess an ETA for the fix.
• If asked about credit: "We will measure the downtime against our agreement and follow the contractual process. We will come back to you on that once service is stable." Do not debate liability during the incident.

How SLA credit is calculated

Uptime SLAs are usually monthly. A 99.9% promise in a 30-day month allows (30 days times 24 hours times 60 minutes = 43,200 minutes in the month):

$43{,}200 \times 0.001 = 43.2 \text{ minutes}$

Suppose the outage lasted 120 minutes. Availability is:

$1 - \frac{120}{43{,}200} \approx 99.72\%$

That is below 99.9%, so a credit tier applies. A credit tier is a table in the contract mapping availability to a refund of that month's fee. For example (illustrative, not from any real contract): below 99.9% but at least 99.0% earns a 10% credit; below 99.0% earns 25%. At 99.72% the customer falls in the first band, so on a $5,000 monthly fee the credit is 10% times $5,000 = $500. Tier percentages differ by contract, so quote them from the actual agreement, not from memory. Credits often require a customer claim, but for top accounts many companies apply them automatically as goodwill.

Recommendation

Apply the contractual credit proactively for high-profile customers once the numbers are confirmed. It costs little against the relationship, and the decision belongs to finance and legal, with the TPM preparing the numbers. Offer anything beyond contract only with executive approval, and after the fix.

Trade-offs and pitfalls

• Promising credit early can be read as admitting fault and complicate legal position.
• Refusing to discuss credit at all reads as stonewalling. Acknowledge the topic and give a timeline.
• The TPM should not be both incident commander (the person directing the technical response) and communicator; delegate the fix so someone owns the messages.

Direct answer. Judge incident communication on four questions: was it fast, was it steady, was it accurate, and did it help the reader act. Cover both sides: customer-side signals (support tickets and a sample customer survey) and a responder-side signal (a survey of the people running the response), and never trust a single number, because each can be gamed or can move for reasons unrelated to quality.

Definitions (numerator, denominator, window).

• Time to first public update = minutes from detection (the alert firing or the first customer report, whichever is earlier) to the first status page post (the public web page where the company posts live service status). Per incident. Report the median over a quarter.
• Update cadence adherence = updates or resolutions posted within the promised interval divided by all intervals between consecutive posts, for SEV1/SEV2 incidents (the two most severe tiers, e.g. SEV1 = checkout is down for most customers). The final interval, from the last update to resolution, counts as a gap too.
• Accuracy = number of public updates later corrected or retracted divided by total public updates in the window.
• Customer effort = support tickets that mention the incident, per 1,000 affected customers, counted from declaration to 24 hours after resolution. Lower is better only if the outage size is held constant.
• Responder load signal = post-incident survey of the comms lead (the person who writes and sends outward messages) and the IC (incident commander, who runs the response) (1 to 5: "I had the facts I needed when I needed them").

Worked example (data illustrative, arithmetic executed). Three incidents, promised cadence 30 minutes, times as HH:MM. The last interval runs from the final update to resolution.

import statistics


def m(t):
    h, mi = t.split(":")
    return int(h) * 60 + int(mi)

# each row: (name, detected, first_post, all_updates, resolved, promised_cadence_min)
incidents = [
    ("INC-1", "10:00", "10:12", ["10:12", "10:40", "11:10", "11:38"], "11:50", 30),
    ("INC-2", "14:00", "14:35", ["14:35", "15:50", "16:20"], "16:30", 30),
    ("INC-3", "09:00", "09:08", ["09:08", "09:38"], "09:45", 30),
]
total_gaps = on_time = 0
for name, det, first, updates, res, cad in incidents:
    times = [m(u) for u in updates] + [m(res)]
    gaps = [b - a for a, b in zip(times, times[1:])]
    ok = sum(1 for g in gaps if g <= cad)
    total_gaps += len(gaps)
    on_time += ok
    print(name, "time_to_first_update_min =", m(first) - m(det), "gaps =", gaps, "on_time =", ok, "/", len(gaps))
print("cadence adherence =", on_time, "/", total_gaps, "=", round(100 * on_time / total_gaps), "%")
ttfus = [m(i[2]) - m(i[1]) for i in incidents]
print("median time to first update =", statistics.median(ttfus), "min")

Output:

INC-1 time_to_first_update_min = 12 gaps = [28, 30, 28, 12] on_time = 4 / 4
INC-2 time_to_first_update_min = 35 gaps = [75, 30, 10] on_time = 2 / 3
INC-3 time_to_first_update_min = 8 gaps = [30, 7] on_time = 2 / 2
cadence adherence = 8 / 9 = 89 %
median time to first update = 12 min

What the code does, in plain words:

• m() converts "HH:MM" into minutes since midnight so times can be subtracted.
• zip(times, times[1:]) pairs each post time with the next one, so b - a is the silence between consecutive posts. The resolution time is appended to the list so the wait between the last update and resolution is also checked.
• A gap counts as on time if it is at most the promised 30 minutes. 8 of the 9 gaps qualify (only INC-2's 75-minute gap fails), so 8/9 = 89%.
• Time to first update is first post minus detection: 12, 35 and 8 minutes; sorted (8, 12, 35), the middle value is 12. statistics.median also handles an even number of incidents correctly (it averages the two middle values: for 8, 12, 35, 40 it gives 23.5), which indexing at len // 2 would get wrong (it would return 35).

Reading it: the headline 89% looks healthy and the median of 12 minutes looks good, yet INC-2 went 35 minutes before anything was posted and had a 75-minute silence. The median hides the one incident customers remember, so also report the worst case (the maximum, or the 90th percentile, the value that 9 out of 10 incidents come in under, once you have enough incidents, say 20 or more).

How to collect. Timestamps come from the status page's own post history plus the incident tool's declared time (no manual entry). Ticket counts come from support tagging tied to the incident ID. Correction counts come from a "was edited/retracted" flag on posts. Surveys go out within 48 hours to responders and to a sample of affected customers, with one open text question ("what did you not know that you needed to?"). That open text is the qualitative signal, and it is where you learn why the numbers moved.

Which signals mislead.

Senior point. Publish the metrics as a set reviewed in the postmortem (the written review after an incident), never as a personal target for the comms lead, or you will get gaming instead of communication.

Direct answer

Because the outage has plateaued (stable, not getting worse), I would recommend the slower staged hotfix (a hotfix is a small urgent code fix shipped forward; staged means released to a slice of traffic at a time, here 10%, then 50%, then 100%), and say so plainly: downtime is recoverable, permanent data loss is not. I would present it as a decision with named risks and a checkpoint (a pre-agreed time or health signal at which we stop and re-decide), not as a certainty. I would switch to rollback if impact starts to grow, if the hotfix slips past a time limit the business cannot tolerate, or if the data at risk turns out to be small or recoverable.

How I would reason about it

1. Reversibility first. Lost data cannot be undone; an extra hour of degraded service can be apologised for and credited.
2. Quantify both sides (illustrative numbers, stated as assumptions).
3. Look for a third option. Can we export or snapshot the at-risk writes first, so a rollback becomes lossless? That can turn the trade-off into a plan.
4. Set a checkpoint. "If the hotfix is not at 50% of traffic by 15:30, we reconsider."

Worked example (illustrative assumptions)

• A migration is a change to the structure of the database. Rolling back returns the database to how it was before that change, so every order written since the deploy is discarded. Assumption: 300 orders per minute, and 40 minutes have passed since the 13:40 deploy, so the orders written since then number:

$300 \times 40 = 12{,}000 \text{ orders lost permanently}$

• Staged hotfix takes 90 minutes. During that time 20% of orders fail, but customers can retry.

$300 \times 0.2 = 60 \text{ failed attempts per minute}, \quad 60 \times 90 = 5{,}400 \text{ failed attempts}$

The two numbers count different kinds of harm, so I do not subtract them. A failed attempt is a customer who can retry and probably will. A lost order is one the customer thought was complete and now has no record of. The safe comparison is a worst-case bound: even if every one of the 5,400 failed attempts became a permanently lost sale, that is still fewer than 12,000. Realistically many retry, so the true permanent loss is smaller still. So hotfix wins unless the business says it cannot survive 90 more minutes.

Explaining it to each audience

What I would not say: "This is safe", "should be fixed soon", or "we have no choice". State confidence honestly: "I am fairly sure about the cause, less sure about the timing."

Trade-offs and pitfalls

• If lost data can be rebuilt from logs or a payment provider, the calculation changes; verify that before deciding.
• Do not let a stressed executive pick the option in the abstract. Present the recommendation plus the trigger that overturns it.
• Record the decision and reasoning in the incident document for the review.

Direct answer

Governance for incident communication should answer three questions in a short written policy: who is allowed to speak for the company, what needs a second pair of eyes before it goes out, and how we check afterwards. The trick to not slowing responders is to make speed the default: pre-approved wording flows immediately, and only the risky kinds of statements wait for review.

The rules (an excerpt you could adopt)

1. Who posts. Responders never post to customers. During an incident, one named communications lead (appointed by the incident commander, the person coordinating the response) posts to the public status page (a public web page showing current service health) and to customer channels. Backup: a named deputy from a short list.
2. Two lanes of content.
   • Pre-approved lane: messages built from templates whose wording legal and support already signed off ("We are investigating degraded performance in [service]. Next update by [time]"). The communications lead posts these with no review.
   • Reviewed lane: anything that states a root cause, assigns fault, mentions data exposure or customer data, promises credits or dates for a fix, or names a customer. These go to a designated reviewer (legal or the security lead) who commits to answer within 10 minutes; if the reviewer is silent, the update falls back to the pre-approved wording rather than waiting.
3. Internal versus external. The responders' channel is unrestricted. Stakeholder and executive updates follow the template but need no review.
4. Facts. Every claim of impact or cause carries a source ("error rate dashboard, 14:12 UTC"), or is labelled as a hypothesis.
5. Audit. The status-page and messaging tools record author, time and text automatically. The postmortem (the blameless written review after an incident: it looks for weak process, not for someone to blame) includes a check: were all public posts by an authorised poster, and did any reviewed-lane content skip review?
6. Exceptions. If nobody with posting rights is reachable in SEV1 (the most severe level, full outage or major customer harm), any incident commander may post the template lane and records it as a break-glass use for review. Break-glass means an emergency override of the normal rule, like breaking the glass on a fire alarm: allowed when needed, but always logged and reviewed afterwards.

How to enforce without friction

• Use tool permissions, not memory: only the communications roster has publish rights on the status page.
• Keep templates one click away so the compliant path is also the fastest.
• Review a random sample of posts monthly; feedback goes to the process, not to the individual.

Worked example (illustrative)

At 14:20 a database problem slows checkout. The communications lead posts the template at 14:26 with no waiting. At 15:10 the lead wants to write "caused by a faulty configuration change". That is the reviewed lane; the lead submits the draft at 15:10, the reviewer answers at 15:16 (6 minutes) and asks to say "we identified a configuration change as the likely cause", and the lead edits the wording and posts at 15:18 (2 minutes to apply the change). Responders were never interrupted.

What the records look like (illustrative)

An automatic audit record, one line per post:

14:26 UTC | status page | author: A. Rivera (comms roster) | lane: pre-approved (template T1) | review: not required
15:18 UTC | status page | author: A. Rivera (comms roster) | lane: reviewed | reviewer: legal, approved 15:16

A reviewed-lane template for a security or customer-data case: "We are investigating a possible security issue affecting [service]. We will share confirmed details when available. Next update by [time]." Any wording beyond that (what data, how many customers) goes to the reviewer first.

Trade-offs

• Tight review protects against wrong or legally risky statements but slows updates; the two-lane split is the compromise.
• Wide posting rights are fast but produce inconsistent, sometimes incorrect statements. I would start narrow and widen with training.
• What would change my call: a regulated business or a security incident with breach risk moves more content into the reviewed lane.

Direct answer. Stop the public argument first, then fix the structure that let it happen. Move disagreement out of the public channels, restate one shared source of truth, remind everyone of the rule (facts and next steps only, no attribution of fault during the incident), and put the comms lead (the one person who writes and sends outward messages, whether to customers or to the wider company) in charge of what appears publicly. The incident commander is the person running the response. Blame gets handled afterwards in the blameless postmortem (a written review focused on system causes, not people).

What I would do, in order.

1. Intervene once, in public, briefly and neutrally. Post: "We are keeping this channel to verified facts and next steps until the incident is resolved. Questions about cause and ownership will be handled in the postmortem. Next update at 16:00." No lecture, no naming.
2. Take the argument private. Call the two or three leads involved (a short call, not more chat). Listen for what each is really worried about: usually being blamed, or missing information. Give each a route to be heard: a dedicated internal channel where cause discussion is allowed, and a promise that their perspective goes into the postmortem.
3. Establish who speaks. One comms lead publishes to customers; teams feed them facts through the operations channel (the working chat where responders share verified facts, with a written ops log of entries). Ask the incident commander (the person running the response) to reaffirm this so it is not just my opinion.
4. Restore the shared picture. Publish a single timeline of confirmed facts, for example: "14:05 API release began. 14:20 checkout errors rose. 14:32 release rolled back. 14:50 errors back to normal." Disputes are usually about who knew what and when. A timeline replaces recollection and shortens the fight.
5. Lock or moderate the channel if needed. Restrict posting to the response roles and a read-only audience, with a separate Q&A thread. This is a last resort but acceptable during a live outage.
6. Escalate if it continues. If a leader keeps posting blame, the incident commander or an executive sponsor (a senior leader who backs the response and can lean on other leaders) speaks to that leader's manager. Doing this privately avoids escalating in front of customers.
7. After resolution. Feed the friction into the postmortem: unclear ownership of a shared dependency is a system problem, not a personality problem.

Illustrative exchange. A platform lead writes "the API team's release broke us." The reply from the comms lead, in the public channel: "Thanks. We are still confirming the trigger and will share it in the review. For now, checkout is degraded; next update 16:00." Privately, the comms lead calls both leads and says something like: "I am not deciding whose fault this is. That is for the review. I need each of you to add your timeline entries to the ops log by 15:45 so the record is fair to both teams, and I will make sure both views go into the postmortem." Each agrees to send their timeline entries to the ops log by 15:45.

Trade-offs. Firm moderation can feel like censorship and may hide real safety information; keep a clear route for facts that matter. Moving discussions private reduces transparency, so the postmortem must record what was decided and why.

What would change my approach. If the finger-pointing is a symptom of an ambiguous ownership boundary, I would ask leadership to name a single owner for the shared component right now, mid-incident, rather than wait.