Incident Communication and Stakeholder Management Questions

Direct answer

Do not give an estimated time to fix (ETA) you cannot support. Give what executives actually need in order to plan: what you know, what you do not, what is being done, when they will hear from you next, and the moment you expect to have a defensible estimate. Commit to the next update, not to the fix time. Then offer a planning assumption so they are not left empty-handed.

Message I would send (illustrative)

Update 14:40 UTC. Checkout is degraded for what looks like about half of customers (a rough estimate from support reports, not confirmed). We do not yet know the cause, so we cannot honestly give a fix time; a guess now would likely be wrong and you would plan around it.

What we are doing: two engineers are checking recent changes and one is adding logging to see what fails, because our current visibility is limited.

What I can commit to: another update at 15:10 UTC, and I will give you a first time estimate as soon as we have a confirmed cause. Expect that to be by 15:30 UTC or I will tell you it is not ready.

For planning today, please assume the problem may last through the afternoon and prepare customer messaging on that basis. If that assumption becomes too pessimistic I will say so early.

Why this works

• It says "unknown" once, plainly, with the reason. Executives trust "we do not know yet, here is when we will" more than a number that later slips.
• The date it gives is the date of the next information, which the responder controls, unlike the fix time, which they do not.
• The planning assumption gives leaders something to act on (customer messaging, staffing) and shifts them from demanding a number to making decisions.
• What is being done shows motion without listing every action.

Handling the pressure around it

• Give executives one channel to follow so they stop asking responders directly. The person writing updates absorbs the questions.
• If someone demands a number anyway, offer a range with the condition attached: "if it is the recent change, tens of minutes after we confirm; otherwise unknown." Say what would change the range.
• Never trade certainty for calm: a wrong estimate costs far more than a slow one.

Trade-offs and pitfalls

• "Working on it" with no next update time is the weakest reply; it creates the follow-up messages you are trying to avoid.
• Giving a best-case time to look decisive turns into a broken promise.
• Being unreachable feels safe to the writer but reads as loss of control to leadership.

Direct answer. Keep one timestamped, append-only log (you only add new lines at the bottom and never edit or delete old ones, so the record of what people knew at each moment stays intact) in a shared place (an incident channel or doc) that records what was observed, what was decided, what was done, and who did it, as it happens. The stakeholder updates (messages to customers via the status page, the public web page for live service status, and to internal leaders) are then written from that log, not from memory, so each update is a summary of verified entries.

What to record. Use one line per entry, with a time (write the time zone, ideally UTC, the universal reference time, so responders in different regions agree).

1. Observations: symptoms, graphs, error messages, customer reports, with links.
2. Hypotheses: what people think is wrong, marked as unconfirmed.
3. Actions: what was changed (rollback, config flip, restart), by whom, at what time.
4. Decisions: who decided and why, especially the ones that trade risk for speed.
5. Impact facts: which customers or regions, since when, and how you know.
6. Comms sent: what was told to whom and when, so the next update does not contradict it.
7. Open questions and owners.

Worked example.

14:02 UTC  [obs]     Checkout error rate rising on the payments dashboard (link)
14:06 UTC  [hyp]     Possible bad deploy from 13:55 (UNCONFIRMED, Priya)
14:09 UTC  [action]  Rollback of 13:55 deploy started (Priya)
14:12 UTC  [action]  Rollback of 13:55 deploy completed (Priya)
14:15 UTC  [obs]     Error rate falling, not yet at baseline
14:16 UTC  [comms]   Status page: "Investigating failed checkouts. Next update 14:45" (Sam)

How notes feed updates. The comms lead (the person who writes and sends the outward messages) takes only entries tagged as observed facts or completed actions, drops hypotheses, and writes: what customers see, what we know, what we are doing, when you will hear next. From the log above, the 14:45 update can say "we rolled back a recent change and error rates are falling", but must not say "caused by a bad deploy" until someone marks it confirmed.

Pitfalls. Logging conclusions as facts; one person both debugging and scribing (assign a dedicated scribe, a person whose only job is keeping the log, if you can); no timestamps, which makes the postmortem (the written review after the incident) timeline unrecoverable; pasting secrets or customer data into a log that will later be shared widely.

Direct answer

A good status update opens with what customers experience and its current stage, then scope, workaround, what we are doing, and when the next update comes. It stays honest by stating only verified facts with timestamps, and it avoids legal and public relations (PR) risk by describing impact and actions rather than fault, cause or promises. Each part earns its place because it answers a question a reader would otherwise ask support.

Order and justification

The four stages mean: Investigating (something is wrong, cause unknown), Identified (cause found, fix underway), Monitoring (fix applied, watching to confirm it holds), Resolved (confirmed stable).

Keeping it honest without legal or PR risk

• Facts, not causes: "Login failures for EU customers" is safe; "a bug in our vendor's code" may be wrong and may assign blame.
• No promises you cannot keep: avoid fix times and words like "guarantee". Service credits (compensation defined in an SLA, the service-level agreement with the customer, for example a partial refund on that month's bill) are handled through the support process, not announced in the post.
• Pre-agree wording with legal and PR once, so that during an incident the communications lead picks from approved phrases instead of asking for sign-off each time.
• Escalate to legal only for suspected data exposure, regulated customers or contractual notice duties.
• Never delete or rewrite history: correct mistakes with a visible new update.

Worked example (illustrative sequence)

Investigating | 11:05 UTC
Some customers in the EU cannot log in since 10:40 UTC. Customers already signed in are unaffected. Workaround: none yet. Next update by 11:35 UTC.

Identified | 11:35 UTC
We have found the affected component and are applying a fix. Login remains impacted for some EU customers. Next update by 12:15 UTC.

Monitoring | 12:10 UTC
A fix has been applied and login is working again for EU customers as of 12:10 UTC. We are watching to confirm it stays stable. Next update by 12:40 UTC.

Resolved | 12:40 UTC
Login has been stable for all customers since 12:10 UTC. Affected window: 10:40 to 12:10 UTC (1 hour 30 minutes). A written review (a postmortem, a blameless account of what happened and what will change) will follow.

The duration in the resolved post is derived from the timestamps shown. The post is factual, has no blame, and has a next-update time until the end.

Trade-offs and pitfalls

• Too much technical detail confuses customers and creates risk.
• Vague apologies without substance read as PR. Say what happened and what you did.
• Marking Resolved too early forces an embarrassing reopen.

Direct answer

Because the outage has plateaued (stable, not getting worse), I would recommend the slower staged hotfix (a hotfix is a small urgent code fix shipped forward; staged means released to a slice of traffic at a time, here 10%, then 50%, then 100%), and say so plainly: downtime is recoverable, permanent data loss is not. I would present it as a decision with named risks and a checkpoint (a pre-agreed time or health signal at which we stop and re-decide), not as a certainty. I would switch to rollback if impact starts to grow, if the hotfix slips past a time limit the business cannot tolerate, or if the data at risk turns out to be small or recoverable.

How I would reason about it

1. Reversibility first. Lost data cannot be undone; an extra hour of degraded service can be apologised for and credited.
2. Quantify both sides (illustrative numbers, stated as assumptions).
3. Look for a third option. Can we export or snapshot the at-risk writes first, so a rollback becomes lossless? That can turn the trade-off into a plan.
4. Set a checkpoint. "If the hotfix is not at 50% of traffic by 15:30, we reconsider."

Worked example (illustrative assumptions)

• A migration is a change to the structure of the database. Rolling back returns the database to how it was before that change, so every order written since the deploy is discarded. Assumption: 300 orders per minute, and 40 minutes have passed since the 13:40 deploy, so the orders written since then number:

$300 \times 40 = 12{,}000 \text{ orders lost permanently}$

• Staged hotfix takes 90 minutes. During that time 20% of orders fail, but customers can retry.

$300 \times 0.2 = 60 \text{ failed attempts per minute}, \quad 60 \times 90 = 5{,}400 \text{ failed attempts}$

The two numbers count different kinds of harm, so I do not subtract them. A failed attempt is a customer who can retry and probably will. A lost order is one the customer thought was complete and now has no record of. The safe comparison is a worst-case bound: even if every one of the 5,400 failed attempts became a permanently lost sale, that is still fewer than 12,000. Realistically many retry, so the true permanent loss is smaller still. So hotfix wins unless the business says it cannot survive 90 more minutes.

Explaining it to each audience

What I would not say: "This is safe", "should be fixed soon", or "we have no choice". State confidence honestly: "I am fairly sure about the cause, less sure about the timing."

Trade-offs and pitfalls

• If lost data can be rebuilt from logs or a payment provider, the calculation changes; verify that before deciding.
• Do not let a stressed executive pick the option in the abstract. Present the recommendation plus the trigger that overturns it.
• Record the decision and reasoning in the incident document for the review.

Direct answer

Start from the facts once, then translate them per audience by asking "what does this person need to do or say?" Sales needs what to tell prospects, support needs what customers will see and what to say, executives (senior leaders such as the chief operating officer, COO) need business impact and decisions, and none of them need the mechanism. Keep every version consistent with the same confirmed facts and the same next-update time.

The initial technical update (the source)

Checkout API returning 500 errors since 13:52 UTC because the payments service connection pool is exhausted after deploy 4127. About 40% of requests fail. Rolling back deploy 4127. Next update 14:30 UTC.

Plain reading of the source, so you can check the translations below: the connection pool is the limited set of open lines the payments service keeps to its database, and "exhausted" means all are in use so new payment requests are turned away; a 500 error is the generic "the server failed" response; deploy 4127 is release number 4127 of new code, and rolling back means reverting to the previous release.

Versions by audience (illustrative)

Support team (needs symptoms and a script):

Some customers cannot complete a purchase; they see an error at the payment step. Do not promise refunds or times. Tell them: "We are aware and working on it. Please try again later; you can follow status.example.com." Next update 14:30 UTC. Log affected tickets with the tag CHECKOUT-OUTAGE (a label added to each ticket so the team can count and find them all together).

Sales (needs what to tell prospects and what to avoid):

Our checkout is partly down: some customers cannot pay. [Add "customers can still browse" only if engineering confirms it; the technical update above says nothing about browsing.] If a prospect asks: we are aware and working on it, and will send an update by 14:30 UTC. Please do not speculate on cause or timing, and hold any demos that require live checkout.

Executives (concise):

Checkout is failing for about 40% of purchase attempts since 13:52 UTC. Cause is a recent software change; we are undoing it. Revenue impact is ongoing but we do not yet have a figure. Next update 14:30. No decision needed from you now.

Account manager for a key customer:

We had a problem at checkout affecting some payments from 13:52. It is being fixed now. I will confirm to you personally once recovery is verified and send a written summary later. Your data is not affected [only after this is verified].

Engineering versus the COO (the same fact, two ways)

• Engineer: "Connection pool exhausted after deploy 4127; rolling back."
• Chief operating officer (COO): "A change we released this afternoon overloaded our payment system. We are reverting it. Some customer orders are failing to go through until it is done (about four in ten attempts, estimated); a failed attempt is not necessarily a lost order, because some customers retry, but revenue impact is not yet measured. Do not say orders are being lost, and do not say they are safe, until it is."

Technique

1. Replace mechanism with symptom: not "connection pool exhausted", but "some payments fail".
2. Replace percentages with words only if you have no confirmed number; use the number for executives and support triage.
3. Say the customer or business effect first, the action second.
4. Give each group one instruction: support gets a script, sales gets a boundary, executives get "no decision needed" or a decision.
5. Keep terms constant across groups (do not call it "an outage" to sales and "a degradation" to executives).

Trade-offs and pitfalls

• Over-simplifying turns into inaccuracy: "the system is fine" is worse than saying nothing.
• Adding reassurance you cannot verify ("no customer data affected") can become a public falsehood; add it only after verification.
• Sending the technical update to everyone invites questions the responders then have to field.