Incident Communication and Stakeholder Management Questions

Direct answer

Say it in four short parts (a few plain sentences each): what changed, who is affected, what we are doing, and what we can and cannot promise. Use one everyday analogy, avoid model jargon, and be clear that fixing the cause is quick but confirming quality has recovered is not. Promise the next update time and the next concrete step, not the date the quality will be back.

What I would say (illustrative, spoken)

1. What happened. "Earlier today a change to the data feeding our recommendation model altered how one field is filled in. The model learns from that data, so it started making worse suggestions, like a chef given mislabelled ingredients."
2. Who is affected. "New users, about one in five sessions, get the worst results. Returning users see a smaller effect. We know this from comparing today's results against last week on a test set (offline evaluation)."
3. What we are doing. "First we revert the data change so bad data stops flowing. Then we rebuild the affected data and retrain the model (teach it again on the corrected data). Then we test it before it goes live."
4. What I can and cannot promise. "I can promise the revert is done within about an hour, and an update at 3 p.m. I cannot yet promise when quality is back, because retraining time depends on how much data we must rebuild. After the first rebuild run I will give you a firm estimate."

Why it is worded this way

• Analogy over mechanism. A product manager can act on "worse suggestions for new users", not "feature distribution shift".
• Scope in user terms. Say who and how many, not metrics names.
• Two clocks. Stopping the damage is controlled by us; recovery depends on retraining and validation, so it has uncertainty. Separating them makes the promise honest.
• No fabricated numbers. Give the ranges you know; label estimates as estimates.

What the PM may need to decide

• Whether to pause the affected feature or show a simple fallback (for example, popular items) meanwhile.
• Whether support or customers should be told.
• Whether to delay a launch that depends on the model.

Trade-offs and pitfalls

• Do not say "fixed" when only the pipeline is reverted; the model is still trained on bad data until retrained.
• Do not promise a timeline before you know how long the rebuild takes.
• Do not bury the impact in jargon like drift, embeddings or loss.
• Avoid blame ("the data team changed it"); say what changed and how you will prevent it, such as a data check in the pipeline.

Direct answer

Treat the ten minutes as a decision meeting, not a technical review. Open with the headline and the ask, tell one simple causal story with a single picture, and protect the last four minutes for discussion and the decision itself: the ask is stated on slide 6 inside the six talking minutes, and the four minutes are where leadership questions it and commits. Cut the tooling detail, the minute-by-minute log, and anything that invites a debate about how the fix was done.

Slide structure (six slides, six minutes talking: 1+1+1+1+1+1, leaving four for questions and the decision)

What you ask of leadership (be concrete)

Pick the two decisions that matter most for this incident (the headline promises two); the list below is the menu to choose from, not a list to ask for all at once.

• A resourcing or priority decision: for example, pausing feature work for two weeks to build the missing safeguard, or funding a second vendor.
• A risk decision: accept, fund, or defer a remaining risk, with the cost of each stated.
• An owner for cross-team actions that no single team controls.
• For customer-facing consequences, approval of the customer message and who speaks to top accounts.

What to cut

• Tool names, log excerpts and architecture detail beyond the one diagram.
• The full timeline (keep it in an appendix), personal names, and the debate over alternatives.
• Anything defensive. Leaders trust a plain "we missed this" more than a long explanation.

Data-incident variant

If the incident involved data (loss, corruption, exposure), replace slide 4's picture with a data-flow visual: where the data lives, which stores were touched, the time window affected, and how many records or customers. Add a small "recoverable or not" column for each store. The asks then include: approve customer and regulator notification with legal, fund the verification and restoration work, and decide how long the affected customers are offered monitoring or remediation.

Worked example (illustrative)

A release removed a database index, and search slowed until the site timed out. Slide 3 says: "Tuesday 14:02 a release went out; 14:19 we noticed; 15:07 we rolled back; 15:40 caches recovered." Slide 4 shows one chain: release, slow queries, timeouts, no alarm on query time. The ask on slide 6: "Approve two weeks of one team's time to add automated performance checks before release, and name a single owner for release safety."

Trade-offs and pitfalls

• Ten minutes runs short: rehearse the talk to the six minutes in the table so questions do not push the ask off the agenda.
• Defending instead of deciding: if the CTO starts probing the fix, offer to take detail offline and return to the ask.
• Asking for "support" with no specifics gets sympathy and no action.

Direct answer

Tell the CTO (chief technology officer, the executive accountable for the systems) first, and quickly, with a short holding message that separates what is confirmed from what is suspected. Then, in order: the people who can fix or measure the problem, the internal teams and customers who rely on the system, and last, the vendor (the outside company supplying the data or service) with hard evidence rather than an accusation. The CTO comes first because they must never be surprised and they authorise everything else; the people who can fix it are contacted right after, within minutes. Commit to a fixed update rhythm, a written summary, and a postmortem (a blameless written review), so the CTO is not left guessing between updates.

Order of communication and what each party gets

1. CTO (within the first 30 minutes). A holding message with three labelled parts: confirmed, suspected, unknown, plus the time of the next update. Never a guessed root cause.
2. Core responders. One owner and one channel. Freeze other changes (pause unrelated releases so they cannot confuse the diagnosis), preserve evidence (before and after data, version identifiers, timestamps), and agree how the drop will be measured.
3. Internal dependants (product, support, analytics, customer teams). Tell them the symptom, who is affected, and a workaround if one exists, so they stop reporting different numbers.
4. Customers, only if the drop touches them, and only via whoever owns customer communication and legal. Describe impact and next update time, not the vendor's name.
5. Vendor. Open a ticket at high severity (the vendor's urgent priority tier for support requests) with evidence: the exact change window on their side (the time span in which they changed something; ask for their change log), the metric comparison, and a specific request such as pinning the previous version (freezing on the last known-good version instead of taking automatic updates) or rolling back.

Handling the vendor as a party

The vendor is a hypothesis until proven. Do not post blame externally. Ask for their status, keep contract and account contacts informed, and log downtime for any later credit claim.

Follow-ups to commit to

• Updates every 60 minutes until the cause is confirmed, then at each state change.
• A written summary within 24 hours of stabilisation.
• A postmortem within about five business days, with owned actions.
• A vendor review: notification terms for upstream changes, and a way to pin versions.

Worked example (illustrative)

A recommendation model's click-through quality (the share of shown recommendations that people click) falls sharply at 09:00. The model depends on an upstream data provider (a source that feeds data into our system) and a parser (the code that reads that data into our format). The holding message at 09:25:

"Confirmed: quality on the recommendation service dropped at about 09:00 and is still low. Suspected: an upstream data provider changed a field format at 08:45 (unconfirmed). Unknown: full impact and time to fix. We are comparing inputs before and after the change window and will update at 10:25."

The responders diff sample inputs (compare raw examples of the incoming data side by side) from 08:30 and 09:30 and find a field now arrives in a different unit. The team pins the vendor's previous feed version if the provider allows it (switching back to the last known-good version and holding it there), or ships a parser patch that converts the new unit to the old one. Reverting our own parser would not help, because the change is in the incoming data. Quality recovers. Only then does the vendor ticket name the specific change and ask for advance notice in future.

Trade-offs and pitfalls

• Speed versus accuracy: an early message is worthwhile only if it labels its own uncertainty.
• Blaming the vendor too soon damages the relationship if the fault turns out to be yours (a stale assumption, a missing validation).
• Different audiences hearing different numbers erodes trust faster than the drop itself.

Direct answer

Treat it as a priority incident (one worked immediately and tracked formally, because it has a hard deadline). Escalate immediately to the report's on-call owner and my own manager, open an incident so work is tracked, and get a contingency in hand well before the meeting. Then send the CEO and CFO a short, calm email that does four things: acknowledges the failure, states the mitigation and timing, offers contingency material and commits to a follow-up. Executives need to plan around the meeting, not learn about the cause.

Escalation (in order)

1. Now: page (send an urgent alert to) the data on-call (the engineer on duty to be reached at any hour) and the report owner; the failed job's owner may already know the cause.
2. Within minutes: tell my manager and the CFO's office lead, so the executives are not surprised by a colleague.
3. Open an incident record (a ticket that tracks the problem, owner and timeline in one place) with a named lead and a deadline (report needed by the board start, for example 10:00).
4. Set two tracks: fix the pipeline and, in parallel, prepare a manual or last-known-good report (the most recent version that was generated and checked, for example yesterday's). A fix that misses the deadline is useless without the fallback.
5. Checkpoint times: for example 08:30 decide whether the fix will land, 09:00 lock the final report (the rerun or the fallback). Section below shows where these times come from.

Email to the CEO and CFO (draft)

Subject: Board report delayed: mitigation in progress, update by 08:30

[Names],
The scheduled board report did not generate this morning because of a data
pipeline failure. We are working on it and have a fallback ready.

Mitigation: the data team is rerunning it; I will confirm by 08:30 whether it
will be ready.
Contingency: if not, I will send yesterday's approved report, clearly dated
as of yesterday, plus a one-page table of the five headline figures refreshed
manually and checked by finance.
Next update: 08:30, and a written cause summary by end of day tomorrow.

[Name]

Why the email is built this way

A data pipeline, mentioned in the email, is the automated chain of jobs that collects and prepares the report's data.

• It acknowledges without over-apologising or explaining the pipeline.
• It commits to a time, so the executives can plan.
• The contingency is labelled by its date, so nobody presents stale data as fresh.
• It says who is responsible and when the next word arrives.

Worked example on timing (illustrative)

Job fails at 06:00, discovered at 06:20, board starts at 10:00. Work backwards from the 10:00 start. Executives need 60 minutes to read, so the final report is locked by 09:00. Verifying a rerun and swapping it in takes about 30 minutes, so the decision point is 08:30 (09:00 minus 30 minutes): if the rerun is not confirmed good by then, the fallback becomes the report. Building the fallback takes about an hour, so it starts in parallel as soon as the failure is found (about 06:20) and is comfortably ready by 09:00. If the rerun succeeds at 08:10, you still send a note so they know the numbers are final and current.

Trade-offs and pitfalls

• Do not send the CEO a wall of technical explanation, or promise a fix time you cannot control. Promise a checkpoint time.
• A manual number produced quickly can be wrong; have a second person verify, and label it as manual.
• Do not wait to inform executives until you have a full fix; silence is what damages trust.
• Afterwards, add monitoring so a failed report alerts before humans need it.

Direct answer

Credibility comes from being early, specific and honest about what is wrong, not from hiding it until it is fixed. When I notice the broken pipeline overnight I post a short message to the data team and the downstream consumers' channel, escalate to leadership only if a morning deadline is at risk, and send a proactive summary before people start their day. Then I follow up at a predictable rhythm until it is fixed and confirmed correct.

First message (as soon as I know)

03:10 | Pipeline: daily orders load | FAILED
Impact: tables orders_daily and revenue_summary are stale (last good: yesterday 23:00). Dashboards built on them show yesterday's data.
Cause: not yet known. Investigating.
Safe to use: anything dated before yesterday.
Next update: 05:30, or earlier if fixed.

Late data is annoying; wrong data that people trust is worse. So the message says explicitly what is stale, what is still safe, and what to avoid.

When to escalate

• To my manager or the on-call lead: no clear cause after about 60 to 90 minutes, or I need access or help I lack.
• To leadership and consumers directly: if the fix will land after a hard deadline (for example the executive dashboard is due at 07:00), or if incorrect data was already published.
• Not to wake executives for a stale internal report with no deadline pressure.

Follow-up rhythm

1. Overnight: updates every 2 hours or when something changes (people are asleep, so fewer interruptions).
2. Before people arrive (06:45): a morning summary, written so someone reading it cold understands it.
3. During business hours: hourly until fixed, then a closing note with the verification (row counts match, a backfill (re-run for missed days) is complete).
4. After: a short written review with the cause and the prevention.

Worked example (illustrative timeline)

• 03:10 alert fires, first message posted.
• 05:30 update: "Cause found: upstream schema change. Fix testing. ETA to refresh: not yet confirmed."
• 06:45 morning summary in the leadership channel: "Dashboards show yesterday's data until about 09:00. No wrong numbers were published."
• 08:40 fixed, backfill checked, closing note sent.

Trade-offs and pitfalls

• Silence until it is fixed feels safe but makes people discover the problem themselves.
• Over-promising ("fixed by 7") then missing costs more trust than a wide, honest estimate.
• Remember that downstream teams need instructions, not just status: which tables to avoid.