Technical Writing and Documentation Questions

Direct answer

Below is the entry for a machine-learning inference service (a service that loads a trained model and answers prediction requests) called recommender-inference, running on Kubernetes (a system that runs copies of a service, called pods, and restarts them). The newcomer test I would run first is the triage fork (steps 1 to 3: find the bad pods, read why they failed, choose one of three fixes), because that is where unwritten knowledge hides and where a wrong turn does damage.

The runbook entry

RUNBOOK: recommender-inference instance down or not serving
Owner: ml-platform-oncall   Last verified: <date>, by <name>
Use when: alert "recommender-inference error rate high" or "pods not ready"

SYMPTOMS
  - Pods show CrashLoopBackOff (the pod keeps crashing and being restarted), or Running but never Ready
  - Callers get HTTP 5xx or timeouts from /v1/predict
IMPACT
  - Recommendation widgets fall back to a static list (degraded, not down)

STEP 1. Find the bad pods.
  Run from: any machine with the prod kubeconfig (the cluster credentials file, context prod-ml)
    kubectl --context prod-ml -n ml-serving get pods -l app=recommender-inference
  Expected when healthy: all pods 1/1 Running. Note any that are not.

STEP 2. Read why it failed.
    kubectl --context prod-ml -n ml-serving describe pod <pod>
    kubectl --context prod-ml -n ml-serving logs <pod> --previous --tail=50
  Match:  Reason OOMKilled                 -> ran out of memory, go to 3A
          "failed to load model"           -> model artifact problem, go to 3B
          started right after a deploy     -> go to 3C

STEP 3A. One pod only: replace it (safe to repeat, the Deployment recreates it).
    kubectl --context prod-ml -n ml-serving delete pod <pod>
  If the replacement pod is OOMKilled again, deleting pods will not help: the memory limit or the model size has to change, so escalate to the owner.
STEP 3B. Model artifact missing or wrong version: pin the last good model.
    kubectl --context prod-ml -n ml-serving set env deployment/recommender-inference MODEL_VERSION=<last-good>
STEP 3C. Bad release: roll the deployment back.
    kubectl --context prod-ml -n ml-serving rollout undo deployment/recommender-inference

STEP 4. VERIFY (never skip).
    kubectl --context prod-ml -n ml-serving rollout status deployment/recommender-inference --timeout=180s
    curl -fsSi -X POST -H 'Content-Type: application/json' https://recommender.internal.example/v1/predict -d @sample-request.json
  Expected: rollout reports success; the first line curl prints is the status line (HTTP/1.1 200 OK or HTTP/2 200), followed by a body with a scores array.
  Then confirm the error-rate dashboard is falling for 5 minutes.

ROLLBACK OF THE FIX: re-apply the previous MODEL_VERSION value or run rollout undo again.
ESCALATE: after 30 minutes or if 3A to 3C do not help, page ml-platform-owners.

The service name, namespace, endpoint and environment variable are illustrative; every kubectl form shown is a real command.

Reading the commands in plain words

• --context prod-ml says which cluster to talk to. -n ml-serving picks a namespace, a named partition of the cluster where this team's pods live.
• -l app=recommender-inference is a label selector: it filters to only the pods carrying that tag.
• Running means the container process has started. Ready means it also passed its health check and is receiving traffic, so 1/1 means one of one containers is ready.
• OOMKilled means Kubernetes killed the container for running out of memory. logs --previous shows the output of the instance that just crashed.
• A Deployment is the object that declares how many copies of the pod to keep and recreates any that disappear, which is why deleting one pod is safe.
• set env changes an environment variable and makes the Deployment replace its pods. rollout undo reverts the Deployment to its previous release.
• 5xx means HTTP server-error codes (500 to 599).
• In the curl command, -i prints the HTTP status line and headers (without it curl shows only the body, so you could not see "200"), -f makes curl exit non-zero on a 4xx or 5xx response, -sS hides the progress meter but still shows errors, and -H sets the request's Content-Type header.

Why the entry is shaped this way

• Symptoms and impact come first so the reader confirms they are in the right page.
• Every command states where it runs, and the context flag prevents running against the wrong cluster.
• The machine-learning-specific failure (model file will not load) has its own branch, because a code rollback would not fix a bad model version.
• Verification is a separate step with a concrete pass condition, and each fix has its own undo.

Which part I test first with a newcomer, and why

I would hand the entry to an engineer who joined recently and stage a failure in a test environment (delete a pod, or set a wrong MODEL_VERSION). Then I would watch silently and write down each question they ask. I start with steps 1 to 3 (including the fixes, because I want to see whether the newcomer picks the branch the log evidence points to):

1. They decide which branch to take, so vague wording there sends someone to the wrong fix.
2. Access assumptions (is the kubeconfig set up, what is the context name) are invisible to the author.
3. If steps 1 to 2 fail, the reader never reaches the fixes in step 3, so a defect there blocks everything after it.

Step 4 is tested next, because a reader who cannot tell success from failure will stop too early or keep changing things.

Pitfalls

• Testing only the happy path. The test scenario should hit a branch other than the most common one.
• The author "helping" the newcomer, which hides the exact gaps the test exists to find.

Direct answer

A runbook is a step-by-step operating guide for one specific situation, usually opened from an alert. A tired on-call engineer (the person currently carrying the pager for the team) has almost no spare attention, so a usable runbook needs no interpretation: it starts from the alert they received, gives one action per step with the exact command, shows what a correct result looks like, and says plainly when to stop and call someone. For a data pipeline (a scheduled chain of jobs that moves and transforms data) I write each entry around a symptom such as "table not fresh by 06:00", never around the architecture.

Practices I follow for a pipeline runbook

• One entry per alert, linked from the alert itself. The page that fires should contain the URL of its entry, so nobody searches a wiki at 3 a.m.
• Symptom first, then a short decision path. Step 1 answers "is it still running, failed, or never started?" and each answer points to a numbered step. No background essays inside the steps.
• Exact commands and exact locations. Say where the command runs (a jump host, meaning a locked-down server you log into first to reach production, or a laptop with a named cluster context, the setting that says which cluster a command talks to) and paste it ready to copy, with placeholders in angle brackets.
• State the expected result after each step, so the reader knows whether to continue.
• Say whether an action is safe to repeat. Re-running a load is only calm if the job is idempotent (running it twice leaves the same data as running it once). Write "safe to repeat: yes, the load merges on order_id" or "NOT safe: creates duplicates, ask the owner first".
• Name the blast radius (how much else is affected if this breaks or your fix goes wrong): which dashboards, models or teams read this table, and who to notify.
• Escalation is a step, not an afterthought: after 30 minutes without progress, page the owning team, with the channel and rotation name (the rotation is the schedule that decides who is on call).
• Known failure table: last three real incidents as "log line seen, cause, fix", because pipelines fail in repetitive ways.

Worked example (illustrative excerpt)

ALERT: orders_daily table not updated by 06:00 UTC
Owner: data-platform-oncall   Last verified: 2025-06-02 by a.rivera
Impact: finance dashboard and churn model read this table.

1. Is the run still going? (run from the jump host)
     airflow dags list-runs -d orders_daily --state running
   Rows returned -> wait 30 min, then go to step 4.
   No rows       -> go to step 2.
2. Open the DAG's runs in the orchestrator UI.
     No run at all for today (it never started) -> check the DAG is not paused and the
                                     scheduler is healthy, then go to step 4
     A red (failed) run: open the failed task's log and match the last line:
     "Connection refused ... source-db"  -> source outage, go to step 4
     "duplicate key"                     -> STOP, do not re-run, escalate
3. Clear (re-run) the failed task load_orders. Safe to repeat: yes (MERGE on order_id).
   Expected: task turns green in about one normal run time.
4. Escalate to data-platform-owners if not green after 30 minutes.

Reading the excerpt: Airflow is a scheduler that runs a chain of jobs in order, and a DAG (directed acyclic graph) is one such chain of tasks with a fixed order and no loops. airflow dags list-runs -d orders_daily --state running lists the runs of the orders_daily DAG that are in progress right now. The orchestrator UI is Airflow's web page showing each task's status and logs. To "clear" a failed task means to tell Airflow to run it again. MERGE on order_id is a SQL statement that updates rows that already exist and inserts new ones, matched by order_id, which is why repeating it does not create duplicates.

Fields I validate before publishing (and re-validate on a schedule)

1. A named owner that is a team or rotation, not a person who may leave.
2. A last-verified date and who verified it, with a review interval (for example 90 days) after which the page is flagged stale.
3. Tested commands: someone other than the author ran every command against staging or a copy and the output matched what the step promises.
4. Links resolve, and the alert's runbook link points to this exact page.

Pitfalls

• Paragraph prose instead of numbered steps.
• "Check the logs" without saying which logs and what to look for.
• Commands that only work with context the author carries in their head.
• A runbook that has never been used under pressure. Run a drill and fix every place the reader hesitated.

Direct answer

I would keep one source of truth written for the on-call engineer (the detailed runbook, a step-by-step procedure for handling an operational task or incident) and generate the auditor's view from it, instead of maintaining a second, shorter document. Auditors rarely need less truth, they need a different entry point: which control (a required safeguard, such as "we can recover a failed service") is covered, who owns it, how often it is tested, and where the evidence is. Those are metadata and links, and they can be derived from the runbook, so nothing can drift.

Design

• Docs-as-code (documentation stored as files in version control and reviewed like code). The runbook is a Markdown file in the service's repository, so every change has an author, a review and a timestamp. That history is itself audit evidence.
• Structured metadata at the top of each runbook (front matter, a small block of fields the file starts with):

id: RB-PAY-014
title: Recover a failed payments-api instance
owner: payments-oncall
last_verified: 2025-06-02
verified_by: a.rivera
drill_link: https://wiki.example.com/drills/2025-06-02
control_ids: [IR-2, BC-4]
review_every_days: 90

• A generated audit index. A small script reads the fields and produces a table, so the auditor's page is regenerated on every merge and never hand-edited. The script runs in the same CI job as the checks below, reads every runbook file's front matter in the repository, and writes the table to the audit page. Here are the two runbooks' front matter blocks (the second one differs in id: RB-ORD-003, owner: orders-oncall, last_verified: 2025-02-10, control_ids: [IR-2]) and the whole script, run with a fixed "today" of 2025-08-15. One trap worth knowing: PyYAML's safe_load already turns 2025-06-02 into a Python date, so calling date.fromisoformat on it raises a TypeError.

import glob
from datetime import date
import yaml  # PyYAML

today = date(2025, 8, 15)  # fixed so the output is reproducible
print("| Runbook | Owner | Controls | Days since verified | Status |")
print("|---|---|---|---|---|")
for path in sorted(glob.glob("*.md"), reverse=True):
    front = open(path).read().split("---")[1]   # text between the first two --- lines
    fields = yaml.safe_load(front)
    age = (today - fields["last_verified"]).days
    status = "STALE" if age > int(fields["review_every_days"]) else "current"
    print(f"| {fields['id']} | {fields['owner']} | {', '.join(fields['control_ids'])} | {age} | {status} |")

The script reads every runbook file in the folder and prints one table row per runbook. The output:

| Runbook | Owner | Controls | Days since verified | Status |
|---|---|---|---|---|
| RB-PAY-014 | payments-oncall | IR-2, BC-4 | 74 | current |
| RB-ORD-003 | orders-oncall | IR-2 | 186 | STALE |

Age is the date difference. RB-PAY-014: June 2 to August 15 is 74 days, under its 90-day interval, so current. RB-ORD-003: February 10 to August 15 is 186 days, over 90, so STALE.

• Summary layer by link, not copy. An audit-facing page holds two to three sentences per control plus permalinks (stable links that keep pointing at the same section even if the page is renamed or moved) to the runbook section. It never restates steps.
• Evidence is produced by doing, not by writing. Drill records (dated records of a practice run, such as deliberately failing over a service to prove the steps work), CI results and incident tickets are linked from the metadata. Auditors trust a dated drill link more than a long paragraph.
• Enforcement in CI (the automated checks that run on every change): fail the build if owner or last_verified is missing, and open a ticket when an entry passes its review interval.

Worked example

Auditor asks: "Show me how you recover a failed payments instance and how you know it works." They open the index row for RB-PAY-014, see control IR-2 (an internal control ID), owner, verification date and the drill link, and click through to the exact steps. The on-call engineer never sees the audit wrapper and the auditor never has to read a 4-page procedure unless they choose to.

Trade-offs and what would change my call

• Generation adds tooling to maintain, which is worth it once there are more than a handful of runbooks.
• Metadata can become theatre (a show of compliance: dates bumped without real verification). Guard by requiring a linked drill or run record with the date, not just a field edit.
• If an audit regime (the rules an auditor applies and how often they check) demands frozen point-in-time documents (copies that show exactly what was true on one date and never change afterwards), I would tag a snapshot of the generated view at each audit period, keeping one source of truth.
• If the wiki has no version control, I would use page properties (structured fields attached to a wiki page, which play the role of front matter) plus a scheduled export (a job that copies the pages out on a timer), but I would still refuse two hand-written copies.

Pitfalls

• Writing the "short version" by hand once and promising to update it.
• Making the runbook shorter to please auditors, which harms the reader who needs it most under pressure.

Direct answer

Treat the runbook repository like code: every pull request (PR, a proposed change awaiting review) runs three automated checks in continuous integration (CI, the robot that runs on each PR). A link checker (lychee) catches dead links, a small script validates required front matter (the YAML metadata block at the top of each file, such as owner and last-reviewed date), and markdownlint-cli2 enforces Markdown style. Failures show up as inline annotations on the PR and as a red required check. For emergencies, an on-call engineer adds an emergency-fix label. That label makes only the checks that are cosmetic (lint) or depend on outside websites (external links) non-blocking, keeps the check that protects accountability (front matter) blocking, and automatically opens a follow-up issue so the debt is paid rather than forgotten.

Terms in plain words

• GitHub Actions: GitHub's built-in automation. A YAML file in .github/workflows/ tells GitHub what commands to run on each PR.
• Required status check / branch protection: branch protection is a repository setting on the main branch. Marking a check as "required" there means GitHub refuses to merge a PR until that check is green.
• Triage rights: a GitHub permission level. Only people with at least that level (or higher) on the repository can add or remove labels, so the label cannot be applied by a random outsider.
• Inline annotation: an error message drawn on the exact file and line in the PR's Files tab.
• continue-on-error: a step setting that lets the step fail without turning the whole job red.

Tools and what each one guards

The workflow (GitHub Actions syntax, parsed as valid YAML)

name: runbook-checks
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled]   # 'labeled' re-runs checks when the emergency label is added
    # No `paths:` filter on purpose: a required check that is skipped by path filtering stays "Pending" and blocks the PR
  schedule:
    - cron: "0 6 * * 1"   # weekly full link crawl catches link rot nobody's PR touched
permissions: { contents: read, issues: write, pull-requests: read }
jobs:
  docs-checks:            # the ONE job name set as a required status check
    runs-on: ubuntu-latest
    env:
      EMERGENCY: ${{ contains(github.event.pull_request.labels.*.name, 'emergency-fix') }}
    steps:
      - uses: actions/checkout@v4
      - name: Front matter (always blocking, because no owner means nobody is accountable)
        run: pip install pyyaml==6.0.3 && python scripts/check_front_matter.py $(git ls-files 'runbooks/*.md')
      - name: Markdown lint
        continue-on-error: ${{ env.EMERGENCY == 'true' }}
        run: npx --yes markdownlint-cli2@0.23.3 "runbooks/**/*.md"
      - name: Links
        continue-on-error: ${{ env.EMERGENCY == 'true' }}
        uses: lycheeverse/lychee-action@v2
        with:
          args: --no-progress --max-retries 2 --accept 200,429 "runbooks/**/*.md"
          fail: true
      - name: Open follow-up issue when the emergency path was used
        if: env.EMERGENCY == 'true'
        env: { GH_TOKEN: "${{ github.token }}" }
        run: gh issue create --title "Post-incident doc cleanup for PR ${{ github.event.pull_request.number }}" --label docs-debt --body "Lint and link checks were non-blocking on this PR. Re-run and fix within 2 business days."

Reading the workflow step by step

1. The workflow starts when a PR is opened, updated, reopened or labeled, on every PR (there is deliberately no paths: filter, see the pitfalls). A weekly schedule also runs it.
2. permissions gives the job read access to the code and write access to issues, so it can open the follow-up ticket, and nothing more.
3. EMERGENCY is true only when the PR carries the label emergency-fix (the contains(...) expression asks "is that name in the PR's label list?").
4. Front matter always runs and always blocks: if it fails, the job is red.
5. Lint and links have continue-on-error set to that flag: normally a failure turns the job red, but on an emergency PR the failure is still printed and the job stays green.
6. The last step runs only when the flag is true and files the cleanup issue.

The front-matter check, run for real

import os, sys, pathlib, tempfile
import yaml

REQUIRED = ["title", "owner", "service", "last_reviewed"]

def check(path, text):
    """Return a list of problems for one runbook."""
    text = text.replace("\r\n", "\n")   # files saved on Windows use CRLF line endings
    if not text.startswith("---\n"):
        return ["no front matter block"]
    end = text.find("\n---", 4)
    if end == -1:
        return ["front matter block is never closed"]
    try:
        meta = yaml.safe_load(text[4:end]) or {}
    except yaml.YAMLError as e:
        return [f"front matter is not valid YAML: {e.__class__.__name__}"]
    if not isinstance(meta, dict):
        return ["front matter is not a set of key: value pairs"]
    # a key that is present but empty (`owner:`) is as useless as a missing one
    return [f"missing key: {k}" for k in REQUIRED if meta.get(k) in (None, "")]

def main(paths):
    failures = 0
    for p in paths:
        for problem in check(p, pathlib.Path(p).read_text()):
            print(f"::error file={p}::{problem}")
            failures += 1
    return 1 if failures else 0

if __name__ == "__main__":
    if len(sys.argv) > 1:
        sys.exit(main(sys.argv[1:]))
    with tempfile.TemporaryDirectory() as d:
        os.chdir(d)
        good = pathlib.Path("db-failover.md")
        good.write_text("---\ntitle: DB failover\nowner: team-data\nservice: orders-db\nlast_reviewed: 2026-01-05\n---\n# Steps\n")
        bad = pathlib.Path("cache-flush.md")
        bad.write_text("---\ntitle: Cache flush\nservice: edge-cache\n---\n# Steps\n")
        empty = pathlib.Path("dns-cutover.md")
        empty.write_text("---\ntitle: DNS cutover\nowner:\nservice: dns\nlast_reviewed: 2026-02-01\n---\n# Steps\n")
        crlf = pathlib.Path("vpn-reset.md")
        crlf.write_bytes(b"---\r\ntitle: VPN reset\r\nowner: team-net\r\nservice: vpn\r\nlast_reviewed: 2026-03-01\r\n---\r\n# Steps\r\n")
        code = main([str(good), str(bad), str(empty), str(crlf)])
        print("exit code:", code)

Running it prints:

::error file=cache-flush.md::missing key: owner
::error file=cache-flush.md::missing key: last_reviewed
::error file=dns-cutover.md::missing key: owner
exit code: 1

The third file has owner: with nothing after it, which YAML reads as null. A plain k not in meta test would let it pass, defeating the rule that "no owner means nobody is accountable", so the check tests for an empty value too. The fourth file (Windows line endings) passes because the script normalises \r\n first.

The ::error file=...:: format is what makes GitHub draw the message on the exact file in the PR's Files tab, so the author does not have to open CI logs.

How failures surface

• One job name (docs-checks) is marked as a required status check in branch protection, so a red result blocks merge.
• Each tool writes file-level annotations, so the fix is visible where the author is already looking.
• Link checks accept HTTP 429 (rate limited) and retry twice, because a flaky third-party site should not fail the PR.
• The weekly scheduled run crawls everything, because links rot even when nobody edits the runbook.

Emergency path without gutting the checks

• Only people with triage rights can apply the label. GitHub itself enforces this (the workflow does not need to), so it is not a self-service bypass for anyone.
• Front matter stays blocking: it is local and takes seconds, but the real reason is that an emergency runbook with no owner or review date is the one nobody will fix later, which is worse than a lint warning.
• Lint and external links become non-blocking (continue-on-error). Lint findings are cosmetic and cannot mislead a reader, and link checks depend on third-party sites that may be down or rate-limiting during the same incident, so a red result there does not mean the runbook is wrong. Lint is not slow; the point is that a style complaint should never delay a fix.
• The bypass is not a branch-protection override or a disabled workflow. The checks still run and still report, so the author sees exactly what to clean up.
• The last step files a docs-debt issue with a two-business-day target, and a weekly report counts how many emergency labels were used. A rising count is a signal to fix the checks or the on-call process.

Worked example

At 02:10 on-call needs to change step 3 of a failover runbook. They open a PR, add emergency-fix, and a teammate approves. Front matter passes, lint fails on a trailing-space rule and is shown but non-blocking, and the merge happens in minutes. The follow-up issue lands in the team queue the same night, and the author fixes the lint error next morning.

Trade-offs and pitfalls

• Making every check non-blocking during emergencies is tempting but teaches people to always use the label. Keep the blocking set as small as the cheapest useful checks.
• Do not run link checks against private internal URLs without an allowlist or auth, or every PR fails. Exclude those patterns explicitly and verify them another way.
• Pin the action and tool versions so a tool upgrade does not turn every PR red on the same morning.
• A required check whose workflow is skipped by a paths: filter stays "Pending" forever (GitHub documents this), so a PR touching only a README would be unmergeable. Either drop the filter, as above, or add a tiny always-running job that reports the same check name.
• gh issue create --label docs-debt fails if that label does not exist, which would turn the emergency path red. Create the label once up front, and remember each new push to an emergency PR re-runs the step, so de-duplicate (for example search for an open issue with that title first).
• The workflow file itself is part of the PR, so an author could edit it to weaken the checks. Put .github/workflows/ under CODEOWNERS with "Require review from Code Owners" enabled.
• If the label is used often, the real problem is that the checks are too slow or too strict for normal work.

Direct answer. A one-page postmortem answers, in plain order: what happened and how bad, why it happened, how we responded, and what we will change. A top block written for business readers (impact, duration, cause in one sentence, status) sits above an engineering block (timeline, root cause, contributing factors, action items). It is blameless: it describes systems and decisions, not individuals.

Template

TITLE / date / severity (how bad it was, on a scale such as SEV1 total outage to SEV3 minor) / author / status (draft, reviewed, closed)
SUMMARY (business readers): impact, duration, one-sentence cause, customer action needed
IMPACT: who, how many, what they experienced, measured how (include uncertainty)
TIMELINE (UTC): detection, escalation, mitigation, resolution
ROOT CAUSE and CONTRIBUTING FACTORS: technical cause, why it was not caught earlier
WHAT WENT WELL / WHAT WENT POORLY / WHERE WE GOT LUCKY
ACTION ITEMS: owner, due date, priority, how we will verify it worked

Example: an outage that affected ten percent of users for three hours (illustrative details)

Summary. On 14 March, 10% of users could not complete checkout for 3 hours (10:05 to 13:05 UTC). Cause: a configuration change reduced the connection limit (the cap on simultaneous connections the application may open to the database, also called pool size) on one of the ten database shards (the database is split into ten slices, each holding a share of users). Fixed by reverting the change. No data was lost. Affected customers were offered a retry link.

Impact. Users are assigned to shards by ID, so one shard of ten means about 10% of users. If we had 400,000 daily active users, that is roughly 40,000 users with failed checkouts; the estimate rests on the assumption that shard membership is even, and we checked actual failed-request counts in logs to confirm the range.

Timeline (UTC).

• 09:58 config change deployed to shard 7.
• 10:05 error rate on shard 7 rises; no alert fires (the alert threshold, the error level that triggers a page, covers the global rate only).
• 10:40 support tickets spike; the on-call engineer (whoever is currently responsible for responding to problems) is paged by support, not by monitoring.
• 11:20 root cause identified; 12:50 revert approved; 13:05 recovery confirmed.

Root cause and contributing factors. The root cause is the single change without which the outage would not have happened; contributing factors are conditions that let it last or go unnoticed. The change set the pool size too low for the traffic on that shard: the new limit was below what the shard needs even off-peak, and the old value had been sized for peak. Contributing: the global error-rate alert averages ten shards, so a 10% failure looks like noise; no canary (staged rollout to a small share first); the change review did not include capacity numbers.

What went well/poorly/luck. Well: once the revert was approved at 12:50, recovery was confirmed 15 minutes later. Poorly, three delays: 35 minutes from first errors (10:05) to the first page (10:40); 40 minutes from the page to an identified root cause (11:20), which is 75 minutes after the errors began; and 90 minutes from root cause to an approved revert (11:20 to 12:50). The approval delay is the biggest single lesson, since it was the longest wait and the fix was already known. Luck: it happened off-peak; at peak the same 10% of users would have seen failures on a larger share of their requests, and the pool would have been exhausted sooner.

Action items.

Variants (same skeleton, different emphasis). Data-quality incident: replace "users affected" with which tables, which dashboards, and which decisions used bad numbers, and add a backfill plan. ML outage or recurring degradation: the impact includes a metric with a range ("precision fell 3 to 5 points, estimated from a labeled sample of 400 predictions"), the timeline includes feature snapshots and drift checks before and after, and the root-cause section separates data drift (inputs changed) from a code or pipeline change. The five-section short form is: what happened, impact, why, what we did, what changes, each opened with one sample sentence like "For 3 hours, 1 in 10 users could not check out."