Technical Writing and Documentation Questions

Direct answer

I would design the document for the hardest reader first, since the fixes overlap: plain sentences, the reason behind each step, the expected result of each step, and information carried by text and structure rather than by colour, screenshots or idiom. Below is one concrete change for each group, all applied to the same runbook (a step-by-step guide for handling a known operational situation).

The starting text

Bounce the box if the queue is stuck (see red graph).

It uses slang, assumes knowledge, and relies on colour.

1. Non-native speakers: plain, literal, consistent language

• Avoid idioms and phrasal verbs ("bounce", "spin up", "kick off"), use one verb per action and one word per thing, and keep sentences short with no ambiguous "it".
• Use the exact label shown in the interface, in bold, so the reader can match it without translating.
• Example: replace "bounce the box" with "restart the worker service". The word "restart" is in dictionaries and translation tools and matches the command.

2. Junior engineers: context, the why, and the expected result

• Say where to run each command (which environment), why, what success looks like, and what to do if it does not.
• Example step: "Restart the worker to clear a stuck queue. This is safe: in-flight jobs are retried. Run kubectl rollout restart deployment/queue-worker -n prod. Expected output: deployment.apps/queue-worker restarted. If you see NotFound, check you are in the production cluster context and ask in the on-call channel."

What a junior needs to know to read that step: kubectl is the command-line tool for Kubernetes (a system that runs applications across many machines). A deployment is the named group of running copies of one application. -n prod picks the namespace (a named partition of the cluster) called prod. The cluster context is the setting that decides which cluster kubectl talks to, so checking it stops you restarting the wrong one. In-flight jobs are jobs being processed at that moment, and the on-call channel is the team chat room watched by the engineer currently responsible for incidents.

3. Accessibility needs: structure, and no information by colour or image alone

• Use real heading levels, descriptive link text (not "click here"), table header rows, and alt text (a short text description of an image) for every diagram. A screen reader (software that reads the page aloud) can only read what exists as text.
• Example: "the graph turns red" becomes "The Queue Depth chart is in alert when the line is above 10,000 messages for 10 minutes; its label then reads ALERT." The threshold is illustrative. It works for a person who cannot distinguish red, or uses a screen reader.

The revised runbook step

1. Open the Queue Depth dashboard.
2. If the line is above 10,000 messages for 10 minutes (label reads ALERT), restart the worker service: kubectl rollout restart deployment/queue-worker -n prod.
3. Expected output: deployment.apps/queue-worker restarted. Check that the depth falls within 5 minutes.
4. If not, escalate to the on-call lead (the person responsible for the incident decision that week).

How I would keep it that way

• A prose linter such as Vale (an open-source tool that checks writing against rules you define) can flag idioms and undefined acronyms in CI (continuous integration, the automatic checks that run on every proposed change).
• Have a junior colleague run the runbook in a test environment. Their questions are the bug list.

Trade-offs and pitfalls

• Over-explaining can insult experts. Keep the why short and let experienced readers skim, using a bold action line first and the context underneath.
• Machine translation can mangle code and labels, so wrap commands in code blocks that are never translated.
• Avoid a separate "accessible version" that will drift. Fix the one document.

Direct answer. All three are documents, but they answer different questions. A README answers "what is this and how do I start?", a runbook answers "what exact steps do I follow for this task or alert?", and a playbook answers "how do we handle this kind of situation, and who decides what?". Naming conventions differ between companies, so I state my definitions when I write one. In everyday terms: a runbook is an operating instruction sheet, like a recipe you follow exactly; a playbook is a team's game plan, which tells people their roles and when to make a call; a README is the front-door sign of a project.

Worked example: a payments service.

• README: "Payments API. Requires Python 3.12. Run make dev. Tests: make test. Docs: link."
• Runbook: "Alert: queue depth above threshold. 1. Check the consumer dashboard. 2. Run kubectl rollout restart deployment/payments-consumer. 3. Confirm the queue drains within 10 minutes. 4. If not, roll back to the previous release (command)." Each step is unambiguous and checkable.
• Playbook: "Suspected payment data leak. Incident commander declares severity. Security lead assesses scope. Legal decides on notification. Communications drafts customer message. Decision point: take the service offline? criteria listed." It links to the runbooks for isolating a host.

Choosing between them. If the reader must think, it is a playbook; if the reader must follow, it is a runbook; if the reader is arriving cold, it is a README. A typical mistake is writing a runbook full of judgement calls (it fails at 3am) or a playbook full of shell commands (it goes stale quickly).

Direct answer

When clusters are disposable, a hand-written runbook is wrong almost immediately, so stop treating the doc as prose and treat it as a build artifact. Generate the environment-specific parts from the infrastructure code (IaC, meaning tools like Terraform or Helm, which define infrastructure in files), keep the doc in the same repository and version as that code, show real captured output rather than invented examples, and run the procedure against a fresh cluster on a schedule. If the run fails, the doc is stale, and you find out from a red build in CI (continuous integration, the automation that runs checks on every change) instead of a bad incident.

Terms in plain words

• Kubernetes cluster: a group of machines that runs containerized applications. A namespace is a named partition inside it (here, everything belonging to payments). A pod is one running copy of an application, and it is Ready when it has started and passes its health check.
• kubectl: the command-line tool for talking to a cluster. A context is a saved "which cluster am I talking to" setting, so use-context switches clusters.
• Velero: a Kubernetes backup and restore tool. velero backup get lists backups and velero restore create --from-backup X recreates the saved resources.
• Terraform: a tool that builds infrastructure from code, and its outputs are named values (cluster name, bucket) it prints after building. A module is a reusable bundle of that code. Helm packages Kubernetes applications the same way.
• Tribal knowledge: things everyone on a team "just knows" but nobody wrote down.

1. Templates: separate the stable steps from the values that change

The sequence of steps (check backup, restore namespace, verify pods) changes rarely. Cluster names, regions and bucket names change with every cluster. Write the steps once as a template with placeholders, and fill the values in from IaC outputs at publish time. A missing value fails the build.

import json, re
from string import Template

# Shape of `terraform output -json`: each output is {"value": ..., "type": ...}
tf_outputs = json.loads("""{
  "cluster_name": {"value": "prod-eu-3", "type": "string"},
  "region": {"value": "eu-west-1", "type": "string"},
  "backup_bucket": {"value": "s3://acme-velero-eu-3", "type": "string"}
}""")
values = {k: v["value"] for k, v in tf_outputs.items()}

runbook = Template("""## Restore namespace `payments` on $cluster_name ($region)
1. `kubectl config use-context $cluster_name`
2. `velero backup get` (pick the newest with status Completed)
3. `velero restore create --from-backup <backup-name from step 2>`
Backups live in $backup_bucket. Owner: $owning_team
""")

def render(t, vals):
    try:
        return t.substitute(vals)
    except KeyError as e:
        # CI fails here: the doc references a value the infrastructure code no longer outputs
        return f"RENDER FAILED: template needs {e} but Terraform does not output it"

print(render(runbook, values))
print(render(runbook, {**values, "owning_team": "platform-sre"}))

It prints:

RENDER FAILED: template needs 'owning_team' but Terraform does not output it
## Restore namespace `payments` on prod-eu-3 (eu-west-1)
1. `kubectl config use-context prod-eu-3`
2. `velero backup get` (pick the newest with status Completed)
3. `velero restore create --from-backup <backup-name from step 2>`
Backups live in s3://acme-velero-eu-3. Owner: platform-sre

The first line shows the point: the template needed an owning_team that Terraform did not provide, so the build fails loudly instead of publishing a runbook with a blank owner.

2. Capturing real cluster state as examples

Invented sample output goes stale and can hide format changes. Instead, a CI job creates a throwaway cluster (for example with kind, a tool that runs Kubernetes inside containers), runs the read-only commands the runbook references (kubectl get nodes, kubectl get pods -n payments), and writes the output to files that the docs include. Each capture is stamped with the commit and date. Scrub secrets and account IDs in the capture step, not by hand.

3. Anchoring docs to infrastructure code

• The runbook lives beside the module that builds the cluster and is released with the same version tag, so "the runbook for module v1.14" is a real thing.
• Runbooks reference IaC names (module outputs, variable names), and a CI check scans the template for every placeholder and confirms each one still exists in the outputs. A minimal version, run for real after the bucket output was renamed:

import re
tf_outputs = {"cluster_name", "region", "velero_bucket", "owning_team"}   # after the rename
runbook = "Restore on $cluster_name ($region). Backups live in $backup_bucket. Owner: $owning_team"
refs = set(re.findall(r"\$(\w+)", runbook))
for name in sorted(refs - tf_outputs):
    print(f"runbook references '{name}' but Terraform no longer outputs it")
print("checked", len(refs), "references")

runbook references 'backup_bucket' but Terraform no longer outputs it
checked 4 references

• A pull request that changes a module output must touch or acknowledge the runbooks that use it (CODEOWNERS, a file that assigns required reviewers by path, makes the doc owner a reviewer). One line does it: runbooks/restore-namespace.md @acme/platform-sre makes that team an automatically requested reviewer on any PR touching the file. It becomes a hard requirement only when branch protection has "Require review from Code Owners" enabled. Because a module-output change would not touch the runbook file, also add a CODEOWNERS line for the module path that requests the runbook owners, and the PR template asks "did you update or confirm the affected runbooks?", which is the acknowledge step.

4. Testing the steps

• Executable steps: each numbered step maps to a script or make target, and a scheduled pipeline builds a new cluster, seeds sample data, runs the restore steps, and asserts the result (for example that the expected pods are Ready).
• Game days (planned drills where a human follows the doc cold) catch what automation cannot: unclear wording, missing permissions, steps that assume tribal knowledge.
• Record time to complete and where the tester hesitated, then fix the doc, not the tester.

Worked example

The nightly pipeline stands up a clean cluster, restores a sample namespace from the newest backup, and asserts three pods reach Ready. One night a Terraform change renames backup_bucket to velero_bucket. The reference check fails the PR that made the rename, because the runbook still uses the old name. The author updates the template in the same PR. Without the check, the mismatch would have surfaced during a real restore.

Trade-offs and pitfalls

• Full automation of every step is expensive. Automate the recovery paths that matter most and rely on game days for the rest.
• Generated docs still need human sentences explaining why and what to do when a step fails. Generate values, hand-write judgement.
• A test cluster can differ from production (size, network, data), so state the gap in the doc rather than implying the drill proves everything.
• Old versions of the runbook must remain reachable for clusters that have not been rebuilt yet.

Direct answer

Below is the entry for a machine-learning inference service (a service that loads a trained model and answers prediction requests) called recommender-inference, running on Kubernetes (a system that runs copies of a service, called pods, and restarts them). The newcomer test I would run first is the triage fork (steps 1 to 3: find the bad pods, read why they failed, choose one of three fixes), because that is where unwritten knowledge hides and where a wrong turn does damage.

The runbook entry

RUNBOOK: recommender-inference instance down or not serving
Owner: ml-platform-oncall   Last verified: <date>, by <name>
Use when: alert "recommender-inference error rate high" or "pods not ready"

SYMPTOMS
  - Pods show CrashLoopBackOff (the pod keeps crashing and being restarted), or Running but never Ready
  - Callers get HTTP 5xx or timeouts from /v1/predict
IMPACT
  - Recommendation widgets fall back to a static list (degraded, not down)

STEP 1. Find the bad pods.
  Run from: any machine with the prod kubeconfig (the cluster credentials file, context prod-ml)
    kubectl --context prod-ml -n ml-serving get pods -l app=recommender-inference
  Expected when healthy: all pods 1/1 Running. Note any that are not.

STEP 2. Read why it failed.
    kubectl --context prod-ml -n ml-serving describe pod <pod>
    kubectl --context prod-ml -n ml-serving logs <pod> --previous --tail=50
  Match:  Reason OOMKilled                 -> ran out of memory, go to 3A
          "failed to load model"           -> model artifact problem, go to 3B
          started right after a deploy     -> go to 3C

STEP 3A. One pod only: replace it (safe to repeat, the Deployment recreates it).
    kubectl --context prod-ml -n ml-serving delete pod <pod>
  If the replacement pod is OOMKilled again, deleting pods will not help: the memory limit or the model size has to change, so escalate to the owner.
STEP 3B. Model artifact missing or wrong version: pin the last good model.
    kubectl --context prod-ml -n ml-serving set env deployment/recommender-inference MODEL_VERSION=<last-good>
STEP 3C. Bad release: roll the deployment back.
    kubectl --context prod-ml -n ml-serving rollout undo deployment/recommender-inference

STEP 4. VERIFY (never skip).
    kubectl --context prod-ml -n ml-serving rollout status deployment/recommender-inference --timeout=180s
    curl -fsSi -X POST -H 'Content-Type: application/json' https://recommender.internal.example/v1/predict -d @sample-request.json
  Expected: rollout reports success; the first line curl prints is the status line (HTTP/1.1 200 OK or HTTP/2 200), followed by a body with a scores array.
  Then confirm the error-rate dashboard is falling for 5 minutes.

ROLLBACK OF THE FIX: re-apply the previous MODEL_VERSION value or run rollout undo again.
ESCALATE: after 30 minutes or if 3A to 3C do not help, page ml-platform-owners.

The service name, namespace, endpoint and environment variable are illustrative; every kubectl form shown is a real command.

Reading the commands in plain words

• --context prod-ml says which cluster to talk to. -n ml-serving picks a namespace, a named partition of the cluster where this team's pods live.
• -l app=recommender-inference is a label selector: it filters to only the pods carrying that tag.
• Running means the container process has started. Ready means it also passed its health check and is receiving traffic, so 1/1 means one of one containers is ready.
• OOMKilled means Kubernetes killed the container for running out of memory. logs --previous shows the output of the instance that just crashed.
• A Deployment is the object that declares how many copies of the pod to keep and recreates any that disappear, which is why deleting one pod is safe.
• set env changes an environment variable and makes the Deployment replace its pods. rollout undo reverts the Deployment to its previous release.
• 5xx means HTTP server-error codes (500 to 599).
• In the curl command, -i prints the HTTP status line and headers (without it curl shows only the body, so you could not see "200"), -f makes curl exit non-zero on a 4xx or 5xx response, -sS hides the progress meter but still shows errors, and -H sets the request's Content-Type header.

Why the entry is shaped this way

• Symptoms and impact come first so the reader confirms they are in the right page.
• Every command states where it runs, and the context flag prevents running against the wrong cluster.
• The machine-learning-specific failure (model file will not load) has its own branch, because a code rollback would not fix a bad model version.
• Verification is a separate step with a concrete pass condition, and each fix has its own undo.

Which part I test first with a newcomer, and why

I would hand the entry to an engineer who joined recently and stage a failure in a test environment (delete a pod, or set a wrong MODEL_VERSION). Then I would watch silently and write down each question they ask. I start with steps 1 to 3 (including the fixes, because I want to see whether the newcomer picks the branch the log evidence points to):

1. They decide which branch to take, so vague wording there sends someone to the wrong fix.
2. Access assumptions (is the kubeconfig set up, what is the context name) are invisible to the author.
3. If steps 1 to 2 fail, the reader never reaches the fixes in step 3, so a defect there blocks everything after it.

Step 4 is tested next, because a reader who cannot tell success from failure will stop too early or keep changing things.

Pitfalls

• Testing only the happy path. The test scenario should hit a branch other than the most common one.
• The author "helping" the newcomer, which hides the exact gaps the test exists to find.

Direct answer

Treat the runbook repository like code: every pull request (PR, a proposed change awaiting review) runs three automated checks in continuous integration (CI, the robot that runs on each PR). A link checker (lychee) catches dead links, a small script validates required front matter (the YAML metadata block at the top of each file, such as owner and last-reviewed date), and markdownlint-cli2 enforces Markdown style. Failures show up as inline annotations on the PR and as a red required check. For emergencies, an on-call engineer adds an emergency-fix label. That label makes only the checks that are cosmetic (lint) or depend on outside websites (external links) non-blocking, keeps the check that protects accountability (front matter) blocking, and automatically opens a follow-up issue so the debt is paid rather than forgotten.

Terms in plain words

• GitHub Actions: GitHub's built-in automation. A YAML file in .github/workflows/ tells GitHub what commands to run on each PR.
• Required status check / branch protection: branch protection is a repository setting on the main branch. Marking a check as "required" there means GitHub refuses to merge a PR until that check is green.
• Triage rights: a GitHub permission level. Only people with at least that level (or higher) on the repository can add or remove labels, so the label cannot be applied by a random outsider.
• Inline annotation: an error message drawn on the exact file and line in the PR's Files tab.
• continue-on-error: a step setting that lets the step fail without turning the whole job red.

Tools and what each one guards

The workflow (GitHub Actions syntax, parsed as valid YAML)

name: runbook-checks
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled]   # 'labeled' re-runs checks when the emergency label is added
    # No `paths:` filter on purpose: a required check that is skipped by path filtering stays "Pending" and blocks the PR
  schedule:
    - cron: "0 6 * * 1"   # weekly full link crawl catches link rot nobody's PR touched
permissions: { contents: read, issues: write, pull-requests: read }
jobs:
  docs-checks:            # the ONE job name set as a required status check
    runs-on: ubuntu-latest
    env:
      EMERGENCY: ${{ contains(github.event.pull_request.labels.*.name, 'emergency-fix') }}
    steps:
      - uses: actions/checkout@v4
      - name: Front matter (always blocking, because no owner means nobody is accountable)
        run: pip install pyyaml==6.0.3 && python scripts/check_front_matter.py $(git ls-files 'runbooks/*.md')
      - name: Markdown lint
        continue-on-error: ${{ env.EMERGENCY == 'true' }}
        run: npx --yes markdownlint-cli2@0.23.3 "runbooks/**/*.md"
      - name: Links
        continue-on-error: ${{ env.EMERGENCY == 'true' }}
        uses: lycheeverse/lychee-action@v2
        with:
          args: --no-progress --max-retries 2 --accept 200,429 "runbooks/**/*.md"
          fail: true
      - name: Open follow-up issue when the emergency path was used
        if: env.EMERGENCY == 'true'
        env: { GH_TOKEN: "${{ github.token }}" }
        run: gh issue create --title "Post-incident doc cleanup for PR ${{ github.event.pull_request.number }}" --label docs-debt --body "Lint and link checks were non-blocking on this PR. Re-run and fix within 2 business days."

Reading the workflow step by step

1. The workflow starts when a PR is opened, updated, reopened or labeled, on every PR (there is deliberately no paths: filter, see the pitfalls). A weekly schedule also runs it.
2. permissions gives the job read access to the code and write access to issues, so it can open the follow-up ticket, and nothing more.
3. EMERGENCY is true only when the PR carries the label emergency-fix (the contains(...) expression asks "is that name in the PR's label list?").
4. Front matter always runs and always blocks: if it fails, the job is red.
5. Lint and links have continue-on-error set to that flag: normally a failure turns the job red, but on an emergency PR the failure is still printed and the job stays green.
6. The last step runs only when the flag is true and files the cleanup issue.

The front-matter check, run for real

import os, sys, pathlib, tempfile
import yaml

REQUIRED = ["title", "owner", "service", "last_reviewed"]

def check(path, text):
    """Return a list of problems for one runbook."""
    text = text.replace("\r\n", "\n")   # files saved on Windows use CRLF line endings
    if not text.startswith("---\n"):
        return ["no front matter block"]
    end = text.find("\n---", 4)
    if end == -1:
        return ["front matter block is never closed"]
    try:
        meta = yaml.safe_load(text[4:end]) or {}
    except yaml.YAMLError as e:
        return [f"front matter is not valid YAML: {e.__class__.__name__}"]
    if not isinstance(meta, dict):
        return ["front matter is not a set of key: value pairs"]
    # a key that is present but empty (`owner:`) is as useless as a missing one
    return [f"missing key: {k}" for k in REQUIRED if meta.get(k) in (None, "")]

def main(paths):
    failures = 0
    for p in paths:
        for problem in check(p, pathlib.Path(p).read_text()):
            print(f"::error file={p}::{problem}")
            failures += 1
    return 1 if failures else 0

if __name__ == "__main__":
    if len(sys.argv) > 1:
        sys.exit(main(sys.argv[1:]))
    with tempfile.TemporaryDirectory() as d:
        os.chdir(d)
        good = pathlib.Path("db-failover.md")
        good.write_text("---\ntitle: DB failover\nowner: team-data\nservice: orders-db\nlast_reviewed: 2026-01-05\n---\n# Steps\n")
        bad = pathlib.Path("cache-flush.md")
        bad.write_text("---\ntitle: Cache flush\nservice: edge-cache\n---\n# Steps\n")
        empty = pathlib.Path("dns-cutover.md")
        empty.write_text("---\ntitle: DNS cutover\nowner:\nservice: dns\nlast_reviewed: 2026-02-01\n---\n# Steps\n")
        crlf = pathlib.Path("vpn-reset.md")
        crlf.write_bytes(b"---\r\ntitle: VPN reset\r\nowner: team-net\r\nservice: vpn\r\nlast_reviewed: 2026-03-01\r\n---\r\n# Steps\r\n")
        code = main([str(good), str(bad), str(empty), str(crlf)])
        print("exit code:", code)

Running it prints:

::error file=cache-flush.md::missing key: owner
::error file=cache-flush.md::missing key: last_reviewed
::error file=dns-cutover.md::missing key: owner
exit code: 1

The third file has owner: with nothing after it, which YAML reads as null. A plain k not in meta test would let it pass, defeating the rule that "no owner means nobody is accountable", so the check tests for an empty value too. The fourth file (Windows line endings) passes because the script normalises \r\n first.

The ::error file=...:: format is what makes GitHub draw the message on the exact file in the PR's Files tab, so the author does not have to open CI logs.

How failures surface

• One job name (docs-checks) is marked as a required status check in branch protection, so a red result blocks merge.
• Each tool writes file-level annotations, so the fix is visible where the author is already looking.
• Link checks accept HTTP 429 (rate limited) and retry twice, because a flaky third-party site should not fail the PR.
• The weekly scheduled run crawls everything, because links rot even when nobody edits the runbook.

Emergency path without gutting the checks

• Only people with triage rights can apply the label. GitHub itself enforces this (the workflow does not need to), so it is not a self-service bypass for anyone.
• Front matter stays blocking: it is local and takes seconds, but the real reason is that an emergency runbook with no owner or review date is the one nobody will fix later, which is worse than a lint warning.
• Lint and external links become non-blocking (continue-on-error). Lint findings are cosmetic and cannot mislead a reader, and link checks depend on third-party sites that may be down or rate-limiting during the same incident, so a red result there does not mean the runbook is wrong. Lint is not slow; the point is that a style complaint should never delay a fix.
• The bypass is not a branch-protection override or a disabled workflow. The checks still run and still report, so the author sees exactly what to clean up.
• The last step files a docs-debt issue with a two-business-day target, and a weekly report counts how many emergency labels were used. A rising count is a signal to fix the checks or the on-call process.

Worked example

At 02:10 on-call needs to change step 3 of a failover runbook. They open a PR, add emergency-fix, and a teammate approves. Front matter passes, lint fails on a trailing-space rule and is shown but non-blocking, and the merge happens in minutes. The follow-up issue lands in the team queue the same night, and the author fixes the lint error next morning.

Trade-offs and pitfalls

• Making every check non-blocking during emergencies is tempting but teaches people to always use the label. Keep the blocking set as small as the cheapest useful checks.
• Do not run link checks against private internal URLs without an allowlist or auth, or every PR fails. Exclude those patterns explicitly and verify them another way.
• Pin the action and tool versions so a tool upgrade does not turn every PR red on the same morning.
• A required check whose workflow is skipped by a paths: filter stays "Pending" forever (GitHub documents this), so a PR touching only a README would be unmergeable. Either drop the filter, as above, or add a tiny always-running job that reports the same check name.
• gh issue create --label docs-debt fails if that label does not exist, which would turn the emergency path red. Create the label once up front, and remember each new push to an emergency PR re-runs the step, so de-duplicate (for example search for an open issue with that title first).
• The workflow file itself is part of the PR, so an author could edit it to weaken the checks. Put .github/workflows/ under CODEOWNERS with "Require review from Code Owners" enabled.
• If the label is used often, the real problem is that the checks are too slow or too strict for normal work.