Technical Writing and Documentation Questions

Direct answer

The proposal (an RFC, "request for comments": a written design put out for peer review before building) that I would describe was an illustrative composite: it fixed repeated overload of a shared job queue. I structured it around evidence from real incidents, three options compared on cost and risk, a phased reversible rollout, and success measures agreed up front. It convinced reviewers because it showed the failure reproduced, offered the cheapest fix first, and could be switched off. Today I would establish the baseline earlier and involve the most affected team sooner.

Structured elaboration (situation, task, action)

Situation. Several teams sent jobs to one shared queue. Roughly every two months, one tenant's (one team or customer sharing the platform) failing jobs retried in a tight loop, filled the queue, and delayed everyone else's work. Each time the on-call engineer (the person responsible for responding to alerts out of hours) drained the queue by hand.

Task. I owned the reliability of that queue and had to move the team from firefighting to a fix that other teams would accept, since some of them had to change their retry code.

Action. I wrote the RFC with these sections:

1. Problem and evidence: three postmortems (written incident reviews) laid on one timeline, showing the same trigger each time. This made it a pattern, not an anecdote.
2. Goals and non-goals: goal, one tenant can no longer delay others. Non-goal, rewriting the queue technology.
3. Options: (a) per-tenant rate limits (a cap on how many jobs one tenant may submit per minute), (b) retry backoff with jitter (random delay so retries do not all fire together) plus a dead-letter queue (a holding area for jobs that keep failing), (c) migrate to a different queue product. I recommended a plus b, and rejected c as costing months for a problem we could fix in weeks.
4. Rollout: a flag, shadow mode first (log what would be limited without limiting), then one tenant, then all.
5. Success measures, defined before rollout: count of overload pages per quarter, and the age of the oldest waiting job. Written with a baseline and a target, for example (illustrative numbers): "baseline: 3 overload pages in the last two quarters, taken from the pager history; target: 0 over the next two; oldest waiting job under 10 minutes." Without the baseline, nobody can say later whether it worked.
6. Risks and reviewers' asks: what each dependent team had to change.

Worked example: what convinced reviewers, and the result

• We reproduced it. I replayed recorded incident traffic in staging and showed the queue saturating (filling to capacity so nothing new could get through), then showed it holding with the limits on.
• Cheapest reversible option first. Shadow mode meant nobody had to trust the design; they could see its decisions before it acted.
• I pre-reviewed with the noisiest team. Their objection (a hard limit could drop legitimate bursts) became a design change: a burst allowance (letting a tenant briefly exceed its cap for a short spike, then refilling slowly).
• Result (qualitative): the overload pages stopped recurring in the following quarters, and the on-call runbook (the step-by-step guide an engineer follows when an alert fires) lost its manual-drain step (emptying the queue by hand). I would not quote a precise percentage I cannot reconstruct.

Trade-offs, pitfalls and what I would change now

• Change 1: capture the baseline before writing, not during rollout, so the "before" number is not a reconstruction.
• Change 2: bring the affected team in at the outline stage, not the draft stage. Their burst objection cost a rewrite that an earlier conversation would have avoided.
• Change 3: the first draft ran to eight pages; reviewers replied to the parts they read. A two-page body with an appendix got better comments.
• Change 4: add a date to review the outcome (say, 90 days) so the proposal is judged against its own success measures.
• A pitfall in telling this story: naming virtues ("I collaborated") instead of the specific action. Interviewers remember the timeline overlay and the shadow mode, not the adjectives.

Direct answer

Below is the entry for a machine-learning inference service (a service that loads a trained model and answers prediction requests) called recommender-inference, running on Kubernetes (a system that runs copies of a service, called pods, and restarts them). The newcomer test I would run first is the triage fork (steps 1 to 3: find the bad pods, read why they failed, choose one of three fixes), because that is where unwritten knowledge hides and where a wrong turn does damage.

The runbook entry

RUNBOOK: recommender-inference instance down or not serving
Owner: ml-platform-oncall   Last verified: <date>, by <name>
Use when: alert "recommender-inference error rate high" or "pods not ready"

SYMPTOMS
  - Pods show CrashLoopBackOff (the pod keeps crashing and being restarted), or Running but never Ready
  - Callers get HTTP 5xx or timeouts from /v1/predict
IMPACT
  - Recommendation widgets fall back to a static list (degraded, not down)

STEP 1. Find the bad pods.
  Run from: any machine with the prod kubeconfig (the cluster credentials file, context prod-ml)
    kubectl --context prod-ml -n ml-serving get pods -l app=recommender-inference
  Expected when healthy: all pods 1/1 Running. Note any that are not.

STEP 2. Read why it failed.
    kubectl --context prod-ml -n ml-serving describe pod <pod>
    kubectl --context prod-ml -n ml-serving logs <pod> --previous --tail=50
  Match:  Reason OOMKilled                 -> ran out of memory, go to 3A
          "failed to load model"           -> model artifact problem, go to 3B
          started right after a deploy     -> go to 3C

STEP 3A. One pod only: replace it (safe to repeat, the Deployment recreates it).
    kubectl --context prod-ml -n ml-serving delete pod <pod>
  If the replacement pod is OOMKilled again, deleting pods will not help: the memory limit or the model size has to change, so escalate to the owner.
STEP 3B. Model artifact missing or wrong version: pin the last good model.
    kubectl --context prod-ml -n ml-serving set env deployment/recommender-inference MODEL_VERSION=<last-good>
STEP 3C. Bad release: roll the deployment back.
    kubectl --context prod-ml -n ml-serving rollout undo deployment/recommender-inference

STEP 4. VERIFY (never skip).
    kubectl --context prod-ml -n ml-serving rollout status deployment/recommender-inference --timeout=180s
    curl -fsSi -X POST -H 'Content-Type: application/json' https://recommender.internal.example/v1/predict -d @sample-request.json
  Expected: rollout reports success; the first line curl prints is the status line (HTTP/1.1 200 OK or HTTP/2 200), followed by a body with a scores array.
  Then confirm the error-rate dashboard is falling for 5 minutes.

ROLLBACK OF THE FIX: re-apply the previous MODEL_VERSION value or run rollout undo again.
ESCALATE: after 30 minutes or if 3A to 3C do not help, page ml-platform-owners.

The service name, namespace, endpoint and environment variable are illustrative; every kubectl form shown is a real command.

Reading the commands in plain words

• --context prod-ml says which cluster to talk to. -n ml-serving picks a namespace, a named partition of the cluster where this team's pods live.
• -l app=recommender-inference is a label selector: it filters to only the pods carrying that tag.
• Running means the container process has started. Ready means it also passed its health check and is receiving traffic, so 1/1 means one of one containers is ready.
• OOMKilled means Kubernetes killed the container for running out of memory. logs --previous shows the output of the instance that just crashed.
• A Deployment is the object that declares how many copies of the pod to keep and recreates any that disappear, which is why deleting one pod is safe.
• set env changes an environment variable and makes the Deployment replace its pods. rollout undo reverts the Deployment to its previous release.
• 5xx means HTTP server-error codes (500 to 599).
• In the curl command, -i prints the HTTP status line and headers (without it curl shows only the body, so you could not see "200"), -f makes curl exit non-zero on a 4xx or 5xx response, -sS hides the progress meter but still shows errors, and -H sets the request's Content-Type header.

Why the entry is shaped this way

• Symptoms and impact come first so the reader confirms they are in the right page.
• Every command states where it runs, and the context flag prevents running against the wrong cluster.
• The machine-learning-specific failure (model file will not load) has its own branch, because a code rollback would not fix a bad model version.
• Verification is a separate step with a concrete pass condition, and each fix has its own undo.

Which part I test first with a newcomer, and why

I would hand the entry to an engineer who joined recently and stage a failure in a test environment (delete a pod, or set a wrong MODEL_VERSION). Then I would watch silently and write down each question they ask. I start with steps 1 to 3 (including the fixes, because I want to see whether the newcomer picks the branch the log evidence points to):

1. They decide which branch to take, so vague wording there sends someone to the wrong fix.
2. Access assumptions (is the kubeconfig set up, what is the context name) are invisible to the author.
3. If steps 1 to 2 fail, the reader never reaches the fixes in step 3, so a defect there blocks everything after it.

Step 4 is tested next, because a reader who cannot tell success from failure will stop too early or keep changing things.

Pitfalls

• Testing only the happy path. The test scenario should hit a branch other than the most common one.
• The author "helping" the newcomer, which hides the exact gaps the test exists to find.

Direct answer. One CI/CD (continuous integration and continuous delivery, meaning automated build, check and release steps triggered by each change) pipeline with a build-and-verify job that fails on any warning, doc tests and link checks in the same job, a publish job that only runs on the main branch, and internal docs built and hosted separately from public docs, never filtered after the fact.

Start with this minimal pipeline; the rest is refinement. Core: build docs with warnings as errors, run the doc tests, publish only from main. Add link checking, internal/public separation and previews once that works.

Terms. Sphinx takes reStructuredText (a plain-text markup format, .rst) and Python docstrings and produces an HTML documentation site. JSDoc reads specially formatted comments in JavaScript source and produces an API reference site. Cross-references are links between documented items, which break when something is renamed. A doctest runs an example written as >>> code and checks the printed result. linkcheck fetches every link and reports dead ones. An artifact is a build output the CI system stores and passes to the next job. A CDN (content delivery network) caches your site on servers near readers. SSO (single sign-on) lets people log in with their company account. An allowlist is a list of exceptions you deliberately permit.

1. Validating the build

• Sphinx (Python): sphinx-build -W --keep-going -b html docs site/python. -W turns warnings (broken cross-references, malformed reStructuredText) into errors, and --keep-going reports all of them at once rather than the first only. Add -b linkcheck for dead links.
• JSDoc (JavaScript): npx jsdoc -c jsdoc.public.json --pedantic -d site/js. --pedantic makes JSDoc errors and warnings fatal. Add eslint-plugin-jsdoc to lint comment tags in the normal lint step.

2. Doc tests

• Python: the Sphinx doctest extension executes >>> examples embedded in docs and docstrings: sphinx-build -b doctest docs docs/_doctest (I ran this, and it reported the example passed).
• JavaScript: JSDoc does not run examples. I keep runnable examples in real test files (Jest or similar) and reference or copy them into @example, so a documented snippet cannot drift from working code.

3. Publishing to a static host
Upload the built site/ folder as an artifact and deploy it on merge to main (GitHub Pages shown; the same idea works with Amazon S3 static hosting plus a CDN, or Netlify). Pull requests build and check but do not publish, and may upload a preview.

name: docs
on:
  pull_request:
  push:
    branches: [main]
jobs:
  build-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: pip install -r docs/requirements.txt
      - run: npm ci
      - run: sphinx-build -W --keep-going -b html docs site/python
      - run: sphinx-build -b linkcheck docs docs/_linkcheck
      - run: sphinx-build -b doctest docs docs/_doctest
      - run: npx jsdoc -c jsdoc.public.json --pedantic -d site/js
      - uses: actions/upload-pages-artifact@v3
        with:
          path: site
  deploy-public:
    if: github.ref == 'refs/heads/main'
    needs: build-docs
    runs-on: ubuntu-latest
    permissions:
      pages: write
      id-token: write
    environment: github-pages
    steps:
      - uses: actions/deploy-pages@v4

Walk-through of the workflow. on: runs it for every pull request and every push to main. Job build-docs checks out the code, installs Python and Node, then runs the same commands as above in order: build the Python docs (fails on any warning), check links, run doc tests, build the JavaScript docs (fails on pedantic errors), and upload the finished site/ folder as the artifact. Job deploy-public waits for build-docs (needs), runs only on main (if), and is the only place with permission to publish (pages: write). Pull requests therefore verify but never publish.

The YAML parses (checked with a YAML parser); I did not run it on GitHub. Check that the action versions are the current majors when you copy it.

4. Separating internal from public docs

• Sphinx: wrap internal content in .. only:: internal and build public with no tag, internal with -t internal. I built both: the public HTML did not contain the internal section, the internal build did.
• JSDoc: mark internals @private (excluded by default) and build internal with a second config plus --private.
• Separate outputs and hosts: the public artifact goes to the public site; the internal build goes to a host behind login (SSO, single sign-on). Never publish the internal build and hide parts with CSS.
• Add a CI check that greps the public output for a marker string used in internal pages, so a mistake fails the build.

Trade-offs. Strict -W can block a release for a trivial warning: fix warnings promptly, do not turn the flag off. Link checks fail on flaky third-party sites: run them on a schedule as well and allowlist known-flaky hosts. Two builds double CI time, which is cheap compared with a leaked internal page.

Direct answer

Treat "people misread our docs" as a defect with causes, not a talent problem. I would propose a standard with four layers: a fixed page shape with required metadata, a short list of wording rules a machine can check, snippets that are executed rather than trusted, and a cold-read test for the few pages where a misreading is expensive. All of it runs in CI (continuous integration, the automatic checks that run on every pull request, meaning a proposed change awaiting review), so quality does not depend on one heroic editor.

1. Start from the misreads, not from a style guide

Pull the last quarter's production bugs that trace back to documentation and sort them by cause: a number without a unit, a missing precondition, a stale version, a snippet that no longer runs, two names for one thing. The standard only needs to attack the causes you actually see. Example: a Kafka topic (a named stream of messages in Kafka, a system that moves event data between programs) setting retention.ms (how long messages are kept) is in milliseconds, and a page saying "set retention to 7 days" led someone to type 7.

2. The standard: rule, then how it is checked

The table has five rows because the first layer, a fixed page shape with required metadata, is split into two checks: one for the heading layout and one for the front matter. The other three layers are one row each, which gives the four layers named above.

Readability scores such as Flesch (a formula that rates how easy text is to read from sentence and word length) measure sentence length, not correctness, so use them only as a weak warning. The "7 days" page would score well and still cause the bug. The fix is Set retention.ms to 604800000 (7 days = 7 x 24 x 60 x 60 x 1000 ms).

3. One convention across a mixed stack (Spark, an engine for processing large data sets; Kafka; Airflow; dbt)

Do not force each tool's native docs into one tool. Airflow (a workflow scheduler), dbt (a tool that builds SQL transformations) and the rest keep their own generated reference. The standard sits above them as a page-level contract: the same format (Markdown), the same required metadata, the same snippet conventions (tagged, pinned, runnable, with placeholders in UPPER_SNAKE_CASE), and a rule that hand-written pages link to generated reference instead of copying it.

4. No heroics: enforce by ratchet

A ratchet is a rule that only tightens: it never lets quality slip back, but it does not demand a big-bang cleanup either. Warnings first, then errors, and only on files a PR touches, so nobody must fix the whole backlog before merging. Pages older than a set age (say 180 days since last_verified) open an automatic ticket to the owner.

Worked example

REQUIRED = {"title", "owner", "audience", "last_verified", "applies_to"}

def parse_front_matter(text):
    lines = text.splitlines()
    if not lines or lines[0] != "---":
        return {}
    fields = {}
    for line in lines[1:]:
        if line == "---":
            break
        key, _, value = line.partition(":")
        fields[key.strip()] = value.strip()
    return fields

def check(name, text):
    missing = sorted(REQUIRED - parse_front_matter(text).keys())
    return f"{name}: FAIL missing {', '.join(missing)}" if missing else f"{name}: ok"

good = "---\ntitle: Backfill a dbt model\nowner: data-platform\naudience: data engineers\nlast_verified: 2026-09-01\napplies_to: dbt 1.8\n---\nBody"
bad = "---\ntitle: Restart a Kafka consumer\nowner: streaming\n---\nBody"
print(check("dbt-backfill.md", good))
print(check("kafka-restart.md", bad))

This prints dbt-backfill.md: ok and kafka-restart.md: FAIL missing applies_to, audience, last_verified. The Kafka page cannot merge until its owner says which audience and version it applies to, which is exactly the information that was missing when someone followed it for the wrong version.

How the script works: partition(":") splits each header line at the first colon into a key and a value. REQUIRED - keys is a set difference, meaning the required fields that are not present. sorted keeps the message in a stable order.

Trade-offs and pitfalls

• Rules that are too strict make people avoid writing docs. Start with the three causes that hurt most.
• Metadata checks prove presence, not truth. A date can be updated without re-verifying, which is why the cold read exists for high-risk pages.
• Measure it: count postmortems (written reviews of an incident after it is resolved) tagged "doc-caused" per quarter and the share of executed snippets that pass. Falling counts are the proof, style-guide adoption is not.
• What would change my call: a group with only a handful of pages should skip the linter and keep the cold read plus snippet execution.

Direct answer

Keep the docs in the same repository, in the folder beside the SQL models and pipeline configs they describe, and change both in one pull request (a proposed change reviewed before merge). Docs-only changes get a lighter path (one reviewer, checks limited to docs, publish on merge without a pipeline deploy). Releases are handled by building the docs site from tags (a tag is a permanent, named label on one commit, such as v2026.10), so each released version has its own frozen docs (a copy that never changes after release), with latest (the always-current docs) following the main branch.

Layout: docs sit next to what they describe

models/
  orders.sql
  customers.sql
  schema.yml          # per-model and per-column descriptions
pipelines/
  nightly_load.yml
docs/
  index.md
  changelog.md
  runbooks/nightly-load.md

Tools such as dbt (a widely used tool for building SQL models) keep descriptions in YAML beside each model and can generate a documentation site from them; the idea works with any tooling.

Workflow

1. Short-lived branches from main (a branch is a parallel line of work created for one change and deleted after it merges). A change to models/orders.sql includes the edit to its description in schema.yml and a line in docs/changelog.md, in the same pull request.
2. A check that every model has a description so a new model cannot merge undocumented. This is a real, runnable check (needs pip install pyyaml):

import pathlib, sys, yaml

def undocumented(models_dir):
    sql = {p.stem for p in pathlib.Path(models_dir).glob("*.sql")}
    documented = set()
    for y in pathlib.Path(models_dir).glob("*.yml"):
        for m in yaml.safe_load(y.read_text()).get("models", []):
            if m.get("description", "").strip():
                documented.add(m["name"])
    return sorted(sql - documented)

if __name__ == "__main__":
    missing = undocumented("models")
    print("models without a description:", missing)
    sys.exit(1 if missing else 0)

Run it against this fixture (a small folder made up for the test), where orders has a description and customers does not:

models/orders.sql
models/customers.sql
models/schema.yml

# models/schema.yml
models:
  - name: orders
    description: One row per order
  - name: customers

The script collects the names of the .sql files (orders, customers), collects the names in the YAML that have a non-empty description (orders), and prints the difference. It prints:

models without a description: ['customers']

and exits with status 1, which fails the pull request.
3. CODEOWNERS (a file naming who must approve changes to which paths) makes the model owner review both the SQL and its description.

Doc-only changes

• Use a commit prefix (the first word of the commit message) such as docs:, from the Conventional Commits convention (a naming scheme for commit messages; release tools that follow it do not bump a version for such commits) and a docs label.
• One reviewer is enough, and CI (continuous integration, automated checks on every change) runs only the docs checks (lint, links, build), not the expensive data tests.
• One caution: a path filter is a rule that starts a check only when certain files change. On GitHub, if a check is marked required but its path filter stops it from ever starting, the pull request waits forever for a result that never arrives. The fix is a check that always starts on every pull request and decides inside the job what to run. A minimal sketch of that decision (run_data_tests stands for your own command):

changed=$(git diff --name-only origin/main...HEAD)
if echo "$changed" | grep -qvE '^docs/'; then run_data_tests; else echo "docs only, skipping data tests"; fi

The check itself always reports success or failure, so the required-check rule is always satisfied.

• Merge publishes the site immediately. No pipeline deployment is needed, because nothing executable changed.

Releases

• Tag the data-model release (for example v2026.10). The docs site is built from the tag and published under its own path, while the main branch feeds latest. MkDocs (a static site generator that builds a website from Markdown) users often use a tool called mike for this versioned publishing.
• A wrong doc in an already released version is fixed on main, and cherry-picked (copying that one commit onto another branch) to the release branch (a long-lived branch holding one released version so it can still receive fixes) only if readers of that version are affected.
• The changelog is the bridge: every schema change has a dated line saying what changed and which version introduced it.

Trade-offs and pitfalls

• Descriptions in YAML beside the model stay accurate but are less friendly to non-engineers than a wiki. Generate a browsable site so they read it there.
• Do not version prose that has no relationship to a release (meeting notes, planning). Keep those elsewhere.
• Requiring docs on every trivial refactor creates noise. Allow an explicit docs-not-needed: reason in the pull request, and review those reasons occasionally.