Technical Writing and Documentation Questions

Direct answer

An ETL pipeline (extract, transform, load: pulling data from sources, reshaping it, and storing it for use) is easiest to learn as a small map, a working sandbox, and a set of real tasks, not as a long prose document. The doc set has five parts: orientation, architecture and data lineage, a data dictionary, operations, and a guided first-change tutorial. Beyond prose, supply things a newcomer can run or look at: a lineage diagram, a small sample dataset with a one-command local setup, schema and data-contract files, and recorded examples of past failures. I would judge success by how quickly new engineers reach real milestones, compared with the last cohort.

What the documentation set contains

Also record key design decisions (a short list of "why we did it this way") so people stop re-litigating them.

Concrete artifacts beyond prose

• Lineage diagram generated from the orchestrator (the scheduler that runs the stages), so it cannot drift from reality.
• Sample dataset plus make sandbox that runs the pipeline locally on a few hundred rows in minutes.
• Schema and contract files (column names, types, allowed nulls) that tests enforce, doubling as documentation.
• Failure library: three real past incidents, each with the symptom, the log line, and the fix.
• Starter tickets labelled "good first issue" that exercise the tutorial.

Measuring whether onboarding improved

Pick milestones before writing anything, and record them for the current cohort as the baseline:

• Days from start date to first merged pull request (a proposed change accepted into the codebase).
• Days to first solo on-call shift.
• Number and type of questions asked in the team channel in the first month, categorised by whether the answer existed in the docs.
• A short survey after two weeks: what was missing or confusing?

Worked example (illustrative numbers)

Before the docs, four engineers reached their first merged pull request after 14, 17, 19 and 22 days. Sorting them gives a median of (17 + 19) / 2 = 18 days. After the docs, the next four took 8, 9, 11 and 12 days, a median of (9 + 11) / 2 = 10 days. That is a drop of 8 days. With only four people per cohort, this is a signal, not proof, so I would confirm it with the survey and channel questions and check that the new hires were similar in experience.

Trade-offs and pitfalls

• Time-to-first-PR can be gamed by picking trivial tickets. Pair it with a review of what the change was.
• A pipeline that changes weekly makes hand-drawn diagrams rot, which is why the diagram and dictionary should be generated.
• Do not measure onboarding once. Repeat per cohort and treat each newcomer's confusion as bug reports on the docs.
• Too much detail overwhelms. Link to deeper references and keep orientation to one page.

Direct answer

I would rewrite the README around one goal: a newcomer runs a tiny, real training run and one prediction within about fifteen minutes, with exact commands and example files. Then I would keep it true by making the quick start executable: the README calls a single script that CI (continuous integration, automated checks on every change) runs on every pull request, so a broken quick start fails a build instead of surprising a reader.

The rewritten skeleton

# <model-name>: one sentence on what it predicts
Status, owner, model card link.

## Quick start (about 15 minutes, CPU only)
1. Set up: `make setup`   (pinned lockfile, Python version stated)
2. Get sample data: `make sample-data`  (small file in data/sample/)
3. Train a tiny run: `make train-tiny`  (1 epoch on 200 rows)
4. Predict: `python predict.py --input examples/input.json`
Expected output: a JSON object with fields `label` and `score`.

## Example input
examples/input.json (committed, shown in full below)

## Full training (reproducing the released model)
Config file, data snapshot id, hardware, expected metric with tolerance.

## Evaluate
`make eval`  and the metric it prints, with the reference value.

## Repository layout
## Troubleshooting (top 3 errors and fixes)
## Contributing and contacts

Terms in the skeleton: a Makefile is a file that gives short names (make targets) to shell commands, so make train-tiny runs whatever command is written under that name. A lockfile lists the exact version of every package, so everyone installs the same ones (that is what "pinned" means: fixed to an exact version). An epoch is one full pass over the training data. A model card is a short document stating what the model is for, what data it learned from and its known limits. Illustrative Makefile lines:

setup:       pip install -r requirements.lock
train-tiny:  python train.py --config configs/tiny.yaml --epochs 1 --max-rows 200

The two sections left abstract above would be filled like this (illustrative):

## Full training
python train.py --config configs/full.yaml    # data snapshot churn-2025-06-30, 1 GPU, about 2 hours
Expected metric: accuracy 0.91 +/- 0.01 on the holdout file

## Evaluate
make eval    # prints "accuracy: 0.91"; reference value is stored in eval/reference.json

What the original lacked and this fixes:

• No reproducible training command: the quick start gives an exact make train-tiny and the full run points at a committed config.
• No example inputs: examples/input.json is checked in and displayed, with the expected output so success is unmistakable.
• Expected results: each step states what the reader should see, so they know whether it worked.

Keeping the quick start from going stale

1. One source of truth for commands. The README calls make targets. The steps live in the Makefile, so there is nothing to copy wrongly.
2. A CI smoke job runs the quick start. Same commands, on a clean runner, on every pull request and on a nightly schedule (to catch drift in external dependencies).
3. Pin what can drift: a lockfile for packages, a fixed Python version, a named sample-data version.
4. Assert the output. The smoke test checks that predict.py returns the fields promised, not merely that it exits.
5. Ownership rule: a change to CLI flags or config keys must update the docs in the same pull request (docs-as-code: documentation is stored beside the code and reviewed in the same process).
6. A "last verified" line generated by CI, so a reader can see it is fresh.

name: quickstart-smoke
on:
  pull_request:
  schedule:
    - cron: "0 5 * * *"
jobs:
  quickstart:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make setup && make sample-data && make train-tiny
      - run: python predict.py --input examples/input.json | grep '"label"'

What the workflow file does, line by line:

• on: pull_request runs the job on every pull request (a proposed code change). schedule with cron: "0 5 * * *" also runs it daily. The five cron fields are minute, hour, day of month, month and day of week, so this means 05:00 UTC every day.
• runs-on: ubuntu-latest starts a fresh clean machine each time, so nothing from a developer's laptop can hide a problem.
• The first run executes the setup, sample-data and tiny-training commands. The second runs the prediction and pipes it to grep '"label"', which fails the step if that field is missing.
• This is a smoke test: a quick check that the main path works at all, not a full quality check.

Worked example

Suppose someone renames --input to --path. Without the smoke job the README breaks silently and the next new hire loses an hour. With it, the pull request fails at the predict.py step, and the author fixes the README in the same change.

Trade-offs and pitfalls

• Keep the tiny run truly tiny, or the job becomes slow and people disable it.
• A green smoke test proves the quick start works, not that the model is good. Full reproduction is verified separately.
• Do not paste large outputs into the README. Show the shape and a small example.

Direct answer

Another team must act on this without my context, so the ticket has to carry the evidence, the impact, a precise ask and a way to confirm the fix. I would write it with facts separate from guesses, include a query they can run in their own tools, and add a short primer if they have never seen the pipeline.

Evidence I would gather first (runnable, sample data)

The first query answers "which days are missing rows?". The WITH RECURSIVE days part builds a calendar with one row per date from 2025-03-01 to 2025-03-07 (a starting date, then repeatedly adding one day). We need it because a day with no rows at all cannot show up in a plain GROUP BY on the table. The LEFT JOIN keeps every calendar day even when nothing matches, COUNT(f.user_id) then gives 0 for it, and HAVING n < 4 keeps only days with fewer rows than expected. The Python only gathers evidence; the ticket contains the query and its output, not the Python setup. The second query is a separate, smaller demonstration used in the variant below.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE daily_features (feature_date TEXT, user_id INTEGER, sessions_7d INTEGER)")
# 4 users for each date from 2025-03-01 to 2025-03-07, except 03-04 (nothing) and 03-06 (only 2 users)
for day in range(1, 8):
    d = f"2025-03-0{day}"
    users = 0 if day == 4 else (2 if day == 6 else 4)
    for u in range(users):
        db.execute("INSERT INTO daily_features VALUES (?, ?, ?)", (d, u, 3 + u))

print("rows per date (expected 4):")
for r in db.execute("""
    WITH RECURSIVE days(d) AS (
        SELECT '2025-03-01' UNION ALL
        SELECT date(d, '+1 day') FROM days WHERE d < '2025-03-07')
    SELECT days.d, COUNT(f.user_id) AS n
    FROM days LEFT JOIN daily_features f ON f.feature_date = days.d
    GROUP BY days.d HAVING n < 4 ORDER BY days.d"""):
    print(" ", r)

# dashboard discrepancy variant: same metric from two sources, per day
db.execute("CREATE TABLE dash_a (day TEXT, signups INTEGER)")
db.execute("CREATE TABLE dash_b (day TEXT, signups INTEGER)")
db.executemany("INSERT INTO dash_a VALUES (?, ?)", [("2025-03-01", 120), ("2025-03-02", 131), ("2025-03-03", 118)])
db.executemany("INSERT INTO dash_b VALUES (?, ?)", [("2025-03-01", 120), ("2025-03-02", 124), ("2025-03-03", 118)])
print("days where the dashboards disagree:")
for r in db.execute("""SELECT a.day, a.signups AS dash_a, b.signups AS dash_b, a.signups - b.signups AS diff
                       FROM dash_a a JOIN dash_b b USING (day) WHERE a.signups <> b.signups"""):
    print(" ", r)

Output:

rows per date (expected 4):
  ('2025-03-04', 0)
  ('2025-03-06', 2)
days where the dashboards disagree:
  ('2025-03-02', 131, 124, 7)

The ticket

Title: daily_features missing rows for 2025-03-04 (all) and 2025-03-06 (half)

Impact: the churn model reads daily_features each morning. On the missing days it
scores stale features, and retention emails were sent to the wrong audience (stale features means values from an earlier day were used, so users were scored on old behaviour).
Priority: high, next model run is tomorrow 06:00 UTC.

What we see (facts):
  - 2025-03-04: 0 rows (expected 4 in this sample, normally one per active user)
  - 2025-03-06: 2 rows (expected 4)
  - Other days in 03-01..03-07 complete. Query and output attached.
When it started: first missing day is 2025-03-04.

What we checked: our jobs read the table only, no writes. Upstream job runs
show green on both days (the jobs that feed this table reported success, so it did not fail loudly).

Ask: (1) find why rows did not land, (2) backfill the two dates (reload the missing rows for those dates),
(3) tell us when done so we re-score (run the model again on corrected inputs).
How to confirm: rerun the attached query, no dates returned.
Hypothesis (unverified): partial load on 03-06 after a retry.
Contacts: <name> for model questions, #ml-features for follow-up.

Adapting it for a team that has never seen the pipeline

• Note that the daily_features table holds the model's inputs (feature values are the per-user numbers, such as sessions in the last 7 days, that the model reads).
• Add a 5-line primer at the top: what the table is, what reads it, where the upstream data comes from, with a diagram or link.
• Define our vocabulary ("active user", "sessions_7d") in one line each, and use their table names when possible.
• Give the check as a self-contained query they can run without our tools, and say what output means "fixed".
• Lead with impact and the ask, keep history below.
• Offer a 15-minute walkthrough, but never make the ticket depend on it.

Variant: two dashboards that keep disagreeing

Same structure, with reproduction steps (which dashboards, filters, date range) and a query that compares them day by day. The comparison in the script above returned one disagreement, on 2025-03-02 (131 versus 124, a difference of 7), which points the receiving team at a single day instead of "the numbers don't match".

Pitfalls

• Asking "please investigate" with no dates, no query, no deadline.
• Blaming the receiving team's job in the title.
• Leaving out who is affected and by when.

Direct answer

Reproducible means a stranger, with only what you left behind, can rebuild the same inputs, run the same code in the same environment and get the same result (or a result inside a stated tolerance). The minimum I always record is five things: the exact code version, the exact data version, the environment, the randomness controls, and the question plus the result and decision. Everything else can be found from those five.

The minimum record (one per experiment)

1. Question and decision: what you were testing and what you concluded. Without this, six months later nobody knows why the run exists.
2. Code: the git commit hash (a unique ID for a saved version of the code) and whether there were uncommitted changes. A notebook name alone is not a version.
3. Data: a snapshot identifier or query plus date, row counts, and how the train/validation/test split was made. Provenance (where each input came from and how it was transformed) includes the feature pipeline: which script and commit built the features, since features are often rebuilt from a moving table.
4. Environment: Python version and a lockfile (an exact list of package versions), plus hardware where it matters.
5. Randomness: seeds (the number that starts a random generator, so shuffles and initial weights repeat) for every library used, and a note on operations that are not deterministic anyway (some GPU operations sum in a different order and vary in the last decimals).

Then the result (metric, baseline, tolerance, where the saved model or output lives).

Worked example (illustrative values)

Metric used below: recall@top10pct means rank customers by the model's predicted churn risk, flag the top 10%, and measure what share of the customers who actually churned were in that flagged group. 0.61 means 61% of real churners were caught.

experiment: churn-model-2026-09-14-run3
question: Does adding the 30-day support-ticket count improve churn recall on new customers?
code:
  repo: github.com/example-org/churn-model
  commit: 3f9a1c2
  uncommitted_changes: none
environment:
  python: "3.12.4"
  lockfile: requirements.lock (sha256 a41c...e07)
  hardware: 1x GPU, driver 550
data:
  snapshot: warehouse.churn_features@2026-09-01
  row_count: 184230
  query_or_pipeline: pipelines/build_features.py @ 3f9a1c2
  split: by customer_id hash, 80/10/10, seed 42
randomness:
  numpy_seed: 42
  model_seed: 42
  nondeterministic_ops: GPU sum order (results vary in 4th decimal)
result:
  metric: recall@top10pct
  value: 0.61
  baseline_value: 0.58
  artifact: s3://ml-artifacts/churn/run3/model.pkl
decision: Keep feature; re-check on the Q4 snapshot.

This record is a short file stored next to the run (or in an experiment tracker such as MLflow or Weights & Biases; either tool is optional, a plain YAML file committed next to the code already meets the bar). Six months later, checking out commit 3f9a1c2, installing the lockfile, and rebuilding the 2026-09-01 snapshot reproduces the run within the stated tolerance.

Documenting a reproducible investigation of a claimed segment-level failure

Suppose a stakeholder says "the model is bad for customers on the annual plan". Write the investigation so it can be re-run:

• Claim in one sentence, with who made it and when.
• Definition of the segment as code or a query (not "annual customers"), plus its size.
• Same-snapshot comparison: the model's metric on the segment versus everyone else, computed on a named snapshot, with a confidence interval (a range the true value plausibly lies in, wider for small samples) or at least the segment's row count, because a small segment gives noisy numbers.
• What was ruled out: label quality (are the churn labels for this segment correct, or do annual customers show as "not churned" only because their contract has not ended yet), leakage (a feature that accidentally contains the answer, or information from after the prediction date, so offline scores look better than reality), a train/serve preprocessing mismatch, and a change in the data feed.
• Verdict and next action, and the exact command to reproduce the table.

Filled-in segment example (illustrative)

• Claim: "the model is bad for annual-plan customers" (raised by the retention lead, 2026-09-20).
• Segment: plan_type = 'annual' on warehouse.churn_features@2026-09-01; 1,840 customers, of whom 220 churned.
• Result: the model flagged 108 of those 220 churners in its top 10%, so recall@top10pct = 108/220 = 0.49. Everyone else: 0.62.
• Uncertainty: with 220 churners, the rough 95% interval on 0.49 is about 0.42 to 0.56, which does not reach 0.62, so the gap is unlikely to be noise.
• Ruled out: labels re-checked on 50 sampled annual accounts (all correct); no leaked feature found; the same preprocessing function is used in serving.
• Verdict: real weakness. Next action: add annual-specific features and re-measure; reproduce the table with python analysis/segment_recall.py --snapshot 2026-09-01.

Trade-offs and pitfalls

• Notebooks are the usual culprit: cells run out of order and hold hidden state. Re-run top to bottom before saving, and export the final analysis as a script or clean notebook.
• Over-documenting kills the habit. Automate capture (log the commit, lockfile hash and seed from code) so the human writes only the question and the decision.
• "Reproducible" has levels: same numbers exactly, same numbers within a tolerance, or same conclusion. Say which one you promise.

Direct answer

Keep the docs in the same repository, in the folder beside the SQL models and pipeline configs they describe, and change both in one pull request (a proposed change reviewed before merge). Docs-only changes get a lighter path (one reviewer, checks limited to docs, publish on merge without a pipeline deploy). Releases are handled by building the docs site from tags (a tag is a permanent, named label on one commit, such as v2026.10), so each released version has its own frozen docs (a copy that never changes after release), with latest (the always-current docs) following the main branch.

Layout: docs sit next to what they describe

models/
  orders.sql
  customers.sql
  schema.yml          # per-model and per-column descriptions
pipelines/
  nightly_load.yml
docs/
  index.md
  changelog.md
  runbooks/nightly-load.md

Tools such as dbt (a widely used tool for building SQL models) keep descriptions in YAML beside each model and can generate a documentation site from them; the idea works with any tooling.

Workflow

1. Short-lived branches from main (a branch is a parallel line of work created for one change and deleted after it merges). A change to models/orders.sql includes the edit to its description in schema.yml and a line in docs/changelog.md, in the same pull request.
2. A check that every model has a description so a new model cannot merge undocumented. This is a real, runnable check (needs pip install pyyaml):

import pathlib, sys, yaml

def undocumented(models_dir):
    sql = {p.stem for p in pathlib.Path(models_dir).glob("*.sql")}
    documented = set()
    for y in pathlib.Path(models_dir).glob("*.yml"):
        for m in yaml.safe_load(y.read_text()).get("models", []):
            if m.get("description", "").strip():
                documented.add(m["name"])
    return sorted(sql - documented)

if __name__ == "__main__":
    missing = undocumented("models")
    print("models without a description:", missing)
    sys.exit(1 if missing else 0)

Run it against this fixture (a small folder made up for the test), where orders has a description and customers does not:

models/orders.sql
models/customers.sql
models/schema.yml

# models/schema.yml
models:
  - name: orders
    description: One row per order
  - name: customers

The script collects the names of the .sql files (orders, customers), collects the names in the YAML that have a non-empty description (orders), and prints the difference. It prints:

models without a description: ['customers']

and exits with status 1, which fails the pull request.
3. CODEOWNERS (a file naming who must approve changes to which paths) makes the model owner review both the SQL and its description.

Doc-only changes

• Use a commit prefix (the first word of the commit message) such as docs:, from the Conventional Commits convention (a naming scheme for commit messages; release tools that follow it do not bump a version for such commits) and a docs label.
• One reviewer is enough, and CI (continuous integration, automated checks on every change) runs only the docs checks (lint, links, build), not the expensive data tests.
• One caution: a path filter is a rule that starts a check only when certain files change. On GitHub, if a check is marked required but its path filter stops it from ever starting, the pull request waits forever for a result that never arrives. The fix is a check that always starts on every pull request and decides inside the job what to run. A minimal sketch of that decision (run_data_tests stands for your own command):

changed=$(git diff --name-only origin/main...HEAD)
if echo "$changed" | grep -qvE '^docs/'; then run_data_tests; else echo "docs only, skipping data tests"; fi

The check itself always reports success or failure, so the required-check rule is always satisfied.

• Merge publishes the site immediately. No pipeline deployment is needed, because nothing executable changed.

Releases

• Tag the data-model release (for example v2026.10). The docs site is built from the tag and published under its own path, while the main branch feeds latest. MkDocs (a static site generator that builds a website from Markdown) users often use a tool called mike for this versioned publishing.
• A wrong doc in an already released version is fixed on main, and cherry-picked (copying that one commit onto another branch) to the release branch (a long-lived branch holding one released version so it can still receive fixes) only if readers of that version are affected.
• The changelog is the bridge: every schema change has a dated line saying what changed and which version introduced it.

Trade-offs and pitfalls

• Descriptions in YAML beside the model stay accurate but are less friendly to non-engineers than a wiki. Generate a browsable site so they read it there.
• Do not version prose that has no relationship to a release (meeting notes, planning). Keep those elsewhere.
• Requiring docs on every trivial refactor creates noise. Allow an explicit docs-not-needed: reason in the pull request, and review those reasons occasionally.