Technical Writing and Documentation Questions

Direct answer. A handoff document must capture what only the current owners know: why it was built this way, what it silently depends on, how it fails, what "healthy" looks like in business terms, and who to call. The receiving team can read the code and the schema themselves. They cannot recover intent, tribal knowledge, known landmines or agreements with stakeholders.

Terms. A runbook is a step-by-step operating guide for a routine task or alert, like a recipe. Tribal knowledge is what team members know but never wrote down. Shadowing means the receiving team watches, then does the work with the previous owners on hand. A KPI (key performance indicator) is a business number the team is judged on. A feature store is a shared table of prepared model inputs; feature drift is when those inputs change in distribution over time so the model sees data unlike its training data. A secrets manager is a locked vault for passwords and API keys. Rollback means returning to the previous working version. A label definition states exactly what counts as a churner, the thing the model predicts.

Template

1. Purpose and consumers   what it is for, who uses it, business decision it supports
2. Owners and contacts     current owner, receiving owner, escalation, stakeholders
3. Architecture at a glance diagram, inputs, outputs, schedules, hosting
4. Dependencies            upstream data sources and owners, credentials/where secrets live, downstream consumers
5. Acceptance criteria     the measurable conditions under which the receiving team accepts the handoff
6. Monitoring and KPIs     what is monitored, thresholds, and how each maps to a business KPI
7. Known issues and gotchas quirks, workarounds, past incidents, decisions NOT to revisit and why
8. Operations              runbook link, retraining/refresh cadence, on-call expectations, rollback
9. Open items and owners   research-to-production checklist with a named owner and date per item
10. Handoff timeline       shadowing period, sign-off, date ownership formally transfers

If time is short, keep these: 1 (purpose), 2 (owners), 4 (dependencies), 5 (acceptance criteria), 7 (known issues) and the runbook link from 8. The rest can follow within a sprint.

The acceptance criteria let a data scientist hand a model to ML engineers with no ambiguity ("done" means numbers, not opinion). The monitoring-to-KPI mapping serves SRE (site reliability engineering) and product together. Item 9, with a named owner and date on every line, keeps the research-to-production checklist from becoming orphaned advice.

Filled example for one item: a churn-prediction model (illustrative).

• Purpose: ranks subscribers by cancellation risk weekly; the retention team calls the top 500.
• Owners: current, Data Science lead; receiving, ML Platform on-call; escalation, Head of Retention.
• Dependencies: reads the subscriptions and support_tickets tables (owner: Data Engineering); needs the feature store job to finish by 02:00 UTC; API key stored in the secrets manager under churn-model.
• Acceptance criteria: reproduces the recorded validation score (same data version and seed) within the tolerance agreed in the handoff meeting, for example a recorded AUC (a 0 to 1 ranking-quality score) of 0.86 must reproduce between 0.855 and 0.865; scoring job completes within its agreed window on production data, for example under 45 minutes so the call list is ready before the retention team starts work on Monday; rollback tested once. The point is that the receiving team can pass or fail each line with a number, not an opinion.
• Monitoring to KPI: feature-drift alert on days_since_last_login (a leading sign the model is scoring stale behavior) maps to the KPI "retention calls that reach at-risk users"; a scoring-job failure maps to "retention team has no call list Monday".
• Known gotchas: trial accounts must be excluded or the model predicts them all as churners; the July training data has a billing migration gap, so it is deliberately skipped. Do not switch to daily scoring: the retention team can only act weekly.
• Open items: replace a manual CSV export (owner: ML Platform, next sprint); document the label definition (owner: Data Science).

Pitfalls. A handoff that is only a link to the repo; monitoring with no owner or KPI link; no shadowing period. The strongest check: ask the receiving team to run the runbook once, unaided, before sign-off.

Direct answer

Generate the facts, write the meaning. A pipeline extracts everything a machine can know reliably (schemas, column statistics, lineage, which is the record of which tables and columns feed which others, and freshness, meaning when the data was last updated) into a registry, a machine-readable catalogue with one record per table, model or feature, and humans add hand-written descriptions in separate files keyed to the same identifiers. A build step merges both into one site. Trust at scale comes from three rules: generated facts always win over prose, hand-written text is flagged stale when the thing it describes changes, and every page shows when it was generated and how complete it is.

Architecture

The core of this design is warehouse tables, their columns and their lineage. Experiments, model registries and feature stores are an extension that reuses the same record shape. In the diagram, the dbt manifest is the file the dbt transformation tool writes describing how each table is built from others; a column profiling job is a scheduled job that measures each column (how many values are missing, how many distinct values). Mermaid is a text format that renders as a diagram.

flowchart LR
  W[Warehouse metadata] --> R[Registry build]
  P[Column profiling job] --> R
  L[Lineage from query logs or dbt manifest] --> R
  X[Extension: experiments, model registry, feature store, monitoring] --> R
  H[Hand-written annotations in Git] --> R
  R --> V{Validate and flag stale}
  V --> S[Static docs site]
  V --> Q[Coverage and freshness report]

The registry record (machine-readable schema)

Every entity, whether a table, a model, a feature or a monitor, is emitted in one shared shape, so adapters (small converters, one per source system, that translate that system's metadata into the shared shape) write the same fields. Illustrative record:

id: warehouse.sales.orders
type: table
owner: data-platform@example.com
tier: 1
description_source: hand-written
columns:
  - name: order_total
    type: numeric(12,2)
    null_rate: 0.0
    profiled_on: 2026-09-27
    sample: false
    upstream: [warehouse.staging.stg_orders.order_total]
    description: Gross order value in EUR before tax and discounts.
    annotation_for_definition_hash: 8837df
lineage:
  parents: [warehouse.staging.stg_orders]
  confidence: parsed

How to read the record: tier: 1 marks a business-critical table; numeric(12,2) means a number with up to 12 digits, 2 after the decimal point; null_rate: 0.0 is the share of empty values; sample: false says the statistics came from the full table, not a sample; upstream lists the column this one is copied or derived from; confidence: parsed says lineage was read out of SQL text; annotation_for_definition_hash is a fingerprint of the column definition the description was written against (explained under stale detection).

Experiments, the model registry (a catalogue of trained models), the feature store (a shared store of model input columns) and monitoring dashboards (including a drift monitor, which watches for input data changing over time) each get an adapter that fills the same id, owner, columns or fields, lineage and freshness fields, so one site can show, say, a feature's source table, the models that consume it, and its drift monitor.

Combining generated and hand-written content

• Generated fields are read-only on the page: types, null rate, row counts, lineage, last profiled date. Nobody edits these, so they cannot lie.
• Hand-written fields hold meaning: what "order_total" includes, known quirks, how to join. They live in per-table YAML or Markdown in Git, reviewed like code.
• Stale detection: each annotation stores a hash (a short fixed-length fingerprint computed from text; any change to the text changes it) of the column definition (name and type) it was written against. For example, the SHA-256 fingerprint (first six hex characters) of the text order_total:numeric(12,2) is 8837df, while order_total:numeric(14,2) gives 3a6b4a; you can reproduce both with printf '%s' 'order_total:numeric(12,2)' | shasum -a 256. If the hash differs at build time, the page shows "description may be out of date" and opens a ticket for the owner.
• One automation rule: if a column is a pass-through in lineage (copied unchanged from the parent table, same name, no transformation) and has no description of its own, inherit the upstream column's description and label it "inherited from stg_orders.order_total". This lifts coverage cheaply without inventing text.

Trust at scale

• Tiering: hand-written docs are required only for tier-1 tables (the most business-critical tier, for example those feeding finance dashboards). Everything else gets auto-generated pages, clearly labelled.
• Freshness and coverage: every page shows generated-at and profiled-on dates; a report shows the percentage of tier-1 tables with owner, description and lineage.
• Honest statistics: profile incrementally (only tables whose last-modified time changed) and by sample on the largest tables, marked sample: true. Never show top values for columns tagged as personal data.
• Lineage confidence: label lineage as parsed from SQL, declared by a tool, or inferred, since parsers miss dynamic SQL (queries assembled as text at run time, so no parser can see which tables they touch).
• CI validates every registry record against the schema, so a broken adapter fails a build rather than publishing bad pages.

Worked example

A finance analyst opens sales.orders. The generated part shows 40 columns, types, null rates dated yesterday, and a lineage graph back to the staging table. The hand-written part explains that order_total excludes refunds. Last week an engineer changed order_total from numeric(12,2) to numeric(14,2); the build recomputes the fingerprint, gets 3a6b4a instead of the stored 8837df, and sees a mismatch, so the description carries a stale banner and the owner has a ticket, instead of the analyst trusting an old statement.

Trade-offs and pitfalls

• Auto-generating everything yields a huge site nobody trusts because the descriptions are empty. Coverage reporting and tiering keep effort on what matters.
• Ownership drifts when people leave; flag tables whose owner no longer exists.
• Profiling cost scales with table size; budget it rather than running everything nightly.

Direct answer

Assume anything typed into docs will be public and permanent, so real credentials and real personal data never go in. The approach has four layers: use obviously fake, reserved-for-examples values while writing, scan for leaks before merge, scan what gets published, and have a rehearsed response for the day something slips through. A checker in CI (the automatic checks run on every proposed change) makes this routine instead of relying on people remembering.

1. Write with safe values

• Credentials: show environment variables and placeholders such as YOUR_API_KEY, never a working token. Where a shaped example is needed, use one the vendor documents as a placeholder, for example the AKIAIOSFODNN7EXAMPLE key ID that AWS publishes for examples.
• Personal data (also called PII, personally identifiable information: names, emails, phone numbers, addresses): use synthetic records generated by a seeded script (a fixed starting value so the output repeats). Use reserved names: example.com (RFC 2606), IP ranges 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24 (RFC 5737). An RFC (Request for Comments) is a numbered internet standards document. These two reserve names and ranges for examples only, so they are guaranteed never to belong to a real person or machine, which is what makes them safe to publish. The /24 means the first 24 bits fix the network, leaving 256 addresses (192.0.2.0 to 192.0.2.255). Use fictional phone numbers such as 555-0100. An "anonymised" copy of production data is not safe, because rows can often be re-identified (matched back to real people by combining columns such as postcode and birth date).
• Hidden channels: screenshots, notebook outputs, log excerpts, curl examples with tokens in headers, and query results. These leak more often than the prose does.

2. Detect before merge

A scanner (open-source tools such as gitleaks or detect-secrets, or a small script like this one) runs as a pre-commit hook (a script git runs automatically just before a commit is recorded) and again in CI, over the source and the built site:

import re

RULES = {
    "aws-access-key-id": re.compile(r"\bAKIA[0-9A-Z]{16}\b"),
    "private-key-block": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    "bearer-token": re.compile(r"Bearer\s+[A-Za-z0-9\-_.]{20,}"),
    "real-looking-email": re.compile(r"[\w.+-]+@(?!example\.(?:com|org|net)\b)[\w-]+\.[\w.]+"),
}
ALLOWED = ("AKIAIOSFODNN7EXAMPLE",)  # the placeholder key AWS documents

def scan(text):
    findings = []
    for lineno, line in enumerate(text.splitlines(), start=1):
        for rule, pattern in RULES.items():
            for match in pattern.finditer(line):
                if match.group(0) not in ALLOWED:
                    findings.append((lineno, rule))
    return findings

page = """\
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_ACCESS_KEY_ID=AKIAZ3MPLEFAKE234567
curl -H "Authorization: Bearer abcdefghijklmnopqrstuvwx" https://api.example.com/v1/me
Contact jo.smith@corp-mail.io or support@example.com
"""
for lineno, rule in scan(page):
    print(f"line {lineno}: {rule}")

It prints line 2: aws-access-key-id, line 3: bearer-token and line 4: real-looking-email. Line 1 (the documented placeholder) and support@example.com pass. How to read the rules: AKIA[0-9A-Z]{16} means the letters AKIA followed by exactly 16 capital letters or digits, the shape of an AWS access key ID, and \b marks a word edge. In the email rule, (?!example\.(?:com|org|net)\b) is a negative lookahead: it matches only when the domain is not example.com, .org or .net.

The allowlist is tiny and reviewed, so a real key cannot hide behind it.

3. Stop it reaching the docs site

• Build the site in an environment with no production credentials, and run notebooks or examples against a sandbox, so generated output cannot contain live values.
• Scan the built output and the search index, not only the Markdown.
• Require a PR field: "Contains no real customer data, and how you know."

4. If something leaks

Assume compromise. Revoke and rotate the credential first (rotate means issue a new one and disable the old one), then remove the content, purge caches and history if needed (delete copies held by the search index, CDN caches and git history), and check access logs. Deleting the page alone fixes nothing, since the secret was already copied.

Trade-offs and pitfalls

• Scanners are noisy. Keep a small reviewed allowlist, or people will disable the tool.
• Pattern scanners find keys but not a real name in a screenshot. Human review still covers images and sample datasets.
• What would change my call: for an internal-only wiki, keep the scan and the placeholder rules but relax the publishing layer.

Direct answer

Another team must act on this without my context, so the ticket has to carry the evidence, the impact, a precise ask and a way to confirm the fix. I would write it with facts separate from guesses, include a query they can run in their own tools, and add a short primer if they have never seen the pipeline.

Evidence I would gather first (runnable, sample data)

The first query answers "which days are missing rows?". The WITH RECURSIVE days part builds a calendar with one row per date from 2025-03-01 to 2025-03-07 (a starting date, then repeatedly adding one day). We need it because a day with no rows at all cannot show up in a plain GROUP BY on the table. The LEFT JOIN keeps every calendar day even when nothing matches, COUNT(f.user_id) then gives 0 for it, and HAVING n < 4 keeps only days with fewer rows than expected. The Python only gathers evidence; the ticket contains the query and its output, not the Python setup. The second query is a separate, smaller demonstration used in the variant below.

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE daily_features (feature_date TEXT, user_id INTEGER, sessions_7d INTEGER)")
# 4 users for each date from 2025-03-01 to 2025-03-07, except 03-04 (nothing) and 03-06 (only 2 users)
for day in range(1, 8):
    d = f"2025-03-0{day}"
    users = 0 if day == 4 else (2 if day == 6 else 4)
    for u in range(users):
        db.execute("INSERT INTO daily_features VALUES (?, ?, ?)", (d, u, 3 + u))

print("rows per date (expected 4):")
for r in db.execute("""
    WITH RECURSIVE days(d) AS (
        SELECT '2025-03-01' UNION ALL
        SELECT date(d, '+1 day') FROM days WHERE d < '2025-03-07')
    SELECT days.d, COUNT(f.user_id) AS n
    FROM days LEFT JOIN daily_features f ON f.feature_date = days.d
    GROUP BY days.d HAVING n < 4 ORDER BY days.d"""):
    print(" ", r)

# dashboard discrepancy variant: same metric from two sources, per day
db.execute("CREATE TABLE dash_a (day TEXT, signups INTEGER)")
db.execute("CREATE TABLE dash_b (day TEXT, signups INTEGER)")
db.executemany("INSERT INTO dash_a VALUES (?, ?)", [("2025-03-01", 120), ("2025-03-02", 131), ("2025-03-03", 118)])
db.executemany("INSERT INTO dash_b VALUES (?, ?)", [("2025-03-01", 120), ("2025-03-02", 124), ("2025-03-03", 118)])
print("days where the dashboards disagree:")
for r in db.execute("""SELECT a.day, a.signups AS dash_a, b.signups AS dash_b, a.signups - b.signups AS diff
                       FROM dash_a a JOIN dash_b b USING (day) WHERE a.signups <> b.signups"""):
    print(" ", r)

Output:

rows per date (expected 4):
  ('2025-03-04', 0)
  ('2025-03-06', 2)
days where the dashboards disagree:
  ('2025-03-02', 131, 124, 7)

The ticket

Title: daily_features missing rows for 2025-03-04 (all) and 2025-03-06 (half)

Impact: the churn model reads daily_features each morning. On the missing days it
scores stale features, and retention emails were sent to the wrong audience (stale features means values from an earlier day were used, so users were scored on old behaviour).
Priority: high, next model run is tomorrow 06:00 UTC.

What we see (facts):
  - 2025-03-04: 0 rows (expected 4 in this sample, normally one per active user)
  - 2025-03-06: 2 rows (expected 4)
  - Other days in 03-01..03-07 complete. Query and output attached.
When it started: first missing day is 2025-03-04.

What we checked: our jobs read the table only, no writes. Upstream job runs
show green on both days (the jobs that feed this table reported success, so it did not fail loudly).

Ask: (1) find why rows did not land, (2) backfill the two dates (reload the missing rows for those dates),
(3) tell us when done so we re-score (run the model again on corrected inputs).
How to confirm: rerun the attached query, no dates returned.
Hypothesis (unverified): partial load on 03-06 after a retry.
Contacts: <name> for model questions, #ml-features for follow-up.

Adapting it for a team that has never seen the pipeline

• Note that the daily_features table holds the model's inputs (feature values are the per-user numbers, such as sessions in the last 7 days, that the model reads).
• Add a 5-line primer at the top: what the table is, what reads it, where the upstream data comes from, with a diagram or link.
• Define our vocabulary ("active user", "sessions_7d") in one line each, and use their table names when possible.
• Give the check as a self-contained query they can run without our tools, and say what output means "fixed".
• Lead with impact and the ask, keep history below.
• Offer a 15-minute walkthrough, but never make the ticket depend on it.

Variant: two dashboards that keep disagreeing

Same structure, with reproduction steps (which dashboards, filters, date range) and a query that compares them day by day. The comparison in the script above returned one disagreement, on 2025-03-02 (131 versus 124, a difference of 7), which points the receiving team at a single day instead of "the numbers don't match".

Pitfalls

• Asking "please investigate" with no dates, no query, no deadline.
• Blaming the receiving team's job in the title.
• Leaving out who is affected and by when.

Direct answer. The template forces the author to state, in this order: the decision the experiment informs, the hypothesis and success criterion written before seeing results, the setup, the result with its uncertainty, and the recommendation. The point is that a product manager can read the top three lines and act, while an engineer can find everything needed to reproduce or challenge it below.

Template

1. Decision this informs:   (ship / don't ship / pick config; who decides, by when)
2. Hypothesis and success criterion (written before the run):
   metric, minimum effect that matters, guardrail metrics that must not regress
3. Setup:  population and split, sample size, duration, seed(s), code commit, data version
4. Results: primary metric per arm, difference, 95% confidence interval, guardrails
5. Interpretation: what it does and does not show; known threats (novelty effect, leakage, peeking)
6. Recommendation: ship / iterate / stop, plus follow-up actions and owner
7. Reproduce: command, config file, environment lock, link to raw data

Terms. A guardrail metric is a metric that must not get worse even if the main one improves (for example page load time). The novelty effect is users reacting to something just because it is new, which fades. Leakage is test-set information accidentally used in training or tuning, which makes results look better than they are. Peeking is checking results repeatedly and stopping the moment they look good. Selection bias here means reporting only the best of many tries. A seed is the number that starts a random number generator: the same seed reproduces a run, different seeds give slightly different results. Percentage points are the plain difference between two percentages (12.9% minus 12.0% is 0.9 points; relative to 12.0% that is a 7.5% increase).

A 95% confidence interval is a range of effects consistent with the data: if it includes 0, the experiment did not show a reliable difference.

Filled example (A/B test, illustrative). Decision: ship a redesigned checkout button by Friday. Hypothesis: conversion (orders per visitor) rises by at least 0.5 percentage points; guardrail: page load unchanged. Setup: 10,000 visitors per arm, 14 days, random assignment by user ID.

Difference: 0.9 percentage points. Standard error: $\sqrt{\frac{0.12 \times 0.88}{10000} + \frac{0.129 \times 0.871}{10000}} \approx 0.0047$, so the 95% interval is $0.009 \pm 1.96 \times 0.0047$, about -0.02 to +1.82 percentage points. In words, the standard error measures how much a difference between two conversion rates would wobble from sample to sample: the first half of the square root is the control rate's variance divided by its visitors (0.12 x 0.88 / 10,000 = 0.00001056), the second is the same for treatment (0.129 x 0.871 / 10,000 = 0.0000112359); they add to 0.0000217959, whose square root is 0.00467 (0.47 points). Then 1.96 x 0.00467 = 0.00915, and 0.009 - 0.00915 = -0.00015 (-0.015 points) to 0.009 + 0.00915 = 0.01815 (+1.815 points). Interpretation: the point estimate clears the 0.5-point target, but the interval includes 0, so the data do not yet prove an effect. Recommendation: do not ship on this evidence. Note the test was underpowered for its own target: detecting a 0.5-point lift on a 12% baseline at 95% confidence and 80% power needs about 66,000 visitors per arm (2 x 0.12 x 0.88 x (1.96 + 0.84)^2 / 0.005^2, computed), not 10,000. Do not simply keep running this test and re-check it, because that is peeking and inflates false positives. Instead pre-declare a new run with the computed sample size (or use a sequential design built for early looks), then decide. This is exactly the case where a PM reading only "12.9 vs 12.0" would get it wrong, so the template forces the interval into the main line.

Hyperparameter-trial variant (metadata, top results, reproducibility, conclusions). After 50 trials of a model, the same skeleton becomes: metadata (dataset version, search space, tracker run IDs, code commit, hardware, seed count); top results (best 3 trials with the metric averaged over several seeds and its spread, plus the default configuration as baseline); reproducibility notes (exact config files, seed list, command); conclusions (which hyperparameters mattered, which did not, whether the top gap exceeds seed noise, next search). Rule: if the gap between rank 1 and rank 3 is smaller than the seed-to-seed spread, the report must say "not distinguishable".

Worked numbers (illustrative). Each top trial was run with 3 seeds. Trial A scored 0.837, 0.841, 0.845 (mean 0.841); trial B mean 0.839; trial C scored 0.834, 0.838, 0.842 (mean 0.838); the default configuration averaged 0.820. The seed-to-seed standard deviation is about 0.004 (for A: the scores sit 0.004 either side of the mean). The gap between rank 1 and rank 3 is 0.841 - 0.838 = 0.003, which is less than 0.004, so the report says "A, B and C are not distinguishable; all three beat the default by about 0.02". Only the last statement is supported by the data.

Gaming and guards. Reporting only the best of 50 trials inflates results (selection bias): require the count of trials run and a final untouched test-set score. Peeking and stopping early when the number looks good: the pre-declared duration and criterion. Changing the metric after the fact: criterion locked in section 2.