Technical Writing and Documentation Questions

Direct answer

Write the metric as a versioned definition that an outsider can execute without asking you anything: a plain-English sentence, the exact query, the time boundaries, what is excluded, where the data comes from, who owns it, and one small worked example with a known answer. MAU (monthly active users) is a good test case because "active", "user" and "month" each hide a choice, and two analysts will pick different ones unless the document forces a single reading.

This kind of write-up is a KPI (key performance indicator) definition: the agreed recipe for a number the business tracks.

What the definition must pin down

• Unit: distinct user_id, not accounts, devices or sessions.
• "Active": the exact event types that count (say login, page_view, purchase) and the ones that do not (background heartbeat pings).
• Time window: calendar month in UTC, written as a half-open range (start included, next month's start excluded), so an event at exactly midnight cannot land in two months.
• Exclusions: internal and test accounts, and events with no user id (anonymous traffic).
• Source and freshness: which tables, and the "data as of" timestamp of the run, because late-arriving events can change last month's number after you reported it. State the restatement policy (for example, "the prior month is re-run for five days, then frozen").
• Owner, version and changelog: who to ask, and a changelog (a dated list of what changed and why), one note each time the definition changes, so an old figure can be matched to the definition that produced it.
• Worked example: a tiny dataset with the expected result, so a reviewer can check the definition without production data.

Worked example (run against the tiny dataset below)

metric: monthly_active_users
definition: Distinct non-internal users with at least one qualifying event in a calendar month (UTC).
qualifying_events: [login, page_view, purchase]
window: "event_ts >= month_start AND event_ts < next_month_start"
excludes: [internal accounts, events with NULL user_id, heartbeat events]
owner: product-analytics
version: 3   # v3: heartbeat events stopped counting

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE users  (user_id INTEGER PRIMARY KEY, is_internal INTEGER NOT NULL);
CREATE TABLE events (user_id INTEGER, event_type TEXT NOT NULL, event_ts TEXT NOT NULL);
INSERT INTO users VALUES (1,0),(2,0),(3,0),(4,1),(5,0),(6,0),(8,0);
INSERT INTO events VALUES
 (1,'login','2026-08-03T09:00:00'), (1,'purchase','2026-08-20T12:00:00'),
 (2,'page_view','2026-08-31T23:30:00'),
 (3,'heartbeat','2026-08-10T10:00:00'),
 (4,'login','2026-08-11T10:00:00'),
 (5,'login','2026-07-31T23:59:59'),
 (6,'login','2026-09-01T00:00:00'),
 (NULL,'login','2026-08-12T10:00:00'),
 (8,'login','2026-08-15T08:00:00');
""")

MAU_SQL = """
SELECT COUNT(DISTINCT e.user_id)
FROM events e JOIN users u ON u.user_id = e.user_id
WHERE u.is_internal = 0
  AND e.event_type IN ('login','page_view','purchase')
  AND e.event_ts >= :month_start AND e.event_ts < :next_month_start
"""
print("MAU 2026-08:", db.execute(MAU_SQL, {"month_start": "2026-08-01T00:00:00",
                                          "next_month_start": "2026-09-01T00:00:00"}).fetchone()[0])

Output:

MAU 2026-08: 3

Why 3: users 1, 2 and 8 count (user 1 has two events but is counted once). User 3 only sent heartbeats, user 4 is internal, user 5's event is 23:59:59 on 31 July, user 6's event is exactly 00:00:00 on 1 September (both outside August), and the NULL-id row is anonymous. Each exclusion is a documented rule, so an auditor who reruns the query gets 3, and can see why every other row was left out.

Variant: a new KPI such as net revenue retention (NRR)

NRR measures how much recurring revenue you keep from customers you already had, including upgrades. MRR is monthly recurring revenue. In the formula below, expansion is extra MRR from upgrades by existing customers, contraction is MRR lost to downgrades, and churn is MRR lost to cancellations. A production definition needs these elements, each for a reason:

$\text{NRR}=\frac{\text{MRR}_{start}+\text{expansion}-\text{contraction}-\text{churn}}{\text{MRR}_{start}}$

$\frac{100{,}000+8{,}000-3{,}000-5{,}000}{100{,}000}=\frac{100{,}000}{100{,}000}=1.00$

That means the starting customers are worth exactly what they were a period ago (100%): upgrades of 8,000 cancelled out downgrades of 3,000 plus cancellations of 5,000.

A finished NRR definition, in the same style as the MAU one (illustrative):

metric: net_revenue_retention
question: How much recurring revenue do we keep from last quarter's customers, upgrades included?
formula: (mrr_start + expansion - contraction - churn) / mrr_start
cohort: customers with mrr > 0 on the first day of the quarter (new customers excluded)
excludes: [one-time fees, usage overage]
currency: converted at the rate on the last day of the quarter
sources: billing.subscriptions, billing.invoice_lines
owner: finance-analytics
approver: vp-finance
restatement: recomputed for 10 days after quarter end, then frozen
version: 2   # v2: overage excluded
test_value: "start 100000, +8000, -3000, -5000 => 1.00"

Pitfalls

• A definition that lives only in prose or in someone's head cannot be re-run. Keep the definition and the query in one place.
• Leaving the time zone or boundary unstated is the most common cause of "our numbers are off by a few".
• Documenting the query but not the data snapshot means you can no longer reproduce a number after late data lands.

Direct answer. The template forces the author to state, in this order: the decision the experiment informs, the hypothesis and success criterion written before seeing results, the setup, the result with its uncertainty, and the recommendation. The point is that a product manager can read the top three lines and act, while an engineer can find everything needed to reproduce or challenge it below.

Template

1. Decision this informs:   (ship / don't ship / pick config; who decides, by when)
2. Hypothesis and success criterion (written before the run):
   metric, minimum effect that matters, guardrail metrics that must not regress
3. Setup:  population and split, sample size, duration, seed(s), code commit, data version
4. Results: primary metric per arm, difference, 95% confidence interval, guardrails
5. Interpretation: what it does and does not show; known threats (novelty effect, leakage, peeking)
6. Recommendation: ship / iterate / stop, plus follow-up actions and owner
7. Reproduce: command, config file, environment lock, link to raw data

Terms. A guardrail metric is a metric that must not get worse even if the main one improves (for example page load time). The novelty effect is users reacting to something just because it is new, which fades. Leakage is test-set information accidentally used in training or tuning, which makes results look better than they are. Peeking is checking results repeatedly and stopping the moment they look good. Selection bias here means reporting only the best of many tries. A seed is the number that starts a random number generator: the same seed reproduces a run, different seeds give slightly different results. Percentage points are the plain difference between two percentages (12.9% minus 12.0% is 0.9 points; relative to 12.0% that is a 7.5% increase).

A 95% confidence interval is a range of effects consistent with the data: if it includes 0, the experiment did not show a reliable difference.

Filled example (A/B test, illustrative). Decision: ship a redesigned checkout button by Friday. Hypothesis: conversion (orders per visitor) rises by at least 0.5 percentage points; guardrail: page load unchanged. Setup: 10,000 visitors per arm, 14 days, random assignment by user ID.

Difference: 0.9 percentage points. Standard error: $\sqrt{\frac{0.12 \times 0.88}{10000} + \frac{0.129 \times 0.871}{10000}} \approx 0.0047$, so the 95% interval is $0.009 \pm 1.96 \times 0.0047$, about -0.02 to +1.82 percentage points. In words, the standard error measures how much a difference between two conversion rates would wobble from sample to sample: the first half of the square root is the control rate's variance divided by its visitors (0.12 x 0.88 / 10,000 = 0.00001056), the second is the same for treatment (0.129 x 0.871 / 10,000 = 0.0000112359); they add to 0.0000217959, whose square root is 0.00467 (0.47 points). Then 1.96 x 0.00467 = 0.00915, and 0.009 - 0.00915 = -0.00015 (-0.015 points) to 0.009 + 0.00915 = 0.01815 (+1.815 points). Interpretation: the point estimate clears the 0.5-point target, but the interval includes 0, so the data do not yet prove an effect. Recommendation: do not ship on this evidence. Note the test was underpowered for its own target: detecting a 0.5-point lift on a 12% baseline at 95% confidence and 80% power needs about 66,000 visitors per arm (2 x 0.12 x 0.88 x (1.96 + 0.84)^2 / 0.005^2, computed), not 10,000. Do not simply keep running this test and re-check it, because that is peeking and inflates false positives. Instead pre-declare a new run with the computed sample size (or use a sequential design built for early looks), then decide. This is exactly the case where a PM reading only "12.9 vs 12.0" would get it wrong, so the template forces the interval into the main line.

Hyperparameter-trial variant (metadata, top results, reproducibility, conclusions). After 50 trials of a model, the same skeleton becomes: metadata (dataset version, search space, tracker run IDs, code commit, hardware, seed count); top results (best 3 trials with the metric averaged over several seeds and its spread, plus the default configuration as baseline); reproducibility notes (exact config files, seed list, command); conclusions (which hyperparameters mattered, which did not, whether the top gap exceeds seed noise, next search). Rule: if the gap between rank 1 and rank 3 is smaller than the seed-to-seed spread, the report must say "not distinguishable".

Worked numbers (illustrative). Each top trial was run with 3 seeds. Trial A scored 0.837, 0.841, 0.845 (mean 0.841); trial B mean 0.839; trial C scored 0.834, 0.838, 0.842 (mean 0.838); the default configuration averaged 0.820. The seed-to-seed standard deviation is about 0.004 (for A: the scores sit 0.004 either side of the mean). The gap between rank 1 and rank 3 is 0.841 - 0.838 = 0.003, which is less than 0.004, so the report says "A, B and C are not distinguishable; all three beat the default by about 0.02". Only the last statement is supported by the data.

Gaming and guards. Reporting only the best of 50 trials inflates results (selection bias): require the count of trials run and a final untouched test-set score. Peeking and stopping early when the number looks good: the pre-declared duration and criterion. Changing the metric after the fact: criterion locked in section 2.

Direct answer

Reproducible means a stranger, with only what you left behind, can rebuild the same inputs, run the same code in the same environment and get the same result (or a result inside a stated tolerance). The minimum I always record is five things: the exact code version, the exact data version, the environment, the randomness controls, and the question plus the result and decision. Everything else can be found from those five.

The minimum record (one per experiment)

1. Question and decision: what you were testing and what you concluded. Without this, six months later nobody knows why the run exists.
2. Code: the git commit hash (a unique ID for a saved version of the code) and whether there were uncommitted changes. A notebook name alone is not a version.
3. Data: a snapshot identifier or query plus date, row counts, and how the train/validation/test split was made. Provenance (where each input came from and how it was transformed) includes the feature pipeline: which script and commit built the features, since features are often rebuilt from a moving table.
4. Environment: Python version and a lockfile (an exact list of package versions), plus hardware where it matters.
5. Randomness: seeds (the number that starts a random generator, so shuffles and initial weights repeat) for every library used, and a note on operations that are not deterministic anyway (some GPU operations sum in a different order and vary in the last decimals).

Then the result (metric, baseline, tolerance, where the saved model or output lives).

Worked example (illustrative values)

Metric used below: recall@top10pct means rank customers by the model's predicted churn risk, flag the top 10%, and measure what share of the customers who actually churned were in that flagged group. 0.61 means 61% of real churners were caught.

experiment: churn-model-2026-09-14-run3
question: Does adding the 30-day support-ticket count improve churn recall on new customers?
code:
  repo: github.com/example-org/churn-model
  commit: 3f9a1c2
  uncommitted_changes: none
environment:
  python: "3.12.4"
  lockfile: requirements.lock (sha256 a41c...e07)
  hardware: 1x GPU, driver 550
data:
  snapshot: warehouse.churn_features@2026-09-01
  row_count: 184230
  query_or_pipeline: pipelines/build_features.py @ 3f9a1c2
  split: by customer_id hash, 80/10/10, seed 42
randomness:
  numpy_seed: 42
  model_seed: 42
  nondeterministic_ops: GPU sum order (results vary in 4th decimal)
result:
  metric: recall@top10pct
  value: 0.61
  baseline_value: 0.58
  artifact: s3://ml-artifacts/churn/run3/model.pkl
decision: Keep feature; re-check on the Q4 snapshot.

This record is a short file stored next to the run (or in an experiment tracker such as MLflow or Weights & Biases; either tool is optional, a plain YAML file committed next to the code already meets the bar). Six months later, checking out commit 3f9a1c2, installing the lockfile, and rebuilding the 2026-09-01 snapshot reproduces the run within the stated tolerance.

Documenting a reproducible investigation of a claimed segment-level failure

Suppose a stakeholder says "the model is bad for customers on the annual plan". Write the investigation so it can be re-run:

• Claim in one sentence, with who made it and when.
• Definition of the segment as code or a query (not "annual customers"), plus its size.
• Same-snapshot comparison: the model's metric on the segment versus everyone else, computed on a named snapshot, with a confidence interval (a range the true value plausibly lies in, wider for small samples) or at least the segment's row count, because a small segment gives noisy numbers.
• What was ruled out: label quality (are the churn labels for this segment correct, or do annual customers show as "not churned" only because their contract has not ended yet), leakage (a feature that accidentally contains the answer, or information from after the prediction date, so offline scores look better than reality), a train/serve preprocessing mismatch, and a change in the data feed.
• Verdict and next action, and the exact command to reproduce the table.

Filled-in segment example (illustrative)

• Claim: "the model is bad for annual-plan customers" (raised by the retention lead, 2026-09-20).
• Segment: plan_type = 'annual' on warehouse.churn_features@2026-09-01; 1,840 customers, of whom 220 churned.
• Result: the model flagged 108 of those 220 churners in its top 10%, so recall@top10pct = 108/220 = 0.49. Everyone else: 0.62.
• Uncertainty: with 220 churners, the rough 95% interval on 0.49 is about 0.42 to 0.56, which does not reach 0.62, so the gap is unlikely to be noise.
• Ruled out: labels re-checked on 50 sampled annual accounts (all correct); no leaked feature found; the same preprocessing function is used in serving.
• Verdict: real weakness. Next action: add annual-specific features and re-measure; reproduce the table with python analysis/segment_recall.py --snapshot 2026-09-01.

Trade-offs and pitfalls

• Notebooks are the usual culprit: cells run out of order and hold hidden state. Re-run top to bottom before saving, and export the final analysis as a script or clean notebook.
• Over-documenting kills the habit. Automate capture (log the commit, lockfile hash and seed from code) so the human writes only the question and the decision.
• "Reproducible" has levels: same numbers exactly, same numbers within a tolerance, or same conclusion. Say which one you promise.

Direct answer

A notebook is a lab diary: it records what was tried, in the order it happened. A post is an argument: it states one conclusion and shows only what supports it. My editorial job is to find the conclusion, decide who needs it, and cut everything else, then check every claim I keep against the notebook by rerunning it.

Editorial steps

1. Interview the author (15 minutes) and read the notebook end to end. Ask "what did you find and what should the reader do about it?" Write the answer in one sentence. That is the headline.
2. Name the audience and the decision. Engineers want what changed and how to use or integrate it; product managers want impact, risk, confidence and next step. If both, lead with the result and put method below the fold.
3. Sort every section into three piles: keep (supports the headline), move to an appendix or link (needed to reproduce), delete (dead ends, debugging, repeated plots). Dead ends survive only if they change what the reader should believe (for example "we tried X and it did not help, so do not retry").
4. Rerun the notebook top to bottom in a clean environment so every number and chart is the one you publish. Hidden state and stale outputs are common.
5. Choose 2-4 figures. Each gets a title that states the takeaway ("Recall rises for new customers"), labelled axes, and units. Redraw plots that were made for exploring.
6. Translate the jargon. Define the metric (for example, recall: the share of real positives the model finds), give the baseline, and state uncertainty honestly, including sample size.
7. Write to a fixed skeleton: result and recommendation, why it matters, what we did (short), evidence (figures), caveats, next steps, link to the notebook and the data snapshot.
8. Cut again and have the author fact-check it. They approve accuracy; you own clarity. Then test it on one engineer and one PM who were not involved: can each say the conclusion back?

Worked example (illustrative excerpt)

Headline: Adding support-ticket counts to the churn model finds more at-risk customers among new accounts, at the cost of a slower nightly build.
Recommendation: Ship it for new accounts; hold for older accounts until we re-test in Q4.
Evidence: (Figure 1) recall by account age; (Figure 2) build time before and after.
Caveats: The new-account group is small, so the gain has wide uncertainty.
Reproduce it: link to the notebook, commit and data snapshot.

Of the roughly 500 lines, what survives is the two figures and about five lines of setup; the feature-engineering experiments live behind the link.

Trade-offs and pitfalls

• Cutting can distort: do not drop a caveat because it complicates the story.
• Do not paste code into the post unless the audience will run it.
• The author must approve; the editor's polish should never change a claim.

Direct answer

The balance depends on the reader's task and on the cost of getting it wrong. When the reader is trying to get started or to do a common task, brevity wins: one clear path, and everything else linked. When the reader is looking something up or doing something risky or irreversible, completeness wins: every parameter, default, unit and failure case. So I organise documentation by purpose instead of picking one length for everything.

How I decide

• Split by reader need. The Diataxis framework (a documentation approach by Daniele Procida) separates tutorials (learning), how-to guides (doing a task), reference (looking up facts) and explanation (understanding why). Tutorials and how-to guides should be short. Reference should be complete.
• Ask what happens if the reader misses this. If the answer is "a minor delay", cut it or link it. If it is "an outage or lost data", include it, even when it makes the page long.
• Simplify without dumbing down. Put the simple path on the main page and the nuance one link away. The reader gets the short version, but the exact version exists.
• Decide what not to document. Skip things the code or tool makes obvious, generated details that change weekly and can be generated on demand (for example, a full list of flags a tool prints itself), and internal history nobody acts on.

Case for brevity: a quickstart

A data pipeline library gets a quickstart of five steps: install, configure one source, run, see the output, next steps. It does not list all 40 options. The reader's goal is a first success in minutes, and each extra option delays it.

Case for exhaustive detail: a reference and an irreversible procedure

The retry setting on that same library gets a reference table:

How a reader uses that row: with retries 3 and backoff_seconds 2, the waits before the three retries are 2 s, 4 s and 8 s, so a persistent failure is reported after about 14 seconds of waiting (2 + 4 + 8).

A step that deletes a production table gets equal completeness: preconditions, exact command, how to verify, how to undo. A person following it at 3 a.m. cannot ask you what you meant.

Trade-offs and pitfalls

• Exhaustive everywhere buries the one fact people need; brevity everywhere forces them to guess. Layering avoids both.
• Completeness decays: every detail you write is a detail that can go stale. Generate reference from code where you can.
• Brevity is not vagueness. A short sentence still needs the exact command and the exact condition.
• What would change my call: for an audience of experts under time pressure (an on-call runbook for a familiar system; a runbook is a step-by-step guide for handling a known operational situation, and on-call means the engineer currently responsible for responding to alerts), lean shorter, since the reader already knows the background.