Technical Writing and Documentation Questions

Direct answer

They answer different questions for different readers. The model card explains what the model is for and how far to trust it (readers: product owners, risk, regulators, reviewers). The README (the front page of the code repository) explains how to build, run and change the project (readers: engineers who work on it). The API specification (a machine-readable contract such as OpenAPI, a standard format for describing HTTP endpoints) defines exactly how another system calls the model (readers: engineers integrating with it).

Terms used below: a schema is the agreed list of fields and their types; a datasheet is the equivalent document for a dataset (where it came from, how it was collected); drift means the model's live behaviour or input data slowly moving away from what it was tested on; integration tests are automated tests that call the real service the way another system would; an on-call engineer is the person currently responsible for responding when the service breaks. An HTTP 422 response ("Unprocessable Content") means the request was well formed but failed the service's validation rules, for example an empty text field.

Comparison

Where each stakeholder question belongs

Worked example

A product owner asks, "Will it work on Spanish text?" That is a model-card question: the card's data section should say whether Spanish was in training and the evaluated accuracy. An engineer asks, "What does the endpoint return for empty text?" That is the API spec: a response schema and an example. Putting either answer in the wrong file means the right reader never finds it.

What each looks like in practice (illustrative). A single model-card line for the product owner:

Languages: trained on English and Spanish. Accuracy is reported separately per language in the evaluation table. Other languages are out of scope.

And an API-spec fragment (OpenAPI, written in YAML, a plain-text key: value format) for the empty-text case:

paths:
  /classify:
    post:
      requestBody:
        content:
          application/json:
            schema:
              type: object
              required: [text]
              properties:
                text: {type: string, minLength: 1, maxLength: 5000}
      responses:
        "200": {description: Label and score}
        "422": {description: "Validation failed, for example empty text"}

The card says what to trust; the spec says exactly which request is accepted (a text of 1 to 5,000 characters) and what comes back when it is not.

Pitfalls

• Cramming everything into the README so the model card is skipped.
• Repeating numbers in three places; they diverge. Keep each fact in one place and link to it.
• Treating the API spec as documentation of model quality; it documents shape, not correctness.

Direct answer

Define the score as a weighted checklist of things that can be verified from the model's repository and registry, cap it by freshness, and cross-check the claims against real evaluation data so filling in a field is not enough to score. Then say explicitly that the score is a screening signal for triage, not a goal to reward.

Definition (numerator, denominator, window)

A model card is a short document stating a model's intended use, training data, evaluation results and limitations. The model registry is the shared catalog where a team records every trained model version: its owner, its version number and its evaluation runs (a saved test of that version on held-out data, with the metrics it produced). The deployment config is the file that says which version is live in production. The score for one model is:

$\text{score} = \min\!\Big(\sum_i w_i c_i,\ \text{freshness cap}\Big)$

where each check c_i is 1 if passed and 0 if not, the weights w_i sum to 100, and the cap is 60 if the card was last reviewed more than 180 days ago. Intuition: a full, verified card scores 100; an old card cannot score above 60 however complete it looks. Reading it with numbers: checks passed with weights 20 + 20 + 20 + 5 = 65, but the card is 212 days old, so min(65, 60) = 60.

Worked example (pinned date, run it)

import datetime as dt

TODAY = dt.date(2026, 9, 1)  # pinned so the result never changes

# Rubric: each item is a yes/no check a script or a reviewer can answer.
# Weights sum to 100.
RUBRIC = {"intended_use": 20, "training_data": 20, "eval_by_slice": 25,
          "limitations": 20, "owner_and_contact": 5, "version_matches": 10}

cards = {
    "fraud-v3":  {"intended_use": 1, "training_data": 1, "eval_by_slice": 1, "limitations": 1,
                  "owner_and_contact": 1, "version_matches": 1, "reviewed": dt.date(2026, 8, 10)},
    "churn-v2":  {"intended_use": 1, "training_data": 1, "eval_by_slice": 0, "limitations": 1,
                  "owner_and_contact": 1, "version_matches": 0, "reviewed": dt.date(2026, 2, 1)},
    "rank-v7":   {"intended_use": 1, "training_data": 0, "eval_by_slice": 0, "limitations": 0,
                  "owner_and_contact": 1, "version_matches": 1, "reviewed": dt.date(2026, 8, 20)},
}

def score(card):
    base = sum(w for k, w in RUBRIC.items() if card[k])
    age_days = (TODAY - card["reviewed"]).days
    stale = age_days > 180          # freshness cap: stale cards cannot score above 60
    return base, min(base, 60) if stale else base, age_days

for name, card in cards.items():
    base, final, age = score(card)
    print(f"{name}: checklist={base} final={final} days_since_review={age}")

Output:

fraud-v3: checklist=100 final=100 days_since_review=22
churn-v2: checklist=65 final=60 days_since_review=212
rank-v7: checklist=35 final=35 days_since_review=12

Check the arithmetic: churn-v2 passes intended use, training data, limitations and owner (20 + 20 + 20 + 5 = 65), fails slice evaluation and version match, and its last review was 212 days before the pinned date, so the cap of 60 applies. rank-v7 gets 20 + 5 + 10 = 35.

What data feeds it

Card files in the repository (parsed by a script), the model registry (versions, owners, latest evaluation run), and deployment config (which version is live). Aggregate the score per team on the dashboard, and show the distribution of scores and the reason for each failing check, not one average.

How to stop gaming (Goodhart's law: when a measure becomes a target, it stops being a good measure)

• Verify, don't just detect a field: the evaluation numbers on the card must match the registry's latest run; the version must match what is deployed. Copying template text scores zero.
• Sampling audit: a human reviews a random 5 percent of high scorers each quarter; a score that a reviewer disagrees with triggers a rule change.
• Freshness cap so old cards decay.
• Do not attach bonuses or punishment to the raw number; use it to route help to the teams with low scores, and publish the checks so the goal is the document's quality.
• Outcome tie-back: watch whether low-score models have more incidents or slower onboarding (new team members taking longer to become productive on that model); if the score does not predict anything, revise it.

Trade-offs and pitfalls

A checklist measures presence, not truth or clarity. Keep the number small, explain each point deducted, and accept that the score triages effort rather than proving a document is good.

Direct answer

Assume anything typed into docs will be public and permanent, so real credentials and real personal data never go in. The approach has four layers: use obviously fake, reserved-for-examples values while writing, scan for leaks before merge, scan what gets published, and have a rehearsed response for the day something slips through. A checker in CI (the automatic checks run on every proposed change) makes this routine instead of relying on people remembering.

1. Write with safe values

• Credentials: show environment variables and placeholders such as YOUR_API_KEY, never a working token. Where a shaped example is needed, use one the vendor documents as a placeholder, for example the AKIAIOSFODNN7EXAMPLE key ID that AWS publishes for examples.
• Personal data (also called PII, personally identifiable information: names, emails, phone numbers, addresses): use synthetic records generated by a seeded script (a fixed starting value so the output repeats). Use reserved names: example.com (RFC 2606), IP ranges 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24 (RFC 5737). An RFC (Request for Comments) is a numbered internet standards document. These two reserve names and ranges for examples only, so they are guaranteed never to belong to a real person or machine, which is what makes them safe to publish. The /24 means the first 24 bits fix the network, leaving 256 addresses (192.0.2.0 to 192.0.2.255). Use fictional phone numbers such as 555-0100. An "anonymised" copy of production data is not safe, because rows can often be re-identified (matched back to real people by combining columns such as postcode and birth date).
• Hidden channels: screenshots, notebook outputs, log excerpts, curl examples with tokens in headers, and query results. These leak more often than the prose does.

2. Detect before merge

A scanner (open-source tools such as gitleaks or detect-secrets, or a small script like this one) runs as a pre-commit hook (a script git runs automatically just before a commit is recorded) and again in CI, over the source and the built site:

import re

RULES = {
    "aws-access-key-id": re.compile(r"\bAKIA[0-9A-Z]{16}\b"),
    "private-key-block": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    "bearer-token": re.compile(r"Bearer\s+[A-Za-z0-9\-_.]{20,}"),
    "real-looking-email": re.compile(r"[\w.+-]+@(?!example\.(?:com|org|net)\b)[\w-]+\.[\w.]+"),
}
ALLOWED = ("AKIAIOSFODNN7EXAMPLE",)  # the placeholder key AWS documents

def scan(text):
    findings = []
    for lineno, line in enumerate(text.splitlines(), start=1):
        for rule, pattern in RULES.items():
            for match in pattern.finditer(line):
                if match.group(0) not in ALLOWED:
                    findings.append((lineno, rule))
    return findings

page = """\
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_ACCESS_KEY_ID=AKIAZ3MPLEFAKE234567
curl -H "Authorization: Bearer abcdefghijklmnopqrstuvwx" https://api.example.com/v1/me
Contact jo.smith@corp-mail.io or support@example.com
"""
for lineno, rule in scan(page):
    print(f"line {lineno}: {rule}")

It prints line 2: aws-access-key-id, line 3: bearer-token and line 4: real-looking-email. Line 1 (the documented placeholder) and support@example.com pass. How to read the rules: AKIA[0-9A-Z]{16} means the letters AKIA followed by exactly 16 capital letters or digits, the shape of an AWS access key ID, and \b marks a word edge. In the email rule, (?!example\.(?:com|org|net)\b) is a negative lookahead: it matches only when the domain is not example.com, .org or .net.

The allowlist is tiny and reviewed, so a real key cannot hide behind it.

3. Stop it reaching the docs site

• Build the site in an environment with no production credentials, and run notebooks or examples against a sandbox, so generated output cannot contain live values.
• Scan the built output and the search index, not only the Markdown.
• Require a PR field: "Contains no real customer data, and how you know."

4. If something leaks

Assume compromise. Revoke and rotate the credential first (rotate means issue a new one and disable the old one), then remove the content, purge caches and history if needed (delete copies held by the search index, CDN caches and git history), and check access logs. Deleting the page alone fixes nothing, since the secret was already copied.

Trade-offs and pitfalls

• Scanners are noisy. Keep a small reviewed allowlist, or people will disable the tool.
• Pattern scanners find keys but not a real name in a screenshot. Human review still covers images and sample datasets.
• What would change my call: for an internal-only wiki, keep the scan and the placeholder rules but relax the publishing layer.

Direct answer. The template forces the author to state, in this order: the decision the experiment informs, the hypothesis and success criterion written before seeing results, the setup, the result with its uncertainty, and the recommendation. The point is that a product manager can read the top three lines and act, while an engineer can find everything needed to reproduce or challenge it below.

Template

1. Decision this informs:   (ship / don't ship / pick config; who decides, by when)
2. Hypothesis and success criterion (written before the run):
   metric, minimum effect that matters, guardrail metrics that must not regress
3. Setup:  population and split, sample size, duration, seed(s), code commit, data version
4. Results: primary metric per arm, difference, 95% confidence interval, guardrails
5. Interpretation: what it does and does not show; known threats (novelty effect, leakage, peeking)
6. Recommendation: ship / iterate / stop, plus follow-up actions and owner
7. Reproduce: command, config file, environment lock, link to raw data

Terms. A guardrail metric is a metric that must not get worse even if the main one improves (for example page load time). The novelty effect is users reacting to something just because it is new, which fades. Leakage is test-set information accidentally used in training or tuning, which makes results look better than they are. Peeking is checking results repeatedly and stopping the moment they look good. Selection bias here means reporting only the best of many tries. A seed is the number that starts a random number generator: the same seed reproduces a run, different seeds give slightly different results. Percentage points are the plain difference between two percentages (12.9% minus 12.0% is 0.9 points; relative to 12.0% that is a 7.5% increase).

A 95% confidence interval is a range of effects consistent with the data: if it includes 0, the experiment did not show a reliable difference.

Filled example (A/B test, illustrative). Decision: ship a redesigned checkout button by Friday. Hypothesis: conversion (orders per visitor) rises by at least 0.5 percentage points; guardrail: page load unchanged. Setup: 10,000 visitors per arm, 14 days, random assignment by user ID.

Difference: 0.9 percentage points. Standard error: $\sqrt{\frac{0.12 \times 0.88}{10000} + \frac{0.129 \times 0.871}{10000}} \approx 0.0047$, so the 95% interval is $0.009 \pm 1.96 \times 0.0047$, about -0.02 to +1.82 percentage points. In words, the standard error measures how much a difference between two conversion rates would wobble from sample to sample: the first half of the square root is the control rate's variance divided by its visitors (0.12 x 0.88 / 10,000 = 0.00001056), the second is the same for treatment (0.129 x 0.871 / 10,000 = 0.0000112359); they add to 0.0000217959, whose square root is 0.00467 (0.47 points). Then 1.96 x 0.00467 = 0.00915, and 0.009 - 0.00915 = -0.00015 (-0.015 points) to 0.009 + 0.00915 = 0.01815 (+1.815 points). Interpretation: the point estimate clears the 0.5-point target, but the interval includes 0, so the data do not yet prove an effect. Recommendation: do not ship on this evidence. Note the test was underpowered for its own target: detecting a 0.5-point lift on a 12% baseline at 95% confidence and 80% power needs about 66,000 visitors per arm (2 x 0.12 x 0.88 x (1.96 + 0.84)^2 / 0.005^2, computed), not 10,000. Do not simply keep running this test and re-check it, because that is peeking and inflates false positives. Instead pre-declare a new run with the computed sample size (or use a sequential design built for early looks), then decide. This is exactly the case where a PM reading only "12.9 vs 12.0" would get it wrong, so the template forces the interval into the main line.

Hyperparameter-trial variant (metadata, top results, reproducibility, conclusions). After 50 trials of a model, the same skeleton becomes: metadata (dataset version, search space, tracker run IDs, code commit, hardware, seed count); top results (best 3 trials with the metric averaged over several seeds and its spread, plus the default configuration as baseline); reproducibility notes (exact config files, seed list, command); conclusions (which hyperparameters mattered, which did not, whether the top gap exceeds seed noise, next search). Rule: if the gap between rank 1 and rank 3 is smaller than the seed-to-seed spread, the report must say "not distinguishable".

Worked numbers (illustrative). Each top trial was run with 3 seeds. Trial A scored 0.837, 0.841, 0.845 (mean 0.841); trial B mean 0.839; trial C scored 0.834, 0.838, 0.842 (mean 0.838); the default configuration averaged 0.820. The seed-to-seed standard deviation is about 0.004 (for A: the scores sit 0.004 either side of the mean). The gap between rank 1 and rank 3 is 0.841 - 0.838 = 0.003, which is less than 0.004, so the report says "A, B and C are not distinguishable; all three beat the default by about 0.02". Only the last statement is supported by the data.

Gaming and guards. Reporting only the best of 50 trials inflates results (selection bias): require the count of trials run and a final untouched test-set score. Peeking and stopping early when the number looks good: the pre-declared duration and criterion. Changing the metric after the fact: criterion locked in section 2.

Direct answer

Reproducible means a stranger, with only what you left behind, can rebuild the same inputs, run the same code in the same environment and get the same result (or a result inside a stated tolerance). The minimum I always record is five things: the exact code version, the exact data version, the environment, the randomness controls, and the question plus the result and decision. Everything else can be found from those five.

The minimum record (one per experiment)

1. Question and decision: what you were testing and what you concluded. Without this, six months later nobody knows why the run exists.
2. Code: the git commit hash (a unique ID for a saved version of the code) and whether there were uncommitted changes. A notebook name alone is not a version.
3. Data: a snapshot identifier or query plus date, row counts, and how the train/validation/test split was made. Provenance (where each input came from and how it was transformed) includes the feature pipeline: which script and commit built the features, since features are often rebuilt from a moving table.
4. Environment: Python version and a lockfile (an exact list of package versions), plus hardware where it matters.
5. Randomness: seeds (the number that starts a random generator, so shuffles and initial weights repeat) for every library used, and a note on operations that are not deterministic anyway (some GPU operations sum in a different order and vary in the last decimals).

Then the result (metric, baseline, tolerance, where the saved model or output lives).

Worked example (illustrative values)

Metric used below: recall@top10pct means rank customers by the model's predicted churn risk, flag the top 10%, and measure what share of the customers who actually churned were in that flagged group. 0.61 means 61% of real churners were caught.

experiment: churn-model-2026-09-14-run3
question: Does adding the 30-day support-ticket count improve churn recall on new customers?
code:
  repo: github.com/example-org/churn-model
  commit: 3f9a1c2
  uncommitted_changes: none
environment:
  python: "3.12.4"
  lockfile: requirements.lock (sha256 a41c...e07)
  hardware: 1x GPU, driver 550
data:
  snapshot: warehouse.churn_features@2026-09-01
  row_count: 184230
  query_or_pipeline: pipelines/build_features.py @ 3f9a1c2
  split: by customer_id hash, 80/10/10, seed 42
randomness:
  numpy_seed: 42
  model_seed: 42
  nondeterministic_ops: GPU sum order (results vary in 4th decimal)
result:
  metric: recall@top10pct
  value: 0.61
  baseline_value: 0.58
  artifact: s3://ml-artifacts/churn/run3/model.pkl
decision: Keep feature; re-check on the Q4 snapshot.

This record is a short file stored next to the run (or in an experiment tracker such as MLflow or Weights & Biases; either tool is optional, a plain YAML file committed next to the code already meets the bar). Six months later, checking out commit 3f9a1c2, installing the lockfile, and rebuilding the 2026-09-01 snapshot reproduces the run within the stated tolerance.

Documenting a reproducible investigation of a claimed segment-level failure

Suppose a stakeholder says "the model is bad for customers on the annual plan". Write the investigation so it can be re-run:

• Claim in one sentence, with who made it and when.
• Definition of the segment as code or a query (not "annual customers"), plus its size.
• Same-snapshot comparison: the model's metric on the segment versus everyone else, computed on a named snapshot, with a confidence interval (a range the true value plausibly lies in, wider for small samples) or at least the segment's row count, because a small segment gives noisy numbers.
• What was ruled out: label quality (are the churn labels for this segment correct, or do annual customers show as "not churned" only because their contract has not ended yet), leakage (a feature that accidentally contains the answer, or information from after the prediction date, so offline scores look better than reality), a train/serve preprocessing mismatch, and a change in the data feed.
• Verdict and next action, and the exact command to reproduce the table.

Filled-in segment example (illustrative)

• Claim: "the model is bad for annual-plan customers" (raised by the retention lead, 2026-09-20).
• Segment: plan_type = 'annual' on warehouse.churn_features@2026-09-01; 1,840 customers, of whom 220 churned.
• Result: the model flagged 108 of those 220 churners in its top 10%, so recall@top10pct = 108/220 = 0.49. Everyone else: 0.62.
• Uncertainty: with 220 churners, the rough 95% interval on 0.49 is about 0.42 to 0.56, which does not reach 0.62, so the gap is unlikely to be noise.
• Ruled out: labels re-checked on 50 sampled annual accounts (all correct); no leaked feature found; the same preprocessing function is used in serving.
• Verdict: real weakness. Next action: add annual-specific features and re-measure; reproduce the table with python analysis/segment_recall.py --snapshot 2026-09-01.

Trade-offs and pitfalls

• Notebooks are the usual culprit: cells run out of order and hold hidden state. Re-run top to bottom before saving, and export the final analysis as a script or clean notebook.
• Over-documenting kills the habit. Automate capture (log the commit, lockfile hash and seed from code) so the human writes only the question and the decision.
• "Reproducible" has levels: same numbers exactly, same numbers within a tolerance, or same conclusion. Say which one you promise.