Technical Writing and Documentation Questions

Direct answer. A whitepaper for three audiences works when it is layered: a self-contained front section for customers, a core architecture section for architects, and implementation detail for developers pushed into appendices and linked resources. Before publication it runs a claims audit, a technical review by the people who built the system, a security and legal check, and a copy edit, each with a named sign-off.

Terms. A hybrid-cloud architecture combines on-premises or private infrastructure with a public cloud. A whitepaper is a long-form, authoritative document that argues for an approach. An SME (subject matter expert) is the person who actually knows the area deeply.

More terms. Data residency means a legal or policy requirement that data stays in a specific country or region. A reference architecture is a proven blueprint others can copy. Trust and network boundaries are the lines on a diagram where one security zone ends and another begins. Shared responsibility spells out which security tasks the provider handles and which the customer handles. A context diagram shows the whole system as one box with the people and systems around it; a component diagram then opens the box. The benchmark method is how a performance test was set up, so others can repeat it.

Structure by audience

Each section opens with "who this is for and what you will get", so readers can skip safely.

What the one-page executive summary looks like (illustrative)

Run regulated workloads in the cloud without moving your data out of your data center. Companies in finance and healthcare often cannot move customer records off their own hardware, yet want cloud scale for peaks. This paper describes a hybrid design that keeps sensitive data on-premises and runs burst processing in the public cloud, connected by an encrypted link. It suits teams with an existing data center and a legal requirement to keep records in-country. It is a poor fit if you are starting from scratch with no on-premises systems. Next step: read the reference architecture (section 3) with your architect, or start the 30-minute walk-through in section 4.

It states the problem, the outcome, who it fits (and who it does not), and the next step in about 100 words, with no product jargon.

Editorial and peer-review steps, in order

1. Audience and message brief signed by the product owner, before writing. Decide the one claim the paper defends.
2. Outline review with two architects, cheap to change now.
3. Technical review by SMEs who did not write it, checking every diagram against the real system.
4. Claims audit. Every number, comparison and "supports X" statement gets a source or a reproducible method. Benchmarks state the setup. Anything unverifiable gets softened or cut.
5. Security and legal/compliance review, including customer names, trademark use and regulatory statements.
6. Developer walk-through: someone follows the implementation section on a clean environment.
7. Copy edit against a style guide: terminology consistency, defined acronyms, accessible diagrams with alt text.
8. Final gate: product owner and engineering lead sign off; the launch date does not override an open claims-audit finding.
9. After publication: a named owner, a version number, and a review date, so it can be corrected when the product changes.

If time is short: steps 3 (SME technical review), 4 (claims audit) and 5 (security and legal check) are never cut, because errors there hurt customers or the company. Steps 2, 6 and 7 can be compressed; step 9 is cheap and should not be skipped.

Pitfalls. Marketing claims that engineering has never seen; one voice trying to serve all three audiences in every paragraph; a public paper that reveals unreleased roadmap or internal topology. If the schedule slips, I cut scope (fewer appendices), never the claims audit.

Direct answer

I expand everything an engineer must build and test against: the user problem in concrete terms, acceptance criteria (the conditions that must be true for the work to count as done), constraints, edge cases (unusual or failure situations, such as a feed being down), interfaces, dependencies, non-goals (things this work explicitly will not do) and how success is measured. I keep short the parts that motivate but do not constrain implementation: market context, strategy narrative and long-range roadmap. To turn a business goal into a testable requirement, I state the goal, pick a measurable outcome, then write behaviour in observable Given/When/Then form (a fixed sentence pattern: the starting situation, the action, the visible result) with explicit thresholds.

Expand versus keep short

Turning one goal into a testable requirement

Steps: goal, outcome metric, behaviour, threshold, how it will be verified. In the bullets below, behaviour and threshold appear together in the requirement bullet, so five steps fill four bullets.

• Goal: fewer "where is my order?" support contacts.
• Outcome metric (defined precisely): support contacts tagged "late delivery" per 1,000 orders per week, measured from the support tool over the four weeks before and after launch. Illustrative numbers: before launch, 600 late-delivery contacts on 20,000 orders per week is 600 / 20,000 x 1,000 = 30 per 1,000 orders; four weeks after, 420 contacts on 21,000 orders is 420 / 21,000 x 1,000 = 20 per 1,000, a one-third fall. Using a rate per 1,000 orders keeps the comparison fair when order volume changes.
• Requirement in testable form (illustrative values):
  • Given an order whose estimated delivery time has passed, when the customer opens the order page, then the page shows "delayed" with a new estimated date.
  • Given an order delayed by one day or more, when the delay is detected, then the customer receives one email, and never more than one per order.
  • Given the carrier feed is unavailable, when the customer opens the page, then the page shows the last known status with its timestamp, not an error.
• Verification: each line maps to an automated or manual test an engineer can write without asking me.

The engineer version of the brief also lists which service owns delay detection and what is out of scope (no refunds logic in this release).

Pitfalls

• Requirements that restate the goal ("make delivery tracking better") cannot fail a test, so they are not requirements.
• Prescribing the solution when only the behaviour matters. Give engineers the problem and constraints, and let them choose the implementation.
• Vanity or unmeasurable metrics, and forgetting a guardrail (for example, do not raise contacts on other topics).
• Thresholds like "one day" are placeholders; agree the real values with engineering and support.

Direct answer

The summary asks for approval without giving a reviewer anything to evaluate: no measurable claim, no numbers, no plan, no decision requested. "Costs: unknown", "Rollout: TBD" and "Risks: some" are placeholders, not content. The most important flaw to fix first is that the goal and the decision are unmeasurable: with no baseline, target or stated request, no reviewer can judge anything else. A good rewrite states the decision needed, the problem with a baseline, the cost and timeline as ranges with a confidence level, a staged rollout, named risks with mitigations, and a fallback.

Critique, line by line

• "Move to service-oriented architecture (splitting one large application into independently deployable services) to improve deploy velocity": SOA is a means, not a goal. "Velocity" has no baseline, no target and no measure, so nobody can later say whether it worked.
• "Costs: unknown": unknown is only acceptable with a plan to find out and bounds on the range. Reviewers cannot approve an open cheque.
• "Rollout: TBD": a migration without sequence or rollback is a risk in itself.
• "Risks: some": names none, so it reads as if none were examined.
• Missing altogether: the decision being requested, the alternatives considered (for example a modular monolith, where code is separated inside one deployable unit), who is affected and by when.

Rewrite (all figures illustrative; replace with measured values)

"Decision requested: approve a six-month pilot extracting (moving out of the main application) the billing and notification modules into separate services, with a go/no-go review (a scheduled meeting where the team decides to continue or stop based on agreed evidence) at month three.

Problem: today every change ships in one weekly release train (a fixed schedule on which all finished changes ship together), and median time from merge to production is about five days (baseline to be confirmed from deploy logs). Two teams have blocked each other three times this quarter waiting on a shared release.

Goal: billing and notifications deploy independently, target median merge-to-production under one day for those modules, with change failure rate (share of deployments causing an incident or rollback) no worse than today.

Cost: estimated ten to fourteen engineer-weeks for the two extractions, plus ongoing operations work for two more services. Confidence: medium; a spike (a short, time-boxed investigation to answer one question) of two weeks resolves the main uncertainty (shared database tables) before commitment.

Rollout: extract billing first behind a feature flag (a switch that turns new code on or off without redeploying), run old and new paths side by side for two weeks, cut over by traffic percentage (send 5%, then 25%, then 100% of requests to the new path), keep the old path for one release as rollback.

Risks and mitigations: distributed failures (add timeouts and retries), harder debugging (shared request IDs in logs), data ownership disputes (billing owns its tables before extraction).

Alternative considered: modular monolith, cheaper but does not remove release coupling between teams."

Pitfalls

Do not invent precision to fill the gaps. Where a value truly is unknown, write what is unknown, the owner, and the date it will be known. A reviewer can act on that; they cannot act on "TBD".

Direct answer

Treat "people misread our docs" as a defect with causes, not a talent problem. I would propose a standard with four layers: a fixed page shape with required metadata, a short list of wording rules a machine can check, snippets that are executed rather than trusted, and a cold-read test for the few pages where a misreading is expensive. All of it runs in CI (continuous integration, the automatic checks that run on every pull request, meaning a proposed change awaiting review), so quality does not depend on one heroic editor.

1. Start from the misreads, not from a style guide

Pull the last quarter's production bugs that trace back to documentation and sort them by cause: a number without a unit, a missing precondition, a stale version, a snippet that no longer runs, two names for one thing. The standard only needs to attack the causes you actually see. Example: a Kafka topic (a named stream of messages in Kafka, a system that moves event data between programs) setting retention.ms (how long messages are kept) is in milliseconds, and a page saying "set retention to 7 days" led someone to type 7.

2. The standard: rule, then how it is checked

The table has five rows because the first layer, a fixed page shape with required metadata, is split into two checks: one for the heading layout and one for the front matter. The other three layers are one row each, which gives the four layers named above.

Readability scores such as Flesch (a formula that rates how easy text is to read from sentence and word length) measure sentence length, not correctness, so use them only as a weak warning. The "7 days" page would score well and still cause the bug. The fix is Set retention.ms to 604800000 (7 days = 7 x 24 x 60 x 60 x 1000 ms).

3. One convention across a mixed stack (Spark, an engine for processing large data sets; Kafka; Airflow; dbt)

Do not force each tool's native docs into one tool. Airflow (a workflow scheduler), dbt (a tool that builds SQL transformations) and the rest keep their own generated reference. The standard sits above them as a page-level contract: the same format (Markdown), the same required metadata, the same snippet conventions (tagged, pinned, runnable, with placeholders in UPPER_SNAKE_CASE), and a rule that hand-written pages link to generated reference instead of copying it.

4. No heroics: enforce by ratchet

A ratchet is a rule that only tightens: it never lets quality slip back, but it does not demand a big-bang cleanup either. Warnings first, then errors, and only on files a PR touches, so nobody must fix the whole backlog before merging. Pages older than a set age (say 180 days since last_verified) open an automatic ticket to the owner.

Worked example

REQUIRED = {"title", "owner", "audience", "last_verified", "applies_to"}

def parse_front_matter(text):
    lines = text.splitlines()
    if not lines or lines[0] != "---":
        return {}
    fields = {}
    for line in lines[1:]:
        if line == "---":
            break
        key, _, value = line.partition(":")
        fields[key.strip()] = value.strip()
    return fields

def check(name, text):
    missing = sorted(REQUIRED - parse_front_matter(text).keys())
    return f"{name}: FAIL missing {', '.join(missing)}" if missing else f"{name}: ok"

good = "---\ntitle: Backfill a dbt model\nowner: data-platform\naudience: data engineers\nlast_verified: 2026-09-01\napplies_to: dbt 1.8\n---\nBody"
bad = "---\ntitle: Restart a Kafka consumer\nowner: streaming\n---\nBody"
print(check("dbt-backfill.md", good))
print(check("kafka-restart.md", bad))

This prints dbt-backfill.md: ok and kafka-restart.md: FAIL missing applies_to, audience, last_verified. The Kafka page cannot merge until its owner says which audience and version it applies to, which is exactly the information that was missing when someone followed it for the wrong version.

How the script works: partition(":") splits each header line at the first colon into a key and a value. REQUIRED - keys is a set difference, meaning the required fields that are not present. sorted keeps the message in a stable order.

Trade-offs and pitfalls

• Rules that are too strict make people avoid writing docs. Start with the three causes that hurt most.
• Metadata checks prove presence, not truth. A date can be updated without re-verifying, which is why the cold read exists for high-risk pages.
• Measure it: count postmortems (written reviews of an incident after it is resolved) tagged "doc-caused" per quarter and the share of executed snippets that pass. Falling counts are the proof, style-guide adoption is not.
• What would change my call: a group with only a handful of pages should skip the linter and keep the cold read plus snippet execution.

Approach

Python is the better choice over Bash here, because correct heading extraction needs a small amount of state (are we inside a fenced code block, a code sample wrapped in triple backticks?) plus Unicode-safe file reading (it copes with non-English characters such as é or 日本語 instead of crashing).

Core versus optional. The core ideas are the fence flag, the sorted walk and the duplicate-anchor counter. Setext headings, front matter and the complexity note are extras you can add later. Plan:

1. Walk the repo, keep every .md file, skip .git, node_modules, virtual environments and hidden directories, and skip the output file itself so re-runs do not index their own output.
2. Sort paths so the output is deterministic (the same input always gives the same output) and diff-friendly (a re-run changes only lines that truly changed, so version control shows a small diff).
3. For each file, read line by line, toggle a "in fenced code" flag on lines that start with three backticks or three tildes (so # comment lines inside code are not mistaken for headings), and match ATX headings (the #-prefixed kind: # Title, ## Subtitle, ### Section).
4. Print directories as nested bold entries, each file as a link, and each heading as a nested link using a GitHub-style anchor (an anchor is the #section-name part of a link that jumps to a heading). Duplicate headings in one file get -1, -2 suffixes, as GitHub does. A slug is the link-friendly form of a heading, for example Getting Started becomes getting-started.

Code

#!/usr/bin/env python3
"""Build TOC.md from the Markdown headings (levels 1-3) of every .md file under a repo."""
import re
import sys
from pathlib import Path

SKIP_DIRS = {".git", "node_modules", ".venv", "venv", "__pycache__"}
FENCE = re.compile(r"^\s{0,3}(`{3}|~{3})")
HEADING = re.compile(r"^ {0,3}(#{1,3})\s+(.+?)\s*#*\s*$")


def slugify(text):
    """GitHub-style anchor: lowercase, drop punctuation, spaces to hyphens."""
    text = re.sub(r"[`*_]", "", text.strip().lower())
    text = re.sub(r"[^\w\s-]", "", text)
    return re.sub(r"\s", "-", text)


def headings(path):
    """Yield (level, title, anchor) for one file, skipping fenced code blocks."""
    seen, fence = {}, None
    for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
        m = FENCE.match(line)
        if m:
            fence = None if fence == m.group(1) else (fence or m.group(1))
            continue
        if fence:
            continue
        h = HEADING.match(line)
        if h:
            title = h.group(2)
            base = slugify(title)
            n = seen.get(base, 0)
            seen[base] = n + 1
            yield len(h.group(1)), title, base if n == 0 else f"{base}-{n}"


def build(root, out_name="TOC.md"):
    root = Path(root)
    files = sorted(
        p for p in root.rglob("*.md")
        if p.name != out_name
        and not any(part in SKIP_DIRS or part.startswith(".") for part in p.relative_to(root).parts[:-1])
    )
    lines, shown_dirs = ["# Table of contents", ""], set()
    for f in files:
        rel = f.relative_to(root)
        for depth, d in enumerate(rel.parents[::-1][1:]):   # each ancestor dir, top-down
            if d not in shown_dirs:
                shown_dirs.add(d)
                lines.append("  " * depth + f"- **{d.name}/**")
        base = "  " * (len(rel.parts) - 1)
        lines.append(f"{base}- [{rel.name}]({rel.as_posix()})")
        stack = []   # open ancestor levels, so a skipped level (h1 then h3) still nests one step
        for level, title, anchor in headings(f):
            while stack and stack[-1] >= level:
                stack.pop()
            stack.append(level)
            lines.append(f"{base}{'  ' * len(stack)}- [{title}]({rel.as_posix()}#{anchor})")
    (root / out_name).write_text("\n".join(lines) + "\n", encoding="utf-8")
    return len(files)


if __name__ == "__main__":
    n = build(sys.argv[1] if len(sys.argv) > 1 else ".")
    print(f"indexed {n} files")

Worked example (run it exactly as shown)

mkdir -p demo/docs/api demo/.git
printf '# Acme\n\n## Install\nRun it.\n' > demo/README.md
printf '# Guide\n\n## Setup\n~~~bash\n# not a heading\n~~~\n\n## Setup\n\n#### Too deep\n' > demo/docs/guide.md
printf '# Auth\n### Tokens ###\n' > demo/docs/api/auth.md
echo '# hidden' > demo/.git/x.md
python3 make_toc.py demo && cat demo/TOC.md

Output:

indexed 3 files
# Table of contents

- [README.md](README.md)
  - [Acme](README.md#acme)
    - [Install](README.md#install)
- **docs/**
  - **api/**
    - [auth.md](docs/api/auth.md)
      - [Auth](docs/api/auth.md#auth)
        - [Tokens](docs/api/auth.md#tokens)
  - [guide.md](docs/guide.md)
    - [Guide](docs/guide.md#guide)
      - [Setup](docs/guide.md#setup)
      - [Setup](docs/guide.md#setup-1)

Note that # not a heading inside the code fence and #### Too deep (level 4) are both absent, the second "Setup" got the -1 anchor, the trailing ### in ### Tokens ### was stripped, and the .git/x.md file was ignored.

Reading the two regular expressions

FENCE:

^\s{0,3}(```|~~~)

• ^ start of the line; \s{0,3} up to three spaces or tabs of indentation.
• The bracketed group matches either three backticks or three tildes, captured as group 1. The code remembers which one opened the block and only a matching one closes it.

HEADING:

^ {0,3}(#{1,3})\s+(.+?)\s*#*\s*$

• ^ {0,3} start of line, up to three spaces.
• (#{1,3}) one to three # characters, captured: the count is the heading level. Four hashes fail to match, which is why #### Too deep is dropped.
• \s+ at least one space, so #NoSpace is not a heading.
• (.+?) the title, captured lazily (as short as possible) so it does not swallow the closing hashes.
• \s*#*\s*$ optional spaces, optional closing hashes as in ### Tokens ###, then the end of the line.

Key points

• rel.parents[::-1][1:] gives each ancestor directory top-down (dropping the empty root), so each directory is printed once, preserving structure.
• The stack makes nesting follow the heading hierarchy even if a document jumps from # to ###.
• Links are relative to the repo root where TOC.md is written, so they work on GitHub.

Complexity

Big-O notation describes how work grows with input size. Time is O(total size of all Markdown files) to read them, plus O(F log F) to sort F file paths (F is the number of files; sorting a list of F items takes a bit more than F steps). Memory is O(F + H) for the sorted path list and the output lines (H is total headings), and the seen-anchor dictionary is per file.

Edge cases and limits

• Setext headings (a title underlined with === or ---) are not handled. Add a regex for the line above such underlines if the repo uses them.
• Anchor slugs follow GitHub's rules only approximately (for example, non-ASCII titles and some punctuation differ across renderers). Verify on your docs platform.
• Headings inside HTML blocks or front matter (a YAML block at the top of a file) are ignored or could be misread. Skip a leading --- block if you use it.
• Unreadable or non-UTF-8 files are read with errors="replace" so one bad file does not abort the run.