Documentation and Knowledge Management Questions

Direct answer

Capture tacit knowledge (the know-how people cannot easily write down) by interviewing for it rather than asking seniors to "document everything". Turn each recording into a short written artifact that is searchable, put it into a small set of typed formats, organize it by service and task with owner and verified-date metadata, and keep it current by attaching maintenance to events that already happen: code changes, incidents and onboarding.

Terms

• Runbook: step-by-step instructions for an operational task.
• Postmortem: a written review after an incident covering what happened, why, and what will change.
• README: the front-page file of a code repository that explains what it is and how to use it.
• Shard by tenant: split one large database into separate pieces using the customer (tenant) as the partitioning key, so each customer's data lives together on one piece (a piece may hold one large tenant or many small ones).
• Shadow and pair: watch an expert do real work (shadow), or work alongside them and take turns at the keyboard (pair).
• Distillation step: boiling a long recording down into a short written summary.
• Scribe: the person who writes things down while others talk.
• Timestamped link: a link to an exact moment in a recording, such as minute 12:40.
• Tagging with a service: labelling a page with the service it concerns (here, ingest, the service that loads data) so search can filter by it.

Why recordings alone fail. A one-hour recording has no search, no summary and no owner, and the interesting three minutes are buried. It is raw material, not the product.

1. Capture: pull the knowledge out

• Scenario interviews. Instead of "explain the system", ask "walk me through the last time this broke. What did you check first? What did you rule out and how?" This surfaces judgment, the part that lives in heads.
• Shadow and pair on real work (an incident, a deploy) with a junior writing notes as they go.
• Recorded sessions get a distillation step: auto-transcribe, then a person writes a one-page summary with timestamped links into the video. The summary is what search indexes.
• New-hire read-back: a new hire tries to follow the note; every question they ask becomes an edit. This is the test of whether the knowledge really transferred.

2. Formats: one format per kind of knowledge

3. Organize for findability

• Structure by service and task, not by who wrote it or when.
• Required metadata on every page: owner, backup owner, service, audience, type, last-verified date.
• Linked from where work happens: the service README, the alert, the deploy tool.
• Search tuned with synonyms for the team's own vocabulary.

4. Keep it current

• Event triggers: a pull request that touches a service's configuration prompts "does a doc need updating?"; a postmortem's action list must include the doc fix before it can be closed; every new hire reports what was wrong.
• Calendar trigger: a quarterly verify pass, limited to high-use pages (found from view counts), so the effort follows value.
• Retirement: unowned pages with no views for a year get archived.

Worked example. A senior knows why the nightly data load must never be re-run after 06:00. Step one: a 20-minute interview using "the last time someone re-ran it". Step two: a runbook page "Re-running the nightly load" with the rule, the reason (it would double count that day's orders), the safe alternative and a link to the timestamp in the session recording. Step three: tagged with service ingest, owner, backup, last-verified date. Step four: a new hire follows it in week two and corrects an outdated table name; the date is refreshed.

The finished page looks like this:

Title:    Re-running the nightly load
Type:     runbook     Service: ingest
Owner:    data-platform team   Backup: analytics-eng   Last verified: 2026-09-28
Rule:     Never re-run the load after 06:00 UTC.
Why:      A second run adds that day's orders on top of the first, so they are counted twice.
Safe alternative: use the replace-day option, which deletes the day's rows before loading.
Recording: 12:40 "why the double count happens" (link)

Trade-offs and pitfalls

• Time is the seniors' scarce resource: budget 20 to 30 minutes per topic and let a scribe or a junior write it up.
• Auto-generated summaries need a human check, or they confidently record wrong steps.
• Incentives matter: recognize documentation in performance reviews, otherwise the work is invisible.
• What would change my call: if a topic changes weekly, prefer tests and automation over prose that will rot.

Direct answer

A strong answer tells a specific story: a real reproducibility pain (someone could not rerun a result), the concrete practices you introduced, how you made adoption easy rather than mandatory theatre, and a measure you can state honestly (for example how many past results could be rebuilt from the repo before and after). Below is the shape, followed by a sample story to adapt. The details are illustrative, so use your own real project.

Structured elaboration (STAR, with the detail interviewers look for)

STAR is the standard behavioral-interview format: Situation, Task, Action, Result.

• Situation: a model or analysis whose result could not be reproduced, or a new joiner who lost days rebuilding context. Reproducibility means another person can rerun the work from the repository and get the same result.
• Task: your role in fixing it, and the constraint (no one had time for heavy process).
• Action, what changed: the core is to pin everything a result depends on: the code (Git), the data (dated snapshots, meaning copies that are never edited after saving, or a data version control tool such as DVC), the environment (a lock file, which records the exact version of every library, or a container image, a packaged copy of the whole software setup), the randomness (fixed seeds, meaning setting the random number generator's starting value so runs repeat) and the experiment record (a tracker such as MLflow that stores parameters, metrics and code commit). Named tools like DVC and MLflow are optional examples; the practices are what matter. Document in a README (how to run) and a model card (a short document stating a model's purpose, training data, limits and owner).
• Action, adoption: a project template that already contains these files, a CI (continuous integration) job that runs the pipeline end to end on a small sample, a pull request checklist, a short demo, and pairing with the first two teammates.
• Result: a before/after measure you can defend, plus the behaviour change.

For instance, before the change: "the churn notebook could not be rerun"; after: "a teammate reran it from the README in one sitting".

Worked example

Situation. A churn model owned by a colleague who left could not be retrained. The notebook depended on a table that had changed, and no one knew the library versions used.

Task. As the ML engineer picking it up, I was asked to make it maintainable and to stop it happening to the next model.

Action. I rebuilt the model from the repo history and recorded the gaps: no data version, no environment file, seeds not set. For that one model I added a lock file, dated data snapshots, fixed seeds, an MLflow run per training and a README with a "reproduce in five commands" section, plus a model card. Then I made it cheap for others: I put all of this in a project template, so a new project starts compliant. A CI job retrains on a 1% sample and compares the metric to a stored value within a tolerance, so drift in the environment is caught. I demoed it in the team meeting and paired with two colleagues to port their projects. I did not make it mandatory at first. After it caught a broken dependency upgrade in CI, the team lead made the template the default.

Result (illustrative, adapt to yours). I audited the last five models before the change: two could be rebuilt from the repo alone. After porting, all five could, verified by a teammate running the README commands cold. Onboarding (getting a new teammate productive) onto a model dropped from "several days of asking around" to a first successful rerun on day one. I would state it in that plain form: counts I actually checked, not a percentage I cannot back up.

Trade-offs and pitfalls

• Do not claim precision you did not measure. "Five of five reproduced by a teammate from scratch" is stronger than an invented improvement figure.
• Tools alone do not fix documentation. The template and CI check are what change habits.
• Bit-for-bit reproducibility can be impossible because GPU nondeterminism (parallel hardware adding numbers in a varying order) makes reruns differ slightly. Say what tolerance you define as "reproduced", for example a metric within 0.5 points of the stored value.
• Weak answers name tools but not the resistance you met. Say who pushed back and what changed their mind.

Direct answer

Set a small, tiered minimum standard (a few artifacts every service, pipeline or model must have), keep the docs in the same repository as the code and change them in the same pull request (PR), and enforce the standard with automation and defaults rather than meetings and sign-offs. Freshness comes from ownership plus a review date on every doc, not from goodwill. The test of "not bureaucracy" is that following the standard is the easiest path and the checks are run by tooling.

Structured elaboration

1. The standard itself: small and tiered

• Tier 1 (anything on-call, customer-facing or shared): a README (the front-page file of a repo: what it is, who owns it, how to run it, where to ask), a runbook (step-by-step instructions for responding to an alert or failure), and a decision record (a short note recording what was chosen, the options rejected and why) for hard-to-reverse choices.
• Tier 2 (internal tools, experiments): README only.
• ML and data assets: a model card (short document stating what a model is for, its training data, known limits and owner) or a dataset datasheet, plus a data dictionary (what each column means, its unit and source). For analytics projects add required metadata: owner, refresh schedule, source tables and last-reviewed date.
• One template per artifact, pre-filled, with the required fields marked. Templates are what stop inconsistent deliverables and the rework that follows.
• The ML and data items are optional add-ons for teams that ship models or datasets, not part of the core README, runbook and decision record set.

A minimal template skeleton, so the required fields are concrete (the CI check simply fails if a heading is missing):

README.md            RUNBOOK.md                 decision record
# <service>          # Alert: <alert name>      # Decision: <statement>
Owner: <team>        Severity: <P1-P3>          Status: proposed | accepted | superseded
Last reviewed: <d>   Symptoms: <what you see>   Context / Options / Consequences
## What it does     Diagnosis: <steps>          Owner and review date
## Run locally      Fix or rollback: <steps>
## Ask for help     Escalate to: <who>

2. Governance as one framework (owners, templates, review cadence, tooling, incentives)

• Templates: the pre-filled skeleton above is the only format accepted for each artifact, stored in one template repository, so the CI check has a fixed set of required headings to test.
• Owners: every doc names a team owner (a CODEOWNERS file, which routes PR review to the named owners, plus a service catalog entry, meaning a central list of every service with its owner, docs link and on-call contact). An unowned doc is treated as deletable.
• Review cadence: each doc carries a last_reviewed date. Tier 1 is reviewed at least every 6 months, and also whenever the system changes shape.
• Tooling: docs-as-code (markdown in the repo, reviewed like code), a CI (continuous integration) job that checks required headings, broken links and stale dates, and one searchable site built from all repos.
• Incentives: doc work counts in performance and promotion criteria, authors are credited by name, and "a new hire followed the README and it worked" is celebrated. Punishment alone breeds checkbox docs.

3. Keeping standards current as work changes

• A PR template asks "docs updated or not needed, and why". Changes to alerts, APIs or schemas require touching the linked doc (a CI rule can map code paths to doc paths).
• A bot opens a ticket on the owner when last_reviewed passes its limit. Two missed cycles moves the doc to archived rather than leaving a confident but wrong page.
• The standard itself is versioned and reviewed twice a year by a small docs working group (a few volunteers from different teams who own the standard), and any rule that nobody can justify with an incident or onboarding pain is removed.

4. Measuring compliance and handling non-compliance

• Coverage = Tier 1 assets with all required docs / all Tier 1 assets. Freshness = docs reviewed inside their window / docs required.
• Escalation ladder: bot nudge, ticket on the owner, visible on the team's monthly ops review, and only for Tier 1 a launch-readiness blocker, meaning the service cannot go live until its Tier 1 docs exist. Nothing else is gated.

Worked example

One team owns 15 Tier 1 services. The dashboard shows 12 have a README and runbook, and 9 of those are inside the 180-day review window.

coverage  = 12 / 15 = 80%
freshness =  9 / 15 = 60%   (9 of the 15 required sets are fresh)

The team lead sees two numbers and six named services to fix (three with no docs, three with docs past their review window), not a lecture. The runbook and decision record carry the same Last reviewed field as the README, so the CI stale-date check and the 6-month cadence apply to every Tier 1 artifact, not only the README. Note freshness is computed against the required 15, so a missing doc cannot hide behind a fresh one.

Rollout at scale (500 engineers, ten teams, two regions), phased over 12 months

• Months 1-2: pilot with two willing teams, build templates and the CI check, fix what the pilot hates.
• Months 3-5: add four teams, stand up the single search site and the service catalog, run a 2-hour training per engineer (500 x 2 hours = 1,000 engineer-hours, which is 25 engineer-weeks at 40 hours, a small fraction of the organization's yearly capacity).
• Months 6-9: remaining four teams; migrate only Tier 1 legacy docs (archive the rest, do not migrate everything).
• Months 10-12: switch on the launch-readiness gate for Tier 1, publish the first compliance report, review the standard.
• Rough cost, in people rather than invented dollars: a docs platform owner and a tooling engineer (about 2 full-time), plus one steward per team (the team's named docs champion who helps peers and chases reviews) at about 10% time (ten teams is about 1 full-time equivalent). Two regions: one site, async PR review so no time zone waits on another, and one regional steward each.

Trade-offs and pitfalls

• Too many required docs produces empty template filling. Cut the list until each item has a story of an incident or onboarding delay it prevents.
• Central mandate without team stewards makes the standard feel imposed. Stewards give teams a peer to ask.
• Gating everything on compliance encourages gaming (docs written to satisfy a linter). Gate only Tier 1 and spot-check quality by having a new joiner try the README.
• What would change my approach: for a 15-person team I would skip the catalog and the phases and adopt README, runbook and a PR checkbox in one week.

Direct answer

An assumption log is a living list of the things a project is treating as true without having proved them. A useful entry is specific, falsifiable (written so a test could prove it wrong), owned, backed by stated evidence, and has a date by which it must be validated or dropped. It differs from a risk log (possible future events) and a decision log (choices already made). You update it continuously, in the routines the team already runs, and at every milestone.

Structured elaboration

What makes an entry useful

• Statement: concrete and testable. Bad: "users want reports". Good: "at least half of active admins export a report weekly".
• Owner: the one person who will chase the answer.
• Evidence and confidence: what we know now (source, sample size) and how sure we are (high / medium / low).
• Impact if wrong: what breaks, so we test the expensive ones first. For a data analyst: "if exports are not CSV, the ingestion script must be rewritten, about two weeks of rework". For a business analyst: "if the sponsor has not approved the budget, the scope in the requirements document is void".
• Validation method and date (expiry): how we will test it and when it must be settled. An untested assumption without a date is just an opinion.
• Status: open, validated, invalidated, retired. Plus a link to the decisions that depend on it.

How it differs from a risk log and a decision log

They connect: an assumption is an input to a decision, and an invalidated assumption usually becomes a risk or an issue.

When to update

• In daily scrum (the short daily team check-in), ask one question: did anything block or change an assumption?
• In sprint planning (the meeting where the team picks the work for the next one to two weeks), scan open assumptions and pull validation work in as tickets.
• When new evidence arrives, and at milestones (agreed checkpoints such as "design approved") or phase gates (formal go/no-go reviews between project stages). Surfacing it through the project lifecycle reduces ambiguity, because people stop building on unstated beliefs.
• On the expiry date, when the owner must validate, extend with a reason, or invalidate.

After a discovery call (an early meeting with a customer or stakeholder to learn their needs)
Capture assumptions and open questions the same day. For each, assign an owner (the person best placed to find the answer, not the note taker) and a date. Flag any item that touches contracts, data privacy or regulation for legal review, and any item that would change scope for escalation to the sponsor (the executive who funds the project and must approve scope changes). Example: "the customer assumes we will migrate 10 years of history" is escalated because it could double the effort.

Worked example

If A1 is invalidated (a customer sends 2 GB files), it becomes a risk entry ("upload pipeline may fail on large files") and triggers a decision record about chunked uploads.

Trade-offs and pitfalls

• Vague entries ("system will be fast") cannot be tested. Rewrite until a test exists.
• Logs die when nobody owns them: name one owner for the log itself and review it in planning.
• Do not merge it into the risk log. Mixing them loses the "how do we validate this?" discipline.

Direct answer

A decision record (in software often called an ADR, architecture decision record) is a short note written when a decision is made. It should capture the context, the options considered, the choice, the reasoning and the consequences, so a person six months later can see why without asking. Not every decision needs one: write records for decisions that are hard to reverse, costly, cross-team, surprising or likely to be asked about again. Keep them next to the work (in the repository for engineering choices, linked from the ticket or roadmap for product choices) with one index page.

Structured elaboration

What to capture (one page is plenty)

• Title and date, and status (proposed, accepted, superseded by record N).
• Decision owner and who was consulted.
• Context: the problem, the constraints and the deadline pressure at the time.
• Options considered, with the reason each was rejected. This is the part that stops re-debate, because people who propose the rejected option can see it was already weighed.
• Decision in one sentence.
• Consequences: what gets easier, what gets harder, and what we accept.
• Revisit trigger: the condition that would reopen it ("if traffic exceeds X" or "at the next vendor renewal").

Which decisions deserve one
A quick test: would a new team member reasonably ask "why did we do it this way?" in six months, and would the answer be lost if the two people who decided left? Yes means write it. Typical yes: choosing a database (the reasons live in a few heads and are costly to redo), an API style such as REST versus GraphQL (every client depends on it), build vs buy (the cost, lock-in and staffing reasoning is rarely written elsewhere), a data retention policy (the legal reasoning matters when someone later wants to delete or keep data), a metric definition (dashboards and targets hang on it). Typical no: variable naming, a reversible feature flag default, an option you can change in an afternoon.

Where to keep them

• Engineering decisions: a docs/decisions/ folder in the repository, numbered, reviewed in a pull request (PR). They travel with the code and are found by anyone who opens the repo.
• Product or cross-team decisions: a decision-log space in the wiki, linked from the ticket and the roadmap item.
• One index (a table of number, title, status, date) so people can scan.

Worked example

Decision record 007: Store order data in PostgreSQL, not DynamoDB
Status:        Accepted (2026-03-04)
Owner:         Backend lead. Consulted: data team, finance.
Context:       Finance needs ad hoc joins across orders and refunds.
               Team has strong SQL experience, no DynamoDB experience.
Options:       (a) PostgreSQL. (b) DynamoDB: rejected, ad hoc joins would need extra pipelines.
Decision:      Use PostgreSQL with read replicas.
Consequences:  Easier reporting. Harder to scale writes past one primary.
Revisit if:    Sustained write load outgrows a single primary.

In plain words: PostgreSQL is a relational database queried with SQL, good at joins across tables. DynamoDB is a managed key-value database, fast for simple lookups but awkward for ad hoc joins. A read replica is an extra copy of the database that only serves read queries, which spreads reporting load. A single primary is the one main copy that accepts all writes, so if writes outgrow one machine, the design is harder to scale.

When someone asks in July "why not DynamoDB?", the answer is one link, and the debate is only reopened if the revisit trigger has actually fired.

Trade-offs and pitfalls

• Records that are too long are never written. Cap at one page and accept rough drafts.
• Do not edit an old record to change its meaning. Write a new one that supersedes it.
• Writing records after the fact from memory loses the rejected options, so write them as part of making the decision.
• A record folder nobody links to is invisible: link to it from the README (the repo's front-page file) and the PR template.