Documentation and Knowledge Management Questions

Direct answer

Run a time-boxed remediation program aimed at the runbooks that pages and incidents actually touch. Clean up tier-1 procedures first, detect staleness automatically with checks against real infrastructure and incident history, require a human to walk each procedure before it is marked verified, and stop recurrence by making a working runbook a condition for shipping an alert and for closing a postmortem. Archive everything nobody claims instead of reviewing it.

Terms. A playbook or runbook is a step-by-step procedure for handling an operational event. A postmortem is a written review of an incident; blameless means it looks for system causes, not someone to blame. A tier-1 service is one whose failure pages people or hurts revenue; tier 2 is important but a failure degrades it rather than stopping it (for example an internal reporting tool). A paging alert is an alert that wakes someone up. A staging environment is a safe copy of production for testing. The inventory is the current list of live hosts and services, taken from your configuration database. A decommissioned host is one that has been shut down and removed.

1. What to clean up first

Rank runbooks by consequence, not by age:

1. Runbooks linked from paging alerts on tier-1 services.
2. Runbooks named in the last 12 months of postmortems as missing, wrong or ignored (those are the ones that already repeated an incident).
3. Runbooks for tier-2 services.
4. Everything else: put an archive notice on it and delete or archive after 30 days if no owner claims it.

2. Automatic staleness detection

• Nightly job: links, hostnames, service names and commands referenced in the runbook checked against the current inventory. A reference to a decommissioned host is a hard flag.
• Alert-to-runbook check: every paging alert's runbook URL must resolve and must have a verified date inside its interval.
• Incident hook: responders press "runbook was wrong or missing" during or after an incident, which creates a ticket automatically.
• What the nightly check produces, for one runbook: db-failover step 4 references host db-eu-2 (decommissioned 2026-06-30): HARD FLAG. Ticket opened for payments-team, due in 5 business days.
• Age since last verified, used as a weak signal only (age alone is a poor predictor; broken references and incident flags are stronger).

3. Human review

The owner walks the runbook step by step in a staging environment, or as a tabletop drill (a talked-through rehearsal), with a second person who did not write it. They fix wrong steps, record the date and both names, and only then is the page marked verified. Reading it and clicking "still accurate" does not count.

4. Preventing recurrence

• CI (continuous integration) lint: an automated check that runs on every proposed change and rejects it when a rule is broken, here: an alert definition cannot merge without a runbook link. Illustrative output of a small custom lint script (no standard tool prints exactly this; you define the wording) when it fails:

FAIL alerts/checkout.yaml: "CheckoutLatencyHigh" has severity=page but no runbook_url
FAIL alerts/search.yaml: runbook_url https://wiki.example.com/runbooks/search-old returns 404

• Postmortem action items include "update the runbook" and the postmortem cannot close until the change is merged.
• Runbook lives next to the service code and its owner appears in the repo's CODEOWNERS file (a file that maps paths to owning teams), so changes get reviewed.
• Quarterly drill of the tier-1 runbooks (a game day, a rehearsal where you deliberately trigger a failure).

5. Realistic timeline and staffing

Illustrative sizing: 400 runbooks, of which 40 are tier 1 and 100 are tier 2.

Staffing: one part-time program lead, one engineer for tooling, and a small time budget from each owning team. The human review effort totals about 160 hours (60 for tier 1 plus 100 for tier 2), which is roughly 4 person-weeks spread across teams, not one team's burden.

Success measures. No paging alert without a verified runbook; postmortems citing "runbook wrong" decline quarter over quarter; nightly reference-check failures trend to zero.

Pitfalls

• Reviewing all 400 equally wastes the program's budget on the long tail.
• A one-time cleanup decays; without the CI and postmortem gates you are back here in a year.
• What would change my call: if the team is tiny, skip the tooling and run a monthly tier-1 drill instead.

Direct answer

A decision log becomes searchable and trusted when it has structured metadata, one home, a visible lifecycle, and evidence links that do not rot. I would store each decision as a short record with fixed fields in one repository, index it for search, link each record to its ticket and its evidence at a fixed version, mark superseded decisions instead of deleting them, and run small automated checks so trust does not depend on a curator. At hundreds of entries the risk is not volume, it is stale and unfindable entries.

Structured elaboration

1. One record shape, machine-readable
Every record begins with front matter (a block of structured fields at the top of a markdown file):

id: DEC-0142
title: Use daily partitions for the events table
status: accepted          # proposed | accepted | superseded | deprecated
date: 2026-05-12
owner: data-platform
domain: [warehouse, cost]
systems: [events-table, airflow]
supersedes: DEC-0107
ticket: PLAT-2291
evidence:
  - https://example.com/notebooks/partition-test@a1b2c3d
review_by: 2027-05-12
last_confirmed: 2026-05-12   # date the owner last reconfirmed it
access: internal          # internal | restricted

Free text below the front matter covers context, options, decision and consequences.

What the fields mean: status is the lifecycle stage (superseded means replaced by a newer decision, deprecated means still in place but being phased out); supersedes names the older record this one replaces; domain is the topic area and systems the things it touches, both used for filtering; evidence links to proof; review_by is when the owner must reconfirm and last_confirmed is when they last did. In the evidence link, @a1b2c3d is a commit hash (a fixed ID for one exact saved version of the repository), so the link always shows the notebook as it was when the decision was made. The example is about Airflow (a workflow scheduler that runs data jobs) loading an events table (a table of user actions) split into daily partitions (one chunk of data per day, so queries read only the days they need).

2. Findability

• One place: the repo (or a catalog), not tickets, notebooks and chat. Intake is a ticket template that creates the record skeleton, so decisions made in a thread still land in the log.
• Search over full text plus facets (filters) on domain, system, status and date. A controlled vocabulary (an approved list of tags people must choose from) avoids five spellings of one thing.
• A generated index page and per-system pages, so an engineer looking at the events table sees every decision that touches it.

3. Trust

• Status and supersession: a new decision links to what it replaces. The old record stays, with a banner, so history is intact and no one acts on a replaced decision.
• Review-by dates feed an automatic "needs review" flag. last_confirmed shows when the owner last stood behind it.
• Link checking in CI (continuous integration, automated checks that run on every change) catches dead links.

4. Evidence and ticket linking
Link experiment artifacts, metrics and notebooks with a permalink (a link that always points to the same content) to a commit, run ID (the unique ID a tracking tool gives one experiment run) or dated snapshot, never a moving branch (a branch's content changes as people commit). Add a time-box, a limit on how long the evidence stays valid ("re-test if data volume doubles"). Decisions link back to the ticket, and the ticket links to the record, so either entry point works. Where monitoring depends on a decision's assumption, link the alert or dashboard so a violation is visible.

5. Access control
The repository's permissions are the base. For restricted decisions (a vendor contract, a security choice), publish a stub (title, status, owner, date) that everyone can find, with the body and evidence in a restricted location. Readers then know the decision exists and whom to ask. Evidence links carry their own permissions: a reader without access sees "restricted" rather than a broken link.

6. Scale
Generated indexes, required fields enforced by CI, a quarterly sweep of records past their review date, and a cap on free tags.

Worked example

Suppose the log holds 300 records. The CI report shows: 240 have a valid evidence link, 30 have dead links, 30 have no evidence.

records with valid evidence = 240 / 300 = 80%

The owner of each of the 60 weak records gets a ticket. The next quarter the target is 90%, so at most 30 weak records. Search quality is watched via queries with no click (a sign the record was not found).

Trade-offs and pitfalls

• A central curator does not scale. Owners plus automation do.
• Full history versus clarity: keep superseded records, but hide them from default search results.
• Strict required fields reduce speed of entry. Keep the required set small (id, title, status, owner, date, review_by, one link).
• What would change my call: a very small platform team can use a single table in the wiki with the same columns.

Direct answer

An assumption log is a living list of the things a project is treating as true without having proved them. A useful entry is specific, falsifiable (written so a test could prove it wrong), owned, backed by stated evidence, and has a date by which it must be validated or dropped. It differs from a risk log (possible future events) and a decision log (choices already made). You update it continuously, in the routines the team already runs, and at every milestone.

Structured elaboration

What makes an entry useful

• Statement: concrete and testable. Bad: "users want reports". Good: "at least half of active admins export a report weekly".
• Owner: the one person who will chase the answer.
• Evidence and confidence: what we know now (source, sample size) and how sure we are (high / medium / low).
• Impact if wrong: what breaks, so we test the expensive ones first. For a data analyst: "if exports are not CSV, the ingestion script must be rewritten, about two weeks of rework". For a business analyst: "if the sponsor has not approved the budget, the scope in the requirements document is void".
• Validation method and date (expiry): how we will test it and when it must be settled. An untested assumption without a date is just an opinion.
• Status: open, validated, invalidated, retired. Plus a link to the decisions that depend on it.

How it differs from a risk log and a decision log

They connect: an assumption is an input to a decision, and an invalidated assumption usually becomes a risk or an issue.

When to update

• In daily scrum (the short daily team check-in), ask one question: did anything block or change an assumption?
• In sprint planning (the meeting where the team picks the work for the next one to two weeks), scan open assumptions and pull validation work in as tickets.
• When new evidence arrives, and at milestones (agreed checkpoints such as "design approved") or phase gates (formal go/no-go reviews between project stages). Surfacing it through the project lifecycle reduces ambiguity, because people stop building on unstated beliefs.
• On the expiry date, when the owner must validate, extend with a reason, or invalidate.

After a discovery call (an early meeting with a customer or stakeholder to learn their needs)
Capture assumptions and open questions the same day. For each, assign an owner (the person best placed to find the answer, not the note taker) and a date. Flag any item that touches contracts, data privacy or regulation for legal review, and any item that would change scope for escalation to the sponsor (the executive who funds the project and must approve scope changes). Example: "the customer assumes we will migrate 10 years of history" is escalated because it could double the effort.

Worked example

If A1 is invalidated (a customer sends 2 GB files), it becomes a risk entry ("upload pipeline may fail on large files") and triggers a decision record about chunked uploads.

Trade-offs and pitfalls

• Vague entries ("system will be fast") cannot be tested. Rewrite until a test exists.
• Logs die when nobody owns them: name one owner for the log itself and review it in planning.
• Do not merge it into the risk log. Mixing them loses the "how do we validate this?" discipline.

Direct answer

Keep one small, living assumption and decision register that every stakeholder can open, and make it the single place where a bet, the evidence for it, and the fallback if it fails are written down. Each row has an owner and a "next check" date, so the register drives action instead of gathering dust. Two linked lists sit beside it: a decision log (what we chose and why) and a learning log (experiment results and postmortems).

Terms

• Assumption: something we believe is true but have not verified (for example, "small clinics will pay for online booking").
• Riskiest assumption: the one that, if false, kills the idea. Test it first.
• Fallback (or pivot trigger): what we will do if the evidence says no.
• Postmortem: a written review after something failed or surprised us, focused on lessons.
• Desirability, viability, feasibility: the three kinds of risk. Desirability: do people actually want this? Viability: can the business make it pay and sustain it? Feasibility: can we build it with our skills, time and technology?
• Pivot: a deliberate change of direction on the problem, customer or solution because the evidence said the current one is not working.
• Concierge trial: delivering the service by hand behind the scenes (no real product yet) to test whether people want it.
• Roadmap: the plan of what the team will work on and roughly when. Discovery review: the regular meeting where the team shows what it has learned and what it will test next.

The register (one row per assumption)

Worked example (illustrative)

A team is building appointment booking for small dental clinics.

1. A1 "Clinics lose meaningful revenue to no-shows." Type: desirability. Confidence low, impact high. Test: ten clinic interviews. Threshold: at least six describe it unprompted as a top-three problem. Fallback: narrow to the smaller no-show slice the evidence still supports (reminders only), or pivot to another clinic pain point if even that fails.
2. A2 "Receptionists will adopt a new tool." Type: desirability (a viability row would be "a clinic pays enough per month to cover our costs"). Confidence medium. Test: a two-week concierge trial with three clinics. Fallback: build a lightweight add-on to their existing calendar instead.
3. Interviews come back: four of ten (below the agreed six). A1 flips to invalidated, the fallback fires, and a decision record is written: "Chose to narrow to no-show reminders only, because 4 of 10 cited them, enough to justify a small pilot but below the agreed six needed to commit to a full suite; alternatives considered: full booking suite; revisit if the reminder pilot converts."

A stakeholder asking "why did scope shrink?" gets one link, not a meeting.

Making it findable and acted on

• One home (a shared doc or wiki table), linked from the roadmap, every discovery review and the sprint page. Assumptions get stable IDs (A1, A2) so decision records and tickets can cite them.
• A 15-minute weekly review sorts by next-check date and confidence. Any row past its date is discussed or closed.
• Each decision record has: context, options, choice, owner, review date, and links to the evidence and the assumptions it depends on.

Institutionalizing learning (beyond one team)

• File every experiment result, including negative ones, in the learning log using the same template, with the assumption ID it tested.
• Run a short postmortem when a bet fails or a fallback fires. Blameless: it asks what we missed, not who missed it.
• Drive adoption by making the register the required attachment for discovery reviews and roadmap changes (no register row, no review slot), and by having leaders cite it in updates.
• Measure impact with a few honest signals: share of open assumptions with an owner and a next-check date, how many decisions cite evidence, and how quickly invalidated bets are stopped. For example, with 12 open assumptions, 9 having both an owner and a next-check date gives 9/12 = 75 percent, and if the median time from a failed threshold to the team stopping that bet fell from 21 days to 7, the practice is working (illustrative numbers). Qualitatively, watch whether stakeholder "why" questions get answered by a link.

Trade-offs and pitfalls

• Too heavy a template gets abandoned. Start with five columns and add fields only when a real question demands them.
• Thresholds set after the result is a rationalization, not a test.
• A register that is never re-read is worse than none, because it signals false confidence. The review cadence is the real mechanism.
• Confidence is a judgment, not a measurement. Record the reason next to it.
• What would change my approach: a very small team of two might use a single page, and a regulated product would add formal sign-off on decisions.

Direct answer

Govern the few things that must be consistent and auditable (how artifacts are classified, where they live, who reviews them, and how they are retained) and leave the rest to architect judgment. I would run it as a lightweight, tiered system introduced through a pilot, with templates that make the right thing the easy thing, and review effort scaled to the risk of the artifact.

Terms

• Proof of concept (PoC): a small build proving a design works for one customer's need.
• Artifact: any produced deliverable (diagram, proposal, PoC repo, design decision).
• Audit trail: a record showing who approved what and when.
• C4 model: a diagram convention with four zoom levels (context, container, component, code).
• Data-residency design: an architecture that keeps a customer's data inside a specific country or region because of law or contract.
• Pull request (PR) approval: in a Git repository, a reviewer's recorded "approve" on a proposed change. It is timestamped and tied to a name automatically.
• Retained per policy: kept for the length of time the company's records policy requires, for example seven years, then deleted.
• Short rubric: a checklist of a few yes/no questions reviewers use instead of personal taste.

Step 1: Diagnose before prescribing (first two weeks)
Sample about 15 recent artifacts across the team and score them against three questions: could a stranger find it, could they understand it, could we prove who reviewed it? This shows whether the problem is style, storage or review, and gives the pilot a baseline. Illustrative result: of 15 artifacts, 6 were findable (40 percent), 9 understandable (60 percent) and 3 had provable review (20 percent). Review is the weakest link, so the pilot invests there first.

Step 2: Tier artifacts by risk

Step 3: Make consistency cheap

• One shared, versioned repository or workspace with a fixed folder and naming convention by customer and date.
• Starter templates: a diagram convention (C4 levels, one legend), a proposal outline, a PoC README stating scope, assumptions, results and what was NOT proven. Templates do most of the work, so reviewers do not have to.
• Every artifact carries a small header, for example:

Owner: Ana R.   Customer: Northbank   Status: in review
Tier: 1         Reviewer: J. Osei     Date: 2026-09-14

Step 4: Keep it fast

• A review service-level target: Tier 1 reviewed within two business days, or it proceeds with a logged note that marks it "unreviewed, escalated". Review must then complete within a further two business days, and the unreviewed period stays visible in the audit trail, so speed is protected without leaving a Tier 1 artifact permanently without an approval on record.
• Reviewers check a short rubric, not taste. Example rubric: (1) states scope and assumptions, (2) diagram uses the legend, (3) risks and unknowns listed, (4) residency and security claims cite a source. Four yes/no answers, and any "no" comes back with a comment.
• Sampling for Tier 2 keeps cost flat as volume grows.

Step 5: Audit without paperwork
Approval is a pull request approval or a status field, so the trail is a side effect of normal work. Quarterly, a lead samples Tier 1 artifacts and reports the compliance rate.

Worked example (illustrative)
Ana's PoC for a bank customer used her own diagram style and lived on a personal drive. Under the new scheme she starts from the PoC template in the shared repo, marks it Tier 1 because it commits to a data-residency design, and requests review. A Security Architect approves within the two-day target and the record shows in the repo history. Six months later an auditor asks "who approved the residency claim?" and the answer is one link.

Measuring success: share of new artifacts in the shared home, review turnaround, and architect satisfaction. Do NOT measure by count of documents produced.

Trade-offs and pitfalls

• Over-governing pushes work into private drives, which is worse than variance. If usage falls, the process is too heavy.
• Mandating a tool before agreeing the problem breeds resentment. Involve two or three respected architects in drafting the templates.
• Consistency versus speed: I would relax Tier 2 rules before I would relax the Tier 1 audit trail.
• What would change my call: a regulated customer base pushes more artifacts into Tier 1, and a small team may need only the templates.