Documentation and Knowledge Management Questions

Direct answer

Use one central, versioned metric registry as the single source of truth for definitions of shared, company-level metrics (revenue, active user, churn), and let teams keep federated docs only for local, team-specific metrics that reference the registry. Every shared metric has a named owner, a change process with approval and notice, and enforcement through code: dashboards pull the definition (or the query) from the registry, so a number in a BI (business intelligence, dashboard and reporting) tool cannot silently diverge.

Why the numbers differ (worked example)

Say 100 orders total 1,000 in item price, plus 100 tax, and 50 was refunded. Three tools report "revenue":

• Tool A: sum of amounts charged, including tax = 1,100.
• Tool B: item price only, ignoring refunds = 1,000.
• Tool C: item price minus refunds = 950.

All three are "correct" for their own unwritten definition. The registry entry writes one down:

metric: net_revenue
definition: sum(item_price) - sum(refunds), excluding tax
grain: daily, in UTC (sales dated by order date; refunds dated by refund date since version 3)
owner: Finance Analytics
source table: orders_curated
version: 3 (changed 2026-08-01: refunds now booked at refund date)

Grain means the level of detail a metric is calculated at (here, one number per day). With that entry, only 950 is "net revenue"; the others need different names (gross_charged, item_sales). The 950 example assumes the orders and the refund fall in the same period, so version 2 and version 3 agree there. They differ when a refund lands in a later period:

Order placed 2026-07-30 for 200; refunded 2026-08-03 for 50
Version 2 (refund booked on order date):   July = 150, August = 0
Version 3 (refund booked on refund date):  July = 200, August = -50

The total is 150 either way, but July's published number changes from 150 to 200, which is why this needed a version bump.

Central versus federated

Recommendation: central for the small set of tier-1 shared metrics (the handful that drive company decisions or external reporting) (typically tens, not thousands); federated for the rest, with a rule that a local metric must not reuse a registered name.

Roles and approvals

• Metric owner (one accountable person per metric, from the business function that lives with the number).
• Data steward group (a small cross-team panel of people responsible for data quality and definitions) approves new tier-1 metrics and definition changes.
• Authors propose changes by pull request against the registry file, so history is the audit trail.
• Approval: owner plus one steward for a change; consumers listed in the entry get notified.

Change control

A definition change bumps the version, states the date it takes effect, and states impact ("historical values restated", meaning past numbers are recalculated under the new rule, or "old value frozen", meaning past numbers stay as originally published and only new periods use the new rule). Dashboards show the version. Breaking changes need a notice window before rollout, and a side-by-side period where old and new values appear together. Example: announce on 2026-07-15, show both columns in July, and switch on 2026-08-01, with a note that July net revenue moves from 150 to 200 in the example above.

Enforcement

• Prefer a semantic layer (a shared layer that holds each metric's definition, so it is written once) or governed metric definition (one place where the metric's SQL lives and BI tools query it) so tools consume rather than re-implement.
• CI check: fail if a dashboard defines a field named like a registered metric outside the registry.
• A weekly reconciliation job compares each tool's reported value for tier-1 metrics against the registry query and alerts on mismatch above a tolerance (the difference you accept, for example 0.5%). Example: the registry says 950 and Tool A shows 941; the gap is 9 / 950 = about 0.95%, above 0.5%, so an alert names the tool and the metric.

Onboarding authors

A one-page contribution guide, a template entry, a 30-minute walkthrough, and a first task of documenting one existing metric under a steward's review. Authors learn the naming rule (no silent reuse) early.

Pitfalls

A registry with no enforcement becomes another stale wiki. A registry too large to steward gets bypassed; keep tier-1 small. What would change this: if only one BI tool remains, its native governed layer may make a separate registry unnecessary.

Direct answer

Keep it lightweight by putting each assumption next to the thing it supports (the dashboard definition, the model repo), giving it an owner, a review trigger and an executable check, and letting the check's failure flip the assumption's status and alert the owner. The process is: record, wire to a check, review on a schedule or an event, then retire or invalidate on evidence.

Structured elaboration

1. Record

• One small file per dashboard or model (assumptions.md or YAML in the same repo), or metadata fields on the dashboard itself (description, owner, source tables, refresh schedule). Each entry: statement, owner, why we believed it, how it would be checked, review date.
• Types worth recording: data semantics ("amounts are in USD"), stability ("average session length stays near its training value"), coverage ("all regions send events"), and business rules ("a customer is active if they logged in within 30 days").

2. Tie to code, tests and monitoring (so others can reproduce and validate)

• Data tests in the pipeline (for example tests in dbt, a common SQL transformation tool: accepted_values checks a column holds only the listed values, such as currency in USD or EUR, and not_null checks no row is blank) for semantic assumptions. Failures block the refresh or raise an alert.
• Unit tests for business rules: a fixed input with a known answer.
• Monitoring for stability assumptions. Drift means live data has moved away from what the model or dashboard was built on; a distribution is simply how the values are spread (for example what share of sessions are short, medium or long). Compare live to training and alert past a tolerance. A simple mean-change threshold is a fine start (it is what the script below uses). A more thorough drift metric is the population stability index (PSI), which compares the share of records in each bucket:

PSI = sum over buckets of (live% - train%) x ln(live% / train%)

Example with two buckets: training 50%/50%, live 70%/30%. Bucket 1: (0.70 - 0.50) x ln(1.4) = 0.067. Bucket 2: (0.30 - 0.50) x ln(0.6) = 0.102. PSI = 0.169. A common rule of thumb reads under 0.1 as stable, 0.1 to 0.25 as worth a look, and above 0.25 as a real shift, so this one is worth a look. The intuition: the more the bucket shares move, the bigger the number.

• Every check names the assumption ID in its name or message (A1_currency_is_usd), and the assumption entry links back to the check. Then a failing test tells you which belief broke, and a reader of the assumption can find the evidence.

3. Review

• On a cadence (quarterly for dashboards, at each retrain for models) and on events: an upstream schema change, a new data source, or a product change. Events matter more than the calendar.
• The owner marks each assumption holds, invalidated, or retired (no longer relevant). Retire when the dashboard or feature is removed, not silently.

4. Invalidate
A failing check sets status to invalidated, opens a ticket for the owner, adds a visible banner on the dashboard ("revenue may include non-USD orders") and, for a model, pauses automatic use or falls back to a safe baseline until resolved.

Worked example

This runnable script (it creates a five-row orders table, runs each SQL check and counts violating rows, then compares a live average to a training average) checks two data assumptions on a tiny table and one drift assumption (the numbers are pinned, so the output is reproducible):

import sqlite3

con = sqlite3.connect(":memory:")
con.executescript("""
CREATE TABLE orders(id INTEGER PRIMARY KEY, currency TEXT, amount REAL);
INSERT INTO orders VALUES (1,'USD',20.0),(2,'USD',35.5),(3,'USD',12.0),(4,'EUR',40.0),(5,'USD',18.0);
""")

checks = {
    "A1 revenue is all USD": "SELECT COUNT(*) FROM orders WHERE currency <> 'USD'",
    "A2 amounts are positive": "SELECT COUNT(*) FROM orders WHERE amount <= 0",
}
for name, sql in checks.items():
    bad = con.execute(sql).fetchone()[0]
    print(name, "->", "VIOLATED" if bad else "holds", "(violating rows:", str(bad) + ")")

train_mean, live_mean = 4.0, 5.1
drift = abs(live_mean - train_mean) / train_mean
print("A3 session-length drift:", round(drift * 100, 1), "%", "-> VIOLATED" if drift > 0.20 else "-> holds")

Output:

A1 revenue is all USD -> VIOLATED (violating rows: 1)
A2 amounts are positive -> holds (violating rows: 0)
A3 session-length drift: 27.5 % -> VIOLATED

For A3 the arithmetic is (5.1 - 4.0) / 4.0 = 0.275, above the 20% tolerance, so the model owner is alerted. A1 failing means the EUR order breaks the revenue dashboard's USD assumption: the banner goes up, a ticket is opened, and the fix is either currency conversion or a filter, recorded as a decision.

Trade-offs and pitfalls

• Checks that alert too often get ignored. Set tolerances from past variation and tune them.
• Not every assumption is testable. For those, keep the entry with an owner and review date rather than pretending to automate it.
• A separate assumptions spreadsheet drifts from the code. Keep it in the repo so changes are reviewed together.
• What would change my call: for a one-off analysis I would skip monitoring and record assumptions in the report footer.

Direct answer

An assumption log is a living list of the things a project is treating as true without having proved them. A useful entry is specific, falsifiable (written so a test could prove it wrong), owned, backed by stated evidence, and has a date by which it must be validated or dropped. It differs from a risk log (possible future events) and a decision log (choices already made). You update it continuously, in the routines the team already runs, and at every milestone.

Structured elaboration

What makes an entry useful

• Statement: concrete and testable. Bad: "users want reports". Good: "at least half of active admins export a report weekly".
• Owner: the one person who will chase the answer.
• Evidence and confidence: what we know now (source, sample size) and how sure we are (high / medium / low).
• Impact if wrong: what breaks, so we test the expensive ones first. For a data analyst: "if exports are not CSV, the ingestion script must be rewritten, about two weeks of rework". For a business analyst: "if the sponsor has not approved the budget, the scope in the requirements document is void".
• Validation method and date (expiry): how we will test it and when it must be settled. An untested assumption without a date is just an opinion.
• Status: open, validated, invalidated, retired. Plus a link to the decisions that depend on it.

How it differs from a risk log and a decision log

They connect: an assumption is an input to a decision, and an invalidated assumption usually becomes a risk or an issue.

When to update

• In daily scrum (the short daily team check-in), ask one question: did anything block or change an assumption?
• In sprint planning (the meeting where the team picks the work for the next one to two weeks), scan open assumptions and pull validation work in as tickets.
• When new evidence arrives, and at milestones (agreed checkpoints such as "design approved") or phase gates (formal go/no-go reviews between project stages). Surfacing it through the project lifecycle reduces ambiguity, because people stop building on unstated beliefs.
• On the expiry date, when the owner must validate, extend with a reason, or invalidate.

After a discovery call (an early meeting with a customer or stakeholder to learn their needs)
Capture assumptions and open questions the same day. For each, assign an owner (the person best placed to find the answer, not the note taker) and a date. Flag any item that touches contracts, data privacy or regulation for legal review, and any item that would change scope for escalation to the sponsor (the executive who funds the project and must approve scope changes). Example: "the customer assumes we will migrate 10 years of history" is escalated because it could double the effort.

Worked example

If A1 is invalidated (a customer sends 2 GB files), it becomes a risk entry ("upload pipeline may fail on large files") and triggers a decision record about chunked uploads.

Trade-offs and pitfalls

• Vague entries ("system will be fast") cannot be tested. Rewrite until a test exists.
• Logs die when nobody owns them: name one owner for the log itself and review it in planning.
• Do not merge it into the risk log. Mixing them loses the "how do we validate this?" discipline.

Direct answer

Treat "irrelevant results" as a measurable problem. First collect data on what people search for and what happens next, then sort failures into a few root causes (nothing relevant exists, it exists but is not found, it ranks too low, or it is stale or duplicated), fix the cheapest and most frequent cause first, and keep improving with a small recurring review of failed searches.

Terms

• Rank: a result's position in the list (rank 1 is the top result).
• Index: the searchable copy of your content that the search tool actually looks through; a page missing from it can never be found.
• Synonym list / alias metadata: a table telling search that "k8s" means "Kubernetes", or a field on a page listing other names people use for it.
• Boost: a setting that pushes matching pages up (or down) in the ranking, for example boosting recently updated pages.

Data to collect

• Search logs: query text, number of results, which result was clicked and at what rank, time on page after click, and whether the user searched again immediately (a reformulation).
• Zero-result rate: share of queries returning nothing.
• Abandonment: share of searches that showed results but ended with no click. Zero-result searches also end with no click, but count them under the zero-result rate so the two are not mixed.
• Content facts: last-updated date, owner, page views, duplicates per topic.
• Qualitative: ask ten engineers for the last thing they could not find, and watch them search.

Worked example (small, made-up log of 20 queries)

Rates: zero-result rate is 4 of 20 = 20%; abandonment (results shown, no click) is 4 of 20 = 20%. Together, 8 of 20 searches (40%) ended without a click, which is why the two are tracked separately.

Now trace every failure to a cause. The 4 zero-result queries: "k8s runbook" and "oncall rota" (the pages say "Kubernetes" and "on-call schedule"), a vocabulary mismatch that synonyms fix, plus "vpn setup for contractors" and "ssl cert renewal steps", which match nothing because no such page exists (missing content). The 4 no-click searches: two show a stale 2021 page first (stale content), and two show pages titled "Notes 3" and "Infra misc" that do not say what they cover (weak titles). The 3 low-rank clicks are ranking problems: the right page exists but sits at rank 4 or lower.

Tally: vocabulary 2, missing content 2, stale content 2, weak titles 2, ranking 3. That accounts for all 11 failed searches (9 succeeded). Eight of the 11 are content or vocabulary problems that are cheap to fix without touching ranking, so fix those first.

Most likely root causes and fixes

The first four rows are the core causes; the last two are checks to make once those are clean.

Improving relevance over time

1. Build a test set of the 30 to 50 most common queries with the page that should win, and run it after every change to search settings; the measure is the share where the right page is in the top three.
2. Review the top failed queries weekly for the first month, then monthly, and assign each to an owner.
3. Add ranking signals gradually (freshness, popularity, ownership), one at a time, re-running the test set to see which helped.
4. Keyword ranking formulas (such as BM25, a standard scoring method that favors pages containing rare query words) are often enough; add semantic (meaning-based) search only if failures persist after cleanup.

Pitfalls

• Tuning ranking before cleaning content wastes effort: no ranking can surface a page that is missing or wrong.
• Using click-through alone misleads: a click on a bad page counts as a success unless you also check whether the user searched again.
• What would change my order of work: if the zero-result rate dominates, fix vocabulary and gaps before touching ranking.

Direct answer

Organize the knowledge base around how people look for things, not around who wrote them. Use a few top-level categories, classify each document by a type (what kind of document it is), and let secondary labels such as team or system cut across. A taxonomy matters because people can only use what they can find, and a stable structure lets search, ownership and freshness rules work.

Terms

• Taxonomy: an agreed way to group and name content.
• Dataset doc: describes a table or file (columns, source, refresh, owner).
• Runbook: step-by-step instructions to operate or fix something, usually during an incident.
• Architecture doc: explains how a system is built and why.

Top-level categories for a data platform

1. Getting started: onboarding, access requests, tooling setup.
2. Data catalog: dataset docs, grouped by business domain (orders, payments, customers).
3. Pipelines and systems: architecture docs and design decisions per system.
4. Operations: runbooks, on-call guides, incident reviews.
5. Standards and policies: naming conventions, quality rules, data handling.

How each document is classified

The same page can be reached by category, type or system, but has one home so there is one canonical copy.

Why it matters (worked example, illustrative)
At 2 a.m. an engineer sees the nightly orders load fail. With a good taxonomy she opens Operations, filters by system "orders pipeline", and finds the runbook in two clicks. With no taxonomy she searches "load failed", gets forty results including old design notes and an outdated wiki page, and wastes time or follows stale steps. The taxonomy also tells people where to put new content, which prevents duplicates.

Principles

• Name categories by reader tasks (get started, fix, look up) rather than org charts, because teams reorganize and paths break.
• Keep depth to about two or three levels.
• Use one type list and reuse it everywhere so search filters are predictable.

Trade-offs and pitfalls

• Deep hierarchies hide content; flat structures with rich filters scale better.
• A misc bucket grows without limit, so review it monthly.
• Testing with real users (ask three people to find a given doc) beats debating categories.