Documentation and Knowledge Management Questions

Direct answer

Use one central, versioned metric registry as the single source of truth for definitions of shared, company-level metrics (revenue, active user, churn), and let teams keep federated docs only for local, team-specific metrics that reference the registry. Every shared metric has a named owner, a change process with approval and notice, and enforcement through code: dashboards pull the definition (or the query) from the registry, so a number in a BI (business intelligence, dashboard and reporting) tool cannot silently diverge.

Why the numbers differ (worked example)

Say 100 orders total 1,000 in item price, plus 100 tax, and 50 was refunded. Three tools report "revenue":

• Tool A: sum of amounts charged, including tax = 1,100.
• Tool B: item price only, ignoring refunds = 1,000.
• Tool C: item price minus refunds = 950.

All three are "correct" for their own unwritten definition. The registry entry writes one down:

metric: net_revenue
definition: sum(item_price) - sum(refunds), excluding tax
grain: daily, in UTC (sales dated by order date; refunds dated by refund date since version 3)
owner: Finance Analytics
source table: orders_curated
version: 3 (changed 2026-08-01: refunds now booked at refund date)

Grain means the level of detail a metric is calculated at (here, one number per day). With that entry, only 950 is "net revenue"; the others need different names (gross_charged, item_sales). The 950 example assumes the orders and the refund fall in the same period, so version 2 and version 3 agree there. They differ when a refund lands in a later period:

Order placed 2026-07-30 for 200; refunded 2026-08-03 for 50
Version 2 (refund booked on order date):   July = 150, August = 0
Version 3 (refund booked on refund date):  July = 200, August = -50

The total is 150 either way, but July's published number changes from 150 to 200, which is why this needed a version bump.

Central versus federated

Recommendation: central for the small set of tier-1 shared metrics (the handful that drive company decisions or external reporting) (typically tens, not thousands); federated for the rest, with a rule that a local metric must not reuse a registered name.

Roles and approvals

• Metric owner (one accountable person per metric, from the business function that lives with the number).
• Data steward group (a small cross-team panel of people responsible for data quality and definitions) approves new tier-1 metrics and definition changes.
• Authors propose changes by pull request against the registry file, so history is the audit trail.
• Approval: owner plus one steward for a change; consumers listed in the entry get notified.

Change control

A definition change bumps the version, states the date it takes effect, and states impact ("historical values restated", meaning past numbers are recalculated under the new rule, or "old value frozen", meaning past numbers stay as originally published and only new periods use the new rule). Dashboards show the version. Breaking changes need a notice window before rollout, and a side-by-side period where old and new values appear together. Example: announce on 2026-07-15, show both columns in July, and switch on 2026-08-01, with a note that July net revenue moves from 150 to 200 in the example above.

Enforcement

• Prefer a semantic layer (a shared layer that holds each metric's definition, so it is written once) or governed metric definition (one place where the metric's SQL lives and BI tools query it) so tools consume rather than re-implement.
• CI check: fail if a dashboard defines a field named like a registered metric outside the registry.
• A weekly reconciliation job compares each tool's reported value for tier-1 metrics against the registry query and alerts on mismatch above a tolerance (the difference you accept, for example 0.5%). Example: the registry says 950 and Tool A shows 941; the gap is 9 / 950 = about 0.95%, above 0.5%, so an alert names the tool and the metric.

Onboarding authors

A one-page contribution guide, a template entry, a 30-minute walkthrough, and a first task of documenting one existing metric under a steward's review. Authors learn the naming rule (no silent reuse) early.

Pitfalls

A registry with no enforcement becomes another stale wiki. A registry too large to steward gets bypassed; keep tier-1 small. What would change this: if only one BI tool remains, its native governed layer may make a separate registry unnecessary.

Direct answer

Replace status meetings with a small set of living documents, each with one named owner, a fixed update rhythm, and one index page that everyone bookmarks. Meetings that remain are for decisions, not for reading out updates. Accountability comes from ownership and from visible staleness ("last updated" dates and an automatic nudge), not from having people talk.

The documents (worked example for a model-development team)

An example experiment log entry (so the fields are concrete):

Experiment #41 | Owner: S. Patel | 2026-09-16
Hypothesis: adding last-30-day support tickets as a feature will improve churn prediction
Data version: customers_2026_08 (the frozen snapshot used for training)
Result vs baseline: score 0.82 vs 0.80 for the current production model
Next step: test on live traffic

An example weekly note:

Week of 2026-09-21: churn model v3
Done: retrained on Aug data; offline metric vs baseline: +0.02 (see experiment #41)
Next: shadow test against live traffic
Blocked: feature table refresh owned by Data Eng (needs answer by Wed)
Decision needed: ship threshold 0.4 or 0.5? (PM, by Fri, options in decision log #12)

Plain-language reading for a non-ML stakeholder: the offline metric is the model's score on saved historical data, and +0.02 means it beats the current model by two points (for example 0.80 to 0.82). A shadow test runs the new model quietly next to the live one on real traffic, without affecting customers, to check it behaves. The ship threshold is the score cutoff above which a customer is flagged as likely to churn: 0.4 flags more customers (more catches, more false alarms), 0.5 flags fewer.

Who owns what, and how people find it

• One index page per project links every document above, and each doc header carries owner, last-updated date and status. The index is linked from the team channel topic.
• Stakeholders subscribe to page changes or a weekly digest, so information reaches them without a meeting.
• Naming and location are fixed by a template, so nobody hunts.

Keeping people accountable without meetings

• Comments and questions are answered in the doc thread within an agreed window (for example one business day), and unanswered questions surface in the weekly note.
• A stale doc (not updated within twice its cadence) is flagged automatically to its owner and their manager.
• A 25-minute fortnightly decision review replaces several syncs: only items in the "decision needed" section are discussed.

Trade-offs and pitfalls

• Async docs fail when nobody reads them; ask stakeholders which fields they need, and cut the rest.
• Sensitive or ambiguous topics (a missed target, a conflict) still deserve a live conversation; write-ups can hide disagreement.
• Cutting meetings all at once breaks trust. Pilot for a month, drop the sync that has the lowest attendance value first, and compare stakeholder questions asked in ad hoc messages before and after.
• What would flip it: if stakeholders keep asking the same questions in chat, the docs are not answering what they need, so revise the docs rather than restoring the meeting.

Direct answer

Organize the knowledge base around how people look for things, not around who wrote them. Use a few top-level categories, classify each document by a type (what kind of document it is), and let secondary labels such as team or system cut across. A taxonomy matters because people can only use what they can find, and a stable structure lets search, ownership and freshness rules work.

Terms

• Taxonomy: an agreed way to group and name content.
• Dataset doc: describes a table or file (columns, source, refresh, owner).
• Runbook: step-by-step instructions to operate or fix something, usually during an incident.
• Architecture doc: explains how a system is built and why.

Top-level categories for a data platform

1. Getting started: onboarding, access requests, tooling setup.
2. Data catalog: dataset docs, grouped by business domain (orders, payments, customers).
3. Pipelines and systems: architecture docs and design decisions per system.
4. Operations: runbooks, on-call guides, incident reviews.
5. Standards and policies: naming conventions, quality rules, data handling.

How each document is classified

The same page can be reached by category, type or system, but has one home so there is one canonical copy.

Why it matters (worked example, illustrative)
At 2 a.m. an engineer sees the nightly orders load fail. With a good taxonomy she opens Operations, filters by system "orders pipeline", and finds the runbook in two clicks. With no taxonomy she searches "load failed", gets forty results including old design notes and an outdated wiki page, and wastes time or follows stale steps. The taxonomy also tells people where to put new content, which prevents duplicates.

Principles

• Name categories by reader tasks (get started, fix, look up) rather than org charts, because teams reorganize and paths break.
• Keep depth to about two or three levels.
• Use one type list and reuse it everywhere so search filters are predictable.

Trade-offs and pitfalls

• Deep hierarchies hide content; flat structures with rich filters scale better.
• A misc bucket grows without limit, so review it monthly.
• Testing with real users (ask three people to find a given doc) beats debating categories.

Direct answer

Give every doc or dataset entry a small mandatory core (type, owning team, lifecycle state, sensitivity), a controlled vocabulary for tags with a named steward, and make ownership and access derive from the metadata rather than being maintained separately. Search then works on structured fields first, free-text tags second.

Terms

• Metadata: data about a document (who owns it, what kind it is).
• Tag: a label attached to a doc for search and filtering.
• Controlled vocabulary: an approved list of tag values, changed only through a process.
• Steward: the person responsible for a domain's tags and doc quality.
• ACL (access control list): a rule saying which people or groups may see an item.
• Lifecycle state: draft, active, deprecated or archived.
• Sensitivity: how restricted the content is (public, internal, restricted).
• Model card: a short document describing a machine-learning model's purpose, data and limits.
• Facets: fixed filter categories in search, such as type or domain, shown as checkboxes or dropdowns.
• Alias: an alternative name that points at an approved tag, so "customers" finds docs tagged "customer".
• Recall (in search): how many of the relevant docs a search actually returns. Inconsistent tags hide relevant docs, so recall drops.
• Lineage: the record of where a dataset came from and which datasets or reports are built from it.
• Refresh cadence: how often a dataset is updated (hourly, daily). Schema: the list of columns and their types.
• Identity system: the company directory of people and groups (for example the single sign-on provider).

Mandatory fields (eight rows below, covering eleven individual fields: id, title, type, owner_team, steward, lifecycle, sensitivity, domain, last_reviewed, review_every_days, links; auto-fill most so people complete them)

Type-specific fields sit on top: a dataset adds schema and refresh cadence, a runbook adds the system and severity it covers, a model card adds training data and intended use.

Typed artifact example (sample record)

{
  "id": "doc-4127",
  "type": "runbook",
  "title": "Orders pipeline: recovering a stuck nightly load",
  "owner_team": "data-platform",
  "steward": "priya.n",
  "lifecycle": "active",
  "sensitivity": "internal",
  "domain": "orders",
  "last_reviewed": "2026-03-02",
  "review_every_days": 90,
  "links": {"code": "repo://data-platform/orders-etl", "dataset": "ds-orders-daily"}
}

Finding stale or sensitive docs (runnable, with output)
The query computes days since last review (julianday is SQLite's date-to-number function, so subtracting two of them gives elapsed days) and compares it to each doc's window. On the sample date, 2026-09-28: doc-4127 was reviewed 210 days ago against a 90-day window, doc-2001 331 days ago against 180, doc-3310 100 days against 180 (fine), doc-2050 75 days against 180 (fine).

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""create table docs(id text, type text, owner_team text, lifecycle text,
              sensitivity text, last_reviewed text, review_every_days int)""")
db.executemany("insert into docs values (?,?,?,?,?,?,?)", [
    ("doc-4127", "runbook",    "data-platform", "active",     "internal",   "2026-03-02", 90),
    ("doc-2001", "dataset",    "payments",      "active",     "restricted", "2025-11-01", 180),
    ("doc-3310", "model_card", "ml-platform",   "active",     "internal",   "2026-06-20", 180),
    ("doc-1500", "dashboard",  "bi",            "deprecated", "internal",   "2025-01-10", 90),
    ("doc-2050", "dataset",    "payments",      "active",     "restricted", "2026-07-15", 180),
])

TODAY = "2026-09-28"
print("stale:")
for row in db.execute("""select id, type, owner_team from docs
    where lifecycle = 'active'
      and julianday(?) - julianday(last_reviewed) > review_every_days
    order by id""", (TODAY,)):
    print(row)

print("restricted and not reviewed in 90 days:")
for row in db.execute("""select id, type, owner_team from docs
    where sensitivity = 'restricted'
      and julianday(?) - julianday(last_reviewed) > 90
    order by id""", (TODAY,)):
    print(row)

Output:

stale:
('doc-2001', 'dataset', 'payments')
('doc-4127', 'runbook', 'data-platform')
restricted and not reviewed in 90 days:
('doc-2001', 'dataset', 'payments')

The runbook is flagged because 210 days is over its 90-day window, while the dataset entry doc-2001 is flagged twice: 331 days is past its 180-day window and, being restricted, past the stricter 90-day sensitive review rule. doc-2050 is also restricted but only 75 days old, so it is not flagged.

Stopping tag sprawl

• Two tiers: fixed facets (type, domain, lifecycle, sensitivity are dropdowns, never free text) and a governed tag list that a steward approves.
• Request flow: propose a tag, the steward checks for a synonym, then adds it as an alias (for example "customers" maps to "customer") instead of a new tag.
• Quarterly cleanup: merge tags used by fewer than a handful of docs, and report the tag count.
• Autocomplete from the approved list at authoring time prevents most duplicates.

Connecting tags to ownership and access

• owner_team maps to a group in the identity system, so ownership is reassigned in one place when teams reorganize.
• Sensitivity maps to an access rule: internal means all engineers, restricted means the owner group plus named approvers. Search filters results by the viewer's permissions, so restricted titles never leak in results.
• Access changes are logged, and a restricted doc without an owner is blocked from publication.

Trade-offs and pitfalls

• Too many mandatory fields means blanks or junk values. Auto-fill what you can (owner from the repo, type from the template).
• Free-text tags feel flexible but destroy recall in a year.
• Sensitivity set by the author alone is unreliable, so spot-check it and default new data-related pages to "internal".
• What would change my call: with only a few dozen docs a folder structure is enough.

Direct answer

Treat each production report as a versioned product with an owner. Classify a change by its effect on consumers (cosmetic, compatible, breaking), route approvals by class, keep every change reversible, announce breaking changes ahead of time with a comparison to the old numbers, and retain the previous version for a defined period.

Terms

• Versioning: giving each release an identifier (for example v2.1) so people can tell which one they saw.
• Breaking change: one that alters what a number means or removes something people rely on.
• Rollback: restoring the previous working version.
• Deprecation: announcing something will be removed, with a date.
• Semantic versioning: MAJOR.MINOR.PATCH numbering; MAJOR for breaking changes.
• Changelog: a dated, running list of every change to a report and why it was made.
• Staging copy: a test copy of the report that runs on real data but that no business user reads.
• Validation total: a headline number (for example total revenue) that you compare between old and new versions before release.
• Metric owner: the business person who defines what a number means (for example Finance owns "revenue").

1. Classify changes

The Patch versus Major boundary: ask whether the definition changed. A Patch correction makes the number match the definition everyone already agreed on (the join was wrong, so the number was wrong), so it is announced in the changelog with the before and after values. A Major change alters the definition itself, so the same customers now count differently, and that needs the metric owner's approval and advance notice.

2. Approvals (RACI style)
The analyst who builds is responsible, the report owner is accountable, the metric owner in the business is consulted and must sign off on definition changes (consulted here means their agreement is required, not just their opinion), and affected consumers are informed. (RACI: responsible, accountable, consulted, informed.) Every class gets a second-analyst review: for Patch it is a quick peer check, and for Minor and Major it is a full review before release.

3. Rollback

• Keep the prior version deployable and the query in source control, so rollback means redeploying it, not rebuilding.
• Test in a staging copy against production data before release, and compare key totals side by side.
• Roll back if a validation total differs from the expected one, or if consumers report a wrong number. Decide the rollback owner in advance.

4. Communicating breaking changes
Announce at least two weeks ahead through the channel people actually read (email plus a banner in the dashboard), state what changes, why, when, and show old versus new values for the last three periods. Include a link to the changelog and a named contact. Run old and new versions in parallel where possible.

5. Retention of old versions
Keep the previous Major version accessible read-only for a fixed period (for example 90 days, or through the next month-end financial close, when Finance finalizes its numbers), keep definitions and change logs indefinitely, and archive the rest. The exact period should follow audit and legal requirements.

Worked example (illustrative)
The team wants "active customer" to mean a purchase within 28 days instead of 30. That alters historical counts, so it is Major (v3.0.0). Finance's metric owner approves. Users get notice with a comparison table built from the same customers (the numbers below are illustrative):

Period      Old (30 days)  New (28 days)  Change
Last qtr    12,400         11,900         -500 (about -4%)
Prior qtr   11,800         11,350         -450 (about -4%)

The count drops because customers whose last purchase was 29 or 30 days ago no longer qualify, and the table shows that the drop is a definition effect, not a business decline. v2 remains available for 90 days, and if the Finance total looks wrong at the first close, the team redeploys v2.

Trade-offs and pitfalls

• Heavy approvals push people to silent edits, so keep patch changes light and auditable.
• Communicating only the change and not the numeric impact leads to trust loss.
• Retention costs storage and confusion, so label old versions clearly.
• What would change my call: for self-serve exploratory dashboards, a lighter policy is appropriate. Apply the full one to reports that feed decisions or external reporting.