Technical Writing and Documentation Questions

Scope & purpose:

• Data dictionary: technical, schema-level reference for datasets/tables/fields used by engineers — focuses on structure, types, constraints. Narrow scope (per database/schema).
• Data catalog: broader, user-facing inventory of datasets across the organization — focuses on discovery, lineage, ownership, business context and governance. Enterprise scope.

Typical content:

• Data dictionary contains field-level technical metadata: column name, data type, nullability, default, column-level description, allowed values, column-level lineage references.
• Data catalog contains dataset-level and cross-cutting metadata: dataset description, business glossary terms, owners/stewards, sensitivity/classification, lineage, tags, usage metrics, access instructions, sample queries.

Who maintains:

• Data engineers / DBAs typically create and maintain the data dictionary as part of schema management and ETL development.
• Data catalogs are maintained collaboratively: data stewards, data owners, and data governance teams curate catalog entries; engineers and automated scans populate technical metadata.

Examples — Data dictionary (field/metadata items):

1. Column name: customer_id
2. Data type and length: VARCHAR(36) / UUID
3. Constraints: NOT NULL, FK to customers.id

Examples — Data catalog (dataset/metadata items):

1. Dataset description: "Orders fact table for e-commerce transactions"
2. Owner / steward: analytics-team@example.com
3. Sensitivity / classification: PII / Confidential; access policy and SLA

These complement each other: dictionaries enable correct pipeline code and validations; catalogs enable discovery, governance, and responsible use.

1. Doc Search Success Rate — % of search sessions that end with a click on a doc or “found” signal.
   How to collect: Instrument documentation search (frontend events) to record query → click or “no results” / zero-clicks; correlate with time on doc. Store in analytics (e.g., Snowflake/BigQuery).
   Target: ≥ 70% weekly rolling average for core data platform docs.

2. Time-to-First-Query (TTFQ) — median time from a user landing on docs to running their first data query against the platform.
   How to collect: Link docs sessions to internal query logs via user id; measure time delta between doc session end and first query execution.
   Target: median ≤ 30 minutes for new users; ≤ 5 minutes for experienced users in common flows.

3. Doc-Update Latency — average time between a schema/pipe change and corresponding doc update publication.
   How to collect: Track source-of-truth change events (git commit or schema registry event) and doc update timestamps (PR merge/deploy).
   Target: ≤ 48 hours for critical/data-contract changes; ≤ 7 days for non-critical.

4. First-Contact Resolution Rate — % of support tickets or Slack questions about a dataset answered by linking to docs without follow-up.
   How to collect: Tag support interactions; detect answers that reference docs and check if the issue was closed without follow-ups within 72 hours.
   Target: ≥ 60% for dataset-related queries.

5. Documentation Coverage Index — % of production datasets/pipelines that have required documentation fields (schema, owner, SLAs, lineage, example queries).
   How to collect: Periodic automated scan of metadata store/catalog (e.g., Great Expectations / Data Catalog) to validate presence of fields.
   Target: ≥ 90% for high-priority datasets; overall ≥ 75%.

Notes: Add cohort split (new vs experienced), monitor trends, and combine quantitative KPIs with periodic qualitative surveys (NPS) for completeness.

Approach: Query Postgres information_schema for user tables, left-join your table_descriptions table, and return tables where description is NULL or empty. Exclude system schemas and optionally exclude views or partitions.

-- Lists tables missing descriptions
SELECT
  t.table_schema,
  t.table_name,
  td.description
FROM information_schema.tables t
LEFT JOIN table_descriptions td
  ON td.table_schema = t.table_schema
  AND td.table_name = t.table_name
WHERE t.table_type = 'BASE TABLE'                       -- only real tables
  AND t.table_schema NOT IN ('pg_catalog','information_schema') -- exclude system schemas
  AND (td.description IS NULL OR trim(td.description) = '')  -- missing or empty
ORDER BY t.table_schema, t.table_name;

Key points:

• Use information_schema to discover all tables across schemas.
• table_descriptions is the single source-of-truth for docs; left join finds missing entries.
• Exclude system schemas and choose table_type to avoid views (or include them if you document views too).

Integration into a docs-quality dashboard:

• Schedule the query to run daily via Airflow/cron; store results in a small monitoring table (docs_quality.missing_table_descriptions).
• Dashboard metrics: total tables, tables missing descriptions (count and percent), trend over time, top offending schemas, and age since table creation.
• Add alerting: when percent missing > threshold (e.g., 5%) or critical tables (from a whitelist) lack descriptions.
• UX: link each row to the docs editor or PR template to add descriptions; provide owner/contact field in table_descriptions to route reminders.
• Automate reminders: generate weekly Slack/email digest for table owners with direct links.
• Governance: enforce in CI for new tables—require an entry in table_descriptions before merge.

High-level approach: produce a living “Schema Evolution Runbook” stored in the repo and docs site, plus machine-readable metadata in the Schema Registry. The runbook has four living sections: Compatibility guarantees, Migration steps, Consumer notification & readiness checks, and Rollback plan. Each schema change must include a PR that updates both the registry and runbook entries and passes automated checks.

Compatibility guarantees (documented + enforced)

• State the compatibility mode per schema (Avro/Protobuf): BACKWARD | FORWARD | FULL or NONE. Store this in registry metadata and in the runbook.
• Explicit guarantees: e.g., “BACKWARD: consumers built against vN can read vN+1 produced data” with examples of allowed changes (add optional fields with defaults) and forbidden ones (remove mandatory fields, rename without alias).
• Provide a compatibility checklist and a CLI command to run local schema-compatibility tests (e.g., registry API call or avro-tools).

Migration steps (playbook per change)

• Pre-change: create feature branch, update schema with comments, run unit & integration tests (schema diffs, sample data round-trip), run canary producer/consumer tests against staging registry.
• Dual-write: if changing semantics, produce to new topic (topic-v2) while continuing v1 writes. Use transformers (Kafka Streams/CDC) to populate v2 from v1 for backfilling.
• Backfill: run bounded Spark job to convert historical data, validate counts and checksums, store metrics.
• Cutover: flip consumers in a controlled window (config flag/feature toggle) after readiness checks pass.

Consumer notification & readiness

• Automated consumer registry: require each consumer to register with metadata (team, owner, supported schema versions). This is displayed in docs.
• Notifications: create templated emails + Slack + ticket auto-creation listing change, impact, timeline, migration steps, and required consumer actions.
• Readiness gates: consumers must pass smoke-test consumers (CI job that runs consumer against test topic) and mark readiness in registry UI. Only when all critical consumers green will cutover proceed.
• Provide example migration snippets (e.g., deserialization code with fallbacks, schema alias examples).

Rollback plan (tested, time-boxed)

• Maintain schema immutability: do not delete v1 immediately. If v2 causes regression within the window, toggle producers back to v1 or resume writing to v1 topic (feature flag).
• Consumer-side rollback: deploy previous consumer versions or enable consumer deserialization compatibility fallback.
• Data-level rollback: stop writes to v2, replay v1 data from retention/backup into target topics or restore warehouse partitions from snapshots.
• Postmortem and remediation checklist (logs to collect, checkpoints to rewind, monitoring dashboards to inspect).

Operational artifacts to include in docs

• Schema change template: purpose, owner, semantic version bump, compatibility mode, migration plan, rollback steps, test plan.
• Migration checklist with required signoffs.
• Automated smoke-tests, and links to test reports and dashboards.

Sample semantic versioning policy for schemas

• Format: MAJOR.MINOR.PATCH (e.g., 2.3.1)
• Rules:
  • PATCH: metadata-only changes (docs, description, non-functional metadata). No behavior change. No schema compatibility impact. No consumer action.
  • MINOR: backward-compatible schema changes (add optional fields with defaults, add enum values if allowed by compatibility mode). Bump MINOR. Consumers may adopt at leisure. Producers can deploy after passing CI.
  • MAJOR: any incompatible change (remove or rename required fields, change field meaning, tighten types, change enum value semantics). Bump MAJOR. Requires migration plan, dual-write/backfill, consumer updates, and owner signoff.
• Example: Avro schema v1.4.2 -> add optional field -> v1.5.0 (MINOR). Remove required field -> v2.0.0 (MAJOR).
• Enforce with CI: PR must include new version tag and pass compatibility check against registry; disallow MAJOR without explicit migration-runbook PR template.

Example concise workflow

1. Create schema PR with new version, declare compatibility and bump type.
2. CI: run registry compatibility test, generate schema diff, run smoke consumer tests.
3. Notify consumers via automated channel; require readiness flags.
4. Deploy producers to dual-write if MAJOR; run backfill job.
5. Cutover after readiness; monitor for 2x SLO window.
6. If issues, follow rollback plan (disable v2 writes, replay v1, revert consumers).

This documentation framework makes evolution predictable: machine-enforced checks prevent accidental breaks, the runbook gives teams concrete migration steps, notifications and readiness gates coordinate consumers, and a tested rollback plan minimizes blast radius.

Essential fields for a data-pipeline incident runbook (designed for on-call use), followed by a concise template and which fields should have automated validation.

Essential fields (brief):

• Incident title & severity (S1/S2) — clear triage
• Pipeline name & unique ID
• Affected datasets/tables and downstream consumers
• Owner & escalation contacts (primary, secondary, SME)
• Symptoms & detection signals (alerts, dashboards)
• Quick remediation steps (playbook) with commands/console links
• Safe rollback/stop procedures
• Verification checks post-fix (what to assert)
• Known root causes & historical notes
• Impact & business guidance (who to notify)
• Estimated MTTR and when to declare incident closed
• Links: runbook version, runbook last-tested date, runbook repo/issue
• Postmortem owner & timeline

Runbook template (compact):

• Title: [Pipeline - Job - Component]
• Severity: [S1/S2...]
• Pipeline ID: [...]
• Affected datasets: [...]
• Detection: [Alert name + threshold + metric]
• Immediate actions:
  1. Triage commands/URLs
  2. Restart/stop steps (exact CLI/API with args)
  3. Hotfix steps (if safe)
• Escalation: [Name, Pager, Slack channel]
• Verification: [List of checks to confirm fix]
• Rollback: [When & how]
• Notes/history: [links, prior incidents]
• Post-incident tasks: [cleanup, RCA owner, deadline]

Fields to validate automatically (recommended checks):

• Alert name & threshold mapped to monitoring config (validate existence)
• Pipeline ID matches orchestration system (Airflow job ID / Spark app name)
• Owner contact is up-to-date (on-call rota API)
• Links are reachable (HTTP 200 check)
• Runbook last-tested date within allowed window (e.g., 90 days)
• Verification checks can be executed automatically (smoke tests, row counts, downstream consumer heartbeats)
• Severity-to-SLA mapping consistent with incident policy

Why: Automated validation prevents stale runbooks, ensures runbooks are actionable under pressure, and lets on-call focus on remediation rather than discovery.