Data Governance, Contracts, and Classification Questions

A data contract for a shared dataset needs four things at minimum: a schema (field names, types, and nullability), a freshness guarantee (how stale the data is allowed to get before it's a breach), an owner (a name or team, not just a Slack channel), and compatibility rules for how the schema is allowed to change over time. Everything else (retention, backfill guarantees, semantic definitions) is valuable but secondary to those four.

What goes in the contract

• Schema: field name, type, nullability, and a one-line semantic definition per field (not just a type, but what the field MEANS: is amount pre-tax or post-tax, in what currency).
• Freshness SLA: the maximum allowed lag between an event happening and it landing in the dataset, stated as a number with a unit (for example, 15 minutes for a streaming feature, 24 hours for a daily batch table).
• Backfill and retention: how far back the dataset is guaranteed to be correct if a bug is found and fixed, and how long the data is retained.
• Owner: the team accountable for the contract, and their on-call or escalation path when it's violated.
• Compatibility rules: which kinds of changes are allowed silently (adding a new nullable field), which require a version bump and a deprecation window (renaming or removing a field, tightening nullability), and how consumers are notified.

Worked example

A minimal contract for a user_last_activity feature, expressed as a JSON Schema plus the operational metadata a schema alone can't capture:

{
  "dataset": "user_last_activity",
  "owner": "growth-data-eng",
  "freshness_sla_minutes": 15,
  "schema": {
    "type": "object",
    "required": ["user_id", "event_ts", "activity_type"],
    "properties": {
      "user_id": { "type": "string", "description": "internal user id, never the raw email" },
      "event_ts": { "type": "string", "format": "date-time", "description": "UTC timestamp of the activity" },
      "activity_type": { "type": "string", "enum": ["login", "purchase", "view"] },
      "session_id": { "type": ["string", "null"], "description": "added in v2, optional for backward compatibility" }
    },
    "additionalProperties": false
  },
  "compatibility": "backward",
  "deprecation_notice_days": 30
}

additionalProperties: false is doing real work here: it means a producer adding an undeclared field is itself a contract violation the consumer's schema validator will catch, not just a removal or a type change.

Trade-offs and pitfalls

A contract that's too detailed (every field individually versioned, every consumer's exact query pattern encoded) becomes something nobody maintains and it silently goes stale, which is worse than no contract because it creates false confidence. A contract that's too thin (schema only, no freshness SLA and no owner) doesn't actually prevent the two most common real incidents: silent staleness and nobody picking up the page when it breaks. Also watch for additionalProperties: false being set on a schema whose producer team doesn't actually run CI validation against it: an unenforced contract is a wiki page with extra syntax highlighting.

Data lineage is the recorded trail of where a piece of data came from and every transformation it passed through on the way to where you are looking at it now, a map of what fed into what. It matters for three distinct reasons: it lets you trace a wrong number back to its cause instead of guessing, it lets a stakeholder verify a dashboard's number is trustworthy by seeing its provenance rather than taking it on faith, and it lets you answer a compliance or audit request, "show me everything that fed into this reported figure," without manually reconstructing the pipeline's history.

Why it matters: debugging a wrong number

Without lineage, tracing a bad number backward means manually reading pipeline code and asking around. With it, you follow the recorded chain from the dashboard metric back through each transformation to the source tables, narrowing down where the value diverged from expectation directly, often in minutes instead of hours or days of archaeology.

Why it matters: building trust in a dashboard

A stakeholder who can see, even at a summary level, that a revenue number traces back through a specific, named set of transformations to a specific source table trusts it more than an opaque number with no visible provenance. When the number is questioned, "here is exactly where it comes from" is a much stronger answer than "the pipeline is supposed to be correct."

Why it matters: compliance and audit

Regulators or internal auditors asking "show me what fed into this reported figure" need a defensible, complete answer, and reconstructing that by hand for a complex pipeline is slow and error-prone. Captured lineage turns that into a query rather than an investigation, and provides an auditable record even for figures nobody has looked closely at in months.

Dataset-level versus column-level lineage

Dataset-level lineage tracks dependencies at the table or file level, "table C was built from tables A and B," without saying which specific columns of A and B fed which columns of C. It is cheaper to capture and is enough for coarse-grained impact analysis, such as "if table A changes, table C might be affected."
Column-level lineage tracks the dependency down to individual fields, "column revenue in table C is derived specifically from price and quantity in table A, via this transformation," and not from the other twenty columns table A happens to have. It is more expensive to capture, usually requiring parsing the actual transformation logic rather than just job-level metadata, but it gives a precise answer to "what actually feeds this specific number," which dataset-level lineage can only approximate.

Worked example

An executive dashboard shows quarterly revenue dropped 8% quarter over quarter, and the finance team does not believe it. Dataset-level lineage shows the revenue metric comes from a fct_revenue table, itself built from orders and refunds, which narrows the search to two tables but not further. Column-level lineage shows the dashboard's revenue figure specifically traces to orders.amount_cents minus refunds.refund_amount_cents, aggregated by quarter, and that a transformation two steps upstream started applying a status filter differently after a recent change, causing pending orders to be dropped from the current quarter's total but not the prior quarter's, a data artifact rather than a real revenue drop. Because the lineage was captured at column level, this took one query against the lineage graph instead of re-reading every transformation in the pipeline by hand. The same trace, if requested later by an auditor asking how the reported quarterly revenue figure was derived, is already sitting there as a reusable record rather than something that has to be rebuilt from memory.

Trade-offs and pitfalls

Dataset-level lineage is much cheaper to maintain and is often good enough for impact analysis, such as deciding who to notify before a schema change, so jumping straight to column-level everywhere is over-engineering for teams that mainly need the coarser view; column-level lineage earns its cost specifically for high-stakes, frequently-questioned metrics like revenue or compliance-reported figures. The common pitfall is treating lineage as something you can bolt on retroactively for an audit request; if it is not captured continuously as pipelines run, reconstructing it after the fact for a specific number someone is now questioning is exactly the manual archaeology lineage is supposed to replace.

Direct answer

A lightweight governance policy for a mid-size company should cover four areas: data ownership, schema-change approvals, personally identifiable information (PII) tagging, and lifecycle and retention. To make it stick, encode each one as a check that tooling enforces automatically (a pipeline blocked from deploying, a table blocked from being created) rather than a rule written on a wiki page that depends on people remembering to follow it.

Structured elaboration

Data ownership. Every dataset has a named owner, recorded in the catalog at creation time, not added later if someone asks. Operationalize it by making the owner field a required, non-empty field in the dataset-creation tooling (a table creation script or Terraform module that will not run without it), rather than a policy statement that ownership "should" be recorded.

Schema-change approvals. A breaking schema change (removing a field, tightening nullability, changing a type) requires review before it ships, while an additive, backward-compatible change (adding a new nullable field) can go through automatically. Operationalize it with a CI check that classifies the change (breaking versus additive) by diffing the new schema against the previous one, and only routes breaking changes to a required human approval step, so the policy is enforced by the pipeline, not by hoping the engineer remembers to ask.

PII tagging. Any field containing personally identifiable information (PII), directly identifying (name, email) or indirectly identifying in combination with other fields, is tagged as such in the catalog, and that tag drives masking or access restriction automatically. Operationalize it with an automated PII-detection scan (pattern matching on column names and sampled values) that runs on every new table and flags likely PII for a human to confirm, rather than relying on the table's creator to remember to self-report it.

Lifecycle and retention. Every dataset has a stated retention period, driven by its sensitivity classification and any PII it contains, after which it is automatically archived or deleted. Operationalize it with a scheduled job that reads the retention field from the catalog and acts on it, rather than a policy that retention "should" be respected with no automated enforcement.

Worked example

A new table, signup_events, is created. The creation tooling requires an owner field (the growth team lead is entered) before the table can be provisioned at all. The automated PII scan flags the email column as likely PII based on its name and sampled values; a human confirms it, and the catalog tags the column, which triggers an automatic column-level masking policy for any role without an explicit PII-access grant. Because the table contains PII, its default retention (driven by the sensitivity/PII policy) is set to 18 months rather than the standard non-PII default of 5 years, and a scheduled job will archive rows past that age without anyone needing to remember to run a manual cleanup.

Trade-offs & pitfalls

Automated PII detection based on column names and sampled values produces both false positives (a column named user_ref that is actually an internal, non-identifying ID) and false negatives (a free-text notes field that occasionally contains a customer's name typed in by a support agent, which pattern matching on the column as a whole will likely miss), so it needs a human-review step, not full automation, and the review step is exactly where deadline pressure tends to erode the policy if it becomes a bottleneck. Schema-change classification (breaking versus additive) can also misjudge a case where an additive-looking change actually breaks a downstream consumer's business logic even though it does not break the schema mechanically, tooling catches structural breakage, not semantic breakage. And a retention policy enforced by a scheduled deletion job is unforgiving of a mistake, if a dataset is mis-tagged as containing less-sensitive data than it does, the shorter automated deletion the mis-tag implies can be the safer failure mode, but a longer-than-intended retention on genuinely sensitive data is the more dangerous one, so retention defaults should err conservative when tagging confidence is low.

Direct answer

A practical analyst-facing governance checklist has five items: metric definitions, dataset ownership, access controls, lineage tracking, and periodic audits. For each, the analyst's job is to consume, flag, and enforce at the point of use, while data engineering's job is to build and maintain the underlying system; the split is roughly "engineering builds the pipe, analyst is accountable for what flows through the dashboards they own." When a stakeholder says a dashboard "lacks traceability," the constructive response is to reframe it as a specific, answerable question rather than a vague complaint.

Structured elaboration

Where analyst-engineering collaboration is tightest: sensitivity tags. Sensitivity classification (public, internal, confidential, restricted, or similar tiers) is not something engineering can assign alone, because engineering often does not know the business meaning of a field, and it is not something an analyst can assign alone, because the enforcement (masking, row-level restriction) has to be implemented in the pipeline. The practical split is: the analyst proposes or confirms the sensitivity tag for a field based on what it actually represents (is this column a proxy for a protected characteristic, does it contain contact information), and engineering implements the resulting control and keeps the tag attached as the field flows into derived tables.

Pushing back constructively on "lacks traceability": treat the complaint as underspecified and ask what "traceability" means to the stakeholder concretely, usually one of three things: (1) "I want to see which source tables feed this number," answered by pointing at the lineage graph or model documentation; (2) "I don't trust this number matches another report," answered by pointing at the shared metric definition and where the two reports diverge; or (3) "I want to know who to ask when this changes," answered by pointing at the dataset's documented owner. Responding with "here's exactly where that comes from" (even if the honest answer is "this specific piece isn't tracked yet, and here's the ticket to fix it") is more credible than treating the comment as an attack on the dashboard's quality.

Worked example

A stakeholder says a churn dashboard "lacks traceability." The analyst asks what specifically feels untraceable, and the stakeholder clarifies they saw a different churn number in another report last quarter. The analyst pulls up the metric definition in the shared semantic layer: this dashboard uses the definition "canceled subscription within 30 days," the other report (built before the definition was centralized) used a different window. That is a real, previously undocumented metric-definition mismatch, not a vague trust issue. The analyst logs it, proposes standardizing on one definition, and gets data engineering to update the shared model so both reports pull from the same source going forward.

Trade-offs & pitfalls

Treating the checklist as a one-time setup rather than an ongoing practice is the most common failure: sensitivity tags and ownership records drift as fields get reused for new purposes, and periodic audits are the only thing that catches that drift, so skipping them under deadline pressure quietly reintroduces the exact gaps the checklist exists to close. Analysts sometimes over-index on lineage tracking as purely engineering's problem and stop citing it when explaining a number, which erodes the trust the lineage work was meant to build. Conversely, engineering sometimes assigns sensitivity tags without analyst input to move faster, which produces technically-present but business-meaningless classifications that fail an actual audit.

Across a mixed BI (business intelligence) stack, no single capture method covers everything, so the practical approach combines at least two: parsing dbt's (a SQL-based data transformation tool) own lineage graph for anything modeled in dbt, SQL query parsing for ad-hoc queries run directly against the warehouse from the BI tool, and runtime metadata capture, logging actual read and write events, for Python scripts and Airflow-orchestrated ETL (extract, transform, load) jobs that are not expressible as parseable SQL. Surfacing a "why is this number X" trace for a business user means walking that combined lineage graph from the dashboard's metric definition down through the fact and dimension tables to the source events, and presenting it as a readable chain rather than the raw graph.

Capturing lineage across each part of the ecosystem

• Airflow-orchestrated ETL: capture at the task level via Airflow's (a workflow orchestration tool) lineage hooks, recording which task read which input and wrote which output, giving the coarse backbone of which pipeline stage touched which table.
• Ad-hoc SQL run directly in the BI tool: this is the trickiest case, since it bypasses the orchestrated pipeline entirely. SQL query parsing is the fit here: intercept or log the query text the BI tool sends to the warehouse, most warehouses expose a query history or audit log, and parse it the way a static SQL lineage tool would, extracting which tables and columns the query reads to produce the report or chart.
• Python scripts writing to the warehouse: these usually are not parseable SQL, so runtime metadata capture is the fit: the script, or a lightweight wrapper it calls, explicitly emits a lineage event describing what it read and wrote as it executes.
• Anything modeled in dbt: dbt's own manifest already encodes the model-to-model dependency graph from its reference and source declarations, and dbt's compiled SQL can be parsed for column-level detail, so dbt-managed transformations get lineage largely for free without a bespoke capture method, which is why it is worth naming as its own method distinct from generic SQL parsing.

Combining them

Each of the above methods emits events into one shared lineage store using a common event format, source, target, transformation, timestamp, regardless of which capture method produced it, so the resulting graph spans Airflow tasks, dbt models, ad-hoc BI queries, and Python scripts as one connected structure rather than several disconnected fragments a business user would have to mentally stitch together.

Surfacing a "why is this number X" trace for a business user

Start from the metric as defined in the BI tool, the aggregation and filter logic that turns fact-table rows into the number on screen, and walk backward: the metric definition traces to specific columns in a fact table, which trace through the fact table's build to the dimension tables joined into it, and further back through however many transformation steps the lineage graph records, down to the raw source events that ultimately fed all of it. Present this as a linear, plain-language chain for a non-technical user, for example "this number equals the sum of order amounts, filtered to active customers, from orders placed in the second quarter, sourced from the checkout event stream," rather than the raw dependency graph, with an option to expand any step for someone who wants the underlying SQL or transformation detail. Critically, the trace should also surface when the metric's definition itself changed, for example if "active customer" was redefined last quarter to use a longer lookback window, since a metric-definition change can move a number just as much as a data change can, and a business user asking "why did this change" needs both possibilities distinguished, not just the data lineage.

Worked example

A sales executive asks why "active customers" dropped 15% quarter over quarter on the dashboard. The trace starts at the metric definition: "active customers equals the count of distinct customers with an order in the trailing 90 days," which itself changed from a trailing 30-day window two months ago, visible because the metric's definition is versioned in the BI tool's semantic layer. That alone is a plausible full explanation, since a stricter window produces a smaller count on otherwise-identical data, so the trace surfaces the definition change prominently, dated, before walking any further into the data. The team also checks the data-lineage side to rule out a second cause: the metric traces to fct_orders, built by a dbt model giving exact column lineage for free, which joins dim_customer, populated by an Airflow-orchestrated ETL job, fed ultimately by the checkout event stream. No anomaly is found there; the ad-hoc query the BI tool runs directly against fct_orders for this specific dashboard tile is confirmed, via SQL-parsing of the warehouse's query log, to match the documented metric definition. The answer presented to the executive: the drop is fully explained by the intentional definition change to a stricter 90-day window, not a data quality issue, with the trace itself as the evidence.

Trade-offs and pitfalls

Ad-hoc SQL run directly against the warehouse from a BI tool is the hardest part of this ecosystem to keep in the lineage graph, since it happens outside any orchestrator; relying only on periodic query-log parsing means there is a lag between a new ad-hoc report going live and it appearing in the graph, and a query the BI tool caches instead of re-issuing might not show up in the log at all. The pitfall specific to this "why is this number X" use case is tracing only the data path and never checking whether the metric's own definition changed. A perfectly correct data trace can still mislead a business user if the real cause was a silent redefinition of what the metric means, so the trace has to surface metric-definition history as a first-class part of the answer, not an afterthought.