Data Governance, Contracts, and Classification Questions

Direct answer

Make the case in terms the deadline-driven decision already respects: frame access controls, lineage, and personally identifiable information (PII) handling as risk-reduction with a cost that grows the longer you wait, not as a nice-to-have quality investment competing with the feature. Then pick a first step that piggybacks on work already happening (tagging sensitivity on tables as they get touched during a planned migration, for instance) rather than asking for a dedicated project, so the habit starts without needing a big upfront commitment.

Structured elaboration

Making the case. Governance work is usually deprioritized because its cost is immediate and visible while its payoff is deferred and probabilistic (an incident that might happen). Reframe it concretely: an access-control gap is a specific future incident-response cost (who has access to what, discovered under pressure during an actual incident, is much more expensive than knowing it in advance); missing lineage is a specific future debugging cost (a bad number surfaces and nobody can say quickly where it came from); ungoverned PII is a specific future compliance and trust cost. The pitch is not "governance is good practice," it is "here is the bill we will pay later, and here is the much smaller bill we can pay now instead."

Picking a low-friction first step. The step that works is one that attaches to something already scheduled rather than requesting new headcount or a dedicated sprint: tagging sensitivity classification on tables as part of a migration that is happening anyway, adding an owner field to the catalog entry every time a new dataset gets created (a one-line addition to an existing creation checklist), or capturing lineage automatically as a side effect of adopting a transformation tool the team is already evaluating. The common thread is that it rides on existing momentum instead of asking for new momentum.

What tends to stick versus what does not. A rule enforced by tooling (a required field in a dataset-creation template, a check that blocks a merge if a new table has no declared owner) tends to persist, because it does not depend on anyone remembering. A rule that depends on a person remembering to do an extra step by hand, however well-intentioned at rollout, tends to decay within a quarter once the person who championed it moves to a different priority.

Worked example

A realistic version of this: a team is migrating a set of tables from one warehouse to another anyway. Instead of treating that as a pure lift-and-shift, the migration script is extended to require a sensitivity tag and an owner field before a table is allowed to land in the new warehouse, using values the team already knows (they are moving the table, so they know what it contains and who owns it). No separate governance project gets proposed or approved; the tagging happens as an unavoidable step of work that was already scheduled. Six months later, new tables created after the migration still carry the tag, not because anyone is manually enforcing it, but because the creation template that grew out of the migration script still requires the field. What stuck was the tooling requirement, not a one-time cleanup effort or a policy document asking people to remember.

Trade-offs & pitfalls

The main risk of the "piggyback on existing work" approach is scope creep in the wrong direction: if the governance addition meaningfully slows down the work it is attached to, it becomes the reason the underlying project gets deprioritized instead of quietly succeeding alongside it, so the addition has to be genuinely small. Making the risk-based case too abstract ("this could cause an incident someday") is easy to deprioritize against a concrete deadline; it needs a specific enough scenario to be memorable, without inventing a specific incident that did not happen. And a first step that only touches new work leaves the existing backlog of undocumented, untagged data untouched, so it needs to be paired, eventually, with a plan for the backlog, even if that plan starts smaller and later.

Backward compatibility means a reader using the new schema can correctly read data written with the old schema. Forward compatibility means a reader using the old schema can correctly read data written with the new schema. Full compatibility requires both at once, so producers and consumers can each upgrade independently, in either order, without breaking. Three concrete strategies that keep old and new consumers working as a schema evolves are: only ever adding nullable or optional fields (never new required ones), never repurposing an existing field's type or meaning, and running automated compatibility checks in CI (continuous integration) before a producer's change ships.

The three compatibility modes

• Backward: new schema reads old data. This lets you evolve the schema and still process previously-written data without a rewrite.
• Forward: old schema reads new data. This lets producers roll out a new schema before every consumer has upgraded, because old consumers keep working against new-format data.
• Full: both hold at once. This is the strictest and safest mode for datasets with many, uncoordinated consumers, because it removes ordering assumptions entirely.

Strategy 1: only ever add nullable or optional fields

Adding a required field breaks any reader that does not know to populate or expect it. Adding an optional field with a sensible default (or that is simply absent when missing) is safe in both directions: old readers ignore it, new readers get a default when it is missing from old data.

Strategy 2: never repurpose an existing field's type or meaning

Changing a field's type (a string ID becoming an integer) or its semantics (a percentage field switching from a 0-1 scale to a 0-100 scale) while keeping the same name is the most dangerous class of change, because it can pass a naive type check while silently corrupting every consumer that assumed the old meaning. If the meaning must change, add a new field with a new name rather than mutating the old one in place.

Strategy 3: run compatibility checks in CI before a producer's change ships

A CI job on the producer's pull request calls the schema registry's compatibility-check endpoint with the proposed schema, evaluated against the subject's configured mode (backward, forward, or full), and fails the build on an incompatible change. This catches the break before merge instead of after a downstream job fails in production.

Parquet and Avro specifics, and where enforcement should live

Avro (a data serialization format commonly used for streaming events) resolves fields by name (with optional aliases) between a writer's schema and a reader's schema, so a rename without an alias breaks resolution even though the field's type is unchanged. Parquet (a columnar file format commonly used in data warehouses and data lakes) stores its own schema per file, so a partitioned table can end up with older files under an older schema footer sitting alongside newer ones; a query engine (or a table format like Iceberg or Delta Lake built for schema evolution) has to reconcile that at read time, and adding a nullable column works cleanly here because older files simply return null for it. Because of this, enforcement should not live only at one point: check it at ingestion (reject or quarantine malformed records), at the processing layer (the CI check above, run against the schema registry or the dbt/SQL model definition), and at the serving layer (the API or dashboard reading the table should not silently accept a shape it was not built for).

Worked example

An Avro event user_signup has fields id (string), email (string), country (string, optional). A developer proposes renaming country to country_code. Under a naive field-name match this is not backward compatible: a reader on the new schema looking for country_code finds nothing in old data and has no default to fall back on. Adding "aliases": ["country"] to the country_code field restores backward compatibility, because Avro's reader can resolve data written under the old field name through the alias. A CI job that runs the registry's compatibility check (in BACKWARD mode) against this change fails before the alias is added and passes after, which is exactly strategy 3 in action.

Trade-offs and pitfalls

Full compatibility is the safest choice for a dataset with many uncoordinated consumers, but it is also the most restrictive: a genuinely breaking change (splitting one field into two) cannot be done in place under full compatibility, and needs a new field or a new schema version alongside the old one rather than an in-place mutation. Backward-only compatibility is common and less restrictive, but it assumes consumers upgrade after producers, which is not always true. The compatibility checker itself only catches structural drift; it cannot catch a field that keeps its type but silently changes meaning, which is exactly why strategy 2 (never repurpose meaning) has to be a human-reviewed rule in the contract, not something automated tooling alone can enforce.

A data contract is an explicit, versioned agreement between the team that produces a dataset or event stream and the teams that consume it, specifying what the data looks like and how it is allowed to change, so consumers can build on it without watching the producer's code. It moves the schema from an implicit "whatever the table currently contains" to a documented interface, the same way an API contract does for a service boundary.

What a data contract specifies

• Schema: the field names, types, nullability, and structure of the dataset or event.
• Semantics: what each field actually means (for example, is amount pre-tax or post-tax, is user_id internal or external).
• Freshness or SLA (service-level agreement): how often new data lands and by when (for example, "table refreshed daily by 6am UTC" or "event latency at the 95th percentile under 5 minutes").
• Ownership: who to contact, who approves changes, who gets paged if it breaks.
• Compatibility rules: what kind of changes are allowed without notice (adding an optional field) versus what requires a migration (renaming or removing a field, changing a type).

Where this shows up for ML features specifically

When the "dataset" is a feature powering a model, the contract checklist picks up a few ML-specific fields on top of the general ones: the feature's computation logic (so training and serving compute it identically), a point-in-time correctness guarantee (no leakage of future data into a training example), a PII (personally identifiable information) or sensitivity flag, and an owner responsible for backfills when the definition changes. Feature stores often store these as machine-readable metadata alongside the feature itself.

Why teams benefit, even from an informal one

Without any contract, consumers reverse-engineer the current shape of a table or event and build against it implicitly; the producer has no idea who is downstream, so a "harmless" rename breaks three pipelines nobody warned. A contract, even a one-page document naming the fields, semantics, freshness, and an owner, converts that into an explicit interface: the producer knows who to notify before a change, consumers know what they can safely rely on, and disputes over "was this expected" get resolved by pointing at a written agreement instead of chat-history archaeology. Contracts also make governance easier, because who owns which sensitive field is written down rather than tribal knowledge.

How this actually gets adopted across teams

A contract that lives only in a wiki page decays quickly. What makes it stick is treating it as an artifact next to the code: check it into the same repository as the producing pipeline, version it, and, once the culture is ready, turn its compatibility rules into an automated CI (continuous integration) check on the producer's build so a breaking change fails the build instead of failing in a downstream dashboard three weeks later. Cross-team adoption grows from a few high-traffic, high-incident datasets first, not a company-wide mandate on day one.

Worked example

A orders.order_placed event contract might read:

event: orders.order_placed
owner: checkout-team (#checkout-eng)
schema:
  order_id: string (required)
  user_id: string (required)
  amount_cents: integer (required)   # semantics: total charged, in cents, post-discount, pre-tax
  currency: string (required, ISO 4217 code)
  placed_at: timestamp (required, UTC)
freshness: emitted within 2 seconds of order placement (95th percentile)
compatibility: backward-compatible only; new fields must be optional; no field renames or type changes without a version bump

A downstream fraud-scoring pipeline reads amount_cents and currency from this contract. If checkout later wants to add a discount_cents field, the compatibility rule says that is fine as an optional addition, but changing amount_cents to represent a pre-tax amount would violate the stated semantics and requires a notified migration, not a silent change.

Trade-offs and pitfalls

Writing and maintaining a contract has real overhead, and for a low-traffic, single-consumer dataset a full formal contract can be more process than the risk warrants; an informal one (a short document with an owner and the five elements above) captures most of the value at a fraction of the cost. The common pitfall is writing the document once and never enforcing it: a contract nobody checks against reality is worse than none, because consumers trust it and get burned when the real data has drifted from what is written. The fix is pairing the contract with even a lightweight CI check, not just a policy page.

Data lineage is the map of which datasets and transformations fed a given dataset, and in what order. A catalog's metadata is the descriptive record of what a dataset is, its owner, schema, classification, business description, and usage stats, regardless of how it got there. Data provenance is the fuller chain-of-custody history of a specific piece of data, including who or what actually touched it and under what process, a stricter, audit-grade record than lineage's pipeline-level map. They are related, since provenance overlaps with lineage, but they answer different questions: lineage answers "what pipeline steps produced this," metadata answers "what is this dataset," and provenance answers "can I prove exactly how this specific data came to be, including anything outside the normal pipeline."

When lineage specifically is the tool

An analyst needs to know which dashboards would break if a source table's column is dropped. This is squarely a lineage question: walk the dependency graph forward from the table to find every downstream job, table, and dashboard that reads it, regardless of who owns any of them or what the data actually means.

When catalog metadata specifically is the tool

A new analyst joins and wants to find "the" table with customer lifetime value before building a report, without knowing the schema or which of several similarly-named tables is the maintained one. This is a metadata question: search the catalog by description and tags, check which table has an owner, a certified or deprecated status, documented column meanings, and recent usage stats, none of which lineage, a dependency graph, or provenance, a custody trail, directly answer.

When the lineage-versus-provenance distinction matters for an incident

A payments table has an unexpected duplicate row discovered during a fraud investigation, and the question is whether it came from a legitimate pipeline retry or an unauthorized manual edit. Lineage shows the table is declared as fed by exactly one job, a nightly extract-transform-load pipeline, which is what the pipeline's design says should be true. That is a statement about the intended data flow, not proof of what actually happened to this specific row. Provenance, if captured as an audit trail of actual write events, who or what process wrote each row, with a timestamp and an actor identity, rather than just the pipeline's declared shape, can show that this particular row was inserted outside the normal pipeline, by a different credential entirely, at an unusual hour on a weekend. Lineage alone, the intended pipeline graph, would not catch this, because the graph describes the designed data flow rather than a forensic record of every actual write; provenance is what closes that gap for an investigation where the question is "prove what actually happened," not "what is supposed to happen."

How the three complement each other in a governance program

Metadata makes a dataset discoverable and tells you whether to trust it in principle, who owns it, is it certified, what is it classified as. Lineage tells you its structural dependencies, so you can do impact analysis and understand the pipeline. Provenance provides the audit-grade proof of what actually happened to specific data, which matters for compliance, security incidents, and any claim you need to defend beyond "the pipeline is supposed to work this way." A mature governance program needs all three because they answer different questions an incident, an audit, or a new analyst will eventually ask, and none of the three substitutes for the other two.

Worked example

A retail company's customer_ltv table is flagged in a compliance review as potentially including data for customers in a jurisdiction with a data-retention deadline. The catalog metadata shows the table is owned by the growth-analytics team and tagged as containing personal data, which is how the compliance team found it in the first place. Lineage shows customer_ltv is built from orders and customer_profile, both themselves fed from further upstream tables, letting the team scope which upstream sources also need a retention check. Provenance, captured as a write-audit log, is what ultimately answers the compliance question precisely: for the specific rows flagged, exactly when each was written or last modified, and by which job run, confirming whether any affected customer's row was actually retained past the deletion deadline, something the lineage graph alone cannot attest to, since the graph says the pipeline is supposed to purge on schedule, while the provenance log is what proves whether a specific row actually did.

Trade-offs and pitfalls

Provenance-grade audit logging, recording every actual write event with an actor identity, is meaningfully more expensive to capture and store than pipeline-level lineage, so most teams do not run it on every table; it is worth reserving for datasets where "prove what actually happened" has real stakes, such as compliance, security, or financial reporting, while lineage and catalog metadata are reasonable to run broadly. The common confusion is treating lineage as if it were provenance, assuming the declared pipeline graph is proof of what happened to the data, when it is really a description of the intended flow.

Data lineage is the recorded trail of where a piece of data came from and every transformation it passed through on the way to where you are looking at it now, a map of what fed into what. It matters for three distinct reasons: it lets you trace a wrong number back to its cause instead of guessing, it lets a stakeholder verify a dashboard's number is trustworthy by seeing its provenance rather than taking it on faith, and it lets you answer a compliance or audit request, "show me everything that fed into this reported figure," without manually reconstructing the pipeline's history.

Why it matters: debugging a wrong number

Without lineage, tracing a bad number backward means manually reading pipeline code and asking around. With it, you follow the recorded chain from the dashboard metric back through each transformation to the source tables, narrowing down where the value diverged from expectation directly, often in minutes instead of hours or days of archaeology.

Why it matters: building trust in a dashboard

A stakeholder who can see, even at a summary level, that a revenue number traces back through a specific, named set of transformations to a specific source table trusts it more than an opaque number with no visible provenance. When the number is questioned, "here is exactly where it comes from" is a much stronger answer than "the pipeline is supposed to be correct."

Why it matters: compliance and audit

Regulators or internal auditors asking "show me what fed into this reported figure" need a defensible, complete answer, and reconstructing that by hand for a complex pipeline is slow and error-prone. Captured lineage turns that into a query rather than an investigation, and provides an auditable record even for figures nobody has looked closely at in months.

Dataset-level versus column-level lineage

Dataset-level lineage tracks dependencies at the table or file level, "table C was built from tables A and B," without saying which specific columns of A and B fed which columns of C. It is cheaper to capture and is enough for coarse-grained impact analysis, such as "if table A changes, table C might be affected."
Column-level lineage tracks the dependency down to individual fields, "column revenue in table C is derived specifically from price and quantity in table A, via this transformation," and not from the other twenty columns table A happens to have. It is more expensive to capture, usually requiring parsing the actual transformation logic rather than just job-level metadata, but it gives a precise answer to "what actually feeds this specific number," which dataset-level lineage can only approximate.

Worked example

An executive dashboard shows quarterly revenue dropped 8% quarter over quarter, and the finance team does not believe it. Dataset-level lineage shows the revenue metric comes from a fct_revenue table, itself built from orders and refunds, which narrows the search to two tables but not further. Column-level lineage shows the dashboard's revenue figure specifically traces to orders.amount_cents minus refunds.refund_amount_cents, aggregated by quarter, and that a transformation two steps upstream started applying a status filter differently after a recent change, causing pending orders to be dropped from the current quarter's total but not the prior quarter's, a data artifact rather than a real revenue drop. Because the lineage was captured at column level, this took one query against the lineage graph instead of re-reading every transformation in the pipeline by hand. The same trace, if requested later by an auditor asking how the reported quarterly revenue figure was derived, is already sitting there as a reusable record rather than something that has to be rebuilt from memory.

Trade-offs and pitfalls

Dataset-level lineage is much cheaper to maintain and is often good enough for impact analysis, such as deciding who to notify before a schema change, so jumping straight to column-level everywhere is over-engineering for teams that mainly need the coarser view; column-level lineage earns its cost specifically for high-stakes, frequently-questioned metrics like revenue or compliance-reported figures. The common pitfall is treating lineage as something you can bolt on retroactively for an audit request; if it is not captured continuously as pipelines run, reconstructing it after the fact for a specific number someone is now questioning is exactly the manual archaeology lineage is supposed to replace.