Data Governance, Contracts, and Classification Questions

Direct answer

Build a federated catalog layer that harvests metadata from each provider's native catalog (AWS Glue Data Catalog, Google Cloud's Data Catalog, and Azure Purview, now branded Microsoft Purview) into one normalized index, treat one provider's tag as authoritative source of truth per dataset for ownership and sensitivity rather than trying to keep three independently-editable copies in sync, and resolve description conflicts with an explicit, ranked rule set rather than a silent overwrite.

Structured elaboration

Harvesting into a normalized model. Each cloud catalog (AWS Glue Data Catalog, Google Cloud's Data Catalog, and Azure Purview) has its own schema and terminology for the same underlying concepts (a "table" and a "database" mean roughly the same thing across all three, but field names differ). A harvester per provider maps each into one normalized schema (dataset, owner, sensitivity tag, technical schema, lineage) so the unified search layer queries one consistent model instead of three different ones.

Keeping ownership and sensitivity tags synchronized. The reliable pattern is single-source-of-write per dataset: whichever cloud actually hosts a given dataset is the authoritative source for its native catalog metadata, and the unified layer treats that provider's tag as the source of truth, propagating it outward read-only to the unified index rather than allowing edits in the unified layer to be independently pushed back into three systems (which is where sync conflicts actually originate). A scheduled re-harvest (not a one-time sync) keeps the unified index from drifting from the source of truth as the underlying tag changes.

Resolving conflicts when the same dataset is described differently in two systems. This mainly comes up when a dataset is mirrored or referenced across providers (a table replicated from AWS into a BigQuery-based warehouse, cataloged separately in both). The resolution rule needs an explicit precedence, for example: the system that owns the canonical write path for the data wins for sensitivity classification (since that determines who is actually allowed to touch the source data), while the most recently updated description wins for the human-readable definition (since a fresher definition is more likely to be accurate). Automatic resolution should flag, not silently overwrite, a conflict where the two systems' sensitivity tags actually disagree, since that specific disagreement (one system says internal, another says confidential, for the same underlying data) is a real risk signal worth a human look, not just noise to average away.

flowchart TD
    A[AWS Glue Data Catalog] --> D[Federated search index]
    B[Google Cloud Data Catalog] --> D
    C[Azure Purview] --> D
    D --> E[Conflict resolution + merge rules]
    E --> F[Unified ownership and sensitivity record]
    F --> A
    F --> B
    F --> C

Worked example

A customer_events dataset exists natively in AWS (source of truth, tagged confidential in AWS Glue Data Catalog because it contains raw device identifiers) and is also mirrored into a Google BigQuery dataset for a separate analytics team, cataloged there as internal because the mirroring team was not aware of the sensitivity of the source columns. The unified catalog's harvesters pull both records. The conflict-resolution rule (canonical source wins for sensitivity) flags this: the mirrored copy's internal tag disagrees with the source's confidential tag for what is fundamentally the same data. Rather than silently picking one, the unified catalog surfaces the mirrored dataset as confidential (inherited from source, conflicts with local tag) and opens a review task for the mirroring team, which is the actual finding this whole system exists to catch.

Trade-offs & pitfalls

Treating "most recently updated wins" as the rule for sensitivity tags (instead of only for descriptions) is a common and dangerous shortcut, it lets a careless recent edit silently downgrade a genuinely sensitive dataset's classification. Harvesting on a schedule rather than via native change events introduces a lag window where the unified catalog can show stale information, which matters most exactly when a sensitivity tag has just been tightened for a real reason. And normalizing three providers' different metadata models into one schema always loses some provider-specific nuance, a Purview-specific classification label with no clean AWS or Google Cloud equivalent has to be mapped to the nearest normalized tier, which is a lossy translation the unified layer needs to document, not hide.

Build one governance model with region-aware enforcement rather than two separate governance systems for the EU (European Union) and the US: a shared classification taxonomy and shared access-control mechanism, with region-specific rules (residency, retention, consent basis) expressed as policy parameters rather than as forked infrastructure, spanning product analytics, the ML (machine learning) feature store, and customer records.

Data classification

A single four- or five-tier classification (public, internal, confidential, restricted, and, given the regulatory scope here, a distinct regulated-PII sub-tier for data covered by GDPR (the EU's General Data Protection Regulation) or CCPA (the California Consumer Privacy Act) style rules) applies globally, with a region tag attached alongside the sensitivity tag. A customer record's sensitivity tier doesn't change between the EU and the US, but its region tag determines which residency and retention rules apply to it.

PII detection and masking

Automated PII (personally identifiable information) detection runs across all three domains, product analytics events, ML feature store inputs, and customer records, using a combination of schema-level tagging (known PII fields declared at ingestion) and pattern-based scanning (catching PII that lands in free-text or unexpected fields). Masking defaults follow the classification tier: confidential and restricted fields are masked by default in any cross-team or cross-region view, unmasked only for roles with an approved, region-appropriate business need.

Role-based access control

RBAC (role-based access control) roles are defined once, globally (analyst, ML engineer, finance, compliance), with region as an additional attribute checked alongside the role, effectively a lightweight attribute-based layer on top of RBAC rather than a fully separate ABAC (attribute-based access control) system: an EU-based analyst role and a US-based analyst role have the same permissions structure, but an EU analyst's access to EU customer records is subject to the residency rule that data doesn't leave the EU region's processing boundary, even for another employee with the same job title based in the US.

Lineage and audit trails

End-to-end lineage tracking (capturing how a customer record or an ML feature traces back to its source and forward to every derived dataset or model that consumes it) is essential here specifically because a right-to-erasure or data-subject-access request under GDPR requires being able to answer "everywhere this person's data went," not just where it started. Audit trails record every access, transformation, and cross-region data movement, since cross-region movement is itself a regulated event under most residency regimes, not just another access to log.

SOC 2 and consent-management framing

Beyond the GDPR/CCPA distinction, this architecture should also be evaluated against SOC 2 (System and Organization Controls 2, an audit standard for how a service organization manages customer data) trust-service criteria, since access control, change management, and monitoring requirements from a SOC 2 audit map directly onto the same RBAC, audit-logging, and lineage components described above; building governance to satisfy GDPR/CCPA residency rules largely satisfies SOC 2's security and confidentiality criteria as a byproduct, but SOC 2 additionally expects documented, tested incident-response and change-management processes around this infrastructure, which should be built explicitly rather than assumed.

Consent management is the piece that ties classification to actual legal basis: a customer's consent state (what they've agreed their data can be used for, which can differ between EU and US customers under different regimes) is stored as its own governed record linked to the customer's identifier, and the access-control layer checks consent state, not just role and region, before allowing a customer record into an ML feature pipeline. A pseudonymization/tokenization architecture variant supports this concretely: customer records feeding the ML feature store are pseudonymized (the raw identifier replaced by a stable token) at ingestion, with the reversible mapping held in a separate, tightly-restricted vault, so a consent withdrawal can be enforced by revoking or expiring that customer's token without needing to hunt down every derived feature-store row that used the raw identifier.

Governance decision-making body

A standing governance council, not a single owner, since region-specific legal obligations and cross-functional trade-offs (ML freshness versus residency, self-service speed versus compliance) need multiple perspectives represented:

• Chair: a data governance or privacy lead, owning the overall policy and cadence.
• Legal/privacy counsel for each covered region (EU and US), since the specific obligations differ.
• A security lead, owning encryption, key management, and the SOC 2 control set.
• An ML engineering representative, owning the feature-freshness and pseudonymization trade-offs.
• A product/business representative, ensuring policy decisions account for actual business need, not compliance in a vacuum.

This body meets on a regular cadence (for example monthly, with an expedited path for urgent decisions like a new region's regulatory change) to approve new classification categories, review flagged cross-region data movements, and adjudicate exception requests.

flowchart TB
    subgraph EU[EU processing boundary]
        EUDATA[EU customer records + features]
    end
    subgraph US[US processing boundary]
        USDATA[US customer records + features]
    end
    EUDATA -->|pseudonymized token only, consent-checked| SHARED[Shared aggregate / model artifacts]
    USDATA -->|pseudonymized token only, consent-checked| SHARED
    CONSENT[(Consent record store)] --> RBAC[Region-aware RBAC layer]
    RBAC --> EUDATA
    RBAC --> USDATA
    GOV[Governance council] -->|policy + exception review| RBAC
    GOV --> CONSENT

Keeping ML feature freshness and developer agility workable

The residency and consent checks above add a lookup on the path from raw customer record to a live feature, which risks slowing feature-store updates if done naively. Keep it workable by:

• Checking consent and residency at write-time into the feature store (when a feature is computed), not at read-time on every serving request, so the low-latency serving path stays a simple key lookup with no per-request policy evaluation.
• Giving developers a self-service, pre-approved pattern (a feature-engineering template that already routes through pseudonymization and consent checks correctly) for the common case, so most new features don't require a governance council review individually, only genuinely new data sources or use cases do.

Trade-offs & pitfalls

The biggest risk in this design is a fast, informal "just this once" cross-region data copy for an urgent business need that bypasses the residency boundary; because the architecture makes the compliant path only slightly slower than the non-compliant one (pre-approved templates, write-time checks), there should be little practical incentive to skip it, but the governance council needs visibility into attempted or actual cross-region movements specifically to catch the cases where someone tries anyway.

Design the workspace boundary as the primary isolation control (one tenant's data physically or logically cannot be queried from another tenant's context, regardless of the labeler's role), layer role-based access and audit logging inside each workspace, and measure labeling quality with inter-rater agreement computed entirely within a tenant's own boundary so quality checks never require cross-tenant visibility.

Private per-tenant workspaces

Each tenant's documents, labeling tasks, and annotator assignments live in a workspace scoped by tenant ID, enforced at the data-access layer, not just the application's user interface. Concretely: every query the labeling backend issues includes the tenant ID as a mandatory predicate (similar to a row-level security filter), so even a bug in the application layer's routing logic can't surface tenant B's documents inside tenant A's labeling queue, because the underlying data layer would reject or empty-return a query missing or mismatching the tenant filter. Storage should also be logically or physically partitioned per tenant (separate schemas, buckets, or encryption keys per tenant) so a single compromised credential's blast radius is bounded to one tenant.

Role-based access

Within a workspace, define roles distinct from the tenant boundary itself:

• Labeler: can view and label documents assigned to them, cannot see other labelers' in-progress work or the adjudicated "gold" answer.
• Reviewer/adjudicator: can see multiple labelers' submissions for the same document to resolve disagreements, cannot access documents outside their assigned tenant.
• Workspace admin: manages labeler assignments and views workspace-level (not cross-tenant) quality metrics.

Role assignments are always tenant-scoped: a person who is a reviewer for tenant A's workspace has no implicit access to tenant B's workspace, even if they hold the same job title; access is granted per workspace explicitly, not inherited from a global role.

Audit logging

Every document view, label submission, and adjudication decision is logged with the labeler's identity, the tenant ID, the document ID, and a timestamp, in an append-only log. Logs themselves are also tenant-partitioned so that an audit or compliance review for tenant A never requires exposing tenant B's activity, and a labeler under investigation for one tenant's work doesn't reveal information about their activity in another tenant's workspace as a side effect.

Checking labeling quality without cross-tenant leakage

Inter-rater agreement (a measure of how consistently multiple labelers agree on the same item) is computed entirely inside each tenant's boundary: assign a subset of each tenant's own documents to multiple labelers within that same tenant's workspace, and compute agreement only over that tenant's own labeled set. No pooling of documents or labels across tenants is needed to get a meaningful signal, because agreement is a property of a labeler's consistency within a document population, not something that requires comparing across tenants.

A common agreement measure is Cohen's kappa for two raters, which corrects for the chance agreement expected even from random labeling:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed proportion of items the two raters agreed on, and $p_e$ is the proportion expected to agree by chance given each rater's own marginal label frequencies.

Worked example. Suppose within one tenant's workspace, two labelers each classify the same 100 documents as "sensitive" or "not sensitive," producing this confusion matrix:

Observed agreement:
$p_o = \frac{40 + 45}{100} = 0.85$

Expected agreement by chance, from each rater's marginal totals:
$p_e = \left(\frac{45}{100} \cdot \frac{50}{100}\right) + \left(\frac{55}{100} \cdot \frac{50}{100}\right) = 0.225 + 0.275 = 0.5$

Kappa:
$\kappa = \frac{0.85 - 0.5}{1 - 0.5} = \frac{0.35}{0.5} = 0.7$

A kappa of 0.7 falls in the "substantial agreement" range on the commonly used Landis and Koch scale (0.61 to 0.80), meaning the two labelers agree well beyond what chance alone would produce, without ever needing to see or compare data from any other tenant to compute it.

flowchart TB
    subgraph TenantA[Tenant A workspace]
        DA[Tenant A documents] --> LA1[Labeler 1]
        DA --> LA2[Labeler 2]
        LA1 --> QA[Agreement check: Tenant A only]
        LA2 --> QA
    end
    subgraph TenantB[Tenant B workspace]
        DB[Tenant B documents] --> LB1[Labeler 1]
        DB --> LB2[Labeler 2]
        LB1 --> QB[Agreement check: Tenant B only]
        LB2 --> QB
    end

Trade-offs & pitfalls

The most common failure is a labeler pool shared across tenants for cost efficiency, without the workspace and query-layer isolation described above; sharing labelers is fine (the same person can work for two tenants), but sharing the underlying query surface without a mandatory tenant filter is the actual leak vector, since it relies on the application UI never having a routing bug rather than the data layer structurally preventing the wrong result.

Direct answer

Evaluate policy at every layer where a decision actually gets made (ingest, the catalog, and the query gateway), each with its own policy-enforcement point (PEP, the component that intercepts a request and asks the policy engine for a decision) calling a shared policy engine in the style of Open Policy Agent (OPA, an open-source engine that evaluates access and validation rules written in a dedicated policy language, Rego, against structured input), roll out policy changes the same way as code (through version control, tests, and a staged rollout), and audit enforcement by logging every decision the engine actually made, not just what policies are currently written.

Structured elaboration

Where policies get evaluated.

• At ingest: a new dataset or a new batch of records is checked against structural and classification policy before it lands (does this table have a required owner and sensitivity tag; does this batch violate a schema contract) so ungoverned data cannot enter the lake in the first place.
• In the catalog: policy governs metadata itself, for example, whether a dataset can be marked "production-ready" without a confirmed owner and passing quality checks, gating the catalog's own state transitions.
• At a query gateway: every read is checked against access policy (does this identity have permission for this table, this column, or this row) at the moment of the query, which is the enforcement point that actually protects data in use, independent of what happened at ingest.

Evaluating at all three matters because they catch different failure classes: ingest-time policy stops bad data from entering, catalog-time policy stops it from being labeled trustworthy, and query-time policy stops it from being read by someone who should not see it, even if the first two steps were somehow bypassed.

Rolling out policy changes safely. Treat policy source (Rego files, in OPA's case) exactly like application code: version-controlled, reviewed via pull request, and tested against a suite of example inputs with expected allow or deny outcomes before merge. Roll a changed policy out in a dry-run or shadow mode first, where the engine evaluates the new policy and logs what it would have decided without actually enforcing it, so a team can compare the new policy's decisions against the old one's on real traffic before switching enforcement over, catching an overly broad or overly restrictive policy change before it affects anyone.

Auditing what was actually enforced versus merely written. The policy source in version control tells you what is supposed to happen; it does not tell you what actually happened. Every policy-enforcement point logs each decision (the input, the policy version evaluated, and the allow or deny outcome) to an immutable decision log. An audit compares the two: query the decision log for a specific access event to confirm which exact policy version made the call, and separately check whether a policy that exists in the repository was actually deployed and wired to an enforcement point at all, since a written-but-not-connected policy provides zero real protection while looking, on paper, like coverage exists.

flowchart TD
    A[Policy repo, Rego source] --> B[CI policy tests]
    B --> C[Policy bundle registry]
    C --> D[Ingest-time PEP]
    C --> E[Catalog-time PEP]
    C --> F[Query gateway PEP]
    D --> G[Decision log]
    E --> G
    F --> G
    G --> H[Audit: enforced vs written]

Worked example

A new policy is written requiring any table tagged restricted to deny row-level access to anyone outside the compliance role. It is added to the policy repository, tested against a suite of example queries (a compliance-role user against a restricted table: expect allow; a non-compliance user against the same table: expect deny), and merged. It is deployed first in shadow mode at the query gateway for one week: the engine logs what it would have decided on real query traffic without blocking anything. The shadow log shows two non-compliance users who would have been denied under the new policy but are currently active daily users of that table, a signal the rollout catches before enforcement, prompting a review of whether those two need a role change or the policy needs a documented exception, rather than discovering the break only after enforcement went live and those users were suddenly locked out.

Trade-offs & pitfalls

Evaluating policy at three separate points (ingest, catalog, query gateway) means three separate integrations to build and keep correctly wired, and it is easy for one of them, commonly the catalog-time check, to be treated as optional and quietly skipped under deadline pressure, leaving a real gap that only the audit process would catch, and only if someone runs it. Shadow-mode rollout adds calendar time before a policy actually protects anything, which is a real cost when the policy exists because of an urgent compliance gap. And a decision log that is comprehensive but never actually reviewed provides the appearance of auditability without the substance, the audit step itself needs to be a scheduled, owned task, not a capability that exists in theory.

Direct answer

Build one central catalog and lineage graph fed by automated harvesters from each source system (the lake, the warehouse, and the ML artifact store), with access control and sensitivity tags attached at the dataset and column level as first-class catalog fields rather than a bolt-on, and retention and deletion policy driven off those same tags so a classification change propagates its consequences automatically. Treat the whole thing as a governance program, not just infrastructure: track adoption (searches, dataset views, and completed access requests through the catalog) so you can tell whether teams are actually using it versus tolerating its existence.

Structured elaboration

Architecture. A metadata harvester runs against each source (lake file listings and table formats, warehouse system tables, the ML artifact store's model and dataset registry), pushing technical metadata (schema, size, last-modified, detected lineage from query and job logs) into a central catalog. The catalog stores this alongside curated business metadata and exposes both through one search surface, regardless of where the underlying data physically lives.

Automatic versus hand-curated metadata. Anything mechanically derivable is automated: schema, row counts, freshness, and lineage edges reconstructed from SQL and pipeline job graphs. Anything requiring business judgment is curated by the dataset's owner: what the dataset means, who to contact, and the sensitivity classification of each field. Automation without curation gives you an accurate but meaningless inventory; curation without automation gives you documentation that goes stale within a quarter. The system needs both, with automation covering the majority so curation effort concentrates on the judgment calls.

Access controls and sensitivity tags. Every dataset and, where it matters, every column, carries a sensitivity tag (for example public, internal, confidential, restricted). Access control policy is defined against the tag, not against the individual dataset, so a newly classified table inherits the correct policy immediately instead of waiting for someone to configure access rules by hand. When a table is derived by joining a confidential source with a public one, the derived table inherits the higher (more restrictive) classification by default.

Retention and deletion hook. The same sensitivity tag (plus a data-category tag, such as "contains personally identifiable information," PII) drives a retention policy: a restricted-and-PII dataset gets a shorter, enforced retention window with automated deletion, while an internal-analytics-only dataset follows a longer default. This only works because the tag lives in the catalog and the retention job reads it from there, rather than retention rules being hand-maintained per dataset.

Automated-lineage-plus-RBAC-plus-quality-enforcement as a rollout. Practically, this ships in that order: get automated lineage and cataloging working first (so there is something to attach policy to), then layer access control enforcement on top of the now-populated catalog, then add automated data-quality checks that block a dataset from being marked "production-ready" in the catalog until they pass. Rolling out RBAC before the catalog exists means access decisions have nothing reliable to key off of.

The same system as a governance program (metric consistency, discoverability, ownership). Beyond the technical architecture, this is also a program with three organizational goals: metric consistency (the same business number is defined once, not recomputed differently by five teams), discoverability (someone new to the org can find the right dataset without asking around), and ownership (every dataset has a named, accountable owner, not an implicit one). The catalog is the shared surface those three goals get measured against.

flowchart TD
    A[Data lake] --> M[Metadata harvesters]
    B[Warehouse] --> M
    C[ML artifact store] --> M
    M --> D[Central catalog + lineage graph]
    D --> E[Access control / sensitivity tags]
    D --> F[Retention and deletion policy hook]
    D --> G[Curated business metadata]
    D --> H[Adoption metrics]

Measuring adoption versus tolerance. Track catalog searches per week, the fraction of active datasets with a confirmed (not default) owner, and the fraction of access requests that go through the catalog's workflow versus a side channel (a Slack message to someone who "just has access"). A catalog that exists but where most access still happens outside it is being tolerated, not adopted.

Trade-offs & pitfalls

The harvesters covering three different source types (lake, warehouse, ML artifact store) each need their own integration, and keeping all three in sync in near-real time is genuinely hard; a common failure is one source's harvester silently falling behind, so the catalog looks complete but is quietly stale for that source. Deriving a joined table's classification as "the more restrictive of its inputs" is the right default but is not always correct (an aggregation can sometimes de-sensitize data, for example, a table of counts by region does not carry the same restriction as the row-level source), so the automatic inheritance rule needs a documented, reviewable override path. Measuring adoption by search volume alone can be gamed or misleading, a team using the catalog to browse but still pulling data through an old side channel looks adopted but is not; the side-channel-usage metric is the one that actually tells you if you have won.