Data Governance, Contracts, and Classification Questions

A four-tier scheme covering public, internal, confidential, and restricted gives each tier a distinct access population, storage treatment, and an example field, and the tier is what mechanically determines the controls applied downstream, not a separate manual decision made each time.

The four tiers

How classification drives downstream controls

The tier should not be a label a person reads and manually acts on: it should be a tag the platform enforces automatically.

• Access. The classification tag maps directly to an RBAC policy: restricted-tier datasets require an access grant tied to a specific approved role (for example, "fraud investigator"), while internal-tier datasets are grantable to any authenticated employee by default. The mapping from tag to grantable roles is a lookup table, not a case-by-case decision.
• Encryption. Public and internal data use the platform's default encryption. Confidential and restricted data additionally use customer-managed (or team-managed) encryption keys, so a security team can revoke key access independently of the general access-control system as a second line of defense.
• Retention. The tag also drives a default retention window: restricted PII (personally identifiable information) is retained only as long as a documented business or legal purpose requires, with automatic deletion or anonymization jobs keyed off the tag; public and internal data can default to a longer or indefinite retention window since the cost of over-retaining is low.

The mechanism that makes this "concrete" rather than aspirational is that the tag lives as metadata on the dataset (in the catalog, alongside the schema), and every downstream system, the query engine, the export pipeline, the retention job, reads that tag and applies its rule automatically rather than a human re-deciding it per dataset.

Worked example

A new customers table lands in the warehouse with a ssn column and a signup_date column. An automated classifier scans new columns against known patterns (a 9-digit numeric pattern in a column named or profiled as an identifier) and tags ssn as restricted and signup_date as internal, before any human looks at it. The restricted tag on ssn immediately triggers: masking by default for anyone without the "PII-approved-analyst" role, a 3-year maximum retention timer tied to the last active transaction, and audit logging on every raw read. The signup_date column gets none of that: internal-tier default rules apply, and any employee querying the table sees it unmasked.

Trade-offs & pitfalls

A common mistake is treating classification as a one-time label at ingestion. Downstream joins and aggregations can either elevate or dilute sensitivity (a table joined against ssn inherits restricted status even if none of its own native columns look sensitive), so the classification needs to propagate through derived tables, not just apply at the source. Under-classifying to avoid friction is the more common failure mode in practice than over-classifying, because over-classification is visible and gets complaints, while under-classification is silent until it causes a breach.

RBAC (role-based access control) is the simplest and fastest of the three but the coarsest; ABAC (attribute-based access control) is the most fine-grained and auditable when instrumented well but the most complex to build and test; row- or column-level masking is complementary to both rather than a substitute, since it controls what a granted user sees within a query, not whether they can query at all. A year-long migration from ad-hoc permissions typically lands on RBAC as the structural backbone with masking layered on top, escalating to ABAC only where role explosion would otherwise occur.

Trade-off comparison

Migrating from ad-hoc, undocumented permissions over a year

An organization running on ad-hoc grants (individual users given table access one-off over time, with no record of why) cannot jump straight to ABAC: there's no clean attribute model yet, and building one on top of undocumented existing access just formalizes the mess. A phased approach:

1. Months 1-2: Inventory and freeze. Audit every existing grant: who has access to what, and can anyone currently explain why. Freeze new ad-hoc grants; any new access request from this point goes through a lightweight ticket, even before the target model exists, so the mess stops growing while it's being cleaned up.
2. Months 3-5: Define roles from observed usage, not from an org chart. Cluster the actual query patterns (which tables does each existing user touch) into a small number of roles (analyst, engineer, finance, admin) rather than inventing roles theoretically. Migrate users into these roles, revoking their old ad-hoc grants as each migration completes.
3. Months 6-7: Layer in column-level masking for known-sensitive fields. Independent of the role migration, identify PII and financial columns and apply masking by default, unmasked only for the roles that demonstrably need it. This is often the fastest risk reduction for the least migration effort, since it doesn't require the role structure to be perfect first.
4. Months 8-10: Identify role-explosion pressure points. As the RBAC structure stabilizes, some access needs won't fit cleanly into roles (row-level restriction by customer or region, time-boxed elevated access). These are the specific, narrow places to introduce ABAC policies rather than a blanket ABAC rollout: a policy engine deciding "is this user's territory attribute equal to this row's territory" for the 2-3 places that actually need it.
5. Months 11-12: Audit and steady-state review process. Establish a recurring (for example quarterly) access review where each role's membership and each ABAC policy's effect is re-certified by the relevant data owner, so the system doesn't quietly regress back into ad-hoc grants six months after the migration.

Trade-offs & pitfalls

The main pitfall in this migration is trying to design the "perfect" ABAC policy model up front before any RBAC baseline exists: without a clean role structure, attribute-based rules end up encoding the same undocumented exceptions they were meant to replace. The other common failure is skipping the freeze in month 1; if ad-hoc grants keep happening during the migration, the inventory from month 1 is stale by month 6 and the whole effort has to restart.

Detect PII (personally identifiable information) as a mandatory, automated stage every pipeline passes through, not an opt-in step someone has to remember to add, by combining schema tagging, pattern-based rules, and machine learning (ML) classifiers so no single blind spot in one method leaves a gap, and pair detection with a masking strategy that defaults to safe while giving legitimate exceptions a documented, reviewed path.

Detection approaches, combined

No single technique catches everything on its own, so the three are layered rather than treated as alternatives:

1. Schema tagging. Fields already known to be PII (declared at table or event-schema creation, email, ssn, date_of_birth) are tagged in the schema registry at the point of definition. This is the cheapest and most reliable method, but it only catches PII that someone remembered to declare; it says nothing about a new field added later or a free-text field that happens to contain PII incidentally.
2. Regex and pattern matching. Deterministic patterns (an email format, a national ID number's digit structure, a credit card number's format) run against both structured columns and free-text fields, catching PII that landed somewhere undeclared, for example a support-ticket comment field containing a customer's phone number typed in by an agent.
3. ML-based classifiers. Named-entity recognition or a trained classifier scans less-structured content (chat logs, free-text notes) for PII patterns too varied for a fixed regex to catch reliably, a person's name embedded in a sentence, an address written in an unstructured format. This is the most expensive method computationally and the one most prone to false positives, so it's best applied selectively rather than on every record in a high-volume stream.

Running all three, with schema tagging as the free first pass, regex as a cheap second pass, and ML classification reserved for content the first two can't confidently clear, keeps overall detection cost proportional to how much genuinely needs the expensive method.

Covering both raw lake data and curated warehouse tables

The same three-method detection runs at two different points, not just once:

• At the raw lake ingestion point, detection tags incoming data as it lands, before anyone builds a curated table on top of it, so a downstream table doesn't inherit an undetected PII field from an upstream source that was never scanned.
• At the curated warehouse layer, detection re-runs against derived and joined tables specifically because a join or transformation can combine otherwise-innocuous columns into a newly PII-bearing result (a table joining a customer ID with a separately-innocuous demographic table becomes identifying once joined), which a source-only scan would miss entirely.

Covering both streaming and batch pipelines

• Streaming (for example Kafka, Flink, or Kinesis). Schema tagging and regex matching run inline, on the hot path, since they're cheap enough not to meaningfully add latency; ML classification, being more expensive, runs asynchronously on a side path (writing flagged content to a review queue) rather than blocking the main stream, so a slow classifier doesn't become a throughput bottleneck for the whole pipeline.
• Batch (for example Spark or a scheduled warehouse job). All three methods can run inline as part of the batch job itself, since batch jobs already tolerate longer per-record processing time; this is also where full re-scans of existing tables (not just new data) happen periodically, catching PII in data that landed before detection was in place.

flowchart TB
    subgraph Streaming
        SIN[Streaming ingestion] --> SSCHEMA[Schema tag check, inline]
        SSCHEMA --> SREGEX[Regex check, inline]
        SREGEX --> SML[ML classifier, async side path]
    end
    subgraph Batch
        BIN[Batch ingestion / scheduled scan] --> BALL[Schema + regex + ML, inline]
    end
    SSCHEMA --> POLICY[Policy engine]
    SREGEX --> POLICY
    SML --> POLICY
    BALL --> POLICY
    POLICY --> MASK[Masking / redaction applied]
    POLICY --> LAKE[(Raw lake, tagged)]
    POLICY --> WH[(Curated warehouse, tagged)]

Masking or redaction strategy once something is found

The action taken depends on the field's classification tier and its downstream use, not a single blanket rule:

• Confirmed structured PII (a tagged or high-confidence regex match on a known field type): masked by default in any view accessible outside the field's originating team, using the same tiered masking approach used elsewhere in the platform (partial mask, full redaction, or tokenization depending on sensitivity).
• PII detected in free text by the ML classifier: redacted (the specific span replaced with a placeholder like [NAME] or [PHONE]) rather than the whole record being dropped, preserving the non-sensitive content around it for analysis.
• Low-confidence ML detections: routed to a review queue rather than auto-masked or auto-ignored, since ML detections carry meaningfully more uncertainty than a schema tag or a regex match.

Handling false positives and legitimate exceptions

• False positives (a regex matching a numeric ID that happens to look like a phone number, or an ML classifier flagging a business name as a person's name): a lightweight review workflow lets a data owner mark a flagged field as a confirmed false positive, which suppresses future flags for that specific field or pattern rather than requiring a manual override on every occurrence.
• Legitimate exceptions (a field that is technically PII-shaped but has an approved, narrow business use, for example a fraud-detection model that genuinely needs raw device identifiers): handled through the same exception-request process used for other restricted-data access elsewhere in governance, an explicit, time-boxed, logged approval, not a permanent bypass flag quietly set once and forgotten.
• Feedback loop. Confirmed false positives and approved exceptions feed back into tuning the regex patterns and retraining or recalibrating the ML classifier's confidence thresholds over time, so the false-positive rate should trend down as the system accumulates review history, rather than staying static.

Trade-offs & pitfalls

The most common mistake is running detection only at ingestion and treating a table as permanently cleared once scanned, which misses both new PII introduced by later joins/transformations and PII in data that existed before detection was deployed. Periodic re-scanning of curated tables, not just one-time ingestion-point scanning, is what catches both gaps, at the cost of ongoing compute spent re-checking data that, most of the time, hasn't changed in its PII profile since the last scan.