Data Governance, Contracts, and Classification Questions

A four-tier scheme covering public, internal, confidential, and restricted gives each tier a distinct access population, storage treatment, and an example field, and the tier is what mechanically determines the controls applied downstream, not a separate manual decision made each time.

The four tiers

How classification drives downstream controls

The tier should not be a label a person reads and manually acts on: it should be a tag the platform enforces automatically.

• Access. The classification tag maps directly to an RBAC policy: restricted-tier datasets require an access grant tied to a specific approved role (for example, "fraud investigator"), while internal-tier datasets are grantable to any authenticated employee by default. The mapping from tag to grantable roles is a lookup table, not a case-by-case decision.
• Encryption. Public and internal data use the platform's default encryption. Confidential and restricted data additionally use customer-managed (or team-managed) encryption keys, so a security team can revoke key access independently of the general access-control system as a second line of defense.
• Retention. The tag also drives a default retention window: restricted PII (personally identifiable information) is retained only as long as a documented business or legal purpose requires, with automatic deletion or anonymization jobs keyed off the tag; public and internal data can default to a longer or indefinite retention window since the cost of over-retaining is low.

The mechanism that makes this "concrete" rather than aspirational is that the tag lives as metadata on the dataset (in the catalog, alongside the schema), and every downstream system, the query engine, the export pipeline, the retention job, reads that tag and applies its rule automatically rather than a human re-deciding it per dataset.

Worked example

A new customers table lands in the warehouse with a ssn column and a signup_date column. An automated classifier scans new columns against known patterns (a 9-digit numeric pattern in a column named or profiled as an identifier) and tags ssn as restricted and signup_date as internal, before any human looks at it. The restricted tag on ssn immediately triggers: masking by default for anyone without the "PII-approved-analyst" role, a 3-year maximum retention timer tied to the last active transaction, and audit logging on every raw read. The signup_date column gets none of that: internal-tier default rules apply, and any employee querying the table sees it unmasked.

Trade-offs & pitfalls

A common mistake is treating classification as a one-time label at ingestion. Downstream joins and aggregations can either elevate or dilute sensitivity (a table joined against ssn inherits restricted status even if none of its own native columns look sensitive), so the classification needs to propagate through derived tables, not just apply at the source. Under-classifying to avoid friction is the more common failure mode in practice than over-classifying, because over-classification is visible and gets complaints, while under-classification is silent until it causes a breach.

Detect PII (personally identifiable information) as a mandatory, automated stage every pipeline passes through, not an opt-in step someone has to remember to add, by combining schema tagging, pattern-based rules, and machine learning (ML) classifiers so no single blind spot in one method leaves a gap, and pair detection with a masking strategy that defaults to safe while giving legitimate exceptions a documented, reviewed path.

Detection approaches, combined

No single technique catches everything on its own, so the three are layered rather than treated as alternatives:

1. Schema tagging. Fields already known to be PII (declared at table or event-schema creation, email, ssn, date_of_birth) are tagged in the schema registry at the point of definition. This is the cheapest and most reliable method, but it only catches PII that someone remembered to declare; it says nothing about a new field added later or a free-text field that happens to contain PII incidentally.
2. Regex and pattern matching. Deterministic patterns (an email format, a national ID number's digit structure, a credit card number's format) run against both structured columns and free-text fields, catching PII that landed somewhere undeclared, for example a support-ticket comment field containing a customer's phone number typed in by an agent.
3. ML-based classifiers. Named-entity recognition or a trained classifier scans less-structured content (chat logs, free-text notes) for PII patterns too varied for a fixed regex to catch reliably, a person's name embedded in a sentence, an address written in an unstructured format. This is the most expensive method computationally and the one most prone to false positives, so it's best applied selectively rather than on every record in a high-volume stream.

Running all three, with schema tagging as the free first pass, regex as a cheap second pass, and ML classification reserved for content the first two can't confidently clear, keeps overall detection cost proportional to how much genuinely needs the expensive method.

Covering both raw lake data and curated warehouse tables

The same three-method detection runs at two different points, not just once:

• At the raw lake ingestion point, detection tags incoming data as it lands, before anyone builds a curated table on top of it, so a downstream table doesn't inherit an undetected PII field from an upstream source that was never scanned.
• At the curated warehouse layer, detection re-runs against derived and joined tables specifically because a join or transformation can combine otherwise-innocuous columns into a newly PII-bearing result (a table joining a customer ID with a separately-innocuous demographic table becomes identifying once joined), which a source-only scan would miss entirely.

Covering both streaming and batch pipelines

• Streaming (for example Kafka, Flink, or Kinesis). Schema tagging and regex matching run inline, on the hot path, since they're cheap enough not to meaningfully add latency; ML classification, being more expensive, runs asynchronously on a side path (writing flagged content to a review queue) rather than blocking the main stream, so a slow classifier doesn't become a throughput bottleneck for the whole pipeline.
• Batch (for example Spark or a scheduled warehouse job). All three methods can run inline as part of the batch job itself, since batch jobs already tolerate longer per-record processing time; this is also where full re-scans of existing tables (not just new data) happen periodically, catching PII in data that landed before detection was in place.

flowchart TB
    subgraph Streaming
        SIN[Streaming ingestion] --> SSCHEMA[Schema tag check, inline]
        SSCHEMA --> SREGEX[Regex check, inline]
        SREGEX --> SML[ML classifier, async side path]
    end
    subgraph Batch
        BIN[Batch ingestion / scheduled scan] --> BALL[Schema + regex + ML, inline]
    end
    SSCHEMA --> POLICY[Policy engine]
    SREGEX --> POLICY
    SML --> POLICY
    BALL --> POLICY
    POLICY --> MASK[Masking / redaction applied]
    POLICY --> LAKE[(Raw lake, tagged)]
    POLICY --> WH[(Curated warehouse, tagged)]

Masking or redaction strategy once something is found

The action taken depends on the field's classification tier and its downstream use, not a single blanket rule:

• Confirmed structured PII (a tagged or high-confidence regex match on a known field type): masked by default in any view accessible outside the field's originating team, using the same tiered masking approach used elsewhere in the platform (partial mask, full redaction, or tokenization depending on sensitivity).
• PII detected in free text by the ML classifier: redacted (the specific span replaced with a placeholder like [NAME] or [PHONE]) rather than the whole record being dropped, preserving the non-sensitive content around it for analysis.
• Low-confidence ML detections: routed to a review queue rather than auto-masked or auto-ignored, since ML detections carry meaningfully more uncertainty than a schema tag or a regex match.

Handling false positives and legitimate exceptions

• False positives (a regex matching a numeric ID that happens to look like a phone number, or an ML classifier flagging a business name as a person's name): a lightweight review workflow lets a data owner mark a flagged field as a confirmed false positive, which suppresses future flags for that specific field or pattern rather than requiring a manual override on every occurrence.
• Legitimate exceptions (a field that is technically PII-shaped but has an approved, narrow business use, for example a fraud-detection model that genuinely needs raw device identifiers): handled through the same exception-request process used for other restricted-data access elsewhere in governance, an explicit, time-boxed, logged approval, not a permanent bypass flag quietly set once and forgotten.
• Feedback loop. Confirmed false positives and approved exceptions feed back into tuning the regex patterns and retraining or recalibrating the ML classifier's confidence thresholds over time, so the false-positive rate should trend down as the system accumulates review history, rather than staying static.

Trade-offs & pitfalls

The most common mistake is running detection only at ingestion and treating a table as permanently cleared once scanned, which misses both new PII introduced by later joins/transformations and PII in data that existed before detection was deployed. Periodic re-scanning of curated tables, not just one-time ingestion-point scanning, is what catches both gaps, at the cost of ongoing compute spent re-checking data that, most of the time, hasn't changed in its PII profile since the last scan.

Apply the same classification and controls consistently across all four surfaces (SQL, the BI tool, object storage, and notebooks) rather than securing the warehouse well and treating the other three as afterthoughts, since the surface most likely to leak PII (personally identifiable information) in practice is whichever one gets the least attention.

Policies and technical controls, per surface

Automation to catch accidental PII leakage

Policy alone doesn't catch the actual failure mode, which is usually someone unintentionally exporting or printing raw PII while doing legitimate work, not a deliberate breach attempt. Layer in automated detection:

1. Static scanning of notebook code and BI exports. A scheduled or pre-commit scan for the pattern of a raw PII column name (email, ssn) appearing in an unmasked query result or a notebook's saved output cell, flagging it for review rather than silently allowing it.
2. Data loss prevention (DLP) scanning on object storage writes. Any write to a broadly-readable storage prefix gets scanned for PII-shaped content (matching known patterns: emails, national ID formats, card numbers) even if it wasn't tagged as PII at the source; this catches the case where a PII column got joined into a new derived file nobody classified.
3. Anomalous-export alerting. A single analyst exporting a normal-sized daily report is expected behavior; the same analyst exporting a 500,000-row extract from a PII-containing table is not. Alert on volume and frequency outliers rather than trying to block every export outright, since most exports are legitimate.
4. Automatic quarantine, not automatic deletion. When a scan flags a likely leak, move the artifact (the notebook output, the storage object) into a restricted-access quarantine location and notify the owner, rather than deleting it outright; this preserves the evidence needed to confirm whether it was actually a leak, while immediately limiting exposure.

Letting authorized analysts do their real work

The controls above are only sustainable if the default, masked path is fast enough that nobody has an incentive to route around it:

• Governed views and masked notebook connections should cover the large majority of day-to-day analysis without needing an unmask request.
• For the genuine minority of cases needing raw PII (an investigation, a data quality debug session), provide a fast, logged, time-boxed unmask request rather than either blocking it entirely or leaving raw access permanently open.
• Keep the friction concentrated on the risky action (raw export, bulk pull), not on ordinary querying, so the controls don't train people to find workarounds.

Trade-offs & pitfalls

The most common failure in practice is securing the warehouse thoroughly and leaving object storage or notebooks as the gap, because they're viewed as "just infrastructure" rather than a PII surface in their own right. A raw data lake prefix that predates the classification effort, or a notebook server with a long-lived admin credential baked in from years ago, is often the actual leak path even when the warehouse-layer controls look solid on paper. Auditing all four surfaces with the same rigor, rather than assuming the newest or most-visible surface is the only one that matters, is the discipline that prevents this.

RBAC (role-based access control) is the simplest and fastest of the three but the coarsest; ABAC (attribute-based access control) is the most fine-grained and auditable when instrumented well but the most complex to build and test; row- or column-level masking is complementary to both rather than a substitute, since it controls what a granted user sees within a query, not whether they can query at all. A year-long migration from ad-hoc permissions typically lands on RBAC as the structural backbone with masking layered on top, escalating to ABAC only where role explosion would otherwise occur.

Trade-off comparison

Migrating from ad-hoc, undocumented permissions over a year

An organization running on ad-hoc grants (individual users given table access one-off over time, with no record of why) cannot jump straight to ABAC: there's no clean attribute model yet, and building one on top of undocumented existing access just formalizes the mess. A phased approach:

1. Months 1-2: Inventory and freeze. Audit every existing grant: who has access to what, and can anyone currently explain why. Freeze new ad-hoc grants; any new access request from this point goes through a lightweight ticket, even before the target model exists, so the mess stops growing while it's being cleaned up.
2. Months 3-5: Define roles from observed usage, not from an org chart. Cluster the actual query patterns (which tables does each existing user touch) into a small number of roles (analyst, engineer, finance, admin) rather than inventing roles theoretically. Migrate users into these roles, revoking their old ad-hoc grants as each migration completes.
3. Months 6-7: Layer in column-level masking for known-sensitive fields. Independent of the role migration, identify PII and financial columns and apply masking by default, unmasked only for the roles that demonstrably need it. This is often the fastest risk reduction for the least migration effort, since it doesn't require the role structure to be perfect first.
4. Months 8-10: Identify role-explosion pressure points. As the RBAC structure stabilizes, some access needs won't fit cleanly into roles (row-level restriction by customer or region, time-boxed elevated access). These are the specific, narrow places to introduce ABAC policies rather than a blanket ABAC rollout: a policy engine deciding "is this user's territory attribute equal to this row's territory" for the 2-3 places that actually need it.
5. Months 11-12: Audit and steady-state review process. Establish a recurring (for example quarterly) access review where each role's membership and each ABAC policy's effect is re-certified by the relevant data owner, so the system doesn't quietly regress back into ad-hoc grants six months after the migration.

Trade-offs & pitfalls

The main pitfall in this migration is trying to design the "perfect" ABAC policy model up front before any RBAC baseline exists: without a clean role structure, attribute-based rules end up encoding the same undocumented exceptions they were meant to replace. The other common failure is skipping the freeze in month 1; if ad-hoc grants keep happening during the migration, the inventory from month 1 is stale by month 6 and the whole effort has to restart.

Classify CRM columns by what they reveal, grant access per role against that classification rather than per table, and mask or tokenize on export by default so leaving the platform is never the path of least resistance for raw PII (personally identifiable information).

Classification scheme for the CRM columns

Using contacts and accounts tables as the concrete example:

Column-level access policy per role

The row-level filter for sales reps ("own assigned accounts only") is enforced the same way as the column policy: as a predicate attached to the underlying table or a secure view, not as a client-side filter the reporting tool could be misconfigured to skip.

Masking or tokenization for exports

Any export (a CSV download, a scheduled report to an external tool, an API pull) goes through a masking layer by default:

• Confidential columns default to masked on export. Email becomes domain-only (***@acme.com), phone becomes last-4 visible. A sales rep exporting their own account list for a call sheet needs raw values, so an explicit "export unmasked" action is available but is logged with the requester, timestamp, and row count, distinct from the default masked export.
• Restricted columns default to tokenized, not just masked, on export. A tokenized value (a random surrogate replacing the real value, with the mapping held in a separate, tightly access-controlled vault) preserves the ability to join or de-duplicate records by the token without ever putting the raw restricted value into an exported file. Reversing a token back to the real value requires a separate, audited request against the vault, not just re-running the export.
• Bulk exports get an extra gate. A single row export (one contact's card) and a 50,000-row bulk export carry very different risk, so exports above a configurable row threshold require a second approval regardless of role, since bulk exfiltration is the actual threat this control exists to catch.

Trade-off between tightening this and keeping sales and analytics productive

Tightening column access and masking exports protects the company from a breach or a compliance violation, but every added control point (an approval step, a masked-by-default value, a token that needs unmasking) adds latency to a sales rep who needs a phone number to make a call right now, or an analyst who needs to validate an anomaly against a real record.

The trade-off resolves cleanly along one axis: default to restrictive, make the unrestricted path fast for the common case. A sales rep's own assigned accounts should never feel gated (row-level access plus unmasked confidential fields, no extra click), because that is their actual job. What gets gated is the exceptional case: bulk export, cross-territory access, or restricted-tier fields. If the common case is fast and only the exceptional case has friction, sales and analytics stay productive while the actual risk surface (bulk exfiltration, restricted-field misuse) is where the friction concentrates. Getting this backwards, adding friction to the common case while leaving bulk export easy, is the pattern that both angers users and fails to reduce real risk.