Data Pipeline Monitoring and Observability Questions

Direct answer

A catalog entry for a real table like orders needs fields across three categories: technical (mechanically derived), operational (about how the pipeline behaves), and business (about meaning and accountability). Good search and filtering over those fields turns "browse everything" into "find the one table that fits," which is the entire value of cataloging in the first place.

Structured elaboration

Technical fields:

1. Schema (field names, types, nullability)
2. Row count / table size
3. Primary key and any known unique constraints
4. Storage format and location (which warehouse, schema, or lake path)

Operational fields:
5. Refresh cadence (for example, hourly batch, or a stated freshness service-level agreement, SLA)
6. Last successful load timestamp
7. Upstream lineage (which pipeline or job produces it) and downstream lineage (what consumes it)
8. Data-quality status (passing or failing its checks, and which checks)

Business fields:
9. Owner (a named person or team, with a contact path)
10. Plain-language definition (what one row represents, e.g., "one row per completed checkout, excludes abandoned carts")
11. Sensitivity classification (public, internal, confidential, restricted, or similar)
12. Known consumers or use cases (which dashboards or models rely on it, useful for impact analysis)

That is twelve concrete fields, comfortably above the minimum of eight, spread across all three categories rather than clustered in one.

A minimal way to model this underneath the fields. Rather than one flat record per table, these fields map cleanly onto a small set of entities: a dataset entity (the table itself, holding operational and top-level business fields like owner and sensitivity), a column entity (one row per field, holding its own type and, where relevant, its own sensitivity tag, since a table's classification is often driven by its most sensitive column), an owner entity (a person or team, referenced by the dataset and reusable across many datasets), a tag entity (sensitivity and free-form tags, also reusable), and a sample_query entity (one or more example queries against the table, which double as executable documentation of how the table is actually meant to be used). Modeling it this way, rather than as one wide table of fields, is what makes filtering by owner or by tag efficient at scale, you are joining against a small owner or tag table instead of scanning a text field on every dataset.

How search and filtering over these fields helps. A field-level catalog lets someone search by what they actually know rather than by table name: filter to tables owned by a specific team, filter to tables refreshed at least daily (rules out a stale nightly table for a need that requires current data), filter to tables classified "internal" or lower (rules out anything requiring a special access request for a quick exploratory question), or search the plain-language definition text for "checkout" and find orders even without knowing its exact name. Faceted filtering across technical, operational, and business fields together is what turns a keyword search into "show me the tables that are actually usable for this specific task," not just tables that mention a matching word.

Worked example

An analyst needs a table of completed customer purchases, refreshed at least daily, that they can access without a special request. They search the catalog for "purchase," which full-text matches orders' definition field ("completed checkout"). They then filter by refresh cadence (daily or better, orders refreshes hourly, passes) and by sensitivity (internal or lower, orders is tagged internal, passes; a hypothetical orders_with_card_number table tagged restricted would be filtered out here, saving the analyst from requesting access to a table they do not actually need). The result: one relevant table surfaced from a catalog of hundreds, using three fields (definition text, refresh cadence, sensitivity) none of which is the table's literal name.

Trade-offs & pitfalls

Populating twelve fields per table for hundreds of tables is real curation effort, if the business fields (owner, definition, sensitivity) are left blank because nobody enforces them, search degrades to matching only on technical fields, which is exactly the information a data engineer already had and an analyst did not. Filtering is only as good as the fields are accurate, a table whose refresh cadence field says "hourly" but whose pipeline has silently been failing for a week will pass a freshness filter it should not, so operational fields like "last successful load" need to be kept live, not set once and forgotten. Search relevance also degrades if the definition field is copied boilerplate rather than a genuinely distinguishing description, "customer data" as a definition on fifteen different tables defeats the entire point of full-text search.

Without formal authority, the roadmap has to earn adoption rather than mandate it: start narrow with a team that already feels pain, prove the governance program removes more friction than it adds, and use that visible win to build the social capital needed to expand. Compliance without authority is nominal (teams do the minimum to avoid being flagged); real adoption comes from teams choosing to keep doing it because it made their own work easier or safer.

First six months, roadmap

• Weeks 1-4, listen before proposing anything. Interview the teams whose data causes the most downstream pain and the teams who consume it. The goal is to find a concrete, already-felt problem (a recurring incident, a metric nobody trusts, a compliance near-miss) rather than pitching governance as an abstract good.
• Weeks 5-8, pick one willing pilot team and one narrow, high-visibility problem. Volunteer, not conscript. Co-design a lightweight fix with them (a data contract for their most-consumed table, an ownership assignment, one automated quality check) so it's their solution, not a mandate imposed on them.
• Weeks 9-16, ship the pilot and make the win visible. Get the fix live, then actively publicize the before/after (fewer incidents, faster diagnosis, less firefighting) in whatever forum leadership and peer teams actually pay attention to (an eng-wide demo, a leadership update, a Slack channel with real traffic).
• Weeks 17-24, expand by invitation, not mandate. Approach two or three more teams using the pilot as social proof ("here's what it did for team X"), and start building the lightweight shared tooling (a catalog entry template, a contract checklist) that makes adoption cheaper for each subsequent team than it was for the first.

Change-management tactics and incentives

• Make the easy path also the compliant path. If registering a data contract takes an afternoon and a shared template, teams will do it; if it means a multi-week review process, they'll route around it. The single biggest lever without formal authority is removing friction, not adding enforcement you don't have the standing to apply.
• Tie the ask to something the team already wants. A team drowning in "why does this number look wrong" Slack pings wants faster diagnosis, not "governance"; frame the same contract-and-ownership work as solving their on-call pain, not as compliance.
• Use visible peer example over top-down messaging. A team hearing "team X cut their incident load doing this" from a peer is more persuasive than a policy memo, especially with no authority to back the memo up.
• Recruit an executive sponsor for air cover, not enforcement. A sponsor who occasionally asks "is this dataset governed yet" in a leadership review creates gentle pressure without you personally having to police anyone, which matters because you don't have the standing to police anyone.
• Publicly credit the adopting teams, not the governance function, for the win. Teams that get recognized for the improvement become advocates who bring the next team in on their own.

Measuring trust and adoption along the way

Rather than fabricate a single trust score, track a small set of concrete, observable signals: how many teams volunteer for the next wave without being asked (the clearest real signal, since a coerced team never volunteers), whether teams start registering NEW datasets under the standard without prompting, whether the pilot team keeps the practice going after the initial push ends (durable adoption versus a one-time favor), and whether incident-related pings in the pilot's channels shift from "who owns this" questions to "here's the runbook" answers. Each of these is a direct observation, not a survey score dressed up as data, and each would need to be logged from the actual rollout rather than assumed in advance.

Trade-offs and pitfalls

The main risk of the volunteer-first approach is that it's slow: six months in, you may have covered two or three teams out of dozens, and a leader impatient for broad coverage may read that as failure when it's actually the necessary cost of building durable, non-nominal adoption. The opposite failure, trying to move fast by leaning on an executive sponsor to mandate adoption early, tends to produce exactly the nominal compliance the question asks you to avoid: teams check the box to satisfy the mandate and quietly keep their old workflow for anything that actually matters to them.

Before wiring in a new consumer, register them as a known dependency, not just grant database access; confirm they understand the dataset's actual contract, its schema, freshness, compatibility guarantees, and owner, rather than reverse-engineering current behavior; and make sure the producer's future-change process will actually notify them. Skipping this is exactly how a dataset ends up with an unofficial consumer nobody accounts for when planning a change.

Steps before granting access

1. Confirm there is an actual contract for the dataset: schema, semantics, freshness, owner, and compatibility rules. If there is not one yet, write a minimal one now, since onboarding a new consumer is exactly the moment to do it, not a distraction from it.
2. Have the new team state what specifically they need, which fields, what freshness, what volume, rather than granting broad access "just in case"; this keeps the eventual blast radius of a future schema change smaller and better understood.
3. Add the new consumer to the dataset's registered consumer list or catalog entry, with a contact and a description of their use case.
4. Confirm the new team knows the compatibility guarantee, what kinds of changes they can expect without notice versus what will trigger a migration process, before they start building against it.

Preventing a silent break from a future upstream change

This is what step 3 is actually for: if the producer's change process, its CI (continuous integration) compatibility checks and notification cadence, works off the registered consumer list, then a consumer who is not on that list does not get notified and finds out through a broken dashboard instead of a heads-up email. Registration at onboarding time is the mechanism that keeps that from happening; it costs a few minutes now versus an incident later.

Preventing it from becoming an unofficial, invisible contract

The failure mode without this process usually is not a policy violation, it is just informality: a dashboard gets built against a convenient table, it works, nobody writes it down, and eighteen months later the producing team has no idea that table has a consumer at all when they plan a change. Making registration a required step of granting access, not a follow-up someone can skip, is what keeps every real dependency visible in the catalog, so "who is downstream of this table" is always answerable by looking something up rather than by asking around.

Worked example

The finance team wants to build a new dashboard on the orders table maintained by the checkout team. Before wiring it up: finance states they need order_id, amount_cents, currency, placed_at, and daily freshness, not real-time; checkout confirms the existing contract covers that, BACKWARD compatibility mode with a daily-refresh service-level agreement; finance is added to the table's registered-consumers list in the catalog with a contact and "revenue dashboard" as the use case; and finance is added to the notification list checkout's CI process already uses for compatibility-flagged changes. Months later, checkout proposes widening amount_cents from a 32-bit to a 64-bit integer; the CI compatibility check flags it as low-risk but still notifies registered consumers, and finance gets the heads-up automatically instead of discovering it when their dashboard's numbers look odd.

Trade-offs and pitfalls

For a truly low-stakes, single-use internal query, requiring full contract registration can feel like overkill, and teams will route around a heavy process by just querying the table directly; keep the registration step lightweight, a catalog entry and a name, not a committee review, so it is actually easier to do than to skip. The most common failure is granting database access first and treating registration as an optional follow-up; once access works, there is no forcing function to ever go back and register it properly.

Direct answer

Combine the signals as a weighted linear score: normalize each signal (usage frequency, recency, a data-quality score, an owner trust score, and semantic match to the query) to a common 0-to-1 scale, weight each by how much it should matter, and sum. The weighting matters more than the exact math, a naive ranking that lets semantic match alone dominate will surface a dataset that mentions the right words but is stale, low-quality, or barely used, over one that is actually the right answer.

Structured elaboration

The five signals, and what each one catches:

• Usage frequency: how often the dataset is queried; a strong signal that other analysts have already found it useful for something.
• Recency: how recently it was queried or updated; guards against surfacing an abandoned dataset that used to be popular.
• Data-quality score: a stated pass rate or completeness score from the dataset's automated quality checks; guards against ranking a technically-relevant but unreliable dataset highly.
• Owner trust score: a signal for how consistently the dataset's owner keeps its metadata and quality checks current (this could be as simple as the fraction of the owner's other datasets that pass their quality checks); guards against a one-off well-described dataset from an otherwise unreliable source outranking a consistently-maintained one.
• Semantic match to the query: how closely the dataset's name, description, and column names match the searched business concept; the signal most directly tied to relevance, but the one most easily gamed by a verbose or keyword-stuffed description.

Combining them into a ranking.

$\text{score} = w_1 \cdot \text{usage} + w_2 \cdot \text{recency} + w_3 \cdot \text{quality} + w_4 \cdot \text{trust} + w_5 \cdot \text{semantic}$

where each signal is normalized to the $[0,1]$ range and the weights sum to 1:

$w_1=0.3,\ w_2=0.2,\ w_3=0.2,\ w_4=0.15,\ w_5=0.15,\ \textstyle\sum w_i = 1.0$

Usage gets the largest weight here because it is the strongest observed signal that real analysts found the dataset genuinely useful for a real task, which is a stronger endorsement than any single metadata field. Semantic match and trust get smaller weights deliberately, because both are easier to satisfy without the dataset actually being the best answer (a well-written description does not guarantee good data; a generally reliable owner does not guarantee this specific dataset is the freshest option). These weights are a starting point, not a fixed law, and should be tuned against real search-and-click behavior once the catalog has usage data to tune against.

Worked example

Three candidate datasets return for a search on "customer churn," with normalized signal values:

Applying the weights above:

Enforce the row filter and the column mask as close to the data as possible, at the warehouse layer, so every consumer (SQL client, BigQuery-style authorized view, or Looker's semantic layer) inherits the same rule instead of each client re-implementing it. The three approaches (a database-level row-level security policy, BigQuery-style authorized views, and a BI tool's model-level access) differ in where the enforcement logic lives, which changes both the performance profile and how many places you have to keep in sync.

The core mechanism

Whichever layer enforces it, the mechanism is the same: a predicate that filters rows based on the querying user's identity (row-level security, RLS) and an expression that conditionally transforms a column's value based on the user's role (masking). Below is a runnable, portable demonstration of that mechanism using SQLite so the logic can be verified directly; the mapping to each real product's specific feature is explained afterward.

import sqlite3

conn = sqlite3.connect(":memory:")
cur = conn.cursor()

# Base fact table: one shared underlying table, as in the question.
cur.execute('''
    CREATE TABLE orders (
        id INTEGER PRIMARY KEY,
        region TEXT NOT NULL,
        customer_email TEXT NOT NULL,
        amount_usd REAL NOT NULL
    )
''')
rows = [
    (1, "US", "alice@example.com", 120.00),
    (2, "US", "bob@example.com", 340.50),
    (3, "EU", "carla@example.com", 89.99),
    (4, "EU", "dieter@example.com", 275.25),
    (5, "APAC", "eun@example.com", 512.10),
]
cur.executemany("INSERT INTO orders VALUES (?,?,?,?)", rows)
conn.commit()

def query_as(session_region, session_role):
    # session_region drives the row filter (RLS); session_role drives the column mask.
    mask_expr = (
        "customer_email"
        if session_role == "PRIVILEGED"
        else "substr(customer_email, 1, 2) || '***@' || substr(customer_email, instr(customer_email,'@')+1)"
    )
    sql = (
        f"SELECT id, region, {mask_expr} AS customer_email, amount_usd "
        "FROM orders WHERE region = ? ORDER BY id"
    )
    cur.execute(sql, (session_region,))
    return cur.fetchall()

for r in query_as("US", "ANALYST"):
    print(r)

Output:

(1, 'US', 'al***@example.com', 120.0)
(2, 'US', 'bo***@example.com', 340.5)

A PRIVILEGED session against the same table and region returns the unmasked emails, and an APAC-scoped session never returns US or EU rows at all, even though every session queries the identical underlying table. The full run (including both assertions and a compliance-role session) is in the code artifact used to write this answer; the point being demonstrated is that a single shared table plus a session-scoped predicate and conditional expression is sufficient to produce per-user row and column visibility without duplicating the data.

How this maps to each real layer

• Database-level RLS policy (for example a Snowflake ROW ACCESS POLICY or a masking policy). The predicate above corresponds to a policy object attached directly to the table: CREATE ROW ACCESS POLICY region_policy AS (region STRING) RETURNS BOOLEAN -> region = CURRENT_REGION_CONTEXT(), attached with ALTER TABLE orders ADD ROW ACCESS POLICY region_policy ON (region). Every query against the table, from any tool, is filtered automatically; there's exactly one place to define and audit the rule.
• BigQuery-style authorized views. Instead of a policy object on the base table, you create a view with the filter and masking baked into its SELECT, and grant users access to the view instead of the base table (CREATE VIEW orders_masked AS SELECT ..., CASE WHEN ... END AS customer_email FROM orders WHERE region = SESSION_USER_REGION()). The base table itself stays locked down; the view is the only grantable surface. This is functionally similar to the RLS policy but the enforcement lives in a separate object you must remember to route every consumer through, rather than attaching automatically to the base table.
• Looker's model-level access. Looker sits above the warehouse and expresses the same row filter as an access_filter in its LookML model (access_filter: { field: orders.region, user_attribute: region }), and column-level masking as a restricted "always filter" or a hidden/derived field shown only to certain permission groups. This enforcement only applies to traffic that goes through Looker; a user with direct warehouse SQL access bypasses it entirely.

Integrating with SSO for identity

None of the three layers know who the user is on their own; identity comes from single sign-on (SSO), typically via SAML (Security Assertion Markup Language) or OIDC (OpenID Connect) federated into the warehouse's own user/role system and separately into Looker's user attributes. The pattern: SSO authenticates the user and asserts group/attribute claims (for example region: EU); the warehouse's identity provider integration maps those claims into session context variables (CURRENT_REGION_CONTEXT() in the demo above); Looker separately maps the same SSO claims into its own user attributes via SCIM (System for Cross-domain Identity Management) provisioning, so both layers derive the same attribute from the same source of truth rather than each maintaining its own copy of "which region is this user."

Performance and maintainability trade-offs

Trade-offs & pitfalls

The recurring mistake is enforcing this exclusively at the BI tool layer because it's the easiest place to configure, then discovering that anyone with direct SQL access to the warehouse (a data scientist's notebook, a scheduled export job) bypasses every rule Looker enforces. Enforcing at the warehouse layer is more work up front but is the only approach that holds regardless of which client connects; BI-tool-level policy should be treated as a convenience layer for the common case, not the actual security boundary.