Policy overview: When an event or schema change affects a metric, follow a defined lifecycle: discover impact, notify stakeholders, run a parallel migration with tests, cutover with feature-flagged reporting, then sunset and archive old metric definitions.1) Impact discovery- Run automated lineage and usage scans: - Query BI metadata: which dashboards, Looks, scheduled reports reference metric_name. - SQL example: SELECT dashboard_id, view_name FROM looker_usage WHERE sql_text ILIKE '%metric_name%'; - Scan ETL jobs and data model (LookML/Power BI dataset), saved queries, alerts.- Produce impact matrix: dashboards, owners, viewers, refresh cadence, business criticality (R/I/L).2) Stakeholder notification templateSubject: Action required — Upcoming change to metric "<metric_name>" (effective <date>)Body:- What: schema/event change and rationale- Who/Why: affected dashboards (<list/links>) and business impact- What we need: review reports, test parallel metric by <date>, sign-off- Timeline and rollback plan- Contact: BI owner + engineering contactInclude links to test sandbox and migration docs.3) Migration timeline (example, adjustable by risk)- Week 0: Discovery & notify stakeholders- Week 1–2: Implement new metric in staging; enable parallel reporting- Week 2–4: Stakeholder testing & feedback; run automated reconciliations- Week 4: Feature-flagged cutover for subset of users- Week 6: Full cutover if reconciliations within tolerance- Week 10: Sunset old metric and archive definitions4) Backward compatibility strategies- Parallel reporting: publish new_metric alongside old_metric (same dashboards show both) for comparison.- Feature flags / config: use dashboard-level toggle to show new vs old; flip gradually.- Semantic versioning: change metric name with suffix _v2 until cutover.- Delta monitoring: compare new vs old at daily/hourly granularity; set alert thresholds.5) Automated tests & monitors- Unit: SQL validation tests for edge cases (NULLs, timestamp boundaries).- Regression: daily reconciliation job that computes relative diff and top contributors to discrepancy; fail if >X% or >Y absolute.- Usage detector: CI job that rescans repo for references to deprecated metric and opens ticket if found.- Dashboard smoke tests: run headless render of dashboards and assert no runtime errors and row counts unchanged.Example SQL check:sql
SELECT
date,
old_sum, new_sum,
(new_sum - old_sum) / NULLIF(old_sum,0) as pct_diff
FROM (
SELECT date(trunc(ts,'day')) as date, SUM(old_metric) as old_sum, SUM(new_metric) as new_sum
FROM analytics.events
WHERE date >= current_date - interval '7 days'
GROUP BY 1
) t
WHERE abs(pct_diff) > 0.05;
6) Sunset & archival rules- Grace period: minimum 30 days of parallel reporting plus successful reconciliation.- Owner sign-off: Product/finance + BI owner approve sunset.- Archive: freeze old metric definition in repo, mark deprecated in metadata, add migration notes and example queries; retain raw data and ETL lineage for 1 year (or org retention policy).- Emergency rollback: maintain quick toggle to re-enable old metric for 48 hours post-cutover.Governance: Log every deprecation as a ticket in tracking system, publish status to BI change board, and report post-mortem if discrepancies > threshold. This policy balances transparency, low business risk, and automated verification so stakeholders can trust metric changes.