Situation: A recurring executive dashboard shows a sudden, large shift in a key metric.
Immediate triage (0–2 hours)
- Open an incident channel (Slack/Teams) titled "BI‑Incident: MetricName — YYYYMMDD" and invite analysts, data engineers, product manager, and the dashboard owner.
- Ask analysts to confirm scope: which dashboards/tiles, users affected, and timing of change.
- Verify reproducibility: refresh the dashboard, run the underlying saved query, and check whether the anomaly appears for different time windows or segments.
Technical validation (2–8 hours)
- Analyst: run the raw SQL against source tables to compare historical vs current aggregates; produce row counts, nulls, and distinct key counts.
- Data Engineer: inspect ingestion pipelines / ETL jobs, recent schema changes, job failures, or delayed partitions in last 48 hours; check DAG runs, logs, and deployment history.
- Product: check for product changes or A/B tests that could alter user behavior; provide release notes/toggles active for the period.
Narrow root cause (8–24 hours)
- If raw data confirms drop/rise -> treat as potential business shift. Product to validate causality (release/marketing event).
- If source data is fine but transformed metric is off -> inspect BI transformation logic, looker/expression changes, calculated fields, timezone or cohort definitions (BI to rollback or patch).
- If ingestion or schema changes caused it -> Data Eng fixes pipeline and backfills; BI notes affected dates.
Communication cadence
- Hourly updates for first 6 hours in incident channel; summary email to execs at 8 hours if unresolved.
- If resolved: send final report within 24 hours summarizing cause, fix, affected scope, and mitigation.
- If root cause requires more work: set daily status until resolved.
Postmortem & prevention (within 72 hours)
- Lead a blameless postmortem with stakeholders documenting timeline, root cause, actions taken, and impact (users, revenue, decisions).
- Implement fixes: monitoring/alerts for metric drift, unit tests on transforms, deploy-time checklist, canary dashboards, schema-change notifications, and rollback procedures.
- Update runbooks and assign owners for recurring checks.
Why this works:
- Fast, clear ownership, reproducible checks separate data vs reporting vs product causes, and closes loop with communication, remediation, and prevention.
