Requirements:- Nightly ETL delivers datasets powering critical dashboards by 6:00 AM business time.- Data freshness, completeness, and schema stability are critical for decision-making.- Acceptable business impact: occasional minor delays, but not missing or incorrect data.SLOs (examples):- Availability SLO: 99.9% of nightly ETL runs complete successfully within SLA window (finish by 6:00 AM) per rolling 30 days.- Freshness/latency SLO: 99% of critical tables updated within 2 hours of job start.- Completeness SLO (row-count): 98% of nightly loads have row counts within ±5% of a 30-day moving median.- Schema SLO: 99.9% of runs have no breaking schema changes (column added/removed/type change) detected.What to monitor:- Job status (success/failure) and run time (start/end timestamps).- End-to-end latency: time from source snapshot timestamp to availability in data warehouse.- Row counts and cardinality per critical table; compare to moving median and percent change.- Key metric sanity checks (e.g., total users > 0, revenue >= 0).- Schema fingerprint (hash of column names/types) and DDL diffs.- Data quality checks (null % on required columns, referential integrity).Alerting thresholds & actions:- P0 (Immediate Pager; on-call engineer + data owner): Job failure OR missing critical tables OR schema-breaking change OR latency > 4 hours (i.e., double SLO). Trigger if any run fails or SLA missed.- P1 (Email + Slack): Latency between 2–4 hours OR row-count deviation 5–10% OR data quality checks moderate (null% slightly above threshold). Action: investigate, consider re-run, apply backfill.- P2 (Digest/report): Minor deviations (row-count within 5% but trending) — email to data owner and product manager next business day.Concrete thresholds (example):- Job failure: alert immediately (P0).- Finish time > 6:00 AM by 2 hours (i.e., >8:00 AM): P0.- Finish time 6:00–8:00 AM: P1.- Row-count change >10%: P0. 5–10%: P1. <5%: no alert but logged.- Schema fingerprint change: P0 if breaking; P1 if additive non-breaking column.Escalation path:1. Alert fires to on-call data engineer and data owner via PagerDuty (P0) or Slack + Email (P1).2. If unacknowledged in 15 minutes (P0) escalate to senior data engineer and analytics manager; 30 minutes escalate to Head of Data.3. For P1, if not resolved in 2 hours escalate to on-call lead and stakeholder inbox.4. Post-incident: blameless postmortem within 48 hours for P0; summary to stakeholders for P1.Example monitoring SQL (row-count check):sql
-- compare today's count to 30-day median
WITH recent AS (
SELECT load_date, count(*) AS rows
FROM analytics.critical_table
WHERE load_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY load_date
)
SELECT
(SELECT count(*) FROM analytics.critical_table WHERE load_date = CURRENT_DATE) AS today_rows,
(SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY rows) FROM recent) AS median_30d;
Notes / best practices:- Keep SLOs tied to business impact; involve stakeholders when setting thresholds.- Alert only actionable items to avoid fatigue; use rate limiting and deduplication.- Maintain runbooks for common failures (schema drift, downstream warehouse outages, data source lag) and automate common remediations (retries, targeted backfills).