Objective: explain end-to-end data lineage for the metric "monthly-active-customers (MAC)" so auditors can validate sources, transformations, ownership, retention, and known gaps.
Definition (authoritative): MAC = count of distinct Customer IDs with ≥1 qualifying activity during the 30-day calendar window ending on the report date. Qualifying activities: successful login, purchase, API call with authenticated token, or in-app event "page_view" with user_id. Definition maintained in Analytics Glossary (Owner: Product Analytics).
Lineage (step-by-step):
- Source systems
- Auth Service (AuthDB replica): emits successful_login events with user_id, timestamp. Owner: SRE / Auth Team (alice.auth@company.com). Retention: 1 year raw logs, 90 days indexed.
- Transactional DB (Billing): purchase records (user_id, order_id, ts, status). Owner: Finance (bob.finance@company.com). Retention: 7 years.
- Event Pipeline (Kafka topic events.raw): app and web events (user_id, event_name, ts, properties). Owner: Mobile/Web Eng (carol.events@company.com). Raw retention: 7 days; archived to S3 for 3 years.
- Ingest & raw landing
- Stream consumer (ETL-landing job) ingests Kafka + DB CDC to raw_events table in Data Lake (S3 Parquet) partitioned by date. Owner: Data Platform (dave.platform@). Retention: raw Parquet 3 years.
- Cleansing & identity resolution
- Job: identity_resolve (daily) joins raw_events, auth, billing to produce canonical_customer_id using mapping table (CRM merge keys). Removes events missing user_id; flags anonymous events. Owner: Data Engineering (erin.de@). Retention: processed dataset 2 years.
- Business transformation (authoritative MAC view)
- SQL transformation (analytics.mac_monthly_view) logic:
- Filter canonical events to qualifying event_names and successful status.
- Normalize timestamps to UTC; assign event_date.
- For given report_date D, select distinct canonical_customer_id where event_date ∈ [D-29, D].
- Count distinct customers.
- Implemented in dbt model mac_monthly (versioned in git repo). Owner: Product Analytics (frank.analytics@).
- Schedule: nightly at 02:00 UTC; runs incrementally.
- Aggregation & dashboard
- Aggregated table analytics.mac_agg contains monthly granularity, pre-computed by first-of-month job. Dashboard (Tableau: "Customer Health") reads mac_agg via BI view (read-only). Dashboard owner: Head of BI (gina.bi@). Dashboard refresh: daily.
Data quality & controls
- Row counts and null-rate checks in CI (dbt tests) — failures alert Data Engineering Slack #data-alerts and email to model owner.
- Reconciliation job: compare mac_monthly to raw event distinct counts; tolerance 0.5%. Owner: Product Analytics.
Retention windows (summary)
- Raw Kafka: 7 days; archived Parquet: 3 years
- Auth logs indexed: 90 days (raw archive 1 year)
- Billing DB: 7 years
- Processed canonical data: 2 years
- Aggregates/mac_agg: 5 years (for historical reporting)
Known data gaps & caveats
- Anonymous users: events without user_id are excluded (impact: undercount). A quarterly estimate of anonymous contribution exists (approx +2–4% MAC) — Owner: Product Analytics.
- Cross-device identity resolution is probabilistic; duplicates can cause small overcount (<1% typical).
- Offline/point-of-sale purchases that sync later: if sync timestamp >30 days after event, may be missed in MAC for that window.
- Small ingestion latency (up to 24 hours) means same-day dashboards may be incomplete until next refresh.
- Bot/automated traffic filtered by rule-set; if rules change, historical comparability may be affected. Changes are recorded in Data Change Log (repo).
Artifacts for auditor review (links provided on request):
- dbt model mac_monthly (git commit id)
- SQL of analytics.mac_monthly_view
- Data dictionary entry (Analytics Glossary)
- Alert runbook and SLOs
- Reconciliation reports for last 12 months
Contact list (owners):
If you’d like, I can attach the SQL model, dbt run history, and last reconciliation report for the audit package.