Situation: Unexpected large KPI spike (e.g., daily conversion rate jumped 3x). Goal: quickly determine if spike is real or tracking-related, quantify drivers, and recommend actions.Quick triage (first 30–60 minutes)- Confirm metric definition and time window (UTC vs local).- Check upstream data pipeline / ingestion errors, ETL logs, and last deployment/change log.- Validate against multiple sources: raw events, analytics db, dashboard aggregates, and third-party tags.- Run simple sanity SQL to compare counts and unique users:sql
-- compare event counts and unique users today vs baseline
SELECT
date(event_time) as day,
COUNT(*) as events,
COUNT(DISTINCT user_id) as uniques
FROM events
WHERE event_time >= current_date - interval '14' day
GROUP BY day
ORDER BY day;
Data analysis techniques (deeper, 1–4 hours)- Time series and changepoint: plot minute/hour granularity to find onset time; apply z-score or Bayesian changepoint to locate abrupt change.- Segmentation: by traffic source, country, device, app version, landing page, campaign id to isolate affected slices.- Cohorts and funnel: build user cohorts (first touch date, new vs returning) and examine funnel conversion steps to see where change originates.- Cross-metric validation: check upstream/downstream KPIs (sessions, click-throughs, revenue) for consistent signals.- Sampling and session-level inspection: pull random user sessions in the spike window and inspect events for anomalies (duplicate events, malformed payloads).Tests and statistical checks- Compute relative lift with confidence intervals; run proportion test for conversion rates.- Check for bot/automation: sudden surge in single IPs, user agents, or unusually low session durations.If spike is real (validated)- Identify top contributing segments and hypothesize causes (campaign, UX change, external event).- Notify stakeholders with early-findings dashboard and recommended immediate actions (monitor, scale infra if needed).- Run root-cause experiments: A/B rollback if tied to release, or attribution analysis for marketing.- Track persistence: categorize as transient vs sustained; if sustained, update forecasts and KPIs.If measurement/tracking error- Pinpoint faulty instrument (tagging bug, duplicate processing, timezone bug).- Stop-gap: exclude affected data range from reports; add annotation to dashboards.- Fix code/ETL and reprocess/backfill clean data; validate with tests and peer review.- Implement monitoring/alerting for similar anomalies (data quality checks, cardinality checks, alerts on sudden unique count jumps).- Document incident and preventive actions.Key outputs for stakeholders- Short executive summary (is it real? impact magnitude, top segments, recommended next steps)- Repro steps and technical root cause- Action plan with owners and timelines- Follow-up: postmortem and monitoring improvements.