Data Quality Debugging and Root Cause Analysis Questions

Focuses on investigative approaches and operational practices used when data or metrics are incorrect. Includes techniques for triage and root cause analysis such as comparing to historical baselines, segmenting data by dimensions, validating upstream sources and joins, replaying pipeline stages, checking pipeline timing and delays, and isolating schema change impacts. Candidates should discuss systematic debugging workflows, test and verification strategies, how to reproduce issues, how to build hypotheses and tests, and how to prioritize fixes and communication when incidents affect downstream consumers.

EasyTechnical

43 practiced

You discover an analytics metric has dropped 20% compared to yesterday. Describe a step-by-step triage workflow you would follow to find the root cause. Include the first quick checks, what queries or aggregation patterns you would run, how to segment the data, and how to involve stakeholders so you limit panic while moving fast.

Sample Answer

Start with a calm hypothesis-driven triage: aim to quickly rule out obvious causes, narrow scope, then dig deeper while keeping stakeholders informed.

1) Quick sanity checks (0–15 min)- Confirm metric definition and pipeline: did the metric calculation change (code/deploys) or schema change yesterday?- Check upstream data freshness: are ingestion jobs/airflow tasks healthy?- Verify dashboards/code: caching or dashboard time-window mistakes.- Check alerts/logs for job failures or schema errors.

2) Narrow with fast aggregations (15–45 min)- Compare overall vs prior periods: today vs yesterday vs same weekday last week.- Run simple SQL rollups to find where drop concentrates:

sql

-- overall trend
SELECT date, SUM(metric) as total_metric FROM events WHERE date >= current_date - interval '14 days' GROUP BY date ORDER BY date;

-- by source / cohort
SELECT source, SUM(metric) as total FROM events WHERE date = current_date - interval '1 day' GROUP BY source ORDER BY total DESC LIMIT 20;

- Aggregate by dimensions: user cohort (new vs returning), region, device, model_version, experiment_id, time-of-day.

3) Model-specific checks- Check model-serving logs and latency/error rates; confirm model_version serving matches expected.- Evaluate input distribution drift: compare feature distributions yesterday vs baseline (JS divergence, histograms).- Re-run a sample of recent inputs through a local copy of the model to ensure outputs are sane.

4) Drill down (45–120 min)- If a dimension is implicated, slice further (e.g., region X, browser Y, model_version Z).- Check correlated metrics (traffic, conversions, downstream ML predictions) to see if it's a data issue or behavioral change.- Use cohort retention over time to see if user behavior changed.

5) Communicate and involve stakeholders- Send an initial concise update: what you saw, immediate checks done, next steps, ETA for deeper analysis. Avoid speculation.- Triage owners: assign data-pipeline, infra, and product/analytics leads to respective investigations.- If root cause unknown after 1–2 hours, propose mitigations (rollback, pause experiment) and escalate.

6) Root cause and remediation- Once found, document evidence, fix (reprocess data, patch code, roll back), validate with regenerated metrics and tests, and communicate resolution and postmortem with action items to prevent recurrence.

Key patterns: fast eliminate (pipeline/deploy/config), then segment by dimension to localize, check model serving & input drift, and keep stakeholders informed with clear, evidence-based updates.

MediumTechnical

41 practiced

How would you instrument dataset and job lineage for a complex ML pipeline that spans Kafka topics, Spark jobs, and a data warehouse so you can quickly trace which upstream change caused a wrong prediction? Describe the metadata to capture, tools or standards you would use, and how you would query lineage during an incident.

Sample Answer

Situation: You need to trace a bad prediction back through Kafka → Spark → warehouse → model. My approach instruments lineage end-to-end so you can pivot from a wrong prediction to the exact upstream change.

Metadata to capture (per dataset/job/event):- Logical dataset id (namespace.name) and physical location (topic, table, path)- Run id / job id and parent_run_id- Schema (avro/ddl) + schema hash- Partition/filter predicates and offsets (Kafka partition, offset range) or file ranges- Record-level identifiers or sample hashes (for sampled snapshots)- Checksums/counts/row-level diff stats- Transformation metadata: job name, code commit hash, docker image + image digest, parameters/config, Spark job/app id, UDF versions- Timestamps (ingest, process start/finish)- Feature derivation lineage (feature name → source fields → transformation)- Model artifact id, model version, training data snapshot id, training config/hyperparams, evaluation metrics

Tools / standards:- Use OpenLineage/Marquez as the event schema to emit lineage events; store and visualize with DataHub or Apache Atlas/Amundsen for graph queries.- Ingestion: instrument producers to emit provenance to a lineage collector (Kafka producer hooks or Kafka Connect SMT emitting lineage events).- Spark: attach a SparkListener or use OpenLineage Spark integration to emit run/start/complete with input/output datasets and job id.- Warehouse: emit lineage when writing tables (e.g., capture INSERT/CTAS via hooks or query-engine integration).- Model registry: use MLflow or Sagemaker Model Registry capturing model artifact, training run id, training dataset snapshot id.- Store metadata in a graph DB (Neo4j) or a metadata catalog that supports graph traversal and time-travel queries.

How to query during an incident:1. Start from the bad prediction: identify model_version and inference_run_id, extract input feature values and feature ids.2. Query model registry for training_run_id → training_data_snapshot_id and feature lineage to find upstream datasets.3. Traverse graph upstream (model → features → transformations → Spark jobs → Kafka topics/timestamps) using API/graph queries: e.g., find all jobs that wrote to dataset X between t1–t2 or with code commit = C.4. Filter nodes by schema change (schema hash diff), partition offsets, or job config changes (new image digest or commit). Use checksums/row counts to spot data drift.5. If a candidate Kafka producer commit is found, inspect offsets and replay sample messages to reproduce.

Example query patterns:- Graph API: /lineage/upstream?dataset=prod.features.user_lifetime&since=2025-11-10T00:00Z- SQL against catalog: SELECT job, commit, start_time FROM job_runs WHERE output_table='warehouse.events' AND start_time BETWEEN ... ORDER BY start_time DESC

Operational practices:- Emit lineage synchronously as part of job lifecycle; persist lightweight snapshots for quick triage and full diffs for postmortem.- Alert on schema/hash changes, unusual row counts, or model metric drift and link alerts to lineage traces.- Maintain playbooks: from alert → model_run_id → lineage traversal → identify changed artifact → rollback or rerun.

This gives deterministic, fast root-cause tracing from prediction back to the upstream code or data change.

HardTechnical

45 practiced

Propose an approach to automatically distinguish between label drift and feature drift when you observe production model degradation. Explain how each type affects retraining cadence differently, and design an experiment using logged production data and a small labeled sample to confirm which drift is occurring.

Sample Answer

Situation: Production model accuracy has dropped. We need to determine whether the loss is due to feature (input) distribution drift or label (target) distribution drift — because each implies different mitigation and retraining cadence.

Approach summary- Monitor two families of signals: input-distribution metrics (feature drift) and target-distribution / label-quality metrics (label drift).- Use logged production inputs, model predictions, historical training data, and a small, random labeled sample from recent production to run hypothesis tests and an experiment that isolates causes.

Experiment design (step-by-step)1. Data collection - Pull a recent window of production logs: X_prod (N inputs) and model predictions y_hat_prod. - Retrieve matching historical training/validation data: X_train, y_train, y_hat_train (if available). - Sample and obtain true labels for a randomized subset of production examples: X_lab, y_true (recommended N_label ≈ 500–2000 depending on base rate and power you need).

2. Feature-drift checks (on X_prod vs X_train) - For numeric features: compute univariate KS tests or two-sample Wasserstein distances per feature; compute multivariate tests (e.g., MMD) or classifier-based drift (train a binary classifier to distinguish X_train vs X_prod; AUC >> 0.5 indicates drift). - Use PSI (population stability index) per feature; thresholds: PSI > 0.2 signals significant drift. - Check feature correlation and joint statistics (covariance shifts), and upstream data pipeline integrity (missingness, new categories).

3. Label-drift checks (on y distribution and model performance vs labels) - Compare marginal label distribution: p(y_true | time window) vs p(y_train). Use chi-squared or KL divergence; significant change suggests label prior drift. - Compare conditional p(y_true | model prediction) to historical calibration: for bins of y_hat_prod (or predicted probabilities), compute observed label rates from X_lab; large deviations indicate label drift or calibration loss. - If possible, inspect business metrics or external signals that could change base rates (seasonality, policy).

4. Causal experiment using the labeled sample - Compute model metrics on labeled sample: accuracy, precision/recall, calibration, and confusion matrix. - If feature drift but no label-drift: you expect p(y) similar to training, but model errors concentrated in inputs with high feature divergence (e.g., classifier distinguishing X_prod vs X_train correlates with error). - If label drift: overall model correctness drops but errors are uniformly distributed across regions of input space; calibration shows systematic offset (e.g., model consistently under/overpredicts positive class probability) and p(y_true) differs. - Concretely: regress per-example error indicator (1 if y_hat != y_true) against a “feature-drift score” (distance from training distribution). If errors correlate strongly with drift score → feature drift. If errors do not correlate, but overall label rates changed → label drift.

Statistical tests and thresholds- Two-sample tests: KS p < 0.01 or classifier AUC > 0.7 indicates feature shift.- Chi-square p < 0.01 or KL divergence large and >0.1 indicates label-prior shift.- Correlation (Pearson or logistic regression) between drift score and error rate: significance suggests feature-driven errors.

How each type affects retraining cadence & remediation- Feature drift: - Often fixed by re-training on recent data, domain adaptation, adding new feature engineering, or model architecture changes. - Retraining cadence increases: automate periodic model retraining (e.g., weekly/monthly) or trigger retrain when drift metrics exceed thresholds. - Also add feature validation and upstream fixes to prevent pipeline changes.- Label drift: - If labels’ semantic meaning or base rates changed (e.g., business process change), immediate retraining can help but is not always the right fix—often you must update label definitions, business rules, or post-processing (thresholds/calibration). - Retraining cadence might be less frequent; instead, focus on label collection, recalibration, or adjusting decision thresholds to match new priors (use Bayes rule for posterior correction when p(y) changes). - If label noise increased (annotation quality), invest in labeling quality rather than frequent retrains.

Example outcome interpretation- Case A: X_prod diverged strongly (PSI>0.3 on core features), classifier AUC=0.85, and errors concentrate on high-drift examples → feature drift. Action: collect more labeled data from affected regions, retrain with recent data, consider domain adaptation and tighter monitoring/alerting for feature PSI.- Case B: p(y_true) increased from 5%→12%, model calibration shows consistent underestimation but X_prod ≈ X_train → label-prior drift. Action: adjust decision thresholds or apply posterior correction; add periodic recalibration and update labeling policy; retrain on data with new priors if the relationship x→y changed.

Practical considerations- Choose sample size to get sufficient power for detecting shifts given base rates.- Automate daily computation of PSI, classifier-drift AUC, calibration plots, and a label-sampling pipeline to obtain fresh ground truth.- Use explainability (SHAP/per-feature attribution) to find which features drive changed predictions.- Maintain a playbook: thresholds for triggering labeled sampling, retrain vs recalibrate rules, and rollback procedures.

This approach gives a repeatable experiment and decision criteria to discriminate label vs feature drift, and prescribes different cadences and fixes appropriate to each type.

HardTechnical

36 practiced

You detect a 50% drop in a critical feature's cardinality that feeds an embedding layer, causing model instability. Describe a step-by-step root cause analysis tracing Kafka partitions, transformation stages, deduplication logic, and the feature store. Also propose immediate mitigations to stabilize predictions while you fix the upstream problem.

Sample Answer

Step-by-step RCA plan (trace-first, narrow-to-root):

1. Confirm symptom & scope- Verify cardinality drop (count unique keys feeding embedding) over time, compare baseline window vs current.- Identify affected model versions, deployments, and time range when drop began.

2. Trace Kafka ingestion- Query Kafka topic offsets, partition-level throughput and lag for timestamps around the change.- Check producer logs for errors, throttling, or key-affinity changes (e.g., partitioning key changed).- Run partition-level unique-key counts: consume a sample per partition and compute distinct key counts to see if loss is partition-local.

3. Inspect transformation pipeline- Review streaming job commits (Flink/Beam/Spark): job version, config changes, recent deploys.- Validate deserialization schema and schema registry compatibility—an incompatible schema can null keys.- Re-run the transform function on a historical raw sample to confirm deterministic behavior.

4. Check deduplication logic- Examine dedupe windows and watermark behavior. A window shrink or watermark regressing can drop events.- Search for modified dedupe keys or added joins that may filter keys unintentionally.- Replay raw events through dedupe step in a sandbox to measure unique keys preserved.

5. Verify feature store ingestion & serving- Check feature-store ingestion job logs and success metrics; look for rejected rows or TTL misconfigurations.- Validate join keys between Kafka-derived features and entities; mismatches can appear as missing cardinality.- Inspect store-level compaction/garbage collection that might have aggressively dropped entries.

6. Correlate config/deploy changes- Pull recent PRs/deploys across producer, transformer, deduper, and feature-store in the incident window.- Check infra autoscaling, network ACLs, or resource contention that could cause partial partition writes.

Immediate mitigations to stabilize predictions:

- Fallback embeddings: switch model serving to use last-known-good embedding table snapshot (read-only) or cached embeddings for missing keys.- Input gating: if many keys arrive as null/invalid, apply a validation layer to route invalid events to a quarantine topic and avoid poisoning the store.- Reduce dedupe aggressiveness: temporarily widen dedupe window or disable dedupe for affected keys to avoid unintended drops (with monitoring).- Feature imputation: for missing key embeddings, use a learned "unknown" vector or mean embedding computed from historical data.- Canary & throttles: route a small % of traffic to a canary model using safe fallback logic before rolling to all traffic.

Post-stabilization actions:- Reprocess raw topic from confirmed-good offset to repopulate missing cardinality into the feature store.- Add end-to-end cardinality alerts (partition-level unique-key metrics) and automated runbooks to reduce time-to-detect.- Implement schema and config change gating (CI checks, deploy windows) for producer/transformer/dedupe changes.

This approach isolates whether loss is at Kafka (producer/partitioning), transform/dedupe (logic bug or windowing), or feature-store ingestion, and gives immediate safe fallbacks to keep predictions stable while the upstream bug is fixed.

MediumSystem Design

33 practiced

Design a testing strategy for an ML feature pipeline to integrate with CI/CD. Describe unit tests for transforms, integration tests for the pipeline, data regression tests comparing feature statistics to a golden baseline, and performance tests. Be specific about mocked inputs, sample sizes, and failure modes to catch.

Sample Answer

Requirements & goals: ensure feature transforms are correct, stable vs data drift, performant, and safe to deploy automatically. Integrate tests into CI for fast feedback and CD gates for production.

Unit tests for transforms (fast, run on every commit)- Scope: pure functions/transform classes (scaling, normalization, categorical encoding, date features).- Mocked inputs: minimal handcrafted rows covering normal, edge (nulls, NaNs, extreme values, duplicate IDs, unexpected categories), wrong types.- Sample size: 10–50 handcrafted examples per transform.- Assertions: exact numeric outputs, dtype, shape, preserved ordering, consistent one-hot columns, no leakage (e.g., target not used).- Failure modes caught: off-by-one, wrong null handling, unseen-category crash, dtype casts.

Integration tests for pipeline (CI merge / nightly)- Scope: end-to-end feature pipeline from raw records → feature store export.- Mocked inputs: realistic synthetic dataset (1k–10k rows) generated to match production schema; include proportion of nulls, rare categories, timestamp ranges.- Assertions: schema match, aggregate checks (null counts, cardinalities), idempotence (running twice yields same output), end-to-end consistency with a downstream model stub returning predictions.- Failure modes: pipeline exceptions, serialization errors, stateful ops misbehaving, ordering/race conditions.

Data regression tests vs golden baseline (CD gate / production monitoring)- Baseline: store golden statistics from a stable window (mean, std, percentiles, cardinalities, null rates, histograms) per feature.- Sample size: daily/weekly batches (≥10k rows or entire batch).- Tests: - Statistical tests: population shift checks (KS test for continuous features, chi-squared for categoricals) with configurable alpha (e.g., 0.01). - Threshold checks: relative change limits (e.g., mean change >10% or cardinality increase >50% triggers warning). - Feature-level checks: new/unexpected categories, sudden drop in non-null fraction.- Failure modes: data/schema drift, pipeline regressions that change scaling, upstream bug introducing NaNs.

Performance tests (pre-deploy canary / periodic)- Scope: latency and throughput of batch and streaming transforms.- Workloads: synthetic production-like traffic (e.g., 100k rows for batch; 1k reqs/sec for streaming).- Metrics: p95/p99 latency, memory, CPU, GC pauses, end-to-end SLA (e.g., batch within X minutes).- Failure modes: memory leaks, backpressure, degraded p99 latency under load.

CI/CD integration & automation- Unit tests run on every PR; integration tests on merge to main or nightly; data regression and performance tests as CD gates on pre-production (canary) deployment.- Use feature flags / canary rollout: deploy to small fraction (1–5%) and run live data regression and performance checks before full rollout.- Alerts & rollback: automatic rollback if thresholds breached; create JIRA on failure; logging of diffs and sample rows that failed tests.

Observability & docs- Store golden stats and test results in a central dashboard; version baselines per model/feature release.- Keep test data generators, schema, and thresholds in repo; require PR updates when intentionally changing transforms.

This strategy balances speed (fast unit tests) and safety (integration, regression, and perf tests) and ties tests to CI/CD gates and production monitoring to prevent silent failures.

Unlock Full Question Bank

Get access to hundreds of Data Quality Debugging and Root Cause Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.