Data Quality and Bias Questions

Covers both the conceptual and technical aspects of data quality assessment, bias identification, and remediation. Candidates should be able to recognize common sources of bias including selection bias, confirmation bias, measurement bias, and sample limitations, and describe how these biases and methodological limitations affect conclusions. They should be able to document and communicate caveats and limitations clearly and responsibly. On the technical side, candidates should demonstrate techniques for detecting and handling missing values, duplicates, outliers, and inconsistent data types; explain trade offs between filtering, imputing, and transforming data; and discuss how data cleaning choices influence downstream analysis. Additional expected skills include validating cleaned data against expectations, performing sensitivity analyses to show how results change under different data handling decisions, tracking data provenance, and describing reproducible processes for data quality management.

EasyTechnical

92 practiced

You receive a dataset with many columns. Describe a repeatable approach to calculate per-column missingness rates and propose a sensible threshold policy to decide when to drop a column versus impute it. Include business-oriented considerations that might override a simple numeric threshold.

Sample Answer

A repeatable approach:1. Gather metadata: column name, type, source, cardinality, business owner.2. Compute per-column missingness (count and percent) programmatically.3. Classify columns by type (ID, target, numeric, categorical, timestamp, flag).4. Apply a threshold policy by class with business overrides.5. Log decisions and reasons in a data dictionary for auditability.

Example (pandas):

python

import pandas as pd

def missingness_report(df):
    n = len(df)
    return (pd.DataFrame({
        'dtype': df.dtypes.astype(str),
        'missing_count': df.isna().sum(),
        'missing_pct': df.isna().mean()
    }).sort_values('missing_pct', ascending=False))

# usage:
# report = missingness_report(df)

Sensible threshold policy (defaults for repeatability):- Critical fields (primary key, target, legal): never drop; escalate to business owner.- IDs/timestamps: tolerate low missingness (<1%); otherwise investigate source.- Targets: if missingness > 5% consider dropping rows or imputation only after stakeholder sign-off.- Numeric features: drop if missingness > 50%; else impute (median) and add indicator flag.- Categorical features: drop if missingness > 60%; else impute with "Missing"/mode and flag.- High-cardinality text/features: drop if >80% missing.

Business-oriented overrides:- Regulatory or contractual requirements may force retention despite high missingness.- If a column drives a key KPI or segmentation, prioritize collection/improvement rather than drop.- Cost/benefit: if imputation introduces unacceptable bias for a business decision, prefer dropping or designing new data collection.- Time sensitivity: for quick exploratory dashboards, use conservative thresholds; for production models, tighten rules and require validation.

Auditability:- Store the missingness report, chosen thresholds, imputation method, and stakeholder approvals in a central repository to ensure repeatability and transparency.

MediumBehavioral

67 practiced

Tell me about a time you suggested a change to upstream data collection or instrumentation to reduce bias or improve quality. What was the context, how did you convince stakeholders, what trade-offs did you consider, and what measurable impact did the change produce?

Sample Answer

Situation: At my previous company I was supporting weekly retention and conversion reports used by Product and Marketing. We kept seeing inconsistent lift in campaigns — sometimes large positive signals that disappeared on re-run. I traced the issue to upstream event instrumentation: the same user action was logged from web and mobile with different user_id schemes and no deterministic dedup key, causing overcounting and bias toward channels with heavier duplicate events.

Task: I proposed changing upstream instrumentation to emit a canonical user_id (hashed customer_id), a source_tag (web/mobile), and an event_uuid to allow deduplication within the event pipeline. The goal was to reduce measurement bias and improve quality for experiments and dashboards.

Action:- I ran an analysis showing current overcounting (≈18% inflation in conversions for mobile-heavy cohorts) and presented concrete examples to Product and Engineering.- Built a small proof-of-concept ETL that deduped on (event_uuid OR canonical user_id + timestamp window) and showed corrected metrics.- Met with stakeholders to explain trade-offs: ~2 weeks of engineering work, ~12% increase in event payload size, and slightly higher storage; benefit: more accurate metrics and fewer false-positive experiment results.- Negotiated a phased rollout: add canonical user_id and source_tag first, enable dedup logic in analytics layer, then roll event_uuid once engineering had bandwidth.- Provided acceptance criteria and monitoring: compare pre/post metrics, set anomaly alerts, and a rollback plan.

Result: After phased rollout the apparent conversion inflation dropped from 18% to under 2% for affected cohorts. Experiment false-positives decreased (we stopped 3 erroneous feature launches), and downstream analysts saved ~6 hours/week by removing manual dedup steps. Stakeholders adopted the new fields across products and the instrumentation standard became part of our onboarding docs.

Learning: Quantify the problem, show a low-risk proof, and present trade-offs clearly. That combination built trust and enabled a practical, measurable improvement in data quality.

MediumTechnical

67 practiced

You have a numeric KPI (average order value). Describe a sensitivity analysis plan to demonstrate how different imputation strategies (mean, median, forward-fill, model-based imputation) affect the KPI. Include how you'd present results to stakeholders and what threshold of change would trigger deeper investigation.

Sample Answer

Plan (objective): quantify how AOV (average order value) changes under four imputation strategies (mean, median, forward-fill, model-based) and determine whether differences materially affect conclusions or actions.

Steps:1. Data prep- Identify missingness pattern (MCAR/MAR/MNAR) with rates by customer, product, time.- Split dataset by relevant segments (channel, cohort) to test heterogeneity.

2. Impute & compute KPI- Implement four imputation pipelines: - Mean: replace missing AOV components with global mean - Median: use global median (robust to skew) - Forward-fill: per-customer last-observed value (time-series) - Model-based: train predictive model (e.g., gradient boosting) using features (customer, product, time, promotions) to predict missing values- For each method compute AOV and bootstrapped 95% CI across orders and by segment.

Example (pseudo-Python):

python

# compute AOV per imputation
def compute_aov(df): return df['order_value'].sum()/df['order_id'].nunique()
for method in ['mean','median','ffill','model']:
    df_imp = impute(df.copy(), method=method)
    aov, ci = bootstrap_aov(df_imp)

3. Compare results- Report absolute and relative differences vs. baseline (e.g., complete-case or mean).- Use paired bootstrap tests to assess statistical significance.- Show distributional differences (histograms, QQ-plots) of imputed values.

Visualization & stakeholder presentation- Executive slide: headline table with AOV per method, % change, and whether change exceeds threshold.- Dashboard: interactive selector to view segments; show: - Bar chart of AOV by method with error bars (95% CI) - Tornado/sensitivity plot (methods on y-axis, % impact on AOV) - Time series bands: baseline AOV with shaded regions showing range across imputations - Sample of imputed vs. observed values for model transparency- Include method descriptions, assumptions about missingness, and potential biases.

Decision rule / thresholds- Trigger deeper investigation if: - Relative change in AOV > max(5%, historical month-to-month volatility) OR - Change moves a key decision across a business threshold (e.g., from profitable to unprofitable segment), OR - Statistical significance (p < 0.05) combined with practical significance (>5%).- For borderline cases (3–5%), recommend targeted follow-up: examine high-impact segments, validate model imputations, or collect more data.

Deliverables- Notebook with reproducible pipelines, metrics, and tests- Dashboard for stakeholders with filters and method documentation- Recommendation memo: chosen imputation for production reporting, caveats, and plan for monitoring.

Why this approach- Compares simple robust methods to predictive methods, quantifies uncertainty, surfaces segment-level differences, and ties statistical and business significance to action.

MediumSystem Design

74 practiced

Design a reproducible, auditable data cleaning pipeline for a daily sales ingestion that collects CSVs from three sources, performs type coercion, deduplication, and enrichment, and writes a cleaned table to the warehouse. Describe components (code, tests, metadata), versioning, and how you'd capture provenance and failures.

Sample Answer

Requirements:- Daily ingest CSVs from 3 sources; produce one cleaned, deduped, enriched table in warehouse.- Reproducible, auditable, testable; capture provenance and failures; support re-runs.

High-level architecture:- Orchestrator: Airflow/Prefect schedules DAG to run daily and coordinates tasks.- Ingest layer: Source connectors fetch CSVs (S3/secure FTP). Save raw files to immutable "raw/" storage with filename, checksum, timestamp.- ETL code: Modular Python (pandas or PySpark depending on volume) in versioned repo. Steps: schema coercion, validation, deduplication, enrichment (lookup joins), write to "staging/" then to warehouse (e.g., BigQuery/Redshift).- Metadata & catalog: Store run metadata in a metadata DB (Postgres) or data catalog (Amundsen) recording run_id, source_files (paths + checksums), git commit, config, start/end times, row counts, diff stats.- Tests: Unit tests for functions, integration tests using fixture CSVs, and data quality checks (Great Expectations) executed in DAG; failing checks stop the pipeline.- Versioning: Code and configs in Git. Every scheduled run records git commit hash and pipeline config version in metadata. Data artifacts stored with run_id and checksum.- Provenance & auditing: Every output row includes provenance columns (ingest_run_id, source_file, source_row_hash). Metadata table links run_id → files → transformations. Store processing logs and stack traces in centralized logging (ELK).- Failure handling: Airflow retries for transient errors, immediate failure for validation errors with alerting (Slack/email). Support manual re-run by run_id; reprocessing uses same raw files and recorded git commit to guarantee reproducibility.- Example metadata record:{"run_id":"2025-12-01T02:00Z","git":"abc123","files":[{"path":"raw/sourceA/2025-12-01.csv","sha256":"..."}],"rows_in":12000,"rows_out":11850,"dq_pass":true}

Trade-offs:- Pandas is simpler for small data; Spark required for scale.- Immutable raw storage + checksums increases storage but ensures auditability.

This design ensures reproducible outputs (same inputs + git commit → same cleaned table), auditable lineage, automated QA, and clear failure/provenance visibility for analysts and stakeholders.

MediumTechnical

72 practiced

You discover that your customer dataset overrepresents urban users relative to the true population. Design a case study to adjust analyses for this sampling bias so that national-level revenue estimates are more accurate. Describe statistical techniques you'd use (weighting, stratification, post-stratification) and how you'd validate the adjusted estimates.

Sample Answer

Situation: Our user table (n=200k) is skewed 70% urban vs true population 50% urban -> national revenue estimates will be biased upward if urban users have higher ARPU.

Approach (case-study steps):1. Define target population and auxiliary variables: obtain national population margins for strata (urban/rural, age group, gender, region) from census or a trusted survey.2. Exploratory analysis: compare sample vs population margins; estimate raw revenue per-stratum (e.g., revenue_urban, revenue_rural).3. Construct weights: - Base weight = 1 / probability of selection if known; otherwise start unweighted. - Post-stratification: adjust weights so weighted sample margins match population margins for key strata. Use cell weights w_i = N_cell / n_cell. - If many cells or sparse cells, use raking (iterative proportional fitting) to match marginal distributions across multiple variables. - Alternative: propensity-score weighting — model P(sample | X) with logistic regression (X = demographics, region, engagement), and set weight = 1 / P̂. Combine with calibration to match population totals.4. Apply weights to compute national revenue estimate: total_revenue = sum_i w_i * revenue_i.5. Variance estimation: use Taylor linearization or bootstrap with weights to get standard errors and CIs.

Validation and robustness:- Benchmark weighted demographic margins against external sources not used in weighting (e.g., income distribution).- Split-sample validation: hold out a subset, train weights, and compare estimates.- Sensitivity analysis: compare estimates using different methods (post-stratification vs raking vs propensity) and different variable sets.- External validation: compare weighted national revenue per user to published industry or census-derived economic indicators if available.- Inspect influential weights: trim or winsorize extreme weights and report changes.- Report assumptions, residual bias risk, and present weighted estimate with CI plus sensitivity bounds.

Outcome: This pipeline produces adjusted national revenue estimates with quantified uncertainty, defensible assumptions, and validation against independent benchmarks — enabling stakeholders to make better-informed decisions.

Unlock Full Question Bank

Get access to hundreds of Data Quality and Bias interview questions and detailed answers.

Join thousands of developers preparing for their dream job.