Data Quality and Bias Questions

Covers both the conceptual and technical aspects of data quality assessment, bias identification, and remediation. Candidates should be able to recognize common sources of bias including selection bias, confirmation bias, measurement bias, and sample limitations, and describe how these biases and methodological limitations affect conclusions. They should be able to document and communicate caveats and limitations clearly and responsibly. On the technical side, candidates should demonstrate techniques for detecting and handling missing values, duplicates, outliers, and inconsistent data types; explain trade offs between filtering, imputing, and transforming data; and discuss how data cleaning choices influence downstream analysis. Additional expected skills include validating cleaned data against expectations, performing sensitivity analyses to show how results change under different data handling decisions, tracking data provenance, and describing reproducible processes for data quality management.

EasyTechnical

90 practiced

Explain how and why you would use stratified sampling for train/validation/test splits. Provide a short scikit-learn example that performs a stratified split on a multi-class label and discuss limitations (for example, tiny subgroup sizes and when stratification is not sufficient).

Sample Answer

Stratified sampling ensures that train/validation/test splits preserve the label distribution (class proportions). I use it when class imbalance exists or when each class must be represented in every split so metrics and model selection are reliable. For multi-class problems, stratification prevents rare classes from disappearing in a split.

Approach:- Use scikit-learn's train_test_split with stratify (or StratifiedShuffleSplit) and a fixed random_state for reproducibility.- If you need three splits, do two-stage splitting (train / temp, then val / test) while stratifying both times.

Example:

python

from sklearn.model_selection import train_test_split
import numpy as np

X = np.arange(100).reshape(100, 1)        # dummy features
y = np.array([0]*70 + [1]*20 + [2]*10)    # imbalanced 3-class labels

# Train / temp (20% holdout)
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# Validation / test split from temp (each 10% of original)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42
)

Key points:- Always stratify on the target when the target distribution matters for training/validation.- Use random_state to make splits reproducible.

Limitations and caveats:- Tiny subgroup sizes: if a class has fewer samples than the number of folds/splits, stratification will fail or produce unstable estimates. Consider up/down-sampling, stratified k-fold with careful grouping, or combine rare classes.- Stratification only preserves marginal label distribution; it won’t preserve covariate distributions or interactions (covariate shift). If temporal/order effects exist, use time-based splits instead.- Cannot stratify on multiple columns directly; create a combined key (e.g., class + demographic bucket) but beware of explosion of strata and tiny cells.- Avoid leakage: ensure stratification doesn’t break grouping constraints (use GroupKFold when samples are not independent).

EasyTechnical

69 practiced

Explain, in your own words, the following data biases as they apply to machine learning systems: selection bias, confirmation bias, measurement bias, and sample limitations. For each bias give a concrete example from a typical ML pipeline (data collection, labeling, training) and describe how that bias can distort model outcomes or evaluations.

MediumTechnical

94 practiced

Fairness metrics question: For a binary classifier, compare and contrast statistical parity, equalized odds, equal opportunity, and calibration by group. Provide scenarios where optimizing for one metric can worsen another and give practical guidance for choosing a metric for deployment.

Sample Answer

Definitions (brief):- Statistical parity (demographic parity): P(Ŷ=1 | A=group1) = P(Ŷ=1 | A=group2). Positive prediction rates equal across groups.- Equalized odds: For each true label y∈{0,1}, P(Ŷ=1 | Y=y, A=group1) = P(Ŷ=1 | Y=y, A=group2). Equal TPR and FPR across groups.- Equal opportunity: Relaxed equalized odds requiring only equal TPRs: P(Ŷ=1 | Y=1, A=...) equal across groups.- Calibration by group: For each score s, P(Y=1 | score=s, A=group1) = P(Y=1 | score=s, A=group2). Predicted probabilities reflect true risk within each group.

Key trade-offs and conflicts:- Impossibility result: If base rates P(Y=1 | A) differ across groups, you generally cannot satisfy both calibration-by-group and equalized odds simultaneously (except trivial cases). Calibration + different prevalences implies different error rates.- Optimizing statistical parity can increase harmful errors: forcing equal positive rates may raise FPR for a low-risk group or lower TPR for a high-risk group, hurting accuracy and individuals who need positive outcomes.- Pursuing equalized odds can break calibration: post-processing that enforces equal FPR/TPR (thresholding per group) typically produces group-specific score-to-probability mappings, harming calibration.- Equal opportunity focuses on TPR parity; it may still allow unequal FPRs, which could create more false positives in one group.

Practical guidance for deployment:1. Clarify harm model: Is false negative or false positive more harmful? If denying beneficial services is critical, prefer equal opportunity (equalize TPR). If harms from false positives matter (e.g., wrongful detention), prioritize FPR constraints or equalized odds.2. Legal/regulatory constraints: Some jurisdictions require demographic parity or prohibit use of protected attributes—align metric choice with legal context.3. Use calibrated scores for decision-making when risk ranking matters (e.g., resource allocation), but accept you may need additional constraints to control disparate errors.4. Multi-objective approach: Combine optimization (regularization during training), thresholding, and post-processing with stakeholder cost functions; measure downstream impact, not only metrics.5. Monitor continuously: track group-wise calibration, TPR, FPR, and base rates post-deployment; retrain when data shifts.6. Transparency and governance: document trade-offs, provide explanations for chosen metric, and allow human oversight for high-stakes cases.

Example scenario: Credit scoring with different default rates. Calibrated scores let lenders price risk fairly, but equalized odds would force lenders to accept more high-risk applicants from low-default groups or reject more from high-default groups—leading to revenue or fairness concerns. Choose based on whether you prioritize risk-based pricing (calibration) or parity of outcomes (equalized odds/equal opportunity).

MediumTechnical

72 practiced

Implement and describe an approach to detect near-duplicate textual records at scale (tens of millions) using MinHash or locality-sensitive hashing. Describe the data structures, memory and time trade-offs, parameters (shingle size, number of hash functions), and how to evaluate precision and recall.

Sample Answer

Approach (summary): Use shingling → MinHash signatures → LSH banding to bucket similar signatures. At scale, stream documents, compute k-shingles, update compact MinHash signatures, and use hash tables (buckets) per band to emit candidate pairs for exact Jaccard verification.

Code (sketch, Python—illustrative, not fully optimized):

python

import mmh3
import random

def shingles(text, k=5):
    s = set()
    text = text.lower()
    for i in range(len(text)-k+1):
        s.add(text[i:i+k])
    return s

class MinHash:
    def __init__(self, num_hash=128, max_hash=2**32-1):
        self.num_hash = num_hash
        self.hash_a = [random.randint(1, max_hash) for _ in range(num_hash)]
        self.hash_b = [random.randint(0, max_hash) for _ in range(num_hash)]
        self.max_hash = max_hash

    def signature(self, shingles):
        sig = []
        for a,b in zip(self.hash_a, self.hash_b):
            minv = min(((a * mmh3.hash(s, signed=False) + b) % self.max_hash) for s in shingles)
            sig.append(minv)
        return sig

def lsh_banding(sig, bands=32):
    rows = len(sig)//bands
    buckets = []
    for b in range(bands):
        start, end = b*rows, (b+1)*rows
        band_hash = mmh3.hash_bytes(bytes(str(sig[start:end]),'utf8'))
        buckets.append((b, band_hash))
    return buckets

Key design choices & parameters:- Shingle size k: 3–6 (characters) or 5–10 (words). Character k=5 often balances noise vs. sensitivity for short text.- Num hash functions (signature length): 64–256. More hashes → better estimate of Jaccard, higher memory/time.- Bands and rows for LSH: choose bands*b = num_hash. Trade-off: similarity threshold t ≈ (1/b)^(1/rows). E.g., 128 hashes, 32 bands × 4 rows → threshold ≈ 0.7.- Data structures: store signatures as uint32 arrays (memory compact). LSH index: per-band hash tables mapping bucket -> list of doc IDs; consider Redis or RocksDB for persistence. Use probabilistic counters or bloom filters for memory reduction.- Memory/time trade-offs: Increasing num_hash improves accuracy but increases memory (num_docs * num_hash * 4 bytes). Example: 50M docs × 128 hashes × 4B ≈ 25.6 GB (plus index). For very large scale, compute signatures on-the-fly and store only band buckets or use compressed signatures (e.g., 64-bit).

Scalability:- MapReduce/beam: map compute signature per doc, emit (band, bucket) -> doc_id, reduce to assemble candidate lists. Use blocking to avoid quadratic comparisons.- Streaming: maintain per-band buckets in sharded key-value stores; periodically expire low-frequency entries.

Evaluation (precision/recall):- Build labeled ground-truth pairs (sampling + manual labeling or synthetic duplicates with noise).- Precision = TP / (TP+FP) among reported near-duplicates; recall = TP / (TP+FN) relative to ground truth.- Vary (bands, rows, num_hash) and shingle size, plot precision-recall curve. Also measure candidate reduction factor = (#candidates)/(N choose 2).- Use ROC-like curves for thresholding on estimated Jaccard from MinHash.

Edge cases & improvements:- Very short documents: shingling noisy—use word-level or whole-document tokens.- Near-duplicates with reordering: use token-level shingles.- Use weighted MinHash / SimHash for cosine similarity if features are TF-IDF vectors.- Post-filter with exact Jaccard on original shingles to remove false positives.

This design balances accuracy and scalability by tuning signature length and banding; implement with distributed storage and streaming pipelines for tens of millions of records.

MediumTechnical

92 practiced

Describe how you would design an A/B experiment to evaluate whether removing outliers from training data improves live model performance. Specify randomization unit, offline vs online metrics to monitor, sample size and power calculation considerations, and safety guardrails to detect harm quickly.

Sample Answer

Situation: We want to test whether removing outliers from training data (e.g., top/bottom 1% by target or feature z-score > 3) improves live model performance.

Randomization unit:- Use user or session as the randomization unit depending on serving granularity to avoid cross-contamination (e.g., user-level if model personalization exists). Randomize at request/user ID and persist assignment in serving to ensure consistent treatment throughout the experiment.

Offline vs online metrics:- Offline (pre-deploy) metrics: validation AUC/ROC, calibration (Brier score), bias/variance diagnostics, uplift in held-out test set, and per-segment performance to check distributional effects.- Online (primary) metrics: business-facing key metric (conversion rate, revenue per user, MAE for regression), plus model-quality metrics: prediction accuracy, calibration drift, latency, and downstream metrics (e.g., retention). Monitor safety metrics (error rate on critical segments, false positive rate for protected groups).

Sample size & power:- Estimate baseline rate and variance from historical logs. Choose Minimum Detectable Effect (MDE) meaningful to business (e.g., 1% relative lift). Use standard two-sample proportion or mean power formulas with alpha=0.05 and power=0.8–0.9. Account for clustering (intra-class correlation) if randomizing sessions nested in users; inflate sample size by design effect = 1 + (m-1)*ICC. Also account for expected traffic allocation (50/50 or weighted) and potential data loss.

Safety guardrails / monitoring:- Pre-registered primary metric and one-sided hypothesis if justified. Implement near-real-time monitoring dashboards with automated alerts for: - Statistically significant negative drop beyond a safety threshold (e.g., >0.5*MDE or >3σ) on primary business metric. - Increases in error rates for critical or minority segments. - Latency or resource regressions.- Use sequential testing controls (alpha spending / O’Brien-Fleming) for interim looks, or Bayesian stopping rules to avoid inflated false positives.- Canary rollout: start with small % of traffic (e.g., 1–5%), validate no regressions, then ramp to full experiment.- Automatic rollback: if safety conditions trip, instantly route traffic to control and trigger incident process.

Additional notes:- Stratify randomization on key covariates to balance known confounders.- Log enough telemetry for offline analyses and to investigate heterogenous treatment effects.- After lift is confirmed, run longer-term monitoring post-rollout to catch delayed harms (concept drift, fairness issues).

Unlock Full Question Bank

Get access to hundreds of Data Quality and Bias interview questions and detailed answers.

Join thousands of developers preparing for their dream job.