Data Preprocessing and Handling for AI Questions

Covers the end to end preparation of raw data for analysis and modeling in machine learning and artificial intelligence. Topics include data collection and ingestion, data quality assessment, detecting and handling missing values with deletion or various imputation strategies, identifying and treating outliers, removing duplicates, and standardizing formats such as dates and categorical labels. Includes data type conversions, categorical variable encoding, feature scaling and normalization, standardization to zero mean and unit variance, and guidance on when each is appropriate given model choice. Covers feature engineering and selection, addressing class imbalance with sampling and weighting methods, and domain specific preprocessing such as data augmentation for computer vision and text preprocessing for natural language processing. Emphasizes correct order of operations, reproducible pipelines, splitting data into training validation and test sets, cross validation practices, and documenting preprocessing decisions and their impact on model performance. Also explains which models are sensitive to feature scale, common pitfalls, and evaluation strategies to ensure preprocessing does not leak information.

HardTechnical

86 practiced

Hard: Case study — After merging two datasets for feature enrichment, downstream model accuracy dropped despite no change in model code. Describe a systematic investigation plan: which data checks to run, how to compare pre-merge and post-merge feature distributions, how to use unit tests and shadow runs to locate the issue, and how to communicate findings to stakeholders.

Sample Answer

Approach (overview): Treat this as a data-quality/regression investigation. Aim to (1) verify merge correctness, (2) detect distribution/feature shifts, (3) isolate which feature(s) harmed the model, (4) remediate and communicate clearly with stakeholders.

Step 1 — Immediate sanity & schema checks- Validate row counts pre/post-merge and expected join cardinality (inner/left/right). Alert on unexpected growth/shrinkage.- Verify join keys: nulls, duplicates, many-to-many vs expected one-to-one. Check foreign-key integrity.- Type & unit checks: datatypes, timezones, numeric scales (e.g., cents vs dollars), categorical encoding changes.

Step 2 — Data-drift and distribution comparison- For each feature: compare pre-merge vs post-merge using summary stats (mean, std, quantiles) and visualize (histograms, boxplots).- Statistical tests: KS-test for continuous, chi-square for categorical, Population Stability Index (PSI) for production drift.- Track missingness per-feature and per-segment (user cohorts, time buckets).- Compute correlation/covariance change and feature importance (SHAP/weights) differences.

Step 3 — Isolate impact with targeted experiments- Shadow runs (offline): score a held-out validation set with pre-merge features vs post-merge features; compute per-sample delta in prediction and loss.- Ablation tests: incrementally add merged features back to the original feature set to find which feature(s) cause the drop.- Feature toggle / canary in production: serve model to a small % using post-merge features and compare metrics against control.

Step 4 — Automated tests & CI- Unit tests for merge logic (deterministic small fixtures verifying expected output).- Data contracts: assertions for row counts, key uniqueness, acceptable null thresholds, allowed categories.- Regression tests: replay historical data through the new pipeline and check that metrics (AUC, RMSE) do not regress beyond threshold.

Step 5 — Root cause hypotheses & remediation- Common causes: incorrect join (duplicated rows), join key mismatch (shifting population), unit/scale mismatch, label leakage introduced, or feature distribution shift for critical features.- Remediate: fix join logic, dedupe, correct scaling/encoding, drop/leak-filter offending features, retrain if feature semantics changed.

Communication plan- Immediate: short incident summary to stakeholders (what changed, observed impact, initial mitigation like rollback or canary), estimated timeline.- Findings: present evidence—key checks, visual diffs, ablation results, and recommended fix with risk assessment.- Next steps: short-term (rollback/patch), medium (tests & CI), long-term (monitoring, data-contracts).- Artifacts: attach reproducible notebooks, logs, unit test results, shadow-run metrics, and a rollout/validation checklist.

Metrics to report: change in model metric (delta AUC/accuracy), percent of affected records, sample counts, and confidence intervals from shadow runs. Conclude with clear ask (approval for rollback, engineering time, or retraining).

HardTechnical

67 practiced

Case study: A retail recommendation model starts returning worse results after adding a new preprocessing step that buckets prices into 10 bins. Walk through how you'd diagnose whether bucketing or a downstream change caused the regression. Include specific analyses, checks, and rollbacks you would perform.

Sample Answer

Clarify scope & impact- Ask: when did regression start, which metric(s) dropped (CTR, conversion, NDCG), percentage change, affected cohorts, and deploy timeline (bucketing-only vs any downstream code). Get deployment logs and commit IDs.

Form hypotheses- H1: bucketing caused regressions (feature discretization loss, boundary effects, information leakage).- H2: downstream change caused regressions (scoring, ranking, business rules).- H3: interaction between bucketing and downstream logic.

Immediate checks (fast, low-cost)- Reproduce locally: run current model code on pre-change and post-change data with/without bucketing.- Sanity stats: compare feature distributions before/after bucketing (counts per bin, empty bins), check for NaNs/inf, label distribution per bin.- Data-drift and leakage: verify bucketing used training-time bin edges vs production edges; ensure same encoder parameters.

Ablation & isolation experiments- Backfill: run the new preprocessing (bucketing) with the previous downstream code; run the new downstream code without bucketing. Compare offline metrics (validation loss, AUC, NDCG) to isolate which change moves metrics.- Shadow/canary: deploy bucketing-only to a small fraction of traffic (or shadow it) and compare online metrics against control.

Model explainability & diagnostics- Feature importance / SHAP before vs after: did bucketed price contribute less or introduce spurious effects?- Per-bin lift: compute CTR/conversion by price bin to see where performance diverges.- Calibration and score distribution: check predicted score histograms and conversion calibration per bin.

End-to-end A/B test- If offline tests inconclusive, run an A/B: control (no bucketing) vs treatment (bucketing) with only that change; monitor core metrics and guardrail metrics.

Rollback plan & remediation- If bucketing is root cause, roll back preprocessing or use finer bins / target-encoded or learned embeddings; retrain model using continuous price or engineered features (log-price, interactions).- If downstream code is culprit, revert or patch logic; add unit/integration tests to prevent regressions (e.g., consistent encoder serialization, bin-edge checks).- Add monitoring: per-feature telemetry, upstream/downstream validation, shadow deployments for future changes.

Prevent recurrence- CI: include offline metric checks for preprocessing changes, schema validators, and automated backtests comparing new preprocessing against baseline on holdout datasets.- Documentation: versioned preprocessing artifacts (bin edges), and stricter change review for feature transformations.

This approach isolates cause via reproducible offline ablations, targeted canary/A-Bing, diagnostics (per-bin and SHAP), and safe rollback with longer-term fixes (better encoding, tests, monitoring).

MediumTechnical

87 practiced

You have a numeric feature with a heavy right skew and outliers. Explain when to use log transform, Box-Cox, or Yeo-Johnson transforms during preprocessing. For each transform, state constraints (e.g., input domain) and how you'd validate that applying the transform improved model performance.

Sample Answer

Use-case summary: all three transforms aim to reduce skew and stabilize variance so models that assume normality or linear relationships (linear models, Gaussian processes) perform better and outliers have less leverage. Choice depends on data domain (positivity) and flexibility needed.

Log transform- When: heavy right skew, strictly positive values, simple, interpretable (multiplicative effects).- Constraint: input must be > 0 (or shift by a constant if zeros present: log(x + c) with c>0).- Pros/cons: very effective for multiplicative relationships; can underfit if negative or zero values or if skew is complex.- Validation: compare pre/post histograms, skew/kurtosis, Q–Q plots; train with cross-validation and compare metrics (RMSE/AUPRC/AUC depending on task), check residual homoscedasticity, and whether coefficients become more stable/interpretable. Back-transform predictions and report error on original scale (e.g., use median or bias-corrected retransformation for RMSE).

Box–Cox- When: you need a parametric family that includes log (λ→0) and other power transforms; data strictly positive.- Constraint: x > 0; requires estimating λ (maximum likelihood typically).- Pros/cons: flexible, often yields near-normality; less interpretable than raw log but more adaptable. Needs care if optimal λ is near 0 (log-like) or extreme.- Validation: inspect optimized λ and resulting distribution (histogram, skewness), use likelihood or AIC on transformed data, run CV comparing model metrics and residual diagnostics; ensure transformation stable across folds and not overfitting.

Yeo–Johnson- When: data contains zeros or negatives but you still want a power-like family; generalized Box–Cox for nonpositive values.- Constraint: accepts x ∈ R (including 0 and negatives); estimates λ similarly.- Pros/cons: flexible with broader domain; slightly more complex inverse transform; may be preferred when shifting is undesirable.- Validation: same process—check distributional improvement, CV model metrics, residual plots, and consistency of λ. For tree-based models (random forests, gradient boosting) transformations may be less beneficial; validate empirically.

Practical validation checklist (apply in a pipeline):1. EDA: histograms, log plots, skew/kurtosis, outlier inspection.2. Fit transform inside cross-validation (use Pipelines so transform is learned on train only).3. Compare baseline vs transformed models using appropriate CV metrics and statistical tests (paired t-test or Wilcoxon on fold metrics).4. Check residuals, calibration, and feature importance stability.5. Back-transform predictions and evaluate on original scale.6. If model performance or stability doesn't improve, prefer simpler options (no transform or robust scaling) or use models robust to skew (tree ensembles).

This approach balances domain constraints, interpretability, and empirical validation to choose the right transform.

EasyTechnical

64 practiced

Easy: For a numeric column with occasional extreme outliers and many zeros (zero-inflated), explain simple preprocessing options (e.g., winsorization, clipping, log(1+x), indicator for zero). Recommend a default approach for tree models and a default for linear/GLM models, with brief justification.

EasyTechnical

76 practiced

Implement in Python a function detect_outliers_iqr(df, column, k=1.5) that returns the indices of rows considered outliers using the IQR method (below Q1 - k*IQR or above Q3 + k*IQR). The function should handle NaNs gracefully and work for numeric columns stored as strings. Provide a simple example and mention limitations of IQR for skewed distributions.

Sample Answer

Approach: convert the column to numeric (coerce invalid values to NaN), compute Q1/Q3 on non-NaN values, compute IQR, then find indices where values < Q1 - k*IQR or > Q3 + k*IQR. Return original row indices; ignore NaNs (they are not flagged as outliers).

python

import pandas as pd
from typing import List, Union

def detect_outliers_iqr(df: pd.DataFrame, column: str, k: float = 1.5) -> List[Union[int, str]]:
    """
    Return list of row indices in df considered outliers by the IQR method.
    Handles numeric columns stored as strings and NaNs gracefully.
    """
    # Copy column and coerce to numeric (invalid -> NaN)
    col = pd.to_numeric(df[column], errors='coerce')
    
    # Compute Q1 and Q3 excluding NaNs
    q1 = col.quantile(0.25)
    q3 = col.quantile(0.75)
    iqr = q3 - q1
    
    # If iqr is zero (all equal or insufficient data), return empty list
    if pd.isna(iqr) or iqr == 0:
        return []
    
    lower = q1 - k * iqr
    upper = q3 + k * iqr
    
    # Boolean mask for outliers; NaNs will be False
    mask = (col < lower) | (col > upper)
    
    # Return indices (preserves original index type)
    return list(df.index[mask])

Example:

python

df = pd.DataFrame({'value': ['10', '12', '11', '300', '13', None, '9']})
print(detect_outliers_iqr(df, 'value'))  # might return [3] as 300 is an outlier

Key points:- Time complexity O(n) for a single pass and quantile computation.- Space complexity O(n) due to creating a numeric series.

Limitations:- IQR is robust to moderate outliers but can be misleading for heavily skewed or multimodal distributions — it may label many typical tail observations as outliers or miss context-specific anomalies. Consider transformations (log), robust z-scores, or model-based methods for skewed data.

Unlock Full Question Bank

Get access to hundreds of Data Preprocessing and Handling for AI interview questions and detailed answers.

Join thousands of developers preparing for their dream job.