Data Preprocessing and Handling for AI Questions

Covers the end to end preparation of raw data for analysis and modeling in machine learning and artificial intelligence. Topics include data collection and ingestion, data quality assessment, detecting and handling missing values with deletion or various imputation strategies, identifying and treating outliers, removing duplicates, and standardizing formats such as dates and categorical labels. Includes data type conversions, categorical variable encoding, feature scaling and normalization, standardization to zero mean and unit variance, and guidance on when each is appropriate given model choice. Covers feature engineering and selection, addressing class imbalance with sampling and weighting methods, and domain specific preprocessing such as data augmentation for computer vision and text preprocessing for natural language processing. Emphasizes correct order of operations, reproducible pipelines, splitting data into training validation and test sets, cross validation practices, and documenting preprocessing decisions and their impact on model performance. Also explains which models are sensitive to feature scale, common pitfalls, and evaluation strategies to ensure preprocessing does not leak information.

HardTechnical

84 practiced

Explain the concept of calibration in preprocessing (e.g., probability calibration) and whether any preprocessing steps can affect downstream model calibration. Provide examples of steps that improve or degrade calibration.

MediumTechnical

82 practiced

You have a categorical feature representing country codes. Some codes are invalid or obsolete. Propose a validation and standardization pipeline to map values to ISO-3166 codes and handle unknown/misspelled entries at scale.

MediumTechnical

85 practiced

Explain how you would detect and treat outliers in a numerical feature used for a regression model. Include both univariate and multivariate approaches and how treatment might differ by model type.

HardTechnical

87 practiced

You need to impute MNAR (missing not at random) data for a medical dataset where sicker patients are less likely to have follow-up lab tests. Discuss advanced strategies to handle MNAR data and how to evaluate their effectiveness ethically and statistically.

Sample Answer

Challenge: MNAR where sicker patients are less likely to have follow-up labs — missingness depends on unobserved health state. Strategies:

1) Explicit missingness modeling / selection models- Model the missingness mechanism jointly with outcome using a selection model (Heckman-style) or joint likelihood (e.g., pattern-mixture or selection models). Estimate parameters that link probability of missingness to latent health indicators.- Pros: Accounts for non-random missingness; can reduce bias.- Cons: Requires strong assumptions or instruments and careful identifiability checks.

2) Use proxies / auxiliary variables & causal inference- Leverage auxiliary data correlated with sicker status (medication, hospitalization, vitals) to model missingness. Use inverse probability weighting where the probability of being observed is modeled using proxies.- Pros: Reduces MNAR bias if proxies are good; more robust.- Cons: Needs rich covariates; risk of model misspecification.

3) Multiple imputation under MNAR (pattern-mixture, delta adjustment)- Perform multiple imputation with sensitivity analysis: use MNAR models by shifting imputed values (delta adjustments) and report range of downstream results.- Pros: Quantifies sensitivity to MNAR assumptions; communicates uncertainty.- Cons: Results depend on chosen deltas; requires stakeholder agreement.

4) Semi-supervised / generative models- Train generative models (e.g., VAE with missing-data likelihood) that model joint distribution and missingness indicator; incorporate prior beliefs about sicker patients.- Pros: Flexible; can leverage unlabeled patterns.- Cons: Complexity and potential identifiability issues.

Evaluation (ethical & statistical):- Sensitivity analysis: Report how inferences change under plausible MNAR scenarios; present ranges rather than single-point estimates.- External validation: Validate imputations against a subset where follow-up labs exist or against chart review; if possible, prospectively collect follow-up for a random sample.- Calibration & fairness checks: Ensure imputation doesn't systematically bias subgroups (age, ethnicity); check downstream model calibration and subgroup performance.- Transparency & consent: Document assumptions and limitations; involve clinicians to validate plausibility of imputed values.

Recommendation: Combine proxy-based missingness modeling with sensitivity analysis. Prioritize external validation on a labeled subset and clearly communicate uncertainty and assumptions to stakeholders for ethical deployment.

MediumTechnical

76 practiced

You have free-text product descriptions and need to create numerical features for a gradient-boosted tree model. Describe feature engineering approaches you would try (at least five), and explain which require training data leakage precautions.

Unlock Full Question Bank

Get access to hundreds of Data Preprocessing and Handling for AI interview questions and detailed answers.

Join thousands of developers preparing for their dream job.