Data Preprocessing and Handling for AI Questions

Covers the end to end preparation of raw data for analysis and modeling in machine learning and artificial intelligence. Topics include data collection and ingestion, data quality assessment, detecting and handling missing values with deletion or various imputation strategies, identifying and treating outliers, removing duplicates, and standardizing formats such as dates and categorical labels. Includes data type conversions, categorical variable encoding, feature scaling and normalization, standardization to zero mean and unit variance, and guidance on when each is appropriate given model choice. Covers feature engineering and selection, addressing class imbalance with sampling and weighting methods, and domain specific preprocessing such as data augmentation for computer vision and text preprocessing for natural language processing. Emphasizes correct order of operations, reproducible pipelines, splitting data into training validation and test sets, cross validation practices, and documenting preprocessing decisions and their impact on model performance. Also explains which models are sensitive to feature scale, common pitfalls, and evaluation strategies to ensure preprocessing does not leak information.

EasyTechnical

84 practiced

Which classes of machine learning models are sensitive to feature scale and why? Give examples of models that require scaling (k-NN, SVM, K-means, logistic regression, neural networks) and models where scaling is less critical (tree-based methods). Provide one short example where failing to scale harms model performance.

HardTechnical

69 practiced

A numeric feature is heavily right-skewed and causing model instability. Discuss candidate transformations (log, Box-Cox, Yeo-Johnson), how to choose among them (diagnostics, normality checks), handling zeros/negatives, and how to compute and store parameters to allow inverse transforms in production for interpretability.

Sample Answer

Start with the goal: reduce right skew to stabilize variance, make relationships more linear, and improve robustness. Candidate transforms:- Log(x + c): simple, widely used. Works well when distribution is positive and multiplicative effects dominate.- Box–Cox: finds power λ to maximize normality (requires strictly positive data). More flexible than log; λ=0 corresponds to log.- Yeo–Johnson: like Box–Cox but supports zero and negative values.

How to choose:- Visual diagnostics: histogram, Q–Q plot before/after transform.- Statistical checks: Shapiro–Wilk or Anderson–Darling for normality, skewness/kurtosis statistics, and examine residuals from model (homoscedasticity).- Predictive diagnostics: cross-validated model performance (RMSE/AUC) and stability of coefficients or feature importance.- Prefer the simplest transform that achieves acceptable diagnostics (e.g., log) unless Box–Cox/Yeo–Johnson yields materially better predictive/stability metrics.

Handling zeros/negatives:- For log/Box–Cox (positive required): add a shift c = max(ε, 1 - min(x)) so min(x + c) > 0. Choose ε small (e.g., 1e-6) but consider interpretability.- Yeo–Johnson handles zeros/negatives without shifting; use when shifting would distort meaning.

Computing & storing parameters for production:- Fit transform on training data and save: - For log: saved shift c. - For Box–Cox: saved λ and applied shift c (if used). - For Yeo–Johnson: saved λ (no shift).- Also save training-set mean and std (if you standardize after transform).- Store metadata in a small JSON or DB table: {feature_name, transform_type, lambda, shift, trained_on_date, training_stats}.- In production apply exactly the same steps (shift → transform → optional scaler). For interpretability, implement and expose inverse transforms using stored params to convert model outputs or feature effects back to original units.

Example Python (scikit-learn/scipy):

python

from scipy import stats
import joblib
# fit Box-Cox
x = train['feat'].values
shift = max(0, 1 - x.min())
x_pos = x + shift
x_bc, lam = stats.boxcox(x_pos)
joblib.dump({'type':'boxcox','lambda':lam,'shift':shift}, 'feat_transform.pkl')

# inverse
info = joblib.load('feat_transform.pkl')
orig = stats.boxcox_inv(transformed, info['lambda']) - info['shift']

Practical notes: document the business meaning of any shift, validate on recent data for drift, and prefer transforms that stakeholders can interpret when reporting metrics.

HardTechnical

69 practiced

You must combine images from different sources with varied resolutions and color spaces. Propose a preprocessing strategy that standardizes inputs while preserving label signal: include resizing policy (same shorter side vs fixed size), aspect-ratio handling, interpolation method, color-space conversion, mean/std normalization, and when to apply per-dataset domain adaptation.

Sample Answer

Start by clarifying labels and downstream model expectations (classification, detection, OCR) because preserving label signal depends on task.

Preprocessing pipeline (order matters):1. Ingest & metadata: record original size, source, color-space, DPI — useful for domain adaptation.2. Color-space conversion: convert everything to a canonical space (usually sRGB) using ICC profiles when available; if not, assume source and convert conservatively. For grayscale sources, replicate channel or keep single-channel depending on model input.3. Resize policy: resize by preserving aspect ratio using "shorter-side scaling" (scale so shorter side = S, e.g., 256 or 512) then apply a center/random crop to fixed model input (e.g., 224x224). This preserves composition and avoids distortion that could shift label signal (objects becoming unnatural). Use fixed-size direct warp only when geometric proportions are irrelevant.4. Aspect-ratio handling: avoid non-uniform scaling. For extreme aspect ratios, pad with mean/reflect padding to reach target then crop; keep padding flag in metadata.5. Interpolation: use bicubic (better visual fidelity) for up/down sampling; use area interpolation for aggressive downscaling to avoid aliasing. For label maps (segmentation), use nearest-neighbor.6. Mean/std normalization: compute per-channel mean/std on a representative subset (not mixing wildly different domains). Apply (x - mean) / std. If using pretrained models, match their normalization stats.7. Per-dataset domain adaptation: apply when performance shows domain shift (validate per-source). Techniques: per-dataset normalization (compute and apply dataset-specific mean/std), histogram matching to reference distribution, color jitter augmentation, or style-transfer-based augmentation. Prefer lightweight adaptation (per-dataset norm + augment) first; escalate to domain-adversarial or fine-tuning if labels available and gap persists.8. Validation & monitoring: keep per-source metrics (accuracy, calibration) and visualize samples pre/post processing; iterate.

This balances standardization with preserving signal and gives hooks for pragmatic domain adaptation.

EasyTechnical

87 practiced

As a Data Analyst, how do you document preprocessing steps to ensure reproducibility and stakeholder trust? Be specific about what to include in README/pipeline metadata (code version, transformer parameters, seeds, summary statistics before/after, sample rows, schema), and how to present this to non-technical stakeholders.

Sample Answer

Situation: For every analytics pipeline I build, I document preprocessing so results are reproducible and trusted by both engineers and business stakeholders.

What I include in README / pipeline metadata:- Identification: git repo + commit hash, pipeline version, release tag, author, timestamp.- Environment: python/R version, package list (requirements.txt / lockfile), container image or conda env.- Inputs/outputs: source tables/files with paths, data extraction timestamps, row counts.- Schema: column names, types, nullable flags, semantic descriptions (data dictionary).- Transformations: step-by-step list (e.g., "drop_nulls on col X", "one-hot encode col Y"), with transformer parameters (imputation strategy, thresholds, scaling method) and code snippet or function name.- Randomness: explicit seeds for sampling, train/test split, any shuffled operations.- Data quality & checks: summary statistics before/after (count, mean, std, missing%); anomaly flags; record-level checksum or sample hashes.- Samples: representative sample rows (before and after) and diff highlights.- How to run: exact commands, config file location, expected runtime, CI status / test coverage.- Provenance: links to raw data, upstream owners, and downstream reports/dashboards.

How I present to non-technical stakeholders:- One-page executive summary: purpose of preprocessing, main changes, and business impact (e.g., "Imputation increased usable rows from 72% to 98%, changing conversion rate estimate by +0.8%").- Visuals: before/after histograms, missingness heatmap, and a small table with key metric shifts.- Glossary: short, non-technical definitions for terms like "imputation" or "one-hot encoding".- Trust anchors: link to reproducible artifact (notebook/script + commit) and a short note on validation (e.g., "sanity checks passed, CI green on 2025-11-20").- Next steps / caveats: limitations and who to contact.

This mix of machine-readable metadata and plain-language summaries ensures reproducibility for engineers and clear trust for business stakeholders.

EasyTechnical

88 practiced

Define data leakage in the context of model training and validation. Provide a concise business example (e.g., including 'days_to_cancel' as a feature when predicting churn), explain how you'd detect such leakage during exploratory analysis, and list practical mitigation strategies.

Unlock Full Question Bank

Get access to hundreds of Data Preprocessing and Handling for AI interview questions and detailed answers.

Join thousands of developers preparing for their dream job.