Feature Engineering and Selection Questions

Covers the end to end process of transforming raw data into predictive and stable model inputs and choosing the most useful subset of those inputs. Topics include generating features from domain signals and timestamps, numerical transformations such as scaling binning and logarithmic transforms, categorical encodings including one hot and target encoding, creation of interaction and polynomial features, construction of dense feature vectors for model consumption, handling missing values and outliers, and strategies for class imbalance. Also includes feature selection and dimensionality reduction methods such as filter techniques statistical tests wrapper methods embedded model based selection mutual information analysis and tree based importance measures. Emphasis is placed on avoiding data leakage validating feature stability over time interpreting feature contributions and documenting rationale for feature creation or removal. For senior roles include designing feature engineering best practices mentoring others and considering feature impact on model interpretability and business metrics.

MediumTechnical

23 practiced

Explain practical steps to avoid target leakage when constructing features and labels for time-series forecasting and time-dependent supervised tasks. Cover creating explicit cut-off times, lagging features, rolling-window label creation, backtesting with rolling windows, use of purge and embargo in CV, and checks you would run to validate no look-ahead information slipped into features.

Sample Answer

**Situation & goal**I'll describe concrete steps I use as an applied scientist to prevent target leakage in time-dependent supervised tasks and time-series forecasting.

**1) Explicit cut-off times**- Define a single cut-off timestamp per training example that represents the moment features are observed and label is unknown.- Build a feature extraction pipeline that only reads rows with timestamp <= cut-off. Treat cut-off as the source of truth in joins (e.g., join on entity and timestamp <= cut-off, then aggregate).

**2) Lagging features**- Compute time-lagged versions of predictors (t-1, t-7, etc.) using groupby+shift so values come strictly from prior periods.- Avoid using rolling aggregates that include the current period unless you explicitly exclude it (use min_periods and window offsets).

**3) Rolling-window label creation**- Create labels by evaluating outcomes over a future window relative to cut-off (e.g., label = event occurs in (cut-off, cut-off + H]).- Store label generation logic separately and compute labels only after features are frozen to avoid mixing.

**4) Backtesting with rolling windows**- Use expanding or rolling backtest splits where each validation fold uses cut-offs later than any training cut-offs.- Example: training cut-offs up to T1, validation cut-offs in (T1, T2], test after T2.

**5) Purge and embargo in cross-validation**- When using overlapping samples (e.g., event windows), purge training samples that overlap validation label windows to remove direct leakage.- Apply an embargo period after each validation cut-off: exclude a short temporal buffer from training to avoid near-future leakage from data propagation or reporting delays.

**6) Validation checks for look-ahead leakage**- Forward-only pipeline test: re-run feature generation in a time-ordered loop and assert no future timestamps are accessed.- Time-shifted feature importance: recompute permutation importance after shifting features forward; features that suddenly gain predictive power may leak.- Correlation by lag: compute cross-correlation between each feature and the label across lags; unusually high correlation at lag 0 or negative lags signals leakage.- Unit tests with synthetic data: insert a synthetic future-only signal and ensure pipeline cannot use it.- Inspect model coefficients/SHAP: high weight on features that are aggregated from future windows is a red flag.- Data freshness audit: check for joins on last-observed timestamps (e.g., last_event_time) that implicitly encode future availability.

**Practical notes & trade-offs**- Keep feature-engineering code deterministic and versioned; run offline replay tests.- Purge/embargo reduces training data; choose window lengths based on domain (reporting delays, action latency).- Document cut-off semantics and enforce via CI tests before model promotion.

This approach ensures every model input is causally consistent with the prediction time and provides reproducible defenses against subtle look-ahead leakage.

EasyTechnical

22 practiced

How would you construct dense feature vectors for a deep learning model from tabular data containing numeric, categorical, and short-text fields? Describe preprocessing for numeric features, embedding approaches for categoricals, text vectorization options, handling missing values, and efficient storage/serialization formats for training and serving (for example TFRecords or compact flat arrays).

Sample Answer

**Overview / goal**Produce a single dense vector per row that preserves signal from numeric, categorical and short-text fields while being compact and fast for training and serving.

**Numeric preprocessing**- Scale to zero mean/unit variance or use quantile/robust scaler for heavy tails.- Optionally bucketize (learnable or fixed) and one-hot or embed buckets.- Add per-feature missing-value indicator bits instead of imputing blindly.

**Categorical embeddings**- Map categories to integer IDs; use learned embeddings (size rule: min(50, 1.6 * num_unique^0.56) or 8–64 typical).- Handle rare categories via "other" or hashing; maintain OOV id for inference.- Consider multiple-lookups (multi-hot) for multi-valued fields and sum/mean or attention pooling.

**Text vectorization**- For short text (<50 tokens): pretrained sentence encoders (MiniLM, SBERT) for rich dense vectors.- Lighter options: TF-IDF reduced with PCA/SVD or learned CNN/RNN/Transformer trained end-to-end.- Normalize text, remove PII, and optionally keep length/mask features.

**Missing values**- Numeric: impute (median) + missing flag; or use model-native handling if supported.- Categorical/text: use special token/id; keep presence flags.- Maintain consistent pipelines for train/serving.

**Concatenation & interaction**- Concatenate scaled numerics + embeddings + text vectors; optionally feed into a small dense projection layer to produce final packed vector and enable cross-feature interactions.

**Storage & serialization**- Training: TFRecords (sharded, Example protos) or Parquet with fast ingestion. Store dense precomputed vectors as float32 arrays, optionally quantized to float16 or int8.- Serving: compact flat arrays (row-major float16), memory-mapable numpy .npy/.npz or flatbuffers/Protobuf, or store embeddings in vector DB (FAISS, Milvus) for retrieval.- Version feature schemas and keep mapping artifacts (vocab, scaler params, tokenizer, embedding dims) in model registry.

This pipeline balances expressiveness and efficiency: choose heavier text encoders for accuracy and lighter preprocessing for large-scale low-latency use-cases.

MediumTechnical

23 practiced

Describe practical approaches to extract predictive features from short free-text fields for use in structured models and for deep learning models. Cover TF-IDF and n-grams for light-weight models, pretrained sentence or token embeddings (e.g., sentence-BERT) as dense features, learned embeddings or fine-tuning for deep models, and operational considerations such as caching, quantization, and latency.

Sample Answer

**Approach overview (why it matters)** For short free-text (titles, search queries, comments) you want features that are both predictive and efficient in production. I prioritize simple sparse features for baseline models and dense embeddings for richer models; trade-offs are accuracy vs latency/compute.

**Light-weight models: TF‑IDF + n‑grams**- Use character and word n‑grams (char 3–5, word 1–2) to capture misspellings and short phrases.- TF‑IDF with L2 normalization works well with linear models (Logistic Regression, GBT).- Feature selection: document frequency thresholds, mutual information or L1 regularization to reduce sparsity.- Advantages: interpretable, fast, tiny memory footprint.

**Pretrained dense features**- Use sentence-BERT or other sentence encoders to produce fixed-length vectors (e.g., 384/768 dims).- Use as input to tree/linear models or concatenate with TF‑IDF.- Good when semantic similarity matters (paraphrase, intent).

**Deep models / learned embeddings**- Two options: (1) Use pretrained transformer as frozen encoder + train task-specific head; (2) Fine-tune end-to-end when you have labeled data.- For very short text, lightweight models (DistilBERT, MiniLM) often suffice.- Learned token embeddings (train from scratch) only if domain vocabulary is specialized.

**Operational considerations**- Cache embeddings for frequent strings (hash + store) to avoid repeated encoding.- Quantization (8-bit quant / int8) and model distillation reduce model size and inference cost.- Batch requests, use GPU for throughput, CPU + optimized ONNX for low-latency CPU inference.- Measure p99 latency and memory; set SLAs and fallbacks (use TF‑IDF baseline if encoder unavailable).

**Example pipeline**- Precompute TF‑IDF sparse vector and cached SBERT embedding; feed both to a small MLP or GBDT. If label volume increases, experiment with fine-tuning a small transformer and compare recall/latency trade-offs.

This balances predictive power with production efficiency and is aligned with applied research and deployment constraints.

HardTechnical

26 practiced

Case study: A model meets accuracy SLA but feature computation cost and latency are high. Design an experimental protocol to find the minimal subset of features that maintains SLA while reducing cost. Include strategies for search (greedy ablation, importance-based pruning, recursive elimination), cross-validation and holdout design to control variance and false positives, statistical tests to compare models, cost metrics, and stopping criteria for the search.

Sample Answer

**Clarify objective & constraints**- Goal: find smallest feature subset that keeps accuracy ≥ SLA (e.g., AUC ≥ 0.82) while lowering compute cost/latency.- Constraints: production latency budget, cost model (per-feature compute/time), retraining budget, permissible risk (false-positive rate for SLA breaches).

**Overall protocol**1. Baseline: record full-model performance (metric ±CI), per-feature compute cost (CPU ms, memory, I/O), and end-to-end latency distribution.2. Split data: stratified time-aware split — temporal train (70%), validation (15) for search, final holdout (15%) frozen for confirmatory testing to control leakage and concept drift.

**Search strategies (use ensemble)**- Importance-based pruning: compute permutation/importances and rank features; try top-K thresholds (K deciles).- Greedy ablation (backward): iteratively remove the feature whose removal least degrades metric per unit cost saved (delta_metric / delta_cost).- Recursive feature elimination (RFE): retrain with cross-validated model, remove lowest-importance chunk, repeat.- Hybrid: start with importance pruning to remove obvious low-value features, then greedy ablation for fine tuning.

**Cross-validation & variance control**- Within validation: use repeated stratified K-fold CV (e.g., 5x3 repeats) to estimate metric variance per subset.- Use the frozen holdout only for final confirmation to avoid p-hacking.

**Statistical tests**- For candidate subset vs baseline, use paired tests across folds/samples: - Paired bootstrap CI for metric difference. - If normal: paired t-test; else Wilcoxon signed-rank.- Equivalence test: two one-sided tests (TOST) to assert metric difference is within SLA margin (e.g., -0.01 AUC).- Adjust for multiple comparisons (Benjamini-Hochberg) when evaluating many subsets.

**Cost metrics**- Primary: expected per-request compute time (ms) and dollar cost per 1M requests.- Secondary: 95th percentile latency, memory, feature extraction IO.- Define a composite objective: maximize cost_savings subject to metric >= SLA_margin with statistical confidence (e.g., 95% CI above SLA).

**Stopping criteria**- Stop when removing any additional feature either: - Causes metric to drop below SLA_threshold with statistical significance (e.g., lower 95% CI < SLA), or - Yields negligible extra cost saving (< X% or < Y ms) relative to engineering/retraining cost.- Also stop if search budget (retrain rounds) exhausted.

**Deployment & monitoring**- Promote candidate subset to shadow deployment; monitor live metrics and latency for a burn-in period.- If live metric CI remains ≥ SLA and latency/cost improved, roll out; otherwise revert and iterate.

This protocol balances systematic search, statistical rigor, and production constraints appropriate for an applied scientist.

HardTechnical

27 practiced

Compare using frozen pre-trained dense embeddings (e.g., sentence or entity embeddings) as features versus fine-tuning those embeddings end-to-end in a limited-data applied setting. Discuss trade-offs in expected accuracy gains, overfitting risk, compute and memory cost, update and deployment complexity, and propose decision criteria for choosing one strategy over the other in production.

Unlock Full Question Bank

Get access to hundreds of Feature Engineering and Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.