Feature Engineering and Selection Questions

Covers the end to end process of transforming raw data into predictive and stable model inputs and choosing the most useful subset of those inputs. Topics include generating features from domain signals and timestamps, numerical transformations such as scaling binning and logarithmic transforms, categorical encodings including one hot and target encoding, creation of interaction and polynomial features, construction of dense feature vectors for model consumption, handling missing values and outliers, and strategies for class imbalance. Also includes feature selection and dimensionality reduction methods such as filter techniques statistical tests wrapper methods embedded model based selection mutual information analysis and tree based importance measures. Emphasis is placed on avoiding data leakage validating feature stability over time interpreting feature contributions and documenting rationale for feature creation or removal. For senior roles include designing feature engineering best practices mentoring others and considering feature impact on model interpretability and business metrics.

MediumTechnical

28 practiced

You're given two very high-cardinality categorical features A (1M unique) and B (500k unique). Describe pragmatic approaches to create interaction features between A and B without exploding cardinality or memory: include frequency thresholding, hashing, bucketization, learned pair embeddings, and sampling-based selection. Provide pseudocode for generating the top-K pairwise interactions ranked by lift or pointwise mutual information.

Sample Answer

When crossing two extremely high-cardinality categoricals, the goal is to capture useful interaction signal while keeping memory and compute reasonable. Pragmatic approaches:

- Frequency thresholding: keep only values of A/B above a count threshold; map rare values to "OTHER". This reduces candidate pairs sharply.- Pair frequency thresholding: only consider pairs seen >= t times in data (co-occurrence pruning).- Hashing (feature hashing): hash pair (A,B) into a fixed-size space (e.g., 2^20) to avoid explicit enumeration; collisions add noise but bound memory.- Bucketization / target-guided bins: map A and B into lower-cardinality buckets by target-statistics (e.g., mean label) or k-means on learned embeddings.- Learned pair embeddings: learn separate embeddings for A and B and combine (concat, hadamard) in a model instead of explicit cross; optionally learn a small projection for pairwise interactions.- Sampling-based selection: sample a subset of A or B (or use importance sampling weighted by frequency/variance) to evaluate candidate interactions cheaply.- Hybrid: first filter candidate values with frequency/target-statistics, then compute exact pair metrics for surviving pairs; use hashing for all others.

Pseudocode: compute top-K interactions by lift or PMI using map-reduce style counting, with candidate pruning by marginal frequency.

python

# python-like pseudocode
def top_k_pairs(records, K, min_count=10, min_pair_count=5):
    # records: iterable of (A, B)
    # 1. marginal counts
    countA = Counter()
    countB = Counter()
    total = 0
    for a,b in records:
        countA[a]+=1
        countB[b]+=1
        total+=1

    # 2. prune low-frequency margins
    validA = {a for a,c in countA.items() if c>=min_count}
    validB = {b for b,c in countB.items() if c>=min_count}

    # 3. pair counting only for surviving margins (streaming)
    pair_count = Counter()
    for a,b in records:
        if a in validA and b in validB:
            pair_count[(a,b)] += 1

    # 4. compute score (lift or PMI) and keep top-K
    heap = []  # min-heap of (score, (a,b))
    for (a,b), c_ab in pair_count.items():
        if c_ab < min_pair_count: 
            continue
        p_ab = c_ab / total
        p_a = countA[a] / total
        p_b = countB[b] / total
        # lift = p_ab / (p_a * p_b)
        # pmi = log2(p_ab / (p_a*p_b))
        score = p_ab / (p_a * p_b)  # lift
        if len(heap) < K:
            heappush(heap, (score, (a,b)))
        else:
            if score > heap[0][0]:
                heapreplace(heap, (score, (a,b)))

    return sorted(heap, reverse=True)

Notes and trade-offs:- Use Count-Min Sketch for marginals when memory tight.- Use stratified sampling if scanning full data is expensive.- Hashing is simplest for model inputs; explicit top-K pairs found above can be used as features while hashing the rest.- Learned embeddings avoid explicit crosses and generalize to unseen pairs but require model capacity and training data.- Monitor collisions (hash) and overfitting (rare pair memorization).

HardTechnical

20 practiced

You have baseline feature set A and candidate set B (A∪B is larger). Design an offline and online experimental plan to decide whether to deploy B: include offline cross-validation and holdout tests, uplift estimation, A/B rollout strategy with guardrail metrics, sample size calculation, monitoring windows, and rollback criteria if KPI degrades. Explain how to handle interaction effects and seasonality.

Sample Answer

Requirements & objective:- Decide whether to add candidate feature set B to baseline A by measuring causal uplift on primary KPI (e.g., conversion, revenue per user) while protecting guardrail metrics (latency, quality, retention, safety).

Offline phase (model & validation):1. Define datasets: train/val/test splits by user-time to avoid leakage (time-based holdout). Reserve a final untouched temporal holdout cohort for offline uplift estimation.2. Cross-validation: use k-fold time-series CV (rolling window) to estimate performance stability. Evaluate both predictive accuracy and treatment-effect models (if using uplift modeling).3. Uplift estimation: fit causal models (e.g., doubly-robust, causal forests, meta-learners) to estimate conditional average treatment effect (CATE). Report average treatment effect (ATE) and quantiles; provide confidence intervals via bootstrap.4. Simulated holdout: apply B vs A on held-out cohort in-silico using logged policy / inverse propensity scoring if prior logs exist (off-policy evaluation) to estimate expected online uplift and variance.5. Safety checks: measure latency, resource usage, failure modes on offline / staging runs.

Sample-size & statistical plan:- Determine minimum detectable effect (MDE) for primary KPI (relative or absolute), desired power (80–90%), alpha (0.01–0.05). Account for multiple comparisons (if testing subgroups) with Bonferroni or FDR.- Compute sample per arm: n = ( (Z_{1-α/2}+Z_{1-β})^2 * (σ^2_treatment + σ^2_control) ) / MDE^2. Inflate for expected attrition and seasonality.

Online A/B rollout:1. Stage 0 — Canary (1% users): short window (1–3 days) focusing on system stability, guardrails (errors, latency, anomaly in distributions).2. Stage 1 — Ramp to 10% over 3–5 days if no alerts.3. Stage 2 — Ramp to 50% and then 100% if uplift significant and guardrails pass.Randomization: user-level persistent assignment. Stratify by key covariates (region, device, user cohort) to control imbalance.

Metrics & guardrails:- Primary metric(s): KPI with pre-registered analysis plan, test statistic (t-test/GLM/bootstrapped CI) and one-sided vs two-sided decision.- Secondary & guardrail metrics: latency, error rate, retention, NPS, safety (toxicity), fairness metrics. Pre-specify thresholds for acceptable degradation.- Sequential testing: use alpha-spending (O’Brien-Fleming) or Bayesian decision rules to allow interim checks without inflating false positives.

Monitoring windows & cadence:- Real-time monitoring for system health and guardrails (alerts fire immediately).- Short-term evaluation window: first-week rolling metrics to catch early regressions.- Statistical evaluation window: until sample-size target met plus stability buffer (e.g., one business cycle / 7–14 days) to capture weekly seasonality.- Longer-term follow-up: 30/60/90-day cohorts for retention/longitudinal effects and model drift.

Rollback criteria:- Immediate rollback triggers: crash rate increase > X%, latency > threshold, critical safety violation.- KPI rollback triggers: primary KPI drops beyond pre-defined negative MDE with statistical significance (adjusted for peeking), or persistent degradation over monitoring window (e.g., >2 consecutive days beyond tolerance).- Automated rollback pipeline that can revert config and notify stakeholders; include manual override for measured trade-offs.

Handling interaction effects:- Pre-specify subgroup analyses (new vs returning users, region, device). Use interaction terms in GLM or CATE models to detect heterogenous effects.- If interactions significant, consider targeted rollout (rollout B only to segments with positive uplift) and run follow-up experiments to validate personalization policy.- Correct for multiplicity when interpreting subgroup tests.

Handling seasonality:- Use time-based blocking in randomization so treatment/control are balanced across days/weeks.- Ensure experiment spans at least one full seasonality period (e.g., full week, or business cycle) before final decision.- Adjust analyses with time-series models (difference-in-differences, holiday covariates) and compare post/pre deltas.

Decision rule & governance:- Pre-register success criteria: minimum positive ATE, no guardrail breaches, acceptable cost/latency.- If criteria met after statistical window, promote B to prod; if partial wins, consider hybrid or targeted deployment.- Post-deployment: continued monitoring for model drift, feedback loops, and scheduled re-evaluation.

This plan balances rigorous offline causal estimation with staged online validation, clear statistical controls, safety guardrails, and provisions for interactions and seasonality—suitable for production AI features.

MediumTechnical

20 practiced

Given a pandas DataFrame events with schema: user_id (int), event_time (ISO8601 UTC string), event_type (string), value (float). Implement a Python function (using pandas) that produces per-user features as of a reference date: hour_of_day (cyclical sin/cos aggregated over last 7 days), day_of_week frequency, time_since_last_event (seconds), count_last_7_days, count_last_30_days. Ensure your function handles missing timestamps, timezones, and scales to tens of millions of rows (avoid Python-level loops).

Sample Answer

Approach: parse and normalize timestamps to UTC, filter events up to (but not after) the reference date, compute features using vectorized pandas operations (no Python-level per-user loops). For cyclical hour_of_day use sin/cos of hour; day_of_week frequency as normalized counts over last 7 days. Use groupby aggregations on filtered slices for windowed counts (7d, 30d) and last-event time.

python

import pandas as pd
import numpy as np

def build_user_features(events: pd.DataFrame, reference_date: str):
    """
    events: DataFrame with columns ['user_id','event_time','event_type','value']
    reference_date: ISO8601 string in UTC (e.g. "2025-01-01T00:00:00Z")
    Returns DataFrame indexed by user_id with features:
      - hour_sin, hour_cos (aggregated over last 7 days)
      - day_of_week_freq_0..6 (fractions over last 7 days)
      - time_since_last_event (seconds) or NaN if none
      - count_last_7_days, count_last_30_days
    """
    # Parse timestamps robustly, force UTC
    ts = pd.to_datetime(events['event_time'], utc=True, errors='coerce')
    df = events.loc[:, ['user_id', 'event_type', 'value']].copy()
    df['event_time'] = ts
    ref = pd.to_datetime(reference_date, utc=True)

    # Drop rows with invalid timestamps or timestamps after reference
    df = df.dropna(subset=['event_time'])
    df = df[df['event_time'] <= ref]

    if df.empty:
        # return empty features frame if no valid events
        return pd.DataFrame(columns=[
            'hour_sin','hour_cos',
            *[f'dow_freq_{i}' for i in range(7)],
            'time_since_last_event','count_last_7_days','count_last_30_days'
        ]).astype(float)

    # Precompute time-based columns
    df['hour'] = df['event_time'].dt.hour.astype(np.int8)
    df['dow'] = df['event_time'].dt.weekday.astype(np.int8)  # 0=Mon..6=Sun
    # cyclical encoding
    radians = 2 * np.pi * df['hour'] / 24.0
    df['hour_sin'] = np.sin(radians)
    df['hour_cos'] = np.cos(radians)

    # Window masks (vectorized filtering)
    last7_cut = ref - pd.Timedelta(days=7)
    last30_cut = ref - pd.Timedelta(days=30)

    df_last7 = df[df['event_time'] >= last7_cut]
    df_last30 = df[df['event_time'] >= last30_cut]

    # Aggregations for hour sin/cos over last 7 days
    agg_hour = df_last7.groupby('user_id')[['hour_sin','hour_cos']].mean().rename(
        columns={'hour_sin':'hour_sin','hour_cos':'hour_cos'}
    )

    # Day-of-week frequency over last 7 days: compute counts per (user_id, dow) then normalize
    if not df_last7.empty:
        dow_counts = (df_last7.groupby(['user_id','dow'])
                               .size()
                               .unstack(fill_value=0)
                               .reindex(columns=range(7), fill_value=0))
        dow_freq = dow_counts.div(dow_counts.sum(axis=1).replace(0, np.nan), axis=0)
        # rename columns
        dow_freq.columns = [f'dow_freq_{i}' for i in dow_freq.columns]
    else:
        dow_freq = pd.DataFrame(index=agg_hour.index, columns=[f'dow_freq_{i}' for i in range(7)])

    # time_since_last_event: for events <= ref take the max timestamp per user, subtract from ref
    last_ts = df.groupby('user_id')['event_time'].max()
    time_since = (ref - last_ts).dt.total_seconds().rename('time_since_last_event')

    # counts
    count7 = df_last7.groupby('user_id').size().rename('count_last_7_days')
    count30 = df_last30.groupby('user_id').size().rename('count_last_30_days')

    # Combine all features into single frame (outer join to include users with any activity)
    out = pd.concat([agg_hour, dow_freq, time_since, count7, count30], axis=1)
    # Ensure numeric dtypes
    out = out.sort_index()
    return out.reset_index().rename(columns={'index':'user_id'}).set_index('user_id')

# Example usage:
# feats = build_user_features(events_df, "2025-12-01T00:00:00Z")

Key points and reasoning:- Parsing to UTC (pd.to_datetime(..., utc=True)) handles timezone-aware and naive ISO strings consistently; errors='coerce' removes bad rows.- Use vectorized groupby and boolean masks to compute windowed features without per-user loops. Filtering slices (df_last7, df_last30) reduces work for aggregations.- Cyclical hour features use sin/cos so ML models learn circularity (midnight adjacent to 23:00).- Day-of-week frequencies produced as normalized counts per user over last 7 days.- time_since_last_event uses max timestamp <= reference_date for correctness.

Scalability:- For tens of millions of rows ensure enough memory; consider processing by user shards, using Dask or PySpark (same logic but distributed), or storing pre-partitioned parquet and computing per-partition aggregates then reducing. Groupby on a large single-machine frame may require >RAM; using categorical user_id or sorting by user_id helps if streaming/rolling approaches are used.

Complexity:- Time: O(N) to parse and filter, groupby cost depends on number of distinct users U (roughly O(N) hashing + O(U) aggregation).- Space: O(N) for input; intermediate filtered views reduce work but still memory-bound.

MediumTechnical

34 practiced

You have 2000 sparse TF-IDF features. Compare TruncatedSVD, feature hashing, and a simple autoencoder as dimensionality reduction approaches before a downstream dense model. Discuss reconstruction/retained variance, interpretability, training and inference cost, ability to handle streaming data, and how each integrates into a retrain-deploy pipeline.

Sample Answer

High-level summary: all three reduce 2000 sparse TF‑IDF features to a dense lower-dim representation for a downstream dense model, but they trade off reconstruction/variance, interpretability, compute, streaming capability, and pipeline complexity differently.

1) TruncatedSVD (approx. PCA for sparse data)- Reconstruction / retained variance: Optimizes for maximum explained variance in low-rank linear subspace; you can measure explained variance ratio and choose k to retain X% variance.- Interpretability: Components are linear combinations of original terms; you can inspect top loadings but components are dense and can be hard to label.- Training & inference cost: Training is O(n_features * k^2 + n_samples * k) with efficient randomized SVD implementations; inference is a cheap matrix multiply (sparse × dense).- Streaming: Not naturally streaming, but incremental / randomized SVD and frequent re-fit strategies exist; handling new vocabulary requires re-fit or projection with consistent vocabulary.- Pipeline integration: Deterministic; store component matrix and mean; retrain on schedule (e.g., weekly) and version artifacts; low infra complexity.

2) Feature hashing- Reconstruction / retained variance: Not reconstructive—it's lossy by design; collisions mix features but preserves inner products in expectation (Johnson-Lindenstrauss-like).- Interpretability: Poor: hashed dimensions map multiple terms; hard to trace back originals.- Training & inference cost: Extremely cheap—no training, only hashing function; inference is O(nnz) per sample and memory minimal.- Streaming: Excellent—naturally handles new terms and streaming data without retraining.- Pipeline integration: Simplest to deploy; deterministic hashing seed must be fixed and versioned; retraining downstream model as data drifts; no artifact to version beyond hash params.

3) Simple autoencoder (shallow neural)- Reconstruction / retained variance: Can learn nonlinear manifolds, potentially better reconstruction for same latent size if data has nonlinear structure; optimize reconstruction loss (MSE or KL using TF‑IDF).- Interpretability: Low; latent dims are abstract; techniques like feature attribution or probing can help but costly.- Training & inference cost: Higher training cost (epochs on GPU/CPU), inference cost moderate (dense forward pass). Requires hyperparameter tuning and regularization to avoid overfitting.- Streaming: Can be adapted with online learning or periodic fine-tuning; more delicate with catastrophic forgetting—need replay or incremental schemes.- Pipeline integration: More complex: model checkpoints, training pipeline, monitoring for drift, GPU resources. Version autoencoder + downstream model together; schema changes (vocab) require embedding layer changes.

Practical guidance for production:- If you want explainability and fast inference, start with TruncatedSVD; pick k by explained variance and monitor reconstruction on holdout.- If you need robustness to new vocabulary and minimal infra, use feature hashing (good baseline).- If empirical downstream performance is limited by linear projections, try a compact autoencoder (or supervised bottleneck) but factor in higher ops cost and greater deployment complexity.- In any pipeline: version transform artifacts, fix random seeds, include unit tests for transform parity, and automate retrain triggers (data drift, vocab growth, performance decay).

EasyTechnical

23 practiced

List and compare practical strategies to handle missing values for both numerical and categorical features in production ML: simple imputations (mean/median/mode), constant imputation with indicator flags, model-based imputation, forward/backward fill for time-series, treating missingness as a separate category, and approaches for MNAR. For each strategy describe pros/cons and suggested validation checks to ensure you aren't leaking label information.

Sample Answer

Brief overview: below are practical strategies for numerical and categorical features, with pros/cons and validation checks to avoid label leakage.

1) Mean/Median (numeric) & Mode (categorical)- Pros: Simple, fast, deterministic; works well when missingness is MCAR or few values.- Cons: Underestimates variance, biases distributions, can distort relationships.- Validation: Fit/impute using training data only (save statistics); compare distribution pre/post imputation; verify no uplift on validation that suggests leakage.

2) Constant imputation + indicator flag- Pros: Preserves missingness signal explicitly; easy to implement; indicator helps model learn missingness effect.- Cons: Choice of constant can create artificial outliers; interactions may be distorted.- Validation: Train with and without indicator, check importance of indicator; ensure indicator doesn't encode label (e.g., missingness correlated with target due to leakage).

3) Model-based imputation (KNN, iterative/ML imputer)- Pros: Uses other features to create more realistic values; reduces bias vs simple imputation.- Cons: Can leak label if target included in imputation; computationally heavier; overfitting risk.- Validation: Impute within cross-validation folds (no peeking); exclude target from imputation model; compare performance when imputing separately on train/test.

4) Forward/backward fill (time-series)- Pros: Preserves temporal continuity; realistic for regularly sampled sensors.- Cons: Propagates stale values across gaps; fails if non-stationary.- Validation: Use time-based splits; simulate gaps and measure forecasting error; ensure lookahead is not used.

5) Treat missing as separate category (categorical)- Pros: Captures informative missingness directly (MNAR-ish); simple.- Cons: Inflates cardinality; may mask underlying distribution.- Validation: Check target rates by category including missing; verify missing category isn’t just a proxy for label due to pipeline bugs.

6) Approaches for MNAR- Options: model missingness (missingness indicator as target), pattern-mixture models, multiple imputation with sensitivity analysis, causal methods (instrumental variables).- Pros: Explicitly acknowledges dependence between missingness and unobserved value.- Cons: Requires domain assumptions; more complex; may not fully identify effect.- Validation: Perform sensitivity analyses (vary imputation assumptions), compare models that include missingness mechanism vs ignore it; check stability of conclusions under alternative MNAR models.

General validation checks to prevent label leakage- Always fit imputers/statistics on training fold only; apply to test/production without refit.- Use time-based split for temporal data; avoid random splitting if missingness correlates with time.- Exclude target (and leaked future features) from imputation models.- Permutation/feature-shift tests: permute target or missingness flags—if model still gains, suspect leakage.- Backtest: simulate production stream and measure performance; monitor feature distributions and model reliance on missingness indicators.- Holdout sanity checks: compare model performance on rows with/without imputed values; large discrepancies warrant investigation.

Unlock Full Question Bank

Get access to hundreds of Feature Engineering and Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.