Decision Trees and Ensemble Methods Questions

How decision trees recursively split data. Hyperparameters: max depth, min samples split, criterion. Ensemble methods: random forests, gradient boosting. Understanding why ensembles work (combining weak learners). Trade-offs: complexity, interpretability, bias-variance. When to use trees vs. linear models.

MediumTechnical

101 practiced

Handling high-cardinality categorical features: Discuss techniques for encoding high-cardinality categorical variables for tree ensembles and linear models (target encoding, frequency encoding, hashing, one-hot), explain leakage risks, and recommend best practices for training and serving in production.

Sample Answer

Definition and goal:High-cardinality categorical features (thousands–millions of levels) need encodings that preserve signal without exploding dimensionality or leaking target information.

Common techniques and when to use them:- One-hot: precise and interpretable but impractical for very high cardinality (sparse huge vectors). Good for tree models with small cardinality only.- Frequency / count encoding: replace category with its frequency or log-frequency. Simple, fast, no leakage, works well for both trees and linear models as a proxy for rarity.- Target (mean) encoding: replace with target mean per category (optionally smoothed). Very powerful for both trees and linear models because it captures category-target relationship.- Hashing (feature hashing): maps categories into fixed-size buckets; controls dimensionality, fast, and stateless. Collisions add noise but often acceptable for large cardinality.

Leakage risks and mitigation:- Target leakage: naive target encoding computed on full training set leaks label information => overoptimistic metrics and poor generalization.- Mitigations: use k-fold out-of-fold encoding (compute encodings using only training folds), leave-one-out (with regularization), Bayesian smoothing (blend category mean with global mean based on count), add noise/stratified folds, and compute encodings using only historical data for time-series.

Training and serving best practices:- Compute encodings deterministically and persist mapping tables or hashing seed to ensure identical behavior in training and production.- For target encoding store counts and smoothed means; at serving, apply global prior for unseen or rare categories.- Use cross-validation-aware encodings to get reliable CV estimates and to choose smoothing hyperparameters.- Monitor distribution drift of categories and handle unseen/new categories with reserved bucket or global/default value.- Prefer frequency/hashing for high throughput, target encoding when accuracy gain justifies complexity, and trees with embeddings/learned encodings for deep models.- Log and version encoding logic and artifacts alongside model for reproducibility and audits.

Summary: choose encoding by model, cardinality, and latency constraints; avoid leakage via out-of-fold or temporal schemes; persist deterministic mappings and handle unseen categories in production.

MediumTechnical

73 practiced

Coding: Implement two Python functions: gini_impurity(y) and information_gain(y_parent, y_left, y_right). Arrays y contain class labels (integers). Ensure functions are correct for binary and multi-class labels and run in O(n) time relative to input sizes.

Sample Answer

To compute Gini impurity and information gain in O(n) time, count class frequencies once per input and use the formula Gini = 1 - sum(p_c^2). Information gain = Gini(parent) - (n_left/n_parent)*Gini(left) - (n_right/n_parent)*Gini(right).

python

from collections import Counter

def gini_impurity(y):
    """
    y: iterable of integer class labels
    Returns Gini impurity (float)
    O(len(y)) time, O(k) space where k = number of classes
    """
    n = len(y)
    if n == 0:
        return 0.0
    freqs = Counter(y)
    # sum of squared class probabilities
    sum_p2 = sum((count / n) ** 2 for count in freqs.values())
    return 1.0 - sum_p2

def information_gain(y_parent, y_left, y_right):
    """
    y_parent, y_left, y_right: iterables of integer class labels
    Returns information gain (float)
    O(len(y_parent) + len(y_left) + len(y_right)) time
    """
    n_parent = len(y_parent)
    if n_parent == 0:
        return 0.0
    g_parent = gini_impurity(y_parent)
    g_left = gini_impurity(y_left)
    g_right = gini_impurity(y_right)
    n_left = len(y_left)
    n_right = len(y_right)
    # Weighted impurity after split
    weighted = (n_left / n_parent) * g_left + (n_right / n_parent) * g_right
    return g_parent - weighted

Key points:- Works for binary and multi-class labels.- Handles empty arrays: returns 0 impurity and 0 gain when parent empty.- Time complexity linear in input sizes; space proportional to number of distinct classes.- Numeric stability: probabilities computed as floats; for very large counts consider using integers then normalize.

MediumTechnical

83 practiced

Interpretability: Explain Partial Dependence Plots (PDP) and Individual Conditional Expectation (ICE) plots for interpreting feature effects in tree ensembles. Describe how correlated features affect PDPs and propose robust alternatives for interpreting effects under correlation.

Sample Answer

Partial Dependence Plots (PDP) and Individual Conditional Expectation (ICE) plots visualize how a feature x_j affects model predictions.

- PDP: plots E[f(x) | x_j = v] across v — practically, for each v you replace x_j in the dataset with v, predict, and average. PDP shows the average marginal effect; useful for global insights in tree ensembles.- ICE: plots f(x^{(i)} with x_j=v) for each instance i (no averaging). ICE reveals heterogeneity and interactions that PDP can hide (crossing lines indicate interactions).

How correlated features break PDPs:- PDP assumes independence between x_j and other features. If x_j is correlated with x_k, replacing x_j with out-of-distribution values creates unrealistic samples, producing biased/unreliable estimates (spurious effects or attenuation).- Averaging in PDP can mask opposing instance-level effects when correlations induce interaction patterns.

Robust alternatives / mitigations:- Centered ICE (c-ICE): subtract each ICE curve’s value at a reference v to focus on shape differences — helps detect heterogeneity but still uses OOD points.- Conditional PDP: compute E[f(x) | x_j=v] integrating over the conditional distribution p(x_{-j}|x_j=v) (use KDE, conditional density estimation, or nearest-neighbors) so replaced values are realistic.- Accumulated Local Effects (ALE): estimates local gradients of f with respect to x_j using data intervals; ALE is unbiased under correlated features and fast for tree ensembles.- SHAP dependence plots: plot SHAP values vs x_j (optionally color by a second feature) to show marginal contributions while accounting for feature interactions and coalitions.- Causal/structural approaches: when causality matters, use causal inference to isolate effect.

Recommendation for ML engineers: start with ICE to check heterogeneity, avoid vanilla PDP when features are correlated, prefer ALE or conditional PDP (or SHAP dependence) for robust interpretation, and always validate explanations by checking data support for perturbed points.

EasyTechnical

123 practiced

Compare Gini impurity and Shannon entropy as split criteria for classification trees. Explain their definitions, how they influence split choices in practice, computational differences, and whether one is generally preferred over the other in production.

MediumTechnical

75 practiced

Describe how Out-Of-Bag (OOB) error is computed in Random Forests. What are the advantages and limitations of using OOB error instead of k-fold cross-validation for hyperparameter tuning and model evaluation in production?

Unlock Full Question Bank

Get access to hundreds of Decision Trees and Ensemble Methods interview questions and detailed answers.

Join thousands of developers preparing for their dream job.