Model Selection and Hyperparameter Tuning Questions

Covers the end to end process of choosing, training, evaluating, and optimizing machine learning models. Topics include selecting appropriate algorithm families for the task such as classification versus regression and linear versus non linear models, establishing training pipelines, and preparing data splits for training validation and testing. Explain model evaluation strategies including cross validation, stratification, and nested cross validation for unbiased hyperparameter selection, and use appropriate performance metrics. Describe hyperparameter types and their effects such as learning rate, batch size, regularization strength, tree depth, and kernel parameters. Compare and apply tuning methods including grid search, random search, Bayesian optimization, successive halving and bandit based approaches, and evolutionary or gradient based techniques. Discuss practical trade offs such as computational cost, search space design, overfitting versus underfitting, reproducibility, early stopping, and when to prefer simple heuristics or automated search. Include integration with model pipelines, logging and experiment tracking, and how to document and justify model selection and tuned hyperparameters.

HardTechnical

89 practiced

A product manager asks you to deliver higher recall for a model even though it may reduce fairness and explainability. As head of ML engineering, how would you approach this request? Describe stakeholder communication, risk assessment, experiments you would run (including fairness-aware hyperparameter constraints), and how you would document the decision for compliance.

Sample Answer

Situation: A PM requested we push for higher recall in a production classifier (to reduce false negatives for user safety), knowing it could worsen fairness (disparate false positive rates across groups) and reduce explainability (more complex ensemble).

Task: As Head of ML Engineering, I had to evaluate whether to prioritize recall, quantify trade-offs, run controlled experiments, and make a documented, auditable decision that aligned with legal and ethical requirements.

Action:- Stakeholder alignment: I convened a brief cross-functional meeting (PM, Legal, Privacy, Trust & Safety, Product Design, and an external ethicist when needed). I clarified the product goal (why recall matters), acceptable cost thresholds (max allowable drop in fairness/explainability), regulatory constraints, and user safety implications.- Risk assessment: I led a risk workshop to map harms (who is harmed by false negatives vs. who is harmed by increased false positives), legal exposure, reputational risk, and operational costs (support load). We scored risks and set quantitative guardrails (e.g., max 5% increase in demographic parity difference; minimum SHAP feature-stability score).- Experimentation plan: - Baseline: current model metrics by subgroup (precision/recall/FPR/FNR), calibration, and explanation consistency. - Recall-focused variants: tune thresholds and train architectures with recall-weighted loss (e.g., focal loss or class-weighting) and ensembles to boost recall. - Fairness-aware constraints: run constrained optimization (e.g., add Lagrangian penalty/or post-processing like equalized odds adjustment) and hyperparameter searches where fairness metrics are bounded (reject configs where demographic parity diff > threshold or FNR gap > threshold). - Explainability controls: track explanation stability (SHAP/Integrated Gradients variance) and include simpler surrogate models for auditability. - A/B test in staging with held-out slices, measure downstream user impact, support/appeal rates, and false positive/negative costs.- Decision criteria: choose model maximizing recall subject to pre-agreed fairness/explainability constraints; if no model meets constraints, escalate to product/legal to accept residual risk or modify product requirements.- Communication: I provided clear trade-off dashboards, visualizations by subgroup, and a recommendation with quantified impacts and mitigation steps (monitoring, user appeal flow, human-in-loop flagging).- Documentation & compliance: I created an audit package stored in our governance repo containing: requirements doc, meeting minutes, risk assessment, experiment logs, hyperparameter ranges (including fairness bounds), code and seeds to reproduce training, evaluation notebooks with subgroup metrics, stress tests, rollout plan, monitoring alerts, rollback criteria, and sign-offs from PM, Legal, and Trust & Safety.

Result: This process produced a recall-improved model that stayed within agreed fairness/explainability guardrails for a safe pilot. The documented decision path enabled rapid approval from Compliance and a clear rollback plan, and the monitoring pipeline caught an unexpected subgroup drift early, allowing a targeted mitigation.

This approach balances product needs with ethical and legal responsibilities through transparent stakeholder engagement, constrained experimentation, and thorough documentation.

HardTechnical

87 practiced

How would you incorporate explicit fairness constraints into hyperparameter tuning (for example, minimize validation loss subject to demographic parity difference < epsilon)? Propose an optimization strategy (constrained optimization, penalty methods, or multi-objective search), tooling choices, and how to validate the final model's compliance.

Sample Answer

Clarify objective and constraint:- Functional: minimize validation loss (or maximize accuracy/utility).- Fairness constraint: demographic parity difference ≤ ε (DP = |P(Ŷ=1|A=0) − P(Ŷ=1|A=1)|).- Non-functional: latency, model size, training budget.

Optimization strategy (recommended hybrid):1. Constrained optimization as primary: treat fairness as a hard constraint using Lagrangian relaxation (constrained min L_val + λ·max(0, DP−ε)). Optimize λ adaptively (dual ascent). This yields models that respect ε with interpretable trade-offs.2. Use a two-stage search: inner loop trains model given hyperparams and current λ; outer loop tunes hyperparams (architecture, lr, regularization) using constrained-aware search (Bayesian optimization with constraint handling, e.g., Gaussian Process with feasibility model).3. Fall-back multi-objective search: if constraints are soft or nonconvex, run Pareto-front search (NSGA-II or multi-objective BO) to produce trade-off frontier.

Tooling:- Training: PyTorch/TensorFlow; fairness test hooks in training loop.- Constrained optimizer: implement Lagrangian with SGD for primal, Adam for λ; or use CVXPY for convex proxies.- Hyperparameter tuning: Optuna or Ax for constrained Bayes; Ray Tune for distributed runs; Nevergrad/DEAP for evolutionary Pareto search.- Fairness libs: Fairlearn (mitigation & metrics), AIF360 (metrics, pre/post-processing baselines).- Orchestration: Kubeflow or Ray for scale; Weights & Biases for logging metrics + constraint traces.

Validation & compliance:- Holdout test + separate fairness test set (stratified by sensitive attribute).- Report: primary metric, DP, statistical significance (bootstrap CIs), subgroup metrics, calibration by subgroup, confusion matrices.- Stress tests: simulate distribution shift, check fairness across intersectional groups, check robustness under sampling variation.- Automated alerts: block deployment if DP > ε or CI overlaps threshold.- Documentation & audit trail: dataset provenance, exact metric definitions, hyperparams, λ schedule, and random seeds.

Example quick workflow:- Define ε and DP estimator.- Run constrained hyperparam search (Optuna with penalty Lagrangian).- For top candidates, run large-seed retrain and bootstrap CI on DP.- Deploy only models with DP ≤ ε and acceptable primary metric loss.

Trade-offs:- Hard constraints may degrade accuracy; choose hard vs soft based on policy.- DP is one fairness notion—consider alternatives (equalized odds) depending on harm model.

MediumTechnical

117 practiced

Implement a random search tuner in Python that accepts an sklearn-style estimator, a parameter-space specification (supporting ranges for integers, floats with log-uniform sampling, and categorical choices), and n_iter trials. The function should run cross-validation for each sampled configuration and return the best configuration and score. Focus on correct sampling, seed handling, and clarity.

Sample Answer

Approach: sample n_iter configurations from the provided search space using a reproducible RNG, evaluate each with cross-validation (sklearn's cross_val_score), and track the best configuration and score. Support: int range (inclusive), float range with optional log scale, and categorical list.

python

import math
import numpy as np
from sklearn.model_selection import cross_val_score
from copy import deepcopy

def sample_param(spec, rng):
    t = spec.get("type")
    if t == "int":
        low, high = spec["low"], spec["high"]
        return int(rng.randint(low, high + 1))
    if t == "float":
        low, high = spec["low"], spec["high"]
        if spec.get("log", False):
            # sample uniformly in log-space
            return float(math.exp(rng.uniform(math.log(low), math.log(high))))
        return float(rng.uniform(low, high))
    if t == "categorical":
        choices = spec["choices"]
        return deepcopy(rng.choice(choices))
    raise ValueError(f"Unknown param type: {t}")

def random_search_tuner(estimator, param_space, X, y, n_iter=50, cv=3, scoring=None, seed=None):
    rng = np.random.RandomState(seed)
    best_score = -np.inf
    best_cfg = None
    for i in range(n_iter):
        cfg = {}
        for k, spec in param_space.items():
            cfg[k] = sample_param(spec, rng)
        # set params (works for sklearn estimators)
        model = deepcopy(estimator).set_params(**cfg)
        scores = cross_val_score(model, X, y, cv=cv, scoring=scoring)
        mean_score = float(np.mean(scores))
        if mean_score > best_score:
            best_score = mean_score
            best_cfg = cfg
    return best_cfg, best_score

Key points:- Uses numpy RandomState for reproducible sampling with seed.- Log-uniform sampling done by sampling uniformly in log-space then exponentiating.- Deepcopies estimator to avoid parameter bleed across trials.- Evaluates with cross_val_score; scoring and cv are passed through.Complexity: O(n_iter * cv * training_time). Edge cases: validate ranges (low>0 for log), categorical empty, k-fold > data size.

EasyTechnical

109 practiced

Explain k-fold cross-validation and stratified k-fold cross-validation. Describe advantages, limitations, and when to prefer stratification. Also summarize why k-fold CV is inappropriate for time-series forecasting and what alternatives should be used for temporal data.

HardTechnical

63 practiced

You need to fine-tune a large transformer on limited GPU resources. Explain how to set and tune hyperparameters such as base learning rate, layer-wise learning-rate decay, optimizer hyperparams (AdamW betas, epsilon), batch size with gradient accumulation, warmup steps, and mixed-precision settings. Provide practical initial ranges and monitoring signals to detect instability.

Sample Answer

Situation: You need to fine‑tune a transformer on limited GPU memory. Below is a practical, role‑appropriate recipe for setting and tuning the key hyperparameters, plus monitoring signals and mitigations for instability.

Base learning rate- Full fine‑tune: start 1e‑5 → 5e‑5 (often 2e‑5 or 3e‑5 is stable for BERT/GPT‑style models on moderate datasets).- Parameter‑efficient methods (LoRA/adapters/freeze most layers): start higher 1e‑4 → 5e‑4.- Tuning: if validation loss decreases too slowly, raise ×2; if loss spikes/NaNs, drop ×3–10.

Layer‑wise learning‑rate decay (LLRD)- Multiply factor per layer from output→input: 0.8–0.95.- Typical: head layers lr = base_lr, lower layers = base_lr × (decay)^(layer_index).- Use stronger decay (0.8–0.9) when freezing early layers is desirable or dataset small.

Optimizer hyperparams (AdamW)- Betas: common choices (0.9, 0.999). For very noisy gradients try (0.9, 0.98).- Epsilon: 1e‑8 (PyTorch default) to 1e‑6. If fp16 instability, increase to 1e‑6.- Weight decay: 0.01 for encoder weights; exclude biases and layernorm.

Batch size & gradient accumulation- Per‑GPU micro‑batch: 1–8 depending on memory. Use gradient accumulation to reach effective batch 16–512.- Rule: larger effective batch allows larger LR; but very large batches may hurt generalization—monitor val loss.- If memory tight: use gradient_checkpointing and mixed precision first before shrinking micro‑batch.

Warmup and schedulers- Warmup: linear warmup 0.03–0.1 of total steps (e.g., 500–2000 steps) or 1000 steps minimum for small data.- Scheduler: linear decay or cosine; prefer linear for stable fine‑tuning.

Mixed precision- Prefer bf16 on Ampere/BNL hardware and frameworks that support it (more stable).- If only fp16: use torch.cuda.amp with dynamic loss scaling, set initial_loss_scale default and enable dynamic.- Monitor for Inf/NaN gradients; if seen, increase eps, enable grad clipping, or use bf16.

Other stabilizers- Gradient clipping: 0.1–1.0 (often 1.0).- Use label smoothing or small weight decay to regularize.- Freeze embeddings / lower half of layers for very small data or tiny GPUs.- Use parameter‑efficient fine‑tuning (LoRA) to massively reduce memory and allow larger LR.

Practical tuning workflow1. Start with conservative config: base_lr=2e‑5, betas=(0.9,0.999), eps=1e‑8, wd=0.01, LLRD=0.9, micro_batch=4, accum_steps to reach effective 32, warmup=500, fp16 with dynamic loss scaling.2. Train 1–2 epochs; monitor signals below.3. If stable and underfitting: raise lr ×2 or reduce LLRD (closer to 1.0). If unstable: lower lr ×3, increase eps, or reduce warmup length? (actually increase warmup length).4. If OOM: reduce micro batch, enable gradient_checkpointing, switch to LoRA/adapters.

Monitoring signals for instability- Training loss spikes or sudden NaNs/Inf → too large LR / FP16 underflow/overflow / small eps → immediately reduce LR, increase eps, enable dynamic loss scale or switch to bf16.- Grad norm exploding → enable/adjust gradient clipping (start at 1.0), reduce LR.- Validation loss increases while training loss decreases → overfitting: increase weight decay, add dropout, reduce LR, or freeze lower layers.- Very noisy validation metric across batches → increase smoothing (more warmup), larger effective batch, or reduce LR.- Slow convergence → LR too small, or scheduler too aggressive; try 2× LR or longer training.

Final tips- Use a small pilot run (5–10% of steps) to validate stability before full runs.- Prefer bf16 if available; for fp16 use dynamic loss scaling.- Consider parameter‑efficient fine‑tuning (LoRA/adapters) to save memory and simplify hyperparameter search.

Unlock Full Question Bank

Get access to hundreds of Model Selection and Hyperparameter Tuning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.