Deep Learning Model Evaluation Questions

Covers evaluation strategies and debugging techniques specific to deep learning models during development and training. Topics include selection and interpretation of evaluation metrics for classification, regression, and generative models (for example Frechet Inception Distance, Inception Score, BLEU), validation strategies and cross validation where applicable, learning curves and validation curves to detect underfitting and overfitting, bias variance tradeoffs, regularization techniques such as dropout, weight decay, early stopping, and data augmentation, and approaches to debug model failures (gradient checking, loss surface inspection, vanishing or exploding gradients, numerical stability). Includes hyperparameter search diagnostics, model interpretability checks, unit tests for model components, sanity checks for data labels and preprocessing, and experiment tracking to reproduce and compare results.

HardTechnical

53 practiced

You're training a large transformer with mixed-precision (FP16) and experience sudden NaNs and instability. Explain numerical issues with FP16, how automatic mixed precision (AMP) frameworks manage loss scaling, and list concrete debugging and mitigation steps: dynamic/static loss scaling, checking for inf/nan in activations/gradients, enabling skip_nan_gradients, verifying custom ops for AMP safety, and adjusting optimizer parameters.

Sample Answer

FP16 numerical issues — short summary:- FP16 has ~11 bits exponent range ±65,504 and 10 bits mantissa → much smaller dynamic range and precision than FP32. Underflows (gradients/activations become zero), overflows (large accumulations -> Inf/NaN), and rounding errors are common. Small gradients can be flushed to zero; sums/variance estimates lose precision. These cause training instability and sudden NaNs.

How AMP (automatic mixed precision) manages loss scaling:- AMP runs forward in FP16 where safe (matrix multiplies, convs) and keeps some ops/weights in FP32 (master weights, BatchNorm). Because FP16 can underflow small gradients, AMP multiplies the scalar loss by a large scale factor before backprop (loss scaling) so gradients are larger and don’t underflow to zero.- After backprop, gradients are unscaled (divided by the same factor) before updating FP32 master weights.- Two loss-scaling strategies: - Static loss scaling: fixed scale chosen by user. Simple but brittle — too large -> overflow/NaN; too small -> underflow. - Dynamic loss scaling: start with a large scale, reduce it when inf/NaN detected in gradients, increase it slowly if no overflows for N steps. This is the default in modern AMP implementations (e.g., torch.cuda.amp.GradScaler).

Concrete debugging & mitigation checklist (step-by-step):

1) Reproduce and isolate- Run a few steps with amp but in a controlled setting: smaller batch, single GPU, deterministic seeds.- Enable gradient/activation checks every step.

2) Detect where NaN/Inf appears- Add runtime checks after forward and after backward:

python

def has_inf_nan(tensor):
    return torch.isinf(tensor).any() or torch.isnan(tensor).any()

# check activations (example)
for name, act in model.named_buffers(): pass
# simpler: check gradients after backward
for p in model.parameters():
    if p.grad is not None and has_inf_nan(p.grad):
        print("NaN in grad:", p.name)

- Log layer outputs and gradients to find first layer producing Inf/NaN.

3) Loss-scaling actions- Use dynamic loss scaling (recommended). In PyTorch: GradScaler() handles this; in TF-MixedPrecision use LossScaleOptimizer with "dynamic".- If using static scaling (e.g., for reproducibility), pick conservative smaller scale (e.g., 128, 256) and test. If you see frequent overflow, reduce the scale.

4) If NaNs persist, enable framework option to skip NaN gradients- Some frameworks offer skip_nan_gradients / skip_if_overflow behavior that prevents optimizer.step when grads contain NaN/Inf and optionally zeros them. This avoids corrupting master weights. Use it as a temporary mitigation while you find root cause. Note: skipping updates can stall training for a while; monitor scale reductions and frequency.

5) Verify custom ops and third-party layers- Custom CUDA kernels or ops often aren’t AMP-safe (might assume FP32 or use inadequate accumulation). Ensure: - Kernels support FP16 inputs and use FP32 accumulation where needed (e.g., in reduction sums). - Autograd implementations correctly propagate gradients and preserve dtype semantics. - Add dtype casts explicitly around unsafe ops: cast inputs to FP32, perform op, cast back to FP16 or keep FP32 outputs.- Run with AMP’s “debug/verbose” flags if available to see which ops are auto-promoted or excluded.

6) Optimizer and hyperparameter adjustments- Use FP32 master weights (standard AMP practice). Ensure optimizer updates happen on FP32 params.- Reduce learning rate and/or grad clipping threshold — large updates are more likely to produce overflow in FP16 pathways.- Try switching optimizer (e.g., AdamW vs SGD) or reduce Adam’s betas / eps to more stable defaults. For Adam, increasing eps (e.g., 1e-6 -> 1e-4) can stabilize denomations.- Enable gradient clipping (clip by norm) before optimizer.step.- Use larger batch normalization eps or convert BatchNorm to FP32 (many frameworks already do this automatically).

7) Numerical robustness practices- Keep certain ops in FP32 (LayerNorm/BatchNorm/softmax/logsumexp/normalization and loss computations).- Use mixed-precision safe implementations (cuBLAS, cuDNN, fused kernels from vendor libraries).- Accumulate gradients (gradient accumulation) with smaller micro-batches if batch size causes instability.

8) Logging & metrics- Track frequency of scale reductions (if using dynamic scaling), number of skipped steps, and loss growth patterns. If scale quickly collapses to 1 and overflows persist, likely a bug in model or custom op.

Example troubleshooting sequence (practical):1. Switch to AMP dynamic scaling (GradScaler). Run with smaller LR and batch size.2. Add checks to identify first NaN-producing layer.3. If NaN found in custom op, cast to FP32 inside op or rewrite op accumulation in FP32.4. If NaNs are sporadic, enable skip_nan_gradients to avoid corrupt updates while continuing training and monitoring scaler behavior.5. Adjust optimizer eps / weight decay / gradient clipping as additional mitigation.

Why these help — reasoning:- Loss scaling addresses underflow by making gradients representable in FP16; dynamic scaling prevents overflow by adapting scale when overflows occur.- Checking activations/gradients pins down the faulty location — often a single op causes blowup.- Converting numerically sensitive ops to FP32 or using FP32 accumulation preserves precision and avoids catastrophic cancellation or overflow.- Optimizer parameter tweaks reduce update magnitude and denominator instability, reducing propensity for Inf/NaN.

Edge cases & final notes:- Intermittent NaNs can come from data (labels, targets, invalid values) — validate dataset for extremes.- Mixed precision uncovers bugs that FP32 masks; treat NaNs as signals to inspect model math and custom kernels.- Always keep master FP32 copy of weights and prefer dynamic loss scaling as the first line of defense.

EasyTechnical

45 practiced

Describe Inception Score (IS) and Frechet Inception Distance (FID) used to evaluate image generative models. Explain how they are computed (feature extraction via Inception network), what each measures (quality vs distributional similarity), and list key limitations such as sensitivity to dataset biases and inability to capture human-perceived fidelity in all cases.

EasyTechnical

42 practiced

Explain what learning curves (training and validation loss/metric vs epochs or vs. training set size) reveal about model fit. Describe typical curve shapes for underfitting, good fit, and overfitting. Explain concrete actions you would take in each case (e.g., gather more data, increase capacity, regularize) and why.

MediumTechnical

47 practiced

Explain methods to estimate predictive uncertainty for deep models: Monte Carlo Dropout, deep ensembles, Bayesian neural networks, heteroscedastic outputs, and approaches to produce predictive intervals for regression. For each method discuss computational cost, calibration quality, and practical evaluation metrics (NLL, Brier score, calibration plots).

Sample Answer

Brief overview: predictive uncertainty splits into aleatoric (data noise) and epistemic (model uncertainty). Good practice: evaluate with negative log-likelihood (NLL), Brier score (for classification/probabilistic events), calibration plots/reliability diagrams, expected calibration error (ECE), and for regression use prediction intervals coverage/probability (PICP) and interval width (MPIW).

1) Monte Carlo Dropout- What: Use dropout at test time and draw T stochastic forward passes; predictive distribution = empirical mean/variance.- Computational cost: T× inference cost (light extra memory); easy to implement in existing models.- Calibration quality: Improves epistemic uncertainty moderately; can under/over-estimate unless dropout rate tuned.- Metrics: NLL decreases with T often; use calibration plot and ECE; for regression compute PICP and MPIW.

2) Deep ensembles- What: Train M independent networks (different seeds, data shuffles); aggregate predictive mean/variance.- Computational cost: M× training and inference (can be parallelized); highest resource cost.- Calibration quality: State-of-the-art in practice — captures multimodal/posterior diversity and gives strong calibration.- Metrics: Typically best NLL/Brier; use reliability diagrams and PICP/MPIW for regression.

3) Bayesian neural networks (BNNs)- What: Place priors on weights and infer posterior (VI, MCMC, or Laplace).- Computational cost: VI/MCMC can be expensive; Laplace or mean-field VI is cheaper but approximative.- Calibration quality: Principled epistemic estimates; quality depends on inference fidelity—mean-field VI often underestimates uncertainty.- Metrics: NLL/Brier; posterior predictive checks; calibration plots.

4) Heteroscedastic outputs (aleatoric)- What: Model output both mean and input-dependent variance (e.g., network predicts μ(x), σ²(x) with NLL loss).- Computational cost: Minimal (single model); small extra head for variance.- Calibration quality: Good for modeling data noise but cannot capture epistemic uncertainty; combining with ensembles/MC-dropout improves results.- Metrics: NLL directly measures combined uncertainty; for regression check PICP/MPIW.

5) Predictive intervals for regression (practical approaches)- Direct parametric: Predict μ and σ and assume Gaussian → compute intervals μ ± z*σ.- Quantile regression: Train to predict α and 1−α quantiles directly (no distributional assumption) → calibrated empirical coverage.- Bootstrap/ensembles: Use ensembles or bootstrapped models to derive empirical quantiles of predictions.- Conformal prediction: Model-agnostic method that gives finite-sample marginal coverage using residuals from calibration set.- Trade-offs: Parametric is efficient but miscalibrated if assumption fails; quantile and conformal are robust; ensembles give epistemic+aleatoric but costlier.

Practical evaluation: report NLL and calibration plots, ECE/Brier (classification), and for regression report PICP, MPIW, and sharpness. Also run out-of-distribution tests to assess epistemic behavior.

MediumTechnical

44 practiced

Your training logs show training loss steadily decreasing while validation loss increases after a few epochs (classic overfitting). Provide a prioritized diagnostic checklist and the concrete interventions you would try in order (for example: data augmentation, weight decay, dropout, reduce model capacity, early stopping) and how you would measure the effect of each intervention to be confident it helped.

Sample Answer

1) Quick sanity checks (highest priority)- Verify no data leakage: ensure train/val split by time/entity if needed, labels not copied, preprocessing identical. Measure: reproducing leak removal should immediately raise val loss or change gap.- Confirm metric/logging correctness (loss vs. metric), consistent batchnorm/dropout train/eval modes. Measure: toggling eval mode fixes discrepancy.

2) Training hyperparams- Check learning rate / optimizer: too high LR can overfit noise or cause weird curves. Intervene: reduce LR or use LR schedules. Measure: smoother validation curve, lower final val loss; compare with same training length.

3) Data & splits- Increase validation robustness: use k-fold or larger holdout to ensure signal is real.- Add data augmentation (task-appropriate). Intervene: augmentations for images/text. Measure: reduced val loss, improved val metric; run A/B with/without augment.

4) Regularization techniques (apply one at a time)- Weight decay (L2): tune via small grid. Measure: decreased generalization gap, better val metric.- Dropout: add or increase rate modestly. Measure: slower train loss drop with improved val.- BatchNorm/LayerNorm tweaks if applicable.

5) Model capacity- Reduce capacity: fewer layers/units or prune. Intervene: train smaller model. Measure: reduced overfitting (smaller train-val gap) while maintaining acceptable val metric.

6) Optimization of training process- Early stopping based on validation metric with patience. Measure: earlier checkpoint yields best val score.- Increase batch size or use gradient noise to regularize; measure effect.

7) Data-centric fixes- Get more labeled data or use semi-supervised/self-supervised pretraining. Measure: val curve shifts down, gap narrows.

8) Advanced: ensembling, mixup, label smoothing- Try mixup/label smoothing if appropriate. Measure: improved calibration and val loss.

Experiment methodology (must do):- Change one factor at a time; keep random seeds; run 3 seeds for significance.- Track: train loss, val loss, primary metric, calibration, and generalization gap across epochs.- Use learning curves and delta between final train/val metrics; prefer intervention that lowers val loss/raises val metric and narrows gap without collapsing train performance.- Final validation on untouched test set or cross-val to confirm real improvement.

Unlock Full Question Bank

Get access to hundreds of Deep Learning Model Evaluation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.