Deep Learning Model Evaluation Questions
Covers evaluation strategies and debugging techniques specific to deep learning models during development and training. Topics include selection and interpretation of evaluation metrics for classification, regression, and generative models (for example Frechet Inception Distance, Inception Score, BLEU), validation strategies and cross validation where applicable, learning curves and validation curves to detect underfitting and overfitting, bias variance tradeoffs, regularization techniques such as dropout, weight decay, early stopping, and data augmentation, and approaches to debug model failures (gradient checking, loss surface inspection, vanishing or exploding gradients, numerical stability). Includes hyperparameter search diagnostics, model interpretability checks, unit tests for model components, sanity checks for data labels and preprocessing, and experiment tracking to reproduce and compare results.
MediumTechnical
57 practiced
Design a validation and cross-validation strategy for a deep learning image classification problem with only 2,000 labeled images. You can use transfer learning and data augmentation, but compute budget is limited to 10 full fine-tune runs. Explain your choice of folds, augmentation regime, and hyperparameter tuning approach.
Sample Answer
Situation: We have only 2,000 labeled images, limited compute (10 full fine-tune runs). Goal: get reliable estimate of generalization and select a robust transfer-learning configuration.Validation strategy (folding and holdout)- Hold out a small final test set first: reserve 10% (200 images) as a locked test set for the final evaluation only. This prevents repeated peeking.- Use stratified 5-fold CV on the remaining 1,800 images (≈360 images/fold). 5 folds gives a good bias–variance trade with small data; stratification preserves class balance.- With 10 full fine-tune runs budget: use a two-stage tuning protocol (see below) so only the top candidates consume full-fine-tune runs. Plan: 5 full fine-tunes for the 5-fold CV of the chosen final configuration (train on 4 folds, validate on 1 fold rotated) OR alternatively, average metrics from the 5 folds while performing full fine-tune on each fold (5 runs). Use remaining 5 runs for final model variants / ensemble / re-runs with best hyperparams trained on full 1,800 + optionally the 200 test held-out if releasing model.Augmentation regime- Train-time augmentations (on-the-fly, randomized): moderate but realistic augmentations to increase effective data without changing label semantics: - Geometric: random horizontal flip, small rotations (±15°), random resized crop (scale 0.8–1.0), slight translations - Photometric: color jitter (brightness/contrast/saturation small), Gaussian blur or sharpening rarely - Regularization: Cutout or random erasing with small masks (helps robustness) - Keep augmentation strengths conservative (no extreme transforms) so validation remains representative.- Validation/test-time: NO stochastic augmentation. Use deterministic preprocessing: resize + center crop + normalization. This ensures consistent, comparable metrics.- Use same augmentation pipeline for all candidate experiments to isolate hyperparameter effects.Hyperparameter tuning approach (compute-efficient)Stage 1 — cheap search (many candidates, low compute):- Linear probe / head-only training: freeze backbone and train only the classifier head (and maybe batch-norm) for a small number of epochs (5–10) over the 5-fold CV or a single validation split. This is fast and discriminative for hyperparameters like learning rate, weight decay for the head, and augmentation choices.- Use a small random/grid search over learning rates (e.g., 1e-4–1e-1 log scale), weight decay, and dropout. Evaluate on 5-fold CV or holdout validation; pick top 3 candidates.Stage 2 — focused full fine-tune (expensive, limited runs):- For the 3 best candidates from stage 1, run progressive unfreezing fine-tuning: - Option A: Full fine-tune per candidate across the 5 folds (would be 3×5 = 15 runs — too many). Instead: - Option B (recommended given 10-run cap): For each top candidate, perform a full fine-tune on 2 folds (2 runs each → total 6), plus one final full train on all 1,800 for the best candidate (1 run) and use remaining runs for a second-best ensemble or repeat to check stability. Or: run full fine-tune across all 5 folds but only for the single best candidate (5 runs), reserve remaining 4 runs for ablations or re-trains.- Use learning rate schedule (cosine or step) and discriminative LR for backbone vs head (lower LR for pretrained layers).- Apply early stopping on validation loss/patience to avoid wasting runs.- Use gradient accumulation if batch size limited by GPU memory.Other practical choices- Metric: use class-balanced metric (F1 or balanced accuracy) if classes imbalanced; track per-class performance.- Seed and checkpoint: save checkpoints and run each full fine-tune with 2–3 different seeds if budget allows to estimate variance (use one of the reserved runs).- Ensembling: if two or three top full-trained models are available, average logits for a small ensemble to improve robustness.- Avoid leakage: ensure no augmentations that mix labels, and that preprocessing is identical between validation folds.Why this design- Reserve test set prevents information leakage.- 5-fold gives stable estimates with small data.- Two-stage tuning (cheap linear probe then focused full fine-tune) makes efficient use of limited full-fine-tune budget while exploring many hyperparameter choices cheaply.- Conservative augmentation improves generalization without corrupting labels; deterministic validation ensures reliable comparisons.- Early stopping, discriminative LRs, and progressive unfreezing maximize gains from transfer learning with limited compute.Expected compute plan (example)- Stage 1: head-only runs across folds — cheap, many candidates.- Stage 2: 5 full fine-tunes for best configuration across 5 folds (5 runs).- Final: train on full 1,800 (+/−200 locked test) for deployment (1 run), and use remaining runs for stability checks or small ensemble (3 runs).This balances reliable validation, sensible augmentation, and an efficient hyperparameter search within a 10 full fine-tune-run budget.
EasyTechnical
51 practiced
List common data augmentation techniques for image and text tasks and explain how augmentation affects evaluation metrics. When and how would you apply test-time augmentation (TTA), and how should TTA results be reported in papers or production dashboards?
Sample Answer
Common augmentations- Image: random crop/resize, horizontal/vertical flip, rotation, color jitter, brightness/contrast, Gaussian noise, Cutout/Random Erasing, MixUp/CutMix, elastic transform, histogram equalization, SmartCrop, Test-time multi-scale / center-crop ensembles.- Text: synonym replacement, random deletion/swap/insertion, back-translation, paraphrasing (beam/seq2seq), token masking, span corruption, word dropout, character-level noise, entity replacement, contextual augmentation with language models.How augmentation affects evaluation metrics- During training: augmentations increase data diversity, reduce overfitting, improve generalization — typically raise validation/test accuracy and lower variance. They can change calibration (often better if augmentation matches real variation) and can bias metrics if augmentations alter label semantics (e.g., heavy crop removes class cues → label noise).- During evaluation: you must be consistent. If the eval set is augmented in a way that changes label distribution, reported metrics become incomparable. Always report whether metrics use clean data or augmented inputs.Test-time augmentation (TTA): when and how- When: useful when small but consistent gains in accuracy or robustness are needed (vision models, limited ensembles), or to smooth predictions under input variability. Avoid if latency/cost crucial.- How: generate K transformed versions of each test input (e.g., flips, crops, scales, light jitter, back-translations for text), run model on each, aggregate predictions via average of logits/probabilities (preferred) or majority vote. For calibration-sensitive tasks, average logits then temperature-scale. For detection/segmentation, aggregate boxes/masks with NMS or pixel-wise averaging and thresholding.Reporting TTA results (papers & dashboards)- Always report both: baseline (no TTA) and TTA numbers.- Specify: exact transforms, parameter ranges, K (number of variants), aggregation method (logit/probability average, majority vote), any post-processing (NMS, thresholding), and compute cost (inference-time multiplier, latency).- Report confidence intervals or standard deviation across runs; show per-class effects if relevant.- For production dashboards: display baseline vs TTA metrics, average latency, cost (CPU/GPU seconds), and fallback policy. If TTA used in A/B tests, show user-impact metrics and degradation risks.Practical tips- Prefer probability/logit averaging over hard-vote.- Limit TTA to transforms seen in training domain.- For text, ensure semantic label preserved (avoid paraphrases that change intent).- Balance accuracy gain vs latency/cost; consider caching or lightweight ensembling alternatives.
EasyTechnical
57 practiced
Describe the standard train/validation/test split strategy for deep learning model development. When would you prefer an explicit validation set over k-fold cross-validation? What are common sources of data leakage and how do you prevent them when splitting datasets (give concrete rules/examples)?
Sample Answer
Standard split: reserve three disjoint sets—training for learning model parameters, validation for tuning hyperparameters/early stopping and model selection, and test for final unbiased evaluation. Common ratios: 70/15/15 or 80/10/10; for large datasets you can use a smaller validation (e.g., 90/5/5). Ensure the test set is held out until final evaluation.When to prefer explicit validation over k-fold CV:- Deep learning with large datasets and long training times: an explicit validation set is faster and realistic for early stopping and frequent checkpointing.- When using large-scale hyperparameter search (e.g., many trials, Bayesian/ASHA), repeated k-fold would be computationally prohibitive.- Use k-fold CV when data is limited or when you need more reliable variance estimates for small models/datasets.Common sources of data leakage and prevention (concrete rules/examples):- Temporal leakage: future data in train. Rule: split by time (train on earlier timestamps, validate/test on later).- Patient/user-level leakage: same user appears in train and test. Rule: split by user_id/patient_id so all records for an entity stay in one set.- Feature leakage: features derived from target (e.g., rolling future mean). Rule: only compute features using past information available at prediction time.- Preprocessing leakage: fitting scalers/encoders on full data. Rule: fit preprocessors only on training set and apply to val/test.- Duplicate or near-duplicate samples across splits. Rule: deduplicate and group similar items (e.g., images from same scene) before splitting.Always document splitting rules and run sanity checks: label distribution, leakage tests (train predictions on val/test should not be unrealistically high), and reproducible seed for splits.
MediumTechnical
55 practiced
Implement a function in Python that computes Expected Calibration Error (ECE) and produces a reliability-diagram-style binned summary for a multiclass classifier given logits and true labels. Specify the binning approach (equal-width vs equal-mass) and how you aggregate per-class vs overall ECE.
Sample Answer
To compute multiclass ECE and produce a binned reliability summary, use predicted probabilities (softmax of logits), assign each prediction to a bin by confidence, then for each bin compute accuracy and average confidence. For multiclass you can compute: (A) top-label (confidence = max prob) ECE (common) or (B) classwise ECE (treat each class as a binary calibration target) and aggregate (average or weighted by class support). I prefer equal-mass binning (quantile) for stable estimates, but equal-width is simpler.Approach:1. Softmax logits -> probs and predicted class, and confidence = max prob.2. Create bins (equal-mass or equal-width).3. For each bin compute mean_confidence, mean_accuracy (1 if pred==true), bin_weight.4. ECE = sum_bin bin_weight * |mean_confidence - mean_accuracy|.Code (top-label ECE + binned summary, supports equal-mass/equal-width and optional per-class ECE):Key points:- Time O(N*C) for softmax; binning O(N log N) if quantiles; overall O(N*C).- Equal-mass reduces empty/low-count bins; equal-width is intuitive for plots.- For deployment, prefer top-label ECE to evaluate decision calibration; use per-class ECE when class-specific calibration matters. Edge cases: ties, single-class predictions, empty bins — handle by fallbacks and reporting counts.
python
import numpy as np
from scipy.special import softmax
def ece_multiclass(logits, labels, n_bins=10, binning='equal-mass', per_class=False):
"""
logits: (N, C) numpy
labels: (N,) integer class indices
binning: 'equal-mass' or 'equal-width'
per_class: if True compute classwise ECE dict (average of binary ECEs weighted by support)
Returns: ece (float), bins_summary (list of dicts), class_ece (dict or None)
"""
probs = softmax(logits, axis=1)
conf = probs.max(axis=1)
pred = probs.argmax(axis=1)
N = len(labels)
# Determine bin edges
if binning == 'equal-width':
edges = np.linspace(0.0, 1.0, n_bins+1)
else: # equal-mass quantile bins
quantiles = np.linspace(0, 1, n_bins+1)
edges = np.unique(np.quantile(conf, quantiles))
if len(edges)-1 < n_bins:
# fall back to equal-width if not enough unique edges
edges = np.linspace(0.0, 1.0, n_bins+1)
ece = 0.0
bins_summary = []
for i in range(len(edges)-1):
lo, hi = edges[i], edges[i+1]
# include rightmost edge
if i == len(edges)-2:
mask = (conf >= lo) & (conf <= hi)
else:
mask = (conf >= lo) & (conf < hi)
if not np.any(mask):
bins_summary.append({'count':0,'mean_conf':None,'acc':None,'weight':0.0})
continue
mean_conf = conf[mask].mean()
acc = (pred[mask] == labels[mask]).mean()
weight = mask.sum() / N
ece += weight * abs(mean_conf - acc)
bins_summary.append({'count':int(mask.sum()), 'mean_conf':float(mean_conf), 'acc':float(acc), 'weight':float(weight)})
class_ece = None
if per_class:
class_ece = {}
C = probs.shape[1]
for c in range(C):
# treat class c as positive: confidence = prob[:,c], label_bin = (labels==c)
conf_c = probs[:, c]
label_bin = (labels == c).astype(int)
# reuse equal-mass edges computed on conf_c if desired; here use equal-mass on conf_c
edges_c = np.unique(np.quantile(conf_c, np.linspace(0,1,n_bins+1)))
if len(edges_c)-1 < n_bins:
edges_c = np.linspace(0.0,1.0,n_bins+1)
ece_c = 0.0
for i in range(len(edges_c)-1):
lo, hi = edges_c[i], edges_c[i+1]
if i == len(edges_c)-2:
mask = (conf_c >= lo) & (conf_c <= hi)
else:
mask = (conf_c >= lo) & (conf_c < hi)
if not np.any(mask): continue
mean_conf = conf_c[mask].mean()
acc = label_bin[mask].mean()
ece_c += (mask.sum()/N) * abs(mean_conf - acc)
class_ece[c] = float(ece_c)
# optionally compute macro / weighted aggregate
class_counts = np.bincount(labels, minlength=C)
weighted = sum(class_ece[c]*class_counts[c]/N for c in range(C))
class_ece['macro'] = float(np.mean([class_ece[c] for c in range(C)]))
class_ece['weighted'] = float(weighted)
return float(ece), bins_summary, class_eceHardTechnical
54 practiced
Design a test suite to ensure numerically stable training when switching to mixed-precision (FP16) or quantized training. Include checks for NaNs/Infs, gradient underflow/overflow, dynamic loss-scaling, and differences in final model accuracy. Describe automated thresholds and remediation steps.
Sample Answer
Situation: We need an automated test suite that verifies numeric stability when switching training from FP32 to mixed-precision (FP16) or to quantized training. The suite should catch NaNs/Infs, gradient under/overflow, verify dynamic loss-scaling, and validate no unacceptable accuracy regression.Test plan (automated):1. Smoke tests (per-step checks)- Run 5–10 training steps on a small deterministic batch.- Assertions: - No NaN/Inf in parameters, activations, gradients. - Per-tensor max/min and L2-norms within expected ranges (see thresholds).- Thresholds (example): - Any NaN/Inf -> fail immediately. - Gradient L2-norm > 1e6 or < 1e-8 -> flag numeric instability.2. Dynamic loss-scaling tests (for mixed-precision)- Simulate scenarios where gradients become very small or large by using scaled synthetic losses.- Verify: - Loss scaler increases when no overflow for N consecutive steps (e.g., 200 steps). - Loss scaler decreases on overflow detection. - Final scaler stays within [2^−8, 2^8] relative to initial.- Remediation: if scaler never increases/decreases as expected, assert misconfiguration of scaler (disable static scaling, use auto-scalers like NVIDIA/Apex/torch.cuda.amp).3. Long-run convergence and accuracy parity (integration)- Train full model for a small but meaningful number of epochs on representative dataset.- Compare metrics vs FP32 baseline: - Final validation accuracy / loss difference within relative tolerance (e.g., ≤0.5% absolute accuracy drop or ≤1% relative loss change). - If outside tolerance, run 3 seeds to rule out noise; if consistent regression, flag.- Remediation steps: enable higher precision for critical layers (master weights in FP32), adjust optimizer hyperparameters (LR scaling, weight decay), enable gradient clipping, tune loss-scaler, or fallback to mixed-precision with selective FP32 casts.4. Gradient distribution and underflow/overflow detection- Instrument per-layer gradient histograms and count of tiny (<1e-7) and huge (>1e5) values.- Automated thresholds: - >1% gradients < underflow threshold -> suspect underflow; recommend enabling gradient accumulation or higher loss-scaler. - >0.1% gradients > overflow threshold -> suspect instability; recommend gradient clipping or reducing LR.5. Quantization-specific tests- Check dequantize/quantize round-trip error on representative activations (max absolute error < 1e-2 for 8-bit per-layer or configurable).- Retrain or fine-tune with fake-quant aware training; compare final accuracy within tolerance above.- Remediation: use per-channel quantization, higher bitwidth for sensitive layers, fine-tuning longer, bias correction.6. CI and monitoring- Integrate tests into CI; fast smoke must run on every PR, longer convergence tests nightly.- On failure: capture model state, gradients, scaler history, and replayable seed + config.- Alerting: automatic ticket with attached artifacts and suggested remediation checklist.Implementation notes- Use deterministic seeds, small synthetic datasets for unit tests and real validation set for integration tests.- Log scaler timeline, per-step gradient norms, overflow flags to time-series store for trend analysis.- Provide automated utility that applies remediation steps in order (e.g., enable FP32 master weights → enable grad clipping → reduce LR by factor 2 → repeat test).This suite balances quick safety checks (NaNs/Infs, overflows), dynamic loss-scaler behavior, and end-to-end accuracy parity with clear automated thresholds and stepwise remediation to diagnose and fix numeric issues when moving to FP16 or quantized training.
Unlock Full Question Bank
Get access to hundreds of Deep Learning Model Evaluation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.