MediumTechnical

83 practiced

Explain the difference between calibration and discrimination. Describe how to evaluate calibration using reliability diagrams, Expected Calibration Error (ECE), Brier score and log-loss. When would you prefer temperature scaling over isotonic regression for post-hoc calibration in production, and what are the limitations of each?

Sample Answer

Calibration vs discrimination:- Calibration measures whether predicted probabilities match observed frequencies (e.g., among items predicted 0.7, ~70% are positive). It answers "are probabilities trustworthy?"- Discrimination measures ranking ability (e.g., AUC, accuracy): can the model separate positives from negatives? A model can be well-discriminating but poorly calibrated (confident but biased probabilities), or well-calibrated but low discrimination.

Evaluating calibration:- Reliability diagram: bin predicted probabilities (e.g., 10 bins). Plot average predicted probability vs empirical frequency per bin. A perfectly calibrated model lies on the diagonal; deviations show under- or over-confidence and where it occurs.- Expected Calibration Error (ECE): weighted average absolute difference between bin accuracy and bin confidence. ECE = sum_k (|acc_k - conf_k| * n_k / N). Simple and intuitive but depends on binning and ignores intra-bin variance.- Brier score: mean squared error between predicted probability and true outcome (for binary: (p - y)^2). Combines calibration and refinement; lower is better. It decomposes into reliability, resolution, and uncertainty.- Log-loss (cross-entropy): -1/N sum [y log p + (1-y) log(1-p)]. Penalizes confident wrong predictions heavily; sensitive to extreme probabilities.

When to prefer temperature scaling vs isotonic regression in production:- Temperature scaling (single scalar on logits before softmax) is parametric, low-risk, preserves class ordering (discrimination unchanged), fast to fit, and robust with small calibration sets. Use it when you want a simple, stable post-hoc fix for over/under-confidence in modern neural nets and need reproducible, low-latency production behavior.- Isotonic regression is nonparametric, can model arbitrary monotonic maps, potentially better if calibration error is complex and training data for calibration is large. It can distort class ordering for multiclass (needs one-vs-rest) and may overfit with small calibration sets.

Limitations:- Temperature scaling: can only rescale confidence (global slope) — cannot correct class- or region-specific miscalibration or non-monotonic errors.- Isotonic regression: prone to overfitting on small calibration sets, can produce piecewise-constant mappings, may break ranking consistency across classes, and adds model complexity and storage.- ECE & reliability diagrams: depend on binning (can hide errors), ECE not a proper scoring rule. Brier and log-loss are proper scoring rules but mix calibration with discrimination; interpreting decomposition helps.Practical tip: validate calibration on held-out, stratified calibration sets; prefer temperature scaling for production simplicity and isotonic only when ample calibration data and need for flexible correction.

MediumTechnical

98 practiced

When your target metric is non-differentiable (for example F1 score or top-k accuracy), explain practical strategies to train models: use surrogate losses, structured prediction with task-specific losses, reinforcement learning approaches (policy gradients), or threshold optimization after training. For each strategy, list pros, cons, and production complexity.

Sample Answer

When the target metric is non-differentiable (F1, top-k accuracy, etc.), you can choose among several practical strategies. Below I summarize each approach with pros, cons, and production complexity.

1) Surrogate losses (e.g., cross-entropy, hinge)- Pros: Simple, stable training; well-optimized in libraries; often produces good calibration and ranking.- Cons: Indirect — optimizing surrogate may not align perfectly with final metric; may need class weighting for imbalance.- Production complexity: Low. Standard training pipelines and hardware; only requires post-training evaluation/thresholding.

2) Threshold optimization / calibration after training- Pros: Easy to implement; directly tunes decision boundary to maximize metric on validation set (e.g., F1, precision@k); preserves fast inference.- Cons: Requires representative validation data; may overfit threshold to val set; per-class thresholds add complexity.- Production complexity: Low–Medium. Store thresholds, add small wrapper in serving; re-calibrate with drift detection.

3) Structured prediction with task-specific losses (e.g., CRFs, structured SVM, direct F-measure approximations)- Pros: Models the actual decision structure and dependencies; can improve end-task performance when structure matters.- Cons: More complex training (inference in loop), slower; harder to scale to large label spaces.- Production complexity: Medium–High. Requires inference components in serving, possibly more compute and careful engineering.

4) Reinforcement learning / policy gradients (directly optimize metric)- Pros: Directly optimizes non-differentiable metrics; flexible for sequential/interactive decisions.- Cons: High variance, unstable training, needs reward shaping and large samples; slower convergence.- Production complexity: High. More complex training infrastructure, reproducibility/monitoring challenges; may need online evaluation/serving for policy.

Guidance: start with surrogate + thresholding (low-risk). If gap to target metric persists and validation is reliable, try structured losses; reserve RL/policy methods for sequential or interactive tasks where other methods fail. Always track metric drift and include calibration/validation pipelines in production.

HardTechnical

72 practiced

Design experiments to measure the effect of label smoothing on model calibration and adversarial robustness. Specify datasets, baseline models, metrics to report (ECE, Brier score, adversarial accuracy under PGD/CW), hyperparameters to sweep, and statistical tests to validate whether smoothing helps or harms calibration and robustness.

Sample Answer

Goal: quantify how label smoothing (LS) impacts probabilistic calibration and adversarial robustness across datasets and architectures, with rigorous statistics.

Datasets & splits:- CIFAR-10, CIFAR-100 (standard train/val/test); ImageNet-100 (subset of ImageNet for compute-limited experiments); MNIST for simple sanity checks.- Use standard augmentations (random crop/flip) and a held-out calibration set (10% of train) for temperature scaling.

Baseline models:- ResNet-50 (ImageNet-100), WideResNet-28-10 (CIFAR), a small ConvNet (MNIST).- For robustness experiments also include adversarially trained WideResNet variant.

Training protocol:- SGD w/ momentum 0.9, weight decay 5e-4, 200 epochs (CIFAR), cosine LR or step LR (specify), batch size 128. Run each config with 5 random seeds.

Label smoothing sweep:- LS values: 0 (baseline), 0.05, 0.1, 0.2, 0.3.- Also sweep: learning rate {0.01,0.1}, weight decay {1e-4,5e-4}, and mixup off/on (control).

Calibration & robustness evaluations:- Calibration: Expected Calibration Error (ECE) with 15 bins (report classwise and top-1), Brier Score, Negative Log-Likelihood (NLL). Also report post-hoc temperature scaling improvement.- Robustness: adversarial accuracy under PGD (l_infty eps={2/255,8/255}, steps=40, step=eps/4) and CW L2 (confidence=0, lr tune) attacks. Report clean accuracy too.- Report reliability diagrams and confidences on correct vs incorrect.

Statistical validation:- For each metric compute mean±std over seeds. Use paired tests comparing LS vs no-LS across seeds: paired t-test if normal (Shapiro-Wilk check), otherwise Wilcoxon signed-rank. Complement with bootstrap CIs (10k resamples) for mean differences and Cohen’s d effect size.- For multiple comparisons (multiple LS values, datasets) control FDR via Benjamini-Hochberg.- Declare practical significance thresholds (e.g., ECE delta > 0.5% and Cohen’s d > 0.2).

Additional analyses:- Evaluate calibration before/after temperature scaling to see if LS gives complementary benefit.- Check confidence distributions and severity: does LS reduce overconfidence on misclassified examples?- Ablation: repeat with adversarial training to see interaction effects.

Reporting:- Tables per dataset: clean acc, ECE, Brier, NLL, PGD@eps acc, CW acc, with p-values and bootstrap CIs.- Figures: reliability diagrams, adversarial accuracy vs eps curves, boxplots across seeds.

This design isolates LS effects, controls training variability, and uses robust statistical tests to conclude whether smoothing helps or harms calibration and robustness.

HardTechnical

103 practiced

Derive the gradient of the binary focal loss with respect to logits. Show the algebraic steps that lead from the focal loss definition to the gradient expression, analyze how the focusing parameter γ modifies gradient magnitudes for easy (high p) and hard (low p) examples, and discuss the implications for training dynamics.

HardTechnical

149 practiced

Training loss is decreasing but validation metrics are plateauing or degrading—diagnose a scenario where label noise or memorization is occurring. Provide a step-by-step debugging and remediation plan: per-example loss histograms, small-loss selection, co-teaching, early stopping, semi-supervised learning, relabeling, and production safeguards to prevent deploying a memorized model.

Sample Answer

Situation: You train a classifier; training loss keeps falling but validation metrics stall or worsen. That pattern suggests memorization or label noise.

Step-by-step debugging1. Sanity checks: confirm data splits aren’t leaking, batch sizes, augmentations same for train/val, and labels aligned.2. Per-example loss histogram: compute per-example validation and training losses after each epoch; plot histograms and track persistence of high-loss validation examples (likely noisy or OOD).3. Small-loss selection: rank training examples by recent loss. If the model fits a small subset with low loss while many examples remain high-loss, the model is memorizing noisy labels.4. Confusion analysis: inspect high-loss examples and model predictions; sample and annotate by humans to estimate noise rate and type (systematic vs random).5. Learning curves by complexity: measure train/val gap across model sizes and regularization strengths to confirm memorization.

Remediation plan1. Early stopping & stronger regularization (weight decay, dropout, data augmentation) to reduce overfitting.2. Robust loss/meta-loss: use label-smoothing, symmetric losses (e.g., generalized cross-entropy) or bootstrapped loss that blends label and prediction.3. Small-loss / Co-teaching: train two networks; each selects small-loss examples for the other to update, reducing confirmation of noisy labels.4. Semi-supervised learning: treat suspected noisy examples as unlabeled and use consistency regularization (FixMatch, MixMatch) to leverage clean examples.5. Relabeling pipeline: sample high-uncertainty or high-loss examples for expert relabeling or crowdsourced consensus; update dataset and retrain.6. Curriculum learning: start with confident/clean subset and gradually introduce harder examples.

Production safeguards- Data/versioning and provenance: track dataset versions, labelers, and annotation schemas.- Pre-deploy checks: per-example loss distribution, validation on a curated gold set, held-out human-verified sample.- Monitoring: drift detectors (input distribution, prediction confidence), rolling validation metrics, and trigger automated rollback if production performance diverges.- Human-in-loop: flag low-confidence or high-loss predictions for review; A/B test with safety limits.- Explainability signals: monitor attribution shifts that indicate memorization (e.g., saliency on artifacts).

Metrics & iterative plan- Define thresholds: acceptable train/val gap, % noisy-label estimate.- Iterate: relabel a batch, retrain with co-teaching, measure held-out gold set improvement; repeat until metrics stabilize.

This approach balances detecting noise, reducing memorization through robust training, correcting labels, and preventing unsafe deployment.

Loss Functions, Behaviors & Selection Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank