Model Evaluation and Quality Assessment Questions

Covers evaluation methods, metrics, and quality assessment approaches for machine learning models including both predictive models and generative models. Topics include selecting appropriate metrics such as accuracy, precision, recall, F one score, area under curve for ranking, root mean square error and mean absolute percentage error for regression, and the rationale for using multiple metrics and baselines. For generative and large language models, covers automatic metrics such as BLEU, ROUGE, METEOR, semantic similarity scores, LLM based evaluation techniques, human evaluation frameworks, factuality and hallucination checking, adversarial and stress testing, error analysis, and designing scalable, cost effective evaluation pipelines and quality assurance processes.

EasyTechnical

65 practiced

Explain k-fold cross-validation. What are the main benefits and common pitfalls? When would you prefer stratified k-fold over ordinary k-fold, and what leakage risks must you avoid when using cross-validation?

HardTechnical

74 practiced

Describe methods to estimate predictive uncertainty and build calibrated prediction intervals for neural networks: compare MC dropout, deep ensembles, Bayesian neural networks, and quantile regression. Discuss calibration, computational cost, ease of implementation, and suitability for production systems that require uncertainty estimates.

Sample Answer

Predictive uncertainty methods fall into two groups: probabilistic (estimate full predictive distribution) and non-probabilistic (directly estimate intervals/quantiles). I'll compare MC dropout, deep ensembles, Bayesian NNs, and quantile regression across calibration, cost, implementation, and production suitability.

MC dropout- Idea: keep dropout at inference, average T stochastic forward passes to approximate posterior predictive.- Calibration: decent for epistemic uncertainty if T large; often underestimates uncertainty for complex tasks.- Cost: T× inference cost (cheap to implement, no model changes).- Implementation: trivial on most frameworks (train normally with dropout).- Production: suitable if you can afford multiple forward passes and latency is tolerant; easy and low-risk.

Deep ensembles- Idea: train M independently initialized networks; aggregate predictions (mean/variance).- Calibration: often best empirical calibration and robustness; captures multimodal/posterior mass better.- Cost: M× training and inference cost, but inference can be parallelized.- Implementation: straightforward; scalable with orchestration.- Production: excellent if resources allow; high reliability and simple to deploy as separate models or shard traffic.

Bayesian neural networks (BNNs / variational)- Idea: learn posterior over weights (e.g., variational inference, HMC).- Calibration: principled uncertainty; quality depends on approximate posterior; can be overconfident if VI is poor.- Cost: high (complex inference, slower training/inference).- Implementation: harder—requires probabilistic layers and careful priors.- Production: powerful but engineering-heavy; use when principled Bayesian treatment is required.

Quantile regression- Idea: directly predict quantiles (e.g., 5th/95th) via pinball loss; outputs conditional intervals.- Calibration: can be calibrated for aleatoric (data) uncertainty; doesn't capture epistemic unless combined with ensembles or dropout.- Cost: single forward pass, minimal overhead.- Implementation: easy (modify loss/output heads).- Production: very practical when you need fast, calibrated prediction intervals for observed noise; combine with ensembles to capture model uncertainty.

Practical guidance- If low latency and only aleatoric uncertainty matter: quantile regression.- If best calibration and robustness needed and resources exist: deep ensembles.- If quick, low-effort uncertainty with flexible compute budget: MC dropout.- If you need principled Bayesian inference and can invest engineering effort: BNNs.

Calibration improvements- Post-hoc methods (temperature scaling for probabilistic outputs, isotonic regression, conformal prediction) are valuable to guarantee interval coverage—use conformal prediction to turn any point/quantile predictor into calibrated intervals with finite-sample guarantees.

Edge cases & tips- For OOD detection prefer epistemic methods (ensembles, BNNs, MC dropout) and combine with input-density or trust-score checks.- Measure calibration with reliability diagrams, expected calibration error (ECE), and interval coverage probability.- Monitor compute/latency trade-offs and experiment: often ensembles+conformal gives best practical mix of calibration and reliability for production.

EasyTechnical

71 practiced

Describe how bootstrap resampling can be used to compute confidence intervals for an evaluation metric such as F1 or AUC. Outline the steps of the bootstrap procedure, choices to make (number of resamples, percentile vs bias-corrected intervals), and limitations of bootstrap-based CIs.

Sample Answer

Bootstrap resampling estimates uncertainty for metrics (F1, AUC) by repeatedly sampling with replacement from your test set and computing the metric on each resample to form an empirical distribution.

Procedure:1. Start with your original labeled test set of N examples.2. Repeat B times (e.g., B = 1000): - Draw a bootstrap sample of size N with replacement from the test set. - Compute the evaluation metric (F1, AUC) on that sample. - Store the metric value.3. After B resamples you have an empirical distribution of the metric. Use it to form confidence intervals (CIs) or p-values.

Choices to make:- Number of resamples B: common choices 1000–10,000. More resamples reduce Monte Carlo noise; 1000 is often sufficient for 95% CIs but increase if you need tighter stability.- CI method: - Percentile (simple): take the α/2 and 1−α/2 quantiles of the bootstrap distribution (e.g., 2.5% and 97.5% for 95% CI). - Bias-corrected and accelerated (BCa): adjusts for bias and skewness in the bootstrap distribution; preferred when distribution is asymmetric or estimator biased. - Studentized bootstrap: uses standard errors per resample—more complex but can be more accurate.- Use stratified bootstrap if class imbalance matters (sample within classes) to preserve prevalence.

Limitations:- Assumes test set is representative and i.i.d.; if data are dependent (time series, clustered) standard bootstrap is invalid.- For small N or extreme metrics (near 0 or 1, or F1 with zero positives), bootstrap CIs can be unreliable or discrete.- Computational cost: B evaluations can be heavy for expensive models (though you can cache predictions and recompute metrics fast).- Does not account for model training variability unless you bootstrap the whole train+test pipeline (nested/bootstrap-of-models).

Practical tips:- Cache model scores and only resample predictions+labels (not retrain) unless you want to capture training instability.- Report CI method and B, and visualize the bootstrap distribution to check skewness and multimodality.

MediumTechnical

72 practiced

For a machine translation system, explain strengths and weaknesses of BLEU and ROUGE. Introduce embedding-based metrics like BERTScore and discuss when they better correlate with human judgments. Finally, describe how you would combine automatic metrics with human evaluation in a practical evaluation plan.

Sample Answer

BLEU (n‑gram precision with brevity penalty)- Strengths: fast, deterministic, easy to compute at scale; good for tracking system regressions and corpus-level comparisons.- Weaknesses: surface-form focused (exact n‑gram matches), punishes valid paraphrases, poor sentence-level reliability, sensitive to reference count and tokenization.

ROUGE (recall-oriented n‑gram overlap, variants like ROUGE‑L)- Strengths: emphasizes coverage of reference content, commonly used for summarization; simple and interpretable.- Weaknesses: same lexical-match limits as BLEU; recall bias can reward verbosity; not semantic.

Embedding‑based metrics (e.g., BERTScore)- Method: compute contextual token embeddings (BERT), align/hit-match tokens by cosine similarity, optionally IDF‑weight.- Strengths: captures synonyms, paraphrases and semantic equivalence; better correlates with human judgments at sentence and system level when lexical variation is high or for morphologically rich languages.- Weaknesses: depends on the embedding model, can be fooled by fluent but incorrect (hallucinated) text, less interpretable; may reflect model biases.

When BERTScore outperforms BLEU/ROUGE- High paraphrase rate, diverse valid translations, modern NMT systems producing fluent but lexically different outputs, or evaluation at sentence-level.

Practical evaluation plan (combining automatic + human)1. Automatic pipeline for continuous monitoring: - Compute BLEU/ROUGE for long-term trends and regression alerts. - Compute BERTScore (and COMET if available) to assess semantic fidelity.2. Sampling strategy for human evaluation: - Stratified sample across metric percentiles (low/medium/high BERTScore & BLEU) and by segtypes (length, rare words, language pair).3. Human protocol: - Use Direct Assessment (continuous adequacy and fluency Likert scales) or MQM for error taxonomy. - 3 raters per item, measure inter‑annotator agreement (Krippendorff’s alpha); adjudicate disagreements.4. Analysis: - Correlate automatic metrics with human scores (Pearson/Spearman) to validate which metric aligns best. - Use human labels to calibrate thresholds for automated alerts and to train learned metrics.5. Operationalize: - Run automatic checks on every run; schedule human evaluations for major releases or drift alarms. - Report both metric trends and human-derived error categories to guide prioritization.

This mix leverages automatic metrics for scale and speed, and human evaluation for trust, calibration, and diagnosing semantic/quality issues.

EasyTechnical

68 practiced

Explain precision, recall, specificity (true negative rate), and F1 score for binary classification. For each metric, state the formula using TP, FP, TN, FN; describe a scenario where it is the most important metric; and give one limitation. Provide a small confusion-matrix example (numbers) and compute all four metrics from it.

Unlock Full Question Bank

Get access to hundreds of Model Evaluation and Quality Assessment interview questions and detailed answers.

Join thousands of developers preparing for their dream job.