Model Evaluation and Validation Questions

Start by clarifying objectives and constraints: what business metric(s) matter (accuracy, precision@k, revenue uplift, false positive rate), acceptable risk/tolerance, and monitoring cadence. That sets which slices and actions matter.

1. Define slices consistently, including intersections

• Domain-driven single-factor slices: geography, OS/device, user cohort (new vs returning), traffic source, time-window.
• Intersectional slices: also enumerate deliberate crossings of these factors (e.g., mobile x Region A, new-user x low-income geography). Intersectional cohorts matter because a model can look fine on every single-factor slice while failing specifically on a combination: the classic example is a system with acceptable per-gender and per-skin-tone error rates that is far worse on their intersection. Enumerate the crossings your risk register cares about ahead of time (2-3 factors deep); don't rely on discovering them after the fact, because the number of possible crossings grows combinatorially and most will never accumulate enough traffic to review manually.
• Ensure mutually intelligible naming and stable join keys; persist slice metadata, including which factors were crossed to produce each intersectional slice id.

2. Prioritize slices
   Score each slice (single-factor and intersectional) by a mix of:

• Impact: fraction of traffic or revenue affected.
• Risk: business or fairness sensitivity (e.g., legal/regulatory regions, or protected-attribute intersections).
• Performance variance: historical delta vs baseline.
  Compute priority = weighted(sum(impact, risk, variance, strategic importance)). Focus top X% or cumulative 80% of impact. Intersectional slices are usually lower-traffic than their parent single-factor slices by construction, so a pure impact-based ranking will systematically starve them; give known-high-risk intersections (e.g., regulatory or fairness-sensitive combinations) a floor priority regardless of raw traffic share.

3. Reliable metrics with small samples, including compounding intersections

• Report both point estimate and uncertainty. Use Bayesian smoothing (Beta posterior for rates) or Wilson score for proportions to avoid extreme estimates.
• For continuous metrics, use bootstrapped confidence intervals or hierarchical (partial pooling) models to borrow strength across similar slices.
• For intersectional slices specifically, single-factor partial pooling isn't enough: use a hierarchical model with crossed random effects (e.g., a random intercept per geography, per device, AND per geography-device interaction) so the geo-x-device cell borrows strength from its parent geo and device marginals rather than being estimated in isolation on a handful of points.
• Aggregate / roll up: combine adjacent bins (time or geography) when safe; use minimum effective sample size rule (e.g., require N such that SE < threshold) before taking action.
• Present effect size relative to baseline with CI and p-values adjusted for multiple comparisons (Benjamini-Hochberg) if testing many slices, since intersectional enumeration multiplies the number of hypotheses tested.

4. Thresholds for action
   Define tiered rules:

• Informational: difference > noise but small impact (log for watch).
• Investigate: effect size exceeds practical significance threshold (e.g., delta > X% and CI excludes 0).
• Action: meaningful business impact (priority score high, delta large, lower bound of CI > action threshold).
  Set thresholds using historical variance and business cost model (cost of false positive vs false negative). Example: require lower bound of 95% CI on revenue delta to be > expected deployment cost.

5. Operationalize monitoring and workflow

• Automate slice-level dashboard with estimates, CI, sample size, and priority score, covering both single-factor and the pre-registered intersectional slices.
• Auto-alert when high-priority slice crosses "investigate" or "action" thresholds.
• Triage playbook: reproducibility check -> data pipeline integrity -> model explainability (feature attributions) -> rollback / targeted retrain / business mitigation.
• Periodic audits for fairness-sensitive slices (including their intersections) with stricter thresholds and human review.

6. Example
   If mobile users in Region A (5% traffic, an intersection of the device and geography factors) show conversion drop from 4%->2%:

• Compute a Beta posterior for both groups (or a crossed-random-effects estimate borrowing strength from the mobile marginal and the Region A marginal) to get a 95% credible interval.
• If the lower bound of the delta's CI is below 0 but the effect size is large and Region A is high-risk, flag for immediate investigation even with small N; consider a targeted A/B or data-collection effort to increase confidence, since this cell will rarely accumulate enough native traffic on its own.

This approach balances statistical rigor, business impact, and operational practicality so you act where it matters, deliberately including cohorts defined by more than one attribute, while avoiding overreaction to noisy small-sample signals.

Random train/test splits:

• Randomly sample rows into train and test. Assumes i.i.d. data and exchangeability. Good for static datasets without time dependence but invalid for time-series because it mixes past and future and can produce optimistic estimates.

Holdout test set:

• Reserve a contiguous portion of data (often the most recent period) as a final unseen evaluation set. Used to simulate production performance once models and hyperparameters are fixed. Prevents repeated peeking at test data.

Temporal (rolling / forward-chaining) splits:

• Respect time order. Examples:
  • Holdout: train on t0..tN, validate on tN+1..tM, test on tM+1..tK.
  • Rolling / walk-forward CV: repeatedly train on [t0..ti], validate on [ti+1..ti+h], advance window. Provides robust estimate of performance over time and for model stability.

For next-day demand forecasting I would choose temporal splits (walk‑forward / forward‑chaining) with a final contiguous holdout of the most recent period. Reason: the prediction task is inherently temporal: using future data to predict past inflates performance. Walk‑forward CV mirrors production retraining cadence and reveals degradation over time; the final holdout simulates deployment.

Concrete calendar walkthrough for 3 years of daily demand data (2021-01-01 through 2023-12-31): require at least 12 months of training history per fold so every fold's model has seen a full seasonal cycle (weekly pattern plus the yearly holiday peak) at least once, then advance the window one quarter at a time:

• Fold 1: train 2021-01-01 to 2022-06-30 (18 months), validate 2022-07-01 to 2022-09-30 (Q3 2022).
• Fold 2: train 2021-01-01 to 2022-09-30, validate 2022-10-01 to 2022-12-31 (Q4 2022, which includes the holiday-season demand spike, an important seasonal stress test).
• Fold 3: train 2021-01-01 to 2022-12-31, validate 2023-01-01 to 2023-03-31 (Q1 2023).
• Fold 4: train 2021-01-01 to 2023-03-31, validate 2023-04-01 to 2023-06-30 (Q2 2023).
• Final holdout: 2023-07-01 to 2023-12-31, evaluated once, after all fold-based tuning is frozen.

What a degrading model's walk-forward curve looks like: suppose validation MAPE comes back as 8% (fold 1), 9% (fold 2), 12% (fold 3), 15% (fold 4). A single train/test split would only ever have shown one of these numbers; the walk-forward sequence shows the error creeping from 8% to 15% across four folds, which is the detection mechanism itself; a model whose error is flat or improving across folds is stable, while one that climbs the way this one does is quietly losing relevance to the current data (e.g. because demand patterns have shifted since the earliest training data) and should trigger a retrain or a feature review before it reaches production.

Key leakage risks to avoid:

• Target leakage: including features that contain information that would only be available after the prediction time (e.g., next-day promotions, returns aggregated including the target day).
• Temporal aggregation leakage: engineering aggregates (rolling means, cumulative sums) computed using the full series instead of cutoff at prediction time.
• Leakage via preprocessing: scaling, imputation, or feature selection fit on full dataset rather than within each training fold.
• Data duplication: identical observations across splits (e.g., user IDs repeated) that connect train and test.
• Label propagation: joining external data indexed by future timestamps.

Best practices:

• Always compute features using only past information up to prediction cutoff.
• Use pipeline-aware preprocessing (fit on train fold, apply to validation/test).
• Keep a final chronological holdout for unbiased evaluation.
• Monitor backtests by time slice and track performance drift; retrain frequently if nonstationarity detected.

Start by clarifying the experimental design: are runs paired? (e.g., you trained both models with the same sequence of random seeds / hyper init / data splits). If yes, treat observations as paired: this removes between-seed variance and is more powerful. If runs are independent (different seeds/splits with no pairing), use unpaired methods.

Recommended workflow

1. Exploratory checks

• Plot per-run accuracies (paired lines, violin/boxplots); inspect distributions and outliers.
• Check normality (Shapiro-Wilk: tests whether a sample looks like it was drawn from a normal distribution) only to choose parametric vs nonparametric methods: don’t over-rely on normality tests with n=20.

2. Primary statistical comparison

• Paired case: use paired t-test (for mean difference) if differences look roughly symmetric; otherwise use Wilcoxon signed-rank test (ranks the paired differences and tests whether the ranks skew positive or negative, used instead of the paired t-test when the differences aren't roughly symmetric or normal). Also compute paired permutation test (exact or Monte Carlo: repeatedly shuffle which run's accuracy 'belongs' to Model A vs Model B, recompute the mean difference each time, and see how often a shuffled difference is as large as the one actually observed) as a robust alternative.
• Unpaired case: use two-sample t-test (Welch’s) if variances differ; otherwise Mann–Whitney U or permutation test.

3. Bootstrap confidence intervals

• Use bootstrap (paired-resampling if paired) to estimate 95% CI for the statistic you care about (mean difference, median difference, or percentage improvement). Bootstrapping is recommended because it: (a) directly quantifies uncertainty of the chosen effect metric, (b) works without strong parametric assumptions, and (c) lets you get CIs for nonstandard metrics (e.g., percent change, AUC difference).
• Procedure: resample runs with replacement (preserving pairing when appropriate), compute statistic for each bootstrap sample, take percentile or bias-corrected accelerated (BCa: a bootstrap interval variant that adjusts the percentile cutoffs for skew and bias in the resampled distribution, more accurate than a plain percentile interval when the statistic's distribution isn't symmetric) intervals.

4. Multiple metrics / multiple comparisons

• If you evaluate multiple metrics (accuracy, F1, calibration, latency) correct p-values or control error rate. For a small set of hypotheses, use Holm-Bonferroni (more power than Bonferroni). For many correlated metrics, consider Benjamini-Hochberg to control the FDR (False Discovery Rate: the expected fraction of your 'significant' results that are actually false alarms; controlling it, rather than the stricter family-wise error rate that Holm-Bonferroni controls, trades a few more false positives for much more power to detect real differences across many metrics).
• If reporting multiple effect sizes / CIs, emphasize CIs over binary “significant/not-significant” calls and adjust interpretation rather than overcorrecting (stakeholders care about practical impact).

5. Effect sizes and presentation

• Report:
  • Point estimate: a worked pass on 6 illustrative runs (standing in for the full 20), Model A = [91%, 89%, 93%, 90%, 92%, 88%], Model B = [93%, 90%, 94%, 92%, 93%, 91%], gives per-run paired differences (B-A) = [2, 1, 1, 2, 1, 3] points, mean difference = 1.67 percentage points, sample SD of the differences = 0.82 points.
  • Paired t-test on those 6 differences: t = mean_diff / (sd_diff/sqrt(n)) = 1.67 / (0.82/sqrt(6)) ≈ 5.00 on 5 degrees of freedom, two-sided p ≈ 0.004, comfortably below a 0.05 threshold.
  • 95% bootstrap CI on the mean difference (10,000 resamples of the 6 paired differences): approximately [1.2%, 2.3%], consistent with the t-test in ruling out a difference of zero.
  • Paired Cohen’s d (or hedge’s g) for standardized magnitude (for paired: d = mean(differences)/sd(differences) = 1.67/0.82 ≈ 2.0, a very large standardized effect on this small illustrative sample, since the differences are consistently positive with little run-to-run spread)
  • Probability of superiority or P(X_B > X_A) from bootstrap/permutation for intuitive interpretation
  • Relative improvement (percent) if meaningful
  • Scaling up: with the real 20 runs per model, the same procedure (paired differences, paired t-test or Wilcoxon, bootstrap CI, Cohen's d) applies unchanged; more runs would be expected to tighten the CI and stabilize the effect-size estimate versus this 6-run illustration.
• Visuals:
  • Paired scatter/line plot showing per-seed pairs
  • Violin or boxplots with overlaid bootstrap distribution of the difference
  • Forest plot of point estimates + CIs for multiple metrics
• Communicate uncertainty: present CIs and probability statements (e.g., “There is a 97% estimated probability that Model B’s accuracy exceeds Model A’s by at least 0.2%.”). Avoid over-reliance on p-values; provide p-values but focus on magnitude + CI + business relevance.

Other considerations

• Check independence assumptions (runs must be independent).
• If results hinge on small differences, consider increasing number of runs or doing nested CV to reduce variance.
• Reproduce analyses (seed list, code) and report exact tests used, adjustment method, and assumptions.

Summary: Prefer paired tests when possible. Use permutation tests and bootstrap CIs for robustness. Correct for multiple metrics with Holm or BH depending on goals. Report effect sizes with CIs and intuitive probability statements, and accompany with clear visualizations so stakeholders can judge practical impact.

Investigation plan: goal: find why offline lift (A > B) didn't surface online. I'll run targeted checks in parallel (triage → hypothesis testing → remediation).

1. Reproduce the discrepancy

• Verify offline evaluation uses the exact same model artifacts, preprocessing, feature generation, and decision logic as deployed.
• Run a small shadow test in production (log model A and B decisions on live traffic without affecting users) to compare predictions and inputs.

2. Metric alignment

• Compare offline objective vs online KPI. Map model output → business action → online metric. If offline uses log-loss or AUC but online cares about click-through or conversion, retrain/evaluate using the online metric (or a proxy such as calibrated probabilities leading to same decision thresholds).
• Compute calibration and decision-threshold swept offline performance for the exact online metric.

3. Sampling bias / population shift

• Compare train/validation distribution to online traffic (user segments, time-of-day, geos, devices). Use population statistics (feature marginals), covariate shift tests (KL, PSI), and model performance broken down by segment.
• If bias found, reweight offline test set to match production distribution or evaluate on stratified holdouts.

4. Instrumentation and logging

• Audit feature generation and any upstream transformations in production for missing/lagged features, default fill values, or unit mismatches. Check feature freshness and latency.
• Validate online event instrumentation for the outcome label (deduplication, attribution windows, causal assignment correctness).

5. Feature leakage and training-serving skew

• Scan for features that leak future info in offline pipeline (timestamps, labels-derived features). Run a “time-forward” evaluation where features only use information available at decision time.
• Compare feature statistics between training snapshots and serving logs to catch serving-time approximations.

6. Exposure and feedback loops

• Check whether model A changes downstream user behavior differently (novelty, long-term effects) not captured offline. Run short-term vs long-term metric analysis and look for delayed signals.
• Ensure randomization in A/B test is correct and no spillover between buckets.

Applying this to the stated scenario: suppose Model A scored 0.85 AUC offline versus Model B's 0.80, a 5-point offline gap, but online conversion came back at 3.20% for A versus 3.19% for B, statistically indistinguishable. Step 2 (metric alignment) would check whether that 5-point offline AUC gap actually maps to more relevant top-ranked items in the live product, since AUC rewards correct ranking across the whole score range while conversion only cares about the handful of items actually surfaced to a user; a model can win on AUC by getting the middle of the distribution right while tying on the top-k that users actually see. Step 3 (sampling bias) would check whether the offline evaluation set's traffic mix (say, weighted toward desktop, daytime, US users) matches the live population the A/B test actually ran on; if Model A's 5-point AUC edge came disproportionately from segments underrepresented in live traffic, the offline number overstates what online users would ever notice. Working through even one of these two steps against the actual scenario numbers, rather than leaving them as generic checklist items, is usually enough to locate which of the two explanations (metric mismatch or population mismatch) is driving the specific gap.

Concrete changes to offline evaluation

• Use holdout sets that mimic production sampling (stratified by geo/device/time) and apply importance weighting when distributions differ.
• Evaluate using the online business metric (or a well-validated proxy); perform threshold-based and counterfactual simulations to map offline scores to online decisions.
• Implement production shadowing as standard: daily sample of live traffic scored with both models and logged for retrospective analysis.
• Add automatic checks: calibration, PSI per feature, training-serving skew alerts, and unit tests validating feature parity.
• Remove or flag any features that rely on future signals; enforce “information cutoff” during dataset assembly.
• Periodically run randomized offline experiments (simulate assignment logic) and maintain an issues playbook linking offline failure modes to remediation steps.

Expected outcome: faster root-cause identification, higher offline/online correlation, fewer failed launches.

Brief framing

For summarization we need both token-level and sequence-level metrics because they capture different failure modes: lexical overlap, fluency, and semantic adequacy. We also need a language-model-based sanity check (perplexity) and a plan for what to do when the numbers and the humans disagree.

Token-level / overlap metrics

• Token overlap (precision/recall/F1 on n-grams): simple, fast; useful for exact matching and extractive summaries but penalizes paraphrase.
• ROUGE (R1/R2/L): recall-oriented n-gram and longest-common-subsequence measures; standard for summarization, correlates with content coverage.
• BLEU: precision-oriented n-gram metric from MT; less ideal for single-reference summaries and brevity-sensitive, but useful as a complementary precision signal.

Perplexity

• What it measures: how well a language model predicts the generated text token by token; it is a fluency/well-formedness signal about the OUTPUT text alone, not a comparison to the reference or the source.
• Why (and why not) to use it: low perplexity tells you the summary reads like natural, grammatical text; it says nothing about whether the summary is faithful to the source or covers the right content, so a fluent hallucination scores well. Use it as a cheap automatic gate to catch degenerate or repetitive output (e.g., a model collapsing into repeated phrases will show a perplexity spike relative to its own baseline), not as a quality or faithfulness metric.

Sequence-level / holistic metrics

• METEOR / BERTScore / MoverScore: embedding-aware metrics that capture paraphrase and synonymy; BERTScore often correlates better with human judgments on semantic similarity.
• Fact-based metrics: QAGS, QuestEval: automated factuality via question generation + QA to detect hallucinations.

Recommended evaluation suite (production)

• ROUGE-L and ROUGE-1/2 (coverage baseline)
• BERTScore (semantic similarity)
• QAGS or QuestEval for factuality
• Perplexity (relative to the model's own historical baseline) as a fluency/degeneration tripwire, never as a primary quality score
• Length, novelty (n-gram overlap with source), and readability (FKGL)
• Human evaluation: adequacy, fluency, coherence, factuality, and preference tests

Human-eval protocol

• Stratified sampling across lengths, topics, and model confidence
• 3+ annotators per item, majority vote + Cohen's kappa for reliability
• Use Likert scales for adequacy/fluency + binary factuality checks with evidence highlighting
• Paired A/B preference tests for UX decisions; collect free-text failure descriptions

Reconciling automatic metrics with human judgment when they disagree

• Treat human judgment as ground truth for the release decision; automatic metrics are a cheap, noisy proxy for it, never a substitute.
• When ROUGE/BERTScore says a new model is better but human preference disagrees (or vice versa), bucket the disagreeing examples and read them: this is usually where the automatic metric's known blind spot fires, e.g., a paraphrased-but-faithful summary scoring low on n-gram overlap (ROUGE penalty), or a fluent hallucination scoring high on BERTScore/perplexity but failing factuality.
• Quantify the disagreement, don't just note it: compute the correlation (Spearman/Kendall) between each automatic metric and the human preference on the current sample, and track it release over release. A metric whose correlation with human judgment is dropping is a signal that model changes have started to specifically exploit that metric's blind spot (a Goodhart's-law failure mode), and it should be down-weighted or replaced, e.g., swap in QAGS/QuestEval if factuality is the recurring gap.
• Operationally: never ship on an automatic-metric win alone if the paired human comparison disagrees; require the human preference test to be at least non-inferior (via the paired significance test from the human-eval protocol) before promoting a model, and use the automatic metrics for cheap continuous regression testing between the periodic human evaluations.

Justification
Combining ROUGE (coverage), BERTScore (paraphrase), QAGS (factuality), and perplexity (fluency/degeneration tripwire) balances classical reproducibility with semantic and factual assessment, while treating perplexity as a narrow sanity check rather than a quality signal. Human protocols validate the automated signals and capture nuanced errors (hallucination, incoherence) critical for product safety and user trust, and the reconciliation step keeps the automatic metrics honest as models evolve.