Model Evaluation and Validation Questions

Situation: Timestamp-derived features (e.g., "days since last purchase") are powerful but easy to compute incorrectly for validation sets, causing evaluation leakage and overly optimistic metrics.

Problem (how leakage happens)

• Using a global reference time (e.g., "today") when computing features for validation instead of using each example’s cutoff time.
• Computing features over the full dataset (including events after the validation cutoff) so "last purchase" uses future events.
• Recomputing features with label timestamps (target-time leakage).

Traced example: a customer whose validation cutoff_time is 2024-01-15, and whose last real purchase before that date was on 2023-12-01. The correct days-since-last-purchase feature, computed only from events strictly before cutoff_time, is (2024-01-15 minus 2023-12-01) = 45 days. Now suppose this customer also placed a purchase on 2024-02-01, which is after cutoff_time and should never have been visible to a feature computed for a 2024-01-15 prediction. If the feature pipeline used a global reference time (today, at pipeline-run time) instead of each row's own cutoff_time, it would pick up the 2024-02-01 purchase as the last one and compute (2024-01-15 minus 2024-02-01) = -17 days: a negative, nonsensical value that some pipelines silently clip to 0, while others pass the negative number straight through. Either way the model is being handed information it should not have: the fact that this customer purchases again soon after the prediction date, which is exactly the label-correlated signal validation is supposed to prevent from leaking in. The correct feature value for this row is 45, not -17 or 0.

Safeguards / best practices

• Define a clear per-row cutoff time (prediction time) and compute features using only events with event_time < cutoff_time.
• Build time-aware feature transformers that accept cutoff_time and operate via windowed joins or incremental aggregations.
• Use temporal cross-validation (time-based folds) instead of random splits.
• Use an offline feature store or frozen snapshot versioned per date so training and validation use identical code and data slices.
• Enforce immutable pipelines: feature code should be deterministic and not depend on current system time.

Concrete unit / integration tests (examples)

1. Sanity assertion after feature build:

# features_df has columns: entity_id, cutoff_time, last_event_time
assert (features_df['last_event_time'] < features_df['cutoff_time']).all()

2. Maximum-event-time test:

def test_no_future_events(features_df):
    bad = (features_df['last_event_time'] >= features_df['cutoff_time']).any()
    assert not bad, "Feature used events at or after cutoff_time"

3. Replay test with synthetic clock:

• Create synthetic event history, compute features with known cutoff; compare to expected hand-calculated values.

4. Snapshot reproducibility:

• For a fixed snapshot date, run feature pipeline twice and assert identical outputs (idempotency).

5. Integration fold test:

• For each temporal fold, compute metrics and assert model performance does not drastically exceed performance on later real-time evaluation (detects leakage if train metrics >> production).

Why these work

• They ensure feature calculations are strictly causal relative to cutoff_time and make leakage explicit and automatable. Combining time-aware pipelines, temporal CV, and the shown assertions prevents subtle future-data contamination during validation and deployment.

Approach summary
Start by treating each model as a probabilistic scorer (calibrate if needed), represent predictions plus uncertainty, then fuse using ensemble methods chosen by downstream decision objective (expected utility).

Probabilistic calibration

• Calibrate each model separately: isotonic regression or Platt scaling (two ways of remapping a model's raw scores onto probabilities that actually match observed outcome rates, e.g. so that among all predictions the model called "70% confident" roughly 70% actually turn out positive) for classifiers; temperature scaling (the cheap standard default for calibrating a neural net's confidence: one learned scalar that uniformly softens or sharpens its output probabilities) or beta calibration for NN; histogram/quantile mapping for tree ensembles.
• Validate calibration with calibration curve, Brier score, and expected calibration error (ECE).

Uncertainty propagation

• Capture aleatoric + epistemic: NN: MC dropout (Monte Carlo dropout: run the network many times with different random neurons switched off each pass, and use the spread across those runs as an uncertainty estimate) or ensembles; trees: predictive distributions via quantile regression forests (a variant of a random forest that predicts a full range of outcome quantiles, e.g. the 10th/50th/90th percentile, instead of just a single point estimate, so the spread between quantiles becomes the uncertainty estimate) or bootstrapping.
• Convert outputs to a common probabilistic form (e.g., predictive mean and variance). Propagate through decision function via analytic formulas if linear, otherwise Monte Carlo sampling to estimate downstream expected utility and risk.

Fusion methods

• Stacking: meta-learner trained on out-of-fold predictions; include model uncertainties and features; optimize directly for decision metric (e.g., cost-weighted loss).
• Bayesian Model Averaging (BMA): instead of picking one model's prediction, take a weighted average across every candidate model, where each model's weight is its posterior model probability (how well it has explained the data seen so far); that weight is computed from the marginal likelihood (the probability of the observed data under that model, averaged over all its possible parameter settings) or approximated via variational approximation (a faster, approximate way to estimate that marginal likelihood when the exact calculation is intractable); BMA naturally accounts for model uncertainty but costs more compute than picking one model.
• Hybrid: use BMA priors to regularize stacking weights or Bayesian stacking (optimize the stacking weights themselves under a Dirichlet prior, a probability distribution over sets of weights that sum to 1, which keeps the learned weights well-behaved instead of overfitting to a small validation set).
• When decision utility is asymmetric, train fusion to maximize expected utility or CVaR (Conditional Value at Risk: the average outcome in just the worst-case tail of scenarios, e.g. the worst 5%, used instead of the plain average when you want to be conservative about downside risk rather than optimizing for the typical case).

Computational trade-offs

• Stacking (deterministic) is cheap at inference; requires O(N_meta) extra features and meta-model cost.
• BMA/MC sampling gives better uncertainty but costs M times base-model inference; mitigate with distillation or low-rank approximations.
• Use adaptive evaluation: cheap model first, run expensive model only on uncertain instances (gating).

Validation under covariate shift

• Simulate shifts via importance weighting, label shift correction, or create out-of-distribution holdouts. Use importance-weighted metrics to estimate real-world performance.
• Validate decisions with counterfactual or A/B tests where possible; measure downstream KPIs (expected utility, regret, cost savings), calibration under shift, and robustness metrics (worst-case loss).
• Monitor online: drift detectors on input features and on predicted uncertainty; recalibrate or reweight models when drift detected.

Example (worked numeric trace)

• Tree (quantile forest) raw score 0.70 on this instance, calibrated via isotonic regression to 0.62 (the tree was overconfident on this score region, so calibration pulls it down); its predictive variance from the quantile spread is var_tree = 0.02.
• NN raw score 0.55, calibrated via temperature scaling to 0.58; its predictive variance from MC dropout is var_nn = 0.05 (noisier than the tree here).
• Fuse via inverse-variance weighting (more confident, lower-variance model gets more weight): w_tree = (1/var_tree) / (1/var_tree + 1/var_nn) = (1/0.02) / (1/0.02 + 1/0.05) = 50 / 70 ≈ 0.714, and w_nn = 20/70 ≈ 0.286.
• Fused probability = w_tree * 0.62 + w_nn * 0.58 = 0.714(0.62) + 0.286(0.58) ≈ 0.443 + 0.166 = 0.609.
• Fused variance under the same inverse-variance combination = 1 / (1/var_tree + 1/var_nn) = 1/70 ≈ 0.0143, i.e. a fused standard deviation of about 0.12, tighter than either model alone because combining two independent estimates reduces uncertainty.
• The same mechanics extend to full Bayesian stacking with a Dirichlet prior over more than two models and to importance-weighted expected-utility validation under simulated shifts; deploy with gating (route only uncertain instances, i.e. where fused variance is high, through the expensive NN path) to save compute.

This pipeline balances reliable probabilities, propagated uncertainty, computational cost, and real decision improvement under shift.

Requirements & goals:

• Evaluate ranking quality (relevance), business outcomes (CTR, revenue, retention), and long-term user satisfaction at 100M DAU with low risk.

Offline metrics & protocol:

• Relevance: NDCG@k (Normalized Discounted Cumulative Gain: rewards a ranked list for placing relevant items near the top, discounts relevant items that appear further down, and normalizes against the best possible ordering so scores are comparable across queries), Recall@k (of all the relevant items that exist, the fraction that showed up anywhere in the top k), MAP@k (Mean Average Precision: the average precision computed at each rank position where a relevant item appears in the top k, then averaged across queries) computed on holdout sessions; use session-level aggregation and per-user temporal splits (train on t, test on t+delta).
• Calibration & confidence: compute confidence intervals via bootstrapping by user.
• Diversity & novelty: catalog-based measures (intra-list diversity, coverage).

Correcting position/exposure bias:

• Propensity scoring via logged exposure probabilities (from serving logs): use IPS (inverse propensity scoring) and SNIPS (Self-Normalized IPS: the same reweighting idea as IPS, but rescaled by the sum of the propensity weights rather than by the raw sample count, which reduces the wild variance plain IPS can have when some items had very low exposure probability) to unbiasedly estimate CTR and NDCG.
• Train position-bias models (e.g., an examination model / PBM) to estimate propensities when they aren't logged directly.
• Use doubly robust estimators combining IPS with outcome models to reduce variance.

Online metrics:

• Immediate: raw CTR, conversion rate, revenue per thousand impressions.
• Short-term engagement: session length, day-over-day retention.
• Long-term value: 7/30/90-day retention, LTV, churn rate, downstream purchases.

Logging schema (must be complete & immutable):

• Event id, user_id (hashed), timestamp, session_id, request_id, placement_id, rank_list (item_ids + positions), served_probabilities (model score, softmax prob), exposure_flag per item, click/engagement events with timestamps, item metadata (owner, category), context (device, region), policy_version, experiment_id, traffic_bucket, reward signals (purchase, watch_time), prior user-state feature snapshot. Ensure deterministic replay keys and a sampling indicator for subsampling.

Counterfactual eval & offline simulator:

• Offline simulator: replay logged requests, simulate alternative policies using logged propensity or importance weights. Include synthetic user-response models learned from logs for stress tests (e.g., adversarial content).
• Use IPS/SNIPS/doubly robust for policy evaluation. Validate simulators by backtesting on historical A/B tests.

A/B testing & system components:

• Experiment platform: traffic allocation, randomization (user-level), exposure logging, kill switch.
• Metrics pipeline: near-real-time aggregator for guardrail metrics, weekly cohort analyses for long-term metrics.
• Policy rollout: staged (canary to ramp), automatic risk checks (statistical significance and business bounds).
• Analysis tools: automated uplift estimation, sequential testing with alpha-spending (a rule for how much of your total false-positive budget you're allowed to spend by peeking at a running experiment's results early, so repeated interim looks don't silently inflate the overall false-positive rate the way naive repeated significance testing would), and variance reduction via stratification/ANCOVA (Analysis of Covariance: adjusting an experiment's outcome metric for a pre-experiment covariate, like each user's prior engagement level, to strip out predictable noise before comparing groups, which tightens confidence intervals without needing more traffic).

How offline evaluation, online A/B tests, and champion-challenger deployment fit together:

• Offline metrics are the fast, cheap FILTER: any new policy must beat the current champion on the offline holdout and the offline simulator (using IPS/SNIPS/DR) before it is allowed anywhere near real traffic. This is the stage that screens out most bad candidates for near-zero cost.
• A/B testing is the causal VALIDATION step: a policy that clears the offline bar gets a randomized, low-traffic online test against the current champion, because even debiased offline estimators can miss position-bias or feedback-loop effects that only appear with real exposure.
• Champion-challenger is the ONGOING PRODUCTION pattern once a challenger has won its A/B test: instead of a full one-shot replacement, the challenger is promoted to serve a small, sustained slice of live traffic (e.g. 5-10%) permanently alongside the incumbent champion, with the same online metrics tracked continuously rather than for a fixed test window. This catches slow drift, seasonality, and small regressions a short A/B window would miss, and gives an instant, no-redeploy rollback (shift traffic back to the champion) if the challenger degrades later. Only after a sustained period of the challenger matching or beating the champion does it get promoted to be the new champion, at which point a fresh challenger can be tested against it.
• Together the three form a funnel of increasing cost and decreasing risk: offline (cheap, many candidates screened, imperfect signal) -> A/B (moderate cost, causal, time-boxed) -> champion-challenger (small ongoing cost, the steady-state safety net that a time-boxed test can't provide).

Long-term impact tracking:

• Cohort-based LTV and retention dashboards, causal impact analyses (difference-in-differences, synthetic controls), monitor content-provider effects and feedback loops (popularity bias).
• Periodic offline retraining with debiased labels and causal features to prevent feedback loops.

Trade-offs & operational notes:

• Logging volume: sample some heavy fields but keep deterministic keys for replay.
• Bias-variance: IPS is unbiased but high variance; prefer doubly robust estimators in production.
• Privacy: hash/anonymize PII; consider differential privacy for aggregate dashboards.

This framework provides unbiased offline evaluation, safe online experimentation, and a champion-challenger steady state to iterate recommendation policies at scale with continuous, low-cost safety monitoring.

Stacking can leak the target if base-model predictions used as features for the meta-model were trained on the same rows: causing over-optimistic validation AUC that won't generalize in production. The safe pattern uses out-of-fold (OOF) predictions for training the meta-model so every meta-feature for a training row is produced by a base model that didn’t see that row.

Key leakage risks:

• Training base models on full training data and using their predictions on that same data to train the meta-model.
• Including features derived from future folds or using global statistics computed with target from full data.
• Improper cross-validation split (e.g., time series treated as i.i.d.).

Correct CV-based stacking procedure (concept):

1. Split training data into K folds.
2. For each base model and each fold:
   • Train base model on K-1 folds.
   • Predict on the held-out fold → fill OOF predictions.
3. After all folds, each training row has OOF predictions from each base model → train meta-model on these.
4. For production: retrain each base model on the full training set, produce base predictions for new data, then feed into the trained meta-model.

Pseudocode (safe training & prediction):

import numpy as np

# training
K = 5
oof_preds = {model_name: np.zeros(n_train) for model_name in base_models}
for fold_train_idx, fold_val_idx in KFold(n_splits=K).split(X_train):
    X_tr, y_tr = X_train[fold_train_idx], y_train[fold_train_idx]
    X_val = X_train[fold_val_idx]
    for name, model in base_models.items():
        m = clone(model)
        m.fit(X_tr, y_tr)                          # never sees fold_val rows
        oof_preds[name][fold_val_idx] = m.predict_proba(X_val)[:,1]

# build meta-training set
X_meta_train = concat_columns([oof_preds[name] for name in base_models])
meta_model.fit(X_meta_train, y_train)

# final base models retrained on full data for production
final_base_models = {}
for name, model in base_models.items():
    m = clone(model)
    m.fit(X_train, y_train)
    final_base_models[name] = m

# prediction in production
def predict_production(X_new):
    base_preds = [m.predict_proba(X_new)[:,1] for m in final_base_models.values()]
    X_meta = concat_columns(base_preds)
    return meta_model.predict_proba(X_meta)[:,1]

Additional notes:

• For time-series use time-aware CV (purge, embargo).
• Ensure no target-derived global scalers are fit on full data; fit scalers inside folds or using training-only data then apply.
• Validate stacked pipeline with a held-out test set or nested CV to estimate real-world performance.

Brief framing

For summarization we need both token-level and sequence-level metrics because they capture different failure modes: lexical overlap, fluency, and semantic adequacy. We also need a language-model-based sanity check (perplexity) and a plan for what to do when the numbers and the humans disagree.

Token-level / overlap metrics

• Token overlap (precision/recall/F1 on n-grams): simple, fast; useful for exact matching and extractive summaries but penalizes paraphrase.
• ROUGE (R1/R2/L): recall-oriented n-gram and longest-common-subsequence measures; standard for summarization, correlates with content coverage.
• BLEU: precision-oriented n-gram metric from MT; less ideal for single-reference summaries and brevity-sensitive, but useful as a complementary precision signal.

Perplexity

• What it measures: how well a language model predicts the generated text token by token; it is a fluency/well-formedness signal about the OUTPUT text alone, not a comparison to the reference or the source.
• Why (and why not) to use it: low perplexity tells you the summary reads like natural, grammatical text; it says nothing about whether the summary is faithful to the source or covers the right content, so a fluent hallucination scores well. Use it as a cheap automatic gate to catch degenerate or repetitive output (e.g., a model collapsing into repeated phrases will show a perplexity spike relative to its own baseline), not as a quality or faithfulness metric.

Sequence-level / holistic metrics

• METEOR / BERTScore / MoverScore: embedding-aware metrics that capture paraphrase and synonymy; BERTScore often correlates better with human judgments on semantic similarity.
• Fact-based metrics: QAGS, QuestEval: automated factuality via question generation + QA to detect hallucinations.

Recommended evaluation suite (production)

• ROUGE-L and ROUGE-1/2 (coverage baseline)
• BERTScore (semantic similarity)
• QAGS or QuestEval for factuality
• Perplexity (relative to the model's own historical baseline) as a fluency/degeneration tripwire, never as a primary quality score
• Length, novelty (n-gram overlap with source), and readability (FKGL)
• Human evaluation: adequacy, fluency, coherence, factuality, and preference tests

Human-eval protocol

• Stratified sampling across lengths, topics, and model confidence
• 3+ annotators per item, majority vote + Cohen's kappa for reliability
• Use Likert scales for adequacy/fluency + binary factuality checks with evidence highlighting
• Paired A/B preference tests for UX decisions; collect free-text failure descriptions

Reconciling automatic metrics with human judgment when they disagree

• Treat human judgment as ground truth for the release decision; automatic metrics are a cheap, noisy proxy for it, never a substitute.
• When ROUGE/BERTScore says a new model is better but human preference disagrees (or vice versa), bucket the disagreeing examples and read them: this is usually where the automatic metric's known blind spot fires, e.g., a paraphrased-but-faithful summary scoring low on n-gram overlap (ROUGE penalty), or a fluent hallucination scoring high on BERTScore/perplexity but failing factuality.
• Quantify the disagreement, don't just note it: compute the correlation (Spearman/Kendall) between each automatic metric and the human preference on the current sample, and track it release over release. A metric whose correlation with human judgment is dropping is a signal that model changes have started to specifically exploit that metric's blind spot (a Goodhart's-law failure mode), and it should be down-weighted or replaced, e.g., swap in QAGS/QuestEval if factuality is the recurring gap.
• Operationally: never ship on an automatic-metric win alone if the paired human comparison disagrees; require the human preference test to be at least non-inferior (via the paired significance test from the human-eval protocol) before promoting a model, and use the automatic metrics for cheap continuous regression testing between the periodic human evaluations.

Justification
Combining ROUGE (coverage), BERTScore (paraphrase), QAGS (factuality), and perplexity (fluency/degeneration tripwire) balances classical reproducibility with semantic and factual assessment, while treating perplexity as a narrow sanity check rather than a quality signal. Human protocols validate the automated signals and capture nuanced errors (hallucination, incoherence) critical for product safety and user trust, and the reconciliation step keeps the automatic metrics honest as models evolve.