Recommendation, Ranking, and Personalization Questions

To compute recall@k and MRR for a batch efficiently, we can use vectorized NumPy operations: map ground-truth IDs to boolean matches across the predictions, compute recall per row (did GT appear anywhere in top-k) and reciprocal rank (1 / (rank index + 1) for the first match). The implementation below treats missing ground-truths (None or np.nan) by ignoring those rows in aggregate metrics.

import numpy as np

def batch_recall_mrr(gt_items, pred_topk):
    """
    Compute Recall@k and MRR for a batch.
    gt_items: 1D array-like of ground-truth item IDs (length N). Can contain None or np.nan for missing GT.
    pred_topk: 2D array-like shape (N, k) of predicted item IDs (top-k per session).
    Returns: (recall, mrr, valid_count)
    """
    gt = np.array(gt_items, dtype=object)
    preds = np.array(pred_topk, dtype=object)
    if gt.shape[0] != preds.shape[0]:
        raise ValueError("gt_items and pred_topk must have same batch size")

    # Mask rows with missing ground-truths (None or NaN)
    missing_mask = np.array([(x is None) or (isinstance(x, float) and np.isnan(x)) for x in gt])
    valid_idx = ~missing_mask
    if valid_idx.sum() == 0:
        return 0.0, 0.0, 0

    gt_valid = gt[valid_idx].reshape(-1, 1)            # (M,1)
    preds_valid = preds[valid_idx]                     # (M,k)

    # Vectorized equality across each row -> boolean matrix (M,k)
    matches = (preds_valid == gt_valid)

    # Recall@k: row-wise any match
    recall_per_row = matches.any(axis=1).astype(float)  # 1.0 if GT in top-k else 0.0
    recall = recall_per_row.mean()

    # MRR: find first match index per row, reciprocal rank else 0
    # argmax on matches returns first True if any, but need to mask rows with no match
    first_match_idx = np.argmax(matches, axis=1)        # returns 0..k-1, or 0 if no True
    has_match = matches.any(axis=1)
    rr = np.zeros_like(first_match_idx, dtype=float)
    rr[has_match] = 1.0 / (first_match_idx[has_match].astype(float) + 1.0)
    mrr = rr.mean()

    return float(recall), float(mrr), int(valid_idx.sum())

Key points:

• Fully vectorized with O(N*k) time using NumPy operations.
• Space: O(N*k) for the boolean match matrix.
• Robust to missing GT by excluding those rows from averages and returning valid_count.
• Edge cases: empty batch, k=0, duplicates in predictions (handled), GT not present.
• Alternatives: use sparse/batched operations if k very large, or torch tensors for GPU acceleration.

Requirements & constraints:

• Offline rank ~thousands policies using logged contextual bandit data (context, action, reward, propensity), low-latency decisions for promotion, control false positives, manageable compute/storage costs.

High-level approach:

1. Use IPS and Doubly Robust (DR) estimators per policy to estimate expected reward; DR lowers bias by adding a reward model.
2. Compute estimates in a distributed batch pipeline (Spark/Dataproc or Flink for streaming incremental runs) over partitioned parquet logs keyed by date/shard.

IPS/DR at scale:

• Precompute and store (context, action, reward, propensities, policy_scores) where policy_scores are the candidate policy’s action probability for that context (vectorized).
• For each policy p calculate per-row weight w = p(a|x)/π(a|x) (π = logging propensity). IPS = mean(w * r). DR = mean( q̂(x, p(x)) + w * (r - q̂(x,a)) ), where q̂ is a regression model estimating expected reward.
• Implement vectorized map-side joins: broadcast small policy parameter sets (or evaluate policies as UDFs) and compute per-policy aggregates via map-reduce to avoid repeated scans. Use grouping + combiner to emit sum_w_r, sum_w, sum_dr terms.

Controlling variance:

• Propensity clipping/truncation: cap weights at c (e.g., 10) and track effective sample size (ESS = (sum w)^2 / sum w^2).
• Self-normalized IPS: IPS_sn = sum w r / sum w reduces variance/bias tradeoff.
• Use DR with a well-regularized q̂ (trained on separate holdout fold) to reduce variance.
• Stratify evaluation by important covariates (e.g., context buckets) and compute per-stratum estimates then aggregate — reduces heteroskedasticity.
• Bootstrap or jackknife in distributed fashion to get standard errors; track ESS and CI.

Multiple-comparison corrections & ranking:

• Compute standardized z-scores / t-statistics per policy using robust SE from bootstrap.
• Control FDR using Benjamini-Hochberg (BH) across thousands of policies to select candidates with q < α (e.g., 0.05).
• For top-k picks, run hierarchical testing: first BH to control FDR, then closed testing or permutation tests among finalists to guard family-wise error if needed.
• Use effect size threshold and minimum ESS: require uplift > δ and ESS > threshold to avoid promoting spurious winners.

Promotion decision & canarying:

• Only promote policies passing statistical criteria and business constraints (latency, actions diversity).
• Canary strategy: staged rollout (1% traffic -> 5% -> 20%) with pre-defined metrics and sequential A/B tests; treat offline estimate as prior and monitor online DR/IPS with short windows; have rollback automation if online observed uplift < promised or negative.

Compute & storage architecture:

• Storage: append-only partitioned Parquet on cloud object store (S3/GCS), with compacted daily summaries. Store policy metadata/catalog in a metadata DB (Postgres/Glue).
• Compute: Spark cluster with autoscaling for bulk evaluation; use Spark UDFs (PySpark/Scala) to evaluate many policies per pass; for very large policy sets, shard policies and run parallel jobs. Use vectorized libraries (numba/pandas) when evaluating locally.
• Model serving for q̂: stored as serialized artifacts (ONNX/TensorFlow) and loaded in executors or precompute q̂(x,a) in feature pipeline.
• Orchestration: Airflow/Kubernetes for scheduled runs, with caching layer (Redis) for recent aggregates.
• Monitoring & lineage: track job metrics, ESS, CI sizes, and persist evaluation outputs to a results DB. Keep reproducible pipeline with immutability.

Tradeoffs:

• More aggressive clipping reduces variance but biases down winners; DR mitigates this.
• Full permutation/bootstrap is costly but needed for top candidates—use approximate bootstrap (Poisson) or influence functions to scale.

This design yields scalable, statistically sound offline policy ranking with clear promotion rules and production-ready compute/storage choices.

High-level summary: Transformers (large sequence models) offer high capacity to model complex, long-range user behaviors and context-rich sessions; matrix factorization (MF) is lightweight, fast, and easier to maintain. Choose based on product priorities: accuracy/complexity vs latency/cost.

Compare by dimension:

• Inference latency:

  • Transformer: higher single-request latency, especially for long sequences or large models; may require GPUs, batching, or distillation to meet real-time SLAs.
  • MF: very low latency on CPU; simple dot-products or lookups, easy to serve at scale.

• Training cost:

  • Transformer: expensive GPU training, longer iterations, complex hyperparameter tuning; costly offline experiments.
  • MF: cheap to train, often on CPU or modest GPU, fast retraining and iteration.

• Capacity: long-term preferences vs short-term session signals

  • Transformer: excels at modeling sequential patterns, attention over long histories, session dynamics, and multi-modal inputs (metadata, context).
  • MF: captures stable, global affinities (user/item latent factors) well but struggles with short-term temporal signals and order effects unless augmented.

• Maintainability:

  • Transformer: more engineering overhead (feature pipelines, infra for batching, model monitoring, drift handling). More brittle to schema changes.
  • MF: simpler pipelines, explainable factors, easier A/B testing and rollback.

• Product situations:

  • Favor Transformers when personalization needs sequence/context sensitivity (next-track prediction, mood-aware playlists, cold-start with rich session signals) and product tolerates higher cost/latency or can amortize via batching/approximation.
  • Favor MF when needing low-latency, large-scale ranking (home screen recommendations, offline-generated playlists), rapid iteration, low infra cost, or when user tastes are stable.

Hybrid options: distill transformers into compact models, use two-stage systems (MF/prioritized recall → transformer reranker), or augment MF with session features to get best of both worlds. Trade-offs: two-stage reduces latency/cost while preserving sequence-aware accuracy in top-k results.

Requirements & constraints:

• 10M users, 1M items, low-latency (<100–200ms) personalized recommendations, budget-conscious startup.

High-level two-stage architecture:
User Request → Candidate Generation (retrieval) → Feature Enrichment (feature store) → Ranker → Response

Core components:

1. Candidate generation

• Methods: item-based collaborative filters, popularity/time-decay, user embeddings nearest-neighbor.
• Implementation: offline precompute top-N per user (daily) + online session-based heuristics.

2. Indexing (ANN / inverted indices)

• ANN (e.g., FAISS, HNSW) for dense embedding nearest-neighbor; keeps item vectors sharded by category.
• Inverted index (Elasticsearch) for sparse signals, metadata, and keyword filtering.
• Refresh: lightweight nightly rebuilds; incremental updates for new items.

3. Feature store / enrichment

• Online feature store (Redis/Memcached) for low-latency scalar features (user recency, item popularity, last interactions).
• Offline store (BigQuery/S3) for heavy aggregated features and training data.

4. Ranker

• Lightweight: gradient-boosted model (XGBoost/LightGBM) served as microservice for per-request scoring — runs in CPU, low latency.
• Heavyweight: deep neural network (multi-task or attention models) used for offline training and periodic export; distilled or batch-scored to produce features or re-rank top-k.

Data flow & placement:

• Offline pipeline: collect events → ETL → train heavy models → produce embeddings/top-N lists → push to ANN/index + feature store.
• Online flow: request → fetch user id → get candidates from ANN + filters → pull online features from feature store → rank with lightweight model → return top-K.
• Heavy models run offline on GPU clusters; lightweight models run in CPU autoscaled serving (Kubernetes), or as serverless functions for cost-efficiency.

Scalability & trade-offs:

• Pre-filter to limit ANN queries (category, availability).
• Cache popular user results to reduce load.
• Use model distillation and batch scoring to keep online latency low while retaining heavy-model quality.

Monitoring & Observability Plan (Spotify recommendation models)

Overview: instrument metrics at three levels (model, feature/data, business), detect drift (data & concept), set tiered alerts, and define automated + human-in-loop remediation. Use Prometheus/Grafana for infra + metrics, OpenTelemetry for tracing, and ML-specific tools (Evidently, WhyLabs, Great Expectations, Seldon/Feast) for model/data observability. Log predictions, inputs, confidences, and user-session context to a metrics/store.

Key metrics

• Model-level

  • Accuracy proxies: offline A/B lift, online CTR/engagement delta per cohort
  • Calibration: expected vs observed click probability (Brier score, calibration curve)
  • Confidence distribution / entropy
  • Latency: p95/p99 end-to-end inference latency, model load time
  • Resource: CPU/GPU utilization, memory
  • Fairness: per-group CTR/acceptance parity, disparate impact ratio, demographic FPR/FNR

• Feature/data-level

  • Feature distributions: mean, std, quantiles; missingness rate
  • Schema: column types, new/unexpected categories
  • Correlations: feature covariance changes
  • Throughput: requests/sec, batch sizes
  • Data quality: invalid values, timestamps skew

• Business-level

  • Engagement: session length, streams per user, retention uplift vs control
  • Revenue proxies: premium conversions attributable to recs
  • User experience: skip rate, repeat listens, complaints/ticket volume

Drift detection strategies

• Data drift: continuous statistical tests (KS for continuous, Chi-square for categorical) comparing recent window (last 24–72h) vs baseline; population stability index (PSI) per feature; KL divergence for embedding distributions.
• Concept drift: monitor model performance on near-real-time labeled feedback slices (e.g., immediate interactions) and use detectors like ADWIN or Page-Hinkley on rolling AUC/CTR.
• Label delay handling: use proxy labels (clicks/skips) with caution and weight by confidence.
• Use embedding-level drift: cosine similarity between production embedding distributions and training embeddings.

Alerting thresholds (examples, adapt per model)

• Latency: p95 > 200ms OR p99 > 500ms → SEV-1
• Error rate: prediction errors or HTTP 5xx > 0.5% → SEV-1
• Data drift: PSI > 0.25 or KS p-value < 0.01 on critical features → SEV-2
• Concept drift: rolling CTR drop > 5% absolute or AUC drop > 3% over 24h → SEV-1/2 depending business impact
• Fairness: per-group CTR gap > 10% relative or disparate impact < 0.8 → SEV-2, require review
• Model confidence shift: median confidence drop > 20% → SEV-3

Remediation workflows

1. Automated immediate actions

   • Circuit breaker: fallback to safe baseline recommender (legacy ranking or diversified heuristics) when latency/error/failure thresholds hit.
   • Rate-limit or degrade to cached recommendations for high load.
   • Auto-rollback: if new model deployment triggers sharp performance regressions within canary window, rollback to last stable model.

2. Detection → Triage

   • Alert to on-call ML engineer with dashboard links, recent examples, and top contributing features.
   • Auto-attach sample requests & predictions to ticket (with privacy-aware sampling).

3. Investigation steps (engineer)

   • Verify instrumentation and data integrity (schema, timestamps).
   • Recompute drift tests on different time windows; examine correlated infra incidents.
   • Slice by cohort/geography/device to localize issue.
   • Re-run inference on historical data to check determinism.

4. Remediation actions

   • Quick fixes: re-trigger feature pipeline, rehydrate feature store, fix ETL bug, clear model cache.
   • Retrain: if drift confirmed, kick off automated retrain on latest labeled window with validation and canary rollout.
   • Model patch: apply calibration, update thresholds, or introduce safety constraints (diversification).
   • Fairness mitigation: adjust weighting or rerank to restore parity; involve ethics/PM for policy decisions.

5. Postmortem

   • Root cause analysis, update playbook, add new tests/monitors, and adjust thresholds.

Best practices

• Use correlated signals (metrics + logs + traces + sampled requests) to reduce false positives.
• Tier alerts by business impact and require actionability.
• Maintain an audit trail for automated rollbacks and human approvals.
• Periodically simulate drift scenarios and runbook drills.
• Business stakeholders define acceptable degradation tolerances (SLOs) and fairness SLIs.