Recommendation, Ranking, and Personalization Questions

Clarify objective & constraints: Are we optimizing short-term clicks, long-term retention/ARPU, or multi-step user satisfaction? Is exploration allowed (A/B, randomized)? What latency/scale and safety constraints exist?

Comparison — high level

• Supervised learning (predict clicks):

  • Approach: Train a model to predict immediate outcomes (clicks, CTR) from historical features; serve top predicted items.
  • Strengths: Simple data pipeline (logged context→label), fast to iterate, stable offline metrics (AUC, calibration), easy to deploy and explain.
  • Weaknesses: Myopic — optimizes one-step reward; suffers from selection bias (logs reflect previous policy) and confounding; can amplify feedback loops.

• Bandits / Reinforcement Learning:

  • Approach: Formulate as contextual bandit for per-impression exploration-exploitation, or full RL for sequential long-term rewards (engagement, LTV).
  • Strengths: Explicitly trades exploration and exploitation, can optimize long-term metrics, handles delayed rewards.
  • Weaknesses: Higher data/sample complexity, needs careful exploration strategy, harder to evaluate offline, more brittle in production.

Data requirements

• Supervised: Large logged datasets with context and immediate labels; needs propensity-aware logging if you want unbiased estimates.
• Bandit/RL: Requires logged actions with exploration (randomized or logged propensities) or online experiments; RL needs trajectories and longer horizon labels; more diverse coverage of state-action space.

Evaluation methods

• Supervised: Holdout test, calibration, counterfactual evaluation via inverse propensity scoring if using logged policy correction; online A/B tests measuring CTR and downstream metrics.
• Bandit/RL: Offline policy evaluation (IPS, Doubly Robust) with high variance unless propensities are well recorded; simulate with logged simulators; small-scale online experiments with conservative exploration (epsilon-greedy, Thompson Sampling) and safe policy constraints.

Production risks & mitigations

• Supervised risks: Feedback loops (popular items get more data), distribution shift, measuring wrong proxy. Mitigate with propensity-weighting, diversity/novelty constraints, regularization, and monitoring.
• Bandit/RL risks: Unsafe exploration harming UX, delayed reward credit assignment, policy collapse, engineering complexity. Mitigate with constrained bandits, conservative policy updates, simulators/sandbox, strong offline evaluation, rollout ramps and kill-switches.

Recommendation (practical path)

1. Start with supervised models for stable baseline and fast iteration.
2. Instrument policy to log propensities and richer trajectories.
3. Gradually introduce contextual bandits for controlled exploration to improve long-term metrics.
4. Only move to full RL when you have reliable simulators, robust offline evaluation, and business tolerance for exploration — and keep human-in-the-loop safety guards.

This staged approach balances speed, safety, and the ability to optimize long-term engagement.

Compute cosine similarity by normalizing vectors (with an epsilon to avoid divide-by-zero), then do a single matrix-vector dot to get all similarities. Use np.argpartition to get top-k in O(n) time for selection then sort the top-k. Time complexity: O(n_items * d). Space: O(n_items) for similarity scores.

import numpy as np

def knn_bruteforce(item_embs: np.ndarray, user_emb: np.ndarray, k: int):
    """
    Brute-force k-NN by cosine similarity.
    item_embs: shape (n_items, d)
    user_emb: shape (d,)
    k: number of top items to return
    Returns: numpy array of top-k indices sorted by descending similarity
    """
    if k <= 0:
        return np.array([], dtype=int)
    if item_embs.ndim != 2:
        raise ValueError("item_embs must be 2-D (n_items x d)")
    if user_emb.ndim != 1:
        raise ValueError("user_emb must be 1-D (d,)")

    n, d = item_embs.shape
    if user_emb.shape[0] != d:
        raise ValueError("Dimension mismatch between item_embs and user_emb")

    k = min(k, n)

    # Convert to float64 for numerical stability if not already
    item_embs = item_embs.astype(np.float64, copy=False)
    user_emb = user_emb.astype(np.float64, copy=False)

    # Safe L2 norms with epsilon to avoid division by zero
    eps = 1e-12
    item_norms = np.linalg.norm(item_embs, axis=1)
    item_norms = np.maximum(item_norms, eps)
    user_norm = max(np.linalg.norm(user_emb), eps)

    # Normalize embeddings (broadcasting)
    item_embs_normalized = item_embs / item_norms[:, None]
    user_emb_normalized = user_emb / user_norm

    # Compute cosine similarities (vectorized)
    sims = item_embs_normalized.dot(user_emb_normalized)  # shape (n,)

    # Get top-k indices efficiently
    if k == n:
        topk_idx = np.argsort(-sims)
    else:
        part = np.argpartition(-sims, k-1)[:k]          # unsorted top-k
        topk_idx = part[np.argsort(-sims[part])]       # sort top-k by descending sim

    return topk_idx

Key points:

• Numerical stability: use eps for norms and float64 for accumulation.
• Complexity: Time O(n*d) for normalization + dot; Space O(n) for sims and normalized copy.
• Edge cases: k>n handled, zero vectors handled via eps, and invalid shapes raise errors. Alternative: compute similarities without explicit normalized copy by dividing dot results by norms product to save memory.

Inverse propensity scoring (IPS) is a counterfactual estimator that uses logged bandit feedback to estimate the expected reward of a new (evaluation) ranking/policy using data collected under a logging policy. For each logged impression i where the logging policy chose action a_i and observed reward r_i, IPS weights the observed reward by the inverse of the logging policy probability p_i = P_logging(a_i | context_i):
Ê_IPS = (1/N) * sum_i ( r_i * π_eval(a_i | context_i) / p_i )
For ranking, π_eval(a_i|context) is the probability the evaluation policy would have selected the same action (or slot-aware extensions for lists).

Key assumptions:

• Unconfoundedness / no hidden confounders: logging action is independent of potential rewards given observed context.
• Support / overlap: p_i > 0 whenever π_eval would place positive mass — otherwise unbiased estimation is impossible.

Variance issues:

• Large or tiny propensities lead to huge importance weights, producing high variance and unstable estimates. Rare actions in logs are especially problematic for complex ranking spaces.

Practical adjustments:

• Clipping (weight truncation): replace weight w = π_eval/p with min(w, C) to bound variance at cost of bias.
• Self-normalized IPS (SNIPS): normalize by sum of weights: Ê_SNIPS = sum(r_i w_i) / sum(w_i), which reduces variance and corrects for scale but introduces small bias.
• Doubly Robust (DR) estimators: combine IPS with a reward model to reduce bias when model is good and reduce variance.
• Propensity estimation care: calibrate logging propensities, regularize small p’s, and use stratification or per-slot propensities for lists.
• Practical tips: report confidence intervals via bootstrap, evaluate on multiple clipping thresholds, restrict evaluation to actions with sufficient support, and use DR + SNIPS for best bias-variance tradeoff.

Requirements & constraints:

• Objective: move from A/B to per-user treatment to maximize long-term reward (e.g., revenue, engagement) while controlling risk.
• Constraints: sample size, delayed/partial rewards, privacy, latency, safety limits (max negative impact), regulatory.

High-level path:

1. Validate HTE from A/B:

   • Split-randomized trial analysis: estimate conditional ATEs by segment; check overlap, balance, and power.
   • Use causal trees/forests (e.g., GRF) to discover interpretable HTEs and feature importance.

2. Build uplift models (offline):

   • Candidate algorithms: S-/T-/X-learners, causal forests, doubly-robust learners, and neural uplift nets (Siamese/DR-loss).
   • Train using randomized assignment data; include propensity estimates to reduce bias.
   • Evaluate with uplift-specific metrics (Qini, uplift at k), and use cross-fitting/double ML to control confounding.

3. Or build contextual bandit (online policy learning):

   • Start with a logged-policy dataset (RCT logs) to warm-start; consider LinUCB/Thompson Sampling or policy-gradient bandits for large contexts.
   • Use model-based reward estimators (e.g., outcome model + uncertainty) for Bayesian exploration.

4. Offline policy evaluation (OPE):

   • Use Importance Sampling (IPS), Self-Normalized IPS (SNIPS), Doubly Robust (DR) estimators; consider MAGIC/MRV for variance reduction.
   • Compute confidence intervals via bootstrap or concentration bounds; penalize policies with high variance or low effective sample size.

5. Safe production rollout:

   • Conservative policy: optimize a constrained objective (maximize expected uplift subject to worst-case/CI constraint).
   • Exploration budget: allocate a small ε of traffic to exploration or run a safe contextual bandit with bounded optimism (KL/relative entropy constraints).
   • Safety constraints: hard guardrails (minimum baseline performance), monotonicity checks, and policy sandboxing (canary cohorts).
   • Monitoring: real-time logging of contexts, actions, rewards; automated drift detection, causal metric monitoring, and backstop rollback triggers.
   • Retraining cadence: continuous learning with feature-store snapshots, periodic offline OPE to approve new policies.

Trade-offs:

• Uplift models give quick, deterministic personalization but require stationary environment; bandits adapt online but need exploration and careful safety design.
• Use hybrid: deploy uplift-policy as baseline, run constrained bandit on small traffic to adapt and validate improvements, then promote.

Example implementation notes:

• Store full RCT logs with propensities, features, timestamps.
• Use DR for OPE, threshold by effective sample size > N_min.
• Enforce a production policy that never assigns risky treatment to protected groups beyond RCT proportions.

This path balances causal validity, offline vetting, and safe online adaptation for scalable personalization.

To implement a contextual bandit with Thompson Sampling for Bernoulli rewards, we maintain per-arm Beta(a, b) posteriors (alpha = successes + 1, beta = failures + 1). On each decision, sample theta_i ~ Beta(alpha_i, beta_i) for every arm and pick the arm with highest sampled theta. After observing binary reward r in {0,1}, update selected arm: alpha += r, beta += 1 - r.

import numpy as np

class ThompsonBernoulli:
    def __init__(self, n_arms, prior_alpha=1.0, prior_beta=1.0):
        # Per-arm posterior parameters (alpha = successes + prior_alpha)
        self.n = n_arms
        self.alpha = np.full(n_arms, prior_alpha, dtype=float)
        self.beta = np.full(n_arms, prior_beta, dtype=float)

    def select_arm(self):
        # Sample theta_i ~ Beta(alpha_i, beta_i) for each arm and return argmax
        samples = np.random.beta(self.alpha, self.beta)
        return int(np.argmax(samples)), samples

    def update(self, arm, reward):
        # reward must be 0 or 1
        if reward not in (0, 1):
            raise ValueError("Reward must be 0 or 1 for Bernoulli bandit.")
        self.alpha[arm] += reward
        self.beta[arm] += (1 - reward)

Key points:

• State per arm: two floats (alpha, beta).
• Time complexity per decision: O(K) to sample K Betas and find argmax.
• Space: O(K) for storing alpha and beta.
• Works online, naturally balances exploration/exploitation.
• Edge cases: ensure numeric stability for large counts (use log or float64), handle ties by argmax deterministic choice or random tie-break. Alternative: use approximate sampling for many arms or vectorized sampling for speed.