Experiment Prioritization & Roadmap Questions

Problem formulation

• Finite population N, T total exposure slots (or continuous arrival rate), K experiments queued. Each experiment i has unknown treatment effects θi (could be multi-armed) and a prior p(θi). At each time t you allocate a user to either: run an adaptive bandit inside some experiment i (continuously adapt assignment within i), or run a fixed A/B test for experiment i for a fixed horizon hi (non-adaptive allocation then analyze). Objective: minimize cumulative regret across all users (sum over t of lost reward vs oracle) or equivalently maximize expected information gain (reduce posterior entropy) weighted by downstream impact of decisions.

Mathematical objective (Bayesian)
Minimize E[∑_{t=1}^T (r* - r_t)] where r* is reward under best true policy; decisions influence posteriors. Equivalent: maximize expected value of decisions after experiments (VOI).

Solution approach (principled -> practical)

1. Bayes-optimal DP (theory): state = posteriors {p_t(θi)} and remaining user budget. Bellman recursion gives optimal allocation but is intractable (continuous high-dim state).

2. Index/Restless-bandit approximation:

• Treat each experiment as a restless bandit; derive Whittle index (when relaxable) representing marginal value of allocating one more user to experiment i given its posterior and remaining budget for that experiment. Allocate users to highest index. This approximates optimal policy and handles switching between adaptive bandit inside an experiment vs finishing as fixed A/B.

3. Meta-bandit / Thompson-scheduling heuristic (practical, scalable):

• For each experiment i, maintain Bayesian posterior. For each candidate action (run adaptive bandit step in i vs run a fixed-batch of size b for i), simulate one-step lookahead:
  • Estimate expected marginal utility = expected reduction in regret or increase in expected final decision value after allocating b users, per user.
• Use Thompson sampling at meta-level: sample θi from each posterior, compute which experiment/action would yield highest immediate reward or VOI, allocate next user (or batch) accordingly.
• Or use greedy myopic VOI: compute closed-form VOI for simple models (Bernoulli rewards) and allocate to highest VOI-per-user.

Implementation sketch (heuristic)

• Model rewards per arm with conjugate priors (Beta-Bernoulli or Gaussian-Normal).
• For each timestep:
  • For each experiment i, compute metrics:
    • Expected immediate improvement from adaptive assignment (e.g., TS expected reward gain).
    • VOI of running a fixed A/B batch of size b (expected reduction in decision error).
    • Normalize by users consumed.
  • Allocate next user (or mini-batch) to experiment/action with max metric.
• Periodically re-evaluate; stop experiments when posterior confidence crosses decision threshold.

Evaluation & trade-offs

• Simulate under realistic priors to tune batch sizes and thresholds.
• Complexity: DP exponential; Whittle index and meta-bandit heuristics scale O(K) per step.
• Edge cases: strongly time-limited horizons favor fixed A/B for rapid unbiased estimates; tiny effect sizes favor pooling budget for adaptive methods to reduce regret; interference across experiments (shared users) requires cohort tracking.

Takeaways

• Exact optimum via DP is intractable; good practical policies use index/Whittle approximations or meta-bandit/VOI heuristics with Bayesian posteriors and Thompson-sampling-style exploration. Validate by simulation and encode business utility into VOI to balance regret vs information.

Approach: for each incoming window (control_events, control_trials, treat_events, treat_trials) compute a two-sample z-test for difference in proportions (using pooled variance under H0). To control Type I when repeatedly peeking, apply a Pocock-style alpha-spending function that allocates cumulative alpha by window index; at each look convert the incremental alpha to a one-sided (or two-sided) critical p-value threshold and decide. Stop when min_samples satisfied and p <= threshold (reject) or when stream ends or max_windows reached (continue / no effect).

import math
import numpy as np
from typing import Iterator, Tuple, Dict

def pocock_spent_alpha(window_idx: int, max_windows: int, alpha: float) -> float:
    # Pocock-like spending: monotone concave spending (approximation via log)
    t = min(window_idx, max_windows)
    return alpha * math.log(1 + (math.e - 1) * t / max_windows) / math.log(math.e)

def two_sample_z_pvalue(c_ev, c_tr, t_ev, t_tr) -> float:
    # pooled proportion under H0
    p_pool = (c_ev + t_ev) / (c_tr + t_tr) if (c_tr + t_tr) > 0 else 0.0
    se = math.sqrt(p_pool * (1 - p_pool) * (1.0 / c_tr + 1.0 / t_tr))
    if se == 0:
        return 1.0
    diff = (t_ev / t_tr) - (c_ev / c_tr)
    z = diff / se
    # two-sided p-value
    return 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))

def evaluate_stream(stream_iter: Iterator[Tuple[int,int,int,int]],
                    alpha: float = 0.05,
                    min_samples: int = 100,
                    max_windows: int = 1000) -> Dict:
    window = 0
    for (c_ev, c_tr, t_ev, t_tr) in stream_iter:
        window += 1
        if window > max_windows:
            break
        total_samples = c_tr + t_tr
        p_val = two_sample_z_pvalue(c_ev, c_tr, t_ev, t_tr)
        if total_samples < min_samples:
            decision = "insufficient_samples"
            # continue to next look but we may still report current p-value
        else:
            alpha_spent = pocock_spent_alpha(window, max_windows, alpha)
            # convert cumulative spent alpha to a per-look threshold: using cumulative spending,
            # we require p_val <= alpha_spent to reject at this look
            if p_val <= alpha_spent:
                return {"decision": "reject", "p_value": p_val, "window_index": window}
            decision = "continue"
    return {"decision": decision, "p_value": p_val if 'p_val' in locals() else None, "window_index": window}

Key points:

• Uses pooled-variance z-test for conversion differences (H0: p_treat = p_control).
• Pocock spending implemented as a log-shaped cumulative spend approximation; at look t we compare p-value to cumulative alpha spent so far.
  Time/space: O(W) time (W windows), O(1) memory.

Assumptions:

• Windows are independent aggregates, counts are non-negative integers.
• Two-sided test for difference in proportions; pooled variance is acceptable for large counts.
• max_windows is known/upper bound for spending schedule.

Edge cases:

• Zero trials in either group -> return no decision (p=1).
• Very small counts: normal approximation invalid — prefer exact/binomial or require larger min_samples.
• Stream shorter than expected: spending never exhausts alpha; final decision is "continue" unless rejected earlier.

Requirements:

• Experiment definitions as code (YAML/JSON + Python training script) stored in Git.
• CI pipelines run experiments reproducibly, validate data/model, register artifacts, and deploy with safe rollouts and rollback.
• Automated metric-regression detection and observability.

High-level architecture:
Git (experiments-as-code) → CI/CD (GitHub Actions / GitLab CI / Jenkins) → Containerized training (Docker/Kubernetes) + Feature store / data snapshots → Model evaluation & validation → Model Registry (MLflow/Feast/SageMaker Model Registry) → Deployment (K8s + Seldon/TF-Serving) + Feature flags (LaunchDarkly/FF4J) → Monitoring & rollback.

Core components & flow:

1. Experiment-as-code: repo holds experiment spec (hyperparams, dataset refs, pipeline DAG). Each PR triggers pipeline.
2. CI pipeline stages:
   • Unit tests: small tests for preprocessing, model utils, and deterministic components (fast).
   • Data-quality checks: run Great Expectations or Deequ on sampled data referenced by spec; validate schema, nulls, label distribution drift.
   • Model-sanity tests: smoke-train on toy subset to ensure code runs, deterministic seeds, check loss decreases and basic metric thresholds.
   • Full training (if PR to main or scheduled): run in ephemeral scaled infra (K8s/Spot GPUs) using Docker images + reproducible artifact hashing.
   • Evaluation & regression detection: compute metrics vs baseline from model registry; run statistical tests (e.g., bootstrap confidence intervals) and rule-based checks (no >X% degradation, fairness metrics).
3. Automated metric-regression detection:
   • Define alert rules in pipeline config. On detection: fail pipeline and create issue/notify owners.
   • Use MLflow to compare metrics; CI calculates delta and gates merge.
4. Model registry integration:
   • On passing validations, artifacts (model binary, scoring code, metrics, data snapshot, lineage) are logged to registry with semantic version and tags (canary-ready).
5. Deployment & rollout:
   • Canary/feature-flag rollout: deploy model to canary subset via Kubernetes + service mesh routing or Seldon + percentage traffic rules. Use feature flags to enable model for specific users.
   • Run live A/B tests, collect metrics (latency, business KPIs, ML metrics); traffic ramp controlled by pipeline or orchestration.
6. Safe rollback:
   • Automate rollback when monitoring detects metric drift/regression beyond thresholds or errors: orchestration switches traffic back to previous model version immediately; registry marks version as deprecated.
   • Maintain immutable artifacts and keep previous model images available for instant redeploy.
7. Observability & governance:
   • Monitoring: Prometheus/Grafana for infra, custom metrics export for ML KPIs; use alerts & automated remediation hooks.
   • Lineage and reproducibility: store dataset hashes, code commit, container image digest, and hyperparams.
   • Access controls and approvals: require signed approvals for production deploys; require human review on critical metric regressions.

Scalability & trade-offs:

• Use sample-based quick checks in CI to keep feedback fast; full-scale training runs gated to main or scheduled pipelines.
• Trade-offs between blocking merges on strict checks vs. developer velocity—employ “soft gates” (warnings + required review) for non-critical metrics.
• Cost: prefer spot instances for heavy training; cache datasets and use incremental training where possible.

This design enforces experiment-as-code, reproducible pipelines, automated validation and regression detection, safe canary rollouts controlled by feature flags, and immediate rollback paths—balancing safety and developer productivity.

Type I error (false positive): declaring the treatment (new ML-driven change) better when it’s actually not — probability = alpha (commonly 0.05). Type II error (false negative): failing to detect a real improvement — probability = beta; power = 1 − beta.

Lowering alpha reduces the chance of false positives but requires more evidence to call a result significant. That increases required sample size or test duration for the same power, so experiments run slower. If you don't increase sample size, power falls and beta rises — you risk missing true effects.

Concrete trade-off example (conversion optimization):

• Baseline conversion 10%. You aim to detect a 1% absolute lift (to 11%) with 80% power.
• With alpha = 0.05 you need ~40,000 total users.
• If you lower alpha to 0.01 to be more conservative, sample size rises to ~55,000 — ~37% longer test.
• If you keep the original 40,000 sample but change alpha to 0.01, power drops (beta increases), maybe to ~60%, so you’d often miss the true 1% lift.

Practical guidance: choose alpha based on business cost of false positives vs. false negatives. For high-risk changes (billing, safety), use low alpha and accept longer tests. For rapid product improvements where missing small gains is costly, accept higher alpha or run sequential testing with proper corrections to balance speed and risk.

Clarify objective & constraints:

• Goal: explain offline +8% revenue vs. 0% in A/B—identify whether data, model, serving, or experiment issues.
• Constraints: deployment cadence, logging retention, ability to run shadow traffic.

Prioritized investigation plan (ordered by likely payoff):

1. Quick sanity checks (fast, low-cost)

• Verify experiment setup: correct hypothesis, metric definitions, guardrail metrics, and power/sample-size.
• Confirm randomization and sample checks: treatment/control traffic proportions, cookie/user assignment stability, no leakage between buckets, and identical eligibility filters.

2. Offline-vs-online metric mismatch

• Compare exact metric computation: offline revenue proxy vs. production revenue (time windows, attribution rules, price adjustments, refunds).
• Recompute offline metric using production aggregation code on the holdout/test logs to spot differences.

3. Data preprocessing & feature checks

• Re-run preprocessing used in training against production feature pipeline outputs (scale, impute, categorical mappings).
• Check missingness rates and distributions for key features; ensure same feature engineering code/version deployed.

4. Feature drift & distributional checks

• Compute per-feature drift (KL/divergence, population stability index) between training, offline eval, and live traffic. Flag large shifts.
• Investigate label distribution shift (conversion rates) and covariate shift for top predictive features.

5. Logging / serving differences

• Validate model input parity: log raw inputs, transformed features, and model scores for a sample of live requests.
• Ensure model version, weights, and preprocessing code match what's evaluated offline; check for serialization/deserialization bugs, rounding, or GPU/FP precision issues.

6. Experiment instrumentation & edge cases

• Ensure deterministic tie-breaking, business rules, and fallback logic are identical between offline scoring and production.
• Check for latency/timeouts causing fallback to baseline model.

7. Targeted experiments to isolate cause

• Shadow deploy: run new model in parallel, log predictions and downstream outcomes without affecting decisions; compare to offline expectations.
• Canary / deterministic routing: route a small deterministic user subset to model and run intensive logging.
• Replay offline: replay production request logs through the exact offline pipeline to reproduce expected uplift.
• A/B test variants: test with simplified model (e.g., only top features) to see if complexity interacts with production behavior.

8. Root-cause analysis & remediation

• If drift/preprocessing mismatch → fix pipeline, retrain or calibrate model, and re-deploy.
• If metric mismatch → align evaluation metric and attribution.
• If logging/serving bug → patch serialization or inference code, re-run shadow tests.

Metrics & success criteria:

• Convergence between offline and replayed online metric within noise bounds.
• Shadow deploy shows predicted uplift before turning on treatment.
• Clear logging for reproducibility and a postmortem documenting fixes and safeguards.

Communicate plan, timelines, and roll-back strategy; prioritize reproducible logging and small, reversible experiments.