Experimentation Platforms and Infrastructure Questions

Direct answer

A compact experiment configuration needs the pieces both the SDK and offline analysis depend on: an id, the variants and their allocation percentages, targeting rules, a rollout schedule (which phase is currently live), the metric definitions this experiment is measuring, and a kill switch, structured so an SDK can evaluate it in milliseconds and an analysis job can reconstruct exactly what configuration was live at any point in time.

Structured elaboration

{
  "experiment_id": "checkout_redesign_v2",
  "status": "running",
  "variants": [
    { "key": "control", "allocation": 0.5 },
    { "key": "treatment", "allocation": 0.5 }
  ],
  "targeting": { "country_in": ["US", "CA"], "app_version_gte": "4.2.0" },
  "rollout_phases": [
    { "phase": 1, "traffic_pct": 5, "started_at": "2026-06-01T00:00:00Z" },
    { "phase": 2, "traffic_pct": 50, "started_at": "2026-06-05T00:00:00Z" }
  ],
  "metrics": { "primary": "checkout_completion_rate", "guardrails": ["payment_failure_rate", "page_load_p95"] },
  "kill_switch": false,
  "config_version": 7
}

The SDK only strictly needs targeting, variants/allocations, the current rollout phase's traffic_pct, and kill_switch, since those are the fields that determine "does this user get evaluated at all, and into which bucket," and it needs them cached locally with low latency, since a request-time fetch to a central config store for every evaluation would be too slow at scale. Offline analysis additionally needs the metrics block (to know what to compute), the full rollout_phases history (to know what config was live at any past moment, not just now), and config_version (to detect that a mid-experiment configuration change happened, which the analysis has to account for or exclude).

Worked example

Six weeks into an experiment, someone asks "was this really running at 50% the whole time?" Because rollout_phases records each phase with its start timestamp rather than overwriting a single current_traffic_pct field, the answer is a direct lookup: 5% for the first four days, 50% after, which lets the analysis correctly exclude or separately model the low-traffic ramp period rather than assuming a constant allocation throughout.

Trade-offs and pitfalls

Making the schema too rich (embedding full statistical analysis parameters, verbose free-text descriptions) bloats what has to be fetched and cached by every SDK evaluation, most of which never needs those fields. Making it too thin (no config_version, no rollout_phases history, just a single current allocation) is cheap for the SDK but breaks reproducibility for analysis, since a config that only tracks "what's true now" cannot answer "what was true when this user was actually exposed."

Direct answer

Build four automated checks: a sample-size/power check comparing accumulated traffic against the planned minimum-detectable-effect target, an exposure-imbalance (SRM) test, a metric-instrumentation-drift check comparing current event volume against a recent historical baseline, and an SDK error-rate check, each computed in a scheduled daily job with a concrete alert threshold and a defined corrective action.

Structured elaboration

1. Sample-size/power check (pseudocode):

current_n = count(exposed_users)
required_n = precomputed_required_sample_size(mde, baseline_rate, alpha, power)
if current_n < required_n * 0.5 AND days_elapsed > planned_duration * 0.7:
    alert("underpowered: unlikely to reach target sample size on schedule")

2. Exposure-imbalance (SRM) test (SQL sketch):

SELECT variant, COUNT(DISTINCT user_id) AS n
FROM exposures WHERE experiment_id = :exp_id
GROUP BY variant;
-- feed observed counts into a chi-square goodness-of-fit test against the
-- configured allocation; alert if p < 0.001

3. Metric-instrumentation-drift check: compare today's event volume per event type against the trailing 7-day median, flagging a drop of more than roughly 20% (a threshold tuned per event type's normal day-to-day variance) as a likely instrumentation break rather than a real behavior change.
4. SDK error-rate check: track the error rate reported by each SDK version calling the assignment or exposure-logging endpoints, alerting when any single version's error rate rises meaningfully above its own recent baseline, which isolates version-specific bugs rather than being swamped by normal background error noise.

Corrective actions: an underpowered flag should prompt either extending duration or explicitly labeling the eventual read-out as exploratory; an SRM alert should pause new exposure until root-caused; an instrumentation-drift alert should pause metric reporting (not the experiment itself) until the drift is understood, since the experiment may still be running fine while only the measurement is broken; an SDK error-rate spike should page the owning team for that specific client version.

Worked example

A daily job finds that exposure event volume for one client platform dropped 35% versus its 7-day median, while the other platform's volume is stable. Rather than a real behavior change (which would show up gradually, not as a sudden day-over-day cliff on one platform only), this pattern strongly suggests an instrumentation break specific to that platform's latest release, and the automated action is to flag metric reporting for that experiment as unreliable pending investigation, rather than letting a broken measurement silently continue to feed the dashboard.

Trade-offs and pitfalls

Setting drift thresholds too tight on inherently noisy event types (ones with genuine day-of-week seasonality) produces frequent false alarms; setting them too loose misses real breaks for days. The practical fix is tuning each threshold to that specific event type's own historical variance rather than using one global percentage across every metric, and separating weekday from weekend baselines where seasonality is strong enough to matter.

Direct answer

Batch computation is the right default when you can tolerate metrics being hours old in exchange for full accuracy and simpler correctness guarantees; streaming is worth its extra complexity specifically when a guardrail needs to detect harm within minutes, not hours; a practical hybrid uses streaming for a small set of guardrail metrics that gate automated safety actions, and batch for everything else, including the authoritative primary-metric read-out.

Structured elaboration

• When batch wins: batch jobs can do a full, exact join over complete data (no late-arrival ambiguity, since by the time the batch runs, the data for that period is settled), are simpler to reason about and test, and are the natural fit for the final, authoritative analysis a ship decision rests on.
• When streaming wins: guardrail metrics that need to trigger an automated pause within minutes of a real regression starting cannot wait for a nightly batch job; streaming is the only way to catch harm fast enough to matter for those specific metrics.
• Hybrid design: run a small, curated set of guardrail metrics on a streaming path (accepting some approximation and a bounded late-arrival window), and run the full primary and secondary metric suite on a nightly batch job that reconciles against the complete, settled data. The streaming guardrail numbers are explicitly labeled as provisional/approximate; the batch numbers are the authoritative record.
• Why not stream everything: streaming every single metric adds meaningful engineering complexity (state management, exactly-once semantics, watermarking) for metrics where a few hours of latency genuinely doesn't matter, which is most of them; reserving streaming for the metrics that specifically need it keeps the system simpler overall.

Worked example

A payments-adjacent experiment needs its payment-failure-rate guardrail checked continuously (streaming), since a real regression there should trigger an automatic pause within minutes, not after tomorrow's batch job runs; the same experiment's secondary engagement metrics (time on page, scroll depth) are perfectly well served by the nightly batch, since nobody needs those to update in real time and the batch path is simpler to build and trust.

Trade-offs and pitfalls

The temptation to build everything on the streaming path "since we have the infrastructure anyway" leads to unnecessary complexity for metrics that never needed sub-hour latency; conversely, under-investing in streaming entirely and relying purely on batch for guardrails means a real regression can run for hours before anyone notices, which is a real cost for high-stakes surfaces. The right call is metric-by-metric, driven by how costly a delayed detection actually is for that specific metric, not an all-or-nothing architectural choice.

Direct answer

Before a new metric-computation pipeline ships, run unit tests on the individual transformation logic, integration tests against a realistic end-to-end sample of raw events, golden-file regression tests that lock in known-correct output for a fixed input, and a live A/B smoke test comparing the new pipeline's output against the existing production pipeline on real (but not yet trusted) data before fully cutting over.

Structured elaboration

• Unit tests: verify individual pieces in isolation (the deduplication logic, the attribution-window filter, the aggregation function) against small, hand-constructed inputs with known expected outputs, which is where a logic bug (like an off-by-one in the attribution window) is cheapest to catch.
• Integration tests: run the full pipeline against a realistic synthetic dataset that includes the messy edge cases production data actually has (duplicate events, out-of-order arrivals, users exposed to multiple experiments), verifying the end-to-end output rather than any one component.
• Golden files: a fixed, versioned input dataset paired with its known-correct output, re-run on every pipeline change; if the new code produces a different result on the same input, that's either an intentional, reviewed change or a regression, and the test forces someone to explicitly acknowledge which.
• Live smoke tests: before fully replacing the existing pipeline, run the new one in parallel on real production data and diff its output against the existing pipeline's numbers for a subset of experiments, since a synthetic test dataset, no matter how careful, can miss a real-world data shape nobody thought to construct by hand.

Each layer catches something the others miss: unit tests catch logic bugs cheaply but can't catch an integration mismatch between components; integration tests catch component-interaction bugs but use synthetic data that might not reflect real-world messiness; golden files catch unintentional regressions but only for the specific cases already captured; live smoke tests catch anything that's specific to real production data's actual shape, at the cost of only running after the code is otherwise believed correct.

Worked example

A new metric pipeline passes every unit and integration test cleanly, since the synthetic test data was constructed with clean, well-ordered events. In the live smoke test, its output for one experiment diverges meaningfully from the existing pipeline's, tracing back to a real production pattern the synthetic data never included: a small fraction of users whose exposure event arrives up to 40 minutes after their first conversion event, due to a specific mobile client's batched event upload behavior. This ordering violates an assumption ("exposure always precedes the outcome event") the new pipeline's join logic quietly depended on, which no amount of hand-constructed synthetic testing surfaced.

Trade-offs and pitfalls

Skipping the live smoke-test stage to move faster is the most common shortcut, and it's exactly the stage that catches real-world data shapes nobody anticipated; synthetic tests, however thorough, are bounded by what the test author thought to include. The cost of the smoke-test stage is running two pipelines in parallel for a period, which is real infrastructure overhead, but it's the only layer that validates against ACTUAL data rather than someone's mental model of what the data looks like.

Direct answer

A bandit's adaptive allocation is exactly what makes clean causal inference hard afterward, because later arms are chosen in response to earlier observed rewards, so a naive comparison of average outcomes by arm is biased; preserve the ability to run a genuinely clean A/B test by holding out a small, fixed, non-adaptive control slice of traffic that the bandit never touches, and use estimators specifically designed for adaptively-collected data (like inverse-propensity weighting) for the rest.

Structured elaboration

• Why naive analysis is biased: if the bandit shifts more traffic toward an arm precisely because it looked good early (possibly due to noise), a plain average of outcomes for that arm is contaminated by the same information that caused it to receive more traffic in the first place, inflating its apparent performance relative to a truly randomized comparison.
• Architecture: partition traffic into two slices, a small fixed-percentage holdout that's always randomized uniformly across arms regardless of what the bandit is learning, and the remainder, which the bandit manages adaptively. The holdout gives you a clean, classical A/B-style comparison at any time; the adaptive slice gives you the efficiency benefit of the bandit for the bulk of traffic.
• Data logging: for the adaptive slice, record the actual selection PROBABILITY the bandit assigned to the chosen arm at the time of each decision, not just which arm was chosen, since that propensity is what an inverse-propensity-weighted estimator needs to correct for the non-uniform, response-driven allocation.
• Randomization service: needs to serve both a "just give me the bandit's current best choice" mode for the adaptive slice and a "give me a uniformly random arm regardless of current beliefs" mode for the holdout, from the same underlying arm set, so the two slices are genuinely comparable.
• Analyzing without adaptivity bias: use the holdout for the primary causal claim whenever possible (it's simplest and cleanest), and reserve propensity-weighted estimators for the adaptive slice specifically when you need the extra statistical power that including it provides.

Worked example

A bandit shifts 70% of traffic to Arm B within the first week because early noisy rewards happened to favor it. A naive comparison of average reward by arm over the full period makes Arm B look strong partly because of when it was shown, not just how good it is. The fixed 10% holdout, randomized uniformly the whole time, shows a much smaller (and, in this case, statistically inconclusive) difference between A and B, revealing that some of the apparent gap in the adaptive slice was an artifact of the bandit's own allocation dynamics rather than a true difference in quality.

Trade-offs and pitfalls

Carving out a fixed holdout costs some of the efficiency gain a bandit is meant to provide, since that slice never benefits from adapting away from a worse arm. Skipping the holdout entirely to maximize efficiency is the common mistake, and it leaves the team with no clean way to answer "was this real" when a stakeholder eventually asks, which is a strategic tax paid later for savings gained now.