Experimentation Platforms and Infrastructure Questions

Direct answer

The multiple comparisons problem is that the more experiments and metrics you test, the more likely at least one shows a "significant" result purely by chance, even if nothing real is happening anywhere. Two engineering controls, beyond a statistical correction, are: capping how many metrics any single experiment can designate as "primary" (rather than letting a team declare a dozen primary metrics and cherry-pick whichever moved), and surfacing, alongside every significant result, how many total tests were run in that same batch or time window, so a viewer has the context to judge how surprising a single "win" really is.

Structured elaboration

Statistical corrections (Bonferroni, Benjamini-Hochberg) adjust the math after the fact; engineering controls change the SHAPE of what gets tested in the first place, which is often a cheaper and more durable fix. Limiting primary-metric count forces a real pre-commitment: a team that has to pick one or two metrics as the ones that decide ship/no-ship, with everything else demoted to secondary or guardrail status, structurally can't quietly cherry-pick a lucky metric from a long list after the fact. Surfacing the total-tests-in-context alongside any single result (something like "this metric was 1 of 40 tested today across the company") gives a reader the honest base rate needed to judge whether a single significant finding is more likely a real effect or an expected false positive given how many things were checked.

Worked example

A team runs an experiment with twelve metrics all nominally labeled "primary." One shows p=0.03. Reported in isolation, that looks like a clean win; reported alongside "twelve primary metrics were tested, and roughly one false positive at this rate is expected by chance alone even with zero real effect," the same number reads very differently, and the platform's job is to make that context impossible to miss rather than something a skeptical reader has to independently know to ask for.

Trade-offs and pitfalls

Capping primary-metric count too aggressively (forcing every team down to exactly one metric) can genuinely hurt teams whose product change legitimately affects two truly independent, equally important outcomes. The workable middle ground is a small cap (two or three) rather than one, paired with the requirement that the primary set be declared before launch, not adjusted afterward once results are in.

Direct answer

Control charts and CUSUM are good defaults for catching a sustained shift once you have a stable baseline; EWMA weights recent data more heavily and reacts faster to a moderate, ongoing drift; changepoint detection is the right tool when you don't know in advance what kind of shift to expect and want the algorithm to find where the underlying process changed. Avoid false positives from ordinary noise by requiring a minimum warmup period to establish a trustworthy baseline variance, and by tuning the alert threshold against that metric's own historical noise rather than a one-size-fits-all number.

Structured elaboration

• Control charts (like a Shewhart chart) flag a single point that's an extreme outlier relative to a fixed baseline mean and standard deviation; simple and interpretable, but slow to catch a moderate, gradual shift because each point is judged independently.
• CUSUM (cumulative sum) accumulates small deviations over time, so it's specifically good at catching a persistent, moderate shift that no single point would trigger on its own, at the cost of needing a tuned "slack" parameter that trades detection speed against false-alarm rate.
• EWMA weights recent observations more than older ones (via a decay parameter alpha), giving a faster reaction to a real, ongoing change than a plain control chart while smoothing out single-point noise; the trade-off is a genuine subtlety worth being careful about: the variance estimate used for the threshold has to be built from PAST points only, or a real regression can inflate its own threshold and hide from the detector (a bug we caught and fixed in exactly this way while implementing an EWMA detector for this same problem).
• Changepoint detection looks for the point in a time series where the underlying distribution's parameters genuinely change, without assuming in advance what kind of shift (mean, variance, or both) to expect; it's more flexible but computationally heavier and harder to run cheaply on every metric, every hour.

Avoiding false positives specifically: require a warmup window (typically the first several intervals) before any check is allowed to fire, since the variance estimate is unreliable with too little history; and calibrate the threshold in units of that metric's own recent variability rather than an absolute number, since a metric that's naturally noisier day-to-day needs a wider band than a very stable one.

Any of these detectors improves over time if it's allowed to LEARN a baseline from history rather than using a fixed rule forever: modeling the metric's normal daily and weekly seasonality (a baseline model, not just a flat mean) and feeding confirmed false positives back into the baseline sharpens precision the longer the detector runs. Whatever detector is chosen also needs an explicit escalation path once it fires: a moderate flag pages an on-call owner for a judgment call, while a severe, high-confidence flag should be wired into the same automated pause-or-rollback machinery guardrail breaches use elsewhere, since an anomaly detector that only produces a dashboard entry with no downstream action is exactly as useful as no detector at all.

Worked example

An EWMA-based detector flagged a conversion-rate regression that turned out to be real (a login-flow bug), but the same detector, before a fix, occasionally fired on ordinary noise shortly after warmup because too little history had accumulated to estimate variance reliably yet. Extending the warmup window and using only PAST variance (never the current point's own deviation) to judge the current point resolved both the missed-detection risk and the early false-positive risk in the same fix.

Trade-offs and pitfalls

The core trade-off across all of these is detection speed versus false-alarm rate: a more sensitive detector (tighter threshold, shorter warmup) catches real regressions faster but fires more often on noise; a more conservative one is quieter but slower to react to a genuine problem. There's no threshold-free answer here; the right calibration depends on how costly a missed detection is (a payments regression warrants a more sensitive, noisier detector) versus how costly alert fatigue is for that team.

Direct answer

A compact experiment configuration needs the pieces both the SDK and offline analysis depend on: an id, the variants and their allocation percentages, targeting rules, a rollout schedule (which phase is currently live), the metric definitions this experiment is measuring, and a kill switch, structured so an SDK can evaluate it in milliseconds and an analysis job can reconstruct exactly what configuration was live at any point in time.

Structured elaboration

{
  "experiment_id": "checkout_redesign_v2",
  "status": "running",
  "variants": [
    { "key": "control", "allocation": 0.5 },
    { "key": "treatment", "allocation": 0.5 }
  ],
  "targeting": { "country_in": ["US", "CA"], "app_version_gte": "4.2.0" },
  "rollout_phases": [
    { "phase": 1, "traffic_pct": 5, "started_at": "2026-06-01T00:00:00Z" },
    { "phase": 2, "traffic_pct": 50, "started_at": "2026-06-05T00:00:00Z" }
  ],
  "metrics": { "primary": "checkout_completion_rate", "guardrails": ["payment_failure_rate", "page_load_p95"] },
  "kill_switch": false,
  "config_version": 7
}

The SDK only strictly needs targeting, variants/allocations, the current rollout phase's traffic_pct, and kill_switch, since those are the fields that determine "does this user get evaluated at all, and into which bucket," and it needs them cached locally with low latency, since a request-time fetch to a central config store for every evaluation would be too slow at scale. Offline analysis additionally needs the metrics block (to know what to compute), the full rollout_phases history (to know what config was live at any past moment, not just now), and config_version (to detect that a mid-experiment configuration change happened, which the analysis has to account for or exclude).

Worked example

Six weeks into an experiment, someone asks "was this really running at 50% the whole time?" Because rollout_phases records each phase with its start timestamp rather than overwriting a single current_traffic_pct field, the answer is a direct lookup: 5% for the first four days, 50% after, which lets the analysis correctly exclude or separately model the low-traffic ramp period rather than assuming a constant allocation throughout.

Trade-offs and pitfalls

Making the schema too rich (embedding full statistical analysis parameters, verbose free-text descriptions) bloats what has to be fetched and cached by every SDK evaluation, most of which never needs those fields. Making it too thin (no config_version, no rollout_phases history, just a single current allocation) is cheap for the SDK but breaks reproducibility for analysis, since a config that only tracks "what's true now" cannot answer "what was true when this user was actually exposed."

Direct answer

A streaming ingestion pipeline for experiment telemetry needs a message broker (like Kafka) for durable at-least-once delivery, stream processors that deduplicate by a stable event id and handle schema evolution via a shared schema registry, and a storage/serving layer that produces near-real-time aggregates for monitoring while a separate, slower batch path produces the authoritative, fully-reconciled numbers used for the final read-out.

Structured elaboration

flowchart LR
  Clients[Client/Server SDKs] --> Broker[(Message Broker: Kafka)]
  Broker --> StreamProc[Stream Processor: dedup, enrich]
  StreamProc --> Hot[(Hot store: near-real-time aggregates)]
  StreamProc --> Cold[(Cold store: raw immutable events)]
  Cold --> Batch[Daily batch job: authoritative recompute]
  Batch --> Reporting[Reporting metric store]
  Hot --> Monitoring[Live monitoring dashboard]
  SchemaReg[(Schema Registry)] --- StreamProc

• At-least-once delivery plus deduplication: the broker guarantees no event is silently lost, but that guarantee comes paired with the possibility of duplicate delivery on retry; the stream processor deduplicates using a stable event id (assigned at the client, not at ingestion, so a genuine retry of the same logical event carries the same id), typically via a bounded time-windowed dedup cache rather than an unbounded one that would grow forever.
• Schema evolution: every event carries a schema version, and the schema registry enforces that changes are additive only, so an older event and a newer one can both be processed by the same pipeline code without breaking.
• Late-arrival handling: the hot, near-real-time aggregate accepts events within a bounded lateness window (a watermark), and any event arriving after that window still lands in the cold, immutable raw store and gets folded into the next batch recompute, so late data isn't lost, it's just not reflected in the FAST view.
• Reconciling monitoring versus reporting: the hot store's numbers are approximate and fast, meant for guardrail alerting and live dashboards; the cold store plus batch job produces the authoritative number used for the actual ship decision, and the two are expected to differ slightly, which the platform should surface explicitly rather than let a viewer assume both are the same precision.
• Concrete freshness SLAs: a common split is a roughly 5-minute freshness target for the hot monitoring path (with historical rollups and lineage attached so a stale-looking number can be traced back) and a sub-one-hour freshness target for a daily-batch variant that does an incremental recompute on top of the previous run rather than a full reprocess every time, reserving a full recompute for schema fixes or backfills.
• Exactly-once semantics with named tooling: stream processors like Flink or Spark Structured Streaming provide exactly-once sink guarantees when configured correctly (checkpointing plus an idempotent or transactional write to the sink), which is what keeps the deduplication logic above from having to do all the work alone; watermarking in these frameworks is what implements the bounded lateness window described above.
• Metric registry integration: the same schema registry that governs event shapes should also version the METRIC definitions computed from those events, so a metric's computation logic, not just its event inputs, is tracked and queryable, which is what lets an offline research team later confirm exactly which formula produced a historical number and catch a silent computation bug through reproducibility checks rather than trusting the number by default.

Worked example

At 200k events per second, holding an unbounded deduplication window in memory is infeasible; a bounded window (say, 24 hours) trades a small risk of missing a very late duplicate for a manageable memory footprint, which is the right trade given that the authoritative batch recompute over the full immutable raw log catches any duplicate the bounded streaming dedup missed.

Trade-offs and pitfalls

Trying to make the streaming path both fast and perfectly accurate is the most common architectural mistake at this scale; it's cheaper and more honest to explicitly split "fast and approximate" from "slow and authoritative" than to over-engineer the streaming path to try to be both, which usually ends up being neither fast nor fully correct. The other pitfall is under-sizing the late-arrival watermark window for the traffic pattern actually observed (mobile clients batching events can introduce longer delays than a naive watermark assumes), silently excluding legitimate late data from the fast view more often than intended.

Direct answer

The core instrumentation practices are: consistent naming conventions across teams, explicit schema versioning, idempotent event delivery, event enrichment at ingestion rather than at query time, and backward compatibility whenever a schema changes, because a violation of any one of these quietly corrupts every metric computed downstream without producing an obvious error.

Structured elaboration

• Naming conventions: a shared taxonomy (verb_object style, like purchase_completed rather than one team's buy_done and another's checkout_success for the same underlying action) so a metric defined once can be reused across teams instead of every team redefining "purchase" slightly differently.
• Schema versioning: every event schema carries a version field, and a schema change is additive (new optional fields) rather than repurposing an existing field's meaning, so historical events remain readable by current code.
• Idempotency: events carry a unique event id so a retry (from a flaky network on the client, or a replay after a pipeline failure) can be deduplicated rather than double-counted, which matters enormously for count and sum metrics specifically.
• Event enrichment: joining an event to its experiment exposure, user segment, or session context should happen once, at ingestion, and be stored on the enriched record, rather than re-joined by every downstream query, both for performance and so every team computing a metric from the same enriched table agrees on what "exposed" meant at that moment.
• Backward compatibility: a schema change should never break a metric-computation job that hasn't yet been updated to read the new field, which in practice means never removing or repurposing a field, only adding.

Worked example

A team renames a field from user (an integer id) to user_id (a string id) in place, without a version bump, to "clean things up." Every metric job still filtering on the old field silently sees no matches for new events and reports a metric that quietly stops updating, which looks exactly like "the experiment has no effect" rather than "the pipeline broke," and can go unnoticed for days because nothing errors.

Trade-offs and pitfalls

Enforcing all of this centrally (a single shared schema registry every team must go through) adds process overhead that slows down individual teams shipping new events. The pragmatic middle ground most platforms land on is: a small number of core, centrally-governed event types (exposure, generic conversion) that everyone must use as-is, plus a flexible custom-properties field for team-specific context that doesn't require central review, so the fields that feed universal validity checks (like SRM) stay standardized while teams retain flexibility for their own custom analysis.