Experimentation Platforms and Infrastructure Questions

Direct answer

Run a step-by-step operational response: first quantify the contamination's actual scope (which experiments, which time window, roughly how many users affected), then decide whether affected experiments need to pause, be re-analyzed with an explicit contamination adjustment, or be allowed to continue with the contamination noted as a caveat, and only after containment turn to the longer-term fix (a coordination policy or a traffic-layering change) that prevents the same class of collision from recurring.

Structured elaboration

1. Scope the problem first, before acting: pull the actual user-overlap and timing data across the flags and rollouts implicated, rather than reacting to the qualitative complaint alone. Contamination reports are often vaguer than the actual data ("something feels off") and the first job is turning that into a specific, bounded claim.
2. Triage by severity: an overlap that's small and unlikely to meaningfully bias either experiment's primary metric can often be left running with a documented caveat; a large, high-confidence contamination affecting a decision that's about to be made should pause the affected experiments immediately.
3. Communicate to the teams whose experiments are affected: they need to know before they make a ship decision on data that might be compromised, not after.
4. Fix the immediate cause: this is often an unplanned interaction between two independently-launched flags or a rollout schedule collision rather than a single experiment's fault, so the fix may involve coordinating a schedule change with a different team, not just editing one experiment's configuration.
5. Address the systemic cause afterward: only once the immediate situation is contained does it make sense to invest in the longer-term prevention, whether that's an automated pre-launch overlap check, a shared calendar of high-traffic-surface launches, or a formal coordination/prioritization policy for a specific contested surface (a busy city, a shared page) where conflicts recur.

Worked example

Two rapid feature-flag rollouts on the same page overlap for four days before anyone notices, and a third team's active experiment shares that page too. Quantifying the actual overlap shows it affects roughly 12% of one experiment's traffic during a 4-day window out of its planned 14-day run. Given the size and duration, the team decides to exclude that 4-day window from the final analysis (using the remaining 10 days, which is still enough for the planned power) rather than pausing and restarting, which would have cost the experiment more total time than simply excluding the contaminated window.

Trade-offs and pitfalls

Reacting to every contamination report by immediately pausing everything involved is the safest but most disruptive option, and overusing it trains teams to under-report ambiguous cases to avoid the disruption; underreacting (dismissing reports without actually quantifying scope) risks shipping a decision based on genuinely compromised data. The resolution is making the FIRST step, quantifying actual scope, fast and cheap enough that a team doesn't have to choose between "ignore it" and "stop everything" before they even know how bad it is.

Direct answer

Verify platform-level calibration by running periodic A/A tests (splitting real traffic into two identical, no-difference groups) and checking that the distribution of p-values across many A/A runs is close to uniform, that no more than roughly the nominal false-positive rate (about 5% at alpha=0.05) come back significant, and that assignment stays balanced throughout; if any of those checks fail, the statistical machinery itself is miscalibrated and every real experiment's p-value is suspect until it's fixed.

Structured elaboration

The logic: under a true null (no real difference, which an A/A test guarantees by construction), a correctly implemented significance test should produce a p-value that is uniformly distributed between 0 and 1, and should come back "significant" at alpha=0.05 roughly 5% of the time, purely by chance, no more and no less. Running many A/A tests (across different metrics, different traffic segments, different times) and checking this empirically is a direct test of whether the platform's variance estimation, its handling of dependent observations, or its multiple-testing correction are actually correct, independent of any real product question.

Concrete numeric thresholds worth watching: if noticeably more than 5% of A/A runs come back significant at alpha=0.05 (say, consistently landing above 8 to 10%), that's a strong signal the variance estimate is too small somewhere, commonly from treating correlated observations (repeated visits from the same user, or clustered users) as independent. If the p-value distribution is visibly non-uniform (bunched near 0, or bunched near 1) across many A/A runs, that points at a systematic bug in the test implementation itself rather than random noise.

Worked example

Running 100 independent A/A tests over a quarter, 14 come back significant at alpha=0.05, roughly triple the expected 5. Investigating further shows the variance calculation was using a simple per-event standard error rather than a per-user one, meaning power users who generated many events were effectively counted multiple times in the variance estimate, understating the true standard error and inflating the false-positive rate. This is exactly the kind of miscalibration that would silently make every real experiment's result look more confident than it actually is, until this specific bug is found and fixed.

Trade-offs and pitfalls

Running enough A/A tests to detect a modest calibration problem with confidence takes real, otherwise-unused traffic and time; a platform under pressure to maximize real experiment throughput can be tempted to skip this. The other pitfall is treating a single A/A test's non-significant result as proof of calibration: any one A/A run coming back non-significant tells you almost nothing (that's the expected outcome 95% of the time regardless), the signal only comes from the AGGREGATE behavior across many runs.

Direct answer

Before trusting a company-wide experimentation platform, an analyst should insist on: a fixed, shared standard for event instrumentation (not "however each team happened to log it"), a single place experiment metadata lives that every team can query, an explicit and validated join path from exposures to the business metrics that actually matter, and enforced guardrails against the three classic self-inflicted wounds: peeking, underpowered tests, and cross-experiment interference.

Structured elaboration

• Instrumentation standards: a shared event taxonomy and a required exposure event, so metrics defined once can be reused across teams instead of every team building slightly incompatible versions of "conversion."
• Metadata location: a queryable registry (not a spreadsheet, not tribal knowledge) so an analyst investigating a weird number in March can find every experiment that touched the same surface, in the same window, without asking around.
• Joining exposures to business metrics: this is where a naive implementation goes wrong first, typically through an attribution-window mismatch or double-counting a user exposed via two different entry points; before trusting a number, an analyst should be able to see the actual join logic, not just the output.
• Guardrails against peeking: enforce a pre-registered analysis plan and either disable early "peek" views for a non-technical audience or make any interim look explicitly caveated as such, since repeated informal checking inflates the false-positive rate far above the nominal 5%.
• Guardrails against underpowered tests: block or clearly flag any launch whose planned sample size wouldn't detect the minimum effect the team says they care about.
• Guardrails against cross-experiment interference: either enforce mutually exclusive traffic layers on shared surfaces or run an automated interference check that flags overlapping populations.
• What "enforced" should mean concretely: a Sample Ratio Test on every launch (not just an offered option), an explicit experiment-versioning field so a mid-run configuration change is visible rather than silently overwriting history, and a conflict check that throttles or flags a NEW launch the moment its targeting overlaps a currently-running experiment on the same surface, rather than only detecting the collision after the fact.

Worked example

An analyst is asked to explain why revenue moved in March. Without a metadata registry, answering that question means emailing every team that might have touched checkout that month; with one, it's a single query against the experiment registry filtered by date range and affected surface. That difference is the entire argument for insisting on centralized metadata before trusting a platform's output at all.

Trade-offs and pitfalls

Insisting on all of this before trusting any result can feel like blocking legitimate, fast-moving experimentation. The resolution isn't to lower the bar, it's to make the standard itself cheap to comply with: a shared instrumentation library that makes "do it the standard way" the path of least resistance, rather than a policy document nobody reads until an audit goes wrong.

Direct answer

Deduplicate exposures per user by keeping the earliest one by default, but reassign to the LAST-exposed variant specifically when that later exposure happened before the user's first purchase, since the spec's intent is attributing the purchase to whichever variant the user was actually under at the time they converted, not simply whichever variant they saw first; then compute the 7-day sum of purchase value strictly AFTER, and within 7 days of, whichever exposure timestamp ends up being the assigned one (the later exposure's timestamp if reassigned, the earliest exposure's timestamp otherwise).

Approach and code

Build a per-user candidate assignment using window functions to identify both the earliest and the latest exposure, decide between them based on where the first purchase falls relative to the latest exposure's timestamp, and carry the ASSIGNED exposure's own timestamp (not always the earliest one) forward as the anchor for the 7-day attribution window, with an explicit lower bound so a purchase before that anchor is never included.

WITH first_purchase AS (
  SELECT user_id, MIN(event_ts) AS first_purchase_ts
  FROM events WHERE event_type = 'purchase' GROUP BY user_id
),
ordered_exposures AS (
  SELECT e.*,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY exposure_ts ASC)  AS rn_earliest,
    ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY exposure_ts DESC) AS rn_latest
  FROM exposures e
),
final_assignment AS (
  SELECT a.user_id,
    CASE WHEN le.exposure_ts != a.exposure_ts
          AND fp.first_purchase_ts IS NOT NULL
          AND le.exposure_ts < fp.first_purchase_ts
         THEN le.variant ELSE a.variant END AS assigned_variant,
    CASE WHEN le.exposure_ts != a.exposure_ts
          AND fp.first_purchase_ts IS NOT NULL
          AND le.exposure_ts < fp.first_purchase_ts
         THEN le.exposure_ts ELSE a.exposure_ts END AS assigned_exposure_ts
  FROM (SELECT * FROM ordered_exposures WHERE rn_earliest = 1) a
  JOIN (SELECT * FROM ordered_exposures WHERE rn_latest = 1) le USING (user_id)
  LEFT JOIN first_purchase fp USING (user_id)
)
SELECT fa.user_id, fa.assigned_variant,
  COALESCE(SUM(CASE WHEN ev.event_type='purchase'
    AND ev.event_ts >= fa.assigned_exposure_ts
    AND ev.event_ts <= fa.assigned_exposure_ts + INTERVAL '7 days'
    THEN ev.value ELSE 0 END), 0) AS purchase_7d_sum
FROM final_assignment fa
LEFT JOIN events ev ON ev.user_id = fa.user_id
GROUP BY fa.user_id, fa.assigned_variant;

Key points: window functions identify earliest and latest exposure per user in one pass; the CASE logic implements the "reassign only if the later exposure preceded the first purchase" rule exactly as specified, and now computes the ASSIGNED exposure's own timestamp alongside the assigned variant, rather than always keeping the earliest exposure's timestamp regardless of which variant was assigned; the 7-day window is anchored to assigned_exposure_ts (whichever exposure the user was actually attributed to) with an explicit >= lower bound so a purchase before that exposure is never counted.

Complexity and edge cases

Complexity is O(n log n) for the window-function sort per user, dominated by the exposures and events table sizes; a user with no purchases returns a 0 sum via COALESCE rather than NULL; a user with only one exposure trivially has earliest equal to latest, so the CASE always falls through to the single exposure's variant and timestamp. I re-verified this against a constructed test with disambiguating cases: (a) a user reassigned to a later variant before their first purchase, with a purchase that falls within 7 days of the LATER exposure but outside 7 days of the EARLIER one, which now correctly counts (the original query incorrectly excluded it); (b) a user with a purchase that occurred BEFORE their exposure, which now correctly returns 0 (the original query incorrectly included it); (c) a user whose later exposure arrived AFTER their first purchase, correctly NOT reassigned, with a late purchase correctly excluded by the 7-day cutoff.

Trade-offs and pitfalls

The tempting simpler approach, always keeping the earliest exposure and ignoring later ones entirely, is wrong under this spec because it would misattribute the purchase to a variant the user was no longer actually under by the time they converted. The window-function approach used here scales well in a data warehouse but assumes exposure and event timestamps are already reasonably clean and synchronized; a real production version would need the same timezone and late-arrival handling discussed for the general metrics pipeline before trusting these numbers.

Direct answer

Design the reconciliation around canonical UTC timestamps computed as close to the source as possible, a shared, explicit session-windowing definition both services agree to use (rather than each inferring sessions independently), deduplication keys based on a stable event id rather than a recomputed hash, idempotent processing so a retry doesn't double-count, and automated tests that assert the joined output on a known input produces a known, hand-verified result.

Structured elaboration

• Canonical timestamps: convert every timestamp to UTC at the earliest possible point (ideally at the source service, not downstream), and store the original timezone/offset alongside it for debugging, since a downstream conversion error is much harder to spot after the original context is lost.
• Session windowing: if Service A and Service B use different session-timeout definitions (say, 30 minutes of inactivity versus a fixed calendar day), events that should logically belong to the same session can get split differently by each service; the reconciliation process needs a single, explicitly agreed session definition applied consistently across both, computed at reconciliation time rather than trusting each service's own internal session boundary.
• Deduplication keys: use a stable, source-assigned event id (not a hash of mutable fields, which can change if any field is corrected or reprocessed) so a legitimate retry is recognized as the same event rather than treated as new.
• Late-arriving events: define an explicit lateness tolerance (how long after a session's nominal end an event can still arrive and be included) consistent with the storage-tiering and watermarking approach used elsewhere in the pipeline.
• Idempotency: the join and aggregation logic should produce the same result whether run once or replayed multiple times over the same input, which usually means writing to an idempotent sink (an upsert keyed by the same deduplication key) rather than a naive append.
• Tests for correctness: a golden-file style test with hand-constructed events from both services, including a case that straddles a timezone boundary and a case with a legitimate retry, verifying the joined output matches a manually-computed expected result.

Worked example

Service A logs an exposure at 23:50 local time in a timezone that's UTC+9, while Service B logs the matching conversion event using a server timestamp already in UTC. Without converting Service A's timestamp to UTC before comparison, a naive join could conclude the conversion happened BEFORE the exposure (since 23:50 local time is actually 14:50 UTC the same day, a nine-hour difference that a naive string or local-time comparison would miss entirely), silently excluding a legitimate conversion from the metric.

Trade-offs and pitfalls

Trusting each service's own internal session or timestamp handling instead of establishing one shared, explicit definition at reconciliation time is the root cause of most cross-service join bugs like this; it's tempting because it requires no coordination between the two teams, but it's exactly the coordination that prevents silent, hard-to-detect metric corruption. The cost of building the shared reconciliation layer (agreeing on and enforcing one canonical timestamp and session definition) is real coordination overhead between two teams, but it's cheap relative to the cost of a metric quietly being wrong for months before anyone traces it back to a timezone mismatch.