Approach: I’ll (1) enumerate likely failure modes from cookie-level randomization for logged-in users, (2) quantify impacts on bias, variance and power, and (3) propose concrete mitigations (implementation + analytic fixes) prioritized by impact and effort.Failure modes- Contamination (treatment switching): Users who clear cookies or switch devices can receive different arms across sessions → bias toward null (attenuation) if effects differ by arm or session.- Correlated observations / non-independence: Multiple observations from same user counted as independent → underestimates SEs, inflated significance (Type I error).- Differential attrition & measurement: If cookie loss correlates with outcome or device type, it induces selection bias.- Loss of effective sample size: Re-randomization fragments user exposure, reducing power.- Misattribution in funnels: Cross-device sessions split across arms break user-level funnels.Quantification (example calculations)- Contamination rate r = fraction of exposures where user’s cookie differs from their canonical assignment. If true ATE = δ, observed ATE ≈ (1 − r)δ (attenuation). So 20% contamination reduces effect by 20%.- Effective sample size with clustering: n_eff = n / (1 + (m − 1)ρ), where m = avg observations per user, ρ = intra-class correlation. If m=5 and ρ=0.1, n_eff ≈ n/1.4 → 29% loss.- Variance underestimation: If you treat observations as independent but clustering exists, SEs are underestimated by factor sqrt(1 + (m − 1)ρ).Concrete mitigations (prioritized)1) Switch to user-id randomization (recommended) - Server-side: on first authenticated event, assign user_id → treatment using hashed user_id mod bucket or deterministic assignment service. Persist assignment server-side. - Benefits: eliminates re-randomization across devices, simplifies analysis (unit = user). - Implementation notes: for existing users, backfill assignment from lookup; for new users assign on account creation.2) Server-side assignment with cookie fallback - Assign server-side but for unauthenticated flows use cookie; when user later logs in, reconcile by mapping cookie to user_id and retroactively tagging exposures for deduplication or reclassification.3) Deduplication + canonicalization in analysis (when switching assignment now isn’t possible) - Aggregate to user-level: collapse multiple device/session rows into a single exposure window (e.g., first-assigned treatment in window) and analyze by user. - Use dedupe rules: prefer authenticated assignment; ignore pre-login cookie assignments once user_id appears. - Weighting: use inverse-probability weights if unequal exposure lengths.4) Instrument analytic corrections and robust inference - Cluster standard errors by user_id (or bootstrap by user) to fix SE underestimation. - Intention-to-treat at user-level when possible. - Sensitivity analyses: estimate contamination r using device-change logs; simulate attenuation to bound true effect.5) Monitoring & alerting - Track metrics: fraction of exposures by device vs user_id, rate of cookie-to-user merges, % of users switching arms, and per-arm funnel divergence. - Alert if contamination r > threshold (e.g., 5–10%) or if m or ρ rising.Trade-offs & rollout plan- Quick wins: analytic deduplication + clustering, monitoring.- Medium: server-side user-id randomization for authenticated flows (low risk).- Long-term: fully migrate experiments to server-side deterministic assignment with cross-device mapping and retroactive tagging; maintain cookie fallback for anonymous users.Example SQL snippets (conceptual)- Deduplicate to user-level first-treatment:sql
SELECT user_id,
MIN(event_time) FILTER (WHERE treatment IS NOT NULL) AS first_treat_time,
ANY_VALUE(treatment) -- treatment at first_treat_time
FROM events
GROUP BY user_id;
