Experimentation Platforms and Infrastructure Questions

Addresses the technical and organizational infrastructure required to run experiments at scale. Topics include randomization and assignment strategies, traffic allocation, instrumentation and metric collection pipelines, experiment configuration and rollout systems, experiment tracking and metadata, data quality and monitoring, guardrails to detect interference or contamination, automated validity checks, self service experimentation tooling, governance and permissions, and approaches to scale experimentation across many teams while preserving statistical validity. Senior conversations include designing experiment platforms, enabling self service and observability, and trade offs when scaling experiment velocity across products.

MediumTechnical

103 practiced

Compare streaming (real-time) vs batch (daily) metric computation for experimentation. For a high-traffic product, outline situations where each approach is preferable and propose a hybrid design that balances immediacy and accuracy.

Sample Answer

**Brief framing**As a research scientist I compare streaming vs batch metrics along two axes: freshness (latency) and fidelity (accuracy/stability). High traffic amplifies both opportunity and risk — quick signals but also high variance and potential bias.

**When streaming (real-time) is preferable**- Rapid anomaly detection or safety (eg. spike in error rate, model drift) needing immediate intervention.- Online experiments where fast iteration matters (A/B gating, feature flag rollback).- Use cases tolerating approximate estimates (sketches, exponentially weighted averages).

**When batch (daily) is preferable**- Finalized business metrics requiring full joins, deduplication, late-arriving events, or costly backfill (revenue, retention, attribution).- Low-signal metrics where reduction of variance via aggregation matters.- Experiments needing gold labels that arrive with latency (purchase confirmations, conversions).

**Hybrid design (balance immediacy + accuracy)**- Streaming layer: Kafka -> stream processor (Flink/Beam) computes lightweight, approximate KPIs (counts, rates, online variance) with confidence intervals; emits alerting signals and dashboards.- Batch layer: Event lake -> Spark/Presto performs deterministic joins, de-dup, windowed aggregations, causal joins (user histories), and computes final experiment metrics once per day.- Reconciliation & calibration: nightly job compares stream vs batch, computes bias/correction factors, updates streaming model parameters and error bands.- Product rules: act on streaming alerts when confidence > threshold; otherwise wait for batch-confirmed decisions.- Instrumentation: attach lineage IDs, idempotent writes, and backfill support so batch can correct streaming estimates and re-run experiment analyses.

**Statistical considerations**- Publish CI and sample sizes for streaming signals; use sequential testing corrections if acting early.- Track late-arriving label rates and correct for censoring in streaming estimates.

This hybrid gives fast, actionable signals while preserving final accuracy required for publishable experimental results.

HardSystem Design

65 practiced

Design an observability and lineage system that allows an analyst to trace an unexpected metric uplift from a dashboard down to raw events, transformation jobs, and code deployments that could have caused it. List the artifacts to store, APIs to query, and visualization features you'd provide.

Sample Answer

**Clarifying scope & goals**Trace a metric uplift from dashboard -> derived metrics -> transformations -> raw events -> model/training/code deployments; support reproducibility for experiments and hypotheses testing.

**Artifacts to store**- Metric catalog (definitions, SQL/DSL, owners, expected ranges)- Lineage graph edges (dataset/table/column -> transform/job -> metric) with timestamps- Event store pointers (storage URI, partition keys, provenance tags)- Job metadata (job id, DAG, code commit sha, config, container image, runtime env)- Model/experiment registry entries (model id, hyperparams, training dataset snapshot, seed, eval metrics)- Telemetry logs (alerts, latency, sample payloads)- Deployment records (canary/rollout percentages, infra changes)

**APIs to query**- GET /lineage?metric={id}&timewindow=... -> returns directed graph with timestamps- GET /artifact/{type}/{id} -> fetch definition, code, commit, config- POST /trace/query -> returns filtered paths, root causes scored- GET /samples?dataset={id}&range=...&limit=... -> raw event samples- GET /jobs/{id}/runs -> run logs, inputs, outputs, diffs between runs- GET /models/{id}/explain?metric=... -> feature importance drift, data slice performance

**Visualization features**- Interactive time-travel lineage graph: animate changes across time, highlight divergent paths- Dependency heatmap: edge thickness = contribution score; node color = drift/anomaly score- Side-by-side diff view: two runs/commits showing dataset drift, schema changes, metric deltas- Root-cause scoring panel: ranked hypotheses (code deploy, data drift, feature change) with evidence links- Sample explorer: view raw events tied to offending metric points with ability to replay/aggregate- Integrations: link to code repo, CI/CD, experiment notebooks for one-click checkout and reproduce

Why this fits a Research Scientist- Preserves experimental artifacts (snapshots, seeds) enabling reproducible causal analysis- Enables hypothesis-driven investigation: propose, test, and validate root causes with data and model artifacts.

HardTechnical

73 practiced

For experiments on social platforms where treatment may propagate between users (network effects), describe experimental designs and identification strategies to estimate peer effects. Include ideas like cluster randomization, encouragement designs, and instrumental variables.

Sample Answer

**Brief framing / challenge**Network interference violates SUTVA: one unit’s treatment can affect others. Identification focuses on designs that control or exploit propagation to separate direct treatment effects from peer effects under explicit assumptions (partial interference, exposure mappings).

**Designs**- Cluster randomization: randomize treatment at community/cluster level (graph clustering, modularity-based). Estimates total (direct+indirect) effects; needs many clusters for asymptotics.- Randomized saturation / two-stage randomization: randomly choose clusters’ treatment saturation levels, then randomize individuals within clusters. Allows separation of direct vs. spillover effects by comparing individuals across saturation arms.- Ego-network/seed randomization: randomize seeds (treated influencers) and measure diffusion; good for diffusion dynamics.- Encouragement designs: randomize encouragement (instrument) to take treatment; observe who adopts. Useful when direct assignment impossible or noncompliance high.

**Identification strategies / estimators**- Exposure mapping: define discrete exposure conditions (e.g., treated self, k treated neighbors). Estimate contrasts between exposure groups; requires correctly specified mapping.- Instrumental variables / TSLS for peer effects: use randomized encouragement or cluster-level assignment as instrument for peers’ treatment. Two-stage:

text

Stage 1: Peer_Treatment_i = alpha + gamma * Instrument_for_peers_i + u
Stage 2: Outcome_i = beta_direct * Treat_i + beta_peer * Predicted_Peer_Treatment_i + e

Assumptions: relevance, exclusion (instrument affects outcome only via peers’ treatment), monotonicity for LATE interpretation.

**Assumptions & trade-offs**- Partial interference (interference only within clusters) simplifies identification.- Exclusion restriction is strong — check via placebo outcomes, balance tests.- Spillovers heterogeneous — consider stratified analyses, heterogeneous treatment effect models.- Graph-aware randomization reduces bias but can be logistically hard; saturation designs improve power to estimate spillovers.

**Practical checks**- Pre-register exposure mapping and estimands (direct, indirect, total, overall).- Use permutation/randomization inference for valid p-values under dependence.- Sensitivity analyses: vary exposure definitions, test instruments, simulate diffusion under alternative models.

MediumSystem Design

66 practiced

Design a scalable assignment algorithm for experiments that must support sticky assignment across devices and sessions, deterministic bucketing, and fast evaluation in microservices. Explain key data inputs and how you would ensure low latency and consistency.

Sample Answer

**Clarify requirements & constraints**- Sticky assignment must persist across devices and sessions for the same person when possible.- Bucketing must be deterministic (same inputs → same assignment).- Microservices must evaluate assignments with very low latency (<1–5 ms typical).- Must support experiment config changes, rollouts, and overrides consistently.

**High-level approach**- Use deterministic hashing for default assignment: hash( identity_key + experiment_salt ) → bucket.- Provide a lightweight local SDK in each microservice that computes the hash locally and caches experiment configs.- Maintain a central Assignment Service for sticky records, overrides, and identity resolution (cross-device linking).

**Key data inputs**- Identity key(s): user_id, device_id, login_id, cookie_id (priority list).- Experiment metadata: experiment_id, number_of_buckets, salt, traffic_allocation, start/stop timestamps, seed_version.- Sticky map: {identity_key → (experiment_id, bucket, timestamp, version)} for persisted sticks.- Identity graph: links across devices (user account ↔ device cookies) for cross-device stickiness.

**Assignment flow**1. Client/microservice picks best identity_key (logged-in user preferred; fallback to cookie/device).2. Local SDK checks sticky cache; if present and version matches, return stored bucket.3. Else compute deterministic bucket: - bucket = Hash( identity_key || experiment_salt || seed_version ) mod N_buckets - If bucket falls outside traffic_allocation, return control.4. If sticky should be persisted, call Assignment Service asynchronously to record (write-through for critical paths).5. Assignment Service stores sticky mapping and resolves identity graph merges.

**Low latency & consistency techniques**- Local SDK caches experiment config and recent sticky entries; computes hash locally to avoid RPC on cold path.- Configs pushed via pub/sub (e.g., Kafka) or fast CDN; small JSON with salts and seed_version.- Writes to central sticky store are batched and made idempotent; reads served from low-latency store (Redis with replication + tunable consistency).- Use versioned salts/seed_version so experiments change deterministically; SDK checks version to invalidate cache.- For cross-device merges, use eventual consistency but ensure deterministic re-evaluation using canonical identity (e.g., account_id) and annotate when reassignments occur; prefer server-side reconciliation with monotonic versioning to avoid flip-flops.

**Trade-offs & edge cases**- Fully consistent cross-device stickiness requires synchronous identity resolution (higher latency) — mitigate by returning local deterministic result and reconciling later.- Privacy: limit persistence of PII, support anonymized hashed IDs and user opt-outs.- Ensure reproducibility: log seed_version and identity_key used for each assignment for analysis.

This design balances deterministic, sticky assignments with microservice low-latency requirements by pushing cheap, deterministic computation to the edge and keeping a central service for persistence, reconciliation, and overrides.

EasyTechnical

76 practiced

Explain traffic allocation strategies used in experimentation platforms (static splits, progressive rollouts, bucketing, and weighted allocation). For each, give a short example of when it is most appropriate and the main trade-offs.

Sample Answer

**Overview** Traffic allocation strategies control how users or requests are assigned to experimental conditions. Choice affects statistical power, bias, rollout safety, and user experience.

**Static splits** - Description: Fixed proportion (e.g., 50/50) throughout the experiment. - When appropriate: A/B tests focused on causal inference where balanced groups are required. - Trade-offs: Maximizes power per sample for two-arm tests but risky for exposing large populations to bad variants and inflexible to time-varying effects.

**Progressive rollouts (ramp-ups)** - Description: Start with small fraction (1–5%), gradually increase to target. - When appropriate: Deploying potentially risky model changes (new ranking model) to monitor upstream failures and UX regressions. - Trade-offs: Safer operationally and easier to detect large adverse effects early, but complicates inference (time confounding) and delays reaching full power.

**Bucketing (cohort-based assignment / sticky buckets)** - Description: Assign users to persistent buckets (hashing, geo, cohort) that determine treatment across sessions. - When appropriate: Longitudinal experiments where consistency per user matters (personalization models, reinforcement learning agents). - Trade-offs: Reduces contamination and interference, enables within-subject measurement, but can create imbalance if bucket distribution correlates with covariates and reduces effective sample randomness.

**Weighted allocation** - Description: Non-equal probabilities (e.g., 70% control, 30% variant) or adaptive weighting (bandits). - When appropriate: Quick optimization when multiple arms or when expected uplift is low; adaptive trials to reduce regret. - Trade-offs: Improves user experience/regret or explores many arms efficiently, but complicates power calculations and can bias standard estimators unless properly adjusted.

For a Research Scientist, always quantify implications: compute required N under each scheme, assess time confounding, and plan analysis (covariate adjustment, reweighting) to obtain unbiased estimates.

Unlock Full Question Bank

Get access to hundreds of Experimentation Platforms and Infrastructure interview questions and detailed answers.

Join thousands of developers preparing for their dream job.