Experimentation Methodology and Rigor Questions

Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.

MediumTechnical

74 practiced

You need to test price, UI layout, and onboarding copy simultaneously. Explain how to set up a factorial experiment to estimate main effects and interactions, how to power the test for interactions, and how to interpret interaction terms so you can decide whether marginal effects are meaningful for product decisions.

Sample Answer

Setup:- Use a full factorial 2x2x2 (if each factor has 2 levels) or k-levels per factor. Example: Price (Low/High), UI layout (A/B), Onboarding copy (Concise/Guided) → 8 cells. Randomly assign users to cells to ensure balance; stratify by key covariates (device, geography) if needed.

Estimands:- Main effects: average causal effect of changing each factor marginalizing over others.- Two-way and three-way interactions: whether effect of one factor depends on another (e.g., price sensitivity differs by layout).

Analysis plan:- Pre-specify primary metric (conversion or LTV) and an ANOVA / linear regression model: Outcome ~ Price + Layout + Copy + Price*Layout + Price*Copy + Layout*Copy + Price*Layout*Copy + covariates.- Estimate coefficients for main effects and interactions, with robust SEs.

Powering for interactions:- Interactions typically smaller; power calculations must target the smallest effect size of practical significance. Use factorial power formula or simulate: - Set Minimum Detectable Interaction (MDI) (e.g., 2 percentage points on conversion). - For continuous approx, required N per cell ≈ (Zalpha/2 + Zbeta)^2 * 2 * sigma^2 / MDI^2. - Total N = cells * N_per_cell. If budget limited, prioritize powering key interactions and consider fractional factorials to reduce cells.

Interpreting interactions for product decisions:- If interaction coefficient is statistically significant and practically meaningful (exceeds your MDI), report conditional marginal effects: e.g., effect of price at Layout A vs Layout B. Visualize with interaction plots and marginal effect tables.- If interactions are non-significant and CIs small (exclude practical impact), use marginal (average) main effects to decide.- If interactions are non-significant but CIs wide (underpowered), avoid confident marginal conclusions—either collect more data or treat results as exploratory.- Decision rules (pre-registered): adopt change if (a) main effect beneficial and no harmful significant interactions, or (b) conditional effect positive in segments with business importance.

Practical tips:- Pre-register analysis and MDIs, monitor balance, guard against multiple testing (adjust alpha or focus on predefined contrasts), and report both statistical and practical significance.

EasyTechnical

79 practiced

Define proximal (leading) metrics and distal (business) metrics. Provide three concrete examples of proximal and three of distal metrics for an e-commerce product. Describe one pragmatic approach to demonstrate that an observed lift in a proximal metric is likely to cause a lift in a distal business metric.

EasyTechnical

68 practiced

What is the multiple hypothesis testing problem in product experimentation? Briefly explain Bonferroni correction and Benjamini-Hochberg False Discovery Rate control, and describe one practical policy for when each approach should be applied in a product organization.

MediumTechnical

65 practiced

Design an audit process to validate product metrics used in experiments. Include a set of test queries and monitoring alerts that detect missing or duplicate events, out-of-order timestamps, sudden changes in event counts, and metric leakage. Specify ownership for fixing issues and a cadence for metric audits.

Sample Answer

Audit process overview:- Goal: ensure experiment metrics are accurate, timely, and free from duplicates, missing events, ordering errors, sudden shifts, or leakage into metrics.- Stakeholders: Product Manager (owner of metric definitions & business impact), Analytics/Experimentation Lead (owner of audit cadence & dashboards), Data Engineering (fixes instrumentation/pipeline bugs), Engineering (fixes client/server event code), QA (validates fixes).

Audit steps:1. Define canonical event spec & golden SQL metric (single source of truth).2. Daily automated checks (alerts); weekly manual deep-dive; monthly steward review.3. Incident workflow: alert → analytics triage → assign to Data Eng/Engineering → fix → verify → postmortem.

Sample test queries (SQL pseudocode):- Missing events: count by day vs expected users

sql

SELECT day, COUNT(*) AS events
FROM events
WHERE event_name='signup'
GROUP BY day
ORDER BY day DESC;

- Duplicate events (same user, event_id, timestamp):

sql

SELECT event_id, COUNT(*) c
FROM events
GROUP BY event_id
HAVING c > 1;

- Out-of-order timestamps (client_ts > server_ts or non-monotonic per user):

sql

SELECT user_id, MIN(server_ts - client_ts) AS min_diff
FROM events
WHERE server_ts < client_ts
GROUP BY user_id;

- Sudden change detection (day-over-day % change):

sql

WITH daily AS (SELECT day, COUNT(*) cnt FROM events WHERE name='purchase' GROUP BY day)
SELECT day, cnt, LAG(cnt) OVER (ORDER BY day) prev, (cnt - prev)/NULLIF(prev,0) pct_change FROM daily;

- Metric leakage (conversion events occurring before exposure or in control group):

sql

SELECT e.user_id
FROM events e
LEFT JOIN exposures x ON e.user_id = x.user_id AND x.experiment='expA'
WHERE e.name='conversion' AND x.exposure_ts IS NULL;

Monitoring alerts & thresholds:- Missing events: alert if event count drops >30% vs 7-day rolling mean.- Duplicates: alert if duplicate rate >0.5% of total events or >1000 duplicates/day.- Out-of-order timestamps: alert if >1% events have client_ts > server_ts or negative skew >5s median.- Sudden changes: alert if day-over-day change >50% or statistical anomaly (z-score >4).- Metric leakage: alert if >0 conversions without exposure per 10k users or any conversions in control cohort.Include alert channels (PagerDuty for critical, Slack + email for warnings) and contextual payload (query results, sample user_ids, recent deploys).

Ownership & remediation:- Analytics triage (within 1 hour for critical alerts): confirm noise vs real issue, annotate dashboard.- Data Engineering: fix ETL or dedup logic within SLA (24–72 hours depending severity).- Engineering (client/backend): fix instrumentation or SDK bug within sprint (typically 1–2 weeks) with hotfix option for production-impacting issues.- PM: communicate product/business impact to stakeholders, prioritize fixes in roadmap if recurring.

Cadence:- Continuous automated (daily) checks.- Weekly audit meeting: review alerts, triage false positives, review high-risk metrics.- Monthly steward review: validate metric definitions, ownership, and runbook updates.- Quarterly retrospective: review incidents, refine thresholds and instrumentation standards.

Key governance items:- Maintain metric catalog with definition, SQL, owner, alert thresholds, and runbook.- Require every experiment to reference a canonical metric ID from the catalog.- Automate verification in CI for instrumentation changes (unit tests, event-schema validation).

MediumTechnical

72 practiced

Design a checklist and monitoring approach for launching experiments in a mobile app that experiences intermittent network connectivity and backgrounded sessions. Include instrumentation, event buffering, deduplication, sessionization rules and how to treat late-arriving events in analysis.

Sample Answer

Checklist for launching experiments (mobile app with intermittent connectivity / backgrounding)

Pre-launch instrumentation- Define experiment events: impression, click, exposure, conversion, attribution keys (user_id, device_id, experiment_id, variant, timestamp, event_id).- SDK requirements: persistent local queue, durable storage (SQLite), retry/backoff, write-ahead logging, encryption, battery/network-aware sync.- Include monotonic client event_id (UUID + sequence) and server-assigned ingestion_id when possible.- Add lifecycle hooks: app foreground/background, network-change, session_start/session_end signals.

Event buffering & upload policy- Buffer events locally with size & time thresholds (e.g., flush when >50 events, or every 30s when online).- Prioritize critical events (exposure/conversion) for immediate retry; batch low-priority analytics.- Backoff and jitter for retries; respect battery saver modes.

Deduplication- Client-side: avoid re-sending identical event_id unless confirmed unsent.- Server-side: idempotent ingestion using event_id + device_id; TTL for dedupe window (e.g., 7 days).- Log dedupe metrics (duplicates rate, rejected events).

Sessionization rules- Session start: app foreground OR explicit user interaction after background.- Session end: background >30s (configurable) OR explicit logout.- Merge short background gaps (<5s) into same session for UX continuity.- Tag events with session_id, session_start_ts, session_length.

Monitoring & alerts- Instrument dashboards: event delivery rate, buffer size distribution, failed uploads, dup rate, session lengths, experiment exposure rate by variant.- Alert thresholds: delivery rate <95%, duplicate rate >1%, average buffer size >75th percentile, spike in session length anomalies.- Health checks: sample end-to-end test events from devices in different network conditions.

Analysis & late-arriving events- Implement event-time processing with watermark (e.g., 24–72 hour lookback). Use ingestion_time for real-time dashboards but finalize metrics after lookback window.- Reconcile conversions arriving late: flag as late and run backfill pipelines to update variant metrics; keep audit trail.- Use incremental attribution: once late events arrive, re-run attribution and compute delta; expose both provisional and final experiment results.- Report confidence intervals that account for late arrivals and apply conservative decisions until finalization window passes.

Roles & rollout- PM: define SLAs, approve finalization window, decide on early-stopping rules.- Eng: implement SDK/server dedupe, buffering, monitoring.- Data: implement watermark/backfill, finalization job, validation queries.

This approach balances user/data integrity, battery/network constraints, and accurate experiment results while giving clear operational guardrails.

Unlock Full Question Bank

Get access to hundreds of Experimentation Methodology and Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.