Measurement Design and Analysis Questions

Practical measurement design and analytic techniques for producing reliable metric signals and proving impact. Includes instrumentation and tracking plans, experiment selection and validation, attribution modeling and its limitations, sample size and statistical considerations, identifying confounding variables, and reasoning about correlation versus causation. Also covers tradeoffs in data collection and data quality checks, cohort and segmentation design, baselining and threshold setting, designing dashboards and monitoring cadence, and connecting engineering and telemetry data to business outcomes. Candidates should be able to write clear measurement plans and success criteria, describe experiment and validation approaches, and explain how to operationalize results through reporting and iteration.

MediumSystem Design

33 practiced

Design an A/B test to evaluate a homepage redesign. Specify: primary metric with operational definition, guardrail metrics, randomization approach, sample size/power assumptions, ramp plan, and how you'd validate that the experiment instrumentation is correct before trusting results.

Sample Answer

Requirements:- Goal: Determine whether a homepage redesign increases user engagement and conversions without harming core business metrics.- Scope: 4-week experiment across web desktop and mobile web; exclude logged-out vs logged-in segmentation decisions as needed.

Primary metric (operational definition):- Primary: 7-day new-session conversion rate (user-level): proportion of users who, within 7 days of first exposure, perform the target conversion (e.g., sign-up or purchase). Unit = unique user ID (or cookie if necessary). Measure exposure date t0; count any conversion event in [t0, t0+7 days].

Guardrail metrics:- Revenue per user (7-day) — detects monetization regressions.- Bounce rate on landing session — immediate engagement signal.- Page load time (median) — performance regression.- Error-rate (JS/500s) — reliability.- Retention (7-day return rate).

Randomization approach:- Unit: user-id (persistent cookie or logged-in id) to avoid cross-contamination.- Method: deterministic hash of user-id modulo 1000; assign to control/treatment ensuring even split (e.g., 50/50).- Stratify by platform (desktop/mobile) and country for balanced allocation; implement consistent assignment for the experiment duration.

Sample size & power assumptions:- Baseline 7-day conversion = 2.0%. Detect minimum relative lift = 10% (absolute increase to 2.2%). Alpha = 0.05, power = 0.8.- Using standard proportions formula: required N per arm ≈ 46,000 users. (If traffic lower, extend duration or increase detectable lift.)- Calculate separately per strata if expecting heterogeneous effects; adjust for multiple comparisons if testing segments.

Ramp plan:- Phase 0 (QA, 1-3 days): internal testing with 0% traffic; smoke tests.- Phase 1 (1% traffic, 2 days): monitor instrumentation, performance, error rates.- Phase 2 (10% traffic, 3 days): validate metrics and guardrails under load.- Phase 3 (50% traffic, remaining duration): full experiment once stable.- Abort criteria: any guardrail triggers > predefined threshold (e.g., +5% error rate, -10% revenue/user) or major performance degradations.

Instrumentation & validation before trusting results:- QA checks: verify treatment/control buckets via deterministic hashing on a sample of IDs; test that variant assets load as expected.- Event-level validation: compare raw event counts (impressions, exposures, conversions) in analytics vs server logs; ensure exposure event fires once per user per session.- A/A test: run short A/A (or pre-period equivalence) to confirm randomization yields no baseline differences.- Metric sanity checks: stable pre-experiment CTR/conversion rates across buckets; no large demographic imbalances.- Data lineage checks: ensure deduplication, consistent user-id resolution, timezone alignment, and correct attribution window (7 days).- Monitoring: real-time dashboards for primary and guardrail metrics during ramp; snapshot comparisons between analytics and database aggregates.- Post-hoc: run treatment assignment SQL to verify sample sizes and check for contamination (users in both arms).

Analysis plan:- Use intent-to-treat, compute difference-in-proportions with bootstrap or z-test; report lift, 95% CI, p-value, and practical significance.- Segment results by platform and country; run heterogeneity tests.- If metrics conflict (primary positive but guardrail negative), recommend hold and deeper investigation before rollout.

This design balances statistical rigor, safety via guardrails and ramps, and concrete operational checks to trust the experiment outcome.

HardTechnical

32 practiced

Describe a production-safe procedure to detect and correct sample ratio mismatch (SRM) at scale across thousands of experiments. Include automated checks, priority rules for alerts, instrumentation tests, and how to surface SRM findings to experiment owners.

Sample Answer

Situation summary (goal): ensure across thousands of experiments we quickly detect sample ratio mismatch (SRM) and correct root causes without noisy false positives, while notifying owners and preventing biased results.

Automated checks (daily + real-time):- For each experiment compute observed vs expected assignment by randomization key using a binomial/chi-square test (choose binomial for 2-armed, chi-square for multi-armed). Use sequential testing with alpha-spending (e.g., alpha=0.001 per check) and Benjamini-Hochberg across experiments to control FDR.- Monitor assignment stability over short windows (hourly) and long windows (24h). Flag abrupt deviations (>3σ from historical mean) and persistent deviations (>48h).- Log raw assignment counts, randomization key entropy, and rejected bucketing errors.

Priority rules for alerts:- P0 (Immediate): overall assign fraction deviates >5 percentage points OR randomization key entropy drop >50% OR instrumentation error rate >1% → page engineering on-call.- P1: persistent but smaller drift (1–5 pp for >48h) → notify experiment owner + platform.- P2: transient blips (<6h) or low traffic → email/dashboard.

Instrumentation tests:- Canary synthetic experiments: inject known synthetic users with deterministic keys to verify routing.- End-to-end replay tests: sample production request logs and re-run bucketing code in CI to validate identical assignments.- Unit tests: assert idempotent hashing, consistent seeding, and backward compatibility of bucketing code.- Integration: healthcheck endpoint that returns current assignment distribution.

Remediation and runbooks:- Automated rollback of causal feature flags if P0 triggered.- If instrumentation bug: revert last deploy & run replay to confirm fix.- If traffic skew (CDN/A/B routing): route-fix with network team.

Surface findings:- Central SRM dashboard (filters by team, experiment, severity) showing test statistics, time-series, and raw counts.- Automated Slack/email notifications with summary, link to dashboard, recommended action, and runbook steps.- Weekly SRM report to platform and QA with trends and top recurring causes.

Example alert payload:- experiment_id, expected_split, observed_split, p_value (adjusted), window, owner, suggested_action, link_to_replay_logs.

Key reasoning: combine sensitive statistical checks with multiple-testing control to avoid noise, triage by impact, and use instrumentation + synthetic tests to quickly distinguish code/telemetry vs true traffic issues; surface clear owner-centric actions.

EasyTechnical

32 practiced

Explain last-touch and first-touch attribution models in marketing analytics. For each, describe a scenario where the model will likely mislead decision-making and one simple mitigation to reduce bias.

EasyTechnical

35 practiced

Explain the difference between instrumentation and telemetry in the context of product analytics. Give two examples of good event properties to capture for a user-click event, and one example of an anti-pattern in event instrumentation that causes analysis pain later.

HardSystem Design

33 practiced

Design a measurement and instrumentation plan to support gradual rollout of a feature via feature flags across multiple countries. Address assignment, logging of exposures, throttling, data joins across regions, and how to measure country-specific impacts while minimizing confounding from local campaigns.

Sample Answer

Requirements & constraints:- Gradual rollout by country with independent ramps, low-latency assignment, accurate exposure logging, ability to measure country-specific causal impact while controlling for local marketing/campaigns.

High-level plan:1. Assignment- Use deterministic hashing on stable user_id (or device_id) + feature_flag_id + country to produce bucket [0,1). This guarantees repeatable assignment and per-country independent randomization.- Record bucket value and treatment (on/off) alongside assignment_version.

2. Exposure & logging schema (events for every user interaction)- event_type: flag_exposure, relevant_action, campaign_exposure- user_id, anonymous_id, country, region, timestamp_utc, local_timestamp, timezone- flag_id, flag_version, assignment_bucket, treatment (bool), rollout_pct, rollout_timestamp- context: page, experiment_id, campaign_ids (from ad attribution)- session_id, device_type- ensure logs written to centralized event store (Kafka) and landed to data warehouse with raw event table + denormalized daily tables.

3. Throttling / rollout control- Control rollout via rollout_pct computed from bucket < pct. Enforce server-side caps per-country (max concurrency) and safety kill-switch via feature manager.- Implement staged ramps (1%, 5%, 20%, 50%, 100%) with automated health gates (error rate, key business metrics).

4. Data joins across regions- Use global unique user_id and consistent schema. Harmonize country codes and timezones during ETL.- Store country of assignment and country_of_activity to detect cross-country users; join on user_id for longitudinal metrics.- Use data partitioning by date and country for performant queries.

5. Measuring country-specific impacts & minimizing confounding- Analysis design: - Prefer within-country randomized control (treatment vs control buckets) per country. Where rollout is deterministic by bucket, control exists in each country. - Primary estimator: difference-in-means on key KPIs with clustered standard errors by user. - Complement with difference-in-differences (DiD) using pre-rollout baselines to control trend differences. - For countries without randomization, use synthetic control or propensity-score matched controls across similar countries.- Control for local campaigns: - Log campaign_exposure events and include campaign_ids as covariates. - Define exclusion windows around major local promotions (or include interaction terms). - Use regression adjustment: KPI ~ treatment + campaign_exposure + time_fixed_effects + user_covariates. - Consider interrupted time series / DiD using campaign exposure as control variable. - Pre-register primary metric and analysis plan; run sensitivity checks.

6. Monitoring, validation & stopping rules- Real-time dashboard: exposure rate by country, assignment drift, key metrics vs control, error/latency.- Automated anomaly detection: if adverse signal (p-value threshold or % delta) or system errors, auto-pause rollout.- Verify randomization: check balance of demographics/covariates across buckets per country.

7. Practical considerations- Retain raw logs for reprocessing; store mapping of flag_version to code-release.- Handle GDPR: anonymize PII, respect country opt-outs.- Provide analysts with templated SQL and precomputed aggregates (daily cohort tables) to speed country analyses.

This plan ensures deterministic, auditable assignment; complete exposure logging for joins; controlled throttling; and rigorous country-level causal measurement with protections against local-campaign confounding.

Unlock Full Question Bank

Get access to hundreds of Measurement Design and Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.