Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

HardTechnical

68 practiced

Given a table experiment_results(metric_name text, variant text, p_value numeric), write a Postgres SQL query that applies the Benjamini-Hochberg false discovery rate (FDR) correction and flags metrics with q_value < 0.05. Use window functions and explain each step briefly.

Sample Answer

Approach: implement Benjamini–Hochberg by (1) ranking p-values ascending, (2) computing raw q = p * m / rank, (3) enforcing monotonicity by taking the cumulative minimum of raw q from largest p to smallest, and (4) cap at 1 and flag q < 0.05. We use window functions for m and rank and an ordered window for the cumulative min.

SQL:

sql

WITH ranked AS (
  SELECT
    metric_name,
    variant,
    p_value,
    ROW_NUMBER() OVER (ORDER BY p_value ASC) AS rank,
    COUNT(*) OVER () AS m
  FROM experiment_results
),
raw_q AS (
  SELECT
    *,
    (p_value * m::numeric / rank) AS raw_q
  FROM ranked
),
adjusted AS (
  SELECT
    metric_name,
    variant,
    p_value,
    rank,
    m,
    raw_q,
    LEAST(1.0, MIN(raw_q) OVER (ORDER BY p_value DESC
                                ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)
         ) AS q_value
  FROM raw_q
)
SELECT
  metric_name,
  variant,
  p_value,
  q_value,
  (q_value < 0.05) AS significant
FROM adjusted
ORDER BY p_value;

Brief step explanations:- ranked: compute rank (i) and total tests (m).- raw_q: BH raw estimate q = p * m / i.- adjusted: enforce non-increasing q by taking cumulative MIN over p descending, and cap at 1.- final SELECT: return q_values and significance flag (q < 0.05).

MediumTechnical

66 practiced

A new feature requires explicit user consent to participate in the experiment, creating selection bias. How would you design the experiment to still get valid causal estimates? Discuss invitation design, encouragement design, instrumental variables, and analysis corrections.

Sample Answer

Situation: We need user opt‑in for a new feature, so simple A/B among volunteers gives selection bias and invalid causal estimates.

Design approach (high level): aim to randomize exposure or inducement to participate, then use instrumental-variable (IV) methods and analysis corrections to recover causal effects.

Invitation design:- Randomly sample users and send randomized invitations (treatment = invited, control = not invited). Invitations should be identical except for randomized element.- Track who was invited vs who opted in; invitation is an exogenous push that breaks direct selection on user attributes.- Pre-register primary metrics, subgroups, and intent-to-treat (ITT) estimands.

Encouragement design:- Use randomized encouragement intensity (e.g., email vs. in‑app prompt vs. no prompt). This increases take-up variation while preserving random assignment.- Vary timing/content in factorial design to learn what increases consent.

Instrumental variables:- Use invitation/encouragement as instrument Z for actual participation D to estimate Local Average Treatment Effect (LATE) via two-stage least squares: first stage D ~ Z (+ covariates), second stage outcome Y ~ D_hat.- Verify IV assumptions: relevance (check first-stage F-statistic), exclusion (justify that invitation affects outcome only via participation), monotonicity.

Analysis corrections:- Report ITT (effect of invitation) and IV/LATE (effect on compliers). Adjust for covariates to improve precision.- Use inverse-probability weighting or propensity score methods as sensitivity checks against residual selection bias.- Conduct balance checks on baseline covariates by instrument; run placebo outcomes.- Run robustness: bounds (Manski), complier characteristics, heterogeneous treatment effects, and simulate impact of violation of exclusion.

Operational notes for PM:- Coordinate with analytics/engineering to log invite, prompt, consent timestamps and rich covariates.- Pre-specify estimands, sample size (power accounting for take-up), and stopping rules.- Communicate that LATE applies to compliers; translate to business implications and consider follow-up experiments to generalize.

MediumSystem Design

74 practiced

Describe the critical elements of a metrics instrumentation and monitoring checklist for experiments. Include event schema contracts, idempotency, unique identifiers, real-time monitoring, and offline validation steps you would require before approving experiment launch.

Sample Answer

Goal: ensure experiment telemetry is reliable, timely, and trustworthy so product and data teams can make safe launch decisions.

Requirements & acceptance criteria:- All events conform to a shared schema; no missing critical fields; backwards-compatible changes only.- Events are deduplicable and idempotent.- Every user/event has stable unique identifiers for joining signals across systems.- Real-time alerts for pipeline failures or metric drift; offline validation within SLA before launch.

Instrumentation & monitoring checklist (pre-launch gate):1. Event schema contract- Documented schema (owner, version, field types, nullable rules, enums).- Contract tests integrated into CI that validate sample payloads.- Clear migration policy (additive changes only; deprecations with versioning).

2. Unique identifiers & identity stitching- Use stable user_id, experiment_id, allocation_id; include device/session ids.- Mapping strategy defined for anonymous→logged-in users; TTLs and privacy constraints noted.

3. Idempotency & deduplication- Include event_id and timestamp; enforce idempotency keys at ingestion.- Consumer-side dedupe logic and tests; SLA for max duplicate rate (<0.1%).

4. Real-time monitoring & alerts- Dashboard: event volume, ingestion latency, error rates, missing fields %, sample size by cohort.- Alerts: drops >20% vs baseline, latency >x sec, schema validation errors >0.5%.- Cost/ops guardrails (e.g., sampling controls).

5. Offline validation before approval- Run end-to-end replay test on staging: joinability of events, metric computations match golden dataset.- Sanity checks: cohort counts within expected range (±10%), treatment allocation parity, no post-exposure winsorization issues.- Statistical power check: projected detectable effect and sample completion estimates.

Roles & sign-off- Instrumentation owner (engineering): implements schema and idempotency.- Analytics owner (data): validates offline metrics, power, and joins.- PM: confirms business logic, monitors dashboards, approves go/no-go.- Security/privacy: confirms identifiers comply with policy.

Post-launch: automated canary for first 24–72h, daily validation checks, and a rollback plan tied to alert thresholds.

EasyTechnical

62 practiced

What are stopping rules in experiments and why is 'peeking' at p-values a problem? Provide two defensible stopping rules you would use in commercial product experiments and explain how each controls Type I error or business risk.

EasyTechnical

63 practiced

As a product manager at a streaming service you notice weekly retention dropped 3% after a UI change. Explain the difference between a hypothesis and a guess, then reframe this observation into a clear, testable hypothesis using the if...then...because structure. Include a measurable primary metric and one guardrail you would monitor.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.