Product Metrics and Health Questions

Designing and using product specific metrics to measure user experience product health and business impact. Topics include identifying a north star metric and supporting metrics at company product and feature levels, measuring activation adoption engagement retention daily active users and monthly active users feature adoption rates and time to value, using product telemetry experimentation and funnel analysis to measure feature impact, and connecting product metrics to monetization and strategic objectives. Candidates should be able to propose metrics for new features justify trade offs instrument tracking and explain how product metrics inform prioritization roadmap and stakeholder alignment.

HardTechnical

119 practiced

A new premium feature is launched and you need to measure its impact on LTV while accounting for selection bias (early adopters may be higher-value). Propose a cohort-based analysis to estimate incremental LTV and explain how you'd handle censoring and differential follow-up times.

Sample Answer

Framework / goal:- Estimate incremental LTV attributable to the premium feature while removing selection bias from early adopters and handling censoring/different follow-up lengths.

Step 1 — Define cohorts- Treatment: users who adopted the premium feature within X days of launch.- Control candidates: users who didn’t adopt in that window.- Anchor time: for each user, set t=0 = feature-availability date for that user (launch date or personal signup date if later) so cohorts align on relative time.

Step 2 — Reduce selection bias- Build a propensity score model (logistic regression or gradient-boosted tree) predicting early adoption using pre-treatment features (historic spend, tenure, engagement, demographics).- Use propensity-score matching (1:1 or caliper) or inverse-probability-of-treatment weighting (IPTW) so treated and control have similar pre-treatment distributions.- Validate balance with standardized mean differences.

Step 3 — Estimate incremental LTV over time- Compute cumulative revenue per user in relative time buckets (week 1, 2, … up to T).- Use weighted averages (weights from IPTW) or matched pairs to get treatment vs control cumulative LTV curves.- Incremental LTV(t) = weighted mean LTV_treatment(t) − weighted mean LTV_control(t).

Step 4 — Handle censoring & differential follow-up- Users have different observation windows; treat revenue as time-to-event with right censoring.- Use survival-analysis techniques to adjust cumulative revenue: - Kaplan–Meier / Aalen-Johansen for retention probabilities. - Fit a parametric (Weibull/Exponential) or semi-parametric (Cox) model for churn and combine with conditional spend rates to project expected future revenue per user.- Alternatively, use the Bang–Tsiatis estimator or inverse-probability-of-censoring weights (IPCW) to correct mean cumulative revenue estimates under censoring.- When projecting beyond observed follow-up, transparently report assumptions and wide CIs.

Step 5 — Uncertainty and robustness- Compute bootstrap CIs for incremental LTV(t).- Do sensitivity analyses: different matching calipers, alternate propensity models, excluding power users, time-window variations.- Check heterogeneity: incremental LTV by segment (power users vs casual).

Implementation notes / SQL sketch- Cohort and pre-period features via CTEs; compute propensity scores offline (Python/R) then join weights back to SQL for weighted aggregations:

sql

WITH events AS (...), users AS (...)
SELECT user_id, relative_week, SUM(revenue) AS rev
FROM events
JOIN users ON ...
GROUP BY user_id, relative_week;

Key outputs to present:- Weighted cumulative LTV curves with bootstrap CIs- Incremental LTV at 30/90/365 days- Balance diagnostics and sensitivity table- Clear statement of censoring assumptions and projection method

Why this works:- Propensity weighting reduces selection bias using pre-treatment data; IPCW / survival models properly handle right-censoring so incremental estimates aren’t biased by different follow-up times. Combined, they give a defensible, transparent estimate of feature-driven LTV.

HardTechnical

69 practiced

You run many experiments each quarter. Propose a multi-metric decision framework that handles primary metrics, secondary metrics, and guardrails. Explain how you'd control for false discoveries when running dozens of tests, and provide a concrete policy for rolling out winners.

Sample Answer

Framework overview:- Define metric tiers up front: - Primary (1–2): business-critical (e.g., revenue per user, conversion). These drive the decision. - Secondary (2–4): supporting metrics that show mechanism or leading signals (e.g., engagement, click-thru). - Guardrails: safety metrics that must not degrade (e.g., error rate, page load, churn).

Decision rules (pre-registered):1. Success if primary metric shows statistically significant improvement (one-sided or two-sided as appropriate) and effect size ≥ minimum detectable effect (MDE) pre-specified.2. No serious degradation on any guardrail (no statistically significant negative change beyond a small-sample tolerance).3. Secondary metrics used to validate mechanism: at least one improves OR none conflict strongly with business logic.

Controlling false discoveries when running many tests:- Pre-register hypotheses and families of tests.- Use Benjamini–Hochberg FDR control across all primary tests within a quarter (q ≤ 0.05) to limit expected proportion of false positives.- For guardrails (safety critical), use a more conservative family-wise control (Bonferroni or Holm) to minimize any false negatives on safety signals.- Report adjusted p-values and confidence intervals; emphasize effect sizes and business impact, not p-value alone.

Concrete rollout policy:- Stage 1: Internal holdout (A/B) with full analytics and monitoring — requirement: pass framework above.- Stage 2: Gradual rollout to 10% of traffic for 1 week with automated alerting on guardrails and Bayesian sequential monitoring for early stopping.- Stage 3: 50% for 2 weeks if stable.- Stage 4: 100% if metrics hold and post-launch monitoring for 4 weeks.- Rollback rule: immediate rollback if any guardrail exceeds predefined threshold or if Bayesian posterior probability of harm > 95%.

Operational practices:- Automate pre-registration, p-value adjustment, and dashboards.- Include sample-size calculations and MDEs in experiment spec.- Communicate uncertainty and expected FDR to stakeholders; prioritize experiments by expected value to allocate statistical budget.

MediumTechnical

78 practiced

Explain uplift modeling and when it is preferable to randomized experiments. Describe the data inputs you would need to build an uplift model to target users for a retention campaign, and list two evaluation metrics for such a model.

Sample Answer

Uplift modeling (a.k.a. heterogeneous treatment effect or incremental modeling) predicts the causal effect of an action on each individual — i.e., how much more likely a user is to be retained if they receive an intervention versus if they don’t. Unlike standard propensity or response models that predict outcome under treatment only, uplift models directly estimate treatment minus control effect so you can target users who are positively influenced.

When preferable to randomized experiments:- Use uplift when an RCT is infeasible (costly, slow, or unethical) but you have historical quasi-experimental or logged A/B data.- Use when you need individualized targeting at scale (optimize spend by treating only those with positive uplift).- Note: RCTs remain gold standard; uplift complements RCTs by enabling personalized allocation after initial randomization validates causal assumptions.

Data inputs for a retention campaign uplift model:- Treatment indicator: binary flag whether user received the retention action (email/offer).- Outcome: retention label (e.g., active at 30 days) measured consistently.- Pre-treatment features: user demographics, engagement history (recency, frequency), product usage metrics, cohort, lifecycle stage.- Experiment metadata/timestamps: to ensure outcome window post-treatment and to control for temporal effects.- Instrumental or assignment variables if non-random assignment exists (to adjust / stratify).- Exclude post-treatment features and ensure sufficient overlap between treated and control.

Two evaluation metrics:1. Qini coefficient (or Qini curve / uplift at top decile): measures cumulative incremental gain when targeting top-ranked users by predicted uplift — practical for campaign ROI.2. AUUC (Area Under the Uplift Curve) or uplift@k: overall ranking quality of uplift predictions; uplift@k shows expected incremental retention when treating top k% users.

Practical notes: validate with holdout or randomized test set, check calibration of uplift estimates, and monitor business KPIs (cost per incremental retained user).

EasyTechnical

82 practiced

Describe what 'activation' means in product metrics. For a new social app, propose a concrete activation definition (example: user performs X within Y days of signup), justify the choice of X and Y, and list at least two edge cases you would handle when measuring activation.

Sample Answer

Activation is the moment a new user experiences core product value for the first time — the event(s) that predict higher retention and engagement. For a social app (discovery + messaging + feed), a concrete activation definition:

"User performs at least one of: follows 3 distinct accounts OR likes or comments on 2 distinct posts OR sends a direct message to another user, all within 7 days of signup."

Justification:- X (actions): Following 3 people shows network-building intent; interacting with 2 posts shows content engagement; sending a DM shows intent to communicate. Using an OR across related meaningful actions captures multiple paths to value.- Y (7 days): One-week window balances giving users time to explore while keeping the signal predictive of short-term retention; industry benchmarks often use 7–14 days, but 7 days reduces noise from delayed activation.

Measurement notes & example SQL:

sql

SELECT user_id,
       MIN(event_time) AS signup_time,
       MAX(CASE WHEN event_type='follow' THEN 1 ELSE 0 END) AS any_follow,
       COUNT(DISTINCT CASE WHEN event_type='follow' THEN target_user END) AS follow_count,
       COUNT(DISTINCT CASE WHEN event_type IN ('like','comment') THEN post_id END) AS engage_count,
       COUNT(DISTINCT CASE WHEN event_type='dm_sent' THEN target_user END) AS dm_count
FROM events
WHERE event_time >= signup_time AND event_time < signup_time + interval '7 days'
GROUP BY user_id;

Edge cases to handle:1) Timezone & clock skew: compute windows in UTC using signup timestamp canonicalized server-side; exclude negative or future-dated events.2) Bots / test accounts: filter by known QA/test flags, very-high-frequency patterns, or accounts created from same IP; optionally exclude invite flows or email-verified only.3) Duplicate/migration events: deduplicate events by event_id to avoid inflating counts when clients retry.4) Invite-driven activation: distinguish organic actions from ones triggered by invites or admin scripts if those shouldn't count.

I would validate this definition by A/B-ing or correlating activation with 30-day retention and iterate thresholds (e.g., follow_count 2 vs 3, window 7 vs 14 days) based on predictive power.

EasyTechnical

82 practiced

You have events(user_id, event_time, event_name). Write a SQL query to compute the funnel conversion from 'signup' -> 'onboard_complete' -> 'first_purchase' for users who signed up in the last 30 days. Return counts and conversion percentages between steps and overall conversion from signup to purchase.

Sample Answer

Approach:Filter users who had a 'signup' event in the last 30 days, then for each user find whether they completed the next funnel steps and count unique users at each step. Compute step-to-step and overall conversion percentages.

sql

WITH signup_users AS (
  -- users who signed up in the last 30 days, take earliest signup time per user
  SELECT user_id, MIN(event_time) AS signup_time
  FROM events
  WHERE event_name = 'signup'
    AND event_time >= now() - interval '30 days'
  GROUP BY user_id
),
user_steps AS (
  -- for each user, find the earliest time they completed each step after signup
  SELECT
    s.user_id,
    s.signup_time,
    MIN(CASE WHEN e.event_name = 'onboard_complete' AND e.event_time >= s.signup_time THEN e.event_time END) AS onboard_time,
    MIN(CASE WHEN e.event_name = 'first_purchase' AND e.event_time >= s.signup_time THEN e.event_time END) AS purchase_time
  FROM signup_users s
  LEFT JOIN events e
    ON e.user_id = s.user_id
  GROUP BY s.user_id, s.signup_time
),
aggregates AS (
  SELECT
    COUNT(*) FILTER (WHERE signup_time IS NOT NULL) AS signup_count,
    COUNT(*) FILTER (WHERE onboard_time IS NOT NULL) AS onboard_count,
    COUNT(*) FILTER (WHERE purchase_time IS NOT NULL) AS purchase_count
  FROM user_steps
)
SELECT
  signup_count,
  onboard_count,
  purchase_count,
  ROUND(100.0 * onboard_count / NULLIF(signup_count,0), 2) AS pct_signup_to_onboard,
  ROUND(100.0 * purchase_count / NULLIF(onboard_count,0), 2) AS pct_onboard_to_purchase,
  ROUND(100.0 * purchase_count / NULLIF(signup_count,0), 2) AS pct_signup_to_purchase
FROM aggregates;

Key points:- We anchor events relative to each user's signup_time to ensure correct order.- Use MIN(...) to capture first completion of each step.- NULLIF avoids division-by-zero; percentages rounded.Edge cases:- Users with multiple signups: we use earliest signup in last 30 days.- Events before signup are ignored.- If you want strict ordering (onboard must occur before purchase), confirm purchase_time >= onboard_time in user_steps filter.Alternative:- Use window functions or EXISTS-based counts if dataset is large; consider indexing on (event_name, event_time, user_id) for performance.

Unlock Full Question Bank

Get access to hundreds of Product Metrics and Health interview questions and detailed answers.

Join thousands of developers preparing for their dream job.