Growth and Product Metrics Analysis Questions
Analysis skills specific to growth and product contexts: interpreting funnel metrics, cohort and retention analyses, attribution of acquisition versus activation, detecting seasonality and external event impacts, and diagnosing conversion or engagement changes. Candidates should be able to form hypotheses about what drove changes, propose targeted follow up analyses or A B tests, and identify which additional metrics are needed to evaluate unit economics and growth efficiency.
HardTechnical
45 practiced
Design an anomaly detection and alerting system for key product KPIs (e.g., conversion, DAU, revenue) that minimizes false positives during known seasonality and active experiments. Describe the detection algorithms, how to incorporate experiment and calendar metadata, alert routing, and feedback loops to tune alerts over time.
Sample Answer
Requirements & goals:- Detect true anomalies in KPIs (conversion, DAU, revenue) while minimizing false positives from seasonality and running experiments.- Fast detection for urgent issues, low noise for business teams.- Integrate experiment and calendar metadata, route alerts to correct owners, and learn from feedback.Detection architecture (hybrid, layered):1. Baseline & seasonality removal- Maintain per-metric, per-cohort baselines using time-series decomposition (STL/seasonal_decompose) or Prophet to model daily/weekly/holiday effects and trend. Store expected value and prediction interval (e.g., 95% CI) at hourly/daily granularity.2. Short-term anomaly detectors- Fast detectors for operational incidents: EWMA + robust z-score on residuals (observed - expected)/MAD for low-latency alerts.- Change-point detection (PELT or Bayesian online change point) for abrupt structural shifts.3. Statistical-significance & impact test- For flagged anomalies, run an impact assessment: estimate absolute and relative delta, p-value (bootstrap or permutation on residuals) and business impact (e.g., lost revenue).4. Ensemble decisioning- Combine detectors with weighted voting and require agreement or high-impact threshold to reduce false positives.Experiment & calendar metadata handling:- Ingest experiment metadata: active experiments, cohorts, traffic splits, start/end times, feature tags.- Two approaches: a) Isolation: compute baselines per cohort (exposed vs control); compare within-cohort to avoid confounding. b) Adjustment: remove expected experiment effect by modeling experiment as covariate in forecasting model; use uplift estimates to normalize metric.- Calendar metadata: holiday flags, planned campaigns, deploy windows. During known events, widen prediction intervals or suppress low-impact alerts. Allow scheduled overrides (e.g., blackout windows for marketing campaigns).Alert routing & prioritization:- Enrich alert with context: metric, granularity, expected vs observed, p-value, impact estimate, active experiments/deploys, recent changes, playbook link.- Route to: - Primary owner (product/metric owner) - Experiment owner(s) if experiment overlaps cohort - On-call SRE for infra-related signals - Stakeholder escalation list for high-impact alerts- Prioritize with severity tiers: - P0: large revenue/availability hits → page + Slack + pager - P1: medium impact → Slack + email - P2: informational → dashboard notificationSuppression & safety rules:- Suppress alerts for windows explicitly marked as planned (campaigns, deploys).- Suppress if anomaly aligns entirely with experiment cohort and control shows no shift (i.e., expected).- Implement “cool-down” windows to avoid repeated alerts for same root cause.Feedback loop & continuous tuning:- Human-in-the-loop labeling: allow owners to mark alerts as true/false/expected (reason tags: experiment, holiday, noise, bug).- Store labels to: - Retrain models (adjust seasonality, thresholds) - Update suppression rules and experiment effect priors - Calibrate detector weights in ensemble using supervised learning (e.g., gradient boosting on features: z-score, duration, impact, experiment overlap)- Metrics for monitoring alert quality: precision, recall, mean time to acknowledge, time to resolution, false-positive rate per metric. Target thresholds and periodic review.- A/B test threshold changes in a shadow environment before rolling out.- Automated adaptation: gradually shrink prediction intervals where historic false positives occur, with human approval.Operational & product considerations:- Provide an alerts dashboard showing active alerts, historical labels, experiment overlap, root-cause playbooks.- Version control for detection config, experiment metadata, and holiday calendars.- Governance: define owners for metrics, SLA for responses, and a review cadence (weekly) to prune noisy detectors.- Privacy & sampling: when using user-level data, respect sampling/PII guidelines; use aggregated signals when necessary.Why this design:- Modeling seasonality removes predictable variance; cohort-aware experiment handling avoids confounding; ensemble + impact checks reduce spurious alerts; human feedback closes the loop so system improves over time while keeping teams focused on real issues.
MediumTechnical
52 practiced
Write a SQL query or describe the steps to produce weekly cohort retention curves for 12 weeks using PostgreSQL given users(user_id, created_at) and events(user_id, event_type, occurred_at). The output should show cohort_week, week_number (0..11), cohort_size, retained_users, and retention_rate.
Sample Answer
Approach: create weekly cohorts by users' signup week, then for each cohort count how many unique users perform any event in each week 0..11 after signup. Use date_trunc('week') for cohort_week and compute week_number as weeks between event week and cohort week. Aggregate and compute retention_rate.Key points:- cohort_week is start of signup week.- week_number = difference in weeks between event_week and cohort_week; week 0 counts any activity in the signup week.- Use DISTINCT user counts to avoid double-counting multiple events in same week.- Edge cases: users with no events (retained_users = 0 for weeks >0), timezone handling for timestamps, choosing week start (Postgres week uses Monday by default).
sql
WITH users_cohort AS (
SELECT
user_id,
date_trunc('week', created_at)::date AS cohort_week
FROM users
),
events_week AS (
SELECT
e.user_id,
date_trunc('week', e.occurred_at)::date AS event_week
FROM events e
WHERE e.occurred_at >= (SELECT min(created_at) FROM users) -- optional filter
),
user_activity AS (
-- join users to events and compute week offset relative to cohort
SELECT
u.user_id,
u.cohort_week,
EXTRACT(EPOCH FROM (ew.event_week - u.cohort_week)) / 86400 / 7 AS week_number_float
FROM users_cohort u
LEFT JOIN events_week ew ON u.user_id = ew.user_id
),
user_week AS (
-- convert to integer week number and keep only 0..11
SELECT
user_id,
cohort_week,
FLOOR(COALESCE(week_number_float, 0))::int AS week_number
FROM user_activity
WHERE FLOOR(COALESCE(week_number_float, 0)) BETWEEN 0 AND 11
GROUP BY user_id, cohort_week, FLOOR(COALESCE(week_number_float, 0))
),
cohort_sizes AS (
SELECT
cohort_week,
COUNT(DISTINCT user_id) AS cohort_size
FROM users_cohort
GROUP BY cohort_week
),
cohort_retention AS (
SELECT
uw.cohort_week,
uw.week_number,
COUNT(DISTINCT uw.user_id) AS retained_users
FROM user_week uw
GROUP BY uw.cohort_week, uw.week_number
)
SELECT
cs.cohort_week,
gs.week_number,
cs.cohort_size,
COALESCE(cr.retained_users, 0) AS retained_users,
ROUND(COALESCE(cr.retained_users,0)::numeric / cs.cohort_size, 4) AS retention_rate
FROM cohort_sizes cs
-- generate rows for weeks 0..11 to ensure full matrix
CROSS JOIN LATERAL (
SELECT generate_series(0,11) AS week_number
) gs
LEFT JOIN cohort_retention cr
ON cr.cohort_week = cs.cohort_week AND cr.week_number = gs.week_number
ORDER BY cs.cohort_week, gs.week_number;EasyTechnical
45 practiced
Explain cohort analysis and the difference between time-based cohorts and behavior-based cohorts. Using an example, describe how you would use weekly signup cohorts to evaluate whether a new onboarding flow improved 4-week retention versus the old flow. Explain what metrics you would compute and why.
Sample Answer
Cohort analysis groups users by a shared attribute (e.g., signup week, acquisition source, or behavior) and tracks their metrics over time to reveal retention, engagement, or monetization trends that aggregate metrics hide.Time-based cohorts vs behavior-based cohorts:- Time-based cohorts: users grouped by when they did something (e.g., signed up in Week 1). Good for measuring lifecycle/retention and the effect of product changes launched at known times.- Behavior-based cohorts: users grouped by actions they took (e.g., completed onboarding, made first purchase). Good for causal links between behavior and outcomes.Example: weekly signup cohorts to evaluate a new onboarding flow’s effect on 4-week retention- Setup: Define cohorts by signup week (Week A = old flow, Week B = new flow; if rollout is A/B by week or phased).- Metric(s) to compute: 1. 4-week retention rate = % of users from the cohort active in week 4 (primary outcome). 2. Weekly retention curve (week1, week2, week3, week4) to see trajectory. 3. Activation rate (e.g., completed onboarding steps) as a mediator. 4. Cohort size and conversion funnel to check selection bias. 5. Confidence intervals / statistical test (chi-square or z-test) to assess significance.- Why: 4-week retention directly measures the product stickiness we aim to improve; weekly curves show whether gains are immediate or decay over time. Activation explains mechanism. Statistical tests ensure observed differences aren’t noise.Practical checks: ensure comparable cohorts (similar acquisition channels), control for seasonality, and run sensitivity checks (segment by device, geography). If non-comparable, use randomized A/B or behavior-based cohorts (users who saw new flow vs not) for stronger causal inference.
HardTechnical
45 practiced
You recommend pausing a marketing campaign that increased acquisition by 40% but decreased conversion rate and increased CAC/paid conversions. As the product manager presenting to executives, outline the one-page decision brief you would prepare showing trade-offs, the metrics and visualizations you would include, recommended actions, and the risks of each option.
Sample Answer
Context: Over the last 4 weeks Marketing A ran a paid campaign that increased new user acquisition by +40% but coincided with a drop in on-site conversion rate (from 3.2% → 2.1%) and a 25% increase in CAC per paid conversion. Budget spend = $X/week.Decision question: Pause, optimize, or scale the campaign?Key metrics (current vs baseline)- New users (w/w): +40%- Conversion rate (A→P): 3.2% → 2.1%- CAC / paid conversion: $30 → $37.5 (+25%)- LTV:CAC ratio: 3.0 → 2.4- Quality signals: 30% higher bounce rate, average session duration −22%- Cohort 30-day retention: preliminary cohort down 15%Visualizations (1-page)- Top-left: KPI trend sparkline (Acquisition, Conversion Rate, CAC) last 8 weeks- Top-right: Waterfall showing traffic → activation → paid conversion drop- Mid: Cohort retention comparison (baseline vs campaign cohort)- Bottom: LTV vs CAC scatter and revenue impact sensitivity (best/worst case)Options & trade-offs1) Pause campaign now- Pros: Stop cash bleed, protect conversion funnel, buys time to diagnose- Cons: Lose volume/market share; stops learning about new segments2) Throttle & experiment (recommended)- Pros: Reduce spend to breakeven, run A/B tests on landing, audience, creatives, and funnel fixes; preserve some volume- Cons: Slower data collection; requires close coordination with marketing/analytics3) Scale aggressively- Pros: Max volume, potential rapid acquisition- Cons: Amplifies poor quality traffic, worsens CAC and retention, risks channel overspendRecommendation (1–2 sentence)Throttle to 40–60% spend immediately, pause underperforming audiences/creatives, and run a 14-day test plan: (a) landing page variants focused on intent messaging, (b) refined targeting excluding low-intent segments, (c) adjust bid strategy to maximize conversions not clicks. Continue only if CAC falls < $32 and 30-day retention gap closes to <5%.Risks & mitigations- False negative (pausing loses high-LTV users): mitigate via targeted holdout groups and extended LTV tracking.- Execution risk (tests poorly instrumented): assign analytics owner, QA events, and weekly gating criteria.- Competitive churn: monitor share and be ready to scale if signals improve.Next steps (owners & timeline)- Day 0: Reduce spend to 50% (Marketing)- Day 1–3: Implement tracking and variant pages (Eng + Analytics)- Day 3–17: Run tests, daily dashboards, weekly exec check-ins (PM)- Day 18: Decision meeting against pre-registered success metricsAsk: Approval to throttle and fund the 14-day optimization plan ($Y) and analytics support.
HardTechnical
39 practiced
Write a PostgreSQL approach (SQL or pseudocode) to compute a Kaplan-Meier survival curve for weekly retention from raw events: users(user_id, created_at) and events(user_id, occurred_at). The output should include week_number, at_risk_count, events_count, survival_probability. Explain edge cases and how you handle censoring.
Sample Answer
Approach summary:- For each user define cohort week (signup week) and measure time-to-first-return-event in whole weeks since signup. Users with no event within observation window are censored at the last observed week.- Compute, for each week t, the number at risk (users who have not yet had the event and are still observable at t), the number of events in that week (first-events occurring in week t), and Kaplan–Meier survival as product over weeks of (1 - events_at_t / at_risk_t).SQL (Postgres) — adjust date ranges/window as needed:Key reasoning and details:- We use time-to-first-return-event per user because retention typically measured as time until first return; multiple returns are irrelevant for KM.- At-risk at week t: users whose time-to-event >= t (they haven’t yet had the event before t and are still observable).- Events_count: users whose first-event occurs in that week.- Survival: product over weeks of (1 - d_t / n_t). Implemented via cumulative sum of log to avoid floating underflow.Edge cases and censoring:- Users with no events in window are right-censored at their last observable week (max_followup_weeks). They contribute to at-risk counts until censor week but not to events.- Zero at-risk: avoid division by zero — we use NULLIF(at_risk,0) and coalesce survival to 1 when undefined.- Events occurring the same week as signup: countable (week 0) — decide based on product definition; if you want initial retention excluding week 0, shift week indexing.- Multiple events: use first-event only.- Time granularity: we truncate to calendar weeks; consider timezone consistency and business definition (calendar week vs 7-day windows).- Left-truncation: if only including users after start_date, ensure users' follow-up window is consistent; if users join before observation window, handle separately.Notes for product analysis:- Choose cohort definition (calendar-week signup or rolling) and observation window (how many weeks to follow).- Report confidence intervals (Greenwood's formula) for KM if needed for statistical significance.- For easier business interpretation, you may report 1 - survival as cumulative retention.
sql
-- Parameters
-- :start_date, :end_date define observation window (e.g., cohort signup window and max follow-up)
WITH cohorts AS (
SELECT
u.user_id,
date_trunc('week', u.created_at)::date AS cohort_week,
generate_series(0, floor(EXTRACT(epoch FROM (:end_date::timestamp - u.created_at::timestamp))/604800)::int) AS max_followup_weeks
FROM users u
WHERE u.created_at >= :start_date AND u.created_at < :end_date
),
first_event AS (
-- first event after signup (exclude events before signup)
SELECT
c.user_id,
c.cohort_week,
MIN(date_trunc('week', e.occurred_at) - date_trunc('week', c.cohort_week)) / 7 AS week_of_first_event
FROM cohorts c
LEFT JOIN events e
ON e.user_id = c.user_id
AND e.occurred_at >= c.cohort_week
AND e.occurred_at <= :end_date
GROUP BY c.user_id, c.cohort_week
),
user_time AS (
-- compute time-to-event in weeks (NULL => no event observed => censored at max_followup_weeks)
SELECT
c.user_id,
c.cohort_week,
CASE
WHEN fe.week_of_first_event IS NOT NULL THEN CAST(fe.week_of_first_event AS int)
ELSE c.max_followup_weeks
END AS time_weeks,
CASE WHEN fe.week_of_first_event IS NULL THEN true ELSE false END AS censored
FROM cohorts c
LEFT JOIN first_event fe USING (user_id, cohort_week)
),
weeks AS (
-- possible week indices to report (0 = signup week)
SELECT generate_series(0, (SELECT MAX(max_followup_weeks) FROM cohorts)) AS week_number
),
km_counts AS (
-- for each week, count at-risk and events (first-event exactly at that week)
SELECT
w.week_number,
COUNT(*) FILTER (WHERE ut.time_weeks >= w.week_number) AS at_risk_count,
COUNT(*) FILTER (WHERE ut.time_weeks = w.week_number AND NOT ut.censored) AS events_count
FROM weeks w
CROSS JOIN (SELECT DISTINCT user_id, time_weeks, censored FROM user_time) ut
GROUP BY w.week_number
ORDER BY w.week_number
),
km AS (
SELECT
week_number,
at_risk_count,
events_count,
-- running product for survival: survival(0)=1, then multiply (1 - d/n) each week
exp(sum(ln(1.0 - events_count::float / NULLIF(at_risk_count,0))) OVER (ORDER BY week_number ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)) AS survival_probability
FROM km_counts
)
SELECT week_number, at_risk_count, events_count,
COALESCE(survival_probability, 1.0) AS survival_probability
FROM km;Unlock Full Question Bank
Get access to hundreds of Growth and Product Metrics Analysis interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.