Product Metrics and Key Performance Indicators Questions

Covers designing, implementing, and governing metric frameworks for products. Topics include defining a north star metric that aligns the organization, identifying supporting and diagnostic metrics that drive and explain the north star, and understanding metric types such as engagement, retention, monetization, and quality. Candidates should be able to discuss metric hierarchies, instrumentation and data pipeline considerations, segmentation and cohort analysis, and the use of metrics for experimentation and decision making. Governance topics include ownership, alerting and anomaly detection, preventing metric manipulation, establishing thresholds and statistical rigor, retiring obsolete metrics, and balancing business and product analytics needs across stakeholders.

EasyTechnical

64 practiced

You have an events table:

events(user_id bigint, event_type text, occurred_at timestamptz)

. Write a SQL query (Postgres) to compute DAU, MAU and DAU/MAU (stickiness) for the most recent day in the table. State assumptions about timezones, deduplication, and users with multiple events per day.

Sample Answer

Approach: pick the table’s most recent calendar day (in a chosen timezone), count distinct users active on that day (DAU), count distinct users active in the preceding 30-day window ending that day inclusive (MAU), and compute DAU/MAU. Deduplicate by user_id (one user counts once per day). Assume timestamps are stored as timestamptz; convert to a fixed timezone (UTC or your business timezone) before taking the date.

Assumptions:- Use UTC for date boundaries (change 'UTC' to e.g. 'America/Los_Angeles' if needed).- A user with multiple events in a day is counted once (dedup by user_id).- MAU is distinct users in the 30-day window including the most recent day.- No explicit bot filtering — add WHERE to exclude if needed.

Query (Postgres):

sql

WITH params AS (
  -- determine most recent calendar date in chosen timezone
  SELECT (max(occurred_at AT TIME ZONE 'UTC')::date) AS recent_day
  FROM events
),
dau AS (
  SELECT COUNT(DISTINCT user_id) AS dau_count
  FROM events
  CROSS JOIN params
  WHERE (occurred_at AT TIME ZONE 'UTC')::date = params.recent_day
),
mau AS (
  SELECT COUNT(DISTINCT user_id) AS mau_count
  FROM events
  CROSS JOIN params
  WHERE (occurred_at AT TIME ZONE 'UTC')::date BETWEEN (params.recent_day - INTERVAL '29 days') AND params.recent_day
)
SELECT
  params.recent_day,
  dau.dau_count AS dau,
  mau.mau_count AS mau,
  CASE WHEN mau.mau_count = 0 THEN NULL
       ELSE ROUND( (dau.dau_count::numeric / mau.mau_count) , 4)
  END AS dau_over_mau
FROM params
CROSS JOIN dau
CROSS JOIN mau;

Key points:- Use DISTINCT user_id to dedupe.- Use date boundaries in a consistent timezone to avoid partial-day errors.- Adjust the MAU window (30 vs 28/31 days) per business definition.

MediumTechnical

83 practiced

Design an alerting rule that reduces false positives by combining multiple signals (e.g., conversion rate drop + traffic source shift + increased 5xx errors). Describe the logical/algorithmic approach (AND/weighted-score), thresholds, and how you'd measure alert precision/recall over time.

Sample Answer

Approach: treat alerting as a binary classifier that must combine heterogeneous signals to reduce false positives. I propose a hybrid: per-signal anomaly detectors + a weighted-score decision rule with an optional hard AND for critical signals.

Signals and preprocessing:- Conversion rate (CR) drop: compute week-over-week and short-window z-score on rolling 24h CR; signal value s1 = max(0, z) scaled to [0,1].- Traffic source shift: KL-divergence or Earth-Mover’s Distance between current vs. baseline source distribution; s2 = normalized distance in [0,1].- 5xx error increase: relative change and Poisson test on counts; s3 = p-value→score mapped to [0,1].

Decision logic:- Hard requirement: s3 > 0.6 (increased server errors) OR (s1 and s2 combined) — errors are critical.- Otherwise, compute weighted score S = w1*s1 + w2*s2 + w3*s3 where w sum=1. Trigger alert when S >= T.- Initial weights: w1=0.5 (CR), w2=0.3 (traffic), w3=0.2 (errors) — because CR reflects business impact but errors are binary-critical via hard gate.- Threshold T: start at 0.6; use retrospective tuning on labeled incidents.

Threshold selection & tuning:- Use historical labeled incidents (postmortems) to grid-search weights and T maximizing F1 or precision at a target recall. If labels scarce, augment via weak supervision (on-call notes) or simulate injected faults.

Measuring precision/recall over time:- Treat each alert window as a prediction; maintain a rolling evaluation (30–90 days) computing precision, recall, F1, and time-to-detect.- Track false-positive rate (alerts with no corresponding incident within x hours) and false-negative events (incidents without alert).- Maintain confusion matrix; plot precision-recall curve by varying T; pick operating point by cost (cost_fp vs cost_fn).

Operational feedback loop:- Collect labels from on-call: mark alerts as true/false and annotate root cause.- Retrain/tune weights monthly or when precision drops below SLA.- Add cooldowns, suppression when known experiments or deployments are active (ingest release metadata).- Provide explainability in alerts: show contributing signal scores and top factors to speed triage.

Trade-offs:- AND reduces false positives but risks missed events; weighted score allows graceful degradation.- Hard error gate ensures critical infrastructure failures always alert.

This design balances sensitivity to business impact (CR) with robustness to traffic noise and critical error signals, and uses continuous evaluation to keep precision/recall within targets.

MediumTechnical

91 practiced

Design two dashboards for the same product: one for executives and one for product managers. Specify which metrics, visualizations, and drill-downs each dashboard should include and justify choices based on audience needs and decision timelines.

Sample Answer

Requirements & constraints:- Same product, two audiences with different time horizons and decisions: Executives (strategy, quarterly) vs Product Managers (tactical, weekly-sprint).- Dashboards must support quick signal detection and deeper root-cause drill-downs.

Executive dashboard (high-level, ~KPI-at-a-glance, weekly/quarterly decisions)- Key metrics: Revenue (MRR/ARR), MAUs/DAUs, Growth rate (QoQ), CAC, LTV, Churn rate, NPS, Feature adoption % (top 3).- Visualizations: KPI tiles for topline numbers (with sparkline for trend), cohort growth chart, funnel summary (conversion rates), heatmap of geography revenue.- Drill-downs: Click KPI → opens time-series with annotation (campaigns, releases); Click churn → cohort retention table and top reasons (survey/qual); Click region → list of major accounts.- Justification: Executives need concise signals and strategic levers (growth, unit economics, retention) to allocate investment and set targets.

Product manager dashboard (operational, ~daily/weekly decisions)- Key metrics: Active users by segment, retention curves (D0, D7, D30), feature usage (events per user), funnel step conversion, performance/latency, bug/error rate, experiment results (A/B).- Visualizations: Segmented retention cohort charts, Sankey/funnel visualization, time-series and distribution plots, table of top users by usage, AB test summary with statistical significance.- Drill-downs: Click feature → session-level event stream and user cohorts; Click funnel drop-off → session replay/sample users and error logs; Click experiment → variant-level metrics and confidence intervals.- Justification: PMs need diagnostic detail to prioritize backlog, iterate on features, and decide quick fixes vs experiments. Include tooling links (analytics queries, JIRA tickets).

Shared design considerations- Freshness: Execs: daily/weekly snapshots; PMs: near real-time.- Access & export: PDF summaries for exec meetings; CSV/export & deep-link for PM investigation.- Alerts: Threshold-based alerts for both but tuned differently (exec: major revenue or churn spikes; PM: funnel regressions, test failures).

HardTechnical

74 practiced

You're running an experiment but worry about interference/network effects (users influence each other). Describe experimental designs that address interference (e.g., cluster randomization, geographic holdouts, network-aware randomization), trade-offs, and how you'd analyze results.

Sample Answer

Situation / goal: When users influence each other, the standard SUTVA-based A/B test is biased. The aim is to get unbiased estimates of direct, indirect, and total effects while balancing logistics and power.

Design options:- Cluster randomization: Randomize groups (e.g., households, teams) instead of individuals. Pros: removes within-cluster interference; simple analysis (cluster-level summaries). Cons: fewer independent units → lower power; requires good cluster definitions to contain spillovers.- Geographic or market holdouts: Randomize by regions/markets. Pros: operationally easy, contains many spillovers. Cons: heterogeneity across geos (need stratification), fewer clusters → lower degrees of freedom.- Network-aware randomization: - Graph cluster randomization / community-based clustering (e.g., METIS, Louvain): partition social graph into clusters, randomize clusters. Balances spillover containment and cluster size. - Edge- or exposure-based designs: randomize treatment probability per node or edges (two-stage: assign seed nodes then propagate). Enables estimation of direct vs. indirect effects. - Rerandomization / constrained randomization to balance network statistics across arms.

Trade-offs:- Bias vs. variance: tighter containment (large clusters) reduces bias but reduces number of independent units → higher variance.- Practicality: operational constraints may force geo or simple clusters.- Assumptions: exposure models (e.g., spillover limited to 1-hop) simplify analysis but risk misspecification.

Analysis approach:- Define estimands up front: direct effect (treated node effect holding others fixed), spillover/indirect effect, total effect.- Use randomization-based inference (permutation tests) aligned to design to get valid p-values; estimate effects with Horvitz–Thompson or inverse-probability weighting when assignment probabilities vary.- Fit generalized linear mixed models or GEE with cluster-robust SEs for covariate adjustment, but rely on randomization inference for validity.- For network designs, use exposure mapping: classify units by their exposure (e.g., fraction of neighbors treated) and estimate dose–response; use instrumental-variable logic when treatment assignment to neighbors is randomized.- Report ITT and complier-average effects, confidence intervals via bootstrap or permutation, and sensitivity analyses (vary assumed spillover radius, alternative clusterings).

Diagnostics and robustness:- Check balance on network metrics across arms.- Measure spillover empirically (treatment effect as function of neighbor-treatment fraction).- Run placebo tests on pre-period outcomes.- Power: simulate using observed network to estimate detectable effects under candidate designs.

This combined design + analysis pipeline preserves causal validity under interference while acknowledging trade-offs between bias, power, and operational feasibility.

EasyTechnical

68 practiced

Design a lightweight metric catalog entry for a KPI (e.g., weekly active subscribers). Include canonical SQL, business definition, owner, freshness SLA, known caveats, dependencies (upstream tables/events), and recommended visualizations. Explain why each field is important.

Sample Answer

Metric: Weekly Active Subscribers (WAS)

Business definition:Number of distinct paying subscribers who had at least one product-active event (login, content view, or billing activity) in the trailing 7-day window ending on report_date. Excludes trial-only accounts.

Owner:Analytics / Subscription Metrics — data.owner@example.com

Freshness SLA:Daily refresh by 04:00 UTC; retrospective backfill within 24 hours for late events.

Canonical SQL:

sql

WITH recent_events AS (
  SELECT user_id, event_time::date as event_date
  FROM events.product_activity
  WHERE event_time >= (current_date - interval '7 days')
    AND event_type IN ('login','content_view','billing')
),
paying_users AS (
  SELECT user_id
  FROM subscriptions
  WHERE status = 'active' AND plan != 'trial'
)
SELECT
  current_date AS report_date,
  COUNT(DISTINCT re.user_id) AS weekly_active_subscribers
FROM recent_events re
JOIN paying_users p USING (user_id);

Dependencies:- events.product_activity (event_time, user_id, event_type)- subscriptions (user_id, status, plan)- user_id canonicalization / identity mapping

Known caveats:- Late-arriving events may temporarily undercount until backfill completes.- Users with multiple accounts mapped to one person may inflate count unless identity mapping applied.- Definition excludes trials; business may want alternate metric including trials.

Recommended visualizations:- Daily time series (7-day rolling) to show trend- Week-over-week percent change bar chart- Cohort retention overlay to contextualize activity- Drill-down by plan tier and region

Why each field matters:- Business definition: ensures analysts/product interpret metric consistently.- Canonical SQL: single source of truth for reproducibility and automation.- Owner: clear accountability for fixes/questions.- Freshness SLA: sets expectations for consumers and alerting.- Dependencies: clarifies upstream quality risks and change impact.- Known caveats: prevents misinterpretation and guides appropriate use.- Visualizations: recommended ways to surface trends and root causes.

Unlock Full Question Bank

Get access to hundreds of Product Metrics and Key Performance Indicators interview questions and detailed answers.

Join thousands of developers preparing for their dream job.