Metrics and KPI Fundamentals Questions

Core principles and practical fluency for defining, measuring, and interpreting metrics and key performance indicators, applicable across any professional domain. Candidates should be able to select meaningful metrics aligned to business objectives rather than vanity metrics, explain the difference between a metric and a target, and distinguish leading indicators from lagging indicators. Coverage includes decomposing complex outcomes into actionable component metrics, writing precise metric definitions (for example what counts as an active user, a completed case, a qualified lead, or a resolved ticket, depending on the domain), calculating common rate-based metrics such as engagement rate, churn rate, conversion rate, cycle time, or utilization rate, establishing baselines and sensible targets, and interpreting signal versus noise including awareness of statistical variability. Also includes using segmentation and cohort analysis to diagnose metric movements, and recommending two to three meaningful metrics for a hypothetical problem in the candidate's own domain with justification and action plans.

EasyTechnical

62 practiced

An e‑commerce funnel shows 5,000 unique site visitors, 400 email signups, and 80 purchases in a day. Calculate (a) visitor→purchase conversion rate and (b) signup→purchase conversion rate. Which conversion would you present to the CEO and why?

HardSystem Design

62 practiced

Design an event-based metrics pipeline for a product analytics platform that ingests 1M events/minute, provides near-real-time dashboard freshness (<=60s), and supports historical backfills. Describe architecture components (ingest, stream processing, storage), aggregation strategy, accuracy vs latency trade-offs, and monitoring/backfill approach.

Sample Answer

Requirements clarification:- Ingest 1M events/min (~16.7k/sec), dashboard freshness <=60s, support historical backfills, high availability, and reasonable cost.- SLAs: 99.9% availability for dashboards, eventual consistency for backfills.

High-level architecture:- Edge clients → API Gateway / Kafka Producer → Ingestion (Kafka/Confluent or Pulsar) → Stream processing (Flink/Beam/ksqlDB) → Serving/storage: - Real-time store: OLAP/TSDB for recent data (ClickHouse, Druid, or ClickHouse + materialized views) optimized for sub-60s rollups - Long-term: Partitioned Parquet on S3 / Data Lake + Hive/Trino for historical queries- BI layer: Pre-aggregated materialized views + query engine (e.g., Superset/Metabase)

Ingest:- Frontend clients batch and compress; API gateway with rate-limiting and schema validation (Avro/Protobuf + schema registry).- Kafka partitions sized to handle throughput and parallelism.

Stream processing & aggregation:- Two-tier processing: 1. Real-time micro-aggregations in streaming engine (per-second windows, session/windowing) producing low-latency increments to real-time store. 2. Continuous rollups to coarser buckets (1m,5m,1h) written to both real-time and long-term storage.- Maintain exactly-once or idempotent writes using deduplication keys and watermarking.

Accuracy vs latency trade-offs:- Low-latency path: approximate, incremental aggregates (mergeable sketches like HyperLogLog, t-digest) for fast dashboards (<60s) with bounded error.- High-accuracy path: batch jobs on Data Lake to recompute exact aggregates overnight or on-demand (backfill), reconciling differences with reconciliation pipelines.- Offer dashboard tiers: “Live (approx.)” vs “Final (exact)” with visible confidence/latency.

Backfill & historical:- Backfill jobs read raw events from persisted Kafka or S3 (sink). Or replay via compacted topic. Use scalable Spark/Flink batch to recompute aggregates, mark reconciled windows, and update materialized views; track progress in a backfill metadata store.

Monitoring, observability & SLOs:- Metrics: ingestion rate, lag (consumer offsets), processing latency, error rates, aggregation divergence (compare stream vs batch), storage health.- Alerts on lag > 30s, error spikes, or reconciliation delta > threshold.- Dashboards for data quality (missing partitions, schema drift), per-tenant quotas.

Operational considerations:- Capacity plan: headroom for peak ~3x; autoscale stream processors.- Cost trade-offs: retention in real-time store vs cheaper S3.- Security: encryption, IAM, PII redaction at ingest.

This design balances product needs: sub-60s actionable insights via approximate real-time aggregates, full accuracy through batch recomputation and backfills, with clear SLAs and observability so product and business stakeholders can trust and iterate on metrics.

MediumTechnical

66 practiced

You want to detect a 5% relative lift in conversion with 80% power and alpha=0.05. Describe step‑by‑step how you would estimate the required sample size. What inputs do you need, what assumptions must you validate, and how would you adjust for multiple metrics or sequential testing?

Sample Answer

Step‑by‑step approach

1) Define inputs- Baseline conversion rate p0 (from recent data)- Target relative lift r = 5% → p1 = p0*(1+r)- Alpha (type I) = 0.05 and one‑ or two‑sided test decision- Power (1−β) = 0.80- Allocation ratio (usually 1:1)- Test duration window, expected daily traffic, and churn/attrition rates

2) Choose formula/approachFor large samples use two‑proportion z test. One formula for per‑group sample size n:n = [Z_{1−α/2} * sqrt(2*p̄*(1−p̄)) + Z_{power} * sqrt(p0*(1−p0)+p1*(1−p1))]^2 / (p1−p0)^2where p̄=(p0+p1)/2 and Z are standard normal quantiles.If p0 is small or sample sizes borderline, prefer simulation/bootstrapping or exact methods.

3) Validate assumptions- Independence of observations (no user duplication between groups)- Stable baseline (no strong trend/seasonality) or adjust for it- SUTVA: treatment affects only assigned users- Sufficient sample for normal approximation or else use exact/binomial/simulation- Check expected variance: conversion near 0 or 1 needs care

4) Practical adjustments- Inflate n for expected noncompliance, dropouts, bot traffic: n_adjusted = n / (1 − attrition)- If you will analyze subgroups, power must be calculated per subgroup ⇒ larger N

5) Multiple metrics or multiple tests- Multiple primary metrics: control familywise error with Bonferroni (conservative) or use hierarchical testing (pre‑specify primary metric order) or control FDR (Benjamini‑Hochberg) depending on tolerance for Type I.- Repeated looks / sequential monitoring: do not peek naïvely. Use group sequential methods (O’Brien‑Fleming, Pocock) or alpha‑spending (Lan‑DeMets) to adjust critical values, or adopt Bayesian monitoring which naturally handles continuous looks.- If using sequential plans, compute sample size or timing with the chosen spending function (often requires specialized software or simulation).

6) Implementation advice (PM focus)- Pull a recent 4–8 week baseline to estimate p0 and traffic variance- Run the calculation in your analytics/power tool and validate by simulation- Pre‑register test plan: metric(s), primary hypothesis, stopping rules, and sample size- Communicate assumptions and contingencies to analytics/engineering before launch

Why this matters: explicit inputs and validated assumptions prevent underpowered tests or false positives; adjustments for multiple metrics and sequential testing preserve statistical validity while matching product cadence.

EasyTechnical

72 practiced

Write a precise metric definition for Daily Active User (DAU) and Monthly Active User (MAU) for a cross-device social app. Specify which events count as activity, how you deduplicate users across devices, the timezone/window used, treatment of anonymous sessions, and if multiple events per user per day matter.

EasyTechnical

55 practiced

Explain statistical variability in product metrics: what is 'signal' versus 'noise'? Provide a simple example where daily spikes are noise, and describe two methods (one statistical, one procedural) you would use to distinguish true change from random variation.

Unlock Full Question Bank

Get access to hundreds of Metrics and KPI Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.