Business Metrics Definition and Strategy Questions

Emphasizes defining meaningful metrics and measurement frameworks that answer business questions and drive decisions. Candidates should be able to distinguish between count metrics, ratio metrics, and rate metrics; select appropriate observation windows and time alignment for retention, churn, and conversion analyses; account for multiple user touch points and events when attributing actions; and identify leading versus lagging indicators. This topic covers designing metric definitions that avoid double counting, selecting denominators and numerators that match the business question, segmenting users for insight, and documenting business logic to ensure consistency. At senior levels expect discussion of trade offs between simplicity and fidelity, governance of metric definitions, and how to prioritize which metrics matter for different stakeholders.

EasyTechnical

31 practiced

Double counting often inflates metrics. Describe three real-world scenarios where double counting can occur (e.g., multi-device sessions, repeat purchases in a timeframe, overlapping event definitions) and propose practical approaches to avoid or correct double counting in each scenario.

Sample Answer

Scenario 1 — Multi-device / multi-session usersProblem: A user visits on mobile then desktop; counting “unique sessions” by cookie or device ID inflates active users.Fixes:- Use deterministic identity resolution (login_id) where available; prefer user_id over device_id.- If only device identifiers exist, deduplicate by probabilistic stitching (email hash, IP + user-agent + time window) and assign a single user key.- In SQL: group by user_id and session_date instead of device_id.

sql

SELECT user_id, COUNT(DISTINCT session_id) AS sessions
FROM events
WHERE event_time BETWEEN '2025-01-01' AND '2025-01-31'
GROUP BY user_id;

Scenario 2 — Repeat purchases in a timeframe (inflated conversion rates)Problem: Counting purchase events naively can double-count users who buy multiple times within the reporting window.Fixes:- Define metric precisely: “orders” vs “buyers.” For buyers, count distinct user_id; for orders, count transactions.- Use sessionization and attribution windows (first purchase in period) when measuring acquisition conversions.

sql

-- unique buyers in month
SELECT COUNT(DISTINCT user_id) FROM purchases WHERE purchase_date BETWEEN ...

Scenario 3 — Overlapping or ambiguous event definitionsProblem: Two events (e.g., add_to_cart and pre-order) both increment “engagement” KPI causing overlap.Fixes:- Establish an events taxonomy and hierarchy; document canonical event for each KPI.- Implement event-processing rules: prioritize or map overlapping events into mutually exclusive buckets (e.g., map pre-order -> cart only if cart not present).- In ETL, normalize events to a single column event_type_normalized before aggregation.

General best practices- Maintain a data dictionary and schema registry.- Add unit tests and monitoring (e.g., alert if unique_users / total_events ratio deviates).- Always state metric definitions in reports (what’s deduped, time windows, attribution rules).

EasyTechnical

32 practiced

Given the events table below, write a SQL query (standard ANSI SQL) to compute Daily Active Users (DAU), deduplicating multiple events per user per day. Schema:

events(event_id INT PK, user_id INT, event_type VARCHAR, occurred_at TIMESTAMP)

Return a table with columns: event_date, dau.

Sample Answer

Approach: For DAU we need one unique user per day. Two common ANSI-safe approaches: count distinct users per date, or first deduplicate to (date, user_id) pairs then aggregate. Both produce same result; deduplication is clearer for downstream joins.

sql

-- Approach: deduplicate first, then count users per day
WITH unique_daily_users AS (
  SELECT
    CAST(occurred_at AS DATE) AS event_date,
    user_id
  FROM events
  WHERE user_id IS NOT NULL
  GROUP BY CAST(occurred_at AS DATE), user_id
)
SELECT
  event_date,
  COUNT(*) AS dau
FROM unique_daily_users
GROUP BY event_date
ORDER BY event_date;

Key points:- CAST(occurred_at AS DATE) is ANSI SQL to extract date.- Alternatively: SELECT CAST(occurred_at AS DATE) AS event_date, COUNT(DISTINCT user_id) AS dau FROM events GROUP BY CAST(occurred_at AS DATE).Edge cases:- Exclude NULL user_id; consider timezone normalization before casting if occurred_at is UTC but business day is local.Performance:- COUNT(DISTINCT ...) is simpler; explicit dedupe may be faster if you need to join back to examples or apply additional per-user filters.

MediumTechnical

31 practiced

Design an automated monitoring system to detect anomalies in product metrics (e.g., daily revenue, DAU). What checks would you implement, what alerting thresholds or statistical methods would you use, and how would you reduce false positives?

Sample Answer

Start by clarifying scope and SLAs: which metrics (daily revenue, DAU), data latency, business owners and acceptable sensitivity (how noisy alerts may be). Then implement layered checks: data-quality, statistical anomaly detection, and business-rule checks.

1) Data-quality checks (first line):- Schema/row-count validation, null-rate, source-change detection, and freshness/latency alerts.- Example: fail if daily ingestion < 90% of historical median.

2) Statistical anomaly detection:- Short-term: rolling z-score on de-seasonalized series (remove weekly/holiday pattern using 7- and 28-day medians or STL decomposition). Flag if |z|>3.- Robust alternative: rolling Median Absolute Deviation (MAD) to handle outliers; flag if deviation > 3*MAD.- Trend-aware: EWMA or Holt-Winters forecast with prediction interval; flag if observed value outside 95% prediction interval for k consecutive days (k≥2).- Change-point detection for sudden level shifts (e.g., Python ruptures, Bayesian Online Change Point).- Rate/relative checks: percent change vs same day-of-week baseline; e.g., >20% drop vs prior 4-week same weekday and p-value <0.01.

3) Multi-signal and context enrichment to reduce false positives:- Correlate signals (revenue drop + orders drop + conversion rate) before paging.- Attach context: deployments, marketing campaigns, holidays, ad spend, data-source health.- Require at least two independent signals (metric + upstream system alert) or sustained deviation (2+ periods) for high-severity alerts.

4) Alerting thresholds and routing:- Tiered alerts: informational (single-point outlier), actionable (sustained or multi-signal), critical (major % change affecting KPI).- Use gradual escalation: email/Slack for informational, on-call paging for critical.- Include runbook with checks to run (SQL queries, dashboard links, recent deploys).

5) Reducing false positives:- Use robust statistics (MAD, median) and de-seasonalization.- Backtest detectors on historical labeled incidents to tune thresholds and compute precision/recall.- Implement suppression windows around known events (holidays, planned releases).- Auto-snooze during data issues and allow stakeholders to annotate/disable rules.- Use ensemble scoring: combine detectors into a single anomaly score and threshold that optimizes business-FPR tradeoff.

Implementation notes:- Build detection pipeline in scheduled jobs (Airflow) that writes anomaly records to a table.- Store metadata: metric, detector, score, context, links.- Integrate with monitoring tools (PagerDuty, Slack) and dashboards (Tableau/Looker) for triage.- Regularly review false positives and retrain/tune models.

This layered, context-aware approach balances sensitivity and precision so analysts get meaningful, actionable alerts rather than noise.

EasyBehavioral

30 practiced

Tell me about a time you had to explain a complex metric (with trade-offs) to non-technical stakeholders. Use the STAR method: what was the Situation, your Task, the Actions you took to communicate, and the Result?

MediumTechnical

43 practiced

Write a SQL query that computes a funnel conversion rate for users who visited product page → added to cart → purchased within 14 days. Schema:

events(user_id INT, event_date DATE, event_type VARCHAR, product_id INT)

Requirements: count unique users at each funnel step, ensure ordering (visit before add to cart before purchase) and per-user deduplication. Return funnel_step and unique_users.

Sample Answer

Approach: deduplicate events per user/event_type by earliest date, then join steps ensuring ordering and <=14 days between first visit and purchase. Count distinct users at each step and return funnel_step + unique_users.

sql

WITH first_events AS (
  -- earliest date per user per event_type
  SELECT
    user_id,
    event_type,
    MIN(event_date) AS first_date
  FROM events
  WHERE event_type IN ('visit_product', 'add_to_cart', 'purchase')
  GROUP BY user_id, event_type
),
visits AS (
  SELECT user_id, first_date AS visit_date
  FROM first_events
  WHERE event_type = 'visit_product'
),
adds AS (
  SELECT user_id, first_date AS add_date
  FROM first_events
  WHERE event_type = 'add_to_cart'
),
purchases AS (
  SELECT user_id, first_date AS purchase_date
  FROM first_events
  WHERE event_type = 'purchase'
),
funnel AS (
  -- join and enforce ordering: visit < add < purchase, and purchase within 14 days of visit
  SELECT
    v.user_id,
    v.visit_date,
    a.add_date,
    p.purchase_date
  FROM visits v
  LEFT JOIN adds a
    ON v.user_id = a.user_id
    AND a.add_date > v.visit_date
  LEFT JOIN purchases p
    ON v.user_id = p.user_id
    AND p.purchase_date > COALESCE(a.add_date, v.visit_date)
    AND p.purchase_date <= DATEADD(day, 14, v.visit_date)  -- use DATEADD or equivalent per SQL dialect
)
SELECT funnel_step, COUNT(DISTINCT user_id) AS unique_users
FROM (
  SELECT user_id, 'visited'   AS funnel_step FROM funnel WHERE visit_date IS NOT NULL
  UNION ALL
  SELECT user_id, 'added_to_cart' FROM funnel WHERE add_date IS NOT NULL
  UNION ALL
  SELECT user_id, 'purchased' FROM funnel WHERE purchase_date IS NOT NULL
) t
GROUP BY funnel_step
ORDER BY 
  CASE funnel_step
    WHEN 'visited' THEN 1
    WHEN 'added_to_cart' THEN 2
    WHEN 'purchased' THEN 3
  END;

Key points:- We dedupe per user/event_type using MIN(date).- Joins enforce strict ordering (visit < add < purchase).- Purchase must occur ≤14 days after visit.- Counts unique users per step with DISTINCT.Edge cases & notes:- If multiple adds/purchases, we use earliest after prior step.- Adjust DATEADD syntax for your SQL dialect (e.g., DATE_ADD in MySQL).- If you want users who completed previous steps (strict funnel), change LEFT JOINs to INNER JOINs.

Unlock Full Question Bank

Get access to hundreds of Business Metrics Definition and Strategy interview questions and detailed answers.

Join thousands of developers preparing for their dream job.