Metrics Analysis and Data Driven Problem Solving Questions

Skills for using quantitative metrics to diagnose and solve business, product, or operational problems across functions. Candidates should be able to identify the key performance indicators relevant to their domain (for example: conversion rate, retention, revenue per user, pipeline velocity, response time, or customer satisfaction), detect anomalies and trends in metrics, formulate and prioritize hypotheses about root causes, design experiments and controlled tests (such as A/B tests) to validate hypotheses, perform cohort and time series analysis, evaluate statistical significance versus practical business impact, and implement and monitor data backed solutions. This also includes instrumentation and data collection best practices, dashboarding and visualization to surface insights, trade off analysis when balancing multiple competing metrics, and communicating findings and recommended changes to cross functional stakeholders.

MediumTechnical

36 practiced

A checkout optimization increases conversion but reduces average order value (AOV). Outline an analytical plan to quantify the net revenue impact, including per-user vs per-order metrics, statistical tests or bootstrap methods to compute confidence intervals, segment-level analysis, and decision rules you would recommend to product stakeholders.

Sample Answer

Summary: measure net revenue impact using revenue-per-user (RPU) as primary KPI (accounts for conversion + AOV). Use both per-user and per-order views, and bootstrap CIs because AOV and revenue are skewed.

1) Define metrics- Conversion rate = orders / users (or sessions)- AOV = sum(order_value) / orders (per-order)- RPU = sum(order_value) / users = Conversion × AOV (per-user; primary)- RPO (revenue per order) = AOV (secondary)- Additional: median order value, order count per buyer, LTV forecasts if relevant

2) Analysis plan- Aggregate at user-level (each row = user): conversion indicator, total spend, orders count. This avoids needing to combine two noisy ratios.- Compute differences: ΔRPU = RPU_treatment − RPU_control. Also report ΔConversion, ΔAOV (per-order), ΔOrders-per-buyer.- Use bootstrap (preferred for skewed spend): resample users with replacement, compute ΔRPU and ΔAOV distributions → 95% CI and p-value.- For per-order tests (AOV), bootstrap resample orders, but beware dependence when users have multiple orders — better to cluster by user or use hierarchical bootstrap.

3) Example bootstrap (python)

python

import numpy as np
def bootstrap_diff(users_control, users_treat, n=2000):
    # users_*: array of user total spend
    diffs=[]
    for _ in range(n):
        s1 = np.random.choice(users_control, size=len(users_control), replace=True).mean()
        s2 = np.random.choice(users_treat, size=len(users_treat), replace=True).mean()
        diffs.append(s2 - s1)
    diffs=np.array(diffs)
    return diffs.mean(), np.percentile(diffs,[2.5,97.5])

4) Segment-level analysis- Pre-specified segments: new vs returning, top-of-funnel channel, device (mobile vs desktop), basket size deciles, geography, high-value customers.- Stratified bootstrap or compute per-segment ΔRPU and CIs. Flag segments with opposite sign effects (e.g., conversion up on mobile but AOV down heavily for desktop).

5) Statistical considerations- Power/sample-size: estimate minimum detectable effect on RPU; ensure experiment duration and user counts are sufficient.- Multiple comparisons: control false discovery (Benjamini-Hochberg) when scanning many segments.- Check balance of covariates; consider regression adjustment (e.g., OLS on user spend with treatment and covariates, cluster SEs).

6) Decision rules for stakeholders- Primary rule: accept if lower bound of 95% CI for ΔRPU > 0 (statistically positive net revenue). If CI crosses zero, consider business thresholds: - If mean ΔRPU > minimal ROI threshold (e.g., covers CAC or margin target) and no significant long-term risks, consider longer ramp. - If conversion up but ΔRPU negative and lower bound < −material_threshold (e.g., expected monthly loss), revert.- Use segment-aware rollout: roll out to segments with positive ΔRPU; hold or A/B test variants for segments with negative impact.- Operational: run a longer-duration holdout to monitor retention/LTV impacts before full rollout if short-term ΔRPU ambiguous.

7) Deliverables- Executive dashboard: RPU with CIs, ΔConversion, ΔAOV, segment heatmap, sample sizes/power, recommendation.- Appendix: bootstrap code, data definitions, raw counts.

This approach gives stakeholders a single business-facing metric (RPU) with robust uncertainty estimates and actionable segment-level recommendations.

EasyTechnical

41 practiced

You are building a metrics dashboard for a customer support organization. List the eight most important KPIs you would include (for example: customer satisfaction, response time, resolution rate, FCR). For each KPI provide:1) a precise business definition (numerator/denominator and time window),2) primary instrumentation/events required to measure it,3) one known caveat or bias to watch for.Limit the list to KPIs that executives and managers would find actionable.

Sample Answer

1) Customer Satisfaction (CSAT)- Definition: % positive survey responses = (# responses rating 4–5) / (total survey responses) over last 30 days.- Instrumentation: Post-ticket survey event with ticket_id, score, timestamp.- Caveat: Response bias — satisfied/unsatisfied customers more likely to respond; low sample size for small segments.

2) First Response Time (FRT)- Definition: Median time from ticket creation to first agent reply in hours over last 7/30 days.- Instrumentation: ticket.created_at, message.sent_at (agent), message.role.- Caveat: Automated acknowledgements can mask real human response unless filtered.

3) Resolution Time (Time to Resolution, TTR)- Definition: Median time from ticket creation to ticket.marked_resolved in hours over last 30 days.- Instrumentation: ticket.created_at, ticket.resolved_at, status change events.- Caveat: Reopened tickets inflate TTR; use first vs final resolution consistency.

4) First Contact Resolution (FCR)- Definition: % tickets resolved without reopening within 7 days = (tickets closed and never reopened within 7 days) / (total tickets closed) over last 30 days.- Instrumentation: ticket.close events, reopen events, ticket lifecycle.- Caveat: Some issues require follow-up by design; not all reopens indicate failure.

5) Resolution Rate / SLA Compliance- Definition: % tickets resolved within SLA target (e.g., 48h) = (tickets resolved within SLA window) / (total tickets) over period.- Instrumentation: SLA policy metadata, ticket.resolved_at, ticket.created_at.- Caveat: SLA definitions vary by priority — compare like-for-like.

6) Contact Volume / Tickets per Channel- Definition: Count of new tickets by channel (email, chat, phone) per day/week.- Instrumentation: ticket.created_at, ticket.channel, source metadata.- Caveat: Channel migrations or duplicates can double-count; normalize threads vs tickets.

7) Backlog / Work in Progress (WIP)- Definition: Number of open tickets older than X days (e.g., >3 days) as of reporting time.- Instrumentation: ticket.status, ticket.created_at, last_activity_at.- Caveat: Some long-lived tickets are intentionally on-hold (awaiting customer) — distinguish hold reasons.

8) Agent Utilization & Efficiency- Definition: Avg productive handling time per agent = (sum handle_time minutes) / (agent logged-in hours) over last 7/30 days.- Instrumentation: agent.login/logout, ticket.handle_time events, wrap-up time.- Caveat: Overemphasis can drive rushed interactions; quality metrics must be paired with utilization.

Each KPI should include filters for priority, product line, and cohort and be surfaced with sample sizes and confidence indicators on executive dashboards.

EasyTechnical

32 practiced

You see a 50% overnight drop in reported daily resolved tickets on the executive dashboard. Provide a prioritized checklist to determine whether this is a real business issue or a reporting/data problem. Include fast SQL checks, instrumentation checks, ETL and pipeline health checks, and quick visualizations you would run.

Sample Answer

Top-level goal: quickly decide if this is a real business drop or a reporting/data issue. Prioritize fast, cheap checks first (SQL/observability), then deeper pipeline and instrumentation checks.

1) Quick sanity SQL checks (5–15 min)- Total resolved-ticket count yesterday vs day-before and 7‑day avg:

sql

SELECT date(resolved_at) as day, COUNT(*) as resolved
FROM tickets
WHERE date(resolved_at) >= current_date - interval '8 day'
GROUP BY 1 ORDER BY 1;

- Count by source/system (UI, API, ingestion process):

sql

SELECT source, COUNT(*) FROM tickets WHERE date(resolved_at)=current_date-1 GROUP BY 1;

- Distinct agents/users resolving tickets:

sql

SELECT COUNT(DISTINCT resolver_id) FROM tickets WHERE date(resolved_at)=current_date-1;

- Max/min timestamps and late-arriving rows:

sql

SELECT MAX(created_at), MAX(updated_at) FROM ticket_ingest WHERE date(event_time)=current_date-1;

2) Instrumentation & dashboard layer (5–20 min)- Verify dashboard query returns expected rows directly in DB/Looker/BI SQL runner.- Check metric logic: filters, timezone, business day cutoff, deduplication logic changed?- Confirm no recent dashboard deploys or metric-definition edits (PRs, changelog).

3) ETL / pipeline health (10–30 min)- Check recent job statuses, runtimes, error logs (Airflow/Kube/Cloud Dataflow).- Verify last successful run time and rows processed:

sql

SELECT job_name, status, run_time, rows_processed FROM etl_job_runs WHERE run_time >= current_date-2 ORDER BY run_time DESC;

- Check for failed partitions or connector errors, schema changes, quota limits.

4) Instrumentation at source (10–30 min)- Check upstream system logs (app servers, API gateway) for drops in submit events.- Validate event counts in raw events table vs resolved aggregates:

sql

SELECT date(event_time), COUNT(*) FROM raw_ticket_events WHERE event_type='resolved' AND date(event_time)>=current_date-8 GROUP BY 1;

- Confirm no client-side deploy removed tracking or changed event names.

5) Quick visualizations to run (5–15 min)- Time series: 30‑day resolved count with annotations for deploys/ETL failures.- Source breakdown bar chart (source/system).- Heatmap by hour-of-day x day to spot delayed processing.- Cumulative arrivals vs resolved to check backlog.

6) Triage & next steps- If SQL/raw events show same drop → real business issue (notify stakeholders, investigate product/ops).- If raw events normal but aggregated metric low → bug in ETL/transform or dashboard logic (roll back change, re-run ETL).- If pipelines failed → re-run or repair and backfill; verify after backfill.- Document findings, incident timeline, mitigation, and preventive steps (alerts, SLA tests).

Key notes: check timezones/cutoffs first, compare raw events vs aggregated, and surface pipeline/job errors. Prioritize actions that restore correct metric quickly (rerun/patch) while communicating status to execs.

MediumBehavioral

33 practiced

Tell me about a time you persuaded stakeholders to prioritize building a dashboard or metric they considered low priority. Use the STAR framework and include details on how you measured business impact after delivery.

MediumTechnical

33 practiced

Design a monitoring plan to detect data drift in features used across dashboards and predictive models. Which statistical tests would you run, what thresholds would you use, how frequently would checks run, and how would you prioritize which drifts require human intervention?

Sample Answer

Approach: build an automated, risk-based monitoring pipeline that compares current feature distributions to a baseline (training period or recent stable window), runs appropriate statistical tests, scores drift severity, and routes alerts by business impact and model/dashboard sensitivity.

Which tests and when to use them:- Numerical features: - Population Stability Index (PSI) for a compact drift score (useful for dashboards/models): easy to compute and track trends. - Kolmogorov–Smirnov (KS) test for distribution shift (gives p-value + stat). - Two-sample t-test for mean shifts when normality holds.- Categorical features: - Chi-square test or Cramér’s V for distribution change. - KL-divergence as an additional score if zero/low-prob bins handled.- Rare-event or heavy-tailed features: - Anderson–Darling or bootstrapped permutation tests.- Completeness/format checks: - Null-rate change, cardinality changes, new categories detection.

Thresholds / interpretation (recommendation):- PSI: <0.1 — negligible, 0.1–0.25 — moderate (investigate), >0.25 — high (action).- KS test: p-value < 0.01 (or <0.05) indicates significant shift; combine with effect size (KS statistic).- Chi-square: p-value < 0.01 plus Cramér’s V > 0.1 signals meaningful categorical drift.- Null-rate: absolute increase > 5 percentage points or relative > 50% flagged.Always combine statistical significance with effect size and business context; avoid alerting on tiny but significant changes from huge samples.

Frequency:- Critical-model features and executive dashboard KPIs: daily checks.- Operational dashboards and low-impact features: weekly checks.- Full audits (all features, historical windows): monthly.

Prioritization for human intervention:1. Business impact: features directly tied to revenue, fraud, legal, or executive KPIs — top priority.2. Model sensitivity / feature importance: high SHAP/feature importance weight → escalate.3. Magnitude & persistence: large PSI/KS that persists across multiple runs (e.g., 3 consecutive days) → higher priority.4. Data integrity signals: sudden null spikes, new category values, or schema changes → urgent.5. Traffic/volume: low-sample features require manual review if tests are unstable.

Operational workflow:- Ingest streaming/batch feature snapshots → compute drift metrics vs baseline → score each feature (severity, business impact, persistence) → apply rules to generate alerts.- Alert channels: Slack/email for medium, ticket + on-call for high.- Attach context in alert: trend plot, baseline vs current histogram, sample counts, model feature importance, downstream dashboards/models affected, suggested next steps (check upstream ETL, examine filters, inspect sample rows).- Triage playbook: quick data-check (ETL/schema), root-cause (data source change vs population change), remediate (fix pipeline, add mapping, retrain model, annotate dashboard), and postmortem.

Example prioritization rule (simple):- If PSI > 0.25 OR KS p < 0.001 AND feature importance > 0.05 → P1 alert (human within 4 hours).- If 0.1 < PSI <= 0.25 OR KS p < 0.01 → P2 alert (review within 48 hours).- Else log/monitor.

Final notes:- Combine statistical tests with visualization (histograms, CDFs, time-series) — BI analysts should include drift panels in dashboards.- Re-evaluate baselines periodically (rolling windows) and guard against false positives from seasonality by using seasonally-matched baselines.

Unlock Full Question Bank

Get access to hundreds of Metrics Analysis and Data Driven Problem Solving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.