Metrics Analysis and Data Driven Problem Solving Questions

Skills for using quantitative metrics to diagnose and solve business, product, or operational problems across functions. Candidates should be able to identify the key performance indicators relevant to their domain (for example: conversion rate, retention, revenue per user, pipeline velocity, response time, or customer satisfaction), detect anomalies and trends in metrics, formulate and prioritize hypotheses about root causes, design experiments and controlled tests (such as A/B tests) to validate hypotheses, perform cohort and time series analysis, evaluate statistical significance versus practical business impact, and implement and monitor data backed solutions. This also includes instrumentation and data collection best practices, dashboarding and visualization to surface insights, trade off analysis when balancing multiple competing metrics, and communicating findings and recommended changes to cross functional stakeholders.

MediumSystem Design

19 practiced

Your team is optimizing the ranking of a homepage carousel and wants to continuously improve click-through rate. Would you recommend a classic A/B test or a multi-armed bandit, and why?

MediumTechnical

16 practiced

You report that overall conversion rate improved after a pricing change, but a colleague points out that conversion actually dropped within every individual customer segment. How is that possible, and how would you have caught it before reporting the aggregate number?

Sample Answer

**What's happening: Simpson's paradox**

This happens when the mix of segments shifts between the two periods you're comparing. If a high-converting segment (say, returning customers) makes up a larger share of traffic after the change, the blended average can rise even though every individual segment's conversion rate fell.

**Worked example**

text

Before: New users 80% of traffic at 2% conversion, Returning 20% at 10% conversion
  Blended = 0.8*2% + 0.2*10% = 3.6%
After:  New users 50% of traffic at 1.5% conversion, Returning 50% at 9% conversion
  Blended = 0.5*1.5% + 0.5*9% = 5.25%

Both segments got worse (2% to 1.5%, 10% to 9%) but the blended rate went up because the mix shifted toward the higher-converting segment.

**How to catch it before reporting**

- Always break a headline metric out by the two or three dimensions most likely to shift mix (user tenure, channel, geography) before presenting an aggregate change.- Check whether the underlying population composition changed between the periods being compared, not just the metric value.- When in doubt, report a mix-adjusted (standardized) rate: hold the segment weights constant at the earlier period's mix and recompute the later period's blended rate.

**Common pitfall**

Treating an aggregate metric as ground truth without checking whether the denominator's composition moved is one of the most common ways a real result gets reported backwards.

MediumBehavioral

21 practiced

Tell me about a time your data analysis pointed to a conclusion that a senior stakeholder strongly disagreed with. How did you handle it?

MediumTechnical

21 practiced

A core metric dropped 4% last week and you're slicing it across 40 different segment combinations (device, region, plan tier, signup cohort, etc.) looking for the cause. What's the statistical risk in this process, and how would you guard against it while still doing useful exploratory work?

Sample Answer

**The risk: multiple comparisons / data dredging**

With 40 independent slices tested at a 5% significance threshold, you'd expect roughly 2 of them to look "significant" purely by chance, even if nothing is actually different in those segments. This is the same false-discovery problem as running many A/B tests, except here it's happening informally during root-cause analysis, which makes it easy to seize on a random-looking segment and build a false narrative around it.

text

Expected false positives ~= number_of_tests * alpha

For 40 tests at alpha=0.05, expect about 2 spurious "findings" even with no real effect anywhere.

**How to guard against it while still exploring**

- Treat segment-cut findings as hypothesis generation, not confirmation. A segment that looks different in this slice needs a second, independent check (a different time window, a different metric, or a follow-up query) before you act on it.- Look for effect size and consistency, not just p-value: a segment that's off by a small amount with a huge sample size can look "significant" but be practically meaningless, while a large, consistent effect across adjacent time periods is more trustworthy than a single point-in-time slice.- If you need a statistically defensible answer (e.g., for an exec readout), apply a multiple-testing correction such as Benjamini-Hochberg to control the false discovery rate across the 40 comparisons, rather than reading each p-value at face value.- Prioritize segments with a plausible causal story (a recent deploy that only touched one platform, a pricing change in one region) over the raw ranked list of "most different" segments.

**Common pitfall**

Presenting the single most extreme segment from an unadjusted 40-way scan as "the root cause" without validating it independently is one of the most common ways teams chase a false lead.

HardSystem Design

28 practiced

Your product's north-star engagement metric has looked healthy and stable for months, but you suspect it's blending two very different user experiences: power users whose engagement is rising, and new users whose engagement is quietly declining, in a way that cancels out in the aggregate. How would you design a metrics framework that would surface this without abandoning the north-star metric?

Sample Answer

**Why the aggregate can hide this**

A single blended metric is a weighted average across the whole user base. If power users are a growing share of a shrinking pool while new-user engagement erodes, the two trends can offset almost exactly, producing a flat top-line number that masks two real, opposite-direction problems. This is the same mechanism as Simpson's paradox, but unfolding gradually over time rather than in a single before/after comparison.

**Framework design**

- Keep the north-star metric as the single number for company-wide communication, since a headline number has real organizational value, but never report it without its standard decomposition attached.- Define a fixed, small set of segments (e.g., tenure buckets: 0-30 days, 30-90 days, 90+ days) and track the north-star metric within each segment on the same dashboard, updated on the same cadence as the aggregate.- Track segment mix (the share of total users in each tenure bucket) alongside the segment-level metrics, since a shift in mix combined with stable segment-level numbers is a different problem (a growth or retention issue) than declining segment-level numbers with a stable mix.- Set an explicit trigger rule: if any individual segment's metric diverges from the aggregate trend by more than a defined threshold for two consecutive reporting periods, that triggers a mandatory segment-level review, independent of whether the aggregate itself moved.

**Decision policy for overriding the aggregate**

When segment-level trends diverge meaningfully from the aggregate, I'd escalate the segment-level view, not the aggregate, as the operative signal for decision-making, and explicitly flag in the exec-facing report that the north-star number is currently mixing two divergent stories.

**Trade-off**

Adding this decomposition as standard practice adds dashboard and reporting overhead, and too many tracked segments dilutes attention the same way too many top-level metrics would. I'd cap it at the two or three dimensions most likely to hide real divergence (tenure, plan tier, acquisition channel) rather than slicing everything.

Unlock Full Question Bank

Get access to hundreds of Metrics Analysis and Data Driven Problem Solving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.