Analytical Problem Solving and Hypothesis Testing Questions

Assesses the candidate's ability to convert an ambiguous business problem or signal into structured hypotheses, design analyses or experiments, and draw defensible conclusions. Expect discussion of problem framing, prioritizing hypotheses, selecting data sources, defining cohorts and metrics, designing queries or tests, validating assumptions, controlling for confounders, and communicating actionable recommendations. Core skills include critical thinking, data exploration, statistical reasoning, and translating insights into measurable action plans.

MediumTechnical

33 practiced

An A/B test shows a statistically significant 12% lift in click-through rate for a new recommendation algorithm, but revenue per session shows no significant change. What do you do next?

MediumTechnical

25 practiced

Can you write a query that computes week-over-week retention, meaning the percent of users active in week N who are still active in week N+1?

Sample Answer

**Approach**Build a weekly activity table per user, then self-join each week against the following week.

**Query**

sql

WITH weekly_activity AS (
  SELECT user_id, DATE_TRUNC('week', event_time) AS activity_week
  FROM events
  GROUP BY user_id, DATE_TRUNC('week', event_time)
)
SELECT
  w1.activity_week AS week,
  COUNT(DISTINCT w1.user_id) AS active_week_n,
  COUNT(DISTINCT w2.user_id) AS retained_week_n_plus_1,
  ROUND(COUNT(DISTINCT w2.user_id)::numeric / NULLIF(COUNT(DISTINCT w1.user_id), 0), 3) AS retention_rate
FROM weekly_activity w1
LEFT JOIN weekly_activity w2
  ON w1.user_id = w2.user_id AND w2.activity_week = w1.activity_week + INTERVAL '7 days'
GROUP BY w1.activity_week
ORDER BY w1.activity_week;

**Key points**- `NULLIF` guards against divide-by-zero.- `LEFT JOIN` keeps churned users in the denominator but not the numerator.

**Edge case**The most recent week is artificially low since week N+1 hasn't fully happened yet; exclude it from reporting.

MediumTechnical

29 practiced

Your finance partner tells you revenue per user went up 8% last quarter, but total revenue actually went down. How would you investigate what's going on?

HardTechnical

25 practiced

You're about to present a headline metric to leadership showing overall improvement, but you notice the effect reverses when you segment by user tenure, meaning long-tenured users actually got worse. You have thirty minutes before the meeting. What's your decision framework for what to present?

EasyTechnical

44 practiced

You're looking at a dashboard showing that overall conversion rate went up, but when you break it down by traffic source, conversion rate actually went down for every single source. How is that possible, and what do you call this phenomenon?

Unlock Full Question Bank

Get access to hundreds of Analytical Problem Solving and Hypothesis Testing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.