Business Problem Solving and Recommendations Questions

Frameworks and skills for taking ambiguous business questions through analysis to clear, actionable recommendations. Includes decomposing complex problems into analyzable components, identifying key drivers, selecting focused analyses, synthesizing data backed findings, and articulating specific next steps and implementation considerations. Emphasizes communicating recommendations in business terms, estimating potential impact when possible, acknowledging trade offs and limitations, prioritizing among multiple actions, and tailoring communication to different stakeholders. Covers translating research or analytic results into feasible product or operational changes and defending choices with evidence.

MediumSystem Design

53 practiced

Design a dashboard to help execs decide between two initiatives: (A) improve onboarding flow, (B) invest in a new marketing channel. Specify primary metrics, supporting cohorts and segments, recommended visualizations (with rationale), decision thresholds, required drill-downs, and a template one-page recommendation summary. Explain how you would present uncertainty and trade-offs on the dashboard.

Sample Answer

Requirements:- Goal: Help execs choose between Initiative A (improve onboarding) and Initiative B (invest in new marketing channel).- Time horizon: 3, 6, 12 months.- Audience: execs → need high-level KPIs with ability to drill to drivers, cohorts, and forecasted ROI.- Data sources: product analytics (events), CRM, marketing spend, funnel, retention, LTV models.

High-level layout (single dashboard page with drill panels)1. Executive Summary (top-left)- Primary recommendation tile (A vs B) with top-line expected delta in NPV/ARR over chosen horizon, confidence band.- Supporting bullets (primary metric change, estimated cost, payback period).

2. Primary metrics (center)- For Onboarding: Activation rate (users who reach key activation event within 7 days), 7/30-day retention, Time-to-value (median), 90-day LTV per cohort.- For Marketing: CAC by channel, New users attributable, Conversion-to-activation, 30/90-day retention of acquired users, Channel LTV.

Visualizations & rationale:- Side-by-side KPI cards for each initiative (metric + sparkline + % change vs baseline) — fast comparison.- Forecast chart: scenario fan chart (baseline, optimistic, pessimistic) showing cumulative incremental ARR/NPV over 12 months — communicates growth and uncertainty.- Waterfall chart: incremental value breakdown (conversion ↑, retention ↑, LTV ↑) by initiative — shows where value comes from.- Cohort retention heatmaps (by signup week and acquisition channel) — surfaces retention differences and persistence.- CAC vs LTV scatter by channel with bubble size = volume — highlights profitable channels.- Funnel conversion stacked bars by cohort (organic vs channel X) — shows where onboarding improvements help.

Supporting cohorts & segments:- New user cohorts by acquisition channel, geography, platform, and signup month.- High-value segment (top decile predicted LTV), low-engagement users (no events first 48h).- Enterprise vs SMB, paying vs free-to-paid conversion.

Decision thresholds (example rules to guide decision)- If improving onboarding increases activation by ≥10% and 30-day retention by ≥5% → prefer A (projected payback <6 months).- If a new channel CAC < 0.6 * median LTV and projected monthly net new MRR > X → prefer B.- If confidence interval of projected NPV overlaps and cost delta < 15% → recommend A for lower operational risk.

Required drill-downs- Click primary metric → user-level funnel view with event timelines, segmentation filters.- Drill from channel CAC → campaign-level spend, creatives, and audience lookalikes.- Drill from forecast fan → model assumptions page (conversion lift, decay rates, churn, attribution method), sensitivity analysis sliders.

Presenting uncertainty & trade-offs- Use fan charts and 90%/50% confidence bands for forecasted revenue; show key model assumptions and elasticities with sliders to let execs see sensitivity.- Display probability of beating baseline (e.g., P(ΔARR>0)=72%).- Show trade-off matrix: expected value vs risk (variance) and implementation cost/time.- Annotate major dependencies (engineering effort, data gaps, channel scalability risks).

One-page recommendation template (top of dashboard export)- Title: Recommendation — [A/B/Hybrid] (date)- Executive verdict: single sentence with expected NPV/ARR delta and confidence (e.g., “Recommend A; +$1.2M ARR over 12mo (±$0.4M), 78% probability of positive ROI”)- Key metrics driving decision: Activation +12%, 30-day retention +6%, Cost: $200k, Payback: 4 months- Upside scenarios: best-case numbers and actions to capture- Risks & mitigations: data limitations, implementation dependencies, channel saturation- Next steps: experiments to run (A/B test plan or small-scale channel pilot), measurement plan, decision checkpoint dates

Implementation notes (brief)- Backend: ETL to a metric cube (dbt), forecasts via reproducible notebooks (Python: pandas, Prophet/GLM/bootstrapping), expose to BI (Tableau/PowerBI).- Live filters and parameter controls for sensitivity; save scenario snapshots for audits.

This dashboard gives execs a clear recommendation, transparent assumptions, measurable decision thresholds, and the ability to explore drivers and uncertainty before committing resources.

EasyTechnical

68 practiced

You're evaluating a new feature for a subscription product (monthly and annual plans). Define a set of success metrics to measure whether the feature improved business outcomes: choose a single north-star metric, 3–4 supporting metrics across acquisition/activation/retention/monetization, and 2 guardrail metrics to detect negative side effects. Explain how you'd instrument and compute each metric.

Sample Answer

North-star metric (single): Increase in Net Revenue Retention (NRR) at 90 days for cohorts exposed to the feature.- Why: directly captures monetization + retention for a subscription product and accounts for upgrades/downgrades/churn.- Instrumentation & computation: join user cohort (feature_exposed_date) to subscription ledger. For each cohort: NRR90 = (MRR_day90 + expansion MRR within 90d) / MRR_day0. Use billing table (subscription_start, plan_tier, price, quantity, events: upgrade/downgrade/cancel). Compute per-user MRR then aggregate.

Supporting metrics (acquisition / activation / retention / monetization):1) Activation rate (activation within 7 days) — Activation event defined (e.g., first successful usage of new feature). Instrument via analytics event "feature_used". Compute: activated_users / exposed_users (7-day window).2) Free-to-paid conversion (30-day) — For trial users exposed: paid_subscriptions_30d / trials_exposed. Use subscription table (trial_start → paid_start).3) 30-day churn rate — churned_users_30d / active_users_at_day0. Churn detected by subscription_cancel event or lapsed billing.4) Average Revenue per User (ARPU) at 90 days — total revenue_from_cohort_90d / cohort_size. Use payments and adjustments table.

Guardrail metrics (detect negative side effects):1) Customer support volume related to feature — count of support tickets tagged with feature / total tickets. Instrument via CRM tags; compute weekly ratio; spike indicates UX issues.2) Conversion drop for annual plans — percent change in annual-plan conversions for exposed vs control. Instrument via subscription plan_type field; compute difference-in-differences.

Implementation notes:- A/B test with random assignment; store assignment in user profile table.- Use SQL pipelines to compute daily cohort metrics; schedule ETL to populate a dashboard (Looker/Tableau).- Apply statistical tests (chi-square or bootstrap CIs) for significance; track effect size and runtime metrics.- Segment by monthly vs annual, geography, and platform; monitor leading indicators (activation) to explain NRR changes.

MediumTechnical

73 practiced

Estimate the ROI of building a personalization recommender for an e-commerce site. Describe the analysis steps, data required, realistic assumptions about expected lift in conversion and AOV, development and ongoing operating costs (provide example numbers), and show a back-of-the-envelope 1-year ROI calculation for optimistic/realistic/pessimistic scenarios.

Sample Answer

Approach (steps)- Clarify scope & metric: lift in monthly revenue (driven by conversion rate and AOV), 1-year horizon.- Data & analysis plan: baseline metrics, uplift estimation from experiments/A-B tests or literature, cost estimate (build + run), sensitivity analysis for optimistic/realistic/pessimistic cases.- Compute incremental revenue, subtract costs, report ROI.

Data required- Traffic (unique monthly visitors), baseline conversion rate (CR), baseline average order value (AOV), repeat purchase/CLV data.- Product catalog size, historical co-purchase/click data, session logs, user profiles.- Infrastructure costs: hosting, model serving, feature store, monitoring, annotation/QA.- Team costs: data scientists, ML engineers, product/PM, frontend/backend engineers.

Realistic assumptions (example baseline)- Monthly unique visitors: 100,000- Baseline CR: 2.0% → orders/month = 100,000 * 0.02 = 2,000- Baseline AOV: $80 → revenue/month = 2,000 * $80 = $160,000 → annual = $1,920,000

Expected lifts (industry-informed examples)- Optimistic: +20% CR, +5% AOV- Realistic: +10% CR, +2% AOV- Pessimistic: +3% CR, +0% AOV(Notes: lifts combine better ranking, personalization, and cross-sell; validate via pilot A/B tests.)

Cost assumptions (example numbers, year 1)- Development (MVP, 3–4 people for 4–6 months): $150,000- Data engineering & integration: included above- Ongoing ops (hosting, model serving, monitoring, minor infra): $7,000/month → $84,000/year- Total Year-1 cost = $150,000 + $84,000 = $234,000

Back-of-the-envelope 1-year ROI calculations

Baseline annual revenue = $1,920,000

1) Optimistic- New CR = 2.0% * 1.20 = 2.4% → orders/month = 2,400- New AOV = $80 * 1.05 = $84- New monthly rev = 2,400 * 84 = $201,600 → annual = $2,419,200- Incremental annual revenue = $2,419,200 − $1,920,000 = $499,200- Net year-1 benefit = 499,200 − 234,000 = $265,200- ROI = 265,200 / 234,000 ≈ 113% (positive)

2) Realistic- New CR = 2.0% * 1.10 = 2.2% → orders/month = 2,200- New AOV = $80 * 1.02 = $81.60- New monthly rev = 2,200 * 81.6 = $179,520 → annual = $2,154,240- Incremental annual revenue = 234,240- Net year-1 benefit = 234,240 − 234,000 = $240- ROI ≈ 0.1% (rough break-even in year 1; longer term CLV uplift may improve economics)

3) Pessimistic- New CR = 2.0% * 1.03 = 2.06% → orders/month = 2,060- New AOV = $80- New monthly rev = 2,060 * 80 = $164,800 → annual = $1,977,600- Incremental annual revenue = $57,600- Net year-1 benefit = 57,600 − 234,000 = −$176,400- ROI = −75.4% (loss in year 1)

Key caveats and considerations- Year-1 often includes heavy upfront costs; multi-year ROI looks better because dev cost is mostly one-time while revenue lift recurs.- Include secondary benefits: increased retention/CLV, reduced marketing CAC via better personalization, higher margin via better product mix.- Validate assumptions via a small-scale A/B test or canary to measure actual CR/AOV lift before full rollout.- Sensitivity analysis: small changes in CR or AOV materially affect ROI; prioritize experiments to estimate real uplift.- Consider cheaper incremental options (rule-based recs, hybrid models, product bundles) to reduce upfront cost if risk-averse.

RecommendationRun a 6–8 week pilot A/B test to measure actual conversion and AOV lifts on a representative traffic slice; if realistic lift ≥ ~10% CR or material CLV gains, proceed to production. Use the numbers above to decide funding and timelines.

EasyTechnical

68 practiced

Write a SQL query (standard SQL) to compute weekly retention rates for sign-up cohorts. Schema:

users(user_id PK, signup_date DATE)events(user_id, event_date DATE, event_name TEXT)

Compute, for each weekly cohort (cohort_week = date_trunc('week', signup_date)), the percentage of users active in week 0 (signup week) through week 4. Show sample output format: cohort_week | week_number | retention_rate.

Sample Answer

Approach: build weekly cohorts by truncating signup_date to week, compute each user's week-number relative to their signup week for events, count unique users per cohort and week (0–4), then compute retention rate = active_users / cohort_size.

sql

WITH cohorts AS (
  SELECT user_id,
         date_trunc('week', signup_date)::date AS cohort_week
  FROM users
),
events_weeks AS (
  SELECT
    c.cohort_week,
    c.user_id,
    FLOOR(EXTRACT(epoch FROM (date_trunc('week', e.event_date) - c.cohort_week)) / 604800)::int AS week_number
  FROM cohorts c
  JOIN events e ON e.user_id = c.user_id
),
active_weeks AS (
  -- distinct users active per cohort_week and week_number
  SELECT cohort_week, week_number, COUNT(DISTINCT user_id) AS active_users
  FROM events_weeks
  WHERE week_number BETWEEN 0 AND 4
  GROUP BY 1,2
),
cohort_sizes AS (
  SELECT cohort_week, COUNT(DISTINCT user_id) AS cohort_size
  FROM cohorts
  GROUP BY 1
),
weeks AS (
  -- ensure weeks 0..4 present for every cohort
  SELECT cohort_week, generate_series(0,4) AS week_number
  FROM cohort_sizes
)
SELECT
  w.cohort_week,
  w.week_number,
  ROUND(
    COALESCE(a.active_users,0)::numeric / cs.cohort_size * 100
  ,2) AS retention_rate
FROM weeks w
JOIN cohort_sizes cs USING (cohort_week)
LEFT JOIN active_weeks a USING (cohort_week, week_number)
ORDER BY cohort_week, week_number;

Key points:- date_trunc('week', ...) aligns cohorts to week starts.- week_number computed by difference in weeks (604800 seconds).- generate_series ensures zero counts for missing weeks.- retention_rate in percent (two decimals).

Sample output:cohort_week | week_number | retention_rate2025-01-06 | 0 | 100.002025-01-06 | 1 | 45.002025-01-06 | 2 | 30.00...

HardTechnical

60 practiced

Design an experimentation strategy for rolling out a product change across multiple countries with different baselines, sample sizes, and regulatory constraints. Describe how you would pre-specify analyses, pool results (fixed vs. random effects / meta-analysis), adjust for heterogeneity, define stopping rules, estimate heterogeneous treatment effects, and decide on global vs. local rollouts.

Sample Answer

High-level goal: reliably estimate treatment effect across countries while respecting differing baselines, sample sizes, and regulations, then choose safe, efficient rollout policy (global vs phased local). Key pillars: pre-specification, appropriate pooling, heterogeneity control, valid stopping rules, robust HTE estimation, and practical rollout decision rules.

Pre-specify analyses- Register protocol: primary/secondary outcomes, estimands (ATE within-country and global), ITT vs per-protocol, covariates, handling of missing data, and minimum detectable effect (MDE) per country with power calculations.- Define analysis windows, interim looks, multiplicity corrections, and data governance (what can be pooled/shared given regs).- Pre-specify subgroup analyses (age, platform, baseline conversion), and sensitivity analyses (winsorizing, alternative covariate sets).

Design & sampling- Compute per-country sample sizes to reach country-level MDEs where possible; where infeasible, plan to borrow strength via pooling.- Use stratified randomization within countries to balance key covariates and enable conditional adjustments.

Pooling results: fixed vs random effects / meta-analysis- If effects are expected homogeneous and contexts similar, fixed-effect (inverse-variance weighted) meta-analysis provides efficient pooled ATE.- With differing baselines and likely between-country variation, use random-effects meta-analysis (DerSimonian–Laird or REML) to estimate mean effect and between-study variance tau^2.- Prefer hierarchical (multi-level) models (frequentist or Bayesian) that model country-level effects as draws from a population distribution — this allows shrinkage and principled uncertainty.

Adjusting for heterogeneity- Meta-regression: include country-level moderators (baseline rate, GDP, product penetration, regulatory constraints) to explain heterogeneity.- Individual-level hierarchical model: y_ij = alpha_c + beta_c * treatment_ij + X_ij gamma + eps_ij; model beta_c ∼ N(mu, tau^2). This enables partial pooling: small countries borrow strength.- Check heterogeneity diagnostics: I^2, tau^2, prediction intervals. If heterogeneity large and unexplained, avoid naive global rollouts.- Pre-specify policies for high-heterogeneity (e.g., require subgroup-specific efficacy thresholds).

Stopping rules and interim monitoring- Pre-specify number/timing of looks. Use alpha-spending (O’Brien–Fleming) for frequentist tests, or Bayesian stopping with posterior probability thresholds (e.g., P(ATE > delta | data) > 0.99 to stop for efficacy, <0.01 for harm).- Implement group-sequential designs or fully sequential monitoring with safeguards against peeking.- For multiple countries, monitor both pooled and country-level effects. Require both pooled evidence and lack of contraindicating local signals before global stop.- Predefine safety triggers for local regulatory concerns; allow country-level termination even if pooled positive.

Estimating heterogeneous treatment effects (HTE)- Two complementary approaches: 1) Model-based: hierarchical linear models give country-specific posterior estimates and credible intervals. 2) Machine-learning HTE: causal forests / X-learner to detect individual-level heterogeneity within and across countries; validate with cross-fitting and honest estimation to avoid overfitting.- Pre-specify primary HTE methods and validation (out-of-sample evaluation, calibration).- Present conditional average treatment effects (CATEs) and quantify uncertainty; use these to inform localized rollouts.

Decision rule: global vs local rollout- Define decision criteria ahead: e.g., - Global rollout if pooled mu_hat > business threshold delta_global, tau^2 below threshold, and no country shows harm (posterior P(beta_c < 0) < 0.05). - Phased/local rollout if pooled positive but heterogeneity moderate: rollout to countries with positive estimated beta_c and acceptable uncertainty; collect more data in ambiguous markets. - Local-only if pooled null/negative but specific markets positive.- Factor non-statistical constraints: regulatory approvals, legal data transfer limits, localization costs, and opportunity cost of delay.- Use a value-of-information calculation: weigh expected value of additional experimentation in uncertain countries vs expected gains from immediate rollout elsewhere.

Practical considerations & compliance- Data privacy: if raw data sharing prohibited, perform federated analysis or exchange aggregate summary stats for meta-analysis; pre-specify aggregation protocols.- Regulatory: align pre-spec with local legal/ethics teams; retain audit trails and pre-registration.- Reporting: produce per-country estimates, pooled estimates, heterogeneity stats, and decision rationale; include negative controls and sensitivity checks.

Wrap-up: combine rigorous pre-specification, hierarchical/random-effects pooling, meta-regression for heterogeneity, principled stopping rules, robust HTE estimation, and explicit decision thresholds that incorporate statistical evidence plus regulatory and business constraints. This yields transparent, defensible rollout decisions that balance speed with safety and local adaptation.

Unlock Full Question Bank

Get access to hundreds of Business Problem Solving and Recommendations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.