MediumTechnical

43 practiced

Design a hypothesis tree for a decline in Net Promoter Score (NPS) for a B2B SaaS product. Provide at least three top-level hypothesis branches, two levels of sub-hypotheses, and map each leaf hypothesis to the exact metric or data source you would query to test it.

Sample Answer

Top-level hypothesis tree (3 branches): Product Experience, Customer Success & Support, & Survey/Sampling Bias.

1) Product Experience 1.1) Feature regressions or bugs - Leaf: Recent increase in error rates for core workflows - Metric/data: API error rate, crash rate, failed job rate - Data source: App logs (Datadog/New Relic), backend metrics, Sentry - Leaf: Spike in bug-related releases - Metric/data: # bug PRs/releases last 30/90 days, rollback rate - Data source: GitHub/GitLab release history, JIRA bug tickets 1.2) UX or performance degradation - Leaf: Page load / transaction latency increased - Metric/data: median & 95th percentile page load time, task completion time - Data source: RUM (Google Analytics/Datadog), telemetry - Leaf: Reduced task success or feature adoption - Metric/data: feature usage funnels, drop-off rates in key flows - Data source: Product analytics (Mixpanel/Amplitude), event DB

2) Customer Success & Support 2.1) Increased support friction - Leaf: Longer time-to-resolution (TTR) or SLA misses - Metric/data: median/95th TTR, % SLA breaches - Data source: Zendesk/Intercom ticket logs, support dashboards - Leaf: Rising volume of product-critical tickets - Metric/data: # tickets tagged "blocking"/"production" per week - Data source: Support system tags, JIRA incidents 2.2) Poor onboarding or account health decline - Leaf: Decrease in new customer activation rate - Metric/data: % activated within 14 days, time-to-first-value - Data source: Onboarding funnel in product analytics, CRM (Salesforce) - Leaf: Expansion / renewal risk signals rising - Metric/data: usage decline per account, increase in at-risk accounts, churn intent flags - Data source: Billing system, CRM health scores, account usage tables

3) Survey & Sampling Bias 3.1) Survey delivery or wording issues - Leaf: Lower response rate or shift in respondent cohort - Metric/data: NPS response rate by segment, respondent vs. customer population match - Data source: NPS tool (Delighted/Qualtrics) + CRM join - Leaf: Timing correlated with negative events - Metric/data: % responses collected during incident window - Data source: NPS timestamps + incident logs 3.2) Changes in scoring population - Leaf: More responses from junior users vs. decision-makers - Metric/data: responder role distribution, NPS by role/industry/ARR - Data source: NPS metadata, CRM user profiles

Example quick SQL to test "NPS response rate by customer tier":

sql

SELECT c.tier, COUNT(n.id) AS responses, COUNT(DISTINCT c.customer_id) AS customers
FROM nps_responses n
JOIN customers c ON n.customer_id = c.customer_id
WHERE n.submitted_at >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY c.tier;

How I'd proceed: prioritize branches by data-access speed and impact (start with NPS metadata + CRM to rule out survey bias, then check support tickets & key product metrics). For each confirmed leaf, quantify contribution to NPS decline (e.g., % of detractors tied to accounts with high TTR) and recommend targeted remediation.

HardTechnical

37 practiced

Your CEO asks: 'What is our addressable market?' There is no clear segmentation. Provide a step-by-step market-sizing approach covering top-down and bottom-up methods, data sources you would use, key assumptions you must document, and how you would present a sensitivity analysis to express uncertainty.

Sample Answer

Step-by-step approach (Data Analyst lens)

1) Clarify scope & objective- Ask CEO whether they mean Total Addressable Market (TAM), Serviceable Available Market (SAM) or Serviceable Obtainable Market (SOM), geography, time horizon, and product definition. Document scope.

2) Top‑down method (macro → company)- Source total market size from trusted secondary data: industry reports (Gartner, IDC, Euromonitor), government statistics (Census, BLS, Eurostat), trade associations, company 10‑Ks.- Apply segmentation filters: geography %, industry vertical %, product substitution rates.- Example: TAM_topdown = Global market revenue (from report) × % relevant geographies × % product-relevant share.

3) Bottom‑up method (micro → aggregate)- Use internal and primary data: CRM customer counts, ARPU from billing, web traffic conversion rates (GA/GA4), transaction volume, channel partner reports.- Build model: Segment × (Number of potential customers in segment × penetration rate × annual spend per customer).- Example: TAM_bottomup = Σ segments (Population_segment × %businesses × adoption rate × avg spend).- Implement in SQL/Excel for reproducibility; store assumptions table.

4) Reconcile and validate- Compare top‑down vs bottom‑up within order-of-magnitude; investigate >2x gaps.- Use competitor revenue and market shares as sanity checks.

5) Key assumptions to document- Definitions (TAM/SAM/SOM), time horizon, pricing, adoption/penetration rates, churn, growth (CAGR), addressable geography, channel coverage, seasonality.- Data source, vintage, and confidence level for each input.

6) Sensitivity & uncertainty presentation- Produce scenario analysis (conservative/base/optimistic) by varying 3–5 key drivers (penetration, price, market growth).- Show tornado chart ranking drivers by impact.- Run probabilistic Monte Carlo (if warranted) and show 90% confidence interval.- Deliverables: dashboard (Tableau/Power BI) with interactive sliders for key assumptions, slide summary with headline TAM, SAM, SOM, main assumptions, data sources, and sensitivity visuals.- Recommend next steps: targeted primary research where uncertainty is largest and periodic model refresh schedule.

EasyTechnical

41 practiced

You're working at a subscription SaaS company where monthly churn has increased. Define the MECE principle and create a MECE breakdown of possible causes of churn that is mutually exclusive and collectively exhaustive. For each high-level category, provide one measurable metric you would collect, explain why that metric maps to the category, and state one assumption you made when creating the breakdown.

MediumTechnical

34 practiced

Multiple teams report different definitions of 'active user' (daily vs monthly, device-specific). How would you reconcile these definitions, create a canonical metric specification, and implement governance to prevent divergence across dashboards? Provide a practical rollout plan.

Sample Answer

Situation: Multiple teams (product, growth, BI) use differing "active user" definitions—some count daily logins, others count monthly unique device IDs—causing inconsistent KPIs and conflicting decisions.

Approach / Proposal:1. Clarify requirements: interview stakeholders to capture use-cases (retention, engagement, billing, legal). Document why each definition exists.2. Create a canonical metric spec (single source of truth). Example spec fields:- metric_id: ACTIVE_USERS_DAILY- name: Daily Active Users (DAU)- definition: count of distinct user_id that performed one or more of {login, page_view, transaction} in UTC date window- deduplication key: user_id (not device_id)- event whitelist: login, page_view, purchase- time zone: UTC- aggregation window: 00:00–23:59 UTC- owner: Growth Analytics- computation SQL snippet and freshness SLA3. Provide SQL example to compute DAU:

sql

-- DAU canonical
SELECT event_date, COUNT(DISTINCT user_id) AS dau
FROM events
WHERE event_type IN ('login','page_view','purchase')
GROUP BY event_date;

4. Governance to prevent divergence:- Publish spec in a searchable data dictionary (Confluence/Metabase/DataHub) with versioning.- Assign metric owners and steering committee (product, BI, growth) for approvals.- Require that dashboards reference canonical metric via a managed metrics layer (semantic layer: dbt exposures, metrics in BI tool, or a shared view).- Implement automated tests: freshness, schema, and alerting for drift.- Enforce through change process: any deviation requires proposal, approval, and documented alias mapping.

Rollout Plan (practical):- Week 1–2: Stakeholder interviews + inventory of existing definitions.- Week 3: Draft canonical specs and SQL views; map gaps.- Week 4: Review with steering committee; iterate.- Week 5: Publish data dictionary, create managed views (e.g., dbt models or BI semantic metrics).- Week 6–8: Pilot with 2 dashboards and one team; collect feedback.- Week 9: Company-wide rollout + training sessions and onboarding materials.- Ongoing: Monthly metric review, automated monitoring, and quarterly audits.

Outcome & Rationale:This balances business needs (different use-cases) while enforcing a single authoritative definition downstream. Using a managed metrics layer plus governance reduces duplication, improves trust in dashboards, and makes divergence explicit and intentional.

HardTechnical

39 practiced

Explain methods to detect and adjust for confounding variables in observational analyses of a feature release. Cover propensity score matching, regression adjustment, difference-in-differences, synthetic controls, and instrumental variables. For each method, state data assumptions, strengths, and practical pitfalls with an example.

Sample Answer

Overview: In observational evaluations of a feature release we must address confounding — factors that affect both exposure (who saw the feature) and outcome. Below are five widely-used strategies, with assumptions, strengths, pitfalls, and short examples.

1) Propensity Score Matching (PSM)- Assumptions: Unconfoundedness/selection on observables (all confounders measured), overlap/common support.- Strengths: Balances observed covariates between treated/control; intuitive and easy to implement with SQL/R/Python.- Pitfalls: Fails with unmeasured confounders; poor overlap reduces sample; sensitive to model specification. Example: Match users who saw a new onboarding modal to similar users by age, engagement, device, prior conversion rate, then compare post-release conversion.

2) Regression Adjustment (covariate control)- Assumptions: Correct functional form; no omitted confounders; linearity if using linear models (or correct link for GLMs).- Strengths: Uses full sample; can adjust many covariates and interactions; fast to run.- Pitfalls: Model misspecification biases estimates; multicollinearity; extrapolation outside support. Example: Regress conversion on treatment indicator + covariates (prior behavior, cohort dummies) to estimate adjusted effect.

3) Difference-in-Differences (DiD)- Assumptions: Parallel trends — treated and control would have evolved similarly absent treatment; no time-varying confounders differentially affecting groups.- Strengths: Controls for unobserved time-invariant confounders; simple pre-post setup.- Pitfalls: Violated by differential trends or concurrent interventions; requires multiple pre-periods to test parallel trends. Example: Control = region without rollout; estimate change in conversion pre/post release difference between regions.

4) Synthetic Controls- Assumptions: Control pool can be weighted to reproduce treated unit’s pre-treatment path; no spillovers.- Strengths: Good for single-unit or aggregate rollouts (e.g., one country); data-driven weighted control approximates counterfactual.- Pitfalls: Needs long pre-period and many unaffected units; sensitive to predictor choice; inference nonstandard. Example: Build synthetic country from other countries’ weighted metrics to estimate effect of feature launched only in Country A.

5) Instrumental Variables (IV)- Assumptions: Instrument relevance (correlates with treatment) and exclusion restriction (affects outcome only through treatment); monotonicity for LATE interpretation.- Strengths: Can address unmeasured confounding when a valid instrument exists (e.g., random rollout timing).- Pitfalls: Hard to find credible instruments; weak instruments bias estimates; LATE may not generalize. Example: Use randomized assignment to early rollout cohorts (instrument) when users self-select into exposure — 2SLS estimates causal effect for compliers.

Practical guidance: pre-register analysis plan, test balance and parallel trends, run sensitivity analyses (placebo windows, Rosenbaum bounds, alternative specifications), report assumptions transparently, and triangulate with multiple methods when possible.

Structuring Ambiguous Business Problems Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank