Clarifying Questions and Scoping Questions

Covers the practice of turning vague or open ended prompts into well scoped problems by asking targeted clarifying questions and setting explicit assumptions. Candidates should show how they surface constraints, stakeholders, success metrics, timelines, dependencies, and edge cases; balance seeking information with moving forward; translate discovery into acceptance criteria or an initial experiment; and sequence inquiry to reduce risk. Interviewers evaluate the quality of the questions, the candidate's ability to frame sensible assumptions, and how the candidate converts discoveries into actionable next steps or measurable outcomes.

EasyTechnical

71 practiced

Design a short framework (3-5 steps) you would use to convert a vague business prompt into measurable acceptance criteria for a data science task. Explain the purpose of each step and how it reduces ambiguity.

MediumTechnical

101 practiced

You must translate stakeholder requirements into an experiment that tests whether a recommendation increases repeat purchases. Draft the hypotheses (null and alternative), primary metric, minimal sample size considerations, and clarify dependencies you would ask about before starting.

Sample Answer

Hypotheses- Null (H0): Showing the recommendation does not change the repeat-purchase rate within 30 days (p_treatment = p_control).- Alternative (H1): Showing the recommendation increases the repeat-purchase rate within 30 days (p_treatment > p_control).

Primary metric- Primary: Repeat-purchase rate within 30 days per user (binary: did the user make another purchase within 30 days of index purchase). Report absolute uplift and relative lift.- Secondary: Revenue per user (RPU) within 30 days, average order value for repeat orders, click-through and add-to-cart rates on recommended items, negative signals (uninstalls, complaints).

Minimal sample size considerations (power calc for proportions)- Choose α = 0.05 (two-sided) and power = 0.8 (β = 0.2).- Need baseline p0 (control repeat rate) and Minimum Detectable Effect (MDE) as absolute uplift you care about.- Sample size formula (approx for two proportions): n_per_arm ≈ [ (z_{α/2} √(2 p̄ (1−p̄)) + z_{β} √(p0(1−p0)+p1(1−p1)) )^2 ] / (p1−p0)^2 where p̄=(p0+p1)/2.- Example: if p0=0.10 and we want MDE = 0.02 (to 0.12), z_{α/2}=1.96, z_β=0.84 → n ≈ 3,840 users per arm (so ~7,680 total).- If metric is continuous (e.g., RPU), use t-test sample size with baseline mean, std dev, and desired absolute change.

Practical adjustments- Inflate sample by ~10–20% for non-compliance, instrumentation loss, and attrition.- If randomization unit is session or device, adjust for intra-class correlation (effective sample size reduction) using design effect: n_effective = n_observed / (1 + (m−1) ICC), where m is cluster size.

Dependencies / questions to clarify before starting- Randomization unit: user-level (recommended) vs session/device/account? Are users shared across devices?- Definition of repeat purchase: timeframe (30 days?), what counts as a purchase (refunds, fraudulent orders)?- Baseline rate & historical variance: what is current repeat-purchase rate and variability by cohort/segment?- Traffic & timeframe: expected eligible users per day/week; any business seasonality or upcoming promotions?- Exposure logic: how and when is recommendation shown (frequency caps, position, UI variants)? Are we guaranteed exposure when assigned?- Instrumentation: Are recommendation impressions, clicks, cart adds, and purchases reliably logged and linkable to user IDs?- Overlap with other experiments or campaigns: any concurrent tests or marketing that could confound results?- Segmentation needs: Do stakeholders want results overall and by key segments (new vs returning users, geography, product category)?- Guardrails: acceptable negative impacts (e.g., lower AOV or increased complaints), rollback thresholds.- Privacy/legal constraints: any PII/consent issues with tracking or personalization.

Analysis plan (brief)- Pre-specify primary metric, MDE, test duration, and stopping rule (avoid peeking).- Use ITT analysis; also report treatment-on-treated (exposure) as secondary.- Run balance checks on covariates; if imbalanced, consider covariate-adjusted models (logistic regression for binary outcome).- Report confidence intervals for uplift and p-values, and practical significance.

This gives a testable hypothesis, the metric and sample-size method, and the operational clarifications needed to run a robust experiment.

HardTechnical

99 practiced

You are asked to identify edge cases that could cause a model to fail in production. Describe the clarifying questions you'd ask about data distributions, user behaviors, adversarial inputs, and system failures, and describe how you would prioritize these edge cases for mitigation.

Sample Answer

First I’d clarify scope and success criteria, then ask targeted questions in four areas and describe a prioritization framework.

Clarifying questions- Data distributions - How was training data sampled? Dates, geography, segments? - Have you measured covariate shift / label shift between train/validation/production? - Are there known sparsely represented subgroups or heavy-tailed features? - How often do features become missing or new categories appear?

- User behaviors - What typical user workflows produce inputs? Any seasonal or campaign-driven spikes? - Do users intentionally or accidentally submit malformed or extreme values? - Are there authentication/permission tiers that change available features?

- Adversarial inputs - What incentives exist for gaming the model (fraud, spam, SEO)? - Can inputs be manipulated at scale (bots, API abuse)? - Are there integrity checks or provenance metadata available?

- System failures - What upstream data pipelines can backfill incorrect values or delay data? - What’s the latency SLA and retry logic? How do downstream consumers handle bad predictions? - What monitoring/alerting exists for data drift, feature distributions, and model performance?

Examples of edge cases- Sudden feature distribution shift after a marketing campaign- New categorical level unseen in training- Coordinated adversarial attacks (e.g., automated account creation)- Partial feature outage due to pipeline failure- Extremely imbalanced rare-event subgroup with high business impact

Prioritization approach- Score each edge case on: likelihood, business impact (cost/risk), detectability, and mitigation cost/time.- Rank by expected risk = likelihood × impact, adjusted by detectability (higher priority if hard to detect) and mitigation effort.- Quick wins: add validations and fallbacks (schema checks, default features, reject or route to human review) for high-likelihood/low-effort issues.- High-impact but low-likelihood: build monitoring, holdout “canary” deployments, and adversarial testing (fuzzing, red-team).- For systemic risks (data pipeline bugs): invest in pipeline contracts, linters, and end-to-end tests.

Operationalize- Implement automated drift and feature-check alerts, unit tests for feature engineering, adversarial test suites, and runbooks for failover (safe default prediction, human-in-loop).- Communicate trade-offs and a roadmap to stakeholders: immediate mitigations, monitoring, and longer-term robustness work.

MediumTechnical

89 practiced

You need to scope the KPI and monitoring plan for an experiment that personalizes notifications. Clarify the questions to define engagement metrics, negative side-effects (e.g., churn), statistical thresholds, and how to surface regressions post-launch.

Sample Answer

Situation: We’re running an A/B experiment to personalize push/email notifications and must define KPIs, detect harms (churn, spam complaints), set statistical thresholds, and put post-launch regression detection in place.

Clarifying questions:- Business goal: lift what? DAU, conversion, retention, revenue per user?- Targeting scope: all users or a segment? Frequency caps and channels?- Expected treatment behavior: different message content, timing, or send probability?- Risk tolerance: acceptable uplift tradeoffs vs. churn/spam?- Available data latency and instrumentation (real-time vs batch)?

Engagement metrics (primary + guardrails):- Primary KPI: 7-day active conversion rate (users who opened and completed target action within 7d).- Secondary KPIs: click-through rate (CTR), open rate, session starts per user, time-to-action.- Engagement depth: avg events per DAU, feature-specific events.

Negative side-effects to monitor:- Short-term: opt-outs/unsubscribes rate, spam complaints, increase in “do not disturb” toggles.- Medium-term: 28-day retention, churn rate, revenue churn, support tickets.- Behavioral churn signals: decreased session frequency, longer inter-session time.

Statistical thresholds & analysis:- Minimum Detectable Effect (MDE) computed per KPI given baseline, alpha=0.01 for guardrail metrics (more stringent), alpha=0.05 for primary, power=0.8. Use sequential testing correction (alpha spending, e.g., O’Brien-Fleming) if peeking.- Use uplift with covariate adjustment (e.g., CUPED) to reduce variance.- Test types: two-sided for engagement, one-sided for expected improvements; non-parametric bootstrap for skewed metrics (revenue).

Regression detection & alerting post-launch:- Real-time / daily monitoring pipeline with: - Dashboards tracking primary + guardrails by cohort, country, channel. - Automated statistical checks: compare rolling 7-day windows to pre-launch baseline using z-tests or permutation tests; flag if effect crosses threshold and persists >48 hours. - Health checks: sudden jump in opt-outs > X SDs or relative increase > Y% triggers pager + kill-switch workflow.- Post-launch rollback criteria: significant harm on any guardrail (alpha-adjusted) sustained for 48-72 hours or major revenue impact.- Root-cause tooling: ability to slice by user cohort, device, channel, message template; store event-level logs for quick replay.

Implementation sketch (sample SQL for daily CTR check):

sql

-- daily CTR by cohort
SELECT
  cohort,
  SUM(clicks)::float / NULLIF(SUM(impressions),0) AS ctr,
  COUNT(DISTINCT user_id) as users
FROM notification_events
WHERE event_date BETWEEN current_date-7 AND current_date-1
GROUP BY cohort;

Key rationale:- Combine primary conversion with engagement depth to avoid optimizing superficial signals.- Use strict guardrail thresholds and sequential-corrected tests to avoid false positives from peeking.- Automate detection with clearly defined escalation and rollback to limit user harm while enabling rapid learning.

HardTechnical

71 practiced

A stakeholder requests a model that requires 99.9% precision on fraud alerts. What clarifying questions do you ask to understand feasibility, dataset label quality, sample prevalence, operational costs of false negatives, and whether alternative workflows could satisfy the requirement?

Sample Answer

I’d start by clarifying the business requirement, then drill into data, costs, and alternatives. Key questions and why I’m asking:

1) Business context & tolerance- What exactly does "99.9% precision" mean operationally (per alert, per day, per account)? - Why is precision prioritized over recall? What harm do false positives cause (customer friction, cost to investigate)? Rationale: ensures we optimize the right metric.

2) Label quality & ground truth- How were labels generated? Human-reviewed, heuristics, or post-facto? - What’s label consistency and inter-rater agreement (Cohen’s kappa)? Any known label drift or systematic bias? Rationale: precision target is meaningless if labels noisy.

3) Sample prevalence & dataset size- What’s the historical prevalence of fraud in the dataset? How many positive examples (and unique fraud types)? - Are there holdout/time-split datasets or near-real-time data for validation? Rationale: very low prevalence makes high-precision thresholds easy (but may kill recall) and affects confidence intervals.

4) Evaluation & statistical confidence- Over what sample size should 99.9% be certified? Desired confidence interval (e.g., 95% CI)? Rationale: compute required test positives to statistically claim 99.9% precision.

5) Operational cost of errors- Cost of a false negative (missed fraud): financial loss, regulatory, reputational? - Cost of false positive: investigation cost, customer churn, SLA penalties? Rationale: cost-sensitive loss functions or decision thresholds may be needed.

6) Deployment constraints & latency- Real-time vs batch? Allowed latency for investigations? Integration points and feedback mechanisms for labels? Rationale: affects model architecture and feature availability.

7) Alternative workflows- Can we use tiered responses (automated low-risk alerts + human review for borderline cases)? - Are business rules / deterministic checks acceptable for high-precision subset? - Is a risk-scoring approach with SLA-based routing possible instead of hard threshold? Rationale: sometimes a hybrid workflow meets business goals with lower model demands.

8) Monitoring, retraining & governance- How will we monitor precision post-deployment and refresh labels? SLAs for model performance? Rationale: ensures long-term compliance with the 99.9% target.

Next steps I’d propose: compute baseline precision/recall from historical model or heuristics, estimate sample size needed to validate 99.9% with confidence, run error-cost sensitivity analysis, and prototype a hybrid workflow (high-precision rule-based pass + ML for recall) to compare operational trade-offs.

Unlock Full Question Bank

Get access to hundreds of Clarifying Questions and Scoping interview questions and detailed answers.

Join thousands of developers preparing for their dream job.