Covers the skills and practices used to clarify, diagnose, and scope ambiguous business or product problems into actionable problem statements before proposing solutions. Candidates should demonstrate structured and insightful clarifying questions to understand business context, current and desired states, target users and user needs, success metrics and desired outcomes, constraints such as budget, timeline, technical dependencies, and compliance, stakeholder perspectives, and existing performance baselines. Includes separating symptoms from root causes, surfacing and testing hypotheses, identifying data to collect and analyze, performing root cause analysis, breaking complex problems into prioritized subproblems, and defining acceptance criteria and next steps or experiments to reduce uncertainty. Encompasses discovery techniques and basic user research to surface user pain points and opportunities, requirements scoping including scope boundaries, risks and trade offs, and the ability to write a concise problem statement in your own words. At senior levels also assess strategic framing, avoiding premature solutions, aligning stakeholders, and presenting an executive narrative that links diagnosis to measurable outcomes and implementation trade offs; for junior candidates emphasize curiosity, systematic thinking, and the ability to prioritize information needs rather than jumping to implementation.
MediumTechnical
104 practiced
Given a dataset schema and monthly sample sizes for an app, explain how to estimate whether you have enough data to train a 30-day retention prediction model. Describe the calculations or rules-of-thumb you would use (class balance, MDE, number of features), and what additional data you would request if the estimate is insufficient.
Sample Answer
Situation: We need to decide from monthly sample sizes whether we can train a reliable 30-day retention (binary) model.Approach / rules of thumb1. Estimate positive class count:- retention_rate = historical 30-day retention (e.g., 10%).- monthly_users = given. positives_per_month = monthly_users * retention_rate.2. Minimum positive examples:- Classic rule: for logistic regression, aim for 10–20 events per variable (EPV). For modern ML, 50–200 positives per important feature is safer when features are noisy/nonlinear.- Practical floor: at least ~1k–5k positive examples for a robust classifier and stable validation metrics.3. Statistical precision (MDE approach):- For estimating retention proportion or detecting model-driven treatment effects, use proportion sample-size: n = (Z^2 * p(1−p)) / d^2. For 95% CI (Z=1.96), p=0.1, to get ±0.01 precision: n ≈ (1.96^2*0.1*0.9)/(0.01^2) ≈ 34596 total samples (positives+negatives).4. Train/val/test split and imbalance:- After splits, ensure each set has enough positives (e.g., >=200–500 positives in test for stable AUC/confidence).- If imbalance severe, use stratified splits, class weighting, or resampling.Concrete example- monthly_users=50k, retention=10% → positives=5k/month.- If you need 10k positives for model complexity, two months suffice; for 35k total samples to estimate retention within ±1%, one month (50k) is enough.What to measure vs features- Count of unique users, positives/negatives, features per user, sparsity, missingness.- Compute EPV = positives / num_features. If EPV < 10–20, the dataset is likely underpowered for simple models; for complex models target EPV >50.If estimate is insufficient — request:- More months of data (quantify how many by dividing desired positive count by positives_per_month).- Additional signal-rich features per user: session counts, recency/frequency, in-app events, acquisition source, device, geography, marketing exposure, retention labels with time stamps.- Cohort-level labels and experiment/treatment flags to control confounding.- Label quality diagnostics (how retention was computed, deduping, potential censorship).- If increasing samples not possible: request feature engineering (aggregates, embeddings), labeling proxies (shorter retention windows), or plan active learning / targeted labeling and stronger regularization.This gives a reproducible check: compute positives/month, choose EPV or MDE targets, compare to needs, then ask for months or features accordingly.
EasyTechnical
72 practiced
A PM asks for 'improve recommendations' in the app but provides no details. As the ML engineer, list 8 clarifying questions you would ask the PM or stakeholders to scope the request. For each question, include a one-sentence rationale explaining why it matters (e.g., metrics, constraints, evaluation).
Sample Answer
1) What specific business goal should “improve recommendations” serve (e.g., increase conversion, engagement, retention, revenue)? Rationale: Clear objective determines optimization metric and model trade-offs (precision vs. diversity vs. novelty).2) Who is the target audience or user segment for these recommendations? Rationale: Personalization strategy, required features, and cold-start handling depend on user segmentation.3) Which recommendation surface(s) and placement(s) in the app are in scope (home feed, search, checkout, push notifications)? Rationale: Context and UI constraints change latency, model complexity, and evaluation approach.4) What success metrics and evaluation period should we use (CTR, NR, MAU uplift, revenue per user, A/B test duration)? Rationale: Metrics and timeframe guide offline proxies, experiment design, and statistical power calculations.5) What data is available now (user actions, profiles, item metadata, content embeddings) and what is the data retention/privacy policy? Rationale: Feasibility of algorithms and feature engineering depends on available data and legal constraints.6) Are there latency, throughput, cost, or on-device constraints for serving recommendations? Rationale: Production constraints determine model size, caching, and real-time vs. batch scoring architecture.7) Do stakeholders require explainability, fairness, or business rules (e.g., promoted content, no explicit content)? Rationale: Constraints may mandate interpretable models or post-filtering and affect model choice and monitoring.8) What rollout strategy and guardrails do you want (canary, percentage rollout, fallback behavior, rollback criteria)? Rationale: Safe deployment and reliable A/B testing require a clear rollout plan and failure mitigation.
MediumTechnical
61 practiced
A PM requests full personalization across the product, but resources are limited. Describe how you would determine an MVP scope for personalization: list the decision steps, minimal data and model requirements, expected acceptance criteria, and how you would plan to iterate.
Sample Answer
Decision steps (how I'd decide MVP scope)1. Clarify goals & metrics: meet PM to define primary business objective (engagement, retention, conversion) and one or two success metrics (e.g., +5% CTR or +3% retention at 7 days).2. Prioritize user segments: pick high-impact segments (e.g., top 20% revenue or new users) rather than full-user personalization.3. Scope channels & touchpoints: choose 1–2 product surfaces (homepage and recommendation widget) to limit engineering scope.4. Choose personalization depth: start with feature-ranking or rule-augmented ML (hybrid) before deep models.5. Resource check & timeline: map data availability, infra needs, and estimate 8–12 week MVP.Minimal data & model requirements- Data: user IDs, item IDs, basic interaction logs (views, clicks, purchases), timestamps, a small feature store (device, locale, recency). Even sparse signals okay for collaborative filtering + content features.- Model: lightweight, production-ready model: e.g., matrix factorization or factorization machines / gradient-boosted decision tree for ranking; or session-based nearest-neighbor. Prefer models with fast training and low inference latency.- Infra: simple feature pipeline, batch training job, and online scoring via cache.Expected acceptance criteria (objective)- KPI uplift: statistically significant improvement on primary metric in A/B test (predefined threshold).- Latency & reliability: P95 inference latency under SLA, <1% error rate.- Data coverage: personalization applicable to ≥30% target segment.- Maintainability: retraining automated weekly; model rollback path exists.Iteration plan- Phase 0 (Weeks 0–2): Data audit, feature engineering, baseline (non-personalized) metric.- Phase 1 (Weeks 2–6): Build MVP model + offline evaluation; deploy as shadow (no user impact) and AB-test small cohort.- Phase 2 (Weeks 6–10): Full A/B test on selected segments; monitor metrics, fairness, and cold-start behavior.- Phase 3+: Expand segments, add richer features (demographics, NLP embeddings), upgrade to deeper models if gains justify cost.Trade-offs & rationale- Start simple to minimize infra and explainability risk; prioritize measurable business impact and extensibility. If cold-start is major, include simple content-based rules until model accumulates signals.
EasyTechnical
106 practiced
You are asked to propose success metrics for an email spam classifier feature. Describe at least five metrics (including business-oriented metrics), explain the trade-offs between them (e.g., precision vs. recall), and provide a short example of an acceptance threshold that ties to business impact.
Sample Answer
1) Precision (spam predicted / total predicted spam) — measures false positive rate indirectly. High precision reduces legitimate email lost to spam folder (customer trust).2) Recall (true spam caught / total actual spam) — measures false negatives. High recall reduces user exposure to spam/phishing.3) False Positive Rate (legitimate flagged as spam / total legitimate) — a business-critical metric: misclassified important emails cause support costs and churn.4) False Negative Rate (spam missed / total spam) — correlates to security risk and user annoyance.5) User-reported complaint rate / Unblock rate — real-world feedback; captures cases not visible in labeled test set and indicates business impact.6) Delivery/engagement KPIs: open rate of inbox, click-through rate on legitimate emails — downstream business metrics affected by overaggressive filtering.Trade-offs:- Precision vs Recall: increasing recall (catch more spam) usually lowers precision (more false positives). Choose direction based on business risk: for financial/phishing-heavy segments prioritize recall; for enterprise customers where missing an email is costly prioritize precision.- FPR vs churn/ops cost: small increases in FPR can cause outsized customer support costs.- Lab metrics vs user-reported: optimizing only for test-set F1 can miss real-world feedback.Acceptance threshold example:- SLA: FPR <= 0.5% and Recall >= 92% on production traffic, user complaint rate <= 0.05% monthly. Rationale: at 0.5% FPR across 100M legit emails, expected 500k misplacements — acceptable only if complaint rate stays below 0.05% (50k complaints); otherwise rollback. This ties model quality to support costs and user trust.
MediumSystem Design
62 practiced
Design an A/B test to validate a new ML model intended to reduce cart abandonment. Specify: the primary metric, secondary/guardrail metrics, how you would randomize, treatment assignment rules, a high-level approach to compute minimum detectable effect (MDE), expected experiment duration considerations, and rollout steps for progressive launch.
Sample Answer
Primary metric- Cart conversion rate (CCR): proportion of sessions with an added-to-cart event that convert to purchase within the session or defined window (e.g., 24–72 hrs). Use session-level or user-session-level denominator depending on implementation.Secondary / guardrail metrics- Purchase value / AOV (average order value) — ensure model doesn't reduce basket size.- Checkout completion rate and funnel drop-off at checkout steps.- Refund/chargeback rate and product returns.- Latency/error rate of model-serving (system health).- Customer experience signals: NPS/CSAT (if available) or click-to-purchase latency.Randomization- Unit: user-id (or cookie/device) to avoid cross-contamination. If users frequently use multiple devices, prefer logged-in user_id.- Stratify randomization by key covariates: prior purchase frequency (new vs returning), device type (mobile/desktop), geography and traffic source to balance baseline CCR.- Use deterministic hashing of user_id -> bucket to ensure consistent exposure.Treatment assignment rules- Control: current baseline checkout/model.- Treatment: new ML model that recommends interventions (e.g., personalized promotions, timing of nudges, predictions used to pre-fill/optimize checkout).- Exclude: internal testers, bots, and users with recent experiments; apply feature flags to route traffic to model endpoint.- Ensure fail-safe: if model serving fails, route to control.MDE calculation (high-level)- Inputs: baseline CCR p0, desired statistical power (1-β, typically 80-90%), significance α (two-sided 0.05), allocation ratio r (usually 1:1).- Approximate per-group sample size: n ≈ [ (Z_{1-α/2} * sqrt(2 p̄(1-p̄)) + Z_{1-β} * sqrt(p0(1-p0)+p1(1-p1)) )^2 ] / (p1 - p0)^2 where p̄=(p0+p1)/2 and p1 = p0 + MDE.- Solve iteratively for MDE given expected daily traffic and experiment duration budget; or compute required days given a target MDE.- For small p0, use proportions formula or convert to log-odds for stability. If measuring revenue per user, use t-test sample size with estimated variance.Duration considerations- Minimum duration: run at least 2 full weekly cycles (14 days) to cover weekday/weekend patterns; often 28 days recommended.- Ensure sufficient sample size based on MDE calculation; allow for attrition and multiple testing adjustments.- Monitor sequentially but avoid peeking—use pre-specified sequential methods (alpha-spending, Pocock/BONFERRONI, or Bayesian stopping rules) if early stopping is needed.Progressive rollout steps1. Canary (1% of eligible users) for 48–72 hours to validate correctness, latency, and no obvious negative signals.2. Small ramp (5–10%) for 1–2 weeks; monitor guardrails and key signals.3. Medium ramp (25–50%) for 1–2 weeks; begin powered analysis when sample met.4. Full rollout if statistically significant improvement on primary metric with no harm to guardrails.5. If mixed results, perform segmentation analysis (by device, cohort) before deciding rollback or targeted rollouts.6. Post-launch: monitor long-term metrics (retention, returns) and model drift; schedule A/B regressions periodically.Key practical notes- Pre-register hypothesis, primary metric, analysis plan, and launch criteria.- Use ITT (intention-to-treat) analysis; secondary per-protocol analyses as diagnostics.- Instrument logging for deterministic bucketing, exposures, and model decisions for debugging and causal attribution.
Unlock Full Question Bank
Get access to hundreds of Problem Definition and Framing interview questions and detailed answers.
