Machine Learning Problem Solving Framework Questions
Present a structured end to end approach to machine learning problems: clarify the business goal and constraints, define success metrics, audit and prepare data, design candidate features and baselines, select models and evaluation protocols, iterate on error analysis, and plan deployment and monitoring. Include considerations for trade offs among accuracy, latency, and scalability, and produce a prioritized plan with milestones, experiments, and rollback criteria.
MediumTechnical
17 practiced
For a churn-prediction problem, how would you decide between a simple logistic regression and a gradient-boosted tree model?
Sample Answer
**Start with the decision, not the algorithm**The choice depends less on which model scores marginally higher offline and more on what the business needs from the deployed system: interpretability, latency, maintenance burden, and how much signal is genuinely nonlinear.**When logistic regression wins**- Stakeholders (e.g. compliance, retention team) need to understand *why* a user was flagged, and logistic regression coefficients are directly interpretable.- The relationship between features and churn is roughly monotonic and additive (e.g. more support tickets and less usage both independently increase churn risk).- Training/serving simplicity matters, fewer moving parts, easier to debug and retrain quickly.**When gradient-boosted trees win**- There's a moderate-to-large dataset with meaningful feature interactions (e.g. 'low usage AND recent price increase' matters more than either alone), which trees capture without manual interaction terms.- The team can tolerate less direct interpretability, using SHAP values as a substitute for coefficient interpretation.- The marginal accuracy gain translates to real business value, e.g. a few points of AUC change materially shifts how many customers get a retention offer.**My actual process**I'd build both: logistic regression as an interpretable baseline, then a gradient-boosted model. If the tree model doesn't meaningfully beat the linear model's held-out performance (accounting for the cost of added complexity in serving and monitoring), I'd default to the simpler model. Winning by 0.5 AUC points isn't worth the added operational complexity unless that translates to real dollars.**Example**If logistic regression gets 0.74 AUC and gradient boosting gets 0.76 AUC, but the retention team needs to explain flags to a legal reviewer, I'd likely still ship logistic regression, or a tree model with mandatory SHAP-based explanations attached to each flag.
EasyTechnical
13 practiced
You've just been handed a raw dataset for a new ML project. Before you start engineering features, what are you actually checking?
Sample Answer
**Understand what the data represents**First I confirm how each column was generated: is it logged at event time or backfilled later, does it come from a system prone to outages, and does the label (if any) reflect the outcome I actually care about or a proxy for it.**Data quality checks**- Missingness: which columns have gaps, and is the missingness random or systematic (e.g. a field only populated after a product launch date)?- Duplicates: are there repeated rows or repeated entities that would leak information across train/test splits?- Distribution shift: does the data cover the time period and population the model will actually see in production, or is it a biased historical slice?- Label quality: for a churn label, was it computed consistently, and does the label leak future information (e.g. a 'churned' flag that is only set after a cancellation email, which itself depends on other model outputs)?**Leakage specifically**I check whether any feature is only known after the outcome occurs. For example, 'number of support tickets about cancellation' is a common leakage trap in churn models, it's a symptom of churn, not a cause I can act on in advance.**Volume and granularity**I check whether there's enough history and enough positive examples to train and evaluate reliably, and whether the data is at the right grain (per-user, per-session, per-day) for the decision the model needs to support.**Why this comes first**Skipping this step is how models end up with inflated offline metrics driven by leakage, that then collapse in production. Auditing the data first is cheaper than debugging a failed launch later.
HardTechnical
13 practiced
You have limited compute budget and limited stakeholder patience for a project. How would you prioritize a sequence of experiments across data quality fixes, feature engineering, and model architecture changes to maximize your odds of hitting a target business metric?
Sample Answer
**Principle: spend the cheapest, highest-signal experiments first**With a tight budget on both compute and stakeholder attention, the ordering matters as much as the ideas themselves. I prioritize by expected-impact-per-unit-of-effort, and I sequence so that each experiment tells me something even if it fails, rather than running expensive experiments that leave me no better informed.**Step 1: Data quality audit (cheap, high leverage)**Before any modeling work, I'd spend a small, fixed amount of time (e.g. 1-2 days) checking for the highest-leverage data problems: leakage, mislabeled examples, missing values concentrated in an important segment, or a broken upstream feature. Fixing a data bug often moves the metric more than any amount of model tuning, and it's cheap to check.**Step 2: Baseline and error analysis (cheap, diagnostic)**Build the simplest reasonable model and do error analysis before investing in anything sophisticated. This tells me where the gap to the target metric actually lives: is it concentrated in a segment (points to feature or data work) or spread evenly (points more toward model capacity or fundamentally missing signal)?**Step 3: Feature engineering (medium cost, usually highest ROI next)**Guided by error analysis, I'd prioritize features that address a diagnosed gap over speculative ones. Feature work is typically cheaper than architecture changes and often yields larger gains, because most of the ceiling on a tabular business problem comes from what signal is even present, not how it's modeled.**Step 4: Model architecture changes (highest cost, run last and only if justified)**Only once feature and data avenues are exhausted, or error analysis clearly shows the current model can't capture necessary interactions or patterns, would I invest compute in trying a fundamentally different architecture. I'd run this as a bounded, time-boxed experiment with a clear go/no-go criterion set in advance (e.g. 'if a gradient-boosted model doesn't beat the linear baseline by X on the validation set within one week of tuning, we stop and ship the simpler model').**Communicating the plan**I'd present this as a milestone plan to the stakeholder up front: 'Week 1: data audit and baseline. Week 2: feature iteration based on error analysis. Week 3: model architecture experiment, only if needed, with a defined stopping rule.' That way, if we run out of patience or budget partway through, we've captured the highest-value gains first, and the stakeholder knows exactly what was and wasn't attempted rather than being surprised by an open-ended search.**What would change my ordering**If error analysis in step 2 immediately reveals a clear, well-understood nonlinear pattern the linear baseline structurally can't capture, I would front-load architecture work, this ordering is a strong default, not a rigid rule, and the diagnostic step is what should actually drive the sequence.
HardTechnical
23 practiced
You're asked to predict which users will upgrade to a paid plan, but the team wants results in two weeks and half the users have significant data gaps. How do you scope the project, and what trade-offs would you flag to the stakeholder?
Sample Answer
**Reject the framing that everything must ship in scope**With a two-week deadline and a serious data quality problem affecting half the population, the honest move is not to quietly build the best model I can on the available data, it's to make the trade-off explicit and let the stakeholder choose, since 'two weeks' and 'good predictions for all users' may not both be achievable.**Scoping options I'd present**1. **Scope down the population**: build and ship a model that only serves the half of users with complete data, and flag the rest as 'no prediction available, use existing manual process.' This gets something reliable to production in two weeks, but only covers half the base.2. **Build a two-tier approach**: a full model for users with complete data, plus a coarser model or rule-based heuristic (e.g. based on whatever sparse signals exist) for the gapped half, with a clear confidence flag distinguishing the two.3. **Push the timeline**: explain that reliably fixing the data gap (understanding why it exists, backfilling if possible) likely takes longer than two weeks, and ship a rough estimate later with a better one.**My recommendation and reasoning**I'd lean toward option 2: it delivers value across the full user base within the deadline, while being explicit that predictions for the gapped segment carry lower confidence. I would NOT quietly impute the missing half's data with a generic method and present one model with uniform confidence, that risks the retention/upgrade team acting on garbage predictions without knowing it, which is worse than not having a model for that segment at all.**What I'd communicate to the stakeholder, in numbers where possible**- Expected precision/recall for the complete-data segment versus the estimated reliability for the gapped segment.- The size of the gapped segment (e.g. 'this is 40% of revenue, not a rounding error').- A concrete proposal for closing the data gap over the following month, so the two-tier situation is temporary, not permanent.**Why this is the senior move**A two-week deadline under real data constraints tests judgment, not modeling skill: the risk isn't building a bad model, it's silently shipping unequal reliability across users without saying so, which erodes trust when someone eventually notices the gapped segment's predictions are unreliable.
MediumTechnical
15 practiced
How do you decide on the train/validation/test split strategy for a model that predicts next week's demand for a retail product?
Sample Answer
**Why a random split is wrong here**Demand forecasting is a time series problem, using a random row-level split would let the model 'see the future' during training, for example training on next month's data and validating on last month's, which massively overstates offline performance.**The right approach: time-based splitting**I'd split strictly by time: train on data up through some cutoff date, validate on the period immediately after, and hold out the most recent period as a final test set that mimics exactly how the model will be used, predicting forward from the latest known data.This reflects that in production the model will always predict forward from 'now', never interpolate within a known period.**Additional considerations**- **Multiple time windows**: I'd use rolling-origin (walk-forward) validation, evaluating on several different cutoff points, not just one, so I'm not overfitting to quirks of a single validation window (e.g. a holiday spike).- **Seasonality**: I'd make sure the validation window includes representative seasonal patterns (e.g. don't validate only on a non-holiday month if the model needs to handle holiday demand).- **Entity leakage**: if products or stores share features (e.g. a promotion feature that's known for the whole chain), I'd double check that the split doesn't leak store-level averages computed using future-window data back into earlier training features.**Why this matters**Getting this wrong is one of the most common ways demand-forecasting models look great offline and then embarrass the team when actual forecasts are systematically biased in production, because the offline evaluation never actually tested the model's ability to extrapolate forward in time.
text
[-------- train --------][-- validation --][-- test --]
time increases -->Unlock Full Question Bank
Get access to hundreds of Machine Learning Problem Solving Framework interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.