DoorDash Machine Learning Engineer Interview Preparation Guide (Junior Level)

Direct answer

Before writing any code, look for two structural signals: does the problem have overlapping subproblems and optimal substructure (an optimal solution is built from optimal solutions to smaller versions of itself)? If yes, it is a dynamic programming (DP) problem, not a greedy one. Within DP, whether you reach for memoization (caching recursive-call results, computed top-down) or tabulation (filling a table iteratively, bottom-up) is a secondary implementation choice, not a correctness question: both compute the same recurrence.

Structured elaboration

Signal 1: does a locally optimal choice guarantee a globally optimal one? Greedy algorithms make one irrevocable choice at each step and never reconsider it. That is only correct when the problem has the "greedy-choice property": committing to the best-looking option right now cannot make the final answer worse. You test this by trying to construct a counterexample where the locally-best choice forecloses a better global outcome (an exchange argument): if you can build one, greedy is wrong and you need DP; if every attempt to build a counterexample fails and you can sketch why (an exchange argument that any optimal solution can be rearranged to match the greedy choice without loss), greedy is likely correct.

Signal 2: overlapping subproblems and optimal substructure. If solving the problem for a larger input naturally requires solving the same smaller subproblem many times (for example, "the best way to reach state k" depends on "the best way to reach state k-1", but state k-1 also gets asked about from other paths), you have overlapping subproblems. If, in addition, an optimal solution to the whole problem is composed of optimal solutions to its subproblems (no locally-suboptimal subproblem answer can still lead to a globally optimal whole), you have optimal substructure. Both together mean DP applies: cache each subproblem's answer once, reuse it everywhere it recurs.

Signal 3: what does the recurrence look like? Write the recurrence in terms of "the answer for state X depends on the answer for smaller states Y, Z, ...", before touching code. If you can write this recurrence but it does not have an ordering where "smaller" always resolves before "larger" (a genuine dependency cycle), you likely need a different technique entirely (graph shortest-path with cycles, for instance).

Once you know it's DP: memoization vs tabulation. These are the same recurrence expressed two ways, not two different algorithms:

Worked example

Take "minimum coins to make amount 6 from denominations {1, 3, 4}" (the coin change problem). The recurrence is: minCoins(a) = 1 + min(minCoins(a - c) for c in coins if c <= a), with minCoins(0) = 0. Overlapping subproblems are visible immediately: computing minCoins(6) needs minCoins(5), minCoins(3), minCoins(2); computing minCoins(5) also needs minCoins(2). minCoins(2) gets requested from two different callers, so caching it once and reusing it is exactly what turns an exponential naive recursion into a linear-in-target one. That overlap is the tell that this is DP, not greedy: a greedy "always take the largest coin" would take 4 then 1 then 1 (3 coins), while the true optimum is 3 + 3 (2 coins), because taking the largest coin first forecloses the better pairing, a real exchange-argument counterexample, confirming greedy is unsafe here and DP (with either memoization or tabulation) is required.

Trade-offs & pitfalls

Key points

• The most common mistake is reaching for greedy because a locally-best choice feels right; the discipline is to actively try to break it with a counterexample before trusting it, not to trust it by default.
• A DP recurrence existing does not by itself tell you whether to memoize or tabulate; that choice depends on whether the reachable state space is sparse (favors memoization) or dense with a clean iteration order (favors tabulation), and on language-specific recursion-depth limits.
• Some problems only look like DP: if there is no genuine overlap (each subproblem is only ever needed once), plain recursion or divide-and-conquer is simpler and DP's caching buys you nothing.

Complexity

• These are meta-level signals, not a specific algorithm, so there is no single complexity here; once you commit to DP, complexity is (number of distinct states) times (work per state), whether computed top-down with a cache or bottom-up with a table.

Edge cases

• A problem with optimal substructure but no overlapping subproblems (each subproblem solved once) does not need DP's memoization; plain recursion or divide-and-conquer suffices and adding a cache only adds overhead.
• A problem where you cannot write a clean dependency order for tabulation (irregular, data-dependent state transitions) may force memoization even in a dense-looking state space, since an explicit iteration order is hard to construct correctly.

Problem formalization:

• Environment: M services i∈{1..M}, T decision rounds. Each round t we allocate a vector of discrete resources a_t = (a_{t,1},...,a_{t,M}) from a feasible budget set A (e.g., ∑i c_i a{t,i} ≤ C, per-host caps).
• Reward: stochastic QoS reward r_t = ∑i R_i(a{t,i}, s_{t,i}) where s_{t,i} are unknown service states (load, SLO attainment); we observe rewards (possibly delayed/noisy) after allocation. Objective: maximize cumulative expected reward E[∑_{t=1}^T r_t] (or minimize regret vs best fixed allocation/policy).
• Unknowns: response curves R_i(·) (monotone, possibly concave), cross-service interactions (contention).

Why this is a combinatorial bandit:

• Each action is a combinatorial object (vector allocation under constraints) drawn from an exponential-size set A. Reward decomposes (possibly approximately) over arms (services) but choice couples via budget constraints. Observations may be semi-bandit (per-service feedback) or bandit (only aggregate). This matches combinatorial multi-armed bandits (CMAB).

Algorithmic solutions:

1. Approximate Combinatorial UCB (CMUCB):

• Maintain per-service estimates μ̂_i(a) or parametric models (e.g., diminishing returns curve). Compute optimistic estimates μ̂_i + bonus_i(t) and solve an offline knapsack/ILP to pick allocation maximizing sum of optimistic rewards subject to constraints. Use approximation-aware regret bounds: if offline oracle is α-approx, regret scales O((poly)·T/α). Use structured models (GLM, parametric) to reduce sample complexity.

2. Greedy with submodular objectives:

• If ∑i R_i(a{i}) is monotone submodular in discrete resource units, use greedy hill-climbing per round: iteratively allocate marginal unit with largest estimated marginal gain. With (1-1/e) approximation and confidence bonuses, this gives provable approximate regret. Practical variant: Thompson Sampling + greedy sampling to handle uncertainty.

3. Contextual/parametric bandits:

• Use contextual CMAB: include features (current load, queues). Fit online Bayesian/linear models and use LinUCB/Thompson Sampling over allocations with efficient combinatorial optimization via knapsack solvers.

4. Handling interactions:

• If strong cross-coupling, treat joint arms for small groups or use low-rank factorization of interaction matrix and learn latent factors.

Practical monitoring and risk controls:

• Safety constraints: enforce hard SLO-preserving actions (never drop below reserve allocations), and clipping of exploratory allocations.
• Conservative bootstrapping: begin with off-line A/B tests or simulation to initialize priors; use decaying exploration rates.
• Canary & rollback: stage policy in canary cluster with traffic shadowing; automatic rollback on SLO degradation beyond thresholds.
• Observability: per-service QoS, latencies, error rates, resource usage, and reward attribution with high-cardinality logs; compute regret/expected reward delta vs baseline in real-time.
• Alarm & intervention: require human-in-loop for high-impact allocation changes, throttle exploratory updates when variance or delayed feedback increases.
• Explainability: surface marginal gains used for each allocation decision to operators.

Trade-offs:

• Exact CMAB algorithms give theoretical guarantees but need efficient oracles; greedy/submodular methods scale well and are robust if submodularity approx holds. Contextual models reduce samples but require reliable features.

This design balances provable algorithms with practical safety: start with parametric/contextual CMUCB + greedy allocation, strong monitoring and safety gates, then iterate as more data reduces uncertainty.

Direct answer
Build the vectorizer and classifier as stages of one scikit-learn Pipeline, then pass a parameter distribution dictionary to RandomizedSearchCV using the double-underscore stepname__paramname convention so both stages' hyperparameters are sampled and evaluated jointly, not tuned separately.

Structured elaboration
Tuning the two stages jointly (rather than tuning the vectorizer once and then the classifier on top of that fixed choice) matters because the best vectorizer setting can genuinely depend on the classifier, and vice versa, an interaction a two-stage sequential tuning approach would miss. RandomizedSearchCV treats the whole Pipeline as one estimator, so cross-validation correctly refits the ENTIRE pipeline (including the vectorizer) on each fold's training data, avoiding any leakage of validation-fold vocabulary into the vectorizer's fit.

Worked example (executed)

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

pipe = Pipeline([("tfidf", TfidfVectorizer()), ("clf", LogisticRegression(max_iter=1000))])
param_dist = {
    "tfidf__ngram_range": [(1, 1), (1, 2)],
    "tfidf__min_df": [1, 2],
    "clf__C": loguniform(1e-2, 1e2),
}
rs = RandomizedSearchCV(pipe, param_dist, n_iter=10, cv=4, random_state=0).fit(docs, labels)

Run against a small synthetic text-classification dataset (120 documents, 2 classes): the search found best params {'clf__C': 1.57, 'tfidf__min_df': 2, 'tfidf__ngram_range': (1, 1)} at CV accuracy 1.0 (a clean synthetic separation, as expected for this toy dataset), confirming the joint search correctly samples and evaluates both stages together within cross-validation.

Trade-offs & pitfalls
It's easy to accidentally fit the TfidfVectorizer once on the full dataset BEFORE cross-validation (outside the Pipeline) as a "preprocessing step," which leaks validation-fold vocabulary and IDF statistics into training; keeping the vectorizer INSIDE the Pipeline, refit fresh on each fold, is what prevents this specific and easy-to-miss leakage.

Requirements & constraints:

• Zero-downtime rolling updates for a model server that needs warm-up time and must drain in-flight requests.
• Kubernetes Deployment with probes and lifecycle hooks to avoid dropped requests.

Manifest (key parts):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0     # never take replicas down -> avoid capacity loss
      maxSurge: 1           # allow one extra pod while updating
  minReadySeconds: 30     # ensure pod stays ready for warm-up period
  template:
    metadata:
      labels: { app: model-server }
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: model
        image: myregistry/model-server:latest
        ports: [{ containerPort: 8501 }]
        readinessProbe:
          httpGet:
            path: /ready        # returns 200 only after model warm-up & loaded
            port: 8501
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 6   # allows ~30s of transient failures
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /healthy      # quick health check for process responsiveness
            port: 8501
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - |
                # Ask server to stop accepting new requests and drain
                curl -sS -X POST http://127.0.0.1:8501/drain || true
                # give app time to finish in-flight requests
                sleep 60

Why these choices and how they avoid dropped requests:

• strategy.maxUnavailable: 0 + maxSurge:1 ensures new pods start before old ones are removed, preserving capacity during rollout.
• minReadySeconds enforces that a pod stays ready long enough for warm-up (e.g., model loading) before being considered available to the Service.
• readinessProbe points to an endpoint that returns success only after the model is fully loaded (warm-up). Kubernetes will not send traffic to a pod until this probe passes, preventing requests to a cold model.
• livenessProbe catches crashed/hung processes but has a longer initialDelay to avoid false positives during load time.
• terminationGracePeriodSeconds gives the container time to finish draining; preStop invokes an application-level drain endpoint to stop accepting new requests and then sleeps to allow in-flight requests to complete. The app should handle SIGTERM to stop accepting new connections immediately and complete ongoing work.
• Combined, these settings ensure new pods become READY before they get traffic and old pods stop receiving new traffic and finish in-flight requests before termination, preventing dropped requests during rolling updates.

Operational notes:

• Implement /ready, /healthy, /drain and proper SIGTERM handling in the model server (stop accepting new connections, finish active requests, flush metrics).
• Tune timeouts (minReadySeconds, sleep, failureThreshold) according to model load times and tail-latency SLAs.

Approach

arr[start:stop:step] selects elements starting at index start, stops before index stop (stop is never included), and takes every step-th element; any of the three can be omitted, and each omitted piece defaults based on the sign of step. Slicing a built-in sequence like list always builds and returns a brand-new list; it is never a view into the original, so mutating the slice result never touches the original list and vice versa.

Code (Python 3.12)

arr = [0, 1, 2, 3, 4]

print(arr[1:4])     # [1, 2, 3]
print(arr[-3:])      # [2, 3, 4]
print(arr[::-1])     # [4, 3, 2, 1, 0]
print(arr[:])        # [0, 1, 2, 3, 4]
print(arr[:] is arr)  # False: a distinct new list object
print(arr[:] == arr)  # True: equal contents

copy_ = arr[:]
copy_.append(999)
print(arr)      # [0, 1, 2, 3, 4]        (unchanged)
print(copy_)    # [0, 1, 2, 3, 4, 999]   (only the copy grew)

Output:

[1, 2, 3]
[2, 3, 4]
[4, 3, 2, 1, 0]
[0, 1, 2, 3, 4]
False
True
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4, 999]

Key points

• arr[1:4] walks indices 1, 2, 3 (stopping before 4), giving [1, 2, 3].
• arr[-3:]: -3 counts from the end (index len(arr) - 3 = 2), and an omitted stop defaults to the end of the sequence for a positive step, giving indices 2, 3, 4, or [2, 3, 4].
• arr[::-1]: both start and stop are omitted and step is -1. For a negative step, an omitted start defaults to the last index and an omitted stop defaults to "one before the first element" (conceptually, off the left end), so Python walks the whole sequence right to left, giving the fully reversed list [4, 3, 2, 1, 0].
• arr[:]: every field omitted, step defaults to 1, start to 0, stop to len(arr), giving every element in original order, which is also the idiomatic way to make a shallow copy of a list.
• Slicing always constructs a new list object (arr[:] is arr is False), so mutating copy_ after slicing never touches arr, confirmed above by appending to copy_ and observing arr is unaffected.

Complexity and edge cases

Slicing built-in sequences costs $O(k)$ time and memory, where $k$ is the length of the resulting slice, since every selected element's reference has to be copied into the new list.

• A slice like arr[::-1] on a large list copies the entire list's worth of references at once; for a very large sequence this is a real, if bounded, allocation, not free.
• Indices outside the sequence's bounds do not raise; Python clamps them (arr[100:200] returns [], arr[-100:2] returns [0, 1]), which differs from single-index access (arr[100]), which does raise IndexError.
• This "always a copy" behavior is specific to Python's built-in sequences (list, tuple, str, bytes). It is not a universal rule across every library: NumPy array slicing, for instance, returns a view sharing the same underlying memory by default rather than copying, which is a deliberate, different design choice worth knowing about separately if you work with array-heavy code, but is not how the built-in sequence types behave.

Direct answer

For a feature that needs to generate text grounded in specific private data, retrieval-augmented generation is usually the safer default over a large language model (LLM) alone, because it grounds each answer in retrievable source material rather than relying entirely on what the model happened to learn or memorize.

Structured elaboration

• Factuality and hallucination risk. An LLM used alone has no mechanism to guarantee its output reflects your actual private data; it can produce fluent, plausible-sounding text that's simply wrong. Retrieval augmentation grounds generation in retrieved passages, which substantially reduces (though doesn't eliminate) hallucination, since the model has real source text to draw from.
• Latency. Retrieval adds a lookup step before generation, so it's typically slower than calling the model alone; whether that's acceptable depends on the feature's latency budget.
• Cost. Retrieval augmentation usually means a smaller, more tightly-scoped generation step (since the retrieved context does some of the work), which can be cheaper per request than relying on model scale alone to "know" everything.
• Up-to-dateness. Retrieval naturally handles data that changes frequently (update the retrieval index, no retraining needed), while an LLM alone reflects whatever was true when it was trained or fine-tuned.

Worked example

For generating a description of a specific listing grounded in that listing's actual attributes, retrieval augmentation (pulling the listing's real attributes and feeding them into the generation step) directly prevents the model from inventing plausible-sounding but false details about that specific listing, which an LLM alone would be prone to when the listing's specifics aren't something the base model was trained on. If the underlying data changes frequently (attributes get updated often), retrieval also means the feature stays current without needing to retrain anything.

Trade-offs and pitfalls

The main risk of retrieval augmentation is a bad retrieval step: if the wrong or incomplete source material is retrieved, the generation step will still confidently produce fluent, ungrounded text, just now grounded in the WRONG source. Investing in retrieval quality (making sure the right passages are actually being found) matters as much as the generation step itself, and is often the harder engineering problem in practice.

Direct answer

Choose by the decision the score feeds, not by algorithm preference. Build a ladder of baselines first (a no-model base rate, a one-line business rule, then a regularized logistic regression), then challenge the best of them with a gradient-boosted tree ensemble. A small neural network is the last thing to try on a table of customer features. Ship the simplest model that beats the baseline on a time-ordered holdout by more than the run-to-run noise, on the metric that matches the action (for example, how many real churners are in the slice of users you can afford to contact), and whose probabilities are trustworthy enough to put a price on (calibrated: among customers scored near 20% risk, about 20% really churn).

Terms used below

• Churn: a subscriber cancels or fails to renew within a fixed window after the date the score is computed.
• Regularized logistic regression: a linear model of the log-odds of churn. The L2 penalty shrinks coefficients so that correlated features do not produce huge offsetting weights.
• Tree ensemble (gradient boosting): many shallow decision trees built one after another, each fitted to the mistakes of the ones before. It finds thresholds and interactions without hand-made columns.
• Small neural network: layers of weighted sums with nonlinearities between them. Flexible, but on a table of a few dozen columns it needs scaling, tuning and more data to match boosted trees.
• AUC: the chance that a random churner gets a higher score than a random non-churner. PR-AUC (average precision) summarizes precision against recall and is far more sensitive to the top of the ranking when churn is rare. Brier score: mean squared error of the predicted probabilities, lower is better, and it rewards honest probabilities.
• Calibration: whether scores behave like probabilities. A calibration check groups customers by score and compares the average predicted risk with the share who actually churned. A calibration step is a small correction fitted on held-out data that maps raw scores to observed churn rates (two common ones are Platt scaling, a logistic curve fitted to the scores, and isotonic regression, a monotone step function).
• Reason codes and attribution tooling: a reason code is a short, plain reason for one customer's score (for example "logins fell by 7"). A linear model gives them directly from coefficient times feature value; for a tree ensemble you need attribution methods (for example SHAP values) that split one score into per-feature contributions.
• Time-ordered folds: several train/test splits that all train on earlier months and test on the next block, sliding forward, so each test block is in the future of its training data.

The baseline ladder (what to build first)

1. Base rate. Everyone gets the training churn rate. AUC is 0.5 by construction. This sets the floor for every metric, including Brier.
2. One-line rule, such as "fewer logins in the last 30 days means riskier". Stakeholders already believe it, so any model has to beat it to justify its cost.
3. Regularized logistic regression on a handful of clean features. This is the real baseline: it trains in seconds, its coefficients can be read aloud, and its probabilities are often reasonably calibrated before any extra step.

Only then add the challengers: a tree ensemble, and a neural network if there is a reason (raw text, event sequences, or embeddings that a table does not capture).

How to decide among the three

The deciding rule: take the challenger only if its lead on the action metric survives repeated time-ordered folds (train on earlier months, test on the next one, slide forward) and the team can monitor and explain it. A lead that appears on one split and vanishes on the next is noise.

Worked example

The table below is simulated so that every number can be reproduced. One simulated fact matters for reading the result: new subscribers who also log in rarely are much more likely to churn, an interaction between two columns that a plain logistic regression cannot represent unless someone builds that column. The first 14,000 rows stand in for the earlier period and the last 6,000 for the later period used as the test. After the main table the program also adds the interaction as an explicit logistic column, checks calibration in the top slice, and slides a time-ordered fold forward four times to measure how much the boosted-tree lead moves.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss

rng = np.random.default_rng(0)
n = 20000
tenure = rng.integers(1, 48, n)                      # months subscribed
logins = rng.poisson(8, n).astype(float)             # logins in last 30 days
logins_prev = rng.poisson(8, n).astype(float)        # logins in the 30 days before
tickets = rng.poisson(0.5, n).astype(float)          # support tickets last 90 days
late_pay = rng.binomial(1, 0.1, n).astype(float)     # a failed payment recently
X = np.column_stack([tenure, logins, logins_prev, tickets, late_pay])

drop = logins_prev - logins                           # fall in engagement
z = (-3.4 + 0.25 * np.maximum(drop, 0) + 0.5 * tickets + 1.2 * late_pay
     - 0.02 * tenure + 2.5 * ((tenure < 6) & (logins < 6)))   # new + disengaged interaction
y = rng.binomial(1, 1 / (1 + np.exp(-z)))

split = 14000                                         # rows are ordered; later rows = test
Xtr, Xte, ytr, yte = X[:split], X[split:], y[:split], y[split:]
print("churn rate train/test:", round(ytr.mean(), 3), round(yte.mean(), 3))

def report(name, p, is_prob=True):
    brier = f"{brier_score_loss(yte, p):.4f}" if is_prob else "n/a"
    print(f"{name:26s} AUC={roc_auc_score(yte, p):.3f} PR-AUC={average_precision_score(yte, p):.3f} Brier={brier}")

base_rate = np.full(len(yte), ytr.mean())
report("base rate only", base_rate)
rule_score = 1.0 / (1.0 + Xte[:, 1])   # rule: fewer logins = riskier (AUC is rank-only; not a calibrated probability)
report("rule: fewer logins", rule_score, is_prob=False)
lr = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000)).fit(Xtr, ytr)
report("logistic (L2, C=1)", lr.predict_proba(Xte)[:, 1])
hgb = HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, max_iter=200, random_state=0).fit(Xtr, ytr)
report("gradient boosted trees", hgb.predict_proba(Xte)[:, 1])
mlp = make_pipeline(StandardScaler(), MLPClassifier((16,), alpha=1e-2, max_iter=300, random_state=0)).fit(Xtr, ytr)
report("small neural net (16)", mlp.predict_proba(Xte)[:, 1])

# --- logistic regression with the interaction added as an explicit column ---
def add_interaction(A):
    return np.column_stack([A, (A[:, 0] < 6) & (A[:, 1] < 6)])
lrx = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000)).fit(add_interaction(Xtr), ytr)
report("logistic + interaction col", lrx.predict_proba(add_interaction(Xte))[:, 1])

# --- calibration: in the 10% of users scored highest, does mean predicted risk match actual churn? ---
print("\ncalibration in the top 10% by score (mean predicted vs actual churn rate):")
for name, p in (("logistic", lr.predict_proba(Xte)[:, 1]), ("gradient boosted trees", hgb.predict_proba(Xte)[:, 1])):
    top = np.argsort(-p)[: len(p) // 10]
    act = yte[top].mean()
    se = (act * (1 - act) / len(top)) ** 0.5          # sampling noise of an observed rate in a slice this size
    print(f"  {name:24s} predicted={p[top].mean():.3f}  actual={act:.3f} (+/- {se:.3f})   (all users: predicted={p.mean():.3f} actual={yte.mean():.3f})")

# --- run-to-run noise: slide a time-ordered fold forward (train on everything earlier, test on the next 4,000 rows) ---
print("\nfold  train rows  PR-AUC logistic  PR-AUC boosted  lead of boosted")
leads = []
for start in (4000, 8000, 12000, 16000):
    tr, te = slice(0, start), slice(start, start + 4000)
    a = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X[tr], y[tr])
    b = HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, max_iter=200, random_state=0).fit(X[tr], y[tr])
    pa = average_precision_score(y[te], a.predict_proba(X[te])[:, 1])
    pb = average_precision_score(y[te], b.predict_proba(X[te])[:, 1])
    leads.append(pb - pa)
    print(f"{start // 4000:4d}  {start:10d}  {pa:15.3f}  {pb:14.3f}  {pb - pa:15.3f}")
print(f"lead of boosted trees over the folds: mean {np.mean(leads):.3f}, smallest {min(leads):.3f}, largest {max(leads):.3f}")

churn rate train/test: 0.07 0.064
base rate only             AUC=0.500 PR-AUC=0.064 Brier=0.0602
rule: fewer logins         AUC=0.663 PR-AUC=0.119 Brier=n/a
logistic (L2, C=1)         AUC=0.748 PR-AUC=0.260 Brier=0.0545
gradient boosted trees     AUC=0.768 PR-AUC=0.336 Brier=0.0513
small neural net (16)      AUC=0.752 PR-AUC=0.275 Brier=0.0537
logistic + interaction col AUC=0.764 PR-AUC=0.340 Brier=0.0510

calibration in the top 10% by score (mean predicted vs actual churn rate):
  logistic                 predicted=0.233  actual=0.237 (+/- 0.017)   (all users: predicted=0.070 actual=0.064)
  gradient boosted trees   predicted=0.252  actual=0.270 (+/- 0.018)   (all users: predicted=0.070 actual=0.064)

fold  train rows  PR-AUC logistic  PR-AUC boosted  lead of boosted
   1        4000            0.215           0.283            0.068
   2        8000            0.273           0.313            0.039
   3       12000            0.267           0.318            0.051
   4       16000            0.264           0.327            0.063
lead of boosted trees over the folds: mean 0.055, smallest 0.039, largest 0.068

How to read it. The no-information model sits at AUC 0.500 with Brier 0.0602, and the logistic regression improves Brier to 0.0545. The tree ensemble leads on this table: AUC 0.768 against 0.748, and PR-AUC 0.336 against 0.260. That PR-AUC gap is the one that matters if the retention program contacts only a small top slice of users. Part of the reason is the built-in interaction, so the size of the gap on real data is an empirical question. The logistic (0.748) and neural network (0.752) AUCs differ by 0.004 on a single seed and a single split, which is not evidence that either is better.

Three further lines of output settle the points the table leaves open.

• Interaction column. Adding the explicit column ((tenure < 6) and (logins < 6)) lifts logistic regression to AUC 0.764, PR-AUC 0.340 and Brier 0.0510, level with the boosted trees (0.768, 0.336, 0.0513). So here it recovers the whole gap, which is expected because the simulation has exactly one interaction and we knew where to look. On real data you rarely know which interaction to add, and finding them automatically is what the trees buy you.
• Calibration. Among the 10% of users scored highest (600 users), logistic predicts 0.233 and 0.237 actually churn; the boosted trees predict 0.252 and 0.270 churn. The two gaps, 0.004 and 0.018, are each about one standard error of an observed rate in a slice this size (the program prints +/- 0.017 and +/- 0.018), so this run does not show that either model is miscalibrated. Drawing the simulation again moves each gap by a few hundredths in either direction, and which model looks better calibrated in the top slice changes with it. The practical point is that a calibration check on a slice of a few hundred users cannot resolve a gap of a couple of points: pool the time-ordered test blocks, or use a larger slice, before concluding that a calibration step is needed. The trees have the better overall Brier score because Brier rewards sharper ranking as well as honest levels. If a gap that survives that larger check matters for pricing, fit a calibration step on held-out data and re-check.
• Run-to-run noise. Across four time-ordered folds the boosted-tree PR-AUC lead over logistic is 0.068, 0.039, 0.051 and 0.063: positive in all four here, but it moves by about 0.03 from fold to fold, and on other draws of the simulation the smallest fold lead can fall to roughly zero. So "more than the run-to-run noise" means a lead that stays clearly positive in every fold, not one that is only large on average. A PR-AUC difference of a few thousandths on one split is well inside that spread, and the 0.004 AUC gap between logistic and the neural network above is the same kind of single-split difference: nothing in this run shows it would repeat.

On this table the call is: ship the tree ensemble if its PR-AUC lead repeats across several time-ordered folds and the team can explain it and monitor it. If the lead shrinks to noise, ship the logistic regression with the interaction added as an explicit column, which, as the output above shows, closed the gap on this table and keeps the explanation simple.

What would flip the choice

• A requirement to give each customer reason codes, or an audit that demands a transparent model, favors logistic regression.
• Very few churners in training (a few hundred) makes flexible models unstable; favor regularized logistic regression.
• Strong interactions or thresholds in the data, and plenty of churners, favor boosted trees.
• Raw sequences, text or images as inputs are the case for a neural network.

Common mistakes

• Choosing on a random split instead of a later period, which flatters every model.
• Reporting overall AUC when the business contacts only the top few percent; report the metric at the contact capacity.
• Skipping calibration when the probabilities feed an expected-value calculation.
• Jumping to a neural network without a baseline, so nobody knows what the complexity bought.
• Comparing models on different feature sets or after tuning on the test period.

Direct answer

Scope the MVP to the smallest slice that proves the model's predictive signal actually changes what a real user sees, and pick a single online metric, measured against a real control group, as the pass/fail bar. Cut everything that exists to make the system durable or scalable rather than to answer "does this work"; put the cuts in writing before you build, not after.

Structured elaboration

What to deliberately leave out: automated retraining, a full monitoring/alerting stack, a general-purpose feature store abstraction (a static, one-time feature snapshot is enough), autoscaling and multi-region serving, and a complex model architecture. A strong, simple baseline (logistic regression or gradient-boosted trees) beats a deep model here, because the two-week clock is spent proving a signal exists, not squeezing out the last point of accuracy.

What must NOT be cut, even under a two-week deadline:

• A holdout evaluation built on a time-based split (train on earlier data, evaluate on later data), so the offline number isn't inflated by a split that leaks future information into training.
• A real serving path that returns live predictions to real users, not a static demo or a slide deck of offline numbers.
• A randomized control group that does not see the feature, so any difference observed can be attributed to the feature and not to some other change happening the same week.

Choosing the single success metric: it should be (a) observable inside the two-week window, (b) causally close to the actual decision the feature drives, and (c) resistant to being gamed by how the demo population was chosen. That rules out offline accuracy or area-under-curve as "the" metric: those measure whether the model separates classes well on held-out data, not whether a real user changed their behavior because of the prediction. The right choice is almost always an online lift metric (click-through rate, conversion rate, or an equivalent behavioral signal) compared between a treatment group that sees the feature and a randomized control group that does not.

Communicating the corners cut: write a short scope memo before building anything, shared with whoever is funding the demo, that lists what's out of scope, tags each cut as "fine for the demo, not fine for a real launch," and states the single success metric in advance so nobody can pick a friendlier metric after seeing the results. When presenting the outcome, show the metric result and the list of cut corners side by side, so the audience judges both together instead of treating a good number as proof of production readiness.

Worked example

Suppose the baseline conversion rate without the feature is 5% and the team expects the feature to lift it to 6% (a 20% relative lift, a reasonable target for a first version). To decide how many of the 100,000 users to route into the test, size the experiment for that effect using the standard two-proportion sample-size approximation:

$n = \frac{(z_{\alpha/2} + z_{\beta})^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_2 - p_1)^2}$

With $z_{\alpha/2}=1.96$ (two-sided, 95% confidence) and $z_{\beta}=0.84$ (80% power), $p_1=0.05$, $p_2=0.06$:

$p_1(1-p_1) = 0.05 \times 0.95 = 0.0475, \quad p_2(1-p_2) = 0.06 \times 0.94 = 0.0564$

$n = \frac{(1.96+0.84)^2 \times (0.0475+0.0564)}{(0.06-0.05)^2} = \frac{7.84 \times 0.1039}{0.0001} \approx 8{,}146 \text{ per group}$

So roughly 16,300 users total (about 8,150 per arm) are enough to reliably detect a 20% relative lift, well inside the 100,000-user pool. That means the scope memo can honestly say: "we are testing on ~16% of the user base, sized to detect a 20% relative lift; if the true lift is smaller than that, the test may come back inconclusive, and that is a stated risk, not a hidden one." That sentence is exactly the kind of corner-cutting disclosure the question is asking for, grounded in a number instead of a vague caveat.

Trade-offs & pitfalls

• Picking offline AUC as the headline metric because it's available by day 10 is a common wrong turn: it cannot answer whether any real user behaved differently, which is the question stakeholders will actually ask afterward.
• Skipping the randomized control to save time removes the only way to separate the feature's effect from unrelated fluctuation (day-of-week effects, an unrelated marketing push), which defeats the purpose of the demo.
• Building durable infrastructure (autoscaling, a general feature store) for a two-week demo spends the time budget on properties the demo doesn't need and that will likely be rebuilt once real requirements are known.
• Cutting corners silently, without writing them down, sets the demo up to be judged against a production bar nobody actually committed to meeting.

Direct answer

Expected profit per 100,000 predictions is $175: the extra conversions are worth $375 and serving the model costs $200. Compare two model sizes by running the same calculation for each (value of the extra conversions, minus serving cost, minus any conversions lost to slower responses) and picking the higher expected profit per prediction, after checking that the difference is bigger than your measurement error.

The calculation, step by step

Definitions: lift is the increase in conversion probability the model causes (here from 0.5% to 0.75%, a lift of 0.25 percentage points). The conversion is worth $1.50 (treated here as profit per conversion; if $1.50 is revenue, multiply by the margin first). Every prediction costs $0.002 whether or not it converts.

1. Extra conversions: 100,000 x (0.0075 - 0.005) = 250.
2. Value of extra conversions: 250 x $1.50 = $375.
3. Serving cost: 100,000 x $0.002 = $200.
4. Expected profit: $375 - $200 = $175.

The 500 conversions that would have happened anyway (100,000 x 0.5%) are not credited to the model: only the lift is. Crediting all 750 conversions would give $1,125 - $200 = $925 and overstate the value fivefold.

Break-even: a prediction pays for itself if lift x $1.50 >= $0.002, so lift >= 0.00133, or 0.133 percentage points. This model clears that by a factor of about 1.9 (0.25 / 0.133).

Comparing model sizes

Per 100,000 predictions, profit = (lift x $1.50 x 100,000) - (cost per prediction x 100,000) - latency penalty. The latency penalty is the conversion lost because a slower model makes the page slower. It must come from an experiment, not intuition. In this illustration it is assumed to be 0.02 percentage points of conversion for each extra 50 ms, measured from the small model's 40 ms.

(The medium model's penalty is 0.02 x (90 - 40) / 50 = 0.02 pp, and the large model's is 0.02 x (200 - 40) / 50 = 0.064 pp.)

Reading the table:

• The large model loses money: its extra $0.008 per prediction over the small model buys at most 0.2 percentage points more lift, worth $0.003 per prediction against $0.008 more cost.
• Medium versus small depends on the latency assumption. Ignoring latency, medium earns $200 against $175, a clear win. With the penalty it earns $170 against $175, a tie. The right move is to measure the latency effect in an A/B test before paying for the bigger model.
• Choose the small model, because the gain from medium is inside the uncertainty of the inputs and small is cheaper and faster. Revisit if the value per conversion rises, since a higher value shifts every row toward bigger models.

Trade-offs and pitfalls

• Cost is incurred on every prediction, but the benefit only appears on those that change an outcome. A model is a good business when the benefit per prediction exceeds the cost per prediction, not merely when accuracy rises.
• Lift must come from a randomised comparison against no model (or the previous model). Offline accuracy gains do not translate one-for-one into conversion lift.
• Include fixed costs (training, monitoring, on-call) in a full comparison. Per-prediction cost alone ignores them.
• Compare on the same basis: per 100,000 predictions on both sides, with the value per conversion held constant.

Direct answer
Precise wording matters because a reader acts on what the words literally say, not on what the writer meant; vague or ambiguous phrasing lets each reader fill in the gap with their own assumption, and those assumptions frequently disagree.

Structured elaboration

• Ambiguity creates silent forks in understanding. Two readers of the same imprecise sentence can each walk away confident they understood it, while holding two different, incompatible interpretations, and neither realizes there's a disagreement until it surfaces later, usually at a worse time.
• Vague quantifiers are a common culprit: words like "soon," "significant," or "most" mean different things to different readers and different things in different contexts.
• Precision does not mean verbosity. A precise sentence can be shorter than a vague one; "by Thursday 5pm" is both more precise and no longer than "soon."
• To enforce more precise terminology: agree on and write down a small shared glossary for terms that get used loosely (what counts as "done," what "urgent" means for this team), review drafts specifically looking for vague quantifiers and ambiguous pronouns ("it," "this") whose referent isn't obvious, and normalize asking "what do you mean by X specifically?" in review rather than letting it pass.

Worked example
Imprecise: "We'll ship the fix soon, once we've done a bit more testing."

What actually happened: one stakeholder read "soon" as "later today" and told a customer to expect it that day; the engineering team read it as "sometime this week" because "a bit more testing" meant a multi-day regression pass. The customer was told an incorrect date because two people read the same sentence and reasonably reached different conclusions.

Precise version: "We'll ship the fix by end of day Thursday, pending a two-day regression test that starts tomorrow."

Same information, but now both readers have the same understanding, and if the regression test finds something, "Thursday" is a concrete promise that either holds or needs an explicit update, rather than a vague one that quietly slips.

Trade-offs and pitfalls

• Being maximally precise about everything is exhausting and unnecessary for low-stakes communication; reserve the rigor for statements other people will act on or make commitments based on.
• Precision can be used dishonestly too, to sound more certain than you actually are; if you genuinely don't know the date, the honest and still precise move is "I don't have a firm date yet, I'll confirm by Wednesday," not a confident-sounding guess.
• Enforcing a shared glossary only works if it's actually referenced in practice, not just written once and forgotten; it needs to show up in review habits, not just documentation.