DoorDash Applied Scientist (Senior Level) - Comprehensive Interview Preparation Guide

Direct answer

When low GPU utilization traces specifically to gradient synchronization delays, the three highest-leverage fixes are overlapping communication with backward computation, fusing many small gradient tensors into fewer, larger collective calls, and compressing what's communicated, each attacking a different source of the synchronization overhead.

Structured elaboration

• Overlap compute and communication: issue each parameter's (or bucket's) all-reduce as soon as its gradient is ready during backward, using an async collective on a separate CUDA stream, rather than waiting for the entire backward pass to finish before starting any communication; implemented via PyTorch DDP's built-in gradient bucketing and backward hooks, or manually with async_op=True and stream management.
• Tensor fusion (bucketing): instead of one collective call per parameter tensor (high per-call fixed overhead when there are thousands of small parameters), group multiple tensors into fixed-size buckets and issue one collective per bucket; this amortizes the fixed latency cost of each collective call over more data per call.
• Gradient compression: quantizing gradients to lower precision (or using top-k sparsification) before communication reduces the bytes actually moved across the network, directly cutting the bandwidth-bound portion of synchronization time, at the cost of some additional numerical/convergence trade-off that needs validation against the specific model.

Worked example

For each technique, in a PyTorch codebase: overlap is implemented by registering a gradient hook per parameter that issues dist.all_reduce(grad, async_op=True) on a dedicated stream as soon as that parameter's gradient is computed, deferring .wait() calls until immediately before optimizer.step() (DDP does exactly this internally by default, so this fix is often "verify DDP's overlap is actually working" rather than hand-rolling it). Tensor fusion means configuring DDP's bucket size (bucket_cap_mb) appropriately: too small reintroduces per-call overhead, too large delays when the first bucket's communication can start overlapping with ongoing backward. Gradient compression requires wrapping the communication hook (DDP's register_comm_hook API) with a quantization/sparsification step before the actual collective call, trading some implementation complexity for reduced bytes moved.

Trade-offs & pitfalls

Compute-communication overlap and tensor fusion are close to free wins (no accuracy trade-off, mostly an engineering/configuration exercise) and should be applied first; gradient compression introduces a genuine numerical trade-off (compressed gradients are a biased or noisier estimate of the true gradient) and should be validated against convergence quality on the actual model and dataset before committing to it in production, not assumed safe by default.

Direct answer

Using the accounting (banker's) method, charge each append a constant amortized cost of $c = \frac{b}{b-1}$ for a growth factor b, and show that the credit this overcharge accumulates between resizes is always enough to pay for the next resize's copying cost; that gives a total cost of n appends bounded by $O(n)$, so $O(1)$ amortized per append. Redoing the same argument for $b = 1.5$ gives $c = 3$ instead of $c = 2$ (doubling's value): the amortized bound of $O(1)$ per append still holds for any fixed growth factor greater than 1, only the constant changes.

Structured elaboration

Setup

Assume, for the proof, that capacity grows through the sequence $1, b, b^2, \ldots$ (the standard simplifying assumption; real implementations round capacities to integers, which does not change the asymptotic result). A resize happens each time the array is full and one more element is appended: the m elements already present are copied into a new array of capacity $b \times m$, then the new element is inserted.

Accounting (banker's) method

Charge every append operation, resize or not, a fixed amortized cost $c = \frac{b}{b-1}$. An ordinary append (no resize) has actual cost 1, so it banks a credit of $c - 1 = \frac{1}{b-1}$ on the element just inserted. Between one resize (at capacity m) and the next (at capacity $bm$), exactly $(b-1)m$ ordinary appends happen before the array fills again, each banking $\frac{1}{b-1}$ in credit:
$(b-1)m \cdot \frac{1}{b-1} = m$
That accumulated credit, m, is exactly enough to pay the actual cost of copying the m existing elements at the next resize (the resize's own new element is paid for directly by that operation's own charge $c$). Since every operation's amortized charge is bounded by the constant c and no operation is ever left with a deficit, the total actual cost of n appends is at most $c \cdot n = \frac{b}{b-1} n$, which is $O(n)$, giving $O(1)$ amortized cost per append.

Aggregate-method cross-check

The same bound falls out of the aggregate method directly: resizes happen at capacities $1, b, b^2, \ldots, b^{K-1}$ for some K with $b^{K-1} < n \le b^K$, and the total number of elements ever copied across all resizes is the sum of those capacities:
$\begin{aligned} \text{TotalCopyCost} &= \sum_{k=0}^{K-1} b^k = \frac{b^K - 1}{b - 1} \\ &< \frac{b \cdot b^{K-1}}{b - 1} \le \frac{b}{b-1}\, n \end{aligned}$
Adding the n direct insertion costs, total cost is $O(n)$, matching the accounting-method bound exactly.

Redoing it for b = 1.5 versus b = 2

Plugging each growth factor into $c = \frac{b}{b-1}$:
$b = 2 \Rightarrow \frac{b}{b-1} = 2$
$b = 1.5 \Rightarrow \frac{b}{b-1} = 3$
Both are finite constants, so the amortized bound of $O(1)$ per append holds for either growth factor; what changes is only the constant itself, not the asymptotic class. Doubling copies each element roughly 2 times on average over its lifetime in the array; growth factor 1.5 copies each element roughly 3 times on average, in exchange for a smaller memory ceiling: at any moment, the wasted (allocated but unused) capacity is bounded by $(b-1)$ times the current size, so doubling can waste up to 100% of the array's used size while 1.5x growth wastes up to 50%.

Worked example

Trace the accounting method on a concrete sequence with $b = 2$: starting from capacity 1, resizes occur when appending the 2nd, 3rd, 5th, and 9th elements, at capacities 1, 2, 4, and 8 respectively. Copy costs at those four resizes are 1, 2, 4, and 8 elements respectively (summing the previous capacity each time), totaling $1+2+4+8=15$ copies for 9 appends, i.e., $15/9 \approx 1.67$ copies per append, comfortably under the proof's bound of $c - 1 = 1$ extra copy-credit per ordinary append plus the resize's own direct cost, and well under the $O(n)$ total the proof guarantees.

Trade-offs & pitfalls

The most common wrong turn is trying to bound the worst single append's cost rather than the total cost across all n appends: any individual resize genuinely costs $O(m)$, not $O(1)$, so a correct argument must be amortized (spread across the whole sequence) rather than claiming every append is cheap in isolation. A second common gap is treating "amortized $O(1)$" as meaning every append is fast in practice: a resize append is still a real, comparatively expensive operation the moment it happens (relevant for tail latency, even if the average over many appends is small), which matters if the system cares about worst-case per-operation latency rather than only total throughput. On the growth-factor trade-off itself, a common oversimplification is treating 1.5 as strictly better because it wastes less memory, without naming that it does so by paying for roughly 50% more total copying over the structure's lifetime; the right choice depends on whether the system is more memory-constrained or more copy-cost-sensitive.

Direct answer

Because a tree predicts by falling into a leaf and returning that leaf's training-data statistic, any input beyond the range of values the tree saw during training lands in the same boundary leaf as the most extreme training examples and gets the same flat prediction, no matter how far out it goes. This bites hardest in trending or growing quantities (time, price, size, demand) where production data routinely drifts past the training range. A common fix is a hybrid: a linear (or otherwise extrapolation-aware) component captures the global trend, and a tree ensemble models the nonlinear residual structure on top of it.

Structured elaboration

Where it bites in practice.

• Time-indexed or growth features: a model trained on 2023-2024 sales data has never seen "days since launch = 800"; if the true relationship keeps growing, the tree flatlines at the value of its last training bucket.
• Monotonic business quantities: square footage, years of experience, account age, order volume; any of these can exceed the training range as the business scales, and the tree silently under- or over-predicts by defaulting to boundary-leaf behavior instead of failing loudly.
• Distribution shift in production: even without a true trend, if the training sample happened not to cover the full feature range (a sampling gap, not real extrapolation), a tree behaves identically: flat prediction at the edge, with no signal that it's operating out of its comfort zone.

Why trees behave this way. A regression tree's prediction is the mean of the training targets in the leaf a query point falls into; splits are chosen only within the observed range of each feature. There is no notion of "beyond the last split point, keep going in the same direction," because a tree has no parametric form to project outward with. Gradient boosting inherits the same property leaf by leaf, even though the ensemble as a whole can produce a step-like approximation.

Hybrid linear + tree design. Model the target as $y \approx g(x) + h(x)$, where $g$ is a simple linear (or otherwise monotonic, extrapolation-safe) term capturing the global trend, and $h$ is a tree ensemble capturing nonlinear interactions and local structure. A practical two-stage recipe:

1. Fit $g$ (e.g., ordinary least squares, or a linear model with monotonic constraints on the trending feature) to the full training data.
2. Compute residuals $r_i = y_i - g(x_i)$.
3. Fit a tree ensemble to the residuals $r$, using the remaining features (interactions, nonlinearities the linear term misses).
4. At inference, predict $\hat y(x) = g(x) + h(x)$; the linear term keeps extrapolating sensibly outside the training range, while the tree term contributes whatever local correction it learned, which naturally saturates near its own training boundary rather than dominating out-of-range behavior.

An alternative that avoids a strict two-stage pipeline: feed the linear model's prediction in as an additional feature to the tree ensemble, letting the trees learn corrections on top of it; this is more flexible but requires care that the ensemble doesn't simply relearn (and then flatten) the linear signal itself.

Worked example

Take a toy pricing relationship $\text{price} = 100{,}000 + 200 \times \text{sqft}$, and suppose training data covers sqft in $[500, 2800]$. At the training boundary, $\text{price}(2800) = 100{,}000 + 200 \times 2800 = 660{,}000$; a pure tree's boundary leaf predicts this same $660{,}000$ value for any query at or beyond 2800 sqft, since that leaf's training examples top out there. Query a 5000 sqft property: the tree still outputs $660{,}000$, a flat prediction, while the true (and the linear component's) value is $100{,}000 + 200 \times 5000 = 1{,}100{,}000$, an underprediction of $1{,}100{,}000 - 660{,}000 = 440{,}000$, exactly the extrapolation gap the linear term is meant to close. In the hybrid, the linear term alone accounts for the full trend at 5000 sqft, and the tree-fitted residual term only needs to add whatever local nonlinear correction it learned near the edge of its own training range, not carry the entire extrapolation.

Trade-offs & pitfalls

• The hybrid only helps if the linear term's functional form is roughly right; if the true trend is itself nonlinear beyond the training range, a plain linear extrapolation can be just as wrong as the tree's flat one, only in a different direction.
• Fit the two stages with the same cross-validation discipline as any two-stage pipeline: compute residuals only from a model trained on the current fold's training data, never on data that includes the validation or test fold, or the residual-fitting step leaks information.
• Monotonic constraints (available natively in XGBoost and LightGBM) are a lighter-weight alternative to a full linear+tree hybrid when you only need the tree ensemble itself to keep moving in a known direction past the training range, without a separate linear model.
• In production, the real defense is detection, not just architecture: log when incoming features exceed the training range and flag those predictions as extrapolated, since even a hybrid model's confidence in that regime is lower than the metrics computed on in-range validation data suggest.

Direct answer

With around 100,000 labeled examples for the new domain, fine-tuning a strong pre-trained foundation model is usually the better starting point: it needs far less compute and data to reach a good result than training from scratch, and the transfer risk is manageable at that data volume. Training from scratch only wins when the new domain is so different from the pretraining distribution that transfer actively hurts, or when licensing or architecture constraints rule out using an existing foundation model.

Structured elaboration

• Compute cost. From scratch requires training a full model to convergence, which is dramatically more expensive in GPU-hours than fine-tuning a subset of an already-converged model's parameters.
• Expected performance. At 100,000 examples, a fine-tuned foundation model typically outperforms a from-scratch model of comparable size, because the pretrained weights already encode general structure that a from-scratch model would need far more data to learn on its own.
• Data requirements. Fine-tuning needs enough labeled data to adapt the model without catastrophic forgetting or overfitting to quirks of the small set; 100,000 examples is comfortably in the range where fine-tuning works well for most modalities.
• Transfer-learning risk. The main risk is negative transfer: if the new domain's underlying structure is fundamentally different from what the foundation model saw in pretraining (a highly specialized technical domain with unusual vocabulary or structure), fine-tuning can inherit biases that hurt more than they help.

Decision framework: default to fine-tuning; only justify training from scratch if you can show, with a controlled comparison on a held-out set, that the foundation model's inductive biases are actively hurting rather than helping on your specific data.

Worked example

For a medical-imaging classification task with 100,000 labeled scans, fine-tuning a foundation model pretrained on general images typically converges faster and generalizes better than training a comparable architecture from scratch on the 100,000 examples alone, because the model doesn't have to relearn low-level visual features (edges, textures) from zero. The exception would be a domain with input structure fundamentally unlike anything in the pretraining data (some specialized scientific sensor readings), where the pretrained low-level features may not transfer at all.

Trade-offs and pitfalls

The common mistake is assuming fine-tuning always wins regardless of domain gap; it's worth an actual side-by-side comparison rather than skipping straight to fine-tuning on faith. The opposite mistake is training from scratch by default out of a belief that it gives "more control," when in practice it usually just costs more compute for a worse result at this data scale.

Direct answer

Training-serving skew is when a model's live input features differ, even subtly, from the features it was trained on, so the model performs worse in production than its offline metrics predicted even though the model itself never changed. It is typically caused by two separate code paths, a batch job for training and an online service for serving, that compute "the same" feature slightly differently, or by data that was available at training time but is late, missing, or defined differently at serving time. Because the model artifact is unchanged, this failure is invisible to code review of the model itself and only shows up as a gap between offline evaluation and live outcomes.

Structured elaboration

Common causes:

1. Feature computation divergence: different null handling, rounding, or time windows between an offline Spark-style job and an online service implementing "the same" feature.
2. Data availability and timing skew: a feature depends on data that arrives late in production though it was fully present at training time, or the reverse, a feature that used information not truly available until after the label.
3. Preprocessing divergence: normalization, tokenization, or categorical encoding applied in training but re-implemented, imperfectly, in the serving path.
4. Environment nondeterminism: differing library versions or numeric precision between the training environment and the serving container.
5. Population shift: the live traffic distribution differs from the logged historical traffic the model trained on; not a code bug, but the same symptom of an offline-online gap.

Detection, two complementary strategies:

• Structural, pre-deploy: replay the exact serving code path against historical inputs and diff the resulting feature vectors against what the training pipeline produced for the same keys and timestamps. This catches skew before it ever reaches production.
• Live, statistical: continuously compare each feature's serving-time distribution against its training-time distribution, and separately track the gap between offline evaluation metrics and live business or proxy metrics. A widening offline-online metric gap with an unchanged model version is the single strongest live signal of skew.

A standard distributional-shift metric for the second approach is the population stability index (PSI), computed by bucketing a feature's values and comparing actual (serving) proportions against expected (training) proportions per bucket:

where $a_i$ is the actual (serving) proportion of observations in bucket $i$, $e_i$ is the expected (training) proportion in that same bucket, and $k$ is the number of buckets.

Worked example

Suppose a single feature is bucketed into "low" and "high," with a training (expected) distribution of $e = (0.7, 0.3)$ and an observed serving (actual) distribution of $a = (0.5, 0.5)$.

By the commonly used industry convention, a PSI under 0.1 is treated as negligible shift, 0.1 to 0.25 as a moderate shift worth investigating, and above 0.25 as a significant shift. A PSI of about 0.169 sits in the moderate range: enough to open an investigation into whether this is skew or a genuine, harmless population change, not yet enough on its own to declare the model broken.

Trade-offs and pitfalls

A replay test catches skew introduced by a code change before deployment, but it cannot catch a live shift caused by an upstream data source changing after deployment; only the distributional monitor catches that half. Distributional monitors need a stable reference distribution and get noisy on low-volume or seasonal features, so a naive fixed PSI threshold will false-positive on a real, expected demand shift (a holiday spike, for instance) that is not a bug at all; thresholds need to account for known seasonality rather than firing on every deviation. Numeric nondeterminism skew (library-version or floating-point differences) is the hardest to catch with either approach, since the discrepancy can be too small to move a distributional metric while still compounding through a deep model; that class needs unit-level golden-value tests comparing exact outputs, not distributional monitoring.

Direct answer
Difficult feedback to a mentee works best delivered privately, tied to a specific, observed behavior and its concrete impact, not to the person's character, and followed up on to confirm the message landed and something changed. The delivery mechanics matter less than getting three things right: timing (soon after the behavior, not saved up), specificity (a real example, not a vague pattern), and follow-through (checking back in, not treating the conversation itself as the fix).

Structured elaboration

Before the conversation

• Get the facts straight: what exactly happened, what was the impact, and is this a one-off or a pattern. Vague feedback ("you need to be more careful") is unusable; a mentee can't act on a mood, only on a specific instance.
• Decide the stakes. Not all critical feedback carries the same weight:
  • A routine performance gap (missed a deadline, sloppy code style) can wait for the next scheduled 1:1.
  • An ethical or safety concern (something in the person's work poses real risk if it ships) changes the calculus: it needs to happen immediately, privately, and is often paired with a concrete containment step (pause the change, get a second reviewer), not just a conversation.
• Decide the medium: a private 1:1, not written feedback and not in front of the team, unless the finding also needs to be logged for safety or compliance reasons.

Delivering it

• Lead with the specific behavior and its impact, not a label: "this change would have introduced X" is usable; "this was careless" is not.
• Ask before you assert. The mentee may know something you don't (a constraint you weren't aware of); asking "walk me through the reasoning" often surfaces that before you've over-committed to a judgment.
• Separate the person from the work. The message is "this output has a problem," not "you are the problem."

When the power dynamic is reversed

Feedback isn't always flowing to someone junior. Giving critical feedback to a mentee who is more senior, more tenured, or simply more confident than you requires the same content but a different frame: lead with genuine respect for their experience, be more explicit that you're not questioning their general competence, and expect (and plan for) more pushback. Defensiveness here is a normal reaction to a status threat, not necessarily a sign the feedback was wrong; the skill is staying anchored to the specific evidence instead of either escalating or backing down.

After the conversation

• Confirm shared understanding before ending: ask them to restate what they heard.
• Agree on a concrete next step and a checkpoint to revisit it, not just "let's see how it goes."
• Follow up. Feedback that isn't revisited quietly signals it wasn't actually important.

Worked example

Situation

During a code review, I found a race condition in an interrupt service routine (an ISR, the block of code that runs automatically when a hardware event interrupts normal execution) a mentee had written: a shared buffer was being written from the ISR without disabling interrupts around the critical section (the stretch of code that touches shared data and must not be interrupted mid-update), so a preemption (the interrupt firing and pausing the main code at an unpredictable moment) at the wrong moment could corrupt data intermittently and unpredictably.

Why this wasn't routine feedback

This wasn't a style nitpick. Left unaddressed it was a latent, hard-to-reproduce bug that could surface in the field. That pushed it from "note it for next time" to "we talk today, and the change doesn't merge until it's fixed."

The conversation

I asked the mentee to walk me through what happens if the interrupt fires mid-write, rather than telling them the bug outright. They found the failure mode themselves partway through the explanation, so the fix landed as their own understanding rather than my correction. We then talked through the general pattern (anything touching state shared between an ISR and main-line code needs an explicit critical section) so it would generalize past this one bug.

Follow-through

I asked them to check two other places in the codebase where similar shared state existed, as a way to prove the concept had stuck rather than just fixing the one instance. Both had the same latent issue.

Result

The immediate bug was fixed before merge, and the mentee started flagging similar patterns unprompted in their own future changes, which was the real signal the feedback had generalized rather than just been complied with once.

Trade-offs & pitfalls

• The feedback sandwich dilutes the message. Padding critical feedback between two compliments is a common junior instinct; it often causes the actual point to get lost. Genuine positive feedback is worth giving, but on its own merits, not as camouflage for the critical part.
• Waiting to "collect examples" delays too long. A senior mentor gives feedback close to the event; batching several issues into one big conversation later makes it feel like an ambush and makes each point harder to act on.
• Not distinguishing skill gap from something more serious. A performance gap and an ethical or safety issue call for different urgency and different documentation; treating a safety issue as routine coaching is itself a failure mode worth naming.
• Confusing "they got defensive" with "I was wrong." Especially with a more senior or tenured mentee, defensiveness is a predictable reaction to a status threat. A junior mentor backs off; a senior one stays anchored to the specific evidence while still leaving room for the other person to be right about something they missed.

The idea in plain words

A fixed pool of users is a budget, and every test spends from it. A test's statistical power (the chance it detects a real effect of a given size) grows with the number of users in it. So the real choice is between many weakly powered tests and a few well-powered ones, and the best split depends on how much you trust your ranking of the ideas. Judge a strategy by its regret: the true gain an all-knowing planner (the oracle, which ships exactly the ideas whose true effect is positive and spends nothing on testing) would capture, called the oracle gain, minus the gain your strategy captured, together with how many mistaken ships it makes. Users and calendar time are one budget, because weekly traffic converts one into the other.

The short answer. If you have evidence that your pre-test ranking has skill, fund the top half of the queue at full power, in whole weeks, and refresh the scores after each wave. If the ranking has no track record, screen cheaply and fund about half the queue on fresh data. Splitting the pool across all ten tests at once matches or beats ranking only when the pool is large enough to fund all ten (a tie), when about half the ideas are good (a tie) or when the ranking is weak (a win). In the simulation below, ranked sequential testing captures 80.9% of the oracle gain against 73.4% for an even split, under the stated assumptions.

What an interview round asks. A round on this question expects the power calculation, the allocation strategies, a paired simulation and how the winner changes with the pool, the helpful share and the quality of the score. Regret theory, adaptive allocation and a knapsack-style scheduling rule are deeper material an interviewer may probe afterwards, and reusing the screening data to approve its own winners is a classic trap worth recognising.

Method, in the order I would do it

1. Price one test in users. A standard error is the typical size of the random wobble in a measured lift from one sample to the next, and a z-score is the measured lift divided by its standard error; a test ships an idea when its z-score clears a threshold. For a conversion metric, the users per arm (control and treatment each get this many) needed for a lift of $\delta$ (an absolute change in conversion rate) is

$n_{\text{arm}} \approx \frac{2\,(z_{\alpha}+z_{\beta})^2\,p(1-p)}{\delta^2}$

Here $p$ is the baseline conversion rate, $z_{\alpha}=1.96$ is the ship threshold (a z-score is the estimated lift divided by its standard error; 1.96 one-sided gives a 2.5% false-ship rate per idea that does nothing, where a false ship is shipping an idea whose true lift is zero or negative), and $z_{\beta}=0.84$ buys 80% power. With the assumptions below, $p=0.05$ and $\delta=0.005$:

$n_{\text{arm}} \approx \frac{2 \times (1.96+0.84)^2 \times 0.05 \times 0.95}{0.005^2} = \frac{2 \times 7.84 \times 0.0475}{0.000025} = \frac{0.7448}{0.000025} \approx 29{,}800$

So a test costs about 60,000 users (two arms of 30,000). A pool of 300,000 users funds five such tests, not ten.

2. Turn users into time. At an assumed 25,000 eligible users per week (plausible range 10,000 to 200,000), 300,000 users is 12 weeks. One full-traffic test takes $60{,}000 / 25{,}000 = 2.4$ weeks. If tests must run whole weeks to cover the weekday cycle, each full-traffic test rounds up to 3 weeks and only four fit back to back in 12 weeks ($4 \times 3 = 12$, each getting 75,000 users). The simulation below counts users, not calendar time, so it funds five tests of 60,000; in calendar terms the fifth fits only by running the five side by side on traffic slices that each last whole weeks (20% of traffic for 12 weeks is $5{,}000 \times 12 = 60{,}000$ users per test), at the price of reading nothing until the end. Ten tests in parallel, each on 10% of traffic, use all 12 weeks at 2,500 users per test per week ($30{,}000 / 2{,}500 = 12$).

3. Name the strategies.

That is four strategies plus one deliberately wrong variant.

A toy world to trace by hand. Three ideas, each with a true lift and a pre-test score (the score is a noisy guess at the lift). P has lift +0.006 and score 0.002, Q has lift 0.000 and score 0.006, R has lift +0.009 and score 0.008. Suppose the pool pays for only two full tests (120,000 users, 30,000 per arm), and, to keep the trace short, every funded test reads out correctly: it ships exactly the ideas with a positive lift. The oracle ships P and R, so the oracle gain is $0.006 + 0.009 = 0.015$.

• sequential_random runs the queue in order, P then Q. It ships P and gains 0.006, which is 40% of the oracle gain; R is never tested, so it is a missed winner.
• sequential_ranked sorts by score, R then Q then P, and funds the top two. It ships R and gains 0.009, which is 60% of the oracle gain, so its regret is $0.015 - 0.009 = 0.006$. P is missed because the score ranked it last, and Q used up a test.
• The paired difference on this one world is $0.009 - 0.006 = +0.003$ in favour of ranking.
• equal_split would run all three at two thirds of the sample each ($120{,}000 / 3 = 40{,}000$ users, 20,000 per arm), so no winner is skipped but each real winner is caught less often.

The program below is this same trace with random lifts, noisy readouts and ten ideas, repeated in 4,000 worlds.

4. Simulate worlds, not one world. Draw many imaginary worlds of ten ideas, run every strategy on the same worlds, and compare. The inputs are assumptions, each with a range worth varying:

• Baseline conversion 5% (range 1% to 20%).
• Each idea is helpful with probability 25% (lift uniform from 0.003 to 0.010 absolute), harmful with probability 10% (lift -0.003), otherwise has no effect. Range for the helpful share: 10% to 50%.
• The pre-test score equals the true lift plus noise with standard deviation 0.004 (range 0.002 to 0.010). This stands in for how good your ranking is (ICE and RICE are scoring formulas that turn ratings into one number per idea: ICE multiplies impact, confidence and ease, and RICE multiplies reach, impact and confidence and divides by effort).
• Pool of 300,000 users (range 150,000 to 600,000).
• Gains of shipped ideas add up (no interactions between ideas).

5. Measure four things. Gain captured as a percentage of the oracle gain defined above (so regret is 100 minus that), false ships per world (ideas shipped whose true lift is zero or negative), winners missed per world, and the paired difference between two strategies on identical worlds with its standard error. Paired means: in each simulated world, subtract one strategy's gain from the other's, then average those differences over the worlds. Every world has its own luck (some contain more helpful ideas, which lifts every strategy), and subtracting inside a world cancels that luck, so the average difference is much more precise than comparing two averages from different worlds. Its standard error is the standard deviation of the per-world differences divided by the square root of the number of worlds. Sensitivity is part of the result: vary the pool, the helpful share and the score noise.

The program

import numpy as np
from scipy import stats

P0 = 0.05            # assumed baseline conversion rate
K = 10               # queued experiments
Z_SHIP = 1.96        # ship if z > 1.96 (one-sided, about 2.5% false-ship rate per null test)
SCORE_NOISE = 0.004  # assumed noise in the pre-test score: score = true lift + N(0, 0.004)

def make_worlds(rng, R, p_help=0.25, p_harm=0.10, noise=SCORE_NOISE):
    """R simulated worlds, each with K ideas. Absolute lifts in conversion-rate points."""
    u = rng.random((R, K))
    d = np.zeros((R, K))
    help_ = u < p_help
    harm = (u >= p_help) & (u < p_help + p_harm)
    d[help_] = rng.uniform(0.003, 0.010, help_.sum())
    d[harm] = -0.003
    score = d + rng.normal(0, noise, d.shape)
    return d, score

def arm_counts(rng, d, n):
    """Conversions in control and treatment with n users per arm (n may be an array)."""
    return rng.binomial(n, np.full(d.shape, P0)), rng.binomial(n, np.clip(P0 + d, 0, 1))

def zstat(xc, xt, n):
    pc, pt = xc / n, xt / n
    pp = (xc + xt) / (2 * n)
    se = np.sqrt(pp * (1 - pp) * 2 / n)
    return (pt - pc) / se

def run(strategy, d, score, pool, rng, n1=4000, keep=3):
    R = d.shape[0]
    ship = np.zeros(d.shape, bool)
    if strategy == "equal_split":                 # all K at once, pool/K users each
        n = pool // K // 2
        xc, xt = arm_counts(rng, d, n)
        ship = zstat(xc, xt, n) > Z_SHIP
    elif strategy in ("sequential_random", "sequential_ranked"):
        n = 30000                                  # about 80% power for a 0.005 lift
        m = pool // (2 * n)                        # tests we can fully fund
        order = np.tile(np.arange(K), (R, 1)) if strategy == "sequential_random" else np.argsort(-score, axis=1)
        funded = order[:, :m]
        dd = np.take_along_axis(d, funded, 1)
        xc, xt = arm_counts(rng, dd, n)
        np.put_along_axis(ship, funded, zstat(xc, xt, n) > Z_SHIP, 1)
    elif strategy in ("two_stage", "two_stage_reuse"):
        xc1, xt1 = arm_counts(rng, d, n1)
        z1 = zstat(xc1, xt1, n1)
        top = np.argsort(-z1, axis=1)[:, :keep]
        n2 = (pool - K * 2 * n1) // (2 * keep)
        dd = np.take_along_axis(d, top, 1)
        xc2, xt2 = arm_counts(rng, dd, n2)
        if strategy == "two_stage":                # decide on fresh stage-2 data only
            z = zstat(xc2, xt2, n2)
        else:                                      # wrong version: reuse the stage-1 data that picked it
            xc2 = xc2 + np.take_along_axis(xc1, top, 1)
            xt2 = xt2 + np.take_along_axis(xt1, top, 1)
            z = zstat(xc2, xt2, n2 + n1)
        np.put_along_axis(ship, top, z > Z_SHIP, 1)
    return ship

def summarize(ship, d):
    gain = (ship * d).sum(1)
    oracle = np.where(d > 0, d, 0).sum(1)
    return dict(gain=gain, oracle=oracle,
                false_ships=(ship & (d <= 0)).sum(1),
                missed=((~ship) & (d > 0)).sum(1))

STRATS = ["equal_split", "sequential_random", "sequential_ranked", "two_stage", "two_stage_reuse"]

def experiment(R, pool, p_help, seed, noise=SCORE_NOISE, **kw):
    rng = np.random.default_rng(seed)
    d, score = make_worlds(rng, R, p_help, noise=noise)
    out = {}
    for s in STRATS:
        r2 = np.random.default_rng(seed + 1)       # same data noise stream for every strategy
        out[s] = summarize(run(s, d, score, pool, r2, **kw), d)
    return out

def pct_oracle(o):
    return 100 * o["gain"].sum() / o["oracle"].sum()

# ---- Main comparison: 300,000-user pool, 25% helpful ideas, 4,000 simulated worlds
R, POOL = 4000, 300_000
res = experiment(R, POOL, 0.25, seed=7)
print("pool", POOL, "worlds", R, "| gain = sum of true lifts of shipped ideas, in % of oracle gain")
print(f"{'strategy':18s}{'gain%oracle':>12s}{'false ships':>13s}{'missed':>8s}{'P(>=1 false)':>14s}")
for s in STRATS:
    o = res[s]
    print(f"{s:18s}{pct_oracle(o):12.1f}{o['false_ships'].mean():13.3f}{o['missed'].mean():8.2f}{(o['false_ships']>0).mean():14.3f}")
base = res["equal_split"]["gain"]
for s in STRATS[1:]:
    diff = res[s]["gain"] - base
    print(f"paired gain vs equal_split, {s:18s}: {diff.mean()*1e3:+.3f} (units of 0.001 absolute lift), SE {diff.std(ddof=1)/np.sqrt(R)*1e3:.3f}")

# ---- Sensitivity: pool size and share of helpful ideas (2,000 worlds each)
print("\nsensitivity, gain % of oracle")
print(f"{'pool':>9s} {'p_help':>6s}" + "".join(f"{s[:17]:>19s}" for s in STRATS[:4]))
for pool, ph in [(150_000, .25), (300_000, .25), (600_000, .25), (300_000, .10), (300_000, .50)]:
    r = experiment(2000, pool, ph, seed=11)
    print(f"{pool:9d} {ph:6.2f}" + "".join(f"{pct_oracle(r[s]):19.1f}" for s in STRATS[:4]))

print("\nsensitivity to the quality of the pre-test score (300,000 pool, 25% helpful), gain % of oracle")
for nz in (0.002, 0.004, 0.010):
    r = experiment(2000, POOL, 0.25, seed=13, noise=nz)
    print(f"score noise SD {nz:.3f}: equal_split {pct_oracle(r['equal_split']):5.1f}  sequential_ranked {pct_oracle(r['sequential_ranked']):5.1f}")
print("\ntwo_stage settings (300,000 pool, 25% helpful), gain % of oracle and false ships per world")
for n1, keep in [(4000, 3), (4000, 5), (2000, 5), (6000, 4)]:
    r = experiment(2000, POOL, 0.25, seed=17, n1=n1, keep=keep)["two_stage"]
    print(f"stage-1 users per arm {n1}, keep {keep}: {pct_oracle(r):5.1f}  {r['false_ships'].mean():.3f}")

# ---- Checks
# 1) analytic power for one funded test (n=30,000 per arm, lift 0.005) vs simulation
n, dl = 30000, 0.005
pb = P0 + dl / 2
se = np.sqrt(2 * pb * (1 - pb) / n)
power_formula = 1 - stats.norm.cdf(Z_SHIP - dl / se)
rng = np.random.default_rng(3)
xc, xt = arm_counts(rng, np.full(200000, dl), n)
power_sim = (zstat(xc, xt, n) > Z_SHIP).mean()
print(f"\npower at 0.005 lift, 30,000 per arm: formula {power_formula:.3f}, simulated {power_sim:.3f}")
assert abs(power_formula - power_sim) < 0.01, "simulator disagrees with the power formula"
# 2) null false-ship rate of the fresh-data rule vs the reuse rule
d0 = np.zeros((20000, K)); sc0 = d0.copy()
for s in ("two_stage", "two_stage_reuse"):
    sh = run(s, d0, sc0, POOL, np.random.default_rng(6))
    print(f"all-null worlds, {s:16s}: false ships per world {sh.sum(1).mean():.3f}")
assert run("two_stage", d0, sc0, POOL, np.random.default_rng(6)).sum(1).mean() < 3 * 0.025 * 1.2, "fresh-data rule should stay near alpha"

Reading the program.

• make_worlds builds R simulated worlds of K = 10 ideas. For each idea it draws a number u between 0 and 1: below 0.25 the idea is helpful and gets a random lift from 0.003 to 0.010; from 0.25 to 0.35 it is harmful (lift -0.003); otherwise its lift is zero. The pre-test score is the lift plus normal noise.
• arm_counts simulates the data: binomial(n, p) draws how many of n users convert when each converts with probability p. Control uses the baseline 0.05, treatment uses 0.05 plus the idea's lift.
• zstat turns two conversion counts into a z-score: the difference in rates divided by a standard error estimated from the combined rate.
• run keeps a table ship of True/False, one cell per world and idea. In equal_split every idea gets pool // K // 2 = 15,000 users per arm and ships where z is above 1.96. In the two sequential strategies, order lists for each world the idea numbers in funding order: queue order for random, or argsort(-score) (the idea numbers sorted from highest score to lowest) for ranked. m = pool // (2 * n) is how many full tests the pool pays for (300,000 / 60,000 = 5), order[:, :m] keeps the first five, take_along_axis picks those five ideas' true lifts out of each row, and after the simulated readout put_along_axis writes the ship decisions back into those five columns, so unfunded ideas stay unshipped. In the two-stage strategies every idea first gets 4,000 users per arm, argsort(-z1)[:, :keep] picks the three highest z-scores, and stage 2 spends the rest of the pool on those three; the two versions differ only in which data make the final z-score.
• summarize computes gain (the sum of true lifts of shipped ideas), the oracle gain (the sum of the positive lifts), false ships (shipped with lift zero or below) and missed winners (positive lift, not shipped).
• experiment builds the worlds once and runs every strategy with a data-noise generator started from the same seed, so all strategies face the same worlds and their gains can be paired.

Output:

pool 300000 worlds 4000 | gain = sum of true lifts of shipped ideas, in % of oracle gain
strategy           gain%oracle  false ships  missed  P(>=1 false)
equal_split               73.4        0.165    0.84         0.155
sequential_random         44.3        0.086    1.44         0.083
sequential_ranked         80.9        0.063    0.64         0.061
two_stage                 63.0        0.035    1.06         0.035
two_stage_reuse           64.5        0.075    1.01         0.074
paired gain vs equal_split, sequential_random : -4.664 (units of 0.001 absolute lift), SE 0.110
paired gain vs equal_split, sequential_ranked : +1.204 (units of 0.001 absolute lift), SE 0.079
paired gain vs equal_split, two_stage         : -1.661 (units of 0.001 absolute lift), SE 0.095
paired gain vs equal_split, two_stage_reuse   : -1.425 (units of 0.001 absolute lift), SE 0.094

sensitivity, gain % of oracle
     pool p_help        equal_split  sequential_random  sequential_ranked          two_stage
   150000   0.25               50.3               17.6               53.5               44.1
   300000   0.25               73.6               44.0               81.1               61.9
   600000   0.25               89.9               89.9               90.1               65.3
   300000   0.10               75.5               44.0               84.5               68.7
   300000   0.50               73.7               45.5               73.5               50.6

sensitivity to the quality of the pre-test score (300,000 pool, 25% helpful), gain % of oracle
score noise SD 0.002: equal_split  73.4  sequential_ranked  88.4
score noise SD 0.004: equal_split  73.4  sequential_ranked  81.4
score noise SD 0.010: equal_split  73.4  sequential_ranked  63.8

two_stage settings (300,000 pool, 25% helpful), gain % of oracle and false ships per world
stage-1 users per arm 4000, keep 3:  61.7  0.032
stage-1 users per arm 4000, keep 5:  72.8  0.073
stage-1 users per arm 2000, keep 5:  68.7  0.073
stage-1 users per arm 6000, keep 4:  70.9  0.049

power at 0.005 lift, 30,000 per arm: formula 0.784, simulated 0.784
all-null worlds, two_stage       : false ships per world 0.072
all-null worlds, two_stage_reuse : false ships per world 0.136

The two checks are the safety net. The first compares the simulator with the closed-form power of one funded test (0.784 both ways). The second runs worlds where every idea does nothing: the fresh-data rule ships 0.072 ideas per world, close to the 0.075 expected from three screened tests at 2.5% each ($3 \times 0.025$), while the reuse variant ships 0.136, almost double, and would fail the assertion (0.136 is above the 0.09 limit). The assertion is a small illustration that covers only this all-null setup; a real process would also check the data pipeline, interaction between concurrent tests and drift in the baseline rate.

What the results say

All of this holds for the assumptions above, not for every product.

• Ranking pays when the score has skill. sequential_ranked captures 80.9% of the oracle gain against 73.4% for equal_split. The paired difference is +1.204 (in units of 0.001 absolute lift) with standard error 0.079, many standard errors from zero.
• Order without a ranking is costly. sequential_random captures 44.3% and misses 1.44 winners per world, because five ideas are never tested, against 0.64 missed for the ranked order.
• Splitting thin raises false ships. equal_split makes 0.165 false ships per world, and in 15.5% of worlds at least one, because each of ten noisy tests has its own chance to cross the threshold.
• Two-stage with fresh data is the most conservative. It makes 0.035 false ships per world but captures 63.0%, below equal_split, because screening on 4,000 per arm is noisy and only three ideas can ship. Reusing the stage-1 data lifts the gain only slightly, to 64.5%, and doubles the false ships to 0.075, because ideas that were lucky in stage 1 cross the threshold more easily when their lucky data are counted again.

Sensitivity changes the ranking of the strategies:

• Pool size. At 150,000 users, ranked 53.5, equal 50.3, two_stage 44.1, random 17.6. At 600,000, equal and random tie at 89.9 because all ten ideas are fully funded; ranked is 90.1, which is within sampling noise of the tie: all three strategies then run the same ten tests at 30,000 per arm, so any gap is only the random draw.
• Share of helpful ideas. At 10% helpful, ranked 84.5 against equal 75.5. At 50% helpful, they tie (73.5 against 73.7), because with half the ideas good, funding only five leaves winners untested.
• Quality of the score. With noise SD 0.002, ranked reaches 88.4; at 0.004, 81.4; at 0.010 it falls to 63.8, below equal_split at 73.4. A noisy enough ranking is worse than splitting evenly, because it stops five ideas from running and picks them poorly. Even at 0.010 it stays well above queue order (sequential_random, 44.3%), and a ranking with no skill at all is simply queue order.
• Two-stage settings. Keeping five instead of three (4,000 per arm in stage 1) lifts it to 72.8% with 0.073 false ships per world, about the gain of equal_split with fewer than half its false ships (separate runs on different simulated worlds).

Regret, information and adaptive allocation

Two regrets are in play. Inside one test, a bandit (a method that moves users toward the variant that looks better while the test runs) reduces the user-experience cost of showing a loser. Across the queue, the regret above is the true gain left uncaptured. They differ in what they protect: a bandit minimises exposure to the loser and gives a biased, less precise effect estimate, while a fixed-horizon test (sample size set in advance, read once) gives the clean estimate and interval that a ship decision needs. Two-stage allocation is adaptive across tests: it stops spending on ideas that show nothing and moves their users to the rest. The price is the selection effect the reuse variant shows, which is why the decision uses fresh data.

The information-gain view says a user is worth most where the ship decision is still uncertain. A result far from the threshold at the planned sample size needs few more users; one near it needs many. For example, after a full-size test a z-score of 3.5 is clear (ship), while one at 1.9 is close to a coin flip that extra users can settle, so extra users are worth most there. A stage-1 read is noisier: a real lift of 0.005 has an expected stage-1 z-score of $0.005/\sqrt{2 \times 0.0525 \times 0.9475/4000} \approx 1.0$ with a spread of about 1, so a low stage-1 z-score does not rule a real winner out, which is why the two_stage screen above loses winners.

A scheduling heuristic for the queue

Rank ideas by expected gain per user spent. This is the logic of a knapsack problem, packing a bag of fixed capacity with items that each have a value and a weight: take the items with the highest value per unit of weight first. Here users are the capacity and weight, and expected lift is the value. For idea $i$, let $P_i$ be the assumed chance it is real, $\delta_i$ the lift it would have if real, and $U_i$ the users needed to detect $\delta_i$, which is $2\,n_{\text{arm}}$ from the formula above. Score $= P_i\,\delta_i / (U_i / 10{,}000)$, expected lift per 10,000 users. Using $15.68 = 2 \times 7.84$ (the $2\,(z_{\alpha}+z_{\beta})^2$ of step 1) and $15.68 \times 0.0475 = 0.7448$:

Order: B, A, C. With a 100,000-user pool, B and A cost $14{,}896 + 41{,}378 = 56{,}274$ users, and C's 93,100 does not fit in the remaining 43,726. The heuristic ignores that power is about 80% rather than 100% and that the real lift may be below the design lift, so it ranks ideas and does not forecast gain.

What I would do

Estimate the score's noise first, by checking last year's pre-test scores against what the tests found. If it shows skill, fund the top half of the queue at full power, with each test lasting whole weeks, and refresh the scores after each wave with what was learned; that is the sequential_ranked policy, which wins in the base case. At full traffic only four tests fit back to back in whole weeks here, so the fifth runs in a traffic slice or waits for the next wave. If there is no track record, run the screening version of two-stage with keep set to about half the queue, which gave a gain close to equal_split with fewer false ships. Flip toward equal_split when about half the ideas are plausibly good or the score is weak, and toward testing everything at full power once the pool covers all ten (600,000 users here). To compare strategies, report gain as a percentage of the oracle, false ships, winners missed and the paired difference with its standard error over the same worlds, and vary the pool, helpful share and score noise.

Common mistakes

• Judging strategies on one simulated world, or on unpaired worlds, so luck decides the winner.
• Counting winners found and ignoring false ships; equal_split looks decent until its 0.165 false ships per world are counted.
• Using stage-1 data to both pick the winners and approve them.
• Treating the ICE or RICE score as accurate without ever checking its track record.
• Running ten tests on overlapping users and assuming they do not interact, or splitting the pool when the tests touch different pages and could run on independent randomisations.

Running the code

Save the program above as a Python file and run it with python. The seeds are fixed, so the output is the same on every run.

The idea in plain words

In two weeks with little labeled data, you are not building a model to ship. You are answering one question for stakeholders: "On real examples, does this approach do well enough, often enough, that the full build is worth funding?" So spend the first days on three things that cost almost nothing: a written definition of "good enough", a small hand-labeled evaluation set (examples with the right answer, kept aside and never used to tune anything, so the score is an honest test on examples the approach was not adjusted to), and a baseline (the simplest thing that could do the job: a rule, a keyword match, or an existing pre-trained model, meaning one already trained on a large body of data by someone else, used as it comes). The demo then shows the baseline and the prototype on the same real examples, including the failures, with a number and its uncertainty.

Step 1: scope (days 1 to 2)

Write down, with the stakeholder who will decide:

• The decision: "Continue to a funded build if the prototype gets at least 80% of cases right on real inputs, the lower end of its 95% interval (worked out in Step 4) is at least 75%, and the mistakes it makes are cheap to catch." The 80% bar and the 75% floor are assumptions; the right bar comes from the cost of a mistake in the product. The floor sits about one margin of error under the bar, and that margin is about 5 points at 200 examples; Step 4 shows what the two conditions do together.
• The narrowest slice that still tests the idea: one input type, one language, one customer segment. A narrow slice that works beats a broad one that half works.
• The cost of each error type: a wrong suggestion a user can ignore costs little; a wrong automatic action costs a lot. This decides whether the demo is "suggest, human confirms" or "act automatically".
• What is not in the two weeks: scale, latency, security review, retraining, monitoring. Say so up front.

Step 2: build the evaluation set before any model (days 1 to 3)

Little labeled data means label a small amount well, and use it for testing, not training.

• Pull 200 real examples that reflect real inputs (sample them, do not pick the easy ones).
• Two people label a shared subset of 40 independently. Inter-annotator agreement is how often they give the same label. If two informed humans agree on only 70% of cases, the task itself is ambiguous. A model is scored against these human labels, so if people reproduce each other's labels only 70% of the time, an 80% score against one person's labels mostly measures which labeler's habits the model copied, and the 80% bar cannot be interpreted; fix the label definition first. That is a finding worth showing stakeholders.
• Labeling effort is arithmetic you can check: 200 examples x 30 seconds each = 6,000 seconds = 100 minutes for one person (assuming 30 seconds per example; range 15 to 90 depending on difficulty).

Step 3: check viability with the cheapest approach first (days 3 to 8)

Try approaches in increasing cost and stop at the first that clears the bar:

1. Rules or keyword matching: if this reaches the bar, you do not need ML at all, which is a valid and valuable answer. Tune any rule or prompt on a small separate development slice (for example 30 more labeled examples), not on the evaluation set, or the evaluation score will come out optimistic.
2. A pre-trained model used as it comes (for text, an existing language model prompted with a few examples, meaning you put several worked input-and-answer pairs into the instructions so it imitates them; for images, an existing classifier), scored on the evaluation set.
3. Light adaptation on your few labels (for example, a simple classifier fitted on top of a pre-trained model's representations, the lists of numbers the model produces for each input to summarise it), trained on a separate training slice, never on the evaluation set.

Two diagnostics tell you whether more effort is worth it:

• A learning curve: score the approach after training on 25%, 50% and 100% of your small training slice. If the score is still rising at 100%, more labels would help; if it has flattened below the bar, the approach is the limit, not the data. Example (assumed scores, each the average over five random draws of the training slice, all scored on the same 200 evaluation examples): 70%, 78% and 83% at 25%, 50% and 100% of the slice is still rising, with 5 points gained from the last doubling, so more labels would probably help. 76%, 79% and 79.5% has flattened below the 80% bar: doubling the labels bought half a point, so the approach is the limit. Small gaps need care: on 200 evaluation examples one example is half a point, and a single score carries a margin of about 5 points (Step 4), so a gap of a point or two between two points on the curve can easily be luck in which examples were drawn for training. Average several random draws at each size, and call the curve flat only when the last gain is smaller than the spread between draws.
• Error analysis: read 30 mistakes and sort them into causes (ambiguous label, missing information in the input, genuine model error, input outside the slice). If most are ambiguous or missing-information, the problem is the data or the product design, not the model. Example (assumed): of 30 mistakes, 12 are ambiguous labels, 9 lack the information needed in the input, 6 are genuine model errors and 3 are outside the slice. That is 21 of 30 (70%) caused by the labels or the input rather than the model (3 more are outside the slice, a scoping question), so the next step is the labels and the product design.

Step 4: put a number and an uncertainty on it

Suppose (illustrative) the prototype gets 170 of 200 evaluation examples right: accuracy 170 / 200 = 85%.

With only 200 examples the figure is imprecise. A rough 95% interval (the normal approximation treats the possible accuracy values as a bell curve around the measured one; adequate here because the counts are well away from 0 and 200):

1. Standard error = the square root of (0.85 x 0.15 / 200) = the square root of 0.0006375 = 0.0252.
2. Margin = 1.96 x 0.0252 = 0.0495.
3. Interval = 85% plus or minus 4.9 points = 80.1% to 89.9%.

So "85%" honestly means "somewhere around 80% to 90% on this kind of input". Apply the rule written on day 1: the point estimate, 85%, is above the 80% bar, and the lower end, 80.1%, is above the 75% floor, so the prototype passes. The two conditions do different jobs: the bar is the accuracy the product needs, and the floor says the data must also rule out accuracy below 75%. At 200 examples they nearly coincide: a measured 80% (160 of 200) has a lower end of 74.5% and fails the floor, while 161 of 200 (80.5%) has a lower end of 75.0% and clears it. With fewer examples the floor binds harder: at 50 examples the margin is about 10 points, and 43 of 50 (86%) is the least that clears it. A prototype whose true accuracy is exactly 80% would measure below 80% about half the time and be sent back; that is the cost of a fixed bar, so set the bar where that is acceptable. A stricter rule, such as a lower end of at least 80%, would be met by 170 of 200 only barely (169 of 200 would miss it) and would call for more labels. Either way the rule was set before the number was seen.

How many examples is enough? The margin shrinks with the square root of the count. For an accuracy near 85%, the 95% margin is about 9.9 points with 50 examples, 7.0 with 100, 4.9 with 200 and 2.5 with 800 (1.96 x the square root of 0.85 x 0.15 / n). Halving the margin takes four times the examples, which is why the sprint settles for 200 and the staged ask below includes a larger label set.

Step 5: choose what to demo

Stakeholders are persuaded by evidence they can inspect, not by a polished interface. Show:

1. The decision and the bar (one slide).
2. A live walk-through on 5 to 8 examples drawn at random from the evaluation set, including some the prototype gets wrong, with what the user would see and what a human would do next. Cherry-picked successes destroy trust the first time someone tries their own input.
3. Baseline against prototype on the same 200 examples, with the interval.
4. The error breakdown: which mistakes are cheap, which are costly, which the product design could catch.
5. The ask: a staged next step (for example, 4 more weeks on a larger label set and a shadow run, in which the prototype processes real inputs and its outputs are recorded and compared with what people did but nobody acts on them, beside the current process) with the next decision rule, not an open-ended build.

The judgment

Commit to this order: bar and slice, evaluation set, baseline, pre-trained model, light adaptation only if the curve is still rising. If the baseline already clears the bar, tell stakeholders the cheap solution wins. If agreement between labelers is low, say the task definition is the blocker. If the prototype clears the bar with an interval that leaves room, ask for a staged build. If it does not, a clear "no, here is why" at two weeks is a successful outcome of a validation sprint.

What would flip it: if a mistake is very costly (for example, an automatic action that moves money), require a much higher bar and a human-confirm design even at 95% accuracy. If you cannot get 200 labels at all, say that the prototype can only be a demonstration of the interface, not evidence of viability.

Common mistakes

• Training and testing on the same few examples, which makes any approach look excellent.
• Demoing only the successes.
• Reporting one accuracy figure with no interval or comparison to a baseline.
• Skipping the non-ML baseline.
• Letting the bar move after the result is known.

Direct answer

Periodic retraining is simplest and most predictable but can leave a model stale between schedules; trigger-based retraining reacts faster but needs a reliable trigger signal; continuous/online learning stays freshest but carries the highest infrastructure and stability risk: for a fraud model with seasonal patterns and costly false negatives, a hybrid of scheduled-plus-triggered is usually right.

Structured elaboration

• Periodic (scheduled): retrain weekly or monthly regardless of signals. Ideal use case: stable domains where the underlying relationship changes slowly and predictably. Infrastructure implication: simplest to build and reason about: a cron-style pipeline. Risk profile: can leave the model stale for the entire interval between retrains if something changes suddenly (a real risk for fraud, where fraudsters adapt quickly).
• Trigger-based: retrain when a drift or performance signal crosses a threshold. Ideal use case: domains with irregular, event-driven shifts (a fraud ring adapts, a new product launches). Infrastructure implication: needs reliable monitoring wired directly into the retraining orchestrator, plus guardrails against thrashing (retraining repeatedly on noisy signals). Risk profile: faster response to real problems, but a noisy trigger can cause wasted retrains or, worse, retrains on CORRUPTED data if the trigger itself was caused by a pipeline bug rather than real drift.
• Continuous/online learning: the model updates incrementally as new labeled data arrives, with no discrete "retrain" event at all. Ideal use case: extremely high-velocity domains where even daily retraining is too slow (ad ranking, real-time personalization). Infrastructure implication: the heaviest lift: needs streaming infrastructure, careful learning-rate and regularization control, and safety constraints to prevent a single bad batch of data from corrupting the live model. Risk profile: highest: catastrophic forgetting, instability from noisy individual updates, and the hardest to reproduce or audit after the fact ("which exact update caused this?").

Worked example

For the fraud system specifically: scheduled weekly retraining as the baseline (captures the slow seasonal drift reliably, on a predictable cadence stakeholders can plan around), PLUS a trigger-based override that fires an out-of-cycle retrain if a PSI-style drift signal or a sudden false-negative-rate spike crosses a threshold BEFORE the next scheduled run: this bounds the worst-case staleness (never longer than a week) while still reacting fast to an adapting fraud pattern that can't wait a week. Continuous online learning is deliberately avoided here specifically because of the cost asymmetry: a single corrupted or adversarially-poisoned online update in a continuously-learning fraud model could open a real exploit window, and the instability risk isn't worth the marginal freshness gain over a well-tuned scheduled+triggered hybrid.

For VALIDATION cadence specifically (a close corollary of the retraining-cadence question): scheduled validation (say, daily backtest against a held-out set) catches slow drift reliably and cheaply; continuous/near-real-time validation catches fast-moving problems sooner but costs more compute and is more prone to false alarms from single-window noise: the same hybrid logic applies: scheduled as the baseline, with a lighter-weight continuous check (a cheap proxy metric, evaluated more frequently) as an early-warning layer that doesn't replace the full scheduled validation.

Trade-offs & pitfalls

The common mistake is treating "more frequent" as strictly better across all three options: a trigger-based or continuous system retrained on a pipeline bug's noise (rather than real signal) can actively make the model WORSE, faster than a slower, more deliberate scheduled cadence would have. The freshness-vs-stability trade-off is real in both directions, not just "stale is bad."

Direct answer: Churn and segmentation feature design at scale needs to combine behavioral counts/rates/recency, cross-source interactions, and time-based signals computed from usage logs, support tickets, and demographics, with an explicit leakage check before trusting any offline lift, since churn labels are especially prone to accidentally encoding future information.

Structured elaboration: Eight concrete engineered features, with intuition and how each is computed at scale (roughly two million rows of mixed numeric, high-cardinality categorical, timestamp, and free-text data):

1. Days since last login/usage event (recency) - intuition: a lapsing user's engagement gap widens before they formally churn. Computed via a vectorized groupby-max on the timestamp column per customer, not a row-by-row loop, which is essential at two million rows.
2. Login/event frequency over trailing 7/30/90-day windows - intuition: a declining frequency trend is often more predictive than any single-point count. Computed via grouped, windowed aggregation (e.g. groupby(customer_id).rolling() or a windowed SQL aggregate).
3. Usage-intensity trend (slope of a simple linear fit or week-over-week ratio of event counts) - intuition: distinguishes "still active but declining" from "steady at a low level," which behave differently for churn.
4. Support ticket count and recency - intuition: a recent spike in support contact correlates with dissatisfaction, but must be computed carefully to avoid the leakage case below.
5. Support ticket sentiment or category distribution (from free-text ticket bodies, e.g. via a lightweight TF-IDF or keyword-tagging pass) - intuition: ticket TONE (frustrated vs. neutral) often separates at-risk from routine contacts even at similar ticket counts.
6. Plan/product tenure (time since signup or since last plan change) - intuition: churn risk is often U-shaped or front-loaded relative to tenure.
7. High-cardinality categorical encoding of plan or product identifier - using target encoding (with proper out-of-fold computation to avoid leakage) or frequency encoding rather than naive one-hot, since a high-cardinality plan/product ID field would otherwise blow up the feature space.
8. Cross-source interaction: declining usage AND a recent support contact (an explicit interaction feature, e.g. a flag or product of the two normalized signals) - intuition: the combination is a stronger churn signal than either usage decline or ticket volume alone, since users who complain but keep using the product, or who go quiet without complaining, behave differently than users doing both.

At roughly two million rows, computing these efficiently means vectorized, grouped aggregation (pandas groupby/window functions, or an equivalent SQL/Spark aggregation) rather than row-by-row processing, and encoding the high-cardinality categoricals (plan or product identifiers) with target or frequency encoding rather than naive one-hot encoding, which would otherwise create an unmanageably wide sparse matrix.

Whether to standardize or bin a behavioral signal depends on the downstream model and the signal's distribution: a heavily skewed count (most customers have very few support tickets, a small tail has many) often benefits from binning or a log-style transform for a linear model, while a tree-based model is largely indifferent to the raw scale and doesn't need standardization at all.

Detecting leakage specifically for churn: check whether any candidate feature (feature 4 and 5 above are the highest-risk ones) is computed using data from AFTER the point the churn decision would have actually been observable. A support ticket opened in response to a cancellation the customer had already decided on is a classic hidden leak, since it looks predictive but is really a symptom of the outcome, not a cause.

Worked example: A "days until account closure" style feature (however indirectly encoded, e.g. a support-ticket category that only exists post-cancellation) would show enormous offline lift and be completely unusable in production, since at prediction time you don't yet know whether the customer will churn; catching this requires explicitly auditing what information was genuinely available BEFORE the churn decision point for every candidate feature, not just checking whether the feature "looks reasonable."

Trade-offs and pitfalls: A common trap at this scale is trusting an offline metric improvement without an out-of-time evaluation; a feature that looks powerful on a random split can be capturing something that's specific to the historical period (a promotion that ran during part of the data) rather than a durable churn signal.