DoorDash Applied Scientist (Staff Level) - Comprehensive Interview Preparation Guide

Redesign the input path
I would separate expensive preprocessing from the training loop and make the input stream sharded, cached, and deterministic.

Main changes

• Compact many small object-store files into larger shards so reads are sequential instead of chatty.
• Precompute expensive transforms once, then store the transformed output or an intermediate cache.
• Use a manifest that records shard order, sample IDs, and versioned transform code.
• Prefetch data to local disk or memory so GPUs do not wait on network reads.
• Keep lightweight, deterministic transforms in the training worker so results are reproducible.

Worked example
If the dataset is 10,000 files of 2 MB each, that is 20 GB spread across many requests. Repacking into 200 shards of 100 MB keeps the same 20 GB, but cuts file open and listing overhead dramatically.

Why this helps
The GPUs stay busy because decoding and network latency happen ahead of time. Reproducibility comes from versioned manifests and fixed seeds. Debuggability comes from keeping raw sample IDs so a bad example can be replayed exactly.

This is usually the highest leverage change when preprocessing time dominates training time.

Direct answer

Four things are named: how you investigated and isolated the issue, how you communicated with stakeholders, how you remedied the pipeline, and what long-term controls you put in place. The differentiating move in "isolate" is finding the pattern, which subset of data is affected and what's different about it, rather than just confirming something looks wrong. And a stakeholder update that happens before you have the full fix, with an interim guardrail attached, reads as far more competent than one that waits until everything is resolved.

Structured elaboration

• Investigate and isolate. Show the specific comparison that revealed the pattern (a distribution check, a timestamp audit), not just "I noticed something odd."
• Communicate with stakeholders. Say when you told them relative to when you had the full picture, and what interim guardrail you gave them to act on immediately.
• Remedy the pipeline. Distinguish the fix to the immediate bug from the backfill of already-affected data.
• Long-term controls. The strongest version generalizes past this one feature, since the real gap an incident like this exposes is usually "nobody owns whether this feature is still telling the truth over time," not just "this one pipeline had a bug."

Worked example

I own a pricing-elasticity model (predicts how much demand shifts when we change an item's price) in production. Weeks after launch, I noticed our "competitor price" feature looked suspicious for a subset of items, about twelve percent of the catalog, and the model was recommending prices roughly four percent higher on average for that subset, a real cost in foregone conversion.

To isolate it, I checked the feature's timestamp distribution against the expected scrape cadence and found it was bimodal: most values updated daily as expected, but a cluster stuck at values unchanged for fifteen to twenty-one days. Tracing that back, a scraper fallback path was silently returning the last cached value, tagged with today's date, whenever a live scrape timed out, so nothing downstream could tell fresh data from stale by timestamp alone.

I told the pricing team lead as soon as I'd isolated the pattern, before I had a full fix, with a quantified estimate (twelve percent of items, about four percent average overpricing) and an immediate guardrail: exclude the affected items from automated price changes until it was resolved. I then fixed the scraper to fail loudly on timeout instead of silently returning stale data, added a data-quality gate that flags any feature whose source timestamp is older than its expected freshness window before it reaches the model, and backfilled the roughly three weeks of affected data by re-scraping.

The long-term control was making source freshness a required, monitored property for every external feature feeding the model, not just this one, plus a quarterly feature audit that spot-checks freshness and distribution assumptions across every input, since this incident showed nobody actually owned that question over time. The following quarter, that audit surfaced a similar issue in an unrelated feature, a macroeconomic index that had silently stopped updating after a vendor API change, before it ever affected a price.

Trade-offs and pitfalls

A weak answer treats isolation as "I confirmed the data was wrong" without ever explaining what pattern distinguished the affected records from the rest. A strong answer names the actual diagnostic comparison. Also worth being honest about: the impact estimate here (four percent average overpricing) is based on the affected items, which were not a random sample of the catalog, so a credible answer should acknowledge that uncertainty rather than presenting the number as more precise than it is.

Direct answer. Detecting whether a production sample has drifted from a baseline distribution requires a test matched to each feature's TYPE, a numeric distributional test like Kolmogorov-Smirnov for continuous features and a chi-square test for categorical ones, run per feature, with a correction for the fact that testing many features at once inflates the chance of a false alarm somewhere.

Structured elaboration. For numeric features, the Kolmogorov-Smirnov test compares the full empirical distributions of the baseline and the sample without assuming any particular shape, which makes it a good general-purpose choice when you don't know in advance what kind of shift (mean, spread, skew) might occur. For categorical features, a chi-square goodness-of-fit test compares the sample's category proportions against the baseline's expected proportions. Missingness should be tracked SEPARATELY from the distributional test on the non-missing values, since a feature can have a stable distribution among the values it DOES have while its missing-rate silently climbs, a distinct and equally important failure mode a bare Kolmogorov-Smirnov (KS) or chi-square test on non-null values alone would miss entirely. With many features tested at once, apply a multiple-testing correction (Bonferroni divides the significance threshold by the number of tests) so the overall false-alarm rate across the whole feature set stays controlled, rather than letting a handful of features flag purely by chance.

Worked example (executed in Python with numpy/scipy). Comparing a numeric feature (avg_order_value, baseline mean 50, sample mean 58, 5,000 baseline rows vs 1,200 sample rows) and a categorical feature (region, unchanged mix in both): the KS test on avg_order_value returns statistic 0.309, p-value effectively 0 (well under a Bonferroni-corrected threshold), correctly flagging real drift; the chi-square test on region returns p=0.072, correctly NOT flagging it even without correction, since the region mix genuinely didn't shift in this synthetic example. Missingness delta for both features came back at 0.0, confirming the drift here is purely distributional, not a completeness issue.

Trade-offs and pitfalls. KS and chi-square are both sensitive to sample size: with a very large sample, even a practically trivial shift becomes statistically significant, so pair the p-value with an effect-size measure (like the KS statistic's magnitude itself, or a simple difference-in-means relative to the baseline's spread) rather than treating statistical significance alone as evidence of a PRACTICALLY important drift. For a streaming or very-large-feature-count setting, exact KS computation can be too slow; approximate quantile sketches or chunked/streaming variants trade a small amount of precision for the ability to run continuously rather than only on demand.

I turn a strategic roadmap into a team plan by translating outcomes into sequenced, capacity-aware work.

My approach:

1. Break the roadmap into epics (large chunks of related work broken down into smaller, shippable stories) or milestones tied to measurable outcomes.
2. Identify dependencies across teams and place them on the critical path (the chain of dependent tasks whose combined duration sets the earliest possible finish date, so a slip anywhere in that chain delays the whole plan).
3. Estimate using actual team capacity, not idealized capacity, and reserve buffer for unplanned work.
4. Sequence work so early items unlock later ones and reduce risk quickly.
5. Recheck whether the plan fits the team’s sustainable pace.

I avoid overcommitting by making trade-offs visible. If the roadmap has more work than the team can support, I push for explicit choices: what is in, what is out, and what can move later. I also keep room for learning, because the plan should be realistic enough to execute, not so packed that one surprise breaks everything.

The strongest plans are not the most ambitious ones; they are the ones the team can actually deliver with confidence.

Worked example
Say the roadmap's quarterly goal is "launch self-serve onboarding." I'd break that into three epics: build the account-setup flow, build the guided first-project wizard, and build the in-app upgrade prompt. Against a team of five engineers with a realistic capacity of about 30 person-days a week after accounting for on-call and code review, the account-setup flow (estimated 25 person-days) and the wizard (estimated 20 person-days) sit on the critical path because the upgrade prompt depends on both finishing first, so those two get sequenced first and the upgrade prompt follows in the back half of the quarter, with a one-week buffer reserved before quarter-end for whatever surprise inevitably shows up.

Situation & high-level approach
I split team capacity into three lanes: 60% delivery (short-term product work), 25% exploratory research, 15% infrastructure/ops and skill development. I adjust the split quarterly based on product needs.

Decision criteria

• Impact: expected improvement in core metric (CTR, revenue, latency).
• Risk & uncertainty: technical feasibility and data readiness.
• Time-to-value: how long to a usable prototype.
• Strategic learning: unique IP or long-term differentiation.

Estimating experiment ROI
I estimate ROI with a simple expected-value model:

Expected ROI = Probability(success) * Net benefit per unit time - Cost per unit time

Intuition: multiply likelihood of success by benefit (e.g., revenue uplift or latency reduction), subtract resource costs. I quantify benefits in product KPIs or engineering savings.

Aligning priorities with stakeholders

• Monthly reviews with PMs/engineers to map experiments to KPI owners.
• Define success criteria and fallback plans before kickoff.
• Use a lightweight RACI and show expected ROI and timeline to inform trade-offs.

Measuring progress & outcomes

• Short checkpoints: weekly status + quantitative metrics (validation loss, offline A/B estimates).
• Gate reviews: prototype -> pilot A/B -> full rollout with predefined metrics and significance thresholds.
• Post-mortem: document learnings, negative results, reproducible code and next steps.

Example: For a recommender improvement, I ran a 6-week probe (25% capacity), estimated 10% uplift with 40% success probability, ran an A/B pilot, and only promoted models that passed 95% CI on CTR and retention.

Direct answer

Pick based on two properties of the graph and one property of the query: if every edge has the same weight, breadth-first search (BFS) alone gives shortest paths in linear time; if weights differ but are never negative, Dijkstra's algorithm is the standard single-source choice; if a negative weight is possible (but no negative cycle), Dijkstra can give a wrong answer and Bellman-Ford is required instead; and if you need distances between every pair rather than from one source, Floyd-Warshall (a dynamic-programming algorithm that considers every node in turn as a possible shortcut between every pair) is the natural fit for all-pairs, or equivalently running Dijkstra from every node. A* only changes single-source, non-negative-weight search: it adds a heuristic estimate of remaining distance to prioritize expansion toward a specific goal, and it only helps, versus plain Dijkstra, when that heuristic is admissible (it never overestimates the true remaining cost), since an inadmissible heuristic can cause A* to return a path that is not actually shortest.

Structured elaboration

Decision order:

1. Are all edge weights equal (or is the graph unweighted)? Use BFS: $O(V + E)$, no priority queue needed at all.
2. Do you need distances between every pair of nodes, not just from one source? Use Floyd-Warshall, $O(V^3)$, or repeat Dijkstra from every source if the graph is sparse and there are no negative weights.
3. Otherwise, single-source with weights: are negative edge weights possible? If yes, use Bellman-Ford, $O(V \cdot E)$, which also detects a negative cycle if one exists (a cycle whose total weight is negative, which makes "shortest path" undefined, since you could loop it forever to keep decreasing the cost). If no, use Dijkstra, $O((V+E) \log V)$ with a binary heap.
4. If you additionally have a single, known goal node (not "distances to everywhere"), and a decent estimate of remaining distance, layer A* on top of Dijkstra's non-negative-weight assumption to reduce how much of the graph gets explored.

Why Dijkstra breaks under negative weights: once Dijkstra pops a node off its priority queue, it treats that node's distance as final and never revisits it, on the assumption that nothing already queued could possibly offer a shorter path, since all remaining edges only add non-negative weight. A negative edge violates that assumption directly: a longer-looking path discovered later can still turn out shorter once a negative edge is added to it.

What A's heuristic must guarantee*: admissibility, never overestimating the true remaining cost to the goal. Given an admissible heuristic, A* is guaranteed to still find a shortest path, exactly like Dijkstra, but it explores fewer nodes when the heuristic is informative, because it prioritizes nodes that look closer to the goal rather than merely closer to the start. (A stronger property, consistency, additionally guarantees a node is never re-expanded after being finalized, which is what lets A* implementations skip the "revisit and relax an already-closed node" bookkeeping that a merely admissible-but-inconsistent heuristic would otherwise require.) A straight-line (Euclidean) distance heuristic on a road network or grid is a classic admissible choice, since no real route can be shorter than the straight line.

Worked example

Bellman-Ford correctness under a negative edge, where Dijkstra gets it wrong:

import math

def dijkstra(adj, src):
    import heapq
    dist = {src: 0}
    finalized = set()
    pq = [(0, src)]
    while pq:
        d, u = heapq.heappop(pq)
        if u in finalized:
            continue
        finalized.add(u)
        for v, w in adj.get(u, []):
            if v in finalized:
                continue
            nd = d + w
            if nd < dist.get(v, math.inf):
                dist[v] = nd
                heapq.heappush(pq, (nd, v))
    return dist

def bellman_ford(edges, nodes, src):
    dist = {n: math.inf for n in nodes}
    dist[src] = 0
    for _ in range(len(nodes) - 1):
        for u, v, w in edges:
            if dist[u] + w < dist[v]:
                dist[v] = dist[u] + w
    for u, v, w in edges:
        if dist[u] + w < dist[v]:
            raise ValueError("negative cycle detected")
    return dist

# True shortest A->B is via C: 1 -> 4 + (-10) = -6, but Dijkstra finalizes
# B at distance 1 (direct edge) before C is even processed.
adj = {"A": [("B", 1), ("C", 4)], "C": [("B", -10)]}
edges = [("A", "B", 1), ("A", "C", 4), ("C", "B", -10)]
nodes = ["A", "B", "C"]

print("dijkstra:", dijkstra(adj, "A"))
print("bellman_ford:", bellman_ford(edges, nodes, "A"))

Running this prints:

dijkstra: {'A': 0, 'B': 1, 'C': 4}
bellman_ford: {'A': 0, 'B': -6, 'C': 4}

Dijkstra reports B at distance 1 (wrong: it finalized B via the direct edge before discovering the cheaper route through C), while Bellman-Ford correctly finds -6 via A to C to B.

A exploring fewer nodes than Dijkstra given an admissible heuristic*, on an open 20x20 grid from (0,0) to (5,5), using Manhattan distance as the heuristic:

import heapq

def neighbors(pos, size):
    x, y = pos
    for dx, dy in ((1, 0), (-1, 0), (0, 1), (0, -1)):
        nx, ny = x + dx, y + dy
        if 0 <= nx < size and 0 <= ny < size:
            yield (nx, ny)

def manhattan(a, b):
    return abs(a[0] - b[0]) + abs(a[1] - b[1])

def search(start, goal, size, use_heuristic):
    h = (lambda n: manhattan(n, goal)) if use_heuristic else (lambda n: 0)
    g = {start: 0}
    pq = [(h(start), start)]
    visited = set()
    expansions = 0
    while pq:
        _, node = heapq.heappop(pq)
        if node in visited:
            continue
        visited.add(node)
        expansions += 1
        if node == goal:
            break
        for nb in neighbors(node, size):
            ng = g[node] + 1
            if ng < g.get(nb, float("inf")):
                g[nb] = ng
                heapq.heappush(pq, (ng + h(nb), nb))
    return g[goal], expansions

size = 20
start, goal = (0, 0), (5, 5)
dist_astar, exp_astar = search(start, goal, size, use_heuristic=True)
dist_dij, exp_dij = search(start, goal, size, use_heuristic=False)
print("A* distance:", dist_astar, "nodes expanded:", exp_astar)
print("Dijkstra distance:", dist_dij, "nodes expanded:", exp_dij)

Running this prints:

A* distance: 10 nodes expanded: 36
Dijkstra distance: 10 nodes expanded: 61

Both find the same correct shortest distance (10), but A* reaches it having expanded 36 nodes against Dijkstra's 61, because the heuristic steered expansion toward the goal instead of outward in every direction equally.

Trade-offs & pitfalls

A common mistake is treating A* as "a different algorithm" from Dijkstra rather than as Dijkstra with a heuristic added to the priority; with a heuristic of zero everywhere (as in the comparison above), A* degenerates to exactly Dijkstra, which is a good way to check an A* implementation for bugs. A second pitfall is picking a heuristic that overestimates in some region "because it prunes more nodes": an inadmissible heuristic can make A* return a path that is not actually shortest, so any heuristic must be checked against the admissibility property, not just judged by how much it speeds things up. Practically, Dijkstra's own implementation constant matters too: with a binary heap, each edge relaxation that improves a distance costs $O(\log V)$ to sift, which under many decrease-key-style relaxations is the dominant cost; the common fix is not switching to a Fibonacci or pairing heap (both have amortized $O(1)$ or near-$O(1)$ decrease-key but carry higher constant factors and more complex implementations) but instead allowing duplicate, stale entries in a plain binary heap and lazily discarding them on pop, which is what the Dijkstra implementation above already does and is the standard practical choice.

Direct answer

Training a 100B+-parameter transformer to finish in 2-4 weeks on commodity GPU nodes requires combining three axes of parallelism (data, tensor, pipeline) with aggressive memory optimization (ZeRO/FSDP sharding, activation checkpointing, mixed precision), and treating fault tolerance and checkpointing as first-class requirements rather than an afterthought, since a multi-week job on hundreds to thousands of GPUs will experience hardware failures during the run.

Structured elaboration

• Memory budget first: a 100B-parameter model in bf16 needs ~200GB for weights alone; with Adam's fp32 master weights, momentum, and variance, full unsharded state is on the order of 1.6TB. This is why ZeRO/FSDP-style sharding of optimizer state (and often gradients and parameters too) across the data-parallel group is not optional at this scale.
• 3D parallelism layout: tensor-parallel within a node (NVLink bandwidth), pipeline-parallel across a modest number of node groups (tolerates lower cross-node bandwidth, at the cost of pipeline bubble), data-parallel to scale out the remaining GPU count, with ZeRO stage 1 or 2 layered on the data-parallel dimension for further memory reduction without adding as much communication as ZeRO-3/FSDP would.
• Throughput target math: reaching a 2-4 week finish time for a fixed token budget requires computing the required aggregate FLOPs/sec from the model's FLOPs-per-token and target token count, then sizing the GPU count and expected model-FLOPs-utilization (MFU, typically 30-50% of peak for well-tuned large-model training) to hit that throughput.
• Fault tolerance and checkpointing: on commodity (as opposed to purpose-built, fully redundant) hardware, expect failures during a multi-week run; frequent, cheap incremental checkpoints (sharded across the same topology the model is sharded across) and automated restart-from-checkpoint are required so a single node failure costs minutes, not days.
• Activation checkpointing: recomputing activations during the backward pass instead of storing all of them trades extra compute (roughly 20-30% more FLOPs) for a large reduction in activation memory, which is usually the right trade at this model scale since memory, not compute, is the binding constraint.

Worked example

Target: 100B parameters, 300B training tokens, transformer FLOPs-per-token approximation of 6 x parameters (forward + backward). Total FLOPs $\approx 6 \times 100\times10^9 \times 300\times10^9 = 1.8\times10^{23}$ FLOPs. An A100 delivers roughly 312 TFLOPS of bf16 peak; at 40% MFU that's $\approx 1.25\times10^{14}$ effective FLOPS per GPU. To finish in 21 days (1.8144 x 10^6 seconds): required aggregate FLOPS = $1.8\times10^{23} / 1.8144\times10^6 \approx 9.9\times10^{16}$ FLOPS, requiring roughly $9.9\times10^{16} / 1.25\times10^{14} \approx 793$ GPUs at that utilization, so a cluster in the 800-1000 GPU range with headroom for failures and imperfect scaling efficiency.

Network topology, storage tiering, orchestration, and monitoring

• Network topology: within a node, the 8x A100s connect via NVLink/NVSwitch for tensor-parallel traffic; across nodes, a rail-optimized InfiniBand (or RoCE) fat-tree fabric with GPUDirect RDMA is standard so pipeline- and data-parallel collectives bypass the CPU and avoid becoming the bottleneck at 800-1000 GPU scale, since a flat, oversubscribed Ethernet fabric would leave cross-node all-reduce badly bandwidth-starved.
• Storage tiering: training data and checkpoints don't fit (or shouldn't live) on one tier. Hot, frequently-read shuffled shards of the token dataset live on local NVMe per node for the fastest, lowest-latency reads during data loading; a shared parallel filesystem (e.g. Lustre/GPFS) or high-throughput object store holds the full dataset and is the durable target for periodic sharded checkpoints; and cold, long-term artifact storage (final checkpoints, logs) goes to cheaper object storage (S3-class) once a run's active window has passed.
• Orchestration: a cluster scheduler (Kubernetes with a GPU-aware scheduler, or Slurm, common in HPC-style GPU clusters) handles job placement, GPU allocation, and automatic restart-from-checkpoint on node failure; the orchestration layer needs to cooperate with the training framework's own elastic/fault-tolerant world-size handling so a restart after a node failure resumes at the correct sharded checkpoint rather than restarting the whole job from scratch.
• Monitoring: production runs at this scale track GPU utilization and model-FLOPs-utilization (MFU) per node to catch degraded or straggling hardware early, loss/gradient-norm curves to catch training instability, and per-node health checks (ECC errors, thermal throttling, NVLink/IB link errors) so a failing GPU is drained before it corrupts a step rather than discovered only after a collective hangs.

Trade-offs & pitfalls

The single most common planning mistake at this scale is sizing the cluster off peak FLOPS instead of realistic MFU; a well-tuned large-model training run typically only achieves 30-50% of theoretical peak once communication overhead, pipeline bubbles, and memory-bound operations are accounted for, so plans that assume near-100% utilization consistently under-provision.

Direct answer
Accuracy is misleading here because the fraud class is a tiny fraction of all transactions, so a model that predicts "not fraud" for everything can score extremely high on accuracy while catching zero fraud. Report precision, recall, and precision-recall area under the curve (PR-AUC) instead, and expect all three to look worse as the fraud rate drops further, purely because the detection problem gets statistically harder, not because the model got worse.

Structured elaboration

• Why accuracy fails: if a base rate is 0.5%, always predicting "legitimate" already achieves 99.5% accuracy while catching 0% of fraud. Accuracy rewards the model for getting the overwhelming majority class right and says nothing about performance on the class that actually matters.
• What to report instead: precision (of the cases you flagged, what fraction were really fraud) and recall (of all the real fraud, what fraction did you catch) directly answer the two questions the fraud team actually cares about. PR-AUC summarizes the precision/recall trade-off across every possible threshold, which matters because the "right" threshold is a business decision, not a fixed property of the model.
• Why receiver operating characteristic area under the curve (ROC-AUC) is a weaker choice here: ROC-AUC's false-positive-rate axis is computed against the (very large) legitimate-transaction population, so it can look deceptively good even when precision at any usable operating point is poor. PR-AUC's precision axis is directly sensitive to the class imbalance, which is exactly the property you want a metric to be sensitive to on this problem.
• What changes as the base rate drops: for a FIXED classifier, a lower base rate mechanically lowers precision at any given recall level, because the same absolute number of false positives is now being compared against fewer true positives. A 0.05% fraud rate needs a noticeably more discriminative model (or a stricter operating threshold) than a 1% fraud rate just to hold precision constant.

Worked example
At a 0.5% fraud rate, predicting "not fraud" for every transaction is accurate 99.5% of the time and catches exactly 0% of fraud (recall 0, precision undefined). That single fact, stated plainly, is usually enough to convince a stakeholder that accuracy is the wrong headline number for this problem; it is worth leading a conversation with it before moving on to precision, recall, and PR-AUC.

Trade-offs and pitfalls
Reporting PR-AUC alone is not sufficient either. A single summary number hides where on the curve you actually plan to operate, and a fraud team cares specifically about precision and recall AT the threshold they will actually use, which is a business decision tied to review capacity (see the separate question on choosing an operating threshold under a review-queue capacity constraint). Always pair a summary metric like PR-AUC with the concrete precision/recall pair at the threshold you intend to ship.

Direct answer

A lift that spikes in week one and fades to flat by week two could be three different things wearing the same shape: a real novelty effect decaying to its true (smaller or zero) persistent level, a seasonal pattern that has nothing to do with the treatment and would show up in both arms if you looked, or a selection-bias artifact where the traffic source itself (a holiday push or a marketing campaign) skewed an unrepresentative mix of users into treatment. The fastest way to tell them apart is to check whether the decay tracks calendar time (seasonality), user exposure-age (novelty), or arm composition (selection bias), because each explanation leaves a distinct fingerprint in the data, not just a distinct story.

Structured elaboration

The checklist, in order

Disentangling novelty from selection bias when a marketing campaign is the traffic source

This case is harder because both explanations can be true at once and produce the same decaying shape. A marketing campaign that ramps down after week one changes who is arriving, not just how they behave: if campaign-driven traffic is disproportionately assigned into treatment (for example because the campaign linked directly into the treatment experience, or assignment happened downstream of a referral parameter correlated with arm), the week-one spike partially reflects a different, more click-happy population rather than a within-user novelty effect.

• Design fix: randomize on a unit and at a point in the funnel that is independent of campaign exposure, assigning before the user ever sees the campaign-specific landing experience, so acquisition-channel composition is balanced by construction rather than by hope.
• Post-hoc check: compare the new-vs-returning mix and traffic-source mix between arms week by week; if the treatment arm's share of campaign-driven new users is higher than control's, run a balance check on that specific subpopulation, not just the top-line allocation.
• Post-hoc correction: re-run the primary analysis restricted to users who arrived through non-campaign channels, and separately on campaign-driven users; if the effect concentrates in the campaign-driven segment and that segment is imbalanced between arms, the pooled week-one number is not a trustworthy estimate of the treatment effect for the general population.

Redesigning the experiment or analysis window

• Extend the primary analysis window well past the campaign's active period so the segment mix has time to normalize, and pre-register that the primary readout is a later window, not week one.
• Use a difference-in-differences framing (treatment-arm change from a pre-period baseline, minus control-arm change over the same calendar span) so genuine calendar movement common to both arms cancels out instead of being misread as a treatment effect.
• If holiday timing cannot be avoided, consider a staggered or replicated launch (the same treatment introduced to a second, non-holiday cohort later) so you get a second read that is not confounded with that specific calendar event.

Worked example

Suppose the pooled week-one lift is +8% on a baseline conversion rate of 10%, and segment analysis shows the treatment arm received 60% campaign-driven new users in week one versus 40% in control (stated, illustrative inputs). If campaign-driven new users convert at a naturally higher rate, say 14% versus 9% for organic users (illustrative inputs), the arm-level blended rate is a weighted average:
$\text{Treatment rate} = 0.6 \times 0.14 + 0.4 \times 0.09 = 0.084 + 0.036 = 0.120$
$\text{Control rate} = 0.4 \times 0.14 + 0.6 \times 0.09 = 0.056 + 0.054 = 0.110$
That gives an apparent lift of $(0.120 - 0.110)/0.110 \approx 9.1\%$, almost entirely explained by the composition difference rather than any within-user treatment effect, before even considering a real novelty component. This is why segment analysis has to run before you trust the topline number, not after.

Trade-offs and pitfalls

• Restricting analysis to non-campaign traffic to remove selection bias also shrinks your sample and can push you back into an underpowered read, exactly what the power-review step is meant to catch.
• Difference-in-differences assumes both arms would have moved in parallel absent the treatment; a campaign that specifically targets one arm's users breaks that assumption too, so DiD is not a free fix if the selection bias operates through campaign-to-arm assignment rather than through time.
• Extending the window to wait out both seasonality and novelty delays the launch decision; be explicit with stakeholders that "flat by week two" was never a valid stopping rule for this scenario, so the delay does not read as backpedaling.
• Retiring a variant because week two looks flat, without a power review, risks killing a real but modest persistent effect that a two-week window was never sized to detect in the first place.

Direct answer

I'd separate the timeline question from the model-complexity question: propose a simple analytics baseline that can ship inside the deadline, while being explicit about what it will and won't deliver, and frame the more complex model as a follow-up investment once the baseline's results are in.

Structured elaboration

1. Quantify the time-versus-value trade-off concretely. How much longer would the complex model realistically take, and what specific additional value would it capture that the simpler approach can't? Vague statements like "the ML model would be better" aren't useful to a stakeholder making a timeline call.
2. Estimate resourcing for both paths honestly. The simpler analytics approach's true cost (including validation, not just a quick query) versus the ML path's true cost (data preparation, training, evaluation, and the ongoing maintenance it creates).
3. Propose a phased plan. Ship the simpler approach on the stakeholder's timeline, instrument it well enough to measure its actual impact, and use that measurement to make the case (or not) for investing in the complex model afterward.
4. Keep the stakeholder's actual goal in view. The tight timeline usually reflects a real business pressure (a launch date, a reporting deadline); the plan needs to respect that pressure rather than just arguing it away.

Worked example

For an analysis that needs to inform a go or no-go decision in two weeks, a simple cohort-based analysis with clearly-stated assumptions and confidence bounds, delivered on time, is worth more to the business than a more rigorous model that arrives after the decision has already been made without it. Framing this honestly to the stakeholder (here's what the fast approach gives you, here's what it doesn't, here's what a slower approach would add) turns a potential conflict into a shared decision.

Trade-offs and pitfalls

The risk of always deferring to the tight timeline is under-delivering on decisions that genuinely need more rigor; the risk of insisting on the complex approach regardless of the deadline is missing the actual business window entirely, which can make even a perfect analysis worthless. The honest phased-plan framing avoids both traps by making the trade-off explicit rather than making the call unilaterally.