DoorDash Machine Learning Engineer Interview Preparation Guide - Mid Level

Situation: You're comparing two groups on a skewed, heavy-tailed metric (time-on-site, RPU). Goal: get reliable inference and a meaningful effect-size.

When to transform

• Use a log (or Box–Cox) transform when multiplicative effects are plausible and you care about proportional changes. Log often stabilizes variance and makes means interpretable as geometric means. Check back-transformed interpretation and heteroskedasticity.

Rank-based tests

• Mann–Whitney U (Wilcoxon) tests stochastic dominance / shift in distribution location but does NOT strictly test medians when shapes differ. Use when you want a distribution-agnostic test and are willing to sacrifice interpretability of mean differences.

Permutation (randomization) tests

• Nonparametric, exact (conditional on observed values), and testable for mean or other statistics. Use permutation of group labels to test whether observed difference (mean, median, trimmed mean) could arise by chance. Works well with any test statistic and small-to-moderate samples.

Bootstrap confidence intervals

• Use bootstrap CIs for medians, trimmed means, or ratios (median ratio); prefer bias-corrected and accelerated (BCa) or percentile bootstrap with large resamples (≥10k). For heavy tails, bootstrap the trimmed mean (e.g., 10–20% trimming) to reduce influence of outliers.

Practical recipe

1. Visualize (ECDFs, violin, boxplots on raw & log scales).
2. Decide target estimand: mean (sensitive), median, trimmed mean, or ratio.
3. If multiplicative effects: log-transform and analyze means; report geometric mean ratio with CI.
4. If robust location: estimate median or 20% trimmed mean and compute BCa bootstrap CI; test via permutation on that statistic.
5. Use permutation test for p-values on chosen statistic if distributional assumptions are doubtful.
6. As sensitivity, run both transformed-mean and robust-median analyses; report both.

Effect sizes & reporting

• Report the estimand and units: difference in medians (with bootstrap CI), ratio of geometric means (with CI), or trimmed-mean difference.
• Avoid raw Cohen’s d on skewed data; prefer Cliff’s Delta (probability one obs > other) or Hodges–Lehmann estimator (median of pairwise differences) with CI.
• Always report: estimand, point estimate, CI, p-value (test used), sample sizes, and visualization. Mention sensitivity analyses (transform, trimmed, rank) and robustness to outliers.

Worked example (why this matters, not just definitions). Group A revenue-per-user in dollars = [12, 15, 14, 210, 13] (one whale customer at $210); Group B = [20, 22, 19, 21, 18]. The naive mean comparison is actively misleading here: mean(A) = 264/5 = $52.80, mean(B) = 100/5 = $20.00, so a plain Cohen's d style comparison says A is $32.80 HIGHER, entirely driven by the single $210 outlier. Now compute the two robust alternatives by hand. Cliff's Delta counts, across all 5x5=25 pairwise comparisons, how often A's value exceeds B's versus the reverse: only the 5 comparisons involving the $210 outlier have A>B, while all 20 remaining comparisons have A<B, giving Cliff's Delta = (5 − 20) / 25 = −0.6, meaning B is typically the larger value in a pairwise sense. The Hodges-Lehmann estimator is the median of all 25 pairwise differences (A_i − B_j); sorting those 25 differences (20 negative values from −10 down to −3, plus 5 large positive values from the outlier row) and taking the 13th of 25 gives a median difference of −6, i.e. the robust point estimate is that A's typical value is about $6 LOWER than B's, the opposite sign of the naive mean comparison. This is exactly why the recommendation above is to avoid raw Cohen's d on skewed data: the one outlier flips the naive comparison's sign, while both rank-based estimators correctly reflect that most of Group A sits below Group B.

Why this approach

• Combines interpretability (median/ratio), robustness (trim/boot), and valid inference without relying on normality assumptions (permutation/boot).

First split the problem into two buckets

1. True optimization issues: the model is mathematically harder to train at larger scale.
2. Distributed systems issues: the parallel run is changing the effective training behavior.

Things I would investigate

• Effective batch size and learning rate scaling. A bigger world size often means a larger batch, which can require LR tuning.
• Data sharding. I would verify that each sample is seen once, not duplicated or skipped.
• Gradient synchronization. I would check that all ranks contribute the same gradients and that there are no stale or dropped updates.
• Precision and numerics. Mixed precision can introduce overflow, underflow, or different rounding behavior at scale.
• Stateful layers like batch normalization, which can behave differently across replicas.

How I would separate them
I would compare small-scale and large-scale runs with the same global batch and fixed seeds, then inspect gradient norms, loss curves, and sample hashes.

Worked example
If 8 workers train with a batch of 256 each, the global batch becomes 2048. If loss gets noisier only after that change, it may be an optimization tuning problem. If the run only degrades when messages cross nodes, it is more likely a communication or synchronization problem.

That distinction tells me whether to tune the optimizer or fix the distributed stack.

Direct answer
At high incident volume, right-sizing postmortem effort means reviewing incidents proportionally to their severity and learning value rather than giving every incident the same heavyweight treatment, since a full deep-dive on every minor blip both burns out reviewers and dilutes attention from the incidents that actually deserve it.

Structured elaboration

• Tier the review depth by severity and novelty. High-severity or novel-pattern incidents get the full treatment: timeline reconstruction, root cause and contributing factors, cross-team facilitation. Low-severity, well-understood, or clearly one-off incidents get a much lighter 'mini' review: a short written summary with a root cause and, if warranted, one action item, no meeting required.
• Rotate reviewers rather than relying on the same few people. Concentrating review responsibility on a small group both burns them out and creates a bottleneck; distributing it (with a shared template and light training) keeps quality consistent while reducing individual load.
• Automate triage where the pattern is well understood. If a category of incident has occurred many times with the same known cause, an automated or templated mini-postmortem that flags it as a known, tracked pattern (rather than requiring fresh analysis every time) frees up reviewer time for genuinely novel incidents.
• Track a pattern-level view, not just per-incident. A large volume of small, similar incidents is itself a signal worth its own dedicated (heavier) review, even if none of them individually crossed the severity threshold for a full postmortem, since the aggregate pattern is often more informative than any single instance.
• Measure whether this is actually preserving learning value, not just reducing workload. Track whether recurrence rates for previously-reviewed incident classes stay flat or improve even as review depth for minor incidents drops, to confirm the lighter-touch approach isn't quietly letting real risk go unaddressed.

Worked example
An organization runs roughly 2,000 incidents a month and full postmortems have become a rubber-stamp exercise nobody has time to do well. The fix: define three tiers. Tier 1 (high severity or genuinely novel pattern, maybe 5% of incidents) gets a full facilitated postmortem within a defined turnaround. Tier 2 (moderate severity, somewhat familiar pattern, maybe 25%) gets a lightweight async writeup by the on-call responder, reviewed by a rotating peer within a week, no live meeting required unless something surprising surfaces. Tier 3 (low severity, well-understood and recurring pattern, the remaining ~70%) gets an automated, templated log entry tagging the known category, with no individual analysis required unless the volume of that specific category spikes, which triggers escalation to a full pattern-level review. Reviewer rotation is enforced across teams so no single person is doing more than a defined share of Tier 1 and Tier 2 reviews in a given month.

Trade-offs and pitfalls
The biggest risk of this approach is under-reviewing something that seemed minor in isolation but was actually an early instance of a bigger, developing problem; the pattern-level tracking (watching for a spike in a normally-quiet Tier 3 category) is what catches that, and skipping it is the most common mistake when teams implement tiering purely to save time.

Direct answer

Deciding what to cut under a hard deadline starts by separating what the deadline is actually protecting, the specific business or compliance need, from everything else that got bundled into "the launch," cutting what doesn't serve that specific need, and being explicit with stakeholders about the trade-off rather than quietly shipping less than promised.

Structured elaboration

• Anchor the cut decision to the actual constraint. Ask what specifically makes the date hard: a regulatory deadline, a committed customer date, a dependency another team is waiting on. That answer tells you which piece of scope is truly load-bearing and which is along for the ride.
• Use a simple filter for what to cut: does this piece of scope directly serve the hard constraint, or does it serve a separate, softer goal that could ship later without anyone being harmed. Cut the latter first.
• Communicate the trade-off explicitly and early, not as a surprise at the deadline. Tell stakeholders exactly what's being deferred, why, and when it's coming, and get sign-off from whoever actually owns the constraint, not just informal agreement from the team.
• Protect the core value on the way out the door. After deciding what to cut, sanity-check that what remains still fully satisfies the reason the deadline existed in the first place; a stripped-down feature that no longer serves its original purpose is a failure dressed up as a delivery.

Worked example

Three weeks before a hard compliance deadline requiring customers to be able to reset their own password without contacting support, the original scope also included an admin bulk-reset tool for support staff. The compliance deadline is specifically about the end-user self-service flow, not the admin tool, so the admin bulk-reset tool is the piece to cut: it serves an internal convenience goal, and support already has a slower manual workaround for it. I brought this to the compliance officer and the support lead one week before the deadline, in writing: ship the self-service reset flow now to meet the deadline, and ship the admin bulk-reset tool as a fast-follow within 4 weeks. The compliance officer signed off since they own the actual constraint being protected. We shipped the self-service flow on the compliance deadline, and the admin tool followed 4 weeks later as promised, with support continuing their existing manual process in the meantime.

Trade-offs and pitfalls

A common mistake is cutting whatever is easiest to cut rather than whatever least serves the actual hard constraint, which sometimes means removing something highly visible while leaving in scope creep that has nothing to do with the deadline. Another failure is deciding the cut unilaterally and only informing stakeholders after the fact, which erodes trust even when the decision itself was correct. Watch also for descoping so aggressively that what ships technically hits the date but no longer delivers the actual value the deadline existed to protect, which is a worse outcome than a short, well-communicated delay would have been.

Direct answer
Quantization edge cases come from three interacting error sources: representational error (a continuous value snapped to the nearest representable discrete level always loses some precision), saturation (a value outside the calibrated range clips to the boundary instead of scaling correctly), and rounding propagation (small per-layer errors compound as they flow through a deep network, and can flip a borderline classification decision even when each individual layer's error looks tiny). Per-channel quantization (a separate scale factor per output channel) versus per-tensor quantization (one scale factor for the whole weight tensor) matters because a single outlier-magnitude channel forces per-tensor quantization to use a wide scale that wastes precision on every OTHER channel, while per-channel isolates the damage to just the outlier's own channel.

Structured elaboration

Representational error. Quantizing to INT8 (8-bit integer, 256 representable levels) maps a continuous range onto a small fixed set of discrete steps; the maximum per-value error is half the step size (scale / 2), which is unavoidable at a given bit width and is why the SCALE choice, not the quantization algorithm, is the primary lever for controlling error.

Saturation. The scale factor is derived from a calibration pass (observing typical activation/weight ranges on representative data). Any RUNTIME value outside that calibrated range does not error, it clips silently to the minimum or maximum representable code, which means an out-of-distribution input at inference (a value 10x larger than anything seen during calibration) is not merely imprecise, it is truncated to the boundary value with no signal that clipping occurred.

Rounding propagation and misclassification. Each quantized layer's output error becomes the NEXT layer's input error, and while individual layer errors are small, they do not necessarily cancel, in the worst case they compound. The practical failure mode for classification is not a globally degraded accuracy number, it is a small subset of BORDERLINE-confidence examples (already close to a decision boundary in the full-precision model) flipping to the wrong class, while high-confidence examples are essentially unaffected. This is why aggregate accuracy alone under-reports the risk: a model can show a small overall accuracy drop while a specific, high-value borderline segment (e.g. fraud cases near the decision threshold) degrades much more.

Per-channel versus per-tensor. Per-tensor quantization computes ONE scale from the single largest-magnitude value across the entire weight tensor. If one output channel legitimately has much larger weight magnitudes than the rest (common for a channel tied to a rare-but-important feature), that one channel's magnitude sets a wide scale that then under-resolves every other, well-behaved channel. Per-channel quantization computes a separate scale per output channel, so the outlier channel's wide scale stays isolated to itself.

Outlier handling during calibration. Beyond per-channel scaling, calibration itself needs an explicit policy for outliers: using the raw min/max (sensitive to a single extreme value) versus a percentile-based range (e.g. clipping calibration itself to the 99.9th percentile, deliberately accepting some saturation on the rarest values in exchange for better resolution on the bulk of the distribution) is a real design trade-off, not a default to leave unexamined.

Experiments to detect quantization-induced failures, and how to mitigate them. Two experiments cover the failure modes above: (1) a per-channel-versus-per-tensor error comparison, exactly like the worked example below, run against the model's REAL weight tensors rather than a synthetic example, to detect whether any channel's magnitude is wide enough to be dragging per-tensor precision down for the rest; and (2) a targeted evaluation slice restricted to examples near the full-precision model's decision boundary, to detect misclassification risk that an aggregate accuracy number hides. The corresponding mitigations are directly paired with what each experiment detects: switch the affected layers from per-tensor to per-channel quantization if experiment 1 shows a wide gap, switch calibration from raw min/max to a percentile-clipped range if outliers are driving saturation, and, for models where post-training quantization's borderline-example degradation is still unacceptable after both of those, fall back to quantization-aware training (training with simulated quantization in the loop, letting the model adapt to the rounding it will actually experience at inference), which is a substantially larger intervention and should be reserved for the case where the two lighter mitigations are proven, via experiment 2's evaluation slice, not to be enough.

Worked example (executed): per-tensor versus per-channel error, and a saturation case

import numpy as np
rng = np.random.default_rng(0)
W = rng.normal(0, 0.05, size=(4, 16))
W[1, 3] = 2.0  # one outlier weight in channel 1

def quantize(x, scale):
    q = np.clip(np.round(x / scale), -127, 127)
    return q * scale

def per_tensor(W):
    scale = np.max(np.abs(W)) / 127.0
    return quantize(W, scale)

def per_channel(W):
    scales = np.max(np.abs(W), axis=1, keepdims=True) / 127.0
    return quantize(W, scales)

dq_t, dq_c = per_tensor(W), per_channel(W)
err_t = np.mean(np.abs(W - dq_t), axis=1)
err_c = np.mean(np.abs(W - dq_c), axis=1)
for ch in range(4):
    print(ch, err_t[ch], err_c[ch], err_t[ch] / max(err_c[ch], 1e-12))

# saturation
calibration_max, runtime_outlier = 0.5, 5.0
scale = calibration_max / 127.0
q = np.clip(np.round(runtime_outlier / scale), -127, 127)
print(q, q * scale, abs(q * scale - runtime_outlier) / runtime_outlier)

Executed output: mean absolute quantization error per channel (per-tensor vs per-channel): channel 0: 0.003189 vs 0.000149 (21.4x worse under per-tensor); channel 1 (the outlier): 0.003659 vs 0.003659 (1.0x, identical, since per-channel cannot do better on the outlier itself); channel 2: 0.004101 vs 0.000179 (23.0x); channel 3: 0.003630 vs 0.000173 (21.0x). The minimum ratio across the three healthy channels is 21.0x, confirming per-tensor quantization measurably degrades every channel that is not the outlier. The saturation test: a runtime activation of 5.0 against a calibration range of 0.5 clips to quantized code 127, dequantizing to 0.5, a 90% relative error, with no error raised.

Trade-offs & pitfalls
The most common mistake is evaluating quantization quality only via a single aggregate accuracy metric on a general test set; because rounding-propagation failures concentrate on already-borderline examples, a targeted evaluation slice (examples near the full-precision model's decision boundary, and any known high-value segment) will show a failure that the aggregate metric hides. A second pitfall is choosing per-channel quantization everywhere by default without checking its cost: per-channel scales require more metadata (one scale per channel instead of one per tensor) and, on some hardware/kernel implementations, cannot be as efficiently fused into the matrix-multiply, so the choice is a genuine accuracy-versus-throughput trade-off, not a strict improvement. Third, silent saturation is easy to miss entirely in testing if the test suite only exercises in-distribution calibration-range values; an explicit out-of-calibration-range test case (as in the worked example) is necessary specifically because saturation produces no exception, only a quiet, potentially large error.

Direct answer

A bounded queue (fixed maxsize) applies backpressure automatically: once it is full, put() blocks (threading) or suspends (await put, asyncio) until a consumer frees a slot, so a fast producer is throttled to the consumers' pace without any extra code. queue.Queue gives you this with an internal threading.Condition; asyncio.Queue gives you the same shape with await instead of blocking waits. The wake-up semantics differ by mechanism: a raw condition variable needs you to choose notify() (wake exactly one waiter) versus notify_all() (wake everyone, most of whom will just recheck and go back to waiting), while asyncio.Queue hides that choice entirely, await put/await get suspend and resume the right coroutine without you managing wake-ups by hand.

Structured elaboration

Threading version: queue.Queue(maxsize=N) is backed by a Lock plus two Conditions internally (not-full, not-empty). put() blocks on the not-full condition when the queue is at capacity; get() blocks on not-empty when it is empty. You do not touch the condition variables yourself, Queue already calls notify() correctly.

asyncio version: a coroutine is a function defined with async def that can pause mid-execution, at an await, and let other coroutines run, then resume later exactly where it left off; the event loop is the scheduler that decides which paused coroutine gets to run next. asyncio.Queue(maxsize=N) has the same contract as the threading version, but await q.put(item) suspends the coroutine (yielding control to the event loop, not blocking an OS thread) when full, and await q.get() suspends when empty. Concurrency comes from running many producer/consumer coroutines as tasks on one event loop instead of many OS threads.

Graceful shutdown, both versions: signal producers to stop (an Event), let running producers finish their current item, then drain the queue (q.join(), which waits until every put item has had a matching task_done()), then push one sentinel value per consumer so each one exits its loop cleanly. q.join() hanging is almost always a sign that a consumer path skipped task_done(), most often via an exception on the processing line before it reached the task_done() call.

The harder variant, without queue.Queue/asyncio.Queue: you build the bounded buffer yourself from a Lock plus two Condition objects sharing that lock, one for "not full", one for "not empty." This is exactly what queue.Queue does internally, made explicit:

import threading, collections

class BoundedBuffer:
    def __init__(self, maxsize):
        self.maxsize = maxsize
        self.items = collections.deque()
        self.lock = threading.Lock()
        self.not_full = threading.Condition(self.lock)
        self.not_empty = threading.Condition(self.lock)

    def put(self, item):
        with self.not_full:
            while len(self.items) >= self.maxsize:
                self.not_full.wait()      # releases the lock, sleeps until notified
            self.items.append(item)
            self.not_empty.notify()       # wake exactly one waiting consumer

    def get(self):
        with self.not_empty:
            while not self.items:
                self.not_empty.wait()
            item = self.items.popleft()
            self.not_full.notify()        # wake exactly one waiting producer
            return item

notify() (one waiter) is correct here because exactly one slot opened up, waking everyone with notify_all() would just cause the other waiters to recheck their while condition and go back to sleep, which is harmless but wasteful (a "thundering herd" of wasted wake-ups). notify_all() earns its keep when a single event can satisfy many different waiters at once, which is not the case for a single-slot state change like this.

Worked example

asyncio.Queue-based pipeline with graceful shutdown, verified on CPython 3.12 (produced and consumed counts always match, since q.join() guarantees every item was drained before sentinels are sent; the exact counts vary run to run because they depend on real-time scheduling):

import asyncio, random

SENTINEL = object()

async def producer(pid, q, stop_event, rng):
    i = 0
    while not stop_event.is_set():
        await q.put((pid, i))          # blocks here once the queue is full
        i += 1
        await asyncio.sleep(rng.random() * 0.02)

async def consumer(cid, q, processed):
    while True:
        item = await q.get()
        if item is SENTINEL:
            q.task_done()
            break
        processed.append((cid, item))
        q.task_done()

async def run_pipeline(n_producers=2, n_consumers=3, maxsize=5, run_time=0.3, seed=7):
    rng = random.Random(seed)
    q = asyncio.Queue(maxsize=maxsize)
    stop_event = asyncio.Event()
    processed = []

    producers = [asyncio.create_task(producer(i, q, stop_event, rng)) for i in range(n_producers)]
    consumers = [asyncio.create_task(consumer(i, q, processed)) for i in range(n_consumers)]

    await asyncio.sleep(run_time)
    stop_event.set()
    await asyncio.gather(*producers)
    await q.join()                          # wait until every produced item is processed
    for _ in consumers:
        await q.put(SENTINEL)
    await asyncio.gather(*consumers)
    assert len(processed) == sum(1 for _ in processed)  # every produced item made it through

asyncio.run(run_pipeline())

A representative run reported "total produced: 62, total consumed: 62"; the invariant produced == consumed holds on every run because q.join() will not return until all items are drained, the specific count itself is not reproducible (it depends on wall-clock scheduling) and is not the claim being made here.

Trade-offs & pitfalls

• Complexity: $O(1)$ per item for queue operations; space is bounded by maxsize for the queue itself, plus $O(k)$ for the $k$ in-flight producer/consumer tasks.
• Edge case: if a consumer raises an exception before calling task_done(), q.join() hangs forever; wrap the processing body in try/finally so task_done() always runs.
• Edge case: sending fewer sentinels than there are consumers leaves some consumers blocked on get() forever; one sentinel per consumer is required, not one total.
• Choosing notify() when the change actually affects multiple waiters (rare for a single bounded queue, more common in custom condition-based coordination) silently starves the other waiters; if you are not certain only one waiter's condition changed, notify_all() is the safer default even though it wakes more coroutines/threads than strictly necessary.
• The asyncio version scales to many more concurrent producers/consumers than the threading version for the same memory budget, since coroutines are far cheaper than OS threads, but it only helps if the "work" inside producer/consumer is itself non-blocking; a CPU-bound consumer inside an asyncio pipeline will stall the whole event loop exactly like any other CPU-bound coroutine.

Problem formalization:

• Environment: M services i∈{1..M}, T decision rounds. Each round t we allocate a vector of discrete resources a_t = (a_{t,1},...,a_{t,M}) from a feasible budget set A (e.g., ∑i c_i a{t,i} ≤ C, per-host caps).
• Reward: stochastic QoS reward r_t = ∑i R_i(a{t,i}, s_{t,i}) where s_{t,i} are unknown service states (load, SLO attainment); we observe rewards (possibly delayed/noisy) after allocation. Objective: maximize cumulative expected reward E[∑_{t=1}^T r_t] (or minimize regret vs best fixed allocation/policy).
• Unknowns: response curves R_i(·) (monotone, possibly concave), cross-service interactions (contention).

Why this is a combinatorial bandit:

• Each action is a combinatorial object (vector allocation under constraints) drawn from an exponential-size set A. Reward decomposes (possibly approximately) over arms (services) but choice couples via budget constraints. Observations may be semi-bandit (per-service feedback) or bandit (only aggregate). This matches combinatorial multi-armed bandits (CMAB).

Algorithmic solutions:

1. Approximate Combinatorial UCB (CMUCB):

• Maintain per-service estimates μ̂_i(a) or parametric models (e.g., diminishing returns curve). Compute optimistic estimates μ̂_i + bonus_i(t) and solve an offline knapsack/ILP to pick allocation maximizing sum of optimistic rewards subject to constraints. Use approximation-aware regret bounds: if offline oracle is α-approx, regret scales O((poly)·T/α). Use structured models (GLM, parametric) to reduce sample complexity.

2. Greedy with submodular objectives:

• If ∑i R_i(a{i}) is monotone submodular in discrete resource units, use greedy hill-climbing per round: iteratively allocate marginal unit with largest estimated marginal gain. With (1-1/e) approximation and confidence bonuses, this gives provable approximate regret. Practical variant: Thompson Sampling + greedy sampling to handle uncertainty.

3. Contextual/parametric bandits:

• Use contextual CMAB: include features (current load, queues). Fit online Bayesian/linear models and use LinUCB/Thompson Sampling over allocations with efficient combinatorial optimization via knapsack solvers.

4. Handling interactions:

• If strong cross-coupling, treat joint arms for small groups or use low-rank factorization of interaction matrix and learn latent factors.

Practical monitoring and risk controls:

• Safety constraints: enforce hard SLO-preserving actions (never drop below reserve allocations), and clipping of exploratory allocations.
• Conservative bootstrapping: begin with off-line A/B tests or simulation to initialize priors; use decaying exploration rates.
• Canary & rollback: stage policy in canary cluster with traffic shadowing; automatic rollback on SLO degradation beyond thresholds.
• Observability: per-service QoS, latencies, error rates, resource usage, and reward attribution with high-cardinality logs; compute regret/expected reward delta vs baseline in real-time.
• Alarm & intervention: require human-in-loop for high-impact allocation changes, throttle exploratory updates when variance or delayed feedback increases.
• Explainability: surface marginal gains used for each allocation decision to operators.

Trade-offs:

• Exact CMAB algorithms give theoretical guarantees but need efficient oracles; greedy/submodular methods scale well and are robust if submodularity approx holds. Contextual models reduce samples but require reliable features.

This design balances provable algorithms with practical safety: start with parametric/contextual CMUCB + greedy allocation, strong monitoring and safety gates, then iterate as more data reduces uncertainty.

Direct answer

For heavy-tailed data, reach for the trimmed mean (the mean after dropping a fixed percentage from each tail), the winsorized mean (capping extreme values at a percentile rather than dropping them), and the median absolute deviation, or MAD (a robust measure of spread built from the median rather than the mean). Each protects against exactly what the plain mean and standard deviation are vulnerable to: a small number of extreme values dominating the summary.

What each protects against, specifically

The plain mean weights every value equally in the sum, so one extreme transaction can shift it substantially; a trimmed mean removes the most extreme values entirely before averaging, which is simple but throws away potentially real information at the tails. A winsorized mean keeps every observation but caps the extreme ones at a threshold, preserving the row count while limiting any single point's influence, a useful middle ground when you don't want to discard rows outright. MAD replaces both the center (the median) and the spread calculation with something that isn't sensitive to a handful of extreme values the way variance is, since the median of a bunch of moderate values plus one extreme one barely moves.

Worked example

Twelve values representing transaction sizes: mostly clustered between 11 and 15, with one value of 250. The plain mean comes out to 32.75, badly distorted by that single value; the median is 13.0, essentially unaffected. A 10% trimmed mean comes out to 13.2, very close to the median, having simply excluded the most extreme values from each end before averaging. The winsorized version (capping the top and bottom 10% at the next-most-extreme retained value) gives a mean of 13.25, similarly close to the median while still using every row. The MAD, scaled to be comparable to a standard deviation on normal data, comes out to about 1.48, a genuinely honest measure of the typical spread among the twelve values, in stark contrast to what the standard deviation (dominated by that single 250) would report.

Trade-offs and pitfalls

Trimming and winsorizing require choosing a percentage to trim or cap at, which is itself a judgment call, not free of assumptions; too aggressive a trim can discard real, meaningful extreme values along with true anomalies. Always report which robust method you used and at what threshold, since "robust mean" without that detail hides a real analytical choice.

Direct answer. Scaling up GPU count or enabling mixed precision changes several things simultaneously (effective batch size, gradient synchronization behavior, numeric precision), so a training run that was stable before and diverges after needs a plan that isolates ONE variable at a time rather than reverting the whole change and guessing.

Three terms this depends on. A replica is one full copy of the model living on one GPU. DDP (DistributedDataParallel) is the synchronous data-parallel wrapper that runs one process per GPU and keeps the replicas in step. All-reduce is the collective operation that combines a tensor across every replica and hands every replica back the same combined result, which is how the per-replica gradients become one shared gradient each step.

Plausible causes, and what to test first. The single highest-prior cause when scaling GPU count is learning-rate scaling: if the effective batch size grows with GPU count but the learning rate doesn't, or is scaled incorrectly (linear scaling isn't always the right rule for every optimizer), training can become unstable purely from that mismatch, and it's the cheapest hypothesis to test. Worked with real numbers so "adjusted for the new batch size" is computable: a stable baseline of 8 GPUs at 32 examples per device is an effective batch of 256 at LR 0.1. Scaling to 64 GPUs at the same 32 per device gives an effective batch of 2048, an 8x increase. The linear rule says LR = 0.1 x 2048/256 = 0.8. The square-root rule says LR = 0.1 x sqrt(8) = 0.283. If the job was launched at 0.8 and diverges, the diagnostic rerun is at 0.283, and the sanity rerun (which trades throughput for certainty) is at the original 0.1.

Next, check batch-norm statistics: if BatchNorm computes per-replica statistics independently rather than synchronizing across replicas, a network with BatchNorm can behave differently, and less stably, at high replica counts purely because each replica now sees a smaller effective batch for normalization purposes. Then check all-reduce precision (is gradient synchronization happening in a lower precision than intended) and synchronization/seed-order differences (are all replicas actually initialized identically before training starts).

For mixed precision (AMP) specifically. Gradient-accumulation interactions with AMP are a common culprit. The mechanics, since the ordering is the whole bug: loss scaling multiplies the loss by a large factor S before the backward pass so that small fp16 gradients do not underflow to zero, and the gradients must later be divided by S ("unscaled") before the optimizer step. With gradient accumulation over K micro-batches, the correct order is accumulate-then-unscale: keep every micro-batch's gradients scaled, sum all K of them, and unscale ONCE immediately before the optimizer step. The wrong order, unscale-then-accumulate, divides each micro-batch's gradients by S as it goes and then accumulates already-unscaled values, while the scaler still believes it owes the division, so the effective gradient magnitude ends up off by a factor tied to K and the scaler's overflow check runs against tensors that are no longer on the scale it expects. Confirm which order your framework's AMP integration actually uses before assuming the bug is elsewhere. Separately, some DDP jobs show gradient divergence where SOME workers produce NaN and others don't purely from AMP's dynamic loss scaling adjusting independently per worker if it isn't explicitly synchronized, worth checking whether your framework synchronizes the loss-scale factor across replicas or lets each pick its own.

What you would expect to observe if each cause were the real one. This is the half that turns the list into a diagnosis:

• Learning-rate / batch-size mismatch: divergence arrives within the first few hundred steps and arrives EARLIER the more GPUs you add, and the step at which it blows up moves monotonically as you sweep LR. Rerunning the same 64-GPU job at 0.283 (or at 0.1) with everything else untouched restores stability. Gradient norms grow smoothly over several steps before the blow-up rather than jumping in one step.
• BatchNorm computing per-replica statistics: dump each replica's BatchNorm running mean and running variance buffers and compare them across ranks. Synchronized normalization keeps them near-identical; the unsynchronized failure shows them visibly diverging, with the spread widening as per-device batch size falls. Crucially, the gradients themselves stay finite the whole time, so this looks like instability without any numeric red flag. It gets worse as you hold total batch fixed and add GPUs (shrinking per-device batch), and it disappears when you swap in synchronized BatchNorm or raise the per-device batch back up.
• All-reduce running in a lower precision than intended: checksum or norm each rank's gradient tensor immediately BEFORE the collective and immediately AFTER it. Before, the ranks legitimately differ, since each saw a different data shard. After, every rank must hold a bit-identical tensor. The tell is that post-collective gradients agree only to the first three or four significant digits rather than exactly, and that the disagreement grows with rank count, because more partial sums mean more accumulated rounding. Compare against a reference reduction performed in fp32 to size the error.
• Seed or initialization-order mismatch: hash every replica's parameters at step 0, before any data is seen. They must be identical. If rank 0's hash differs from rank 1's at step 0, the replicas never started from the same weights, and the shared averaged gradient is being applied to divergent parameter sets. The signature is that divergence appears essentially at step 1, not after a hundred healthy steps.
• AMP unscale-then-accumulate ordering: instrument the global gradient norm immediately before the optimizer step, and compare against the same total batch run with K=1 (no accumulation). The two should agree within numerical noise. The ordering bug shows up as a clean multiplicative discrepancy tied to K, a factor of K or 1/K rather than a vague drift, which is what makes it identifiable rather than merely suspicious.
• AMP loss-scale desync across workers: log the current loss-scale value per rank per step. Healthy runs show every rank holding the same value and stepping it in lockstep. The failure shows ranks holding different scales at the same step (one at 1024 while another is at 32768), SOME ranks reporting NaN while others report clean gradients, and the identity of the NaN-reporting ranks changing from run to run.

Why the effective batch size is the thread running through all of this. Scaling GPU count in synchronous data-parallel training multiplies the effective batch size by the number of new replicas, which is exactly what makes the learning-rate hypothesis the highest-prior one and why the arithmetic above matters. It also means that if the instability appears only past a specific scale rather than at every scale, that threshold is itself evidence: a genuine LR/batch mismatch only becomes destabilizing once the step size and the curvature disagree badly enough, whereas a hardware or environment fault would be present regardless of how many replicas are running.

Prioritized order to test. Order by what each check actually costs to run, then by prior probability, because two of these are nearly free and one of them is a precondition for trusting any of the others.

0. Free, so run these before spending a single GPU-hour on a rerun. Hash every replica's parameters at step 0: one hash per rank, zero training steps, and if the hashes disagree you are done, because none of the cross-rank comparisons below mean anything until the replicas provably start from the same weights. Under AMP, also log each rank's loss-scale value per step, which is one scalar per rank and settles the desync hypothesis outright. Neither of these is the most likely cause; they go first because they are cheaper than everything else and because they are the two that can invalidate the other measurements.
1. Learning-rate scaling, the highest-prior cause and the first hypothesis worth real GPU-hours: rerun the 64-GPU job at 0.283, and at the original 0.1 if that is still unstable. One short run, one variable, directly falsifiable.
2. AMP accumulate-then-unscale ordering, whenever gradient accumulation is on: compare the pre-optimizer global gradient norm against the same total batch run at K=1. One extra short run and one scalar, and the tell is a clean factor of K or 1/K rather than a drift.
3. BatchNorm synchronization: dump the running-mean and running-variance buffers on every rank and diff them. More instrumentation than anything above, still far less than the last item.
4. All-reduce precision, last: checksumming every rank's gradient tensor before and after the collective is the heaviest instrumentation in this list, and it is also the least common root cause for a symptom that appeared specifically at a scaling boundary.

Direct answer. Break a large refactor PR into logical AREAS (not just files) and review each area for a DIFFERENT specific risk, starting with the area most likely to hide a behavior change, and require the diff itself to prove equivalence (tests, before/after comparisons) rather than relying on reading alone to catch every issue.

A prioritized review strategy

1. Read the PR description and any linked design doc FIRST to understand intent before touching the diff -- reviewing line-by-line without knowing the GOAL risks nitpicking style while missing whether the refactor actually achieves what it claims.
2. Identify which parts are pure refactor (should be behavior-preserving) versus which parts change actual preprocessing behavior -- these need fundamentally different scrutiny: behavior-preserving changes should be verifiable against existing tests/characterization tests; behavior CHANGES need their own justification and new tests.
3. Review the riskiest area first: for a training pipeline refactor, that's usually wherever DATA SHAPE or ORDERING could silently change (a reordered transform step, a changed aggregation) since these bugs often don't crash, they just quietly produce a different (wrong) model.
4. Check test coverage of the CHANGED areas specifically, not just that 'tests pass' -- passing tests only mean something if they actually exercise the parts that changed; a refactor that touches code no existing test covers needs new tests added as part of the PR, not deferred.
5. Verify claimed behavior-preservation directly if possible: for a preprocessing refactor, request (or run yourself) a before/after comparison on a fixed dataset to confirm outputs match, rather than trusting a visual code read alone to catch every subtle discrepancy.
6. Review readability/structure last, after correctness -- a well-structured refactor that subtly changes behavior is a worse outcome than a slightly-awkward one that's provably behavior-preserving; get correctness confidence before spending review effort on style.

Trade-offs and pitfalls

• Reviewing a 500-line refactor PR as one undifferentiated block invites either rubber-stamping (too much to hold in your head) or excessive nitpicking on the parts that happen to be easiest to spot-check (style) while missing the parts that actually matter (behavior preservation) -- explicit prioritization by risk avoids both failure modes.
• If the PR bundles a genuine behavior CHANGE together with a large refactor, push back on splitting them into separate PRs before continuing review -- reviewing 'is this refactor safe' and 'is this new behavior correct' simultaneously in one diff makes both harder to verify confidently.