DoorDash Machine Learning Engineer Interview Preparation Guide - Mid Level

Situation: You're comparing two groups on a skewed, heavy-tailed metric (time-on-site, RPU). Goal: get reliable inference and a meaningful effect-size.

When to transform

• Use a log (or Box–Cox) transform when multiplicative effects are plausible and you care about proportional changes. Log often stabilizes variance and makes means interpretable as geometric means. Check back-transformed interpretation and heteroskedasticity.

Rank-based tests

• Mann–Whitney U (Wilcoxon) tests stochastic dominance / shift in distribution location but does NOT strictly test medians when shapes differ. Use when you want a distribution-agnostic test and are willing to sacrifice interpretability of mean differences.

Permutation (randomization) tests

• Nonparametric, exact (conditional on observed values), and testable for mean or other statistics. Use permutation of group labels to test whether observed difference (mean, median, trimmed mean) could arise by chance. Works well with any test statistic and small-to-moderate samples.

Bootstrap confidence intervals

• Use bootstrap CIs for medians, trimmed means, or ratios (median ratio); prefer bias-corrected and accelerated (BCa) or percentile bootstrap with large resamples (≥10k). For heavy tails, bootstrap the trimmed mean (e.g., 10–20% trimming) to reduce influence of outliers.

Practical recipe

1. Visualize (ECDFs, violin, boxplots on raw & log scales).
2. Decide target estimand: mean (sensitive), median, trimmed mean, or ratio.
3. If multiplicative effects: log-transform and analyze means; report geometric mean ratio with CI.
4. If robust location: estimate median or 20% trimmed mean and compute BCa bootstrap CI; test via permutation on that statistic.
5. Use permutation test for p-values on chosen statistic if distributional assumptions are doubtful.
6. As sensitivity, run both transformed-mean and robust-median analyses; report both.

Effect sizes & reporting

• Report the estimand and units: difference in medians (with bootstrap CI), ratio of geometric means (with CI), or trimmed-mean difference.
• Avoid raw Cohen’s d on skewed data; prefer Cliff’s Delta (probability one obs > other) or Hodges–Lehmann estimator (median of pairwise differences) with CI.
• Always report: estimand, point estimate, CI, p-value (test used), sample sizes, and visualization. Mention sensitivity analyses (transform, trimmed, rank) and robustness to outliers.

Why this approach

• Combines interpretability (median/ratio), robustness (trim/boot), and valid inference without relying on normality assumptions (permutation/boot).

Approach: implement a LORD-style online FDR controller using a geometric weighting sequence gamma_t = (1 - d) * d^{t-1} (0 < d < 1). Using this exponentially decaying gamma gives an O(1) recursive update for the running “reward” term, so each p-value is processed in amortized O(1) time and O(1) memory. We maintain:

• t: current test index (1-based)
• V: b * sum_{j in rejections} gamma_{t-j} (updated recursively)
• gamma_t = (1-d) * d^{t-1} used to compute alpha_t = gamma_t * w0 + V

Assumptions:

• p-values are independent or meet LORD assumptions for FDR control.
• We set initial wealth w0 and reward b such that w0 + b <= q (common choices w0 = q/2, b = q/2).
• Parameter d controls decay (e.g., 0.95). Smaller d concentrates alpha earlier.

Code (LORD with geometric gamma):

class OnlineFDR:
    """
    LORD-like online FDR controller with geometric gamma:
    gamma_t = (1-d) * d^(t-1)
    alpha_t = gamma_t * w0 + V_t
    V_{t+1} = d * V_t + b * (1-d) * I(reject at t)
    add_pvalue(p) -> True if reject, else False
    """
    def __init__(self, q, w0=None, b=None, d=0.95):
        if not (0 < q < 1):
            raise ValueError("q must be in (0,1)")
        if not (0 < d < 1):
            raise ValueError("d must be in (0,1)")
        self.q = q
        self.d = d
        # default: split initial wealth
        self.w0 = q/2 if w0 is None else w0
        self.b = q/2 if b is None else b
        if self.w0 < 0 or self.b < 0 or (self.w0 + self.b) > q:
            raise ValueError("Require 0 <= w0,b and w0+b <= q")
        self.t = 0
        self.V = 0.0  # running reward term: b * sum_{rejections} gamma_{t - j}
        # Precompute (1-d) to avoid repeating
        self.one_minus_d = 1.0 - d

    def add_pvalue(self, p):
        """
        Process next p-value and return True if rejected (discover), False otherwise.
        Amortized O(1) time, O(1) memory.
        """
        if not (0 <= p <= 1):
            raise ValueError("p must be in [0,1]")
        self.t += 1
        # compute current gamma_t
        gamma_t = self.one_minus_d * (self.d ** (self.t - 1))
        # alpha_t per LORD formula
        alpha_t = gamma_t * self.w0 + self.V
        reject = p <= alpha_t
        # update V for next time: V_{t+1} = d * V_t + b * gamma_1 * I(reject at t)
        # gamma_1 = (1-d)
        if reject:
            self.V = self.d * self.V + self.b * self.one_minus_d
        else:
            self.V = self.d * self.V
        return bool(reject)

Key points:

• Using geometric gamma yields V_{t+1} = d * V_t + b*(1-d)*I(reject), so updates are constant time.
• alpha_t is adaptive: early tests get higher base allocation via gamma_t*w0; past rejections add to V.
• Time complexity: O(1) per p-value. Space: O(1).
• Edge cases: choose d near 1 to spread alpha slowly; d small concentrates alpha early. Validate w0,b choices to respect overall q budget.
• Statistical caveat: theoretical FDR guarantees depend on model assumptions (independence or other conditions required by LORD). Use simulated calibration if assumptions may be violated.

Direct answer
Quantization edge cases come from three interacting error sources: representational error (a continuous value snapped to the nearest representable discrete level always loses some precision), saturation (a value outside the calibrated range clips to the boundary instead of scaling correctly), and rounding propagation (small per-layer errors compound as they flow through a deep network, and can flip a borderline classification decision even when each individual layer's error looks tiny). Per-channel quantization (a separate scale factor per output channel) versus per-tensor quantization (one scale factor for the whole weight tensor) matters because a single outlier-magnitude channel forces per-tensor quantization to use a wide scale that wastes precision on every OTHER channel, while per-channel isolates the damage to just the outlier's own channel.

Structured elaboration

Representational error. Quantizing to INT8 (8-bit integer, 256 representable levels) maps a continuous range onto a small fixed set of discrete steps; the maximum per-value error is half the step size (scale / 2), which is unavoidable at a given bit width and is why the SCALE choice, not the quantization algorithm, is the primary lever for controlling error.

Saturation. The scale factor is derived from a calibration pass (observing typical activation/weight ranges on representative data). Any RUNTIME value outside that calibrated range does not error, it clips silently to the minimum or maximum representable code, which means an out-of-distribution input at inference (a value 10x larger than anything seen during calibration) is not merely imprecise, it is truncated to the boundary value with no signal that clipping occurred.

Rounding propagation and misclassification. Each quantized layer's output error becomes the NEXT layer's input error, and while individual layer errors are small, they do not necessarily cancel, in the worst case they compound. The practical failure mode for classification is not a globally degraded accuracy number, it is a small subset of BORDERLINE-confidence examples (already close to a decision boundary in the full-precision model) flipping to the wrong class, while high-confidence examples are essentially unaffected. This is why aggregate accuracy alone under-reports the risk: a model can show a small overall accuracy drop while a specific, high-value borderline segment (e.g. fraud cases near the decision threshold) degrades much more.

Per-channel versus per-tensor. Per-tensor quantization computes ONE scale from the single largest-magnitude value across the entire weight tensor. If one output channel legitimately has much larger weight magnitudes than the rest (common for a channel tied to a rare-but-important feature), that one channel's magnitude sets a wide scale that then under-resolves every other, well-behaved channel. Per-channel quantization computes a separate scale per output channel, so the outlier channel's wide scale stays isolated to itself.

Outlier handling during calibration. Beyond per-channel scaling, calibration itself needs an explicit policy for outliers: using the raw min/max (sensitive to a single extreme value) versus a percentile-based range (e.g. clipping calibration itself to the 99.9th percentile, deliberately accepting some saturation on the rarest values in exchange for better resolution on the bulk of the distribution) is a real design trade-off, not a default to leave unexamined.

Experiments to detect quantization-induced failures, and how to mitigate them. Two experiments cover the failure modes above: (1) a per-channel-versus-per-tensor error comparison, exactly like the worked example below, run against the model's REAL weight tensors rather than a synthetic example, to detect whether any channel's magnitude is wide enough to be dragging per-tensor precision down for the rest; and (2) a targeted evaluation slice restricted to examples near the full-precision model's decision boundary, to detect misclassification risk that an aggregate accuracy number hides. The corresponding mitigations are directly paired with what each experiment detects: switch the affected layers from per-tensor to per-channel quantization if experiment 1 shows a wide gap, switch calibration from raw min/max to a percentile-clipped range if outliers are driving saturation, and, for models where post-training quantization's borderline-example degradation is still unacceptable after both of those, fall back to quantization-aware training (training with simulated quantization in the loop, letting the model adapt to the rounding it will actually experience at inference), which is a substantially larger intervention and should be reserved for the case where the two lighter mitigations are proven, via experiment 2's evaluation slice, not to be enough.

Worked example (executed): per-tensor versus per-channel error, and a saturation case

import numpy as np
rng = np.random.default_rng(0)
W = rng.normal(0, 0.05, size=(4, 16))
W[1, 3] = 2.0  # one outlier weight in channel 1

def quantize(x, scale):
    q = np.clip(np.round(x / scale), -127, 127)
    return q * scale

def per_tensor(W):
    scale = np.max(np.abs(W)) / 127.0
    return quantize(W, scale)

def per_channel(W):
    scales = np.max(np.abs(W), axis=1, keepdims=True) / 127.0
    return quantize(W, scales)

dq_t, dq_c = per_tensor(W), per_channel(W)
err_t = np.mean(np.abs(W - dq_t), axis=1)
err_c = np.mean(np.abs(W - dq_c), axis=1)
for ch in range(4):
    print(ch, err_t[ch], err_c[ch], err_t[ch] / max(err_c[ch], 1e-12))

# saturation
calibration_max, runtime_outlier = 0.5, 5.0
scale = calibration_max / 127.0
q = np.clip(np.round(runtime_outlier / scale), -127, 127)
print(q, q * scale, abs(q * scale - runtime_outlier) / runtime_outlier)

Executed output: mean absolute quantization error per channel (per-tensor vs per-channel): channel 0: 0.003189 vs 0.000149 (21.4x worse under per-tensor); channel 1 (the outlier): 0.003659 vs 0.003659 (1.0x, identical, since per-channel cannot do better on the outlier itself); channel 2: 0.004101 vs 0.000179 (23.0x); channel 3: 0.003630 vs 0.000173 (21.0x). The minimum ratio across the three healthy channels is 21.0x, confirming per-tensor quantization measurably degrades every channel that is not the outlier. The saturation test: a runtime activation of 5.0 against a calibration range of 0.5 clips to quantized code 127, dequantizing to 0.5, a 90% relative error, with no error raised.

Trade-offs & pitfalls
The most common mistake is evaluating quantization quality only via a single aggregate accuracy metric on a general test set; because rounding-propagation failures concentrate on already-borderline examples, a targeted evaluation slice (examples near the full-precision model's decision boundary, and any known high-value segment) will show a failure that the aggregate metric hides. A second pitfall is choosing per-channel quantization everywhere by default without checking its cost: per-channel scales require more metadata (one scale per channel instead of one per tensor) and, on some hardware/kernel implementations, cannot be as efficiently fused into the matrix-multiply, so the choice is a genuine accuracy-versus-throughput trade-off, not a strict improvement. Third, silent saturation is easy to miss entirely in testing if the test suite only exercises in-distribution calibration-range values; an explicit out-of-calibration-range test case (as in the worked example) is necessary specifically because saturation produces no exception, only a quiet, potentially large error.

Kubernetes provides several autoscaling tools useful for ML inference; choosing and combining them depends on workload patterns (latency-sensitive vs batch).

Key autoscalers:

• Horizontal Pod Autoscaler (HPA): scales replicas based on CPU/memory or custom metrics (QPS (queries per second), latency, GPU utilization). Good for handling increases in concurrent requests. Works well with stateless model servers.
• Vertical Pod Autoscaler (VPA): adjusts container requests/limits (CPU/memory) to match observed usage. Helpful when a single replica needs more resources (heavy single-threaded model), but causes restarts when resizing - so less ideal for strict SLAs.
• Cluster Autoscaler (CA): adds/removes nodes when pods are unschedulable or nodes are underutilized. Essential when HPA increases replicas beyond node capacity (e.g., GPU nodes).
• Custom metrics-based autoscaling: feed application-specific signals (request latency P95, queue length, GPU memory pressure, model cold-start counts) into HPA via Metrics API or KEDA for event-driven scaling (Kafka, Pub/Sub).

Trade-offs:

• Latency-sensitive (real-time inference): prioritize pre-warmed capacity and conservative scaling thresholds to avoid cold starts. Use HPA with low scale-up latency, keep a minimum replica count, combine with VPA (for baseline sizing) and CA (to ensure node capacity). Prefer fast autoscaling paths (scale-up by replicas rather than resizing).
• Batch/jobs: tolerate higher startup latency; scale aggressively to minimize job completion time, rely more on CA to add nodes and HPA/KEDA to spawn workers. VPA can optimize per-pod resource efficiency.

Handling cold starts and warm pools:

• Keep a minimum replica floor (HPA minReplicas) tuned to baseline traffic.
• Warm pools / prewarmed pods: maintain a small set of idle but ready containers (provisioned with model loaded) or use pod “standby” pools. Implement readiness probes that mark pod ready only after model init.
• Use cached model artifacts (local SSD, memory-mapped weights) and lightweight model variants (smaller quantized models) for fast startup.
• Use fast node types (avoid spot/preemptible for critical latency) and use CA warm node pools (keep a few nodes always available).
• Leverage request buffering/queue with autoscaler on queue length (KEDA) to smooth bursts.

Best practices:

• Measure end-to-end P99/P95 latency and use those as autoscaling signals.
• Combine HPA (reactive), VPA (right-sizing offline or during low-traffic windows), and CA (ensure capacity) while enforcing min replicas/nodes for SLAs.
• Load-test scaling behaviors and tune cooldowns, stabilization windows, and scale increments to balance cost vs latency.

Also covers (folded from merged near-duplicates): bb122f13 folds the horizontal-vs-vertical-scaling framing and cost/cold-start trade-off; 87f81511 folds the DS-voice GPU-bound-pod autoscaling specifics (custom metrics, scheduler considerations).

First split the problem into two buckets

1. True optimization issues: the model is mathematically harder to train at larger scale.
2. Distributed systems issues: the parallel run is changing the effective training behavior.

Things I would investigate

• Effective batch size and learning rate scaling. A bigger world size often means a larger batch, which can require LR tuning.
• Data sharding. I would verify that each sample is seen once, not duplicated or skipped.
• Gradient synchronization. I would check that all ranks contribute the same gradients and that there are no stale or dropped updates.
• Precision and numerics. Mixed precision can introduce overflow, underflow, or different rounding behavior at scale.
• Stateful layers like batch normalization, which can behave differently across replicas.

How I would separate them
I would compare small-scale and large-scale runs with the same global batch and fixed seeds, then inspect gradient norms, loss curves, and sample hashes.

Worked example
If 8 workers train with a batch of 256 each, the global batch becomes 2048. If loss gets noisier only after that change, it may be an optimization tuning problem. If the run only degrades when messages cross nodes, it is more likely a communication or synchronization problem.

That distinction tells me whether to tune the optimizer or fix the distributed stack.

Direct answer

A bounded queue (fixed maxsize) applies backpressure automatically: once it is full, put() blocks (threading) or suspends (await put, asyncio) until a consumer frees a slot, so a fast producer is throttled to the consumers' pace without any extra code. queue.Queue gives you this with an internal threading.Condition; asyncio.Queue gives you the same shape with await instead of blocking waits. The wake-up semantics differ by mechanism: a raw condition variable needs you to choose notify() (wake exactly one waiter) versus notify_all() (wake everyone, most of whom will just recheck and go back to waiting), while asyncio.Queue hides that choice entirely, await put/await get suspend and resume the right coroutine without you managing wake-ups by hand.

Structured elaboration

Threading version: queue.Queue(maxsize=N) is backed by a Lock plus two Conditions internally (not-full, not-empty). put() blocks on the not-full condition when the queue is at capacity; get() blocks on not-empty when it is empty. You do not touch the condition variables yourself, Queue already calls notify() correctly.

asyncio version: a coroutine is a function defined with async def that can pause mid-execution, at an await, and let other coroutines run, then resume later exactly where it left off; the event loop is the scheduler that decides which paused coroutine gets to run next. asyncio.Queue(maxsize=N) has the same contract as the threading version, but await q.put(item) suspends the coroutine (yielding control to the event loop, not blocking an OS thread) when full, and await q.get() suspends when empty. Concurrency comes from running many producer/consumer coroutines as tasks on one event loop instead of many OS threads.

Graceful shutdown, both versions: signal producers to stop (an Event), let running producers finish their current item, then drain the queue (q.join(), which waits until every put item has had a matching task_done()), then push one sentinel value per consumer so each one exits its loop cleanly. q.join() hanging is almost always a sign that a consumer path skipped task_done(), most often via an exception on the processing line before it reached the task_done() call.

The harder variant, without queue.Queue/asyncio.Queue: you build the bounded buffer yourself from a Lock plus two Condition objects sharing that lock, one for "not full", one for "not empty." This is exactly what queue.Queue does internally, made explicit:

import threading, collections

class BoundedBuffer:
    def __init__(self, maxsize):
        self.maxsize = maxsize
        self.items = collections.deque()
        self.lock = threading.Lock()
        self.not_full = threading.Condition(self.lock)
        self.not_empty = threading.Condition(self.lock)

    def put(self, item):
        with self.not_full:
            while len(self.items) >= self.maxsize:
                self.not_full.wait()      # releases the lock, sleeps until notified
            self.items.append(item)
            self.not_empty.notify()       # wake exactly one waiting consumer

    def get(self):
        with self.not_empty:
            while not self.items:
                self.not_empty.wait()
            item = self.items.popleft()
            self.not_full.notify()        # wake exactly one waiting producer
            return item

notify() (one waiter) is correct here because exactly one slot opened up, waking everyone with notify_all() would just cause the other waiters to recheck their while condition and go back to sleep, which is harmless but wasteful (a "thundering herd" of wasted wake-ups). notify_all() earns its keep when a single event can satisfy many different waiters at once, which is not the case for a single-slot state change like this.

Worked example

asyncio.Queue-based pipeline with graceful shutdown, verified on CPython 3.12 (produced and consumed counts always match, since q.join() guarantees every item was drained before sentinels are sent; the exact counts vary run to run because they depend on real-time scheduling):

import asyncio, random

SENTINEL = object()

async def producer(pid, q, stop_event, rng):
    i = 0
    while not stop_event.is_set():
        await q.put((pid, i))          # blocks here once the queue is full
        i += 1
        await asyncio.sleep(rng.random() * 0.02)

async def consumer(cid, q, processed):
    while True:
        item = await q.get()
        if item is SENTINEL:
            q.task_done()
            break
        processed.append((cid, item))
        q.task_done()

async def run_pipeline(n_producers=2, n_consumers=3, maxsize=5, run_time=0.3, seed=7):
    rng = random.Random(seed)
    q = asyncio.Queue(maxsize=maxsize)
    stop_event = asyncio.Event()
    processed = []

    producers = [asyncio.create_task(producer(i, q, stop_event, rng)) for i in range(n_producers)]
    consumers = [asyncio.create_task(consumer(i, q, processed)) for i in range(n_consumers)]

    await asyncio.sleep(run_time)
    stop_event.set()
    await asyncio.gather(*producers)
    await q.join()                          # wait until every produced item is processed
    for _ in consumers:
        await q.put(SENTINEL)
    await asyncio.gather(*consumers)
    assert len(processed) == sum(1 for _ in processed)  # every produced item made it through

asyncio.run(run_pipeline())

A representative run reported "total produced: 62, total consumed: 62"; the invariant produced == consumed holds on every run because q.join() will not return until all items are drained, the specific count itself is not reproducible (it depends on wall-clock scheduling) and is not the claim being made here.

Trade-offs & pitfalls

• Complexity: $O(1)$ per item for queue operations; space is bounded by maxsize for the queue itself, plus $O(k)$ for the $k$ in-flight producer/consumer tasks.
• Edge case: if a consumer raises an exception before calling task_done(), q.join() hangs forever; wrap the processing body in try/finally so task_done() always runs.
• Edge case: sending fewer sentinels than there are consumers leaves some consumers blocked on get() forever; one sentinel per consumer is required, not one total.
• Choosing notify() when the change actually affects multiple waiters (rare for a single bounded queue, more common in custom condition-based coordination) silently starves the other waiters; if you are not certain only one waiter's condition changed, notify_all() is the safer default even though it wakes more coroutines/threads than strictly necessary.
• The asyncio version scales to many more concurrent producers/consumers than the threading version for the same memory budget, since coroutines are far cheaper than OS threads, but it only helps if the "work" inside producer/consumer is itself non-blocking; a CPU-bound consumer inside an asyncio pipeline will stall the whole event loop exactly like any other CPU-bound coroutine.

Direct answer

When legal or compliance flags a possible regulatory problem on something about to ship, that flag is new information, not an attack on the project. The first move is to separate the specific risk from the whole feature: find out exactly what triggers the concern, then look for a way to ship everything outside that blast radius (the specific data, users, or markets the flagged concern actually touches) while the risky piece gets handled properly. Treating the flag as either a full block to fight or a formality to route around are both weak answers; the senior move is to make the freeze as small as the actual risk.

Structured elaboration

1. Turn the flag into a scoped, written finding

Ask for the specific clause or regulation, the specific data flow or behavior it applies to, and which markets or user segments are affected. A flag that sounds like 'this violates a regulation' often narrows down to 'this one data field, in these two markets.' Until that scoping happens, nobody can reason about mitigation, they can only argue about the abstract freeze.

2. Sort what's actually blocked from what's just slow

Once scoped, most flags fall into three buckets: genuinely unsafe to ship anywhere (rare, but real, treat it as a hard stop); unsafe in specific markets or for specific data (the common case, often scoped out with a flag or market-level rule); or unsafe as currently designed but fixable with a smaller change than a full freeze (needs a scoped rework, not a blanket delay).

3. Bring a mitigation, not just a constraint

Offer a concrete option: disable the flagged behavior for the affected markets, gate it behind a feature flag (a toggle that turns a piece of functionality on or off without a new deployment), or ship a version that omits the specific data flow while the rest proceeds. This turns the conversation from 'can we go or not' into 'does this mitigation satisfy the concern,' which moves much faster.

4. Get joint, written sign-off before proceeding

Both the business owner and compliance need to agree in writing on what shipped, what did not, the remaining risk, and who owns closing it. This protects everyone if the interpretation is questioned later and prevents the same argument from recurring next release.

5. If a real freeze can't be avoided, negotiate the timeline explicitly

Sometimes there is no safe scoped path and the freeze has to hold for the affected piece. Here the negotiation shifts to: what's the minimum change needed to clear the concern, who is assigned to it, and can the review be fast-tracked with a dedicated reviewer instead of sitting in a general queue. A freeze with a committed, shrinking timeline is a very different conversation from an open-ended one.

Worked example

A team is about to ship a feature that logs a new field for product analytics, and legal flags that collecting that field may violate a data-protection rule in one region. Scoping the flag shows the issue is narrow: one field, one region. Instead of freezing the whole release, the team ships everywhere else immediately, and for the flagged region ships the same feature with that one field's collection disabled behind a config switch. Legal signs off on the scoped version in writing. The team opens a follow-up item, with an owner and a target date, to redesign how that field is collected (for example, aggregating it instead of storing it per user), so the region isn't stuck without the feature indefinitely.

Trade-offs and pitfalls

• Treating every compliance flag as either a full block or a nuisance to route around is the most common mistake here; both extremes erode trust with the compliance function over time.
• Scoped mitigations (flags, market gating, field exclusions) are good short-term tools but can quietly become permanent if nobody owns the follow-up fix. The sign-off should name an owner and a date, not just describe a workaround.
• Escalating past compliance to force a ship date, without addressing the underlying concern, tends to resurface later as a bigger problem: a real violation or a regulator inquiry. Speed gained by skipping the process rarely survives contact with the risk it was protecting against.
• The strongest signal of seniority isn't how fast the team got to yes, it's whether the final decision is something both sides would still defend the same way months later.

Direct answer

For heavy-tailed data, reach for the trimmed mean (the mean after dropping a fixed percentage from each tail), the winsorized mean (capping extreme values at a percentile rather than dropping them), and the median absolute deviation, or MAD (a robust measure of spread built from the median rather than the mean). Each protects against exactly what the plain mean and standard deviation are vulnerable to: a small number of extreme values dominating the summary.

What each protects against, specifically

The plain mean weights every value equally in the sum, so one extreme transaction can shift it substantially; a trimmed mean removes the most extreme values entirely before averaging, which is simple but throws away potentially real information at the tails. A winsorized mean keeps every observation but caps the extreme ones at a threshold, preserving the row count while limiting any single point's influence, a useful middle ground when you don't want to discard rows outright. MAD replaces both the center (the median) and the spread calculation with something that isn't sensitive to a handful of extreme values the way variance is, since the median of a bunch of moderate values plus one extreme one barely moves.

Worked example

Twelve values representing transaction sizes: mostly clustered between 11 and 15, with one value of 250. The plain mean comes out to 32.75, badly distorted by that single value; the median is 13.0, essentially unaffected. A 10% trimmed mean comes out to 13.2, very close to the median, having simply excluded the most extreme values from each end before averaging. The winsorized version (capping the top and bottom 10% at the next-most-extreme retained value) gives a mean of 13.25, similarly close to the median while still using every row. The MAD, scaled to be comparable to a standard deviation on normal data, comes out to about 1.48, a genuinely honest measure of the typical spread among the twelve values, in stark contrast to what the standard deviation (dominated by that single 250) would report.

Trade-offs and pitfalls

Trimming and winsorizing require choosing a percentage to trim or cap at, which is itself a judgment call, not free of assumptions; too aggressive a trim can discard real, meaningful extreme values along with true anomalies. Always report which robust method you used and at what threshold, since "robust mean" without that detail hides a real analytical choice.

Direct answer. Scaling up GPU count or enabling mixed precision changes several things simultaneously (effective batch size, gradient synchronization behavior, numeric precision), so a training run that was stable before and diverges after needs a plan that isolates ONE variable at a time rather than reverting the whole change and guessing.

Three terms this depends on. A replica is one full copy of the model living on one GPU. DDP (DistributedDataParallel) is the synchronous data-parallel wrapper that runs one process per GPU and keeps the replicas in step. All-reduce is the collective operation that combines a tensor across every replica and hands every replica back the same combined result, which is how the per-replica gradients become one shared gradient each step.

Plausible causes, and what to test first. The single highest-prior cause when scaling GPU count is learning-rate scaling: if the effective batch size grows with GPU count but the learning rate doesn't, or is scaled incorrectly (linear scaling isn't always the right rule for every optimizer), training can become unstable purely from that mismatch, and it's the cheapest hypothesis to test. Worked with real numbers so "adjusted for the new batch size" is computable: a stable baseline of 8 GPUs at 32 examples per device is an effective batch of 256 at LR 0.1. Scaling to 64 GPUs at the same 32 per device gives an effective batch of 2048, an 8x increase. The linear rule says LR = 0.1 x 2048/256 = 0.8. The square-root rule says LR = 0.1 x sqrt(8) = 0.283. If the job was launched at 0.8 and diverges, the diagnostic rerun is at 0.283, and the sanity rerun (which trades throughput for certainty) is at the original 0.1.

Next, check batch-norm statistics: if BatchNorm computes per-replica statistics independently rather than synchronizing across replicas, a network with BatchNorm can behave differently, and less stably, at high replica counts purely because each replica now sees a smaller effective batch for normalization purposes. Then check all-reduce precision (is gradient synchronization happening in a lower precision than intended) and synchronization/seed-order differences (are all replicas actually initialized identically before training starts).

For mixed precision (AMP) specifically. Gradient-accumulation interactions with AMP are a common culprit. The mechanics, since the ordering is the whole bug: loss scaling multiplies the loss by a large factor S before the backward pass so that small fp16 gradients do not underflow to zero, and the gradients must later be divided by S ("unscaled") before the optimizer step. With gradient accumulation over K micro-batches, the correct order is accumulate-then-unscale: keep every micro-batch's gradients scaled, sum all K of them, and unscale ONCE immediately before the optimizer step. The wrong order, unscale-then-accumulate, divides each micro-batch's gradients by S as it goes and then accumulates already-unscaled values, while the scaler still believes it owes the division, so the effective gradient magnitude ends up off by a factor tied to K and the scaler's overflow check runs against tensors that are no longer on the scale it expects. Confirm which order your framework's AMP integration actually uses before assuming the bug is elsewhere. Separately, some DDP jobs show gradient divergence where SOME workers produce NaN and others don't purely from AMP's dynamic loss scaling adjusting independently per worker if it isn't explicitly synchronized, worth checking whether your framework synchronizes the loss-scale factor across replicas or lets each pick its own.

What you would expect to observe if each cause were the real one. This is the half that turns the list into a diagnosis:

• Learning-rate / batch-size mismatch: divergence arrives within the first few hundred steps and arrives EARLIER the more GPUs you add, and the step at which it blows up moves monotonically as you sweep LR. Rerunning the same 64-GPU job at 0.283 (or at 0.1) with everything else untouched restores stability. Gradient norms grow smoothly over several steps before the blow-up rather than jumping in one step.
• BatchNorm computing per-replica statistics: dump each replica's BatchNorm running mean and running variance buffers and compare them across ranks. Synchronized normalization keeps them near-identical; the unsynchronized failure shows them visibly diverging, with the spread widening as per-device batch size falls. Crucially, the gradients themselves stay finite the whole time, so this looks like instability without any numeric red flag. It gets worse as you hold total batch fixed and add GPUs (shrinking per-device batch), and it disappears when you swap in synchronized BatchNorm or raise the per-device batch back up.
• All-reduce running in a lower precision than intended: checksum or norm each rank's gradient tensor immediately BEFORE the collective and immediately AFTER it. Before, the ranks legitimately differ, since each saw a different data shard. After, every rank must hold a bit-identical tensor. The tell is that post-collective gradients agree only to the first three or four significant digits rather than exactly, and that the disagreement grows with rank count, because more partial sums mean more accumulated rounding. Compare against a reference reduction performed in fp32 to size the error.
• Seed or initialization-order mismatch: hash every replica's parameters at step 0, before any data is seen. They must be identical. If rank 0's hash differs from rank 1's at step 0, the replicas never started from the same weights, and the shared averaged gradient is being applied to divergent parameter sets. The signature is that divergence appears essentially at step 1, not after a hundred healthy steps.
• AMP unscale-then-accumulate ordering: instrument the global gradient norm immediately before the optimizer step, and compare against the same total batch run with K=1 (no accumulation). The two should agree within numerical noise. The ordering bug shows up as a clean multiplicative discrepancy tied to K, a factor of K or 1/K rather than a vague drift, which is what makes it identifiable rather than merely suspicious.
• AMP loss-scale desync across workers: log the current loss-scale value per rank per step. Healthy runs show every rank holding the same value and stepping it in lockstep. The failure shows ranks holding different scales at the same step (one at 1024 while another is at 32768), SOME ranks reporting NaN while others report clean gradients, and the identity of the NaN-reporting ranks changing from run to run.

Why the effective batch size is the thread running through all of this. Scaling GPU count in synchronous data-parallel training multiplies the effective batch size by the number of new replicas, which is exactly what makes the learning-rate hypothesis the highest-prior one and why the arithmetic above matters. It also means that if the instability appears only past a specific scale rather than at every scale, that threshold is itself evidence: a genuine LR/batch mismatch only becomes destabilizing once the step size and the curvature disagree badly enough, whereas a hardware or environment fault would be present regardless of how many replicas are running.

Prioritized order to test. Order by what each check actually costs to run, then by prior probability, because two of these are nearly free and one of them is a precondition for trusting any of the others.

0. Free, so run these before spending a single GPU-hour on a rerun. Hash every replica's parameters at step 0: one hash per rank, zero training steps, and if the hashes disagree you are done, because none of the cross-rank comparisons below mean anything until the replicas provably start from the same weights. Under AMP, also log each rank's loss-scale value per step, which is one scalar per rank and settles the desync hypothesis outright. Neither of these is the most likely cause; they go first because they are cheaper than everything else and because they are the two that can invalidate the other measurements.
1. Learning-rate scaling, the highest-prior cause and the first hypothesis worth real GPU-hours: rerun the 64-GPU job at 0.283, and at the original 0.1 if that is still unstable. One short run, one variable, directly falsifiable.
2. AMP accumulate-then-unscale ordering, whenever gradient accumulation is on: compare the pre-optimizer global gradient norm against the same total batch run at K=1. One extra short run and one scalar, and the tell is a clean factor of K or 1/K rather than a drift.
3. BatchNorm synchronization: dump the running-mean and running-variance buffers on every rank and diff them. More instrumentation than anything above, still far less than the last item.
4. All-reduce precision, last: checksumming every rank's gradient tensor before and after the collective is the heaviest instrumentation in this list, and it is also the least common root cause for a symptom that appeared specifically at a scaling boundary.

Problem formalization:

• Environment: M services i∈{1..M}, T decision rounds. Each round t we allocate a vector of discrete resources a_t = (a_{t,1},...,a_{t,M}) from a feasible budget set A (e.g., ∑i c_i a{t,i} ≤ C, per-host caps).
• Reward: stochastic QoS reward r_t = ∑i R_i(a{t,i}, s_{t,i}) where s_{t,i} are unknown service states (load, SLO attainment); we observe rewards (possibly delayed/noisy) after allocation. Objective: maximize cumulative expected reward E[∑_{t=1}^T r_t] (or minimize regret vs best fixed allocation/policy).
• Unknowns: response curves R_i(·) (monotone, possibly concave), cross-service interactions (contention).

Why this is a combinatorial bandit:

• Each action is a combinatorial object (vector allocation under constraints) drawn from an exponential-size set A. Reward decomposes (possibly approximately) over arms (services) but choice couples via budget constraints. Observations may be semi-bandit (per-service feedback) or bandit (only aggregate). This matches combinatorial multi-armed bandits (CMAB).

Algorithmic solutions:

1. Approximate Combinatorial UCB (CMUCB):

• Maintain per-service estimates μ̂_i(a) or parametric models (e.g., diminishing returns curve). Compute optimistic estimates μ̂_i + bonus_i(t) and solve an offline knapsack/ILP to pick allocation maximizing sum of optimistic rewards subject to constraints. Use approximation-aware regret bounds: if offline oracle is α-approx, regret scales O((poly)·T/α). Use structured models (GLM, parametric) to reduce sample complexity.

2. Greedy with submodular objectives:

• If ∑i R_i(a{i}) is monotone submodular in discrete resource units, use greedy hill-climbing per round: iteratively allocate marginal unit with largest estimated marginal gain. With (1-1/e) approximation and confidence bonuses, this gives provable approximate regret. Practical variant: Thompson Sampling + greedy sampling to handle uncertainty.

3. Contextual/parametric bandits:

• Use contextual CMAB: include features (current load, queues). Fit online Bayesian/linear models and use LinUCB/Thompson Sampling over allocations with efficient combinatorial optimization via knapsack solvers.

4. Handling interactions:

• If strong cross-coupling, treat joint arms for small groups or use low-rank factorization of interaction matrix and learn latent factors.

Practical monitoring and risk controls:

• Safety constraints: enforce hard SLO-preserving actions (never drop below reserve allocations), and clipping of exploratory allocations.
• Conservative bootstrapping: begin with off-line A/B tests or simulation to initialize priors; use decaying exploration rates.
• Canary & rollback: stage policy in canary cluster with traffic shadowing; automatic rollback on SLO degradation beyond thresholds.
• Observability: per-service QoS, latencies, error rates, resource usage, and reward attribution with high-cardinality logs; compute regret/expected reward delta vs baseline in real-time.
• Alarm & intervention: require human-in-loop for high-impact allocation changes, throttle exploratory updates when variance or delayed feedback increases.
• Explainability: surface marginal gains used for each allocation decision to operators.

Trade-offs:

• Exact CMAB algorithms give theoretical guarantees but need efficient oracles; greedy/submodular methods scale well and are robust if submodularity approx holds. Contextual models reduce samples but require reliable features.

This design balances provable algorithms with practical safety: start with parametric/contextual CMUCB + greedy allocation, strong monitoring and safety gates, then iterate as more data reduces uncertainty.