DoorDash Machine Learning Engineer Interview Preparation Guide - Mid Level

Situation: You're comparing two groups on a skewed, heavy-tailed metric (time-on-site, RPU). Goal: get reliable inference and a meaningful effect-size.

When to transform

• Use a log (or Box–Cox) transform when multiplicative effects are plausible and you care about proportional changes. Log often stabilizes variance and makes means interpretable as geometric means. Check back-transformed interpretation and heteroskedasticity.

Rank-based tests

• Mann–Whitney U (Wilcoxon) tests stochastic dominance / shift in distribution location but does NOT strictly test medians when shapes differ. Use when you want a distribution-agnostic test and are willing to sacrifice interpretability of mean differences.

Permutation (randomization) tests

• Nonparametric, exact (conditional on observed values), and testable for mean or other statistics. Use permutation of group labels to test whether observed difference (mean, median, trimmed mean) could arise by chance. Works well with any test statistic and small-to-moderate samples.

Bootstrap confidence intervals

• Use bootstrap CIs for medians, trimmed means, or ratios (median ratio); prefer bias-corrected and accelerated (BCa) or percentile bootstrap with large resamples (≥10k). For heavy tails, bootstrap the trimmed mean (e.g., 10–20% trimming) to reduce influence of outliers.

Practical recipe

1. Visualize (ECDFs, violin, boxplots on raw & log scales).
2. Decide target estimand: mean (sensitive), median, trimmed mean, or ratio.
3. If multiplicative effects: log-transform and analyze means; report geometric mean ratio with CI.
4. If robust location: estimate median or 20% trimmed mean and compute BCa bootstrap CI; test via permutation on that statistic.
5. Use permutation test for p-values on chosen statistic if distributional assumptions are doubtful.
6. As sensitivity, run both transformed-mean and robust-median analyses; report both.

Effect sizes & reporting

• Report the estimand and units: difference in medians (with bootstrap CI), ratio of geometric means (with CI), or trimmed-mean difference.
• Avoid raw Cohen’s d on skewed data; prefer Cliff’s Delta (probability one obs > other) or Hodges–Lehmann estimator (median of pairwise differences) with CI.
• Always report: estimand, point estimate, CI, p-value (test used), sample sizes, and visualization. Mention sensitivity analyses (transform, trimmed, rank) and robustness to outliers.

Worked example (why this matters, not just definitions). Group A revenue-per-user in dollars = [12, 15, 14, 210, 13] (one whale customer at $210); Group B = [20, 22, 19, 21, 18]. The naive mean comparison is actively misleading here: mean(A) = 264/5 = $52.80, mean(B) = 100/5 = $20.00, so a plain Cohen's d style comparison says A is $32.80 HIGHER, entirely driven by the single $210 outlier. Now compute the two robust alternatives by hand. Cliff's Delta counts, across all 5x5=25 pairwise comparisons, how often A's value exceeds B's versus the reverse: only the 5 comparisons involving the $210 outlier have A>B, while all 20 remaining comparisons have A<B, giving Cliff's Delta = (5 − 20) / 25 = −0.6, meaning B is typically the larger value in a pairwise sense. The Hodges-Lehmann estimator is the median of all 25 pairwise differences (A_i − B_j); sorting those 25 differences (20 negative values from −10 down to −3, plus 5 large positive values from the outlier row) and taking the 13th of 25 gives a median difference of −6, i.e. the robust point estimate is that A's typical value is about $6 LOWER than B's, the opposite sign of the naive mean comparison. This is exactly why the recommendation above is to avoid raw Cohen's d on skewed data: the one outlier flips the naive comparison's sign, while both rank-based estimators correctly reflect that most of Group A sits below Group B.

Why this approach

• Combines interpretability (median/ratio), robustness (trim/boot), and valid inference without relying on normality assumptions (permutation/boot).

Direct answer
Quantization edge cases come from three interacting error sources: representational error (a continuous value snapped to the nearest representable discrete level always loses some precision), saturation (a value outside the calibrated range clips to the boundary instead of scaling correctly), and rounding propagation (small per-layer errors compound as they flow through a deep network, and can flip a borderline classification decision even when each individual layer's error looks tiny). Per-channel quantization (a separate scale factor per output channel) versus per-tensor quantization (one scale factor for the whole weight tensor) matters because a single outlier-magnitude channel forces per-tensor quantization to use a wide scale that wastes precision on every OTHER channel, while per-channel isolates the damage to just the outlier's own channel.

Structured elaboration

Representational error. Quantizing to INT8 (8-bit integer, 256 representable levels) maps a continuous range onto a small fixed set of discrete steps; the maximum per-value error is half the step size (scale / 2), which is unavoidable at a given bit width and is why the SCALE choice, not the quantization algorithm, is the primary lever for controlling error.

Saturation. The scale factor is derived from a calibration pass (observing typical activation/weight ranges on representative data). Any RUNTIME value outside that calibrated range does not error, it clips silently to the minimum or maximum representable code, which means an out-of-distribution input at inference (a value 10x larger than anything seen during calibration) is not merely imprecise, it is truncated to the boundary value with no signal that clipping occurred.

Rounding propagation and misclassification. Each quantized layer's output error becomes the NEXT layer's input error, and while individual layer errors are small, they do not necessarily cancel, in the worst case they compound. The practical failure mode for classification is not a globally degraded accuracy number, it is a small subset of BORDERLINE-confidence examples (already close to a decision boundary in the full-precision model) flipping to the wrong class, while high-confidence examples are essentially unaffected. This is why aggregate accuracy alone under-reports the risk: a model can show a small overall accuracy drop while a specific, high-value borderline segment (e.g. fraud cases near the decision threshold) degrades much more.

Per-channel versus per-tensor. Per-tensor quantization computes ONE scale from the single largest-magnitude value across the entire weight tensor. If one output channel legitimately has much larger weight magnitudes than the rest (common for a channel tied to a rare-but-important feature), that one channel's magnitude sets a wide scale that then under-resolves every other, well-behaved channel. Per-channel quantization computes a separate scale per output channel, so the outlier channel's wide scale stays isolated to itself.

Outlier handling during calibration. Beyond per-channel scaling, calibration itself needs an explicit policy for outliers: using the raw min/max (sensitive to a single extreme value) versus a percentile-based range (e.g. clipping calibration itself to the 99.9th percentile, deliberately accepting some saturation on the rarest values in exchange for better resolution on the bulk of the distribution) is a real design trade-off, not a default to leave unexamined.

Experiments to detect quantization-induced failures, and how to mitigate them. Two experiments cover the failure modes above: (1) a per-channel-versus-per-tensor error comparison, exactly like the worked example below, run against the model's REAL weight tensors rather than a synthetic example, to detect whether any channel's magnitude is wide enough to be dragging per-tensor precision down for the rest; and (2) a targeted evaluation slice restricted to examples near the full-precision model's decision boundary, to detect misclassification risk that an aggregate accuracy number hides. The corresponding mitigations are directly paired with what each experiment detects: switch the affected layers from per-tensor to per-channel quantization if experiment 1 shows a wide gap, switch calibration from raw min/max to a percentile-clipped range if outliers are driving saturation, and, for models where post-training quantization's borderline-example degradation is still unacceptable after both of those, fall back to quantization-aware training (training with simulated quantization in the loop, letting the model adapt to the rounding it will actually experience at inference), which is a substantially larger intervention and should be reserved for the case where the two lighter mitigations are proven, via experiment 2's evaluation slice, not to be enough.

Worked example (executed): per-tensor versus per-channel error, and a saturation case

import numpy as np
rng = np.random.default_rng(0)
W = rng.normal(0, 0.05, size=(4, 16))
W[1, 3] = 2.0  # one outlier weight in channel 1

def quantize(x, scale):
    q = np.clip(np.round(x / scale), -127, 127)
    return q * scale

def per_tensor(W):
    scale = np.max(np.abs(W)) / 127.0
    return quantize(W, scale)

def per_channel(W):
    scales = np.max(np.abs(W), axis=1, keepdims=True) / 127.0
    return quantize(W, scales)

dq_t, dq_c = per_tensor(W), per_channel(W)
err_t = np.mean(np.abs(W - dq_t), axis=1)
err_c = np.mean(np.abs(W - dq_c), axis=1)
for ch in range(4):
    print(ch, err_t[ch], err_c[ch], err_t[ch] / max(err_c[ch], 1e-12))

# saturation
calibration_max, runtime_outlier = 0.5, 5.0
scale = calibration_max / 127.0
q = np.clip(np.round(runtime_outlier / scale), -127, 127)
print(q, q * scale, abs(q * scale - runtime_outlier) / runtime_outlier)

Executed output: mean absolute quantization error per channel (per-tensor vs per-channel): channel 0: 0.003189 vs 0.000149 (21.4x worse under per-tensor); channel 1 (the outlier): 0.003659 vs 0.003659 (1.0x, identical, since per-channel cannot do better on the outlier itself); channel 2: 0.004101 vs 0.000179 (23.0x); channel 3: 0.003630 vs 0.000173 (21.0x). The minimum ratio across the three healthy channels is 21.0x, confirming per-tensor quantization measurably degrades every channel that is not the outlier. The saturation test: a runtime activation of 5.0 against a calibration range of 0.5 clips to quantized code 127, dequantizing to 0.5, a 90% relative error, with no error raised.

Trade-offs & pitfalls
The most common mistake is evaluating quantization quality only via a single aggregate accuracy metric on a general test set; because rounding-propagation failures concentrate on already-borderline examples, a targeted evaluation slice (examples near the full-precision model's decision boundary, and any known high-value segment) will show a failure that the aggregate metric hides. A second pitfall is choosing per-channel quantization everywhere by default without checking its cost: per-channel scales require more metadata (one scale per channel instead of one per tensor) and, on some hardware/kernel implementations, cannot be as efficiently fused into the matrix-multiply, so the choice is a genuine accuracy-versus-throughput trade-off, not a strict improvement. Third, silent saturation is easy to miss entirely in testing if the test suite only exercises in-distribution calibration-range values; an explicit out-of-calibration-range test case (as in the worked example) is necessary specifically because saturation produces no exception, only a quiet, potentially large error.

Kubernetes provides several autoscaling tools useful for ML inference; choosing and combining them depends on workload patterns (latency-sensitive vs batch).

Key autoscalers:

• Horizontal Pod Autoscaler (HPA): scales replicas based on CPU/memory or custom metrics (QPS (queries per second), latency, GPU utilization). Good for handling increases in concurrent requests. Works well with stateless model servers.
• Vertical Pod Autoscaler (VPA): adjusts container requests/limits (CPU/memory) to match observed usage. Helpful when a single replica needs more resources (heavy single-threaded model), but causes restarts when resizing - so less ideal for strict SLAs.
• Cluster Autoscaler (CA): adds/removes nodes when pods are unschedulable or nodes are underutilized. Essential when HPA increases replicas beyond node capacity (e.g., GPU nodes).
• Custom metrics-based autoscaling: feed application-specific signals (request latency P95, queue length, GPU memory pressure, model cold-start counts) into HPA via Metrics API or KEDA for event-driven scaling (Kafka, Pub/Sub).

Trade-offs:

• Latency-sensitive (real-time inference): prioritize pre-warmed capacity and conservative scaling thresholds to avoid cold starts. Use HPA with low scale-up latency, keep a minimum replica count, combine with VPA (for baseline sizing) and CA (to ensure node capacity). Prefer fast autoscaling paths (scale-up by replicas rather than resizing).
• Batch/jobs: tolerate higher startup latency; scale aggressively to minimize job completion time, rely more on CA to add nodes and HPA/KEDA to spawn workers. VPA can optimize per-pod resource efficiency.

Handling cold starts and warm pools:

• Keep a minimum replica floor (HPA minReplicas) tuned to baseline traffic.
• Warm pools / prewarmed pods: maintain a small set of idle but ready containers (provisioned with model loaded) or use pod “standby” pools. Implement readiness probes that mark pod ready only after model init.
• Use cached model artifacts (local SSD, memory-mapped weights) and lightweight model variants (smaller quantized models) for fast startup.
• Use fast node types (avoid spot/preemptible for critical latency) and use CA warm node pools (keep a few nodes always available).
• Leverage request buffering/queue with autoscaler on queue length (KEDA) to smooth bursts.

Best practices:

• Measure end-to-end P99/P95 latency and use those as autoscaling signals.
• Combine HPA (reactive), VPA (right-sizing offline or during low-traffic windows), and CA (ensure capacity) while enforcing min replicas/nodes for SLAs.
• Load-test scaling behaviors and tune cooldowns, stabilization windows, and scale increments to balance cost vs latency.

Direct answer
Build the rubric around whether the writing actually works for its reader: does it state its point clearly, fit the audience it's for, give the reader something to do with it, and use a tone appropriate to its purpose, then justify each criterion by what a failure on it costs the reader.

Structured elaboration
Proposed criteria, with the reasoning for each:

1. Clear main point: can a reader state the document's core message in one sentence after reading it? Justification: this is the single biggest failure mode in unclear writing, so it anchors the rubric.
2. Appropriate structure: does the important information come early, with supporting detail after, rather than requiring the reader to read to the end to find the point?
3. Audience fit: is the level of jargon and assumed background knowledge appropriate for who's actually going to read this, rather than written for the author's own level of familiarity?
4. Concision: is there padding, hedging, or restatement that could be cut without losing meaning?
5. Actionable next step: if the document implies an action or a decision, is that action stated explicitly, rather than left for the reader to infer?
6. Precision: are claims specific and checkable, or do vague quantifiers stand in for actual numbers where numbers were available?
7. Tone fit: is the tone appropriate to the stakes and relationship, neither over-casual for a high-stakes audience nor needlessly formal for a quick internal note?
8. Honesty about caveats: does the document surface real limitations or risks, rather than smoothing them over to look cleaner?

Worked example
Applying this to a short vendor-status email: main point ("vendor is delayed two weeks") is clear in the first sentence; structure is fine; audience fit is appropriate (no unnecessary jargon for a business reader); concision is good at three sentences; the next step (approve a revised deadline) is explicitly stated; precision holds (a specific date is given, not "soon"); tone is appropriately direct without being alarmist; and the caveat (a small risk of a further one-week slip) is honestly included rather than hidden. That's 8 for 8, which is a genuinely well-written status update by this rubric.

Trade-offs and pitfalls

• A rubric with too many criteria becomes tedious to apply consistently; six to eight, as here, is usually enough to catch the failure modes that matter most without turning review into a lengthy checklist exercise.
• Some criteria trade off against each other (concision versus caveats, for instance); the rubric should make clear that cutting a genuine caveat to satisfy concision is a failure, not a win, on this rubric.
• A rubric like this works best as a discussion tool during review, not as a rigid pass/fail gate; a document can reasonably fail one criterion (say, tone) for a good reason specific to its context.

Direct answer
At high incident volume, right-sizing postmortem effort means reviewing incidents proportionally to their severity and learning value rather than giving every incident the same heavyweight treatment, since a full deep-dive on every minor blip both burns out reviewers and dilutes attention from the incidents that actually deserve it.

Structured elaboration

• Tier the review depth by severity and novelty. High-severity or novel-pattern incidents get the full treatment: timeline reconstruction, root cause and contributing factors, cross-team facilitation. Low-severity, well-understood, or clearly one-off incidents get a much lighter 'mini' review: a short written summary with a root cause and, if warranted, one action item, no meeting required.
• Rotate reviewers rather than relying on the same few people. Concentrating review responsibility on a small group both burns them out and creates a bottleneck; distributing it (with a shared template and light training) keeps quality consistent while reducing individual load.
• Automate triage where the pattern is well understood. If a category of incident has occurred many times with the same known cause, an automated or templated mini-postmortem that flags it as a known, tracked pattern (rather than requiring fresh analysis every time) frees up reviewer time for genuinely novel incidents.
• Track a pattern-level view, not just per-incident. A large volume of small, similar incidents is itself a signal worth its own dedicated (heavier) review, even if none of them individually crossed the severity threshold for a full postmortem, since the aggregate pattern is often more informative than any single instance.
• Measure whether this is actually preserving learning value, not just reducing workload. Track whether recurrence rates for previously-reviewed incident classes stay flat or improve even as review depth for minor incidents drops, to confirm the lighter-touch approach isn't quietly letting real risk go unaddressed.

Worked example
An organization runs roughly 2,000 incidents a month and full postmortems have become a rubber-stamp exercise nobody has time to do well. The fix: define three tiers. Tier 1 (high severity or genuinely novel pattern, maybe 5% of incidents) gets a full facilitated postmortem within a defined turnaround. Tier 2 (moderate severity, somewhat familiar pattern, maybe 25%) gets a lightweight async writeup by the on-call responder, reviewed by a rotating peer within a week, no live meeting required unless something surprising surfaces. Tier 3 (low severity, well-understood and recurring pattern, the remaining ~70%) gets an automated, templated log entry tagging the known category, with no individual analysis required unless the volume of that specific category spikes, which triggers escalation to a full pattern-level review. Reviewer rotation is enforced across teams so no single person is doing more than a defined share of Tier 1 and Tier 2 reviews in a given month.

Trade-offs and pitfalls
The biggest risk of this approach is under-reviewing something that seemed minor in isolation but was actually an early instance of a bigger, developing problem; the pattern-level tracking (watching for a spike in a normally-quiet Tier 3 category) is what catches that, and skipping it is the most common mistake when teams implement tiering purely to save time.

Direct answer

A bounded queue (fixed maxsize) applies backpressure automatically: once it is full, put() blocks (threading) or suspends (await put, asyncio) until a consumer frees a slot, so a fast producer is throttled to the consumers' pace without any extra code. queue.Queue gives you this with an internal threading.Condition; asyncio.Queue gives you the same shape with await instead of blocking waits. The wake-up semantics differ by mechanism: a raw condition variable needs you to choose notify() (wake exactly one waiter) versus notify_all() (wake everyone, most of whom will just recheck and go back to waiting), while asyncio.Queue hides that choice entirely, await put/await get suspend and resume the right coroutine without you managing wake-ups by hand.

Structured elaboration

Threading version: queue.Queue(maxsize=N) is backed by a Lock plus two Conditions internally (not-full, not-empty). put() blocks on the not-full condition when the queue is at capacity; get() blocks on not-empty when it is empty. You do not touch the condition variables yourself, Queue already calls notify() correctly.

asyncio version: a coroutine is a function defined with async def that can pause mid-execution, at an await, and let other coroutines run, then resume later exactly where it left off; the event loop is the scheduler that decides which paused coroutine gets to run next. asyncio.Queue(maxsize=N) has the same contract as the threading version, but await q.put(item) suspends the coroutine (yielding control to the event loop, not blocking an OS thread) when full, and await q.get() suspends when empty. Concurrency comes from running many producer/consumer coroutines as tasks on one event loop instead of many OS threads.

Graceful shutdown, both versions: signal producers to stop (an Event), let running producers finish their current item, then drain the queue (q.join(), which waits until every put item has had a matching task_done()), then push one sentinel value per consumer so each one exits its loop cleanly. q.join() hanging is almost always a sign that a consumer path skipped task_done(), most often via an exception on the processing line before it reached the task_done() call.

The harder variant, without queue.Queue/asyncio.Queue: you build the bounded buffer yourself from a Lock plus two Condition objects sharing that lock, one for "not full", one for "not empty." This is exactly what queue.Queue does internally, made explicit:

import threading, collections

class BoundedBuffer:
    def __init__(self, maxsize):
        self.maxsize = maxsize
        self.items = collections.deque()
        self.lock = threading.Lock()
        self.not_full = threading.Condition(self.lock)
        self.not_empty = threading.Condition(self.lock)

    def put(self, item):
        with self.not_full:
            while len(self.items) >= self.maxsize:
                self.not_full.wait()      # releases the lock, sleeps until notified
            self.items.append(item)
            self.not_empty.notify()       # wake exactly one waiting consumer

    def get(self):
        with self.not_empty:
            while not self.items:
                self.not_empty.wait()
            item = self.items.popleft()
            self.not_full.notify()        # wake exactly one waiting producer
            return item

notify() (one waiter) is correct here because exactly one slot opened up, waking everyone with notify_all() would just cause the other waiters to recheck their while condition and go back to sleep, which is harmless but wasteful (a "thundering herd" of wasted wake-ups). notify_all() earns its keep when a single event can satisfy many different waiters at once, which is not the case for a single-slot state change like this.

Worked example

asyncio.Queue-based pipeline with graceful shutdown, verified on CPython 3.12 (produced and consumed counts always match, since q.join() guarantees every item was drained before sentinels are sent; the exact counts vary run to run because they depend on real-time scheduling):

import asyncio, random

SENTINEL = object()

async def producer(pid, q, stop_event, rng):
    i = 0
    while not stop_event.is_set():
        await q.put((pid, i))          # blocks here once the queue is full
        i += 1
        await asyncio.sleep(rng.random() * 0.02)

async def consumer(cid, q, processed):
    while True:
        item = await q.get()
        if item is SENTINEL:
            q.task_done()
            break
        processed.append((cid, item))
        q.task_done()

async def run_pipeline(n_producers=2, n_consumers=3, maxsize=5, run_time=0.3, seed=7):
    rng = random.Random(seed)
    q = asyncio.Queue(maxsize=maxsize)
    stop_event = asyncio.Event()
    processed = []

    producers = [asyncio.create_task(producer(i, q, stop_event, rng)) for i in range(n_producers)]
    consumers = [asyncio.create_task(consumer(i, q, processed)) for i in range(n_consumers)]

    await asyncio.sleep(run_time)
    stop_event.set()
    await asyncio.gather(*producers)
    await q.join()                          # wait until every produced item is processed
    for _ in consumers:
        await q.put(SENTINEL)
    await asyncio.gather(*consumers)
    assert len(processed) == sum(1 for _ in processed)  # every produced item made it through

asyncio.run(run_pipeline())

A representative run reported "total produced: 62, total consumed: 62"; the invariant produced == consumed holds on every run because q.join() will not return until all items are drained, the specific count itself is not reproducible (it depends on wall-clock scheduling) and is not the claim being made here.

Trade-offs & pitfalls

• Complexity: $O(1)$ per item for queue operations; space is bounded by maxsize for the queue itself, plus $O(k)$ for the $k$ in-flight producer/consumer tasks.
• Edge case: if a consumer raises an exception before calling task_done(), q.join() hangs forever; wrap the processing body in try/finally so task_done() always runs.
• Edge case: sending fewer sentinels than there are consumers leaves some consumers blocked on get() forever; one sentinel per consumer is required, not one total.
• Choosing notify() when the change actually affects multiple waiters (rare for a single bounded queue, more common in custom condition-based coordination) silently starves the other waiters; if you are not certain only one waiter's condition changed, notify_all() is the safer default even though it wakes more coroutines/threads than strictly necessary.
• The asyncio version scales to many more concurrent producers/consumers than the threading version for the same memory budget, since coroutines are far cheaper than OS threads, but it only helps if the "work" inside producer/consumer is itself non-blocking; a CPU-bound consumer inside an asyncio pipeline will stall the whole event loop exactly like any other CPU-bound coroutine.

Situation: On a prior project, we uncovered a late dependency issue that would push a release by a few weeks.

Task: I needed to tell stakeholders early, explain the impact clearly, and keep trust intact.

Action: I didn’t wait until we had perfect data. I shared the risk as soon as the pattern was clear, framed it around business impact, and presented options rather than just the problem. I explained what was affected, what was still on track, and what we could do next: reduce scope, add temporary support, or adjust the release sequence. I also set a short update cadence so no one had to guess.

Result: The group made a quick decision on scope, leadership appreciated the early warning, and the conversation stayed focused on trade-offs instead of blame. The key was being direct, specific, and calm.

What I learned is that trust is protected by speed, honesty, and a recommendation. If I bring a risk with a clear path forward, stakeholders usually stay engaged instead of feeling surprised or managed around.

Direct answer

For heavy-tailed data, reach for the trimmed mean (the mean after dropping a fixed percentage from each tail), the winsorized mean (capping extreme values at a percentile rather than dropping them), and the median absolute deviation, or MAD (a robust measure of spread built from the median rather than the mean). Each protects against exactly what the plain mean and standard deviation are vulnerable to: a small number of extreme values dominating the summary.

What each protects against, specifically

The plain mean weights every value equally in the sum, so one extreme transaction can shift it substantially; a trimmed mean removes the most extreme values entirely before averaging, which is simple but throws away potentially real information at the tails. A winsorized mean keeps every observation but caps the extreme ones at a threshold, preserving the row count while limiting any single point's influence, a useful middle ground when you don't want to discard rows outright. MAD replaces both the center (the median) and the spread calculation with something that isn't sensitive to a handful of extreme values the way variance is, since the median of a bunch of moderate values plus one extreme one barely moves.

Worked example

Twelve values representing transaction sizes: mostly clustered between 11 and 15, with one value of 250. The plain mean comes out to 32.75, badly distorted by that single value; the median is 13.0, essentially unaffected. A 10% trimmed mean comes out to 13.2, very close to the median, having simply excluded the most extreme values from each end before averaging. The winsorized version (capping the top and bottom 10% at the next-most-extreme retained value) gives a mean of 13.25, similarly close to the median while still using every row. The MAD, scaled to be comparable to a standard deviation on normal data, comes out to about 1.48, a genuinely honest measure of the typical spread among the twelve values, in stark contrast to what the standard deviation (dominated by that single 250) would report.

Trade-offs and pitfalls

Trimming and winsorizing require choosing a percentage to trim or cap at, which is itself a judgment call, not free of assumptions; too aggressive a trim can discard real, meaningful extreme values along with true anomalies. Always report which robust method you used and at what threshold, since "robust mean" without that detail hides a real analytical choice.

Direct answer. Scaling up GPU count or enabling mixed precision changes several things simultaneously (effective batch size, gradient synchronization behavior, numeric precision), so a training run that was stable before and diverges after needs a plan that isolates ONE variable at a time rather than reverting the whole change and guessing.

Three terms this depends on. A replica is one full copy of the model living on one GPU. DDP (DistributedDataParallel) is the synchronous data-parallel wrapper that runs one process per GPU and keeps the replicas in step. All-reduce is the collective operation that combines a tensor across every replica and hands every replica back the same combined result, which is how the per-replica gradients become one shared gradient each step.

Plausible causes, and what to test first. The single highest-prior cause when scaling GPU count is learning-rate scaling: if the effective batch size grows with GPU count but the learning rate doesn't, or is scaled incorrectly (linear scaling isn't always the right rule for every optimizer), training can become unstable purely from that mismatch, and it's the cheapest hypothesis to test. Worked with real numbers so "adjusted for the new batch size" is computable: a stable baseline of 8 GPUs at 32 examples per device is an effective batch of 256 at LR 0.1. Scaling to 64 GPUs at the same 32 per device gives an effective batch of 2048, an 8x increase. The linear rule says LR = 0.1 x 2048/256 = 0.8. The square-root rule says LR = 0.1 x sqrt(8) = 0.283. If the job was launched at 0.8 and diverges, the diagnostic rerun is at 0.283, and the sanity rerun (which trades throughput for certainty) is at the original 0.1.

Next, check batch-norm statistics: if BatchNorm computes per-replica statistics independently rather than synchronizing across replicas, a network with BatchNorm can behave differently, and less stably, at high replica counts purely because each replica now sees a smaller effective batch for normalization purposes. Then check all-reduce precision (is gradient synchronization happening in a lower precision than intended) and synchronization/seed-order differences (are all replicas actually initialized identically before training starts).

For mixed precision (AMP) specifically. Gradient-accumulation interactions with AMP are a common culprit. The mechanics, since the ordering is the whole bug: loss scaling multiplies the loss by a large factor S before the backward pass so that small fp16 gradients do not underflow to zero, and the gradients must later be divided by S ("unscaled") before the optimizer step. With gradient accumulation over K micro-batches, the correct order is accumulate-then-unscale: keep every micro-batch's gradients scaled, sum all K of them, and unscale ONCE immediately before the optimizer step. The wrong order, unscale-then-accumulate, divides each micro-batch's gradients by S as it goes and then accumulates already-unscaled values, while the scaler still believes it owes the division, so the effective gradient magnitude ends up off by a factor tied to K and the scaler's overflow check runs against tensors that are no longer on the scale it expects. Confirm which order your framework's AMP integration actually uses before assuming the bug is elsewhere. Separately, some DDP jobs show gradient divergence where SOME workers produce NaN and others don't purely from AMP's dynamic loss scaling adjusting independently per worker if it isn't explicitly synchronized, worth checking whether your framework synchronizes the loss-scale factor across replicas or lets each pick its own.

What you would expect to observe if each cause were the real one. This is the half that turns the list into a diagnosis:

• Learning-rate / batch-size mismatch: divergence arrives within the first few hundred steps and arrives EARLIER the more GPUs you add, and the step at which it blows up moves monotonically as you sweep LR. Rerunning the same 64-GPU job at 0.283 (or at 0.1) with everything else untouched restores stability. Gradient norms grow smoothly over several steps before the blow-up rather than jumping in one step.
• BatchNorm computing per-replica statistics: dump each replica's BatchNorm running mean and running variance buffers and compare them across ranks. Synchronized normalization keeps them near-identical; the unsynchronized failure shows them visibly diverging, with the spread widening as per-device batch size falls. Crucially, the gradients themselves stay finite the whole time, so this looks like instability without any numeric red flag. It gets worse as you hold total batch fixed and add GPUs (shrinking per-device batch), and it disappears when you swap in synchronized BatchNorm or raise the per-device batch back up.
• All-reduce running in a lower precision than intended: checksum or norm each rank's gradient tensor immediately BEFORE the collective and immediately AFTER it. Before, the ranks legitimately differ, since each saw a different data shard. After, every rank must hold a bit-identical tensor. The tell is that post-collective gradients agree only to the first three or four significant digits rather than exactly, and that the disagreement grows with rank count, because more partial sums mean more accumulated rounding. Compare against a reference reduction performed in fp32 to size the error.
• Seed or initialization-order mismatch: hash every replica's parameters at step 0, before any data is seen. They must be identical. If rank 0's hash differs from rank 1's at step 0, the replicas never started from the same weights, and the shared averaged gradient is being applied to divergent parameter sets. The signature is that divergence appears essentially at step 1, not after a hundred healthy steps.
• AMP unscale-then-accumulate ordering: instrument the global gradient norm immediately before the optimizer step, and compare against the same total batch run with K=1 (no accumulation). The two should agree within numerical noise. The ordering bug shows up as a clean multiplicative discrepancy tied to K, a factor of K or 1/K rather than a vague drift, which is what makes it identifiable rather than merely suspicious.
• AMP loss-scale desync across workers: log the current loss-scale value per rank per step. Healthy runs show every rank holding the same value and stepping it in lockstep. The failure shows ranks holding different scales at the same step (one at 1024 while another is at 32768), SOME ranks reporting NaN while others report clean gradients, and the identity of the NaN-reporting ranks changing from run to run.

Why the effective batch size is the thread running through all of this. Scaling GPU count in synchronous data-parallel training multiplies the effective batch size by the number of new replicas, which is exactly what makes the learning-rate hypothesis the highest-prior one and why the arithmetic above matters. It also means that if the instability appears only past a specific scale rather than at every scale, that threshold is itself evidence: a genuine LR/batch mismatch only becomes destabilizing once the step size and the curvature disagree badly enough, whereas a hardware or environment fault would be present regardless of how many replicas are running.

Prioritized order to test. Order by what each check actually costs to run, then by prior probability, because two of these are nearly free and one of them is a precondition for trusting any of the others.

0. Free, so run these before spending a single GPU-hour on a rerun. Hash every replica's parameters at step 0: one hash per rank, zero training steps, and if the hashes disagree you are done, because none of the cross-rank comparisons below mean anything until the replicas provably start from the same weights. Under AMP, also log each rank's loss-scale value per step, which is one scalar per rank and settles the desync hypothesis outright. Neither of these is the most likely cause; they go first because they are cheaper than everything else and because they are the two that can invalidate the other measurements.
1. Learning-rate scaling, the highest-prior cause and the first hypothesis worth real GPU-hours: rerun the 64-GPU job at 0.283, and at the original 0.1 if that is still unstable. One short run, one variable, directly falsifiable.
2. AMP accumulate-then-unscale ordering, whenever gradient accumulation is on: compare the pre-optimizer global gradient norm against the same total batch run at K=1. One extra short run and one scalar, and the tell is a clean factor of K or 1/K rather than a drift.
3. BatchNorm synchronization: dump the running-mean and running-variance buffers on every rank and diff them. More instrumentation than anything above, still far less than the last item.
4. All-reduce precision, last: checksumming every rank's gradient tensor before and after the collective is the heaviest instrumentation in this list, and it is also the least common root cause for a symptom that appeared specifically at a scaling boundary.

Direct answer

You measure whether your communication actually works the same way you'd measure any other process: define what 'acted on' looks like concretely, instrument it, and track it over time, rather than assuming a well-received presentation equals a changed decision.

Structured elaboration

1. Separate 'insight was delivered' from 'insight was acted on.'
Most teams only track the former (a deck was presented, a dashboard exists) because it's easy to observe. The real signal is whether a decision, a roadmap item, or a resourcing choice actually changed as a result. That requires deliberately logging each insight or recommendation as a discrete, trackable unit (a ticket, a decision-log entry, a recommendation ID) rather than letting it live only inside a slide deck that nobody revisits.

2. Define 4-6 concrete metrics that make actionability observable.
A reasonable, non-exhaustive set: (a) share of recommendations formally accepted, rejected, or deferred within a defined window (e.g. 30 days) - the acceptance rate; (b) average time from delivery to a decision being made on it - time-to-decision; (c) share of accepted recommendations that were actually implemented, not just approved - the follow-through rate, since approval without implementation is a common failure mode; (d) measured downstream business impact where an accepted recommendation included a predicted effect (did the metric move the way the insight predicted, and by how much); (e) a stakeholder-reported usefulness or trust score, gathered periodically, as a leading indicator; and (f) recurrence rate of the same insight being re-delivered because it was previously ignored, which is a strong negative signal.

3. Build the minimal data collection to make this trackable, not a large new system.
In practice this is a lightweight log: each insight gets an ID, a delivery date, an owner, a decision outcome, and (if applicable) a link to the metric it was supposed to move. This can live in an existing ticketing or decision-log tool rather than requiring new infrastructure; the discipline is in the LOGGING HABIT, not the tooling.

4. Assign ownership and a reporting cadence.
The team that produces insights (analytics, data science, BI) should own tracking whether insights were delivered and understood; the business owner who received the recommendation should own logging the decision outcome, since they are the one who knows whether it was actually acted on. Report the rollup on a cadence that matches how often recommendations are made (commonly monthly or quarterly) rather than in real time, since 'time to decision' for a nontrivial recommendation is naturally measured in weeks, not hours.

Worked example

A data science team delivers 40 recommendations over a quarter (for example: adjust a pricing tier, change an onboarding step, retire an underperforming feature). They log each with an ID and owner. At quarter end: 28 of 40 were formally decided within 30 days (70% decision rate), of which 19 were accepted, 6 rejected, and 3 deferred; of the 19 accepted, 14 were actually implemented within the quarter (a 74% follow-through rate on acceptances); and of those 14, 9 had a predicted metric attached, of which 6 moved in the predicted direction by at least half the predicted magnitude. The team also finds that 5 of the 40 recommendations were substantively the same insight delivered a second time because the first delivery was never decided on, a recurrence signal that prompts them to investigate why certain recommendation types stall (in this case, three of the five involved a cross-team dependency with no clear single decision-owner). That specific finding, a missing decision-owner for cross-team recommendations, becomes the actionable process fix, which is itself an example of the framework working as intended.

Trade-offs and pitfalls

• The biggest pitfall is conflating 'stakeholders liked the presentation' with 'a decision changed'; a positive reaction in the room is not evidence of actionability and should not substitute for the follow-through metrics above.
• Attributing a downstream metric move entirely to one recommendation is often overclaiming, since other changes happen concurrently; where possible, treat the predicted-impact check as a directional signal, not a rigorous causal claim, and say so.
• A high recurrence rate is more informative than a low acceptance rate; recommendations legitimately get rejected for good reasons, but a recommendation that keeps resurfacing because no one ever decided on it points to a process gap, not a communication gap.
• Do not build a heavy new tracking system before establishing the logging habit manually; teams that try to automate this before anyone consistently logs decisions end up with clean-looking dashboards over incomplete data.