End-to-End ML System Design Questions

First split the problem into two buckets

1. True optimization issues: the model is mathematically harder to train at larger scale.
2. Distributed systems issues: the parallel run is changing the effective training behavior.

Things I would investigate

• Effective batch size and learning rate scaling. A bigger world size often means a larger batch, which can require LR tuning.
• Data sharding. I would verify that each sample is seen once, not duplicated or skipped.
• Gradient synchronization. I would check that all ranks contribute the same gradients and that there are no stale or dropped updates.
• Precision and numerics. Mixed precision can introduce overflow, underflow, or different rounding behavior at scale.
• Stateful layers like batch normalization, which can behave differently across replicas.

How I would separate them
I would compare small-scale and large-scale runs with the same global batch and fixed seeds, then inspect gradient norms, loss curves, and sample hashes.

Worked example
If 8 workers train with a batch of 256 each, the global batch becomes 2048. If loss gets noisier only after that change, it may be an optimization tuning problem. If the run only degrades when messages cross nodes, it is more likely a communication or synchronization problem.

That distinction tells me whether to tune the optimizer or fix the distributed stack.

Redesign the input path
I would separate expensive preprocessing from the training loop and make the input stream sharded, cached, and deterministic.

Main changes

• Compact many small object-store files into larger shards so reads are sequential instead of chatty.
• Precompute expensive transforms once, then store the transformed output or an intermediate cache.
• Use a manifest that records shard order, sample IDs, and versioned transform code.
• Prefetch data to local disk or memory so GPUs do not wait on network reads.
• Keep lightweight, deterministic transforms in the training worker so results are reproducible.

Worked example
If the dataset is 10,000 files of 2 MB each, that is 20 GB spread across many requests. Repacking into 200 shards of 100 MB keeps the same 20 GB, but cuts file open and listing overhead dramatically.

Why this helps
The GPUs stay busy because decoding and network latency happen ahead of time. Reproducibility comes from versioned manifests and fixed seeds. Debuggability comes from keeping raw sample IDs so a bad example can be replayed exactly.

This is usually the highest leverage change when preprocessing time dominates training time.

How I would decide
I would optimize the system that is currently the binding constraint on revenue, not just the one that looks cheapest on paper. That means comparing tail latency impact, cost per successful request, and how often spikes happen.

Data I would want

• p50, p95, and p99 latency under normal and peak traffic
• Request arrival patterns and spike duration
• Conversion, churn, or error-rate impact when latency is missed
• GPU cost per accepted request
• How much headroom each system has before it breaks

Decision rule
If latency misses directly reduce conversions or trigger SLO penalties, I would prioritize the system with the latency problem. If the cheaper system only fails during rare spikes, I might first improve buffering, batching, or autoscaling there. If the fast system wastes a lot of GPU time every day, I would target utilization first.

Worked example
If System A meets the SLO but burns extra GPU cost every minute, while System B saves money but misses the p95 target during frequent bursts, I would usually fix B first because revenue loss from slow responses can erase the savings.

I want to choose the fix that improves total business value, not only infra efficiency.

What I would inspect first
I would start with per-step timing on the slow worker versus the rest of the cluster. If the code, model, and data are unchanged, a single lagging worker is usually a host or systems issue, not an ML issue.

Checks in order

• GPU utilization and memory bandwidth on the slow node
• Data loader wait time and local disk throughput
• CPU steal, thermal throttling, and noisy neighbors
• Network errors, packet drops, and collective communication logs
• Kernel and container logs for retries or hardware faults

Mitigation
If the worker is clearly abnormal, I would cordon it, move the job to a fresh node, and keep the training moving. If the slowdown comes from input starvation, I would reduce preprocessing on that host, increase local caching, or lower dataloader contention.

Worked example
If most workers take 180 ms per step but one takes 420 ms and spends 250 ms waiting on input, the bottleneck is the input path, not the model.

The goal is to isolate the bad actor quickly and avoid letting one slow node stall the whole synchronous job.

Approach
I would trace one request from ingress to response using distributed tracing. A trace is a single request’s timeline, broken into spans, which are timed steps like feature retrieval, preprocessing, model inference, and downstream calls. CPU can look normal because the bottleneck may be waiting on network, locks, queueing, or a slow dependency, not raw compute.

How I would isolate it

1. Start with one trace ID from a slow p99 request.
2. Check span timings for each stage: feature store, preprocessing, batching, model execution, postprocessing, and any outbound calls.
3. Compare slow traces to fast ones. I care about where the extra time accumulates and whether it is wait time or compute time.
4. Add or inspect metrics per stage, such as feature fetch latency, batch queue time, inference time, and downstream timeout rate.
5. If tracing is coarse, I would add sub-spans in the service and temporary structured logs with the trace ID to pinpoint the jump.

Worked example
If median requests are 80 ms and p99 is now 160 ms, I might find: feature retrieval 15 ms, preprocessing 10 ms, batch queue 55 ms, model execution 20 ms, downstream call 45 ms. That tells me the issue is batching or a dependency, not the model itself.

Next checks

• Feature retrieval slow only on cache misses, then look at cache hit rate and feature store latency.
• Batching spikes, then inspect queue depth and batch timeout settings.
• Model execution spikes, then profile the serving container.
• Downstream dependency spikes, then check retries, timeouts, and circuit breakers.

If autoscaling did not trigger, I would also verify the scaling signal. CPU-based scaling often misses latency problems, so I would prefer queue depth, request rate, or custom p95/p99 latency signals for this workload.