End-to-End ML System Design Questions

What I would check first
I would measure where time is going in one training step. The three main bottlenecks are input pipeline, compute, and communication.

Definitions

• Input pipeline: loading, decoding, and preprocessing data before it reaches the GPU.
• Compute: the actual forward and backward passes on the GPU.
• Communication: syncing gradients or parameters across devices.

Evidence to collect

• GPU utilization and memory usage
• Step time broken into data wait, compute, and sync time
• Disk and network throughput
• CPU usage on dataloader workers

Worked example
If a step takes 300 ms total and 180 ms of that is waiting for data, then the GPU is idle most of the time and the input path is the bottleneck. If compute is fast but time rises when more workers are added, communication may be the issue.

Rule of thumb
Low GPU utilization usually points to data or host issues. High GPU utilization with slow scaling usually points to synchronization or network overhead.

I would start with the timing breakdown because it tells you which team, storage, compute, or networking, needs to act first.

What I would inspect first
I would start with per-step timing on the slow worker versus the rest of the cluster. If the code, model, and data are unchanged, a single lagging worker is usually a host or systems issue, not an ML issue.

Checks in order

• GPU utilization and memory bandwidth on the slow node
• Data loader wait time and local disk throughput
• CPU steal, thermal throttling, and noisy neighbors
• Network errors, packet drops, and collective communication logs
• Kernel and container logs for retries or hardware faults

Mitigation
If the worker is clearly abnormal, I would cordon it, move the job to a fresh node, and keep the training moving. If the slowdown comes from input starvation, I would reduce preprocessing on that host, increase local caching, or lower dataloader contention.

Worked example
If most workers take 180 ms per step but one takes 420 ms and spends 250 ms waiting on input, the bottleneck is the input path, not the model.

The goal is to isolate the bad actor quickly and avoid letting one slow node stall the whole synchronous job.

Ways to split the work

• Data parallelism. Each GPU holds a full copy of the model and processes different batches. It is simplest, but the model must fit on one GPU.
• Tensor parallelism. Each layer is split across GPUs. This helps when the layers themselves are too large, but it adds communication every layer.
• Pipeline parallelism. Different layers live on different GPUs. This reduces memory pressure, but can leave devices idle if the pipeline is not balanced.
• Sharded optimizers or fully sharded data parallelism. These split parameters, gradients, and optimizer state to reduce memory use while keeping the training code closer to data parallelism.

How I would choose
If the model is only slightly too big for one GPU, I would prefer sharded data parallelism because it is usually the easiest path. If individual matrix multiplies are the memory or compute problem, tensor parallelism may fit better. If the model is very deep and activations dominate, pipeline or hybrid parallelism can help.

Worked example
For an 8 GPU cluster and a model that almost fits on one device, I would first try sharding plus data parallelism before moving to a more complex hybrid design.

The tradeoff is always simplicity versus communication cost versus memory savings.

Design
I would keep a small pool of warm replicas and use dynamic batching, an autoscaler, and request admission control. The goal is to absorb bursts without letting queueing time blow up p95 latency.

Key components

• API gateway or router that sends requests to healthy replicas
• Inference workers with a batcher that waits only a short time before running a batch
• Autoscaler that looks at queue depth, GPU utilization, and recent latency
• Warm capacity so scale-up is not from zero during traffic spikes
• Priority or deadline-aware routing for urgent requests

How I would control latency
I would set a maximum batch wait time so batching improves throughput without causing long queues. During quiet periods, I would scale down gradually but keep a floor of ready GPUs. During spikes, I would prefer a small amount of extra capacity over missing the p95 target.

Worked example
If a batcher waits up to 5 ms and batches up to 8 requests, it can improve GPU efficiency while still keeping queue delay bounded.

Tradeoff
More batching lowers cost per request, but too much batching hurts tail latency. The right answer is a controlled batch size, warm headroom, and fast scaling signals rather than aggressive scale-to-zero.

First split the problem into two buckets

1. True optimization issues: the model is mathematically harder to train at larger scale.
2. Distributed systems issues: the parallel run is changing the effective training behavior.

Things I would investigate

• Effective batch size and learning rate scaling. A bigger world size often means a larger batch, which can require LR tuning.
• Data sharding. I would verify that each sample is seen once, not duplicated or skipped.
• Gradient synchronization. I would check that all ranks contribute the same gradients and that there are no stale or dropped updates.
• Precision and numerics. Mixed precision can introduce overflow, underflow, or different rounding behavior at scale.
• Stateful layers like batch normalization, which can behave differently across replicas.

How I would separate them
I would compare small-scale and large-scale runs with the same global batch and fixed seeds, then inspect gradient norms, loss curves, and sample hashes.

Worked example
If 8 workers train with a batch of 256 each, the global batch becomes 2048. If loss gets noisier only after that change, it may be an optimization tuning problem. If the run only degrades when messages cross nodes, it is more likely a communication or synchronization problem.

That distinction tells me whether to tune the optimizer or fix the distributed stack.