End-to-End ML System Design Questions

Design
I would keep a small pool of warm replicas and use dynamic batching, an autoscaler, and request admission control. The goal is to absorb bursts without letting queueing time blow up p95 latency.

Key components

• API gateway or router that sends requests to healthy replicas
• Inference workers with a batcher that waits only a short time before running a batch
• Autoscaler that looks at queue depth, GPU utilization, and recent latency
• Warm capacity so scale-up is not from zero during traffic spikes
• Priority or deadline-aware routing for urgent requests

How I would control latency
I would set a maximum batch wait time so batching improves throughput without causing long queues. During quiet periods, I would scale down gradually but keep a floor of ready GPUs. During spikes, I would prefer a small amount of extra capacity over missing the p95 target.

Worked example
If a batcher waits up to 5 ms and batches up to 8 requests, it can improve GPU efficiency while still keeping queue delay bounded.

Tradeoff
More batching lowers cost per request, but too much batching hurts tail latency. The right answer is a controlled batch size, warm headroom, and fast scaling signals rather than aggressive scale-to-zero.

Approach
I would make checkpointing a consistent snapshot of everything needed to resume training, not just model weights. That means saving model parameters, optimizer state, learning-rate scheduler state, random number generator seeds, current epoch and batch cursor, and any data sampler state.

Design

• Write checkpoints atomically: first write to temporary storage, validate checksums, then publish a manifest pointer.
• Keep checkpoints incremental when possible, but always make the latest one self-contained for fast restart.
• Trigger an immediate checkpoint on spot interruption notice, then resume from the last good manifest.
• Store code version, config, and data version so the run is reproducible.

Worked example
If I checkpoint every 10 minutes and a preemption happens at minute 37, the restart only redoes at most 7 minutes of work, not the whole job.

Why this works
The manifest guarantees consistency, the RNG and sampler state keep replay deterministic, and the atomic publish prevents half-written checkpoints from being used.

Redesign the input path
I would separate expensive preprocessing from the training loop and make the input stream sharded, cached, and deterministic.

Main changes

• Compact many small object-store files into larger shards so reads are sequential instead of chatty.
• Precompute expensive transforms once, then store the transformed output or an intermediate cache.
• Use a manifest that records shard order, sample IDs, and versioned transform code.
• Prefetch data to local disk or memory so GPUs do not wait on network reads.
• Keep lightweight, deterministic transforms in the training worker so results are reproducible.

Worked example
If the dataset is 10,000 files of 2 MB each, that is 20 GB spread across many requests. Repacking into 200 shards of 100 MB keeps the same 20 GB, but cuts file open and listing overhead dramatically.

Why this helps
The GPUs stay busy because decoding and network latency happen ahead of time. Reproducibility comes from versioned manifests and fixed seeds. Debuggability comes from keeping raw sample IDs so a bad example can be replayed exactly.

This is usually the highest leverage change when preprocessing time dominates training time.

What I would check first
I would measure where time is going in one training step. The three main bottlenecks are input pipeline, compute, and communication.

Definitions

• Input pipeline: loading, decoding, and preprocessing data before it reaches the GPU.
• Compute: the actual forward and backward passes on the GPU.
• Communication: syncing gradients or parameters across devices.

Evidence to collect

• GPU utilization and memory usage
• Step time broken into data wait, compute, and sync time
• Disk and network throughput
• CPU usage on dataloader workers

Worked example
If a step takes 300 ms total and 180 ms of that is waiting for data, then the GPU is idle most of the time and the input path is the bottleneck. If compute is fast but time rises when more workers are added, communication may be the issue.

Rule of thumb
Low GPU utilization usually points to data or host issues. High GPU utilization with slow scaling usually points to synchronization or network overhead.

I would start with the timing breakdown because it tells you which team, storage, compute, or networking, needs to act first.

Core services
I would build a platform with a job API, scheduler, quota service, checkpoint store, metadata catalog, and node agents. The scheduler matches jobs to GPU pools, while the quota service enforces team budgets and priorities.

How jobs get compute
A team submits a job spec with requested GPUs, runtime, and a checkpoint policy. The scheduler grants a lease, which is a time-bounded claim on nodes. Workers heartbeat to renew the lease. On completion, failure, or preemption, the lease ends and the GPUs are released automatically.

Controls

• Per-team quotas and project budgets
• Priority classes for urgent training
• Spot first scheduling with fallback to on-demand
• Mandatory checkpoints for long jobs
• Admission control when the cluster is full

Worked example
A 64-GPU run might get 8 workers with 8 GPUs each and a 30-minute renewable lease. If a node fails, the job restarts from the last checkpoint instead of starting over.

This design keeps cost visible, failures recoverable, and shared capacity fair.