Covers designing and operating machine learning systems to handle growth in data volume, model complexity, and traffic. Topics include distributed training strategies such as data parallelism, model parallelism, and pipeline parallelism; coordination and orchestration approaches like parameter servers, gradient aggregation, and framework tools such as PyTorch distributed, Horovod, and TensorFlow strategies; data pipeline and I O considerations including sharding, efficient formats, preprocessing bottlenecks, streaming and batch ingestion; serving and inference scaling including model sharding, batching for throughput, autoscaling, request routing, caching, and latency versus throughput tradeoffs. Also includes monitoring, profiling, checkpointing and recovery, reproducibility, cost and resource optimization, and common bottleneck analysis across network, storage, CPU preprocessing, and accelerator utilization.
MediumTechnical
8 practiced
Your spot instance training jobs are frequently interrupted, and rerunning from scratch is too expensive. How would you design checkpointing and restart behavior so that recovery is fast, state is consistent, and the training run remains reproducible?
Sample Answer
**Approach**I would make checkpointing a consistent snapshot of everything needed to resume training, not just model weights. That means saving model parameters, optimizer state, learning-rate scheduler state, random number generator seeds, current epoch and batch cursor, and any data sampler state.**Design**- Write checkpoints atomically: first write to temporary storage, validate checksums, then publish a manifest pointer.- Keep checkpoints incremental when possible, but always make the latest one self-contained for fast restart.- Trigger an immediate checkpoint on spot interruption notice, then resume from the last good manifest.- Store code version, config, and data version so the run is reproducible.**Worked example**If I checkpoint every 10 minutes and a preemption happens at minute 37, the restart only redoes at most 7 minutes of work, not the whole job.**Why this works**The manifest guarantees consistency, the RNG and sampler state keep replay deterministic, and the atomic publish prevents half-written checkpoints from being used.
MediumSystem Design
9 practiced
Design a shared ML training platform for multiple teams that need to run large distributed jobs, recover from node failures, and control cost. What core services and controls would you include, and how would jobs acquire and release compute?
Sample Answer
**Core services**I would build a platform with a job API, scheduler, quota service, checkpoint store, metadata catalog, and node agents. The scheduler matches jobs to GPU pools, while the quota service enforces team budgets and priorities.**How jobs get compute**A team submits a job spec with requested GPUs, runtime, and a checkpoint policy. The scheduler grants a lease, which is a time-bounded claim on nodes. Workers heartbeat to renew the lease. On completion, failure, or preemption, the lease ends and the GPUs are released automatically.**Controls**- Per-team quotas and project budgets- Priority classes for urgent training- Spot first scheduling with fallback to on-demand- Mandatory checkpoints for long jobs- Admission control when the cluster is full**Worked example**A 64-GPU run might get 8 workers with 8 GPUs each and a 30-minute renewable lease. If a node fails, the job restarts from the last checkpoint instead of starting over.This design keeps cost visible, failures recoverable, and shared capacity fair.
HardTechnical
8 practiced
Two production deployments are under review. System A is fast for individual requests but keeps GPUs underutilized. System B is much cheaper per request but misses the latency SLO whenever traffic spikes. Given that revenue depends on both responsiveness and margin, how would you decide which system to optimize first, and what data would you want before making the call?
Sample Answer
**How I would decide**I would optimize the system that is currently the binding constraint on revenue, not just the one that looks cheapest on paper. That means comparing tail latency impact, cost per successful request, and how often spikes happen.**Data I would want**- p50, p95, and p99 latency under normal and peak traffic- Request arrival patterns and spike duration- Conversion, churn, or error-rate impact when latency is missed- GPU cost per accepted request- How much headroom each system has before it breaks**Decision rule**If latency misses directly reduce conversions or trigger SLO penalties, I would prioritize the system with the latency problem. If the cheaper system only fails during rare spikes, I might first improve buffering, batching, or autoscaling there. If the fast system wastes a lot of GPU time every day, I would target utilization first.**Worked example**If System A meets the SLO but burns extra GPU cost every minute, while System B saves money but misses the p95 target during frequent bursts, I would usually fix B first because revenue loss from slow responses can erase the savings.I want to choose the fix that improves total business value, not only infra efficiency.
HardSystem Design
12 practiced
An inference service must handle bursty traffic for a large model with a strict p95 latency target and a limited GPU budget. How would you scale serving so that you keep latency under control without wasting capacity during quiet periods?
Sample Answer
**Design**I would keep a small pool of warm replicas and use dynamic batching, an autoscaler, and request admission control. The goal is to absorb bursts without letting queueing time blow up p95 latency.**Key components**- API gateway or router that sends requests to healthy replicas- Inference workers with a batcher that waits only a short time before running a batch- Autoscaler that looks at queue depth, GPU utilization, and recent latency- Warm capacity so scale-up is not from zero during traffic spikes- Priority or deadline-aware routing for urgent requests**How I would control latency**I would set a maximum batch wait time so batching improves throughput without causing long queues. During quiet periods, I would scale down gradually but keep a floor of ready GPUs. During spikes, I would prefer a small amount of extra capacity over missing the p95 target.**Worked example**If a batcher waits up to 5 ms and batches up to 8 requests, it can improve GPU efficiency while still keeping queue delay bounded.**Tradeoff**More batching lowers cost per request, but too much batching hurts tail latency. The right answer is a controlled batch size, warm headroom, and fast scaling signals rather than aggressive scale-to-zero.
HardTechnical
8 practiced
A release improved model quality, but in production the p99 latency doubled and autoscaling did not trigger. The average CPU on the pods still looks normal. How would you trace the request path end to end to isolate whether the slowdown comes from feature retrieval, preprocessing, batching, model execution, or a downstream dependency?
Sample Answer
**Approach**I would trace one request from ingress to response using distributed tracing. A trace is a single request’s timeline, broken into spans, which are timed steps like feature retrieval, preprocessing, model inference, and downstream calls. CPU can look normal because the bottleneck may be waiting on network, locks, queueing, or a slow dependency, not raw compute.**How I would isolate it**1. Start with one trace ID from a slow p99 request.2. Check span timings for each stage: feature store, preprocessing, batching, model execution, postprocessing, and any outbound calls.3. Compare slow traces to fast ones. I care about where the extra time accumulates and whether it is wait time or compute time.4. Add or inspect metrics per stage, such as feature fetch latency, batch queue time, inference time, and downstream timeout rate.5. If tracing is coarse, I would add sub-spans in the service and temporary structured logs with the trace ID to pinpoint the jump.**Worked example**If median requests are 80 ms and p99 is now 160 ms, I might find: feature retrieval 15 ms, preprocessing 10 ms, batch queue 55 ms, model execution 20 ms, downstream call 45 ms. That tells me the issue is batching or a dependency, not the model itself.**Next checks**- Feature retrieval slow only on cache misses, then look at cache hit rate and feature store latency.- Batching spikes, then inspect queue depth and batch timeout settings.- Model execution spikes, then profile the serving container.- Downstream dependency spikes, then check retries, timeouts, and circuit breakers.If autoscaling did not trigger, I would also verify the scaling signal. CPU-based scaling often misses latency problems, so I would prefer queue depth, request rate, or custom p95/p99 latency signals for this workload.
Unlock Full Question Bank
Get access to hundreds of AI System Scalability interview questions and detailed answers.
