Capacity Planning and Resource Optimization Questions

Covers forecasting, provisioning, and operating compute, memory, storage, and network resources efficiently to meet demand and service level objectives. Key skills include monitoring resource utilization metrics such as central processing unit usage, memory consumption, storage input and output and network throughput; analyzing historical trends and workload patterns to predict future demand; and planning capacity additions, safety margins, and buffer sizing. Candidates should understand vertical versus horizontal scaling, autoscaling policy design and cooldowns, right sizing instances or containers, workload placement and isolation, load balancing algorithms, and use of spot or preemptible capacity for interruptible workloads. Practical topics include storage planning and archival strategies, database memory tuning and buffer sizing, batching and off peak processing, model compression and inference optimization for machine learning workloads, alerts and dashboards, stress and validation testing of planned changes, and methods to measure that capacity decisions meet both performance and cost objectives.

EasyTechnical

21 practiced

Define cold-start and warm-up in the context of model serving. Why are these important for capacity planning, and what operational steps can you take to reduce cold-start latency for a serverless or containerized inference service?

Sample Answer

Cold-start: when a model-serving instance (container, function, or VM) is started from a cold state and must load the runtime, model weights, and any JIT/Warm caches before it can serve requests. Warm-up: bringing an instance to a ready state ahead of traffic—loading model, warming caches (e.g., GPU kernels, TF graph optimizations), and optionally running dummy inferences so latency on first real request is low.

Why it matters for capacity planning:- Cold-starts produce high tail latency and can violate SLOs if unaccounted for.- You must provision enough pre-warmed capacity (or tolerate extra latency) to meet peak SLA; poor planning causes either wasted resources or bad user experience.- Predictable traffic patterns, startup times, and model memory/GPU load influence how many warm instances to keep.

Operational steps to reduce cold-start latency (serverless and containerized):- Pre-warming / keep-alive: keep a small pool of warm instances running during expected peaks.- Lazy vs eager loading: load only required parts on demand, but eagerly load critical layers to avoid first-request spikes.- Lightweight bootstrap: reduce container image size, minimize framework overhead (use TF-TRT, ONNX Runtime, or custom C++ runtimes).- Model optimizations: quantization, pruning, and using serialized runtime-optimized formats to speed load time.- Snapshot/fast restore: persist warmed process state (e.g., cached memory maps, model sharding) and restore quickly if supported.- GPU warm-up: run a short dummy inference to initialize GPU kernels and memory allocations.- Traffic shaping: route a small percentage of traffic to new instances to warm them before full load.- Autoscaler tuning: configure scale-up cooldowns, target warm concurrency, and provisioned concurrency (e.g., AWS Lambda Provisioned Concurrency).- Monitoring & testing: instrument start-up latency, cold-start rate, and tail latency; run chaos tests and load tests to validate warm strategies.

These measures let you balance cost vs SLOs by quantifying warm instance counts needed and minimizing unpredictable latency from cold starts.

MediumTechnical

23 practiced

You must estimate runtime and peak network/storage I/O for a distributed training job. Dataset = 40 TB on S3, per-GPU read throughput = 200 MB/s, using 16 GPUs across 4 nodes, epochs = 3, checkpoint size = 10 GB written every hour. Estimate total wall-clock time (approx), required network bandwidth, and storage write throughput during checkpoints. Explain assumptions.

Sample Answer

Assumptions (explicit):- No local caching: training reads full dataset from S3 each epoch.- Dataset = 40 TB = 40,000 GB (for simplicity) — I'll also show value using 1 TB = 1024 GB variant.- Per-GPU sustained read = 200 MB/s (bytes), 16 GPUs total across 4 nodes (4 GPUs/node).- Checkpoint size = 10 GB written every 1 hour; checkpoint write may be bursty (e.g., written in 10–60s).- Ignore CPU/data-augmentation bottlenecks and retry/network overheads; add a small overhead factor later.

1) Total data read and wall-clock time- Total bytes read = 40 TB * 3 epochs = 120 TB = 120,000 GB.- Aggregate read throughput = 16 GPUs * 200 MB/s = 3,200 MB/s = 3.2 GB/s.- Pure read time = total / throughput = 120,000 GB / 3.2 GB/s = 37,500 s ≈ 10.4 hours.- Add ~15–25% overhead for metadata, augmentation, S3 tail latency → estimated wall-clock ≈ 12–13 hours.(If using 1 TB = 1024 GB: 120*1024 GB = 122,880 GB /3.2 = 38,400 s ≈ 10.7 h → with overhead ≈ 12.3–13.5 h.)

2) Required network bandwidth (read)- Aggregate sustained read = 3.2 GB/s = 25.6 Gbit/s.- Per-node sustained read = 4 GPUs * 200 MB/s = 800 MB/s = 6.4 Gbit/s.- Recommendation: provision at least 30–40 Gbit/s from S3 (or host NICs with 40 Gbps) for cluster to avoid throttling and to allow headroom.

3) Storage write throughput during checkpoints- Average write rate (amortized over 1 hour) = 10 GB / 3600 s ≈ 2.8 MB/s (negligible).- Peak write (burst): if checkpoint is uploaded in 30 s → 10 GB / 30 s ≈ 333 MB/s ≈ 2.7 Gbit/s aggregate. If in 10 s → 1,024 MB/s ≈ 8.2 Gbit/s.- Per-node peak (if evenly sharded across 4 nodes) in 30s case ≈ 83 MB/s (~0.66 Gbit/s) each.

Summary (rounded):- Wall-clock (3 epochs): ~12–13 hours (allowing overhead).- Aggregate read bandwidth required: 3.2 GB/s ≈ 25.6 Gbit/s (recommend 40 Gbit/s NIC/backplane).- Checkpoint write: average ~2.8 MB/s; peak depending on flush window: ~333 MB/s (30s) to ~1,000 MB/s (10s). Per-node peak ≈ 80–250 MB/s.

Notes / risks:- If S3 rate limits or per-object parallelism is low, effective throughput will drop — use multipart/parallel reads, prefetch/caching, or a local cache (EBS, instance store) to reduce S3 egress.- Data augmentation or CPU load can reduce effective per-GPU read; verify with profiling.- If using cloud egress metering, factor cost for sustained 25–40 Gbit/s over ~12 hours.

MediumTechnical

27 practiced

Describe how you would design a cost-aware autoscaler that factors in spot instance availability, on-demand fallback, and a monthly budget cap. How would the autoscaler decide between cheaper preemptible capacity and pricer on-demand nodes under varying load?

Sample Answer

Requirements & constraints:- Minimize cost while meeting SLOs (latency/throughput) for GPU/CPU AI workloads.- Use spot/preemptible instances when safe; fall back to on-demand when needed.- Enforce a monthly budget cap with pacing to avoid early burn.- Handle spot preemption events gracefully.

High-level design:- Kubernetes cluster with two node pools: spot (cheaper, preemptible) and on-demand.- A Cost-Aware Autoscaler (CAA) as a Kubernetes controller that watches metrics, node pools, spot market signals, and budget state.- Components: Metrics collector (utilization, queue length, pod priorities), Spot predictor (ML model or heuristics), Decision engine (risk scoring + optimization), Budget manager, Actuator (scale node pools), Preemption handler (eviction/draining policies, checkpointing).

Decision logic:1. Forecast short-term load and spot availability (minutes-hours) using time-series model (ARIMA/LSTM) trained on historical usage and cloud spot reclaim rates.2. Compute marginal cost per useful capacity (cost_per_vGPU * expected_uptime_fraction).3. Assign workloads by priority: - Best-effort (training experiments): prefer spot if expected uptime > threshold T1. - Latency-sensitive/critical inference: prefer on-demand or spot with mirrored fallback.4. Risk score = (1 - expected_uptime) * workload_criticality. If risk_score > threshold -> provision on-demand.5. Maintain a safety buffer: keep X% of capacity on on-demand or warm pool to absorb sudden load or mass spot loss.6. Budget pacing: convert remaining monthly budget to a burn rate; limiter reduces spot/on-demand scale when projected spend would exceed monthly cap (e.g., scale down low-priority jobs, pause noncritical experiments).

Handling preemption:- Use checkpointing, model sharding, or k8s PodDisruptionBudgets to minimize lost work.- On Spot preempt event, CAA immediately: - Recompute needed capacity, spin on-demand nodes for critical pods. - Reschedule best-effort jobs to remaining spot if projected uptime allows.

Implementation notes:- Integrate with Kubernetes Cluster Autoscaler or Karpenter; implement CAA as a controller that influences provisioner requests.- Use cloud APIs for spot pool health and pricing; persist budget state in a DB.- Expose policy knobs: thresholds T1/T2, buffer %, priority classes.- Monitoring & alerts for budget burn, spot churn, SLO violations.

Trade-offs:- Aggressive spot use lowers cost but increases complexity for checkpointing and higher risk of SLO impact.- Predictive models improve decisions but need retraining and can mispredict during market shocks — hence keep conservative safety buffer.

This design balances cost and reliability by combining forecasting, risk-based decisioning, priority-aware placement, and strict budget pacing.

MediumTechnical

30 practiced

Problem-solving: Suppose training time increased by 30% after switching dataset storage from local NVMe to network-attached storage. Describe steps to diagnose the bottleneck (compute, I/O, network), metrics you would collect, and short-term and long-term remediation strategies.

Sample Answer

Situation: Training time rose ~30% after moving dataset from local NVMe to network-attached storage (NAS). I’d treat this as an I/O vs compute vs network bottleneck diagnosis and remediation problem.

Diagnosis steps1. Reproduce & isolate: Run a controlled short training job with identical config on NVMe and NAS to confirm delta.2. Measure end-to-end and per-stage timings: data load/augmentation, CPU preprocessing, GPU compute, GPU idle/wait times.3. Profile storage I/O and network: capture throughput, latency, and IOPS during training.4. Check CPU/GPU utilization and stalls: look for low GPU utilization or high GPU kernel queueing/waiting.5. Verify file access pattern: many small files vs few large files, random vs sequential reads.6. Validate OS and mount settings: SMB/NFS options, caching, read-ahead, parallelism.

Metrics to collect- Throughput: MB/s read from storage (client and NAS side)- Latency: average and tail read latency (ms)- IOPS: reads/sec, writes/sec- GPU metrics: utilization %, memory utilization, GPU compute vs memcpy time (nvprof/nvmetrics)- CPU metrics: load, number of data loader workers, context switches- Disk metrics on NVMe (baseline) and NAS: queue depth, busy %- Network metrics: bandwidth usage, packet loss, retransmits, RTT- Application-level: batch load time, queue fill time, augmentation time, data loader starvation events

Short-term remediation (quick wins)- Increase data loader workers/threads and prefetch/buffer size to hide network latency.- Enable client-side caching or use a local read cache (tmpfs or NVMe cache) for hot shards.- Use larger batch of sequential reads or concatenate small files into tar/LMDB/TFRecords to reduce metadata overhead.- Tune mount options: enable read-ahead, larger rsize/wsize, async, and disable sync where safe.- Throttle network traffic from other jobs/enable QoS.

Long-term remediation (robust fixes)- Implement an efficient data pipeline: sharded/packed files (TFRecords, WebDataset), streaming with parallel prefetch and decompression.- Deploy a distributed cache layer (e.g., memcached, local SSD cache, or specialized data-serving like NVIDIA DALI + cache).- Move dataset to higher-performance storage (object store with optimized gateways, or NVMe over Fabrics) or colocate compute and storage (same rack/zone).- Adopt data versioning and smart sharding to reduce hotspots and enable locality-aware scheduling.- Monitor and alert on I/O and GPU utilization; automate scaling of cache and workers based on observed metrics.

Why: These steps separate whether GPUs are idle (I/O/network) or saturated (compute). Collecting both system and application metrics pinpoints the stage to optimize; short-term fixes hide latency, long-term fixes remove it.

HardSystem Design

21 practiced

Design a global, multi-region inference architecture for a real-time recommendation API that must serve 100M requests/day with p95 latency < 50ms worldwide. Include region placement, global load balancing, model synchronization strategy, consistency vs staleness trade-offs, and capacity planning considerations.

Sample Answer

Requirements & constraints:- 100M requests/day ≈ 1,157 req/s global average, peak factor 3 → ~3.5k rps.- p95 < 50ms end-to-end inference worldwide.- Real-time recommendation API — low compute per request, but model may be large; multi-region to reduce latency and provide availability.

High-level architecture:- Edge global LB (Cloud CDN / Anycast + Global Accelerator) → Regional API gateways → Regional inference clusters (GPUs/TPUs or CPU for small models) → Feature store & local cache → Async event bus for global state.- Regions placed in 6+ major geo zones (NA-east, NA-west, EU, APAC-east, APAC-south, SA) to keep median network RTT < 50ms target.

Region placement & traffic routing:- Use latency-based DNS/Anycast + health-aware global load balancer to route to closest healthy region.- Sticky routing optional: route by user geo + data residency. Failover to nearest healthy region on outage.

Model synchronization strategy:- Push-model: central model training/CI in preferred region → artifact registry → atomic staged rollout to regions.- Model binary + weights distributed via CDN or object store; regions fetch and atomically swap to avoid partial states.- For personalization embeddings: keep primary store in region of user; replicate asynchronously across regions via change-log (Kafka/CDC) for warm-starts.

Consistency vs staleness trade-offs:- Strong consistency for model binaries at rollout (atomic swap). For user features/embeddings accept eventual consistency with bounded staleness (e.g., last-write TTL < few minutes) to prioritize latency.- Use read-through local caches (Redis/Memory) with short TTLs for latest features; write-through to regional persistent store. On cross-region reads, prefer local fallback with best-effort sync to keep p95 low.

Capacity planning & autoscaling:- Size per-region capacity for 99th-percentile traffic per region + 20% buffer. Example: peak global 3.5k rps; if traffic splits unevenly, plan worst-case region ~30% → 1.05k rps. Benchmark model QPS per GPU (e.g., 200 qps/GPU) → need ~6 GPUs plus redundancy → round to 8 nodes.- Use GPU/CPU autoscaling based on queue length, latency SLOs; provide warm pool of instances to avoid cold-start latency (pre-warmed containers).- SLO-driven load shedding: degrade to smaller model or cached recommendations when load spikes, with circuit-breakers.

Operational concerns:- Observability: per-region latency, tail latencies, model version metrics, feature staleness telemetry.- CI/CD: canary deploy across small % of traffic, verify metrics then global rollout.- Security & compliance: encryption in transit, region-specific data controls.

Trade-offs summary:- Favor regional locality and eventual consistency for user features to meet latency SLOs; enforce strong atomicity for model updates. Invest in pre-warming and read caches to hit p95 <50ms globally.

Unlock Full Question Bank

Get access to hundreds of Capacity Planning and Resource Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.