Covers designing and operating machine learning systems to handle growth in data volume, model complexity, and traffic. Topics include distributed training strategies such as data parallelism, model parallelism, and pipeline parallelism; coordination and orchestration approaches like parameter servers, gradient aggregation, and framework tools such as PyTorch distributed, Horovod, and TensorFlow strategies; data pipeline and I O considerations including sharding, efficient formats, preprocessing bottlenecks, streaming and batch ingestion; serving and inference scaling including model sharding, batching for throughput, autoscaling, request routing, caching, and latency versus throughput tradeoffs. Also includes monitoring, profiling, checkpointing and recovery, reproducibility, cost and resource optimization, and common bottleneck analysis across network, storage, CPU preprocessing, and accelerator utilization.
HardTechnical
31 practiced
You must upgrade a model architecture by renaming layers and changing some layer sizes but want to reuse as many weights as possible from older checkpoints. Describe a robust mapping strategy to match old tensors to new ones by name and shape, fallback rules for partial matches, initialization strategies for unmatched layers, and validation steps to ensure transferred weights behave sensibly.
Sample Answer
Strategy summary: build a deterministic, auditable mapping pipeline that matches tensors by canonicalized names and shapes, applies controlled fallbacks for partial matches, initializes unmatched weights with principled schemes, and validates transfer with unit and behavioral tests.1) Canonicalize names and index metadata- Strip version prefixes, module reordering tokens, and common synonyms (e.g., "ffn"↔"mlp", "ln"↔"layernorm") to produce canonical names.- Record original full name, canonical name, shape, dtype, and checkpoint source.2) Exact-match pass- Map tensors where canonical name and shape exactly match. Mark mapped.3) Shape-compatible fallback rules (in order)- Prefix/suffix remapping: allow mapping if canonical names differ only by added/removed module wrappers (e.g., "encoder.block.0" → "encoder.blocks.0").- Name-equals-size fallback: if canonical name matches but shapes differ, attempt partial mapping by aligning overlapping dimensions: - For linear weights: if new dim <= old dim, take corresponding submatrix ([:new_out, :new_in]); if new > old, copy old into top-left and initialize remainder. - For convolution: center-crop or pad kernels along kernel dims. - For embeddings: if vocab shrunk, slice; if expanded, copy old rows and initialize new rows.- Structural reuse: map attention QKV by splitting/concatenating if old stored combined matrix (e.g., W_qkv) and new stores separate W_q, W_k, W_v; reverse if needed.4) Rejection and logging- If no safe fallback applies, leave unmapped. Log decisions with diffs and a short rationale file for reproducibility.5) Initialization strategies for unmatched/partial parts- Linear/Conv: Kaiming/Glorot depending on activation (ReLU → Kaiming, tanh → Xavier). Scale partial copies to preserve variance (rescale copied block by sqrt(old_fan_in/new_fan_in) if fan-in changed).- Biases/LayerNorm: initialize biases to zero; scale/gamma to 1.0.- Embeddings: random normal with small std (0.02) or use averaged vector from existing embeddings to preserve centroid.- Special: for newly expanded attention heads, initialize new heads by copying an existing head’s weights + small noise to break symmetry.6) Validation steps- Parameter-level checks: verify expected mapped fraction, layer-wise Frobenius norm comparisons, and histogram differences logged.- Unit tests: run a fixed set of deterministic inputs through old and upgraded model (with old weights mapped) for shared subgraph outputs; compute max absolute and cosine similarity per tensor.- Sanity training/forward checks: run a short n-step fine-tune on a small dataset; confirm loss decreases and gradients flow into newly initialized parameters.- Downstream metrics: run end-to-end evaluation on a small validation set; expect performance close to baseline (within sensible deltas). If large regressions appear, inspect layers with largest norm changes.- Safety: run gradient-checks (finite grads), ensure no exploding activations (track activation statistics), and verify embeddings normalization if expected.7) Automation & tooling- Expose mapping as a reproducible script/config with mapping rules, logging, dry-run mode, and --strict flag to fail on unmapped tensors.- Keep a migration manifest storing mapping decisions and seed for random init.This approach maximizes reuse, makes partial transfers safe via controlled copying/padding, documents choices, and validates functionally before deployment.
MediumTechnical
33 practiced
GPU utilization is 30% while CPU is 100% because image decoding and augmentation are the bottleneck. Provide a prioritized list of fixes (code-level and infrastructure-level) to reduce CPU bottleneck and increase GPU utilization. For each fix estimate implementation complexity, likely uplift, and discuss trade-offs like reproducibility and latency.
Sample Answer
Priority 1 — Move decoding & augmentations to GPU (NVIDIA DALI / kornia / OpenCV-CUDA)- What: Replace CPU PIL/NumPy pipelines with DALI or GPU-accelerated ops so decoding + augment happen on GPU stream.- Complexity: Medium (integration + minor refactor)- Likely uplift: High (30–60% increase in GPU utilization; CPU drops substantially)- Trade-offs: Must ensure deterministic pipelines (set seeds, deterministic DALI ops) — some ops non-deterministic. Latency per batch may drop; initial integration/debugging time.Priority 2 — Use asynchronous prefetching + multi-threaded/worker loaders- What: Increase num_workers, use pinned memory, prefetch queue (PyTorch DataLoader or TF autotune).- Complexity: Low- Likely uplift: Medium (10–25%)- Trade-offs: Higher memory usage; potential nondeterministic ordering unless controlled; small increase in startup latency.Priority 3 — Convert dataset to efficient on-disk format (TFRecord/LMDB/recordIO) + memory-map- What: Avoid many small file reads; use sharded binary format and sequential reads.- Complexity: Medium- Likely uplift: Medium (10–30%) especially if IO bound- Trade-offs: Preprocessing/convert time and storage; needs sharding for parallel reads; reproducibility unaffected.Priority 4 — Move heavy CPU augmentations to faster libraries / vectorize- What: Replace Python loops with compiled ops (OpenCV, numba, C++ extension) or batch augmentations.- Complexity: Medium- Likely uplift: Medium (10–20%)- Trade-offs: Development time; ensure numerical parity; deterministic behavior depends on implementation.Priority 5 — Scale infrastructure: more vCPUs / dedicated preprocessing nodes- What: Increase CPU cores on instance or run separate data-prep servers colocated with GPUs.- Complexity: Low–Medium (config)- Likely uplift: Variable (10–50% depending on original CPU bottleneck)- Trade-offs: Increased cost; additional network/coordination complexity; reproducibility unaffected.Priority 6 — Caching decoded images / use warm-up & persistent workers- What: Cache decoded tensors in RAM or NVMe, reuse across epochs, keep workers pinned.- Complexity: Low–Medium- Likely uplift: Small–Medium (5–25%), big for repeated epochs- Trade-offs: Memory usage; stale cache if dataset changes; improves latency.Priority 7 — Batch-level and mixed-precision tuning- What: Increase batch size to amortize CPU overhead; use AMP to shorten GPU compute, exposing more capacity for throughput.- Complexity: Low- Likely uplift: Small–Medium- Trade-offs: Affects convergence, memory footprint, possible latency per batch increase.Notes on reproducibility & latency:- Determinism: Bind worker seeds, disable nondeterministic ops, use deterministic algorithms if exact reproducibility required (may reduce throughput).- Latency vs throughput: Many fixes (larger batch, prefetch) raise throughput but can increase single-batch latency. For low-latency serving prefer smaller batches + warm workers + GPU-accelerated ops.- Measure: Profile before/after (nvidia-smi, perf counters, py-spy, DALI profiler) and iterate; prefer GPU-acceleration first for best ROI.
HardTechnical
28 practiced
Training throughput drops 40% when scaling from 64 to 512 GPUs. Provide a step-by-step profiling and root-cause analysis plan covering network, storage I/O, CPU preprocessing, GPU kernel utilization, and scheduler queuing. For each domain list key metrics, low-level tools to use (e.g., nvidia-smi, nsight, iostat, tcpdump), and likely mitigations.
Sample Answer
Approach: Run an incremental, hypothesis-driven profiling sweep from single-node to full 512-GPU to isolate where throughput diverges. For each domain list what to measure, low-level tools, interpretation, and likely mitigations.1) Baseline & orchestration- Metrics: per-GPU batch/sec, global throughput, job start/step latency, task queue lengths.- Tools: scheduler logs (Slurm/Yarn/K8s), custom instrumentation, Prometheus/Grafana.- Mitigations: ensure fair scheduling, probe stragglers, align worker counts to resources.2) GPU kernel utilization- Metrics: GPU utilization %, SM occupancy, achieved TFLOPS, memory bandwidth utilization, kernel launch latency, CUDA streams idle time.- Tools: nvidia-smi, nvprof / Nsight Systems, Nsight Compute, DCGM exporter.- Interpretation: low SM occupancy/low fp32 throughput → small per-GPU batch or kernel serialization.- Mitigations: increase per-GPU batch size, fuse kernels, enable mixed precision, adjust data-parallel sync frequency (grad accumulation).3) CPU preprocessing- Metrics: CPU utilization per node, context switches, queueing latency, disk IO wait, number of data loader workers, serialization overhead.- Tools: top/htop, perf, pidstat, strace, iostat, blktrace.- Interpretation: high CPU wait or many blocked workers → preprocessing bottleneck.- Mitigations: increase loader workers, move transforms to GPU, use shared memory, optimize Python GIL hotspots (multiprocessing or C++).4) Storage I/O- Metrics: read throughput (MB/s), IOPS, latency P50/P99, cache hit ratio, starvation periods.- Tools: iostat, blktrace, fio, sar, s3 CLI + aws s3 cp profiling, ceph/radosctl metrics.- Interpretation: low sustained MB/s or high P99 latency → I/O starving GPUs.- Mitigations: use local SSD cache, parallel prefetch, sharded dataset, TFRecords/LMDB, increase read-ahead, use faster storage tier.5) Network (inter-node and parameter sync)- Metrics: interconnect throughput (GB/s), packet loss, retransmits, latency, all-reduce time, NCCL op times.- Tools: ifstat, iperf, tcpdump, perfquery for InfiniBand, nvidia-smi topo, nccl-tests, collect NCCL logs.- Interpretation: rising all-reduce time or network saturation at scale → communication bottleneck.- Mitigations: switch to hierarchical all-reduce (intra-node then inter-node), gradient compression/quantization, overlap compute+comm, use tuned NCCL params, upgrade fabric or topology-aware placement.6) Scheduler queuing & stragglers- Metrics: task start jitter, imbalance in per-GPU step times, node-level slowdowns.- Tools: scheduler metrics, application logs, Prometheus histograms.- Interpretation: skewed distribution indicates stragglers (noisy neighbors, thermal throttling).- Mitigations: node fencing, topology-aware pod placement, autoscaling with padding, checkpoint/retry.Execution plan:- Stage 1: run microbenchmarks (nccl-tests, fio, iperf) across target nodes.- Stage 2: collect simultaneous traces (Nsight Systems + perf + iostat + tcpdump) during a reduced job (e.g., 8→64→256→512 GPUs).- Stage 3: correlate timelines (all-reduce peaks vs GPU idle vs disk reads) to identify primary cause.- Stage 4: apply mitigations iteratively (cache, grad-accumulation, NCCL tuning) and re-measure.Key principle: correlate time-series across domains to find where GPU-ready time is lost; fix highest-impact bottleneck first.
MediumTechnical
31 practiced
Explain how converting a dataset of millions of small JPEG files into sharded binary formats (TFRecord, sharded tar, or Parquet for tabular features) improves training throughput. Discuss trade-offs around ingestion flexibility, incremental updates, random access patterns, host CPU parsing cost, and a migration plan.
Sample Answer
Converting millions of small JPEGs into sharded binary formats (TFRecord, sharded tar, Parquet for tabular features) greatly improves training throughput by reducing I/O overhead, lowering file-open syscalls, enabling larger sequential reads, and allowing efficient prefetching and parallelism. A few concrete benefits:- Fewer metadata reads and seeks: single large shard => contiguous disk reads and better read-ahead.- Better host CPU utilization: centralized, batched decoding/parsing (or offloading to accelerators) reduces per-file overhead.- Improved network utilization: object stores handle fewer large objects, reducing request overhead and latency.Trade-offs:- Ingestion flexibility: single-file-per-image is easy to update individually; sharded binaries make small updates harder (need rewriting or append strategies).- Incremental updates: support via time-based shards, delta shards, or log of new small-file shards; otherwise full rewrite costly.- Random access patterns: Parquet supports efficient columnar reads for tabular features; TFRecord/tar require index files or sharding strategy to achieve per-sample access.- Host CPU parsing cost: you can precompute and store already-decoded (or resized) tensors in shards to shift work from training hosts; otherwise parsing still required but can be batched with multi-threaded readers.- Operational complexity: format choice affects tooling, validation, and reproducibility.Migration plan:1. Benchmark baseline: measure throughput, CPU, and storage ops on current setup.2. Prototype: convert a representative subset into TFRecord and sharded tar; produce Parquet for labels/metadata.3. Measure benefits and costs: throughput, latency, storage, and CPU parsing.4. Implement tooling: bulk converter with resumable workers, create shard-index and manifest, include checksums.5. Rollout strategy: start training on read-only archived shards; implement incremental ingestion by writing new delta shards and update manifests; maintain backward compatibility by supporting both small-file and shard readers.6. Automate: CI for data validation, monitoring for read errors and performance regressions.7. Full migration: once stable, deprecate small-file pipeline and remove rewrite paths.This approach balances throughput gains with operational flexibility and supports incremental adoption with measurable checkpoints.
MediumTechnical
31 practiced
List practical steps to make distributed training reproducible across runs and clusters: set and record seeds for all RNGs, capture and checkpoint RNG state, use deterministic ops where available, pin CUDA/cuDNN/library versions, log dataset ordering and sharding, and capture hardware topology and hyperparameters. Explain limitations when using mixed-precision and nondeterministic kernels.
Sample Answer
Practical steps to make distributed training reproducible across runs and clusters:- Set and record seeds everywhere: seed Python's random, NumPy, framework RNGs (PyTorch/TensorFlow), and any third‑party libs. Store the seed value in experiment metadata.- Capture and checkpoint RNG state: save framework RNG state (e.g., torch.get_rng_state(), torch.cuda.get_rng_state_all()) alongside model checkpoints so resumed runs continue deterministically.- Use deterministic ops where available: enable deterministic backends (torch.use_deterministic_algorithms(True)), avoid known nondeterministic operators, and prefer deterministic alternatives.- Pin CUDA/cuDNN and library versions: record exact versions (CUDA, cuDNN, NCCL, MKL, OS, Python, framework) and use containerization (Docker) or environment lockfiles to reproduce runtime.- Log dataset ordering and sharding: record file listing, random shuffle seed, sharding strategy per rank (offsets, workers, batch sampler states) and save a manifest or checksums of data partitions.- Capture hardware topology and distributed config: log GPU topology, number of nodes, ranks-per-node, NCCL env vars, network fabric details and hyperparameters (batch size, LR schedule, augmentation params) in experiment logs.- Automate reproduction: provide scripts/containers that reinstantiate the environment, RNG state, and data sharding before training.Limitations and caveats:- Mixed precision (AMP) can introduce nondeterminism due to different rounding, fused kernels, and loss-scaling behavior; results may vary slightly and some ops behave differently in FP16 vs FP32.- Some high-performance kernels (fused/cuDNN/NCCL collectives) are inherently nondeterministic for speed/parallelism reasons; even with seeds and deterministic flags, exact bitwise reproducibility across different hardware, driver versions, or different numbers of ranks may be impossible.- Practical recommendation: aim for statistical reproducibility (consistent metrics/trends) rather than strict bitwise equality; document tolerances and run multiple seeds to estimate variance.
Unlock Full Question Bank
Get access to hundreds of AI System Scalability interview questions and detailed answers.
