Model Optimization, Debugging, and Performance Tuning Questions

Techniques for optimizing machine learning models in production, including hyperparameter tuning, architecture optimization (e.g., pruning, quantization, distillation), and hardware acceleration. Covers profiling and optimizing inference latency, throughput, memory usage, and energy consumption; debugging training instabilities and inference issues; diagnosing data-related problems; ensuring reproducibility and reliability in ML pipelines; and implementing serving optimizations (batching, caching, parallelization) within ML platforms and MLOps workflows.

EasyTechnical

60 practiced

Implement in Python a function analyze_mlp(layers: List[int]) that returns per-layer parameter counts and an estimated total number of multiply-accumulate operations (MACs) for a dense feedforward MLP described by layers (e.g., [784,256,128,10]). Assume each dense layer has a bias and count MACs as input_dim * output_dim (report total MACs). The implementation should be O(n) in number of layers and handle degenerate inputs.

Sample Answer

To analyze an MLP described by layers = [d0, d1, ..., dN], each dense layer i -> i+1 has parameters: weights of shape (d_i, d_{i+1}) and biases of length d_{i+1}. MACs for that layer = d_i * d_{i+1}. We'll scan layers once (O(n)) and compute per-layer params and MACs, handling degenerate inputs.

python

from typing import List, Tuple, Dict

def analyze_mlp(layers: List[int]) -> Dict[str, object]:
    """
    Returns:
      {
        "per_layer": [ {"from": d_i, "to": d_j, "params": p, "macs": m}, ... ],
        "total_params": int,
        "total_macs": int
      }
    Assumptions:
      - Each dense layer has weights (d_i * d_j) and biases (d_j).
      - MACs counted as d_i * d_j (bias adds no MACs here).
      - Degenerate layers (<=0 dims) are skipped.
    """
    per_layer = []
    total_params = 0
    total_macs = 0

    if not layers or len(layers) < 2:
        return {"per_layer": per_layer, "total_params": 0, "total_macs": 0}

    for i in range(len(layers) - 1):
        in_dim = layers[i]
        out_dim = layers[i + 1]
        # skip invalid/degenerate connections
        if not isinstance(in_dim, int) or not isinstance(out_dim, int):
            continue
        if in_dim <= 0 or out_dim <= 0:
            continue

        weight_params = in_dim * out_dim
        bias_params = out_dim
        params = weight_params + bias_params
        macs = weight_params  # by definition

        per_layer.append({"from": in_dim, "to": out_dim, "params": params, "macs": macs})
        total_params += params
        total_macs += macs

    return {"per_layer": per_layer, "total_params": total_params, "total_macs": total_macs}

Key points:- Single-pass O(n) over layers; space O(n) for per-layer list.- Edge cases handled: empty list, single layer, non-integer or non-positive dims are skipped.- Example: analyze_mlp([784,256,128,10]) yields per-layer params [200960,32896,1290] (weights+biases) and total_macs = 784*256 + 256*128 + 128*10.

HardSystem Design

53 practiced

Architect a scalable serving solution for a large language model (LLM) ~175B parameters requiring an average throughput of 10 tokens/sec per request and P95 tail latency constraints. Consider model sharding (tensor/model parallelism), KV-cache sharding for streaming generation, batching strategies, quantization, memory-tiering, multi-region failover, and cost/latency trade-offs. Describe monitoring and fallback mechanisms to smaller models in case of overload.

Sample Answer

Requirements and constraints:- Serve 175B LLM at avg 10 tokens/sec per request with strict P95 latency (e.g., <200–500ms per token depending on SLA) and high availability across regions. Support streaming generation, graceful overload handling, cost efficiency.

High-level architecture:- Frontend API gateway + request router → Inference cluster (region-local) → KV-cache store (sharded, in-GPU + backing DRAM/NVMe) → Autoscaler + orchestration (Kubernetes + custom scheduler) → Metrics/alerting and fallback controller.

Model sharding and placement:- Use pipeline + tensor parallelism (e.g., Megatron-LM style) to split 175B across N GPUs: tensor parallelism for operator parallelism, pipeline parallelism to improve utilization. Favor 8–16-way TP and 4–8-way PP depending on GPU memory (A100/H100) so each replica fits with quantized weights.- Use model replicas for availability and to lower P95 by avoiding cross-node hops when possible. Place replicas across AZs for SLA; across regions for failover.

KV-cache sharding for streaming:- Shard KV-cache across GPUs matching pipeline stages to keep key/values local to attention computation. For streaming long contexts, store active KV-cache primarily in GPU memory; for less-recent tokens spill to host DRAM or NVMe with async prefetch.- Use partitioned-addressing: each request maps to a small subset of GPUs for KV access; fetches for newly needed segments are overlapped with compute.

Batching strategies:- Dynamic micro-batching with adaptive latency-aware batching: group requests within a tight time window (e.g., 5–30ms) up to max batch size constrained by memory and P95. Use priority queues and deadline-aware schedulers to ensure tail-sensitive requests are prioritized. For streaming, keep per-token batching to maximize throughput without violating latency SLOs.- Use per-replica batching and cross-replica cooperative batching to increase GPU utilization while bounding latency.

Quantization and memory-tiering:- Quantize weights to 4-bit (or mixed 8/4-bit with LLM.int8 / QLoRA style) to reduce memory and increase effective batch sizes while maintaining quality. Keep critical layers in higher precision (fp16) as needed.- Memory tiering: hot model weights + KV-cache in GPU HBM; warm KV-cache and embeddings in host DRAM; cold checkpoints on local NVMe or remote object store. Use async prefetch and background eviction.

Multi-region failover and traffic shaping:- Active-active across primary regions for low-latency routing; geo-DNS + latency-based routing. Maintain cold warm replicas in secondary regions to start quickly (warm pools with weights loaded but minimized active KV-cache).- On region failure, traffic reroutes with progressive ramp and circuit-breakers; use request throttling and degrade gracefully to smaller models.

Cost/latency trade-offs:- Maintain a pool of high-performance GPUs for low-latency SLO traffic and cheaper spot/preemptible nodes for background/batch requests.- Tune replica count vs batch window to balance utilization vs latency. More replicas reduce cross-queue wait but increase cost.

Monitoring and observability:- Metrics: per-token latency (p50/p95/p99), batch sizes, GPU utilization, memory pressure, KV-cache hit/miss, network serialization time, error rates, queue lengths.- Traces: distributed tracing from API to GPU stage with timing per pipeline stage and KV fetch times.- Alerts: high P95, KV miss spikes, queue growth, GPU OOMs. Run automated canary tests with synthetic requests and quality checks.

Fallback and overload handling:- Graceful degradation policy: 1) If overloaded, reduce per-request max tokens and tighten generation temperature. 2) Route low-priority requests to smaller distilled models (e.g., 7B/13B) or retrieval-augmented generation backends. 3) If latency SLOs threatened, shed load: reject with informative error, queue with ETA, or return partial streaming tokens.- Automated model-quality gating ensures fallbacks are acceptable; A/B monitored for hallucination spikes.

Final notes:- Prototype with real workloads to tune batch windows, KV eviction policies, and quantization trade-offs. Use load-testing to calibrate autoscaler thresholds and ensure P95 under peak. Continuous profiling and incremental rollouts keep performance predictable.

EasyTechnical

59 practiced

What is mixed-precision training? Explain its benefits for training speed and memory, common numerical issues such as overflow/underflow, and common mitigations (loss scaling, using BF16 vs FP16, ensuring deterministic ops). When would you choose BF16 over FP16 for a production training job on modern GPUs or TPUs?

Sample Answer

Mixed-precision training means using lower-precision floating point (typically FP16 or BF16) for most model tensors and compute (activations, weights, gradients in parts) while keeping crucial values (usually a master copy of weights and some accumulators) in FP32. This lets you leverage faster tensor cores and reduce memory use without losing final model quality.

Benefits:- Speed: Tensor cores on modern GPUs/TPUs deliver much higher throughput for 16-bit ops, reducing training time.- Memory: Half-precision halves activation and parameter memory, enabling larger batch sizes or bigger models.

Numerical issues:- Overflow/underflow: FP16 has a much smaller exponent range than FP32, so very large or tiny values can overflow to Inf or underflow to 0. This primarily affects gradients and accumulation.- Reduced precision: Lower mantissa bits cause higher rounding error, which can destabilize training for some models.

Common mitigations:- Loss scaling: Multiply the loss by a scale factor before backprop so gradients are larger (avoid underflow), then unscale before updating. Use static or dynamic loss scaling (dynamic adjusts scale when Inf/NaN detected). Frameworks: NVIDIA Apex/torch.cuda.amp, TF mixed precision handle this.- FP32 master weights / accumulators: Keep an FP32 copy of model weights to apply updates and accumulate small gradient updates, preventing precision loss.- BF16 vs FP16: BF16 has the same exponent range as FP32 but fewer mantissa bits. That means BF16 avoids overflow/underflow issues common to FP16 while still reducing memory and enabling fast compute on supported hardware.- Deterministic ops: Mixed precision may expose non-determinism (reordering, reduced precision). If reproducibility is required, enable deterministic kernels (framework flags) and be aware that deterministic runs may be slower.

When to choose BF16 over FP16:- Choose BF16 when hardware supports it (TPUs natively, and modern GPUs like A100/H100 with BF16 tensor core support) and you want to avoid loss-scaling complexity or numerical instability—BF16 often trains stably without aggressive loss scaling because of its FP32-like exponent range.- Prefer FP16 when BF16 isn't supported or when you need slightly better mantissa precision behavior in some edge cases—though FP16 generally requires careful loss scaling and FP32 master weights.- In production training for large models (NLP, transformers) on TPUs or BF16-capable GPUs, BF16 is often the default choice: it simplifies pipeline (less tuning), is fast, and reduces memory while maintaining stability.

Practical note: use framework mixed-precision utilities (torch.cuda.amp or tf.keras.mixed_precision) which handle casting, master weights, and dynamic loss-scaling for you.

MediumTechnical

59 practiced

Compare TensorRT optimization to ONNX Runtime optimization and XLA compilation. For inference acceleration across GPU, CPU, and edge devices, explain when you'd pick TensorRT, ONNX Runtime, or XLA, and what trade-offs regarding operator support, dynamic shapes, and maintenance each choice entails.

Sample Answer

Brief summary:TensorRT, ONNX Runtime (ORT), and XLA are complementary tools for inference acceleration with different targets and trade-offs. Choose based on hardware (NVIDIA GPU vs CPU vs edge ASIC), model characteristics (dynamic shapes, custom ops), and maintenance cost.

When to pick:- TensorRT: Best for high-throughput, low-latency inference on NVIDIA GPUs (including Jetson). Use when you can convert model to supported ops and accept an optimization/precision calibration step (FP16/INT8). Great for production GPU services needing max performance.- ONNX Runtime: Flexible multi-backend (CPU, GPU, TensorRT EP, OpenVINO, NNAPI). Good for cross-platform deployments and edge devices where portability matters. Easier integration for models exported to ONNX and when you need fallbacks for unsupported ops.- XLA: Compiler-focused for JAX/TF; good for improving CPU/GPU performance where you control the computation graph and use TF/JAX idioms. Best for training-to-inference parity in TensorFlow ecosystems and for TPU deployments.

Operator support:- TensorRT: Excellent performance for common conv/FC/BN/relu patterns; limited or evolving support for niche ops; custom plugins required for unsupported ops (extra engineering).- ORT: Broad coverage via ONNX spec; extensions via custom ops or execution providers; generally easier to execute models with varied ops.- XLA: Supports ops expressible in TF/JAX graphs; some high-level ops get fused; unsupported or Python-level ops can be hard to compile.

Dynamic shapes:- TensorRT: Historically favors static shapes for max optimization; latest versions improve dynamic shape handling but may reduce some optimizations and increase engine complexity.- ORT: Handles dynamic shapes better out-of-the-box (ONNX supports symbolic dims); better for variable-length inputs (NLP).- XLA: Can compile for dynamic shapes but may require staged compilation or fallback; works well when shapes are known per-trace (JIT trace model).

Maintenance and lifecycle:- TensorRT: Higher engineering upkeep (conversion, calibration, plugin maintenance) but fewer runtime surprises and top performance on NVIDIA hardware.- ORT: Lower maintenance for heterogeneous deployment; active ecosystem and multiple execution providers reduce vendor lock-in.- XLA: Tightly coupled to TF/JAX versions—can be sensitive to framework upgrades; good if your stack is TF/JAX-centric and you can maintain compiler traces.

Trade-offs summary:- Max GPU performance: TensorRT (cost: conversion, plugins, static-shape tuning).- Portability and ease across CPU/edge: ONNX Runtime (cost: potential slower peak perf than vendor compilers).- Framework-level fusion and TPU/TPU-like optimization or JAX/TF workloads: XLA (cost: coupling to framework and potential compile complexity).

Recommendation: Prototype with ONNX Runtime for portability; if NVIDIA GPU latency matters, benchmark TensorRT after model conversion and be prepared to implement plugins. Use XLA when your codebase primarily uses TF/JAX and you need native compiler-level optimizations.

MediumTechnical

51 practiced

Explain causes of vanishing and exploding gradients in deep networks and list practical mitigations. Cover architecture-level solutions (residual connections, normalization), initialization schemes (Xavier/He), optimizer choices, gradient clipping, and when to change depth or width to stabilize training.

Sample Answer

Vanishing and exploding gradients occur when backpropagated gradients shrink toward zero or grow exponentially across many layers, harming learning. Causes:- Activation functions with gradients <1 (sigmoid/tanh) cause repeated multiplicative shrinkage; ReLU reduces vanishing but can deaden neurons.- Poor weight initialization leads to signals shrinking or amplifying layer-by-layer.- Deep sequential multiplications (many layers or RNN timesteps) amplify numerical instability.- Large learning rates or poorly conditioned loss landscapes can cause explosions.

Practical mitigations:

Architecture-level- Residual/skip connections (ResNets): provide identity paths so gradients bypass nonlinearities, stabilizing deep nets.- Normalization: BatchNorm / LayerNorm / GroupNorm reduce internal covariate shift, keep activations in favorable ranges, and smooth optimization.- Use activation choices like ReLU/LeakyReLU/GELU to maintain non-vanishing gradients.

Initialization- Xavier/Glorot for symmetric activations, He initialization for ReLU-family—choose variance scaling so forward/backward signal variances stay ~constant.

Optimizers & training- Adaptive optimizers (Adam, RMSprop) help with gradient scaling and noisy updates; combine with weight decay.- Gradient clipping (global norm or value clipping) prevents exploding gradients in RNNs or high-loss regions.- Learning rate schedules and warmup reduce initial instability.

Model capacity choices- If vanishing persists, reduce depth or add skip connections; if instability stems from too narrow layers, increase width to improve signal propagation. For very deep models, prefer residual architectures plus careful init/norm.

When to change depth/width- If training loss plateaus with near-zero gradients → likely vanishing: add residuals, reduce depth or change activations; increase width if underfitting.- If gradients blow up or loss diverges → exploding: reduce learning rate, add clipping, try smaller initialization, or reduce depth.

Combine these techniques: e.g., He init + BatchNorm + ResNet + Adam + gradient clipping is a common robust recipe.

Unlock Full Question Bank

Get access to hundreds of Model Optimization, Debugging, and Performance Tuning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.