Performance Engineering and Cost Optimization Questions

Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.

MediumTechnical

49 practiced

Tail latency (p99.9) is causing SLAs to be violated even though p95 is fine. Describe techniques to diagnose and mitigate tail latency in ML serving—cover queuing/backpressure, request hedging, prioritized scheduling, circuit breakers, and resource isolation—and recommend which to try first with justification.

Sample Answer

Situation: Your ML service shows healthy p95 but p99.9 breaches SLA — intermittent long tails impacting a small fraction of requests.

Approach: Diagnose where the long tails originate (CPU/GPU contention, GC, I/O, retries, queuing bursts) and apply layered mitigations in order of low-risk to higher-infrastructure changes.

Diagnosis steps:- Correlate slow requests with host, model version, input size, client, and time-of-day using traces and histograms (per-endpoint latency, queue length, CPU/GPU/memory, GC, socket stats).- Instrument end-to-end tracing (distributed traces) and capture CPU/GPU utilization and per-request resource metrics for p99.9 samples.- Reproduce with load tests that mimic production burstiness.

Mitigations (what they do and when to use):1) Queuing & backpressure — enforce a bounded request queue and return 429 when full. Prevents request pile-up that amplifies tails. Low-risk, first-line control to stop overload cascades.2) Resource isolation — dedicate cores, GPUs, or containers for latency-sensitive models; use cgroups, node pools, or dedicated inference nodes. Reduces noisy-neighbor interference causing sporadic long tails.3) Prioritized scheduling — schedule high-priority (SLA) requests ahead of bulk/low-priority ones. Useful when mixed workloads share inference capacity.4) Request hedging — issue duplicate requests to other replicas when a request exceeds a short threshold; reduces p99.9 at cost of extra load. Use carefully after backpressure and isolation to avoid overload.5) Circuit breakers — detect failing/slow upstreams and short-circuit to fast failover or degraded models to protect system health. Use for persistent failures and to avoid cascading retries.

Recommendation (order to try):1. Implement bounded queues + backpressure (fast to deploy, prevents amplification).2. Add tracing and correlate p99.9 causes (diagnostic parallel step).3. Apply resource isolation (pin cores, reserve GPUs, scale replicas) to remove noisy neighbors.4. Add prioritized scheduling for mixed workloads.5. If tails persist, add hedging with conservative timeouts and circuit breakers for failing replicas.

Metrics to validate: p99.9 latency, request rejection rate (429s), system utilization, cost per inference, and error rates. Iterate: monitor trade-offs (hedging increases cost; backpressure increases client 429s) and choose combinations that restore SLA with acceptable cost.

EasyTechnical

59 practiced

Compare online (real-time) feature computation and batch feature computation for online ML inference. For a user-personalization model that needs 100ms p95 latency, list trade-offs in latency, freshness, cost, and operational complexity and recommend when to use hybrid approaches.

HardTechnical

55 practiced

Design a scheduler for large background retraining jobs that minimizes peak compute costs and avoids impacting online services. Include constraints: retraining can use spot instances but must finish within 48 hours, online SLOs must not degrade, and retraining can be preempted and resumed. Outline scheduling algorithm and fault tolerance mechanisms.

Sample Answer

Requirements & constraints:- Retraining jobs (batch) must complete within 48h, may run on spot instances (preemptible), can be paused/resumed, must not degrade online SLOs (latency, throughput).- Goal: minimize peak cloud compute spend while meeting deadlines and safety for online services.

High-level design:- Central Scheduler service that takes jobs with: resource profile (vCPU, RAM, GPU), estimated runtime, priority, deadline, checkpoint frequency.- Cluster split: reserved capacity for online services (hard cap), and a flexible retrain pool backed primarily by spot instances + small on-demand fallback.- Autoscaler that horizontally scales retrain pool based on available spot capacity, cost signals, and current online load.

Scheduling algorithm (deadline-aware, cost-optimized):1. Normalize job sizes to "work units" = vCPU-hours (adjust for GPU).2. For each job compute latest start = deadline - estimated_runtime and slack = latest_start - now.3. Use EDF (earliest deadline first) weighted by cost sensitivity: sort jobs by (slack / work_units). Smaller ratio → schedule sooner.4. Bin-packing with time dimension: place work units into future time slots (hourly) up to target concurrency limit set by online-safe cap. Prefer scheduling into low-cost spot-availability windows (from cloud spot-price API or historical availability model).5. If spot capacity insufficient, attempt checkpoint-based preemption and resume; if projected spot shortfall threatens deadline, dynamically provision on-demand capacity for the minimal remaining work.

Fault tolerance & preemption handling:- Mandatory periodic checkpoints (user-configurable). Jobs run inside containers that persist checkpoints to durable storage (e.g., S3/GCS + metadata in DB).- Use optimistic task granularity: split jobs into many small tasks (e.g., epoch-level or shard-level). Each task checkpointed; retry idempotently.- On spot termination event: cloud hook triggers scheduler to reschedule remaining tasks immediately. Use exponential backoff for retries with jitter.- Metadata store records job state, progress, and lineage; scheduler can reconstruct tasks after complete failure.- For long-running stateful training (e.g., TensorFlow), use model checkpointing + deterministic resume; for non-deterministic, record RNG seeds and data shard ordering.

Protecting online SLOs:- Enforce hard resource floor: autoscaler reserves N% of cluster (or set by SLO) for online services and monitors tail latency; if online latency approaches threshold, throttle retrain concurrency using token bucket: tokens consumed per retrain VM; reduce tokens when latency rises.- Use cgroups/containers to limit CPU/IO or run retrain on separate node pools with different rack/network isolation to avoid noisy-neighbor IO.- Circuit breaker: if SLO degradation detected, immediately drain retrain pool to safe level and pause new retrain tasks until recovery.

Cost optimization strategies:- Prefer spot instances across diverse instance types (mix instances) using spot fleet + bid strategy.- Time-shift flexible work into low-cost windows predicted by historical spot prices.- Consolidate checkpoints to avoid excess storage costs; compress model checkpoints and use lifecycle rules.

Operational considerations:- Monitoring: job metrics, spot interruption rate, SLO metrics; alerts on missed deadlines or high preemptions.- SLAs: configurable priorities; critical jobs get hybrid on-demand fallback.- Testing: chaos experiments (inject spot terminations, increased online load) to validate resilience.

This design balances cost (spot usage, time-shifting, packing) with reliability (frequent checkpoints, small tasks, on-demand fallback) and preserves online SLOs via hard caps, throttling, and node-pool isolation.

EasyTechnical

61 practiced

Explain model quantization and model pruning. For each technique, describe when it is appropriate to apply it, the impact on latency, memory footprint, inference throughput, and potential accuracy loss. Provide examples of models or layers where quantization is risky.

Sample Answer

Model quantization and pruning are complementary compression techniques that reduce model size and improve inference efficiency, with different trade-offs.

Quantization- What: Reduce numeric precision of weights/activations (e.g., float32 → float16, int8, or lower) or use mixed precision.- When to apply: When deployment hardware supports lower-precision arithmetic (mobile, edge, CPUs with AVX512-VNNI, NPUs). Good for large feedforward/CNN models with forgiving numeric ranges.- Impact: - Latency: Usually reduces per-inference latency (hardware-dependent); int8 can give 2–4x speedups on supported hardware. - Memory footprint: Significant (2–4x) reduction in model size and lower activation memory. - Inference throughput: Increases (more ops per second) if hardware supports fast integer ops. - Accuracy loss: Often minimal for int8 on well-calibrated models; can increase for aggressive quantization (4/2-bit).- Risky models/layers: - Small transformer layers with layernorm and softmax (sensitivity to precision). - RNNs/LSTMs with hidden state dynamics. - Layers with very small or highly skewed weight distributions, batchnorm folding issues, or outputs requiring high dynamic range (softmax before argmax). - Quantizing embeddings or tokenizers aggressively can hurt NLP quality.

Pruning- What: Remove weights, neurons, or channels (unstructured or structured) that contribute little, then fine-tune.- When to apply: When memory or compute budget is tight and you can retrain/fine-tune; structured pruning preferred for runtime gains.- Impact: - Latency: Unstructured pruning reduces FLOPs but may not reduce latency without sparse-supporting hardware/libraries; structured pruning (channel/filter removal) reduces latency and model size. - Memory footprint: Lowers parameter storage; sparse formats help but add indexing overhead. - Inference throughput: Improves if runtime supports pruned topology (structured) or sparse execution; otherwise limited. - Accuracy loss: Depends on sparsity; low-to-moderate pruning (20–50%) often acceptable with fine-tuning; aggressive pruning risks larger drops.- Best practices: Combine quantization and structured pruning, profile on target hardware, validate with representative data, and always fine-tune after compression.

MediumTechnical

53 practiced

Describe memory optimization techniques to reduce peak memory usage during batched inference of deep learning models. Discuss zero-copy I/O, memory pooling, tensor memory formats, garbage collection tuning, and model sharding. For each, give one scenario where it's most effective.

Sample Answer

When serving batched inference at scale, reducing peak memory prevents OOMs and improves throughput. Below are five techniques, what they do, why they help, and one scenario where each is most effective.

1) Zero-copy I/O- What/Why: Avoids copying input data between user-space buffers and framework tensors by mapping input payloads (e.g., memory-mapped files, DMA, or framework APIs accepting buffers) directly to model inputs.- Benefit: Lowers transient copies that spike memory during batch assembly.- Best scenario: High-throughput image/video inference where large input batches are read from disk or network (e.g., video frames via mmap or shared memory).

2) Memory pooling- What/Why: Preallocate reusable buffers for tensors to avoid repeated malloc/free and fragmentation; allocate pools per size/class.- Benefit: Reduces peak fragmentation and expensive allocations during bursts.- Best scenario: Real-time inference with variable but repeated tensor shapes (microservices reusing same batch sizes).

3) Tensor memory formats- What/Why: Use compact layouts (NHWC vs NCHW), FP16/BFloat16 quantization, or packed formats that reduce per-element bytes and improve alignment for kernels.- Benefit: Lowers raw tensor footprint and can improve compute locality.- Best scenario: Transformer inference where activations dominate memory—apply mixed precision to cut activation and weight size.

4) Garbage collection tuning- What/Why: In managed runtimes (Python), control GC thresholds, disable during hot loops, or use manual del + torch.cuda.empty_cache(); use context managers to scope tensors.- Benefit: Prevents GC-induced pauses and delayed free that inflate peak memory.- Best scenario: Batch postprocessing pipelines in Python where many short-lived NumPy/PyTorch tensors are created per request.

5) Model sharding (model parallelism)- What/Why: Split model parameters across devices (GPUs/CPUs) so no single device holds entire model; combine with pipeline parallelism to overlap compute.- Benefit: Lowers per-device peak memory allowing larger models/batches.- Best scenario: Serving very large language models that exceed one GPU memory—shard weights across multiple GPUs to support inference.

Trade-offs: pooling and zero-copy add engineering complexity; mixed precision needs validation for accuracy; sharding increases communication overhead. Combine techniques (e.g., FP16 + pooling + zero-copy) for greatest effect while measuring end-to-end latency and accuracy.

Unlock Full Question Bank

Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.