Performance and Code Optimization Questions

Covers techniques and decision making for improving application and code performance across levels from algorithm and memory access patterns to frontend bundling and runtime behavior. Candidates should be able to profile and identify bottlenecks, apply low level optimizations such as loop unrolling, function inlining, cache friendly access patterns, reducing branching, and smart memory layouts, and use compiler optimizations effectively. It also includes higher level application and frontend optimizations such as code splitting and lazy loading, tree shaking and dead code elimination, minification and compression, dynamic imports, service worker based caching, prefetching strategies, server side rendering versus client side rendering trade offs, static site generation considerations, and bundler optimization with tools like webpack Vite and Rollup. Emphasize measurement first and avoiding premature optimization, and explain the trade offs between performance gains and added complexity or maintenance burden. At senior levels expect ability to make intentional trade off decisions and justify which optimizations are worth their complexity for a given system and workload.

HardTechnical

18 practiced

Explain compiler techniques such as operator fusion, kernel auto-tuning, and memory planning (used by XLA or TensorRT), how they reduce inference time and memory footprint, and why hand-optimized kernels sometimes still outperform auto-compilers in practice.

Sample Answer

Operator fusion, kernel auto-tuning, and memory planning are compiler-level optimizations used by XLA, TensorRT, TVM, etc., to reduce inference latency and memory footprint. Briefly:

Operator fusion- What: Merge sequences of primitive ops (e.g., MatMul -> Add -> Activation) into a single kernel so data stays in registers/L1 instead of round-tripping to DRAM.- Why it helps: Reduces memory bandwidth, eliminates intermediate allocations/copies, increases arithmetic intensity and cache locality.- Example: Fusing bias add + GELU into one CUDA kernel reduces launch overhead and halves peak activation memory for that layer.

Kernel auto-tuning- What: Explore implementation variants (tile sizes, unrolling, vector widths, shared-memory layouts) on target hardware and pick the best-performing kernel configuration.- Why it helps: Matches microarchitecture (SM count, cache sizes, warp width) to exploit occupancy, minimize memory stalls, and maximize throughput.- Example: Auto-tuning GEMM tiling for a given GPU can double effective FLOPS versus a generic configuration.

Memory planning- What: Globally schedule lifetimes of tensors, reuse buffers, perform inplace updates, and reorder computations to lower peak memory usage.- Why it helps: Lowers peak activation footprint so larger batches/models fit on-device; reduces page thrashing and allocator overhead.- Example: Liveness analysis + buffer aliasing can cut activation memory by 30–70% for transformer stacks.

Why hand-optimized kernels sometimes still win- Narrow focus: Engineers hand-tune for one shape/hardware and squeeze out microseconds via handcrafted assembly, specialized intrinsics, or algorithmic tweaks (e.g., Winograd for convolutions).- Unknown heuristics: Auto-compilers may miss domain-specific fusion opportunities, numerical stability trade-offs, or beneficial approximations that a human would accept.- Search/compile cost: Auto-tuning has budget limits; it may not explore the best corner cases. Runtime variability and complex kernels make global optima hard to find.- Driver/runtime constraints: Vendor libraries (cuBLAS/cuDNN) receive low-level continual tuning and assembly intrinsics not exposed to generic compilers.- Edge cases: Irregular shapes, very small tensors, or mixed-precision corner cases may be better handled by bespoke kernels.

In practice the best approach is hybrid: use compilers for broad, repeatable gains (fusion, tuning, planning) and plug in hand-optimized kernels or vendor libraries for hotspots where micro-optimizations matter.

EasyTechnical

32 practiced

Briefly explain what 'zero-copy' means for CPU-to-GPU data transfer in an inference pipeline and name one API or technique that enables zero-copy or reduced-copy transfers. Discuss one trade-off of that approach.

EasyTechnical

16 practiced

You need to run inference of an ML model inside the browser using WebAssembly. List concrete steps and optimizations to minimize initial download and startup time for users on slow networks, including model size, lazy loading, caching, and compilation strategies.

Sample Answer

Situation: You need browser-side ML inference via WebAssembly while minimizing initial download and startup latency for users on slow networks.

Concrete steps and optimizations:

1. Reduce model size- Quantize to 8-bit/4-bit (post-training or QAT), prune unimportant weights, and apply weight clustering/distillation to produce a lightweight student model.- Export compact formats (ONNX / TensorFlow Lite / custom flatbuffer) and remove unused ops/metadata.

2. Split and lazy-load- Chunk the model into layers or shards (e.g., embedding + encoder + head). Load only the minimal shard needed for first interaction; fetch additional shards on demand.- Use dynamic import() for JS/WASM code so the main bundle stays tiny.

3. Compression & delivery- Serve WASM and model shards via CDN with Brotli (pre-compressed .br) and proper cache headers.- Use content negotiation and HTTP/2 or HTTP/3 to reduce latency; enable range requests for large shards.

4. Streaming compile & instantiate- Use WebAssembly.instantiateStreaming(fetch(...)) or streaming compilation APIs so compilation starts while downloading.- Provide a tiny JS “bootstrap” that starts streaming compilation and displays an early UI.

5. AOT / SIMD / Threads- Build multiple WASM variants: baseline (small), SIMD-optimized, multi-threaded. Detect client capabilities via feature detection, load the smallest compatible variant.- For threads, ensure COOP/COEP headers and SharedArrayBuffer support; otherwise fall back.

6. Caching & offline- Cache compiled WASM and model shards in IndexedDB (store raw bytes) or use Cache API via Service Worker so subsequent loads skip network and compilation when possible.- Persist a versioned key; validate via ETag or manifest.

7. Fast startup patterns- Keep initial JS minimal; defer heavy initialization. Warm up a WebWorker that handles WASM instantiation off the main thread to avoid blocking UI.- Precompile / cache compiled module for supported browsers using WebAssembly.compileStreaming and store compiled module bytes if available.

8. Progressive inference & fallback- Offer a CPU-lite fast path (smaller model) with optional upgrade to full model in background.- Use WebGL/WebGPU backend as alternative when WASM performance is poor.

9. Observability & user experience- Show progressive loader and ability to cancel downloads. Track metrics (download time, compile time) and adjust behavior (switch to smaller model for slow networks).

Why these work: Reducing bytes reduces network time; chunking + lazy load postpones heavy downloads until necessary; streaming instantiation overlaps network and compilation; caching avoids repeated work; multiple WASM builds let you choose smallest compatible binary. Together these minimize time-to-first-inference on slow connections while preserving accuracy and UX.

MediumTechnical

23 practiced

A newly deployed model server intermittently returns 500 errors and escalates during load spikes. Describe an incident response checklist focusing on performance causes, required instrumentation to root-cause, and short-term mitigations and long-term fixes to prevent recurrence.

Sample Answer

Incident-response checklist (performance-driven 500s) — concise playbook for an AI Engineer.

Immediate triage (first 5–15 minutes)- Confirm scope: which endpoints/models/clients, % of requests failing, duration and start time.- Pager info: collect request IDs, timestamps, and node/pod IDs from error logs.- Short-term mitigation: enable rate-limiting or reject low-priority traffic; switch traffic to prior stable model version or fallback model; scale up replicas/GPUs if capacity is saturated; apply circuit breaker to avoid cascading retries.

Root-cause investigation (15–60 minutes)- Check high-level metrics: - Error rate (5xx) and P50/P95/P99 latencies per endpoint - Throughput (rps), concurrency, queue/backlog length - Resource: CPU, RAM, GPU utilization, GPU memory, PCIe and NVLink utilization - Pod/container restart counts, OOMKill, and node pressure - External dependency latencies (feature store, DB, auth, storage)- Correlate: overlay deployment/event timeline with metric spikes.- Logs & traces: - Pull structured logs with request_id; inspect stack traces for exceptions. - Use distributed tracing (e.g., Jaeger, Zipkin, OpenTelemetry) to find slow spans: model load, preprocessing, postprocessing, feature fetch.- Model-specific telemetry: - Input size/token counts, batch size, batching latency, model load time, memory footprint. - Warm vs cold start ratio; cache/memoization hit rates.- Runtime diagnostics: - JVM/GC pauses, Python GIL contention, thread pool saturation, async queue lengths. - GPU metrics (nvidia-smi / DCGM): memory fragmentation, kernel time, CUDA errors.- Reproduce at lower scale if safe: replay a sample failing request against a test instance to reproduce.

Required instrumentation (if missing, add ASAP)- Structured request logs with request_id, endpoint, model_version, input_size, user_id, latency components (preproc, inference, postproc), response code.- Distributed tracing (capture service, DB, feature-store spans).- High-resolution resource telemetry (CPU, memory, *GPU* metrics per instance).- Queue/backlog and batching metrics (batch size distribution, batch wait time).- Feature-store and external dependency SLIs and traces.- Health checks and readiness probes exposing warm/cold status and model load states.- SLO/SLA dashboards (error budget, p99 latency) + automated alerts.

Short-term mitigations (minutes–hours)- Roll back to last-known-good model/container image.- Scale out/in: increase replicas or GPU nodes; temporarily increase instance size.- Apply ingress throttling and token-bucket rate limits; prioritize critical clients.- Reduce work per request: disable expensive optional features, lower max tokens or summarization lengths, reduce beam width.- Turn on circuit breaker/fallback responses and exponential backoff for clients.- Restart affected pods to clear leaked resources (after confirming no data loss).- Enable “graceful degradation” route that serves cached or simpler responses.

Long-term fixes (days–months)- Capacity planning & autoscaling: - Implement SLO-driven autoscaling (HPA/VPA/custom scaler based on p99 latency, GPU mem). - Right-size instances and GPU families; reserve headroom for spikes.- Model optimization: - Quantize, distill or prune models; reduce memory and latency. - Use model sharding, tensor parallelism where appropriate.- Inference architecture: - Use asynchronous request queue with bounded workers and backpressure. - Efficient batching with adaptive batch policy to optimize p99 latency vs throughput. - Separate heavy preprocessing from inference service; cache repeated feature lookups.- Observability & testing: - Add end-to-end tracing, per-request telemetry, and dashboards for p50/p95/p99 + error budget. - Implement canary/circuit-breaker deployment patterns, progressive rollouts, and automated rollback. - Run load and chaos testing (including GPU stress) before production deploys.- Reliability practices: - Define SLOs, error budgets and runbooks; automate alerts mapped to runbooks. - Harden retry logic (idempotency, bounded retries, client backoff). - Harden memory usage: detect and prevent memory leaks; enforce limits and graceful degradation.- Post-incident: - Conduct blameless postmortem with timeline, root cause, action items, owners, and deadlines. - Track recurrence metrics and verify mitigations.

Why these steps:- Performance 500s usually stem from resource exhaustion, blocking code, large inputs, batching misconfiguration, dependency latency, or memory leaks — the checklist targets each cause with instrumentation and mitigations so you can both stop the outage and remove the root cause.

EasyTechnical

19 practiced

Explain the practical differences between FP32, FP16 and bfloat16 in terms of exponent, mantissa, dynamic range, and precision. For an AI inference workload, when would you prefer mixed precision, and what pitfalls (numerical stability, accumulation) should you watch out for?

Sample Answer

FP formats (practical summary)- FP32 (IEEE-754 single): 1 sign, 8-bit exponent, 23-bit mantissa (fraction), exponent bias 127. Large dynamic range (~1e±38) and ~7 decimal digits precision — good for both range and fine-grained precision.- FP16 (IEEE half): 1 sign, 5-bit exponent, 10-bit mantissa, bias 15. Much smaller dynamic range (~1e±5 to 1e±4) and ~3 decimal digits precision — reduces memory/compute but can underflow/overflow and has larger rounding errors.- bfloat16: 1 sign, 8-bit exponent, 7-bit mantissa, bias 127. Same exponent range as FP32 (so similar dynamic range) but only 7 mantissa bits (~2–3 decimal digits precision). Keeps range benefits of FP32 while halving storage.

Implications for AI inference- Dynamic range: fp16’s narrow exponent can cause saturation/underflow for very large/small activations or logits; bfloat16 avoids that by matching FP32 exponent.- Precision: both fp16 and bfloat16 lose mantissa bits vs FP32; bfloat16 trades more precision (fewer mantissa bits than fp16?) — note: fp16 has 10 mantissa bits vs bfloat16 7, so fp16 has slightly better precision but worse range. In practice bfloat16 is preferred on large models because range problems are more harmful than a few mantissa bits.

When to prefer mixed precision- Use mixed precision (storing activations/weights in FP16/bfloat16 but computing key accumulations in FP32) to get memory and throughput gains while preserving numerical behavior.- For inference: prefer bfloat16 where available for large models/large logits. Use fp16 if hardware supports fast tensor cores and inputs/activations fit range.- For training: mixed precision with FP32 master weights, FP16 activations, and FP32 accumulators + loss scaling is standard (e.g., NVIDIA AMP or PyTorch autocast).

Pitfalls to watch- Numerical stability: reduced precision increases rounding noise — sensitive ops (softmax, LayerNorm, batch-norm, attention score exponentials) can degrade accuracy. Consider keeping them in FP32.- Accumulation error: summing many small terms in FP16 loses precision; always use FP32 accumulation for reductions/GEMM accumulators.- Overflow/underflow: fp16 exponent may overflow for logits/gradients; bfloat16 mitigates this.- Loss scaling: for training in FP16, use static/dynamic loss scaling to avoid gradient underflow.- Non-determinism and calibration: quantization-like effects can shift outputs; validate with representative data and run calibration/accuracy checks.

Practical rules- Prefer bfloat16 where hardware supports it (TPU, recent CPUs/GPUs) for large models.- Use mixed precision with FP32 accumulators and master weights for training.- Keep numerically sensitive ops in FP32, use loss scaling, and validate end-to-end accuracy.

Unlock Full Question Bank

Get access to hundreds of Performance and Code Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.