Technical Tools and Stack Proficiency Questions

Assessment of a candidates practical proficiency across the technology stack and tools relevant to their role. This includes the ability to list and explain hands on experience with programming languages, frameworks, libraries, cloud platforms, data and machine learning tooling, analytics and visualization tools, and design and prototyping software. Candidates should demonstrate depth not just familiarity by describing specific problems they solved with each tool, trade offs between alternatives, integration points, deployment and operational considerations, and examples of end to end workflows. The description covers developer and data scientist stacks such as Python and C plus plus, machine learning frameworks like TensorFlow and PyTorch, cloud providers such as Amazon Web Services, Google Cloud Platform and Microsoft Azure, as well as design tools and research tools such as Figma and Adobe Creative Suite. Interviewers may probe for evidence of hands on tasks, configuration and troubleshooting, performance or cost trade offs, versioning and collaboration practices, and how the candidate keeps skills current.

EasyTechnical

59 practiced

Describe when you should use GPU versus CPU for both training and inference. Give concrete thresholds or heuristics based on model size, latency requirements, throughput, and batch sizes. Call out one common configuration mistake you have seen and how to detect and fix it.

Sample Answer

Use GPUs when you need high parallel compute for large models or large batches; use CPUs when models are small or ultra-low-latency per-request inference is required and throughput is low.

Training heuristics- Prefer GPU if model > ~50–100M parameters (resnets/transformers) or FLOPs per sample are high. GPUs cut epoch time dramatically.- If batch size per GPU ≥ 16–32 (depends on model) you get good utilization; small batches (<8) underutilize many GPUs.- If you require mixed precision (FP16) or distributed data-parallel scaling, GPUs are the default.

Inference heuristics- Low-latency single requests (<10–50 ms tail latency): if model is small (<20–50M params) a well-optimized CPU (AVX2/AVX512) or CPU + quantization may be faster and cheaper than spinning up a GPU.- High-throughput batched inference (throughput > hundreds qps or batch size ≥ 16–64): GPU wins due to parallelism and higher memory bandwidth.- Medium cases: use GPU for mid-sized models (50–200M) when you need both reasonable latency and throughput; consider model conversion (ONNX, TensorRT) and batching.

Concrete numbers (rules of thumb)- Training: choose GPU when model params >100M or per-step FLOPs >1e11; aim batch ≥32 for good GPU efficiency.- Inference: if p99 latency requirement <50ms and model <50M params, try CPU + quantization; for throughput >200 qps or batch≥16, use GPU.

Common configuration mistake- Problem: Model/data still on CPU despite launching GPU; or insufficient DataLoader settings leading to low GPU utilization. Symptoms: GPU utilization ~0–5%, low GPU memory use, high CPU load, long host-to-device copies.- Detect: nvidia-smi shows low GPU % and memory; torch.cuda.is_available() true but model.parameters().device == cpu; profiling shows host->device traffic.- Fix: Move model and tensors to device (model.to(device), input.to(device)), enable pin_memory and num_workers in DataLoader, increase batch size, use mixed precision, and minimize CPU-GPU transfers (keep preprocessing on GPU when possible).

MediumTechnical

77 practiced

Explain model quantization approaches: dynamic quantization, static post-training quantization, and quantization-aware training. For each, describe when it is appropriate, expected accuracy trade-offs, and which PyTorch or TensorFlow tooling you would use to implement it.

Sample Answer

Dynamic quantization- What: Converts weights (and optionally activations at runtime) to lower precision (usually int8) on-the-fly; minimal model rewrite and no calibration dataset.- When appropriate: CPU inference for transformer/LSTM/embedding-heavy models where weights dominate latency; quick wins for PyTorch models in production.- Accuracy trade-offs: Usually very small degradation for many architectures (transformers do well); activations remain in float by default so accuracy impact is low.- Tooling: PyTorch: torch.quantization.quantize_dynamic (or FX/quantization APIs for advanced use). TensorFlow: dynamic-style lowering less common—use float16 mixed precision or TFLite post-training float16.

Static post‑training quantization (PTQ, calibration-based)- What: Converts weights and activations to int8 ahead of time using a representative calibration dataset to compute activation ranges.- When appropriate: When you can afford an offline calibration pass and want better latency/size than dynamic; good for CNNs on mobile/edge.- Accuracy trade-offs: Moderate risk of accuracy drop (0–5% depending on model); depends on quality of calibration data and per-channel quantization reduces degradation.- Tooling: PyTorch: torch.quantization.prepare/convert (eager), or torch.quantization.quantize_fx for FX graph-mode with calibration. TensorFlow: TFLite Converter with post_training_quantize and representative_dataset (full integer or hybrid).

Quantization‑Aware Training (QAT)- What: Simulates quantization effects during training so the model learns to be robust to reduced precision, then converts to int8.- When appropriate: When PTQ causes unacceptable accuracy loss or for highly sensitive models; recommended for top accuracy on edge/int8 hardware.- Accuracy trade-offs: Best accuracy among quantized methods—often matches FP32 within ~1% or less after fine-tuning.- Tooling: PyTorch: torch.quantization.prepare_qat / convert or FX QAT workflows. TensorFlow: TensorFlow Model Optimization Toolkit (tfmot.quantization.keras.quantize_apply or tfmot.quantization.keras.quantize_model) and TFLite QAT flow.

Summary guidance- Try dynamic quantization first for fast wins on CPU. Use PTQ with good calibration for mobile/CNNs. Use QAT when accuracy must be preserved—budget for fine-tuning and longer dev cycle. Always validate on representative input and test on target hardware (NEON, VPU, NPU) because real-world latency/accuracy vary.

MediumSystem Design

49 practiced

Design an inference stack to serve a PyTorch model at 2000 requests per second with p95 latency under 50 ms. Describe model format conversion options, the serving solution you choose, batching strategy, autoscaling rules, cache considerations, and provide a rough resource estimate per replica.

Sample Answer

Requirements & constraints:- 2,000 rps, p95 < 50 ms, assume model single-GPU friendly (e.g., ResNet/transformer small-medium), input size known, no heavy preprocessing in-line.

Model format conversion:- Convert PyTorch .pt to TorchScript for CPU/GPU inference or to ONNX for cross-runtime export.- Prefer TorchScript for dynamic control flow; prefer ONNX + TensorRT (or ONNX-TensorRT engine) when optimizing for latency on NVIDIA GPUs.- For Triton, deliver either TorchScript or ONNX; for max perf use TensorRT plan built from ONNX.

Serving solution:- Use NVIDIA Triton Inference Server (supports TorchScript/ONNX/TensorRT), supports concurrent models, batching, GPU sharing, metrics.- Alternative: TorchServe for simpler CPU deployments, but GPU+high-throughput favors Triton.

Batching strategy:- Use dynamic batching in Triton with max batch size tuned to keep latency under 50ms.- Empirically measure batching curve: small batch sizes (e.g., 1–8) often optimal for p95 latency; enable timeout (e.g., 2–5 ms) to wait briefly for requests.- For transformer-like models, prefer batch size 4–16; for small CNNs, up to 32 may be ok.

Autoscaling rules:- Horizontal Pod Autoscaler (Kubernetes) based on GPU utilization and request latency: - Primary metric: p95 latency > 40 ms triggers scale-up. - Secondary metric: GPU utilization > 70% sustained for 30s. - Scale down when p95 < 30 ms and utilization < 40% for 60s.- Maintain minimum replicas = 2 for fault tolerance and cold-start smoothing.

Cache considerations:- Use an input/result-level LRU cache (e.g., redis) for high-frequency identical requests (fingerprint inputs); ensure cache hit returns within <5ms.- Warm-up model and keep weights resident; preload model on startup to avoid cold GPU transfers.- Use Triton model control to load/unload models and use/disable instance groups to manage memory.

Rough resource estimate per replica (GPU-based):- 1x NVIDIA T4 (or A10) + 4 vCPU + 8–16 GB RAM.- Expected throughput per replica: ~500–1000 rps depending on model; start with conservative 500 rps → need 4 replicas to meet 2000 rps.- Latency components: model compute 10–30 ms, batching/queue 0–10 ms, network/serialization 1–5 ms → fits p95 <50 ms with correct batching and 4 replicas.- If CPU-only: estimate 8–16 vCPU + 32–64 GB RAM per replica, much lower throughput → require many more replicas.

Validation & monitoring:- Load test (JMeter/locust) to tune batch size and timeouts.- Monitor p95 latency, GPU utilization, queue length, Triton metrics, cache hit rate.- Add circuit-breaker to reject when queue depth grows above threshold to protect latency SLA.

Trade-offs:- TensorRT gives best latency but adds build complexity; TorchScript easier but slightly slower.- Larger batches increase throughput but hurt p95 — tune for percentile, not mean.

HardSystem Design

56 practiced

Describe how to build debuggable, efficient ML pipelines using Argo Workflows or Kubeflow Pipelines. Include recommendations on step granularity, caching strategies, parameterization, artifact passing, UI-driven debugging, and developer experience features to speed experimentation while maintaining production quality.

Sample Answer

Requirements & constraints:- Reproducible, debuggable pipelines for experimentation and production; low-cost iterative runs; strong provenance and fast failures; scalable training steps on k8s.

High-level approach:- Use Argo Workflows or Kubeflow Pipelines (KFP) as orchestration, an object store (S3/GCS/MinIO) + artifact metadata (MLMD), and a metadata/DB for lineage. Containerize steps; use SDKs (argo/kfp) for typed components.

Core recommendations1. Step granularity- Make steps small and single-responsibility: data-prep, feature-extract, train, eval, pack. Small steps speed debugging, allow selective reruns and caching.- Group ultra-fast ops to avoid orchestration overhead; heavy ops (train) stay isolated.

2. Caching strategies- Enable pipeline-level caching (KFP cache / Argo artifact checksums).- Use deterministic keys: hash code + input data snapshot + parameters. Store hashes as artifact metadata.- Cache intermediate artifacts in object store with TTL and lifecycle policies.

3. Parameterization- Expose config via typed params: hyperparams, dataset URIs, image tags, resource hints.- Use config files (yaml/ConfigMap) and feature flags for env-specific behavior. Version parameters in MLMD.

4. Artifact passing & provenance- Pass artifacts by reference (URIs) not by-value. Persist artifact metadata to MLMD (or Vertex AI metadata).- Use standardized artifact schemas (training dataset id, schema, hash, model version). Keep manifest files alongside artifacts.

5. UI-driven debugging- Leverage KFP/Argo UIs to inspect logs, retries, step inputs/outputs and MLMD lineage. Integrate with notebook links and tensorboard links in run metadata.- Add failed-step “dump” sidecar that collects crash logs, pod yaml, resource usage for the UI.

6. Developer experience to speed experimentation- Local iteration: provide “mini” pipeline runtime (kfp-tekton local runner or docker-compose) for single-node dev.- Fast feedback loops: smaller datasets, lower epochs, mock components.- Templates & component library: reusable component images, shared SCM + CI that builds images on commit with reproducible tags.- Integrated telemetry: structured logs (JSON), metrics to Prometheus, and automatic TensorBoard/MLflow upload.- Automated tests: unit tests for components, integration tests for pipelines using synthetic data; enforce hermetic containers.

Operational best practices & trade-offs- Resource isolation: set requests/limits; use node selectors/GPU quotas.- Retries and timeouts: set conservative retries, capture non-deterministic side effects.- Security: least-privilege for object-store access, signed URIs.- Trade-offs: finer granularity increases orchestration overhead; balance by grouping trivial steps. Aggressive caching saves cost but requires robust invalidation.

This design gives traceable, fast-to-debug pipelines while supporting production scale and safe experimentation.

EasyTechnical

82 practiced

Explain the difference between batch inference and online inference. For each pattern list typical frameworks, expected latency and throughput characteristics, and an example use case where batch is preferable and another where online inference is required.

Unlock Full Question Bank

Get access to hundreds of Technical Tools and Stack Proficiency interview questions and detailed answers.

Join thousands of developers preparing for their dream job.