Artificial Intelligence and Machine Learning Expertise Questions
Articulate deep expertise in one or more artificial intelligence and machine learning domains relevant to the role. Cover areas such as neural network architecture design, deep learning systems, natural language processing and large language models, generative artificial intelligence, computer vision, reinforcement learning, and full stack machine learning systems. Describe specific projects and products, datasets and data pipelines, model selection and evaluation strategies, performance metrics, experimentation and ablation studies, chosen frameworks and tooling, productionization and deployment experience, scalability and inference optimization, monitoring and maintenance practices, and contributions to model interpretability and bias mitigation. Explain the measurable impact of your work on product outcomes or research goals, trade offs you managed, and how your specialization aligns to the hiring organization needs.
MediumTechnical
81 practiced
Compare batch gradient descent, stochastic gradient descent (SGD), and Adam optimizer. Discuss convergence behavior, sensitivity to hyperparameters, memory requirements, and when you'd prefer each in a large-scale deep learning training job.
Sample Answer
Batch gradient descent, SGD, and Adam each trade off convergence stability, hyperparameter sensitivity, memory use, and practicality for large-scale training.Summary of methods- Batch GD: computes exact gradient over entire dataset each step.- SGD (mini-batch is typical): uses one or small batch per update (noisy gradient).- Adam: adaptive-moment estimator combining momentum and per-parameter learning rates.Convergence behavior- Batch GD: smooth, stable convergence toward minima for convex problems; slow per-step and impractical for large datasets; can get stuck in saddle points.- SGD: noisy updates help escape saddles and often converge faster wall-clock time; requires more iterations and exhibits high variance around minima (may need decaying LR).- Adam: fast initial convergence and robust for ill-conditioned problems; sometimes converges to worse generalization than SGD (can overfit or find sharp minima) unless tuned.Sensitivity to hyperparameters- Batch GD: learning rate crucial but easier to reason about; fewer iterations mean LR schedules matter less frequently.- SGD: sensitive to learning rate, batch size, momentum; benefits from LR decay, warmup, and careful batch-size/step-size scaling.- Adam: less sensitive out of the box (defaults often work), but β1/β2 and eps matter; requires tuning to avoid poor generalization.Memory requirements- Batch GD: needs whole-dataset pass but memory per-parameter is O(1).- SGD: minimal memory per-parameter; compute efficient with small batches.- Adam: extra O(2·N) memory to store first and second moment estimates (per-parameter), important for large models on limited GPU memory.When to prefer each (large-scale settings)- Batch GD: rarely used for deep learning at scale; maybe for small datasets or final fine-tuning where exact gradients are required.- SGD (mini-batch + momentum): preferred for large-scale training when best final generalization is desired (classic choice for CNNs/transformers). Use large batch with linear LR scaling + careful schedule or small/medium batch for stability.- Adam: preferred for quick prototyping, transfer learning, sparse gradients (NLP embeddings), or when training is unstable with SGD. For large-scale production, consider AdamW (weight decay fix) and possibly switch to SGD later (Adam->SGD fine-tune) to improve generalization.Practical tips- Use AdamW instead of Adam for proper weight decay.- Tune batch size and learning-rate schedule together; consider warmup for large-batch training.- Monitor training vs validation gap—if Adam converges fast but generalization lags, try SGD fine-tune or longer training with LR decay.
HardSystem Design
70 practiced
Design a distributed training platform capable of training models up to 1 trillion parameters. Discuss cluster topology, combination of data/model/pipeline parallelism, checkpointing and sharded optimizer state (e.g., ZeRO), fault tolerance, network requirements (RDMA), storage throughput, and cost-control strategies.
Sample Answer
Requirements & constraints:- Train up to 1T parameters with practical throughput (days–weeks), support mixed precision, elastic failure recovery, multi-tenant cost control, and budgeted cloud/hybrid HW.High-level architecture:- Meta-cluster of GPU nodes (A100/H100 or equivalent) grouped into pods of 8–64 nodes connected by a leaf-spine RDMA network (InfiniBand/Slingshot).- Storage layer: distributed NVMe (on-host for checkpoint staging) + parallel object storage (S3/GCS) with high aggregate throughput and multipart transfers.- Orchestration: Kubernetes + custom operator for job lifecycle; scheduler aware of topology and license/budget quotas.Parallelism strategy (combined):- Pipeline parallelism across model stages (e.g., 8–16 micro-stages) to keep device utilization high.- Model (tensor) parallelism within a stage using 3D tensor-slicing (Megatron-style) to split large layers across GPUs.- Data parallelism across replicas for gradient averaging.- Adopt hybrid: e.g., (Pipeline P) × (Tensor T) × (Data D) with P*T*D = total GPU count. Tune micro-batch and gradient accumulation to balance memory and bubble overhead.Sharded optimizer & memory:- Use ZeRO-2/ZeRO-3: shard optimizer state, gradients, and parameter states across data-parallel ranks to reduce per-GPU memory to fit 1T parameters. Combine with activation checkpointing and mixed precision (bfloat16/FP16 + FP32 master params).Checkpointing & consistency:- Incremental checkpoints: shard-aware checkpoints written in parallel to NVMe, then asynchronously upload to object store. Use index files mapping shards->nodes.- Support fine-grained (optimizer shards + metadata) and frequent lightweight checkpoints (only optimizer deltas) plus periodic full checkpoints.- Use deterministic global step and changelogs to ensure consistent restore.Fault tolerance & elasticity:- Node failure: automatic restart using last consistent checkpoint; leverage ZeRO-3’s ability to rebuild parameter shards from survivors or rerun pipeline micro-batches after restore.- Graceful preemption: preemption hooks to flush minimal checkpoint to fast NVMe; opportunistic reallocation of freed GPUs across jobs.- Live reconfiguration: allow shrinking/expanding D or P by re-partitioning shards with rolling checkpoint+restore to enable elasticity.Network & throughput:- RDMA-enabled fabric with low latency (<1–2µs) and high bisection bandwidth (≥100 Gbps per host, ideally 200–400 Gbps in pod).- NCCL/RDMA for all-reduce and shard exchanges, and GPUDirect Storage for direct NVMe→GPU transfers to reduce CPU hops.Storage requirements:- Sustained write throughput for checkpointing: estimate checkpoint size = parameters+optimizer shards; for 1T params (FP16) ≈1.5 TB raw + optimizer → plan 3–6 TB per full checkpoint. Multithreaded NVMe arrays and aggregate object store throughput ≥ several GB/s.- Use parallel upload, compression, and delta encoding to reduce transfer cost.Cost-control strategies:- Mixed instance types: reserve critical training on on-prem/GPU pools; burst to spot/preemptible instances for non-critical replicas.- Gradient accumulation and batch-size tuning to reduce GPU-hours.- Checkpoint frequency tuned by expected preemption MTTR: lightweight incremental checkpoints reduce wasted compute.- Autoscaling scheduler with priority/credit system, and model sparsity/quantization experiments to reduce compute.- Use cost-aware placement to co-locate pipeline stages to minimize cross-rack traffic.Trade-offs:- More pipeline stages reduce memory per GPU but increase bubble latency and complexity of load balancing.- ZeRO-3 minimizes memory but increases all-to-all comms — requires high BW/low-latency fabric.- Checkpoint granularity: frequent small checkpoints cost I/O but reduce lost work.Conclusion:- A practical design uses hybrid P×T×D parallelism, ZeRO-3 sharding, RDMA fabric, parallel NVMe+object store checkpointing, and orchestration that supports elasticity and cost-aware scheduling to reliably train 1T-parameter models.
HardTechnical
77 practiced
Explain and compare techniques for reducing transformer inference latency and memory: post-training quantization, quantization-aware training (QAT), structured and unstructured pruning, knowledge distillation, and mixed-precision. For each, discuss impact on accuracy, hardware compatibility, and retraining cost.
Sample Answer
I'll compare five common techniques for reducing transformer inference latency and memory: post‑training quantization (PTQ), quantization‑aware training (QAT), structured and unstructured pruning, knowledge distillation, and mixed‑precision. For each I cover what it does, accuracy impact, hardware compatibility, and retraining cost.Post‑training quantization (PTQ)- What: Convert weights (and optionally activations) from FP32 to lower precision (INT8, INT4) after training.- Accuracy: Generally small degradation for well‑behaved models; can be larger for sensitive layers (softmax, LayerNorm) or small models.- Hardware compatibility: Excellent for CPUs/accelerators with INT8 support (X86 VNNI, ARM DotProd, GPUs with INT8 tensor cores); INT4 support is emerging.- Retraining cost: None (no retraining), maybe light calibration using a calibration dataset.Quantization‑aware training (QAT)- What: Simulate quantization during training so model learns to be robust to low precision.- Accuracy: Best accuracy among quantization methods, often recovers PTQ losses and enables aggressive quantization (INT8/INT4).- Hardware compatibility: Same as PTQ; requires runtime that supports target lower precision.- Retraining cost: High — requires full or partial retraining/fine‑tuning and careful hyperparameter tuning, but usually fewer epochs than scratch training.Unstructured pruning- What: Remove individual weights (set to zero) based on magnitude or importance.- Accuracy: Can reach high sparsity (30–90%) with moderate accuracy loss if properly fine‑tuned; diminishing returns at extreme sparsity.- Hardware compatibility: Poor. Sparse models need specialized sparse kernels/hardware (NVIDIA Sparse, custom accelerators) to realize speedups; otherwise memory reduces but latency may not.- Retraining cost: Medium to high — iterative prune+fine‑tune cycles typically required.Structured pruning- What: Remove entire structures: heads, attention heads, MLP neurons, layers, or channels.- Accuracy: More stable per-sparsity (easier to predict impact); moderate pruning (e.g., remove redundant heads or small MLP blocks) can preserve accuracy well.- Hardware compatibility: Excellent — leads to smaller dense matrices that map directly to BLAS kernels, giving real latency and memory benefits.- Retraining cost: Medium — often requires fine‑tuning to recover performance.Knowledge distillation- What: Train a smaller (student) model to match teacher outputs/representations.- Accuracy: Can yield student models that approach teacher performance, especially when architecture is well‑matched; often better than naive pruning/quantization alone.- Hardware compatibility: Excellent — student is dense and small, straightforward to run on any hardware.- Retraining cost: High — requires training a new student model, though training time can be less than full teacher training.Mixed‑precision- What: Use FP16/BFloat16 for most compute and keep sensitive ops (LayerNorm, softmax, accumulators) in FP32.- Accuracy: Minimal to no degradation when using BF16/FP16 with loss-scaling; safe for many transformers.- Hardware compatibility: Excellent on modern GPUs/TPUs with native FP16/BF16 support (A100, H100, V100, TPUs).- Retraining cost: Low — usually no retraining; may require minor fine‑tuning or loss-scaling changes.Practical guidance / combinations- Start with mixed‑precision and PTQ + calibration for immediate wins with low cost.- If PTQ accuracy is insufficient, use QAT selectively (quantize-sensitive layers) or hybrid (INT8 weights + FP16 activations).- For maximal latency reduction on standard hardware, prefer structured pruning or distillation to produce smaller dense models.- Combine methods: distill to a smaller architecture, apply QAT or PTQ, and use mixed precision. Use unstructured pruning only when you have hardware that exploits sparsity.- Measure latency end‑to‑end (including memory-bound effects) on target hardware; choose the technique mix that meets your accuracy-latency trade-offs.
MediumTechnical
62 practiced
You're deploying an object detection model to run on mobile devices with a 100ms end-to-end latency SLO. Explain choices for model architecture (backbone, head), quantization strategy, acceleration libraries (e.g., TFLite, CoreML, NNAPI), model update mechanism, and A/B testing considerations for on-device models.
Sample Answer
Requirements & constraints:- 100 ms end-to-end on-device SLO (capture → preprocess → infer → postprocess)- Target mobile CPUs/NPUs across Android/iOS; limited memory and powerModel architecture choices:- Backbone: choose a lightweight, mobile-optimized backbone such as MobileNetV3-Large (or EfficientNet-lite) for best FLOPs/accuracy tradeoff. If NPU available, consider a small ResNet/RegNet or edge-optimized ConvNeXt tiny. Use depthwise separable convolutions and squeeze-excite blocks to keep latency low.- Head: use a single-stage detector (e.g., SSD or YOLOv5n / YOLOv8 Nano / CenterNet variants) rather than two-stage. Prefer anchor-free heads (e.g., FCOS or YOLOX decoupled head) to reduce compute & simplify postprocessing. Keep number of classes and output resolution tuned to target use case to minimize output size.Quantization strategy:- Start with post-training static quantization (8-bit integer) for CPU inference for fastest turnaround. Evaluate accuracy loss; if too high, use quantization-aware training (QAT) to recover accuracy with int8.- For NPUs that prefer mixed precision, consider FP16 or hybrid (weights int8, activations fp16) depending on vendor.- Calibrate using representative on-device input distribution and validate per-class mAP and latency.Acceleration libraries & deployment:- Use platform-specific runtimes for best performance: - Android: TFLite with NNAPI delegate or vendor-specific delegates (Hexagon, GPU, Snapdragon NPU) depending on device. Use TFLite's delegate chaining and GPU delegate for FP16; NNAPI to reach diverse hardware. - iOS: Core ML (mlmodel) with Core ML Tools conversion; use MLCompute & Metal Performance Shaders for GPU/NPU acceleration. Export from ONNX/TensorFlow to Core ML and verify operators. - Consider ONNX Runtime Mobile for cross-platform consistency where supported.- Profile per-device using real hardware profiling (TFLite benchmarking tool / Instruments on iOS). Measure end-to-end from camera frame grab to final boxes.Model update mechanism:- Use versioned model bundles with metadata (schema, input size, preprocessing steps, checksum).- On-device: small delta updates via binary diff (bsdiff) to minimize bandwidth, or host model on CDN and download when on Wi‑Fi and idle.- Validate integrity via signature; keep backward compatibility and a safe rollback path (store last good model).- Feature-flag rollout: enable new model only after compatibility checks (memory footprint, op support) and runtime sanity tests.A/B testing & evaluation:- Do staged rollouts: internal dogfooding → small percent of users (canary) → ramp-up.- Telemetry: collect anonymized metrics (inference latency histogram, memory, top-1/box confidence distribution, failure rates) plus sampled ground-truth comparisons if privacy allows (opt-in).- Use randomized assignment keyed by user/device id to split traffic; ensure statistically significant sample for accuracy and latency metrics.- Evaluate both offline (mAP on collected test sets across device-camera conditions) and online (task-specific KPIs: detection precision/recall, downstream UX metrics, battery impact).- Guardrails: if latency or error rate exceeds thresholds, auto-roll back.Trade-offs and reasoning:- Single-stage + mobile backbone minimizes latency and memory at small accuracy cost; QAT recovers much of that loss. Platform delegates maximize use of specialized hardware but require per-vendor testing and fallback paths. Safe update and A/B testing ensure reliability and measurable improvements without degrading user experience.Key deliverables before release:- Per-device performance matrix (latency, FPS, memory, accuracy)- Quantized model artifacts and conversion tests- Deployment pipeline with signing, diff updates, and rollback- Monitoring dashboards and alerting for SLO violations.
HardTechnical
76 practiced
Describe threat models and mitigations for serving generative models in production: prompt injection, model extraction/stealing, and data exfiltration. Propose runtime defenses (input/output filtering, rate limiting), logging and auditing best-practices, and active tests to detect model extraction attempts.
Sample Answer
Threat models (summary)- Prompt injection: attacker crafts inputs that override system instructions, exfiltrate secrets, or cause unsafe outputs.- Model extraction/stealing: adversary probes the model with many queries to reconstruct functionality, fine-tune a surrogate, or recover weights.- Data exfiltration: model output or logs leak sensitive training or runtime data (PII, API keys) via memorization or prompt-chaining.Mitigations per threat- Prompt injection - Strict prompt layering: separate system/assistant/user contexts; do not concatenate user text into system instructions. - Input sanitization & canonicalization: strip control tokens, instruction-like phrases, embedded code blocks; normalize Unicode. - Output constraints: enforce policy responses via safety module (deny/neutral responses), and canonical templates for sensitive flows (e.g., “I can’t provide that”).- Model extraction/stealing - Rate limits, query budgets, and per-account budgets; throttle high-entropy or high-coverage probing patterns. - Response truncation / sampling noise: add calibrated stochasticity or token-level temperature noise to reduce exact reproducibility. - Watermarking & fingerprinting: embed subtle patterns in outputs to prove provenance of stolen models. - Differential privacy during training / fine-tuning to limit memorization of rare examples.- Data exfiltration - Remove sensitive items from training or apply PII scrubbing and k-anonymization. - Canary secrets: plant unique fake secrets to detect leaks. - Output filtering and redaction: block patterns matching secrets or sensitive formats (SSNs, keys) and run regex/semantic checks before returning.Runtime defenses- Input filtering: rule-based + ML classifier to detect injection patterns, code-injection, or instruction sequences; sandbox code execution; reject or escalate risky inputs.- Output filtering: multi-stage pipeline—fast regex/token checks, ML LLM-based safety classifier, and policy engine that can redact/replace content.- Rate limiting & behavioral throttling: per-IP, per-user, per-API-key quotas; adaptive backoff for suspicious query sequences (high coverage, low repetition).- Response shaping: deterministic templates for high-risk operations; hide internal instructions and reduce verbose completions for unknown users.Logging & auditing best practices- Immutable, tamper-evident logs: append-only storage with integrity checks (signed logs or blockchain-style hashes).- Context-rich logs: record prompt versions, user id (pseudonymized), model config (temperature, top-k), timestamps, response hashes, and decision path (which filter blocked or allowed).- Redaction & retention policies: store minimal PII; encrypt logs at rest with KMS; separate keys per environment.- Centralized SIEM integration and alerting: anomaly detection on query rate, distributional shift in tokens, sudden spikes in sensitive-format outputs.- Audit trails for changes: track model updates, prompt/template changes, and policy updates with reviewers and approvals.Active tests to detect extraction- Canary probing: seeded fake secrets and unique synthetic documents; periodically query model to see if canaries surface.- Fingerprinting probes: send structured probe sets (e.g., many near-neighbor paraphrases, corner-case inputs) and monitor output similarity to detect overfitting/exposure.- Adaptive extraction simulation: run black-box extraction attempts in a controlled environment that emulate greedy/beam attacks, membership inference, and model inversion; measure how many queries to reach defined fidelity.- Monitoring query entropy & coverage: compute empirical token distribution and n-gram coverage per user; flag clients who unlock large portions of distribution rapidly.- Statistical detectors: classifiers trained to detect extraction-like sequences (systematic coverage, high diversity, temperature probing) and trigger throttles or manual review.Trade-offs and operational notes- Noise and truncation reduce utility; tune per-tenant SLAs.- Differential privacy reduces accuracy; apply selectively to sensitive models/datasets.- Logging must balance forensics with privacy/regulatory compliance.This layered approach—prevention (prompt design, DP), runtime controls (filtering, rate limits), detection (canaries, telemetry), and forensic logging—provides defense-in-depth against prompt injection, model theft, and data exfiltration.
Unlock Full Question Bank
Get access to hundreds of Artificial Intelligence and Machine Learning Expertise interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.