Deep Learning and Neural Networks Questions

Foundations and practical considerations of neural networks and deep learning. Includes network structure and components such as input hidden and output layers, activation functions, forward propagation and back propagation, weight initialization, loss functions, and optimization algorithms. Covers common architectures and when to use them, including convolutional neural networks for images, recurrent and sequence models for time series and text, and modern transformer architectures for language and other modalities. Discusses representation learning, data requirements for deep models, regularization techniques, transfer learning and fine tuning, interpretability limitations, and when deep learning is justified versus simpler models.

MediumSystem Design

143 practiced

You must train a 100B-parameter transformer with 8 GPUs per node across 4 nodes. As an AI Engineer propose a distributed training strategy that discusses data parallelism, tensor/model parallelism, pipeline parallelism, optimizer-state sharding (ZeRO), communication backends, and trade-offs in memory and throughput.

Sample Answer

Requirements and constraints:- Train 100B-parameter transformer on 4 nodes × 8 GPUs = 32 GPUs. Aim: fit model+optimizer states, maximize throughput, reasonable memory headroom, stable scaling.

High-level strategy:- Combine tensor (model) parallelism + pipeline parallelism + data parallelism with ZeRO optimizer-state sharding and NCCL/RDMA communication.

Architecture:- Split the transformer layers into L pipeline stages across GPUs within a node and across nodes (e.g., 8–16 stages depending on depth). Within each stage, apply tensor/model parallelism (1D/2D) to shard large weight matrices across the GPUs assigned to that stage. Then apply data-parallel replicas across pipeline+tensor groups.

Details:- Tensor/model parallelism: use 2D mesh (row+column) if available (e.g., Megatron-LM style) to shard large linear projections and attention heads — reduces per-GPU activation and parameter memory and keeps compute balanced.- Pipeline parallelism: partition layers sequentially to increase max model depth per device; use micro-batching and interleaved schedule to fill GPUs and reduce bubble time.- Data parallelism: across replicas of the combined pipeline+tensor group; synchronize gradients across replicas.

Optimizer-state sharding (ZeRO):- Use ZeRO Stage 3 to shard optimizer states, gradients, and parameters across the data-parallel group to minimize memory per GPU. This is important for 100B where full replication is impossible. Combine with offloading (CPU/NVMe) if further headroom needed.

Communication backends:- Use NCCL over high-speed interconnect (HDR InfiniBand/RDMA) for all-reduce and point-to-point; use optimized collective implementations (ring/ hierarchical) and overlap communication with compute. For pipeline activations, use direct GPU-GPU peer transfers; consider CUDA IPC and NVLink within node for lower latency.

Trade-offs:- Memory vs throughput: More pipeline stages reduce per-GPU memory but increase pipeline bubble (reduces throughput) unless using many micro-batches and interleaving. Larger tensor parallel dims reduce memory but increase all-reduce overhead inside stage.- Communication overhead: Tensor parallelism increases collective frequency/size (synchronized matmul shards). ZeRO reduces memory but increases all-to-all/reshard communication on each step—tune group sizes to balance.- Complexity vs performance: Full 2D tensor + pipeline + ZeRO Stage 3 yields best fit and throughput but adds implementation complexity and debugging cost.- Fault tolerance: With sharded states, node failure can be catastrophic; add periodic checkpoints (multi-node stable storage) and consider lightweight redundancy.

Operational notes:- Start with simulation/prototype on smaller scale to choose tensor/pipeline split and micro-batch size. Tune learning-rate scaling, gradient accumulation, and overlap strategies. Monitor GPU utilization, network saturation, and memory headroom; iterate on parallelism configuration.

MediumTechnical

91 practiced

Explain common learning rate scheduling strategies (step decay, exponential decay, cosine annealing, linear warmup followed by decay). For a transformer trained on large text corpora, recommend a scheduler and justify your choice with respect to stability and convergence.

HardTechnical

70 practiced

A multi-node DistributedDataParallel job intermittently deadlocks/hangs during training. Provide a systematic debugging plan listing possible root causes (NCCL failures, mismatched collective calls, non-deterministic control flow across ranks, buffer size mismatches), tools/commands to diagnose the issue, and practical remediation steps.

Sample Answer

Start with a hypothesis-driven checklist, then iterate: reproduce reliably (deterministic short run), gather artifacts, narrow root cause, apply fixes, verify.

Possible root causes- NCCL runtime errors: driver/CUDA/NCCL version mismatch, network fabric issues (RoCE/IB), or transient NIC errors.- Mismatched collective calls: different code paths or conditional collectives causing rank divergence.- Non-deterministic control flow across ranks: data-dependent branching, differing RNG seeds, dataset sharding bugs.- Buffer size / dtype mismatches: tensors with different sizes or unexpected gradients (e.g., unused parameters) causing hang at all-reduce.- Hanging point inside backward due to mixed precision/autocast interactions.- Deadlocks from synchronization primitives (barrier/all_reduce in wrong order) or custom DDP hooks/communication collectives.- Resource exhaustion: OOM leading to stalls, or GPU hangs requiring kernel reset.

Diagnostics & commands- Reproduce with minimal job (few steps, smaller model, same batch-size per GPU).- Set NCCL_DEBUG=INFO (or TRACE) and NCCL_IB_DISABLE=0/1 to toggle IB; capture logs: - export NCCL_DEBUG=INFO; export NCCL_DEBUG_SUBSYS=ALL- Inspect CUDA driver and GPU state: nvidia-smi -q -d MEMORY,UTILIZATION; dmesg | grep -i nvidia- Check versions: python -c "import torch; print(torch.__version__, torch.cuda.nccl.version(), torch.version.cuda)"- Run NCCL tests: nccl-tests (all_reduce latency) across nodes to verify fabric- Enable PyTorch distributed logging: TORCH_DISTRIBUTED_DEBUG=DETAIL- Insert rank-specific logging (including before/after each collective) with timestamps to locate blocking op- Use gdb/py-spy or torch.autograd.profiler to see where threads blocked; nvprof/nsys for GPU traces- Check network: ibstat, ibv_devinfo, ethtool, tcpdump for lost packets/MTU problems- Validate tensors: in debug build, assert tensor.shape/dtype on each rank before collectives- Reproduce under: single-node multi-GPU to isolate inter-node network issues

Remediations- For NCCL/fabric: - Align driver/CUDA/NCCL versions; apply latest recommended patches - Disable problematic transports (NCCL_IB_DISABLE=1) or tune NCCL_SOCKET_IFNAME/NCCL_IB_HCA - Increase NCCL_TIMEOUT or set NCCL_ASYNC_ERROR_HANDLING=1 for better failure handling- For mismatched collectives/control flow: - Audit code to ensure identical collective call order on all ranks; guard collectives with the same conditions - Add barrier() calls temporarily to detect divergence early - Determinize data pipeline: fixed seeds, deterministic sampler, ensure same sharding- For tensor/buffer mismatches: - Use model.register_comm_hook or verify param.requires_grad and that no parameter is skipped unexpectedly - Use find_unused_parameters=True cautiously; prefer explicit gradient accumulation patterns- For AMP/autocast issues: - Try disabling AMP to confirm; upgrade torch/amp or align scaler usage across ranks- For resource issues: - Reduce batch size, monitor memory, set TORCH_DISTRIBUTED_DEBUG to catch hangs- Fallback strategies: - Add watchdog/timeout and restart failing ranks; log stack traces at timeout - Isolate failing nodes by rotating hosts to find hardware/network culprits

Validation- After each fix: run small deterministic job across same nodes multiple times.- Add long-term observability: centralized logs (timestamps, rank ids), health checks, NCCL metrics exported to monitoring.

This systematic approach—reproduce, collect logs, isolate network vs. code divergence, then apply targeted fixes—will locate and resolve intermittent DDP hangs.

EasyTechnical

94 practiced

Transformers are permutation-invariant; explain why positional encoding is necessary. Compare sinusoidal positional encodings with learned positional embeddings, describing pros and cons for transfer learning, extrapolation to longer sequences, and parameter cost.

MediumTechnical

80 practiced

Compare full fine-tuning, head-only fine-tuning, adapters, and LoRA for adapting large pre-trained transformer models. For each approach explain parameter-efficiency, compute and memory costs during training, serving implications, and scenarios where each is preferred in production systems.

Sample Answer

Situation: You need to adapt a large pre-trained transformer for a downstream task. Below compares four common approaches across parameter-efficiency, training compute & memory, serving, and when to prefer each in production.

Full fine-tuning- Parameter-efficiency: Low — all model weights updated (100% of params).- Compute & memory (training): Highest — backprop through entire model, large optimizer state (e.g., Adam requires 2× params), high GPU memory and longer wall-clock time.- Serving: Single model per task; no special runtime changes. Model size increases if storing separate checkpoints per task.- When to prefer: Small models or when maximum task performance is required and resources permit; few tasks, need to tweak internal representations.

Head-only fine-tuning (linear classifier / task head)- Parameter-efficiency: Very high — only small head (e.g., classifier, pooler) updated (<<1%).- Compute & memory: Minimal — frozen backbone reduces compute and memory; fast to train.- Serving: Lightweight; can reuse same backbone for many tasks, only swap small heads.- When to prefer: Low-data regimes, many tasks with limited compute, quick prototyping where backbone features suffice.

Adapters (bottleneck adapter modules inserted per layer)- Parameter-efficiency: Good — adapters add small layers per transformer block (often 1–5% of params).- Compute & memory: Moderate — extra forward/backward through adapters only; optimizer state needed for adapter params (small). Slight increase in latency due to extra ops.- Serving: Need to load backbone plus adapter weights for task; can keep one backbone and swap adapters dynamically (efficient multi-task serving).- When to prefer: Multi-task systems, modular deployments, desire to retain frozen backbone while allowing deeper representational changes with small storage per task.

LoRA (low-rank update of weight matrices)- Parameter-efficiency: Very high — represents updates as low-rank matrices per attention/projection (often <1%).- Compute & memory: Low to moderate — training updates only low-rank factors; can often be fused at inference to avoid extra ops, and optimizer state is small. Backprop still touches some projected activations but overall cheaper than full-tune.- Serving: Two modes: 1) apply LoRA updates to weights (merge) and serve a single fused model, 2) keep separate low-rank deltas and apply at runtime (slightly higher latency). Merging is common for production.- When to prefer: Large models where storage and training budget are constrained, when you want near full-finetune performance with tiny checkpoint sizes, or when maintaining many task variants.

Trade-offs & practical notes- Performance: Full fine-tuning often gives best ceiling; LoRA and adapters typically approach that with far fewer params. Head-only may underperform if task needs representational change.- Checkpointing & CI: Parameter-efficient methods drastically reduce artifact size and enable many task variants. They simplify CI/CD for models.- Hardware: For very large models use LoRA/adapters to avoid OOM and to enable feasible training on fewer GPUs.- Latency: If inference latency is critical, merge low-rank/adapter weights into the backbone to avoid runtime overhead.- Safety & auditing: Freezing backbone and storing small deltas simplifies reproducibility and A/B testing.

Recommendation summary- Prototype / constrained compute: head-only.- Production multi-task / modular deployments: adapters.- Large-model, storage- and cost-sensitive fine-tuning: LoRA.- When absolute best performance and you control full infra: full fine-tuning.

Unlock Full Question Bank

Get access to hundreds of Deep Learning and Neural Networks interview questions and detailed answers.

Join thousands of developers preparing for their dream job.