Deep Learning and Neural Networks Questions

Foundations and practical considerations of neural networks and deep learning. Includes network structure and components such as input hidden and output layers, activation functions, forward propagation and back propagation, weight initialization, loss functions, and optimization algorithms. Covers common architectures and when to use them, including convolutional neural networks for images, recurrent and sequence models for time series and text, and modern transformer architectures for language and other modalities. Discusses representation learning, data requirements for deep models, regularization techniques, transfer learning and fine tuning, interpretability limitations, and when deep learning is justified versus simpler models.

MediumTechnical

92 practiced

Case study: a production image classifier's accuracy has dropped 10% over the last month. Outline a prioritized investigation plan to identify root causes (data distribution changes, preprocessing pipeline changes, label shift, concept drift, training pipeline regressions), how to triage quickly with diagnostics, and recommended remediation steps including safe rollouts.

Sample Answer

Requirements & goal: restore baseline accuracy quickly while diagnosing root cause and avoiding user impact. Prioritized investigation plan (fast triage → deeper root cause):

1. Quick triage (first 1–4 hours)- Check system signals: deployment/change logs, infra alerts, recent releases (code, config, dependencies).- Verify model is serving the intended artifact (model hash/version), correct feature schema and preprocessing code.- Run smoke tests: send known golden inputs (canary set) and compare predictions to expected.- Compute coarse metrics by slice: overall accuracy drop, per-class accuracy, confidence distribution, latency, and rejection rates.

2. Data-distribution checks (next 4–12 hours)- Compare input feature statistics (image size, color histograms, brightness, EXIF, channel mean/std) vs baseline using KS/TV or PSI.- Check upstream data pipeline: new augmentations, resizing, compression, decoding changes, or CDN/image hosting issues.- Check prevalence of new domains/source hosts.

3. Label-shift & concept-drift (12–48 hours)- Sample recent inputs and get human labels for a stratified sample; compute label distribution shift and label noise.- Use model calibration and embedding drift: compare deep feature distributions (e.g., PCA/t-SNE) vs training set.- Run confusion matrix over recent labeled batches.

4. Training/CI regressions (24–72 hours)- Re-run reproducible training/inference locally with archived data and current code to check for bake-time regressions (dependency or seed changes).- Check for model file corruption or quantization/optimization issues (FP16/INT8).

Diagnostics to triage quickly- Canary tests + golden set.- Delta in confidence and entropy distributions.- Per-class accuracy and support.- PSI/KS statistics on key features and pixel distributions.- Human-in-the-loop labeling of 200–500 recent samples.

Remediation & safe rollouts- If pipeline/config regression: roll back to last known-good serving config immediately (safe hotfix).- If data shift transient (e.g., seasonal or domain): deploy model fallback or ensemble with previous version; use traffic splitting (10% → 50%) with monitoring.- If concept drift confirmed: collect new labeled data, retrain (or fine-tune) with importance weighting; use active learning to prioritize uncertain slices.- Apply feature guards: input validation (reject/flag corrupted images), auto-normalization to expected stats.- Use gradual rollout: A/B test with control group, monitor key metrics and business KPIs, implement automatic rollback thresholds.- Improve monitoring: add per-class alerts, embedding-drift alarms, automatic sample capture for flagged inputs, and postmortem on root causes and prevention.

Outcome metrics & timeline- Immediate rollback or canary should recover accuracy within hours.- If retraining required, target 1–2 week cycle for data collection → retrain → validation → staged rollout.- Document findings, add tests to CI (golden images, data-stat checks) to prevent recurrence.

EasyTechnical

134 practiced

Explain forward propagation and backpropagation in a feedforward neural network with one hidden layer. Describe the computations at each layer, how the chain rule is applied to obtain gradients, how gradients of the loss w.r.t. weights and biases are computed in matrix form, and include expected tensor shapes for a batch of inputs.

HardTechnical

77 practiced

Explain techniques for model compression and inference optimization: unstructured vs structured pruning, quantization (post-training quantization vs quantization-aware training), weight clustering, and knowledge distillation. Discuss trade-offs across accuracy, latency, model size, and hardware support (CPU, GPU, mobile NPUs).

Sample Answer

Start with the goal: reduce model size and inference cost while keeping acceptable accuracy. Below I summarize main techniques, how they work, and trade-offs across accuracy, latency, size, and hardware support.

Unstructured pruning- What: remove individual weights (set to zero) based on magnitude/sensitivity (L1, global, second-order).- Pros: high compression ratio without changing architecture, often small accuracy loss after fine-tuning.- Cons: sparse weight patterns require sparse kernels to get latency wins; many CPUs/GPUs don’t get speedup unless using specialized sparse libraries or hardware (sparse NPU).- Trade-offs: good size reduction; latency benefit only with hardware/software that exploits sparsity.

Structured pruning- What: remove entire channels, filters, attention heads, or transformer layers.- Pros: yields dense smaller networks — direct latency and memory improvements on general CPUs/GPUs and mobile.- Cons: often larger accuracy drop per parameter removed; requires architecture-aware pruning and re-training.- Trade-offs: easier deployment and predictable latency gains versus higher accuracy impact.

Quantization- Post-Training Quantization (PTQ): - What: convert weights/activations from FP32 to INT8/FP16 after training (calibration on small dataset). - Pros: fast to apply, minimal engineering, good latency and size improvement on supported hardware. - Cons: can cause accuracy drop on sensitive ops/models (e.g., LSTM, small heads).- Quantization-Aware Training (QAT): - What: simulate low-precision during training; learns scale/clipping and adapts weights. - Pros: near-floating accuracy at INT8; best for aggressive quantization (INT4). - Cons: additional training cost and complexity.- Hardware: INT8 widely supported on mobile NPUs, modern CPUs (AVX512 VNNI), GPUs (Tensor Cores) — but operator coverage matters.

Weight clustering (k-means)- What: reduce unique weight values by clustering and storing centroid + index.- Pros: good model size reduction and improved cache locality; works with entropy coding for storage.- Cons: needs lookup during inference (adds indirection), moderate accuracy loss; less common than quantization.- Hardware: generally software-level benefit; no special hardware acceleration.

Knowledge distillation- What: train a smaller “student” model to match teacher logits/representations.- Pros: retains task accuracy/behaviors, can yield compact models with good generalization and calibrated outputs.- Cons: requires extra training pipeline and design of loss (temperature, layer matching).- Trade-offs: typically best quality-vs-size trade-off; latency reduced by architecture selection.

Combined strategies and practical guidance- Pipeline: prefer QAT or PTQ first for quick wins; apply structured pruning for latency; use distillation to recover accuracy; use clustering/entropy coding for storage-constrained deployments.- Measure end-to-end: evaluate accuracy, throughput (p99 latency), memory footprint, and energy on target hardware.- Consider operator support: verify kernels (e.g., fused attention, group conv) are quantized on target runtime (ONNX Runtime, TensorRT, TFLite, CoreML).- Monitor robustness: compressed models can be more brittle — validate on edge cases and calibration data.

Bottom line: choose techniques by deployment constraints — if latency on standard hardware is critical, prefer structured pruning + quantization (QAT if accuracy-sensitive); if storage only, use weight clustering and PTQ; for best accuracy/size trade-off, combine distillation with QAT and light structured pruning.

EasyTechnical

86 practiced

Explain different weight initialization schemes (random normal, Xavier/Glorot, He/Kaiming, orthogonal). Describe why initialization matters for training deep networks, how it interacts with activation functions, and how poor initialization can lead to vanishing or exploding activations and gradients.

HardTechnical

69 practiced

A model converges locally but training fails to converge in a distributed multi-GPU setup. Provide a step-by-step debugging plan covering nondeterminism, random seeds, batch normalization synchronization (SyncBatchNorm), learning rate scaling, gradient accumulation mismatch, different effective batch sizes, padding/packing inconsistencies, and numerical precision differences across nodes.

Sample Answer

Start with hypothesis-driven isolation: reproduce the problem with minimal changes, then expand. Follow these steps in order (quick checks first, then deeper causes).

1) Reproduce single-GPU baseline- Confirm same code, data, hyperparams converge on one GPU. Save model/optimizer state and a small training log for comparison.

2) Verify deterministic settings & seeds- Set seeds globally on all nodes/processes (example PyTorch):

python

import torch, random, numpy as np
seed=42
random.seed(seed); np.random.seed(seed)
torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
torch.use_deterministic_algorithms(True)

- Log seeds and RNG states per rank. If deterministic mode errors, note non-deterministic ops.

3) Check nondeterminism sources- Inspect data loader workers, shuffling, augmentations, multiprocessing start method. Force worker_init_fn to set per-worker seeds. Disable asynchronous augmentation to test.

4) BatchNorm issues- If using BatchNorm, try SyncBatchNorm vs local BN: - Test with sync off and very large batch per GPU to emulate single-GPU behavior. - If using SyncBatchNorm, ensure correct all-reduce implementation and that all processes participate each step. Log per-rank batch counts; mismatched counts break stats.

5) Effective batch size & gradient accumulation- Compute effective batch size = per-GPU batch * num_GPUs / accumulation_steps.- Confirm learning rate scaling (linear scaling rule) and optimizer state identical. Check if gradient_accumulation loops match across ranks and that gradients are synchronized only when expected.

6) Padding/packing inconsistencies- For variable-length inputs (NLP, audio), ensure padding masks, pack_padded_sequence, and bucketing are identical across ranks. Log input shapes per rank. Mismatched sequence lengths change BatchNorm/statistics and loss.

7) Numerical precision and reduction errors- Compare float32 vs mixed precision (AMP) behavior. Disable AMP to test. Check different NCCL/blas versions across nodes; ensure same GPU drivers, CUDA, cuDNN, and framework builds. Enable deterministic reductions (use torch.distributed.reduce_op and check). Log gradient norms per rank to catch divergences early.

8) Learning rate and optimizer divergence- Compare initial gradient norms, parameter norms, and loss per rank for first few steps. If divergence appears after an epoch, compare dataset sharding and epoch boundary handling (drop_last, shuffle seed per epoch).

9) Instrumentation and bisecting- Insert checkpoints: after forward, after loss, after backward, after optimizer.step — serialize small tensors (loss, grad norm, param sums) per rank and diff them to find the first mismatch step.- Run two-process debug (rank 0 vs rank 1) with identical inputs to isolate communication errors.

10) Remediation and final checks- Once root cause found: fix (seed propagation, correct SyncBN usage, consistent accumulation, align software stacks), retrain with logging. Add unit tests: deterministic tiny-training test, distributed sanity check comparing single-GPU run.

Key rationale: differences usually arise from RNG/augmentation, mismatched batch statistics, or communication/order mismatches. Systematic logging and bisecting the training step will locate the first divergence.

Unlock Full Question Bank

Get access to hundreds of Deep Learning and Neural Networks interview questions and detailed answers.

Join thousands of developers preparing for their dream job.