Training Deep Learning Models Questions
Understand the training process: feeding data through the network, computing loss, backpropagating, updating weights, and iterating until convergence. Know about batching, epochs, validation splits, and early stopping. Understand overfitting, underfitting, and the bias-variance trade-off. Know techniques to address overfitting: regularization, dropout, data augmentation, batch normalization.
HardTechnical
86 practiced
Discuss the trade-offs between small and very large batch sizes. Explain how batch size affects optimization dynamics, gradient noise scale, generalization gap at large batch sizes, and how scaling rules (e.g., linear scaling of learning rate) attempt to compensate. Provide practical recommendations for batch size selection under hardware constraints.
Sample Answer
Small vs very large batch sizes trade off computational efficiency and optimization dynamics.Optimization dynamics & gradient noise:- Small batches (e.g., 32–256) produce high-gradient noise—stochastic updates that inject variance into SGD. This noise acts like a regularizer, helping escape sharp/local minima and improving exploration early in training. Convergence per step is noisy but often reaches flatter minima that generalize better.- Very large batches (thousands to millions) reduce gradient variance, making updates closer to true-batch gradient descent. This yields more stable, larger effective step directions but less exploratory noise; optimization can get trapped in sharper minima and require more steps to generalize.Gradient noise scale:- Define noise scale g = Var(gradient)/||E[gradient]||^2. Noise decreases roughly ~1/B (B = batch size) for i.i.d. samples, so doubling B halves variance. Critical batch size arises where diminishing returns appear: beyond it, wall-clock speedups plateau.Generalization gap at large B:- Empirically, very large batches often increase test error (generalization gap). Explanations: reduced noise leads to convergence to sharper minima with worse generalization; fewer parameter updates per epoch reduce opportunities for implicit regularization. Mitigations include longer training, stronger explicit regularization, or noise injection.Scaling rules:- Linear scaling rule: when increasing B by k, multiply learning rate by k to keep step magnitude per epoch similar. Works well up to a critical B when optimizer dynamics change; otherwise leads to instability. Warmup (gradually increasing lr) stabilizes early training. More advanced: square-root scaling under certain noise-dominated regimes, or using adaptive optimizers and LARS/LAMB for very large batches.Practical recommendations under hardware constraints:- Start with batch sizes that saturate device throughput (maximize GPU memory and compute) but keep B below the critical batch size observed in prior experiments (common practical range 128–4096). Use gradient accumulation to simulate larger B if needed for throughput without changing per-update noise.- If scaling B: apply linear scaling with linear or cosine warmup, monitor training/validation loss and sharpness metrics, and increase regularization (weight decay, augmentation, dropout) or train longer if generalization degrades.- For production: prefer smaller-to-moderate B when model quality matters; use very large B for rapid prototyping or when training time is the bottleneck and compensations (LARS/LAMB, longer schedules) are feasible.Key takeaways: batch size trades off noise (helpful regularization) vs stability and throughput. Use scaling rules plus warmup and tuning to compensate; empirically identify critical batch size and adjust training schedule and regularization accordingly.
EasyTechnical
79 practiced
Explain clearly the difference between epoch, iteration, and batch in deep learning training. Use an example dataset of 12,000 samples and batch size 128 to compute the number of iterations per epoch and total iterations after 10 epochs. Discuss how these terms affect logging, learning rate schedules, and checkpoint frequency.
Sample Answer
Epoch, iteration, and batch refer to different granularity in training:- Batch: a subset of the dataset passed through the model at once (size you choose). Example: batch size = 128.- Iteration: one update step of the model parameters (one forward + backward pass) using one batch. Each batch produces one iteration.- Epoch: one full pass through the entire training dataset (i.e., processing all samples once).With 12,000 samples and batch size 128:- iterations per epoch = ceil(12000 / 128) = ceil(93.75) = 94 iterations (the final batch has 12000 − 128×93 = 96 samples).- total iterations after 10 epochs = 94 × 10 = 940 iterations.How these affect logging, LR schedules, and checkpoints:- Logging: decide per-iteration (detailed, useful for batch-level metrics like loss/stats, useful for progress bars) vs per-epoch (higher-level summaries like epoch validation accuracy). Often log training loss every N iterations and validation metrics every epoch.- Learning rate schedules: can be defined per-iteration (common for linear warmup, cosine decay with total iteration count) or per-epoch (step decay every M epochs). Per-iteration schedules give finer control and smoother transitions, but require converting epoch-based hyperparams to iteration counts.- Checkpointing: frequency can be per-epoch (simple, reproducible) or per-iteration (useful for long epochs or to save best model more frequently). For 94 iterations/epoch, inexpensive to checkpoint every epoch; for huge epochs, prefer per-N-iterations or validation-based checkpoints.Best practice: clarify whether hyperparameters are epoch- or iteration-based, convert between them (iterations = epochs × iterations_per_epoch), and choose logging/checkpoint cadence appropriate to epoch length and resource constraints.
HardTechnical
89 practiced
Implement a PyTorch Dataset.__getitem__ that performs on-the-fly augmentation, caches preprocessed images to disk to speed up repeated epochs, and uses deterministic per-worker random seeds. Show use of worker_init_fn to seed numpy and random per worker and describe cache invalidation strategy when augmentation pipeline changes.
Sample Answer
Approach: use a Dataset that, for each sample, creates a deterministic RNG seeded from a combination of a global base seed, worker id, and sample index so augmentations are reproducible across epochs and workers. Cache the preprocessed (augmented) image to disk using a cache key that includes a hash of the augmentation pipeline configuration and a dataset version; use atomic writes and an LRU/manifest-based invalidation when the pipeline changes.Key points and reasoning:- Deterministic RNG per (base_seed, worker_id, index) ensures reproducible augmentations across epochs and when using multiple workers. Using numpy.RandomState (or np.random.default_rng with integers) isolates per-sample randomness.- worker_init_fn (below) seeds global libraries per worker for any code relying on global np.random, random, or torch.worker_init_fn example (pass to DataLoader):Cache invalidation strategy:- Include an aug_hash derived from aug_config (parameters + explicit 'version' field). When pipeline changes (params or version increment), aug_hash changes and Dataset looks at different cache filenames/manifest so old files are ignored.- Use manifest files keyed by aug_hash so you can list and remove stale caches proactively.- For safety: write cache files atomically; optionally store metadata (creation time, dataset checksum) to allow LRU eviction or TTL-based cleanup.- If augmentation includes non-deterministic components (e.g., external libraries), wrap them to accept RNG-drawn parameters.Complexity:- Disk I/O dominates first epoch cost; subsequent epochs read preprocessed cached files (faster). Storage cost: O(n * image_size). Consider compression, selective caching, or caching only expensive transforms.Edge cases:- Multiple processes writing same cache entry: atomic write + existence check reduce corruption; better to use file locks for high contention.- Changing image sources: include image file mtime/hash in cache key if raw images may change.- Out-of-disk: implement fallback to in-memory augmentation without caching.
python
import os
import hashlib
import json
from pathlib import Path
from PIL import Image
import numpy as np
import random
import torch
from torch.utils.data import Dataset, get_worker_info
import torchvision.transforms.functional as TF
def _hash_obj(obj):
return hashlib.sha1(json.dumps(obj, sort_keys=True).encode()).hexdigest()[:10]
class AugmentedCacheDataset(Dataset):
def __init__(self, image_paths, aug_config, cache_dir='cache', base_seed=42):
self.image_paths = image_paths
self.aug_config = aug_config # dict describing augmentation choices & params
self.cache_dir = Path(cache_dir)
self.cache_dir.mkdir(parents=True, exist_ok=True)
self.base_seed = int(base_seed)
# include a version field in aug_config to force invalidation when pipeline changes
self.aug_hash = _hash_obj(aug_config)
self.manifest_path = self.cache_dir / f'manifest_{self.aug_hash}.json'
# manifest can track cached keys and last-used time for cleanup (optional)
if not self.manifest_path.exists():
with open(self.manifest_path, 'w') as f:
json.dump({'aug_hash': self.aug_hash, 'entries': {}}, f)
def __len__(self):
return len(self.image_paths)
def _cache_path_for_index(self, idx):
# key includes aug_hash and index to ensure updated pipeline invalidates old files
fname = f'{idx}_{self.aug_hash}.npz'
return self.cache_dir / fname
def _atomic_save(self, arr, path):
tmp = path.with_suffix('.tmp')
np.savez_compressed(tmp, arr=arr)
tmp.replace(path) # atomic on same filesystem
def __getitem__(self, idx):
worker = get_worker_info()
worker_id = worker.id if worker else 0
# deterministic per-sample seed combining base, worker, and index
sample_seed = (self.base_seed + idx * 1000003 + worker_id * 10007) & 0x7fffffff
rng = np.random.RandomState(sample_seed)
# cache lookup
cache_path = self._cache_path_for_index(idx)
if cache_path.exists():
with np.load(cache_path) as d:
arr = d['arr']
img = Image.fromarray(arr)
else:
# load original image and apply deterministic augmentations using rng
img = Image.open(self.image_paths[idx]).convert('RGB')
img = self._apply_augmentations(img, rng)
arr = np.array(img)
# save atomically
self._atomic_save(arr, cache_path)
# optional: update manifest (not shown: concurrency control for manifest)
# final transform to tensor
tensor = TF.to_tensor(img)
return tensor
def _apply_augmentations(self, img, rng):
# Example aug_config:
# {'flip_prob':0.5, 'max_rotate':15, 'brightness':[0.8,1.2], 'version':1}
if rng.rand() < self.aug_config.get('flip_prob', 0.0):
img = TF.hflip(img)
angle = (rng.rand() * 2 - 1) * self.aug_config.get('max_rotate', 0.0)
if abs(angle) > 1e-6:
img = TF.rotate(img, angle)
bmin, bmax = self.aug_config.get('brightness', [1.0, 1.0])
b = bmin + rng.rand() * (bmax - bmin)
img = TF.adjust_brightness(img, b)
return imgpython
def worker_init_fn(worker_id):
# torch's default seed can be used as base; ensure different per worker
base = torch.initial_seed() # provided by DataLoader
seed = (base + worker_id) & 0x7fffffff
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)HardTechnical
73 practiced
Explain in detail how backpropagation computes gradients for a 2D convolutional layer and for batch normalization. Describe tensor shapes involved in computing gradients wrt inputs, filters, and BN parameters, and mention common implementation optimizations (im2col, GEMM) used by libraries like cuDNN.
Sample Answer
Start with shapes and forward pass vocabulary. Let input X have shape (N, C_in, H, W), filters W have shape (F, C_in, K_h, K_w), bias b shape (F,), stride/padding produce output Y shape (N, F, H_out, W_out). During backprop you are given upstream gradient dY = ∂L/∂Y with shape (N, F, H_out, W_out).2D convolution backpropagation — gradients and tensor shapes- Gradient wrt filters (dW): each filter element accumulates over examples and spatial locations by correlating input patches with dY. Shape: dW ∈ (F, C_in, K_h, K_w). Computation: for each output channel f, dW[f] = sum_{n, i, j} dY[n, f, i, j] * X_patch[n, :, i, j] where X_patch is the K_h×K_w patch (C_in × K_h × K_w). In matrix-lowering form, X is im2col'd into X_col shape (C_in*K_h*K_w, N*H_out*W_out), dY is reshaped to (F, N*H_out*W_out); then dW_mat = dY @ X_col^T giving (F, C_in*K_h*K_w) → reshape to (F,C_in,K_h,K_w).- Gradient wrt inputs (dX): shape (N, C_in, H, W). This is convolution of dY with filters spatially flipped and channels swapped (conv-transpose). In lowered form: W_mat shape (C_in*K_h*K_w, F) (i.e. transpose of dW_mat layout); dX_col = W_mat @ dY_mat where dY_mat is (F, N*H_out*W_out); then col2im folds dX_col (C_in*K_h*K_w, N*H_out*W_out) back to (N,C_in,H,W) (accumulating overlapping patches).- Gradient wrt bias (db): db ∈ (F,) computed as db[f] = sum_{n,i,j} dY[n,f,i,j].Key intuition: conv-backprop reduces to two GEMMs + im2col/col2im (one for dW accumulation, one for computing dX), or equivalent fused kernels to avoid materializing large temporary matrices.Batch Normalization backpropagation — shapes and formulasForward (per channel c): given X of shape (N, C, H, W), BN typically computes statistics over the axes (N,H,W) for each channel. Let Ns = N*H*W (per-channel element count). For channel c:- μ_c = (1/Ns) Σ x_{n,c,i,j}- σ2_c = (1/Ns) Σ (x - μ)^2- x̂ = (x - μ)/√(σ2 + ε)- y = γ_c * x̂ + β_c (γ,β shape (C,))Backward:- Upstream dY shape (N,C,H,W).- dβ (C,) = Σ_{n,i,j} dY- dγ (C,) = Σ_{n,i,j} dY * x̂- dX: per-channel formula (vectorized across spatial dims): let std = sqrt(σ2 + ε) dX = (1/Ns) * γ / std * (Ns * dY - Σ dY - x̂ * Σ(dY * x̂)) where sums are per-channel over N,H,W. Shape of dX is (N,C,H,W).Reasoning: BN backprop requires accounting for dependency of mean and variance on all inputs in the channel; algebra yields the compact expression above, implemented using three reductions (Σ dY, Σ dY*x̂, and elementwise ops).Implementation optimizations used in libraries (cuDNN, PyTorch, TensorFlow)- Lowering (im2col) + GEMM: transforms convolution into matrix multiply for highly-optimized BLAS/cuBLAS performance. im2col produces X_col (C_in*K_h*K_w, N*H_out*W_out); two GEMMs compute dW and dX (with col2im). Pros: uses extremely optimized GEMM kernels; cons: high memory overhead for X_col.- Fused kernels: modern libs avoid fully materializing im2col by fusing patch extraction into GEMM microkernels or writing custom CUDA kernels that compute outputs/gradients with reduced memory traffic.- Winograd and FFT: reduce arithmetic for small kernels (e.g. 3×3) or large kernels respectively; requires custom backprop transforms for dW/dX.- Autotuning and workspace: frameworks choose best algorithm per layer (GEMM lowering, Winograd, direct convolution) depending on hardware, sizes, memory limits.- Memory and compute optimizations: kernel fusion (combine BN forward/backward with convolution to reduce memory read/writes), using half-precision (mixed precision) with loss scaling, using streaming/async reductions for BN statistics, and reusing temporary buffers to reduce allocations.- Numerical stability: BN uses epsilon in variance, and implementations compute statistics in FP32 accumulation even if storage is FP16.Edge/implementation considerations- Padding/stride affect im2col shapes and col2im accumulation.- Gradient accumulation when multiple threads/processors update dW (need atomic adds or reduce).- For BN in distributed training, global stats require all-reduce (synchronized BN) and change the gradient math slightly (gradients depend on global μ/σ2).- Determinism: some optimizations (e.g. atomics) can produce nondeterministic reductions.Summary: backprop for conv reduces to correlations and transposed convolutions with shapes (dW: F×C_in×K_h×K_w, dX: N×C_in×H×W, db: F), typically implemented via im2col + GEMM or specialized fused kernels/Winograd/FFT in cuDNN. BatchNorm gradients use per-channel reductions producing dγ, dβ (C,) and a closed-form dX that depends on sums Σ dY and Σ(dY*x̂) over Ns = N*H*W, with optimizations focusing on fused ops, stable accumulations, and efficient reductions.
HardTechnical
69 practiced
Derive how Batch Normalization changes gradient flow through a layer and discuss the intuition behind reducing internal covariate shift. Include an explanation of how normalization affects the gradient magnitude and direction and discuss issues such as interaction with dropout, small batch sizes, and moving averages of statistics.
Sample Answer
Situation: We want how Batch Normalization (BN) changes gradient flow and why it reduces "internal covariate shift", plus practical issues.Derivation / gradient effect (sketch):- Forward for a mini-batch of activations x: μ = mean(x), σ^2 = var(x), x̂ = (x−μ)/√(σ^2+ε), y = γ x̂ + β.- Backprop: given ∂L/∂y, chain rule gives ∂L/∂x = γ·(1/√(σ^2+ε))·(∂L/∂y) minus correction terms from the dependence of μ and σ on x. Concretely, ∂L/∂x_i = γ / √(σ^2+ε) [∂L/∂y_i − (1/m)Σ ∂L/∂y − x̂_i (1/m)Σ (∂L/∂y · x̂)].- Intuition: BN rescales gradients by 1/√(variance) and subtracts their batch mean and a covariance-weighted term. So BN both normalizes magnitude (reducing exploding/vanishing) and rotates gradients by removing components correlated with batch mean and variance.Why reduces "internal covariate shift":- By fixing layer input distribution (zero mean, unit variance per batch, then affine γ,β), subsequent layers see more stable input statistics during training steps. This stabilizes gradients and lets higher learning rates be used and parameters converge faster.Effect on gradient magnitude and direction:- Magnitude: division by √(σ^2+ε) rescales gradients inversely with activation spread — large-variance features get smaller gradients.- Direction: subtracting the mean and the x̂-weighted term removes gradient components aligned with batch-statistics, effectively projecting gradient onto subspace orthogonal to trivial shifts in μ/σ.Practical issues:- Interaction with dropout: BN reduces the need for dropout; combining both can be redundant or cause extra noise because dropout changes batch statistics and increases variance of μ/σ. If both used, apply dropout after BN.- Small batch sizes: batch estimates of μ,σ become noisy → noisy gradients and degraded performance. Remedies: use GroupNorm/LayerNorm, virtual batches, accumulate statistics, or larger effective batch via synchronized BN across devices.- Moving averages at inference: BN maintains running averages of μ and σ for inference. If running stats diverge (due to distribution shift, small batches, or training/inference mismatch), outputs are biased. Use correct momentum, warm-up, or recalibrate on a held-out set; for transfer learning, recompute stats on new data.Summary: BN rescales and projects gradients via normalization terms, stabilizing training by controlling gradient magnitude and removing directions tied to batch shifts. Practical deployment requires care with batch size, interaction with stochastic regularizers, and well-managed running statistics.
Unlock Full Question Bank
Get access to hundreds of Training Deep Learning Models interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.