Basic Neural Network Concepts Questions
Conceptual understanding of how neural networks work: neurons, layers, activation functions, forward propagation, backpropagation, and training. Ability to explain why neural networks are used for certain problems. No advanced mathematics required.
MediumTechnical
18 practiced
You have 5,000 labeled images for a classification task. Compare training a model from scratch versus fine-tuning a pre-trained convolutional neural network. Consider expected performance, training cost, risk of overfitting, time-to-production, and deployment implications, and give a recommended approach with justification.
Sample Answer
Framework: weigh expected accuracy, compute/time cost, overfitting risk, time-to-production, and deployment constraints.Analysis:- Expected performance: With only 5,000 labeled images, a model trained from scratch (random init) will likely underfit or overfit depending on capacity. Pre-trained CNNs (ImageNet or similar) provide strong low-level features (edges, textures) and typically yield much higher validation accuracy after fine-tuning on small datasets.- Training cost & time-to-production: Training from scratch requires many epochs, hyperparameter sweeps, and substantial GPU time. Fine-tuning is far cheaper and faster — often converges in a few epochs for classifier head and tens for deeper unfreezing.- Risk of overfitting: Scratch models have high risk of overfitting due to limited labels. Fine-tuning reduces risk because pretrained weights act as regularizers; however, full unfreezing can still overfit if not controlled.- Deployment implications: Pretrained models may be larger; you can mitigate with architecture choice (MobileNet, EfficientNet-lite), pruning, quantization, or knowledge distillation. Training-from-scratch gives full control over architecture but costs more development and validation.Recommended approach (practical steps):1. Start with transfer learning: pick a well-tested backbone (ResNet/ EfficientNet / MobileNet family) pretrained on ImageNet.2. Training plan: - Replace head with task-specific classifier; train only head for several epochs (frozen backbone). - If validation improves, progressively unfreeze last N layers and fine-tune with a low learning rate (e.g., 1e-5–1e-4) and weight decay. - Use strong data augmentation (random crops, color jitter, MixUp/CutMix), class-balanced sampling if needed, and early stopping. - Use cross-validation or stratified holdouts to estimate generalization.3. Regularization & calibration: dropout in head, label smoothing, and mix augmentation. Monitor for overfitting and use learning-rate schedulers.4. If latency/memory is critical: choose a smaller backbone or apply post-training quantization and pruning; optionally distill to a smaller student network.5. When to train from scratch: only if your domain is extremely different from ImageNet (e.g., medical imaging modalities with different low-level statistics) and you can expand labeled data substantially (>>50k) or have access to domain-specific pretraining.Expected outcome & metrics:- Fine-tuning should give higher accuracy and better calibration with far lower compute cost. Track validation accuracy, F1, calibration error, and inference latency/memory.- Deployment: aim for a pruned/quantized fine-tuned model to balance accuracy and production constraints.Justification: Transfer learning leverages useful inductive biases learned from large corpora, reduces overfitting risk on small datasets, shortens development time, and lowers compute cost — making it the pragmatic default for 5,000 labeled images unless clear domain mismatch dictates otherwise.
MediumTechnical
20 practiced
Describe common learning rate schedules: constant, step decay, exponential decay, cosine annealing, warmup, cyclical schedules, and adaptive methods (e.g., reduce on plateau). Explain practical trade-offs, which schedules pair well with large-batch training, and how you would choose or validate a schedule in a production training setup.
Sample Answer
Constant: keep lr fixed. Simple, stable for fine-tuning or convex problems; may waste budget if too large or slow convergence if too small.Step decay: reduce lr by factor at preset epochs (e.g., x0.1 at 30/60/90). Simple and robust; needs tuning of step points; good for reproducible schedules.Exponential decay: continuous multiplicative decay lr = lr0 * exp(-k t). Smooth reduction; fewer hyperparameters but can decay too quickly.Cosine annealing (with or without restarts): lr follows half-cosine from lr_max to lr_min. Encourages exploration early and fine-tuning later; restarts can escape local minima. Works well for modern deep nets.Warmup: start with very small lr and linearly (or gradually) ramp to target over few steps/epochs. Essential for large-batch training and for adaptive optimizers to avoid instability at start.Cyclical schedules: periodically vary lr between bounds (triangular, cosine). Can improve generalization and speed by implicit ensembles; requires cycle length tuning.Adaptive "reduce on plateau": monitor val metric and shrink lr when progress stalls. Convenient automated approach for long runs but reactive and slower to explore.Practical trade-offs:- Simplicity vs adaptivity: fixed/schedule is predictable; adaptive reacts but may overfit to noisy metrics.- Tuning cost: preset schedules need manual knobs; adaptive less manual but can misfire.- Convergence vs generalization: aggressive decay + restarts often improve generalization.Large-batch pairing:- Large batches require smaller initial lr per-example or linear scaling rule with warmup. Warmup + cosine or step decay with long first stage is common. Cosine annealing + warmup (or LR scaling + warmup) works particularly well.Choosing & validating in production:1. Start with known baselines (linear-scaling rule + short warmup, cosine anneal). 2. Use short proxy experiments (smaller dataset or fewer epochs) to scan lr_max, warmup length, and decay hyperparams with Bayesian/ASHA sweeps. 3. Monitor both training loss and validation metrics; use validation-smoothed signals for adaptive steps. 4. Include checkpoints and abort criteria; run ablation comparing final metric and wall-clock cost. 5. For stability, prefer deterministic schedules in critical deployments; enable reduce-on-plateau only if validation is reliable. Validation metrics: final validation score, convergence speed (epochs and wall time), and robustness across seeds.
MediumTechnical
16 practiced
A binary classifier in training always outputs the same probability for every input (for example, 0.5). Provide a structured debugging checklist to identify root causes: data pipeline issues, constant labels, incorrect loss or activation pairing, learning rate or optimizer problems, gradient checks, and logging to add. Include quick unit tests you would run.
Sample Answer
Start with a hypothesis-driven checklist: check data → model → training loop → tooling. For each item I show what to inspect, why it can cause constant outputs, and quick unit tests or logs to run.1) Data pipeline- Check: inputs not all identical / normalized wrongly; labels constant.- Why: identical inputs or constant targets lead to identical outputs or no learning signal.- Logs/tests: - Log batch input mean/std, sample diffs, label unique counts. - Unit test: feed two distinct synthetic samples through preprocessing; assert outputs differ.2) Labels- Check: labels are constant, wrong dtype (e.g., int vs float), or all same class in batch loader.- Why: loss has no gradient when targets constant or mislabeled.- Logs/tests: - Log label distribution per epoch. - Unit test: assert np.unique(labels).size > 1 for training set.3) Activation / loss mismatch- Check: using sigmoid + BCELoss but passing logits; or using BCEWithLogits but applying sigmoid beforehand.- Why: double-sigmoid saturates; wrong input scale yields flat gradients.- Tests: - Unit test: forward known logits through loss functions and confirm expected gradients/signs. - Ensure using BCEWithLogits for logits, or Sigmoid + BCELoss for probabilities.4) Model initialization / architecture- Check: final layer weights zeroed or frozen; batchnorm in inference mode; accidental detach.- Why: zero weights produce constant logits; frozen params prevent updates.- Logs/tests: - Log final-layer weight histogram at start. - Unit test: assert params.requires_grad for model.parameters(); check variance > 0 for final weights.5) Optimizer / learning rate- Check: learning rate zero, wrong parameter group, optimizer not stepping, optimizer state bad.- Why: no parameter updates -> same outputs.- Logs/tests: - Log learning rate per step, parameter norms before/after step. - Unit test: one optimization step on a tiny model + synthetic loss; assert param values change.6) Gradients and training loop- Check: gradients zero due to vanishing, clipping too aggressive, loss.backward() not called, cumulative graph detached.- Why: zero grads -> no updates.- Logs/tests: - Log grad norms per layer, histogram; assert grad_norm > 0. - Unit test (PyTorch): compute forward/backward on toy batch and assert any(p.grad is not None and p.grad.abs().sum()>0).7) Numerical issues / saturation- Check: outputs saturated at 0/1; loss NaN or Inf.- Why: exploding activations or extreme LR.- Logs/tests: - Log batch loss, loss history, check for NaN/Inf. - Unit test: run one batch and verify loss finite.8) Regularization / gradient hooks- Check: custom hooks, zeroing optimizer grads incorrectly, wrong accumulate steps.- Why: inadvertent zeroing stops learning.- Logs/tests: - Review hooks and accumulation logic. - Unit test: minimal loop verifying grads accumulate as expected.Essential logging to add- Batch-level: input mean/std, label counts, logits mean/std, probabilities mean/std, loss.- Model-level: weight histograms, grad histograms, parameter norms, learning rate.- Training loop: loss per step, validation accuracy, examples of worst/best predictions.Quick PyTorch unit tests (small, runnable):Additional small checks- Ensure model.train() is called.- Run a tiny overfit test: train on 10 examples until loss near zero — if it fails, bug in training loop or model.If outputs remain constant after these checks, instrument a minimal end-to-end script (random inputs, random labels) and progressively replace components (loss, optimizer, data loader) until you isolate the failing piece.
python
import torch, torch.nn as nn, torch.optim as optim
# Simple sanity forward/backward test
model = nn.Sequential(nn.Linear(4,1)) # small model
x = torch.tensor([[0.,0,0,0],[1.,2,3,4]])
y = torch.tensor([[0.],[1.]])
opt = optim.SGD(model.parameters(), lr=1.0)
logits = model(x) # raw outputs
probs = torch.sigmoid(logits)
loss = nn.BCEWithLogitsLoss()(logits, y)
loss.backward()
assert any(p.grad is not None and p.grad.abs().sum()>0 for p in model.parameters())
params_before = [p.clone() for p in model.parameters()]
opt.step()
assert any(not torch.allclose(p, pb) for p, pb in zip(model.parameters(), params_before))MediumTechnical
23 practiced
You observe training loss steadily decreasing while validation loss starts increasing after a few epochs. Provide a systematic, step-by-step diagnosis plan to find the cause and a prioritized list of specific fixes you would try (data checks, regularization, early stopping, model capacity changes), explaining why you would try each and how to measure its effect.
Sample Answer
Start with a clear hypothesis: training loss down + validation loss up = overfitting, but could also be data leakage, distribution shift, or metric mismatch. My diagnosis plan (step-by-step) and prioritized fixes:Diagnosis steps1. Quick sanity checks (1–2 hours)- Confirm metrics: ensure training/validation use identical loss function and preprocessing pipelines. Measure: re-run one batch through both pipelines and compare inputs/features.- Check data leakage: verify no overlapping samples/labels between train and val. Measure: compute hashes or sample IDs intersection.2. Inspect learning curves and per-class performance (1–2 hours)- Plot loss & accuracy over epochs for train/val; plot confusion matrix, per-class precision/recall. Measurement: gap magnitude and which classes fail.3. Validate dataset distribution (2–4 hours)- Compare feature distributions (histograms, PCA/t-SNE) between train and val/test. Measurement: KS-test or Wasserstein distance.4. Model behavior checks (2–4 hours)- Overfit a tiny subset (e.g., 100 examples). If model can’t fit tiny set → underfitting or bugs.- Check gradients, weight norms, learning rate dynamics.5. Review complexity & capacity (1–2 hours)- Count params vs data size; check training speed and saturation.Prioritized fixes and why/how to measure1. Early stopping (fast, low risk)- Stop when val loss stops improving (patience). Measure: validation loss/metric stabilized and simpler checkpoint chosen.2. Regularization (dropout, weight decay) (medium)- Add/raise L2 or dropout. Why: reduces co-adaptation and large weights. Measure: narrower train/val gap, improved val metric.3. Data augmentation (medium)- Apply relevant augmentations (images: flips, color jitter; text: paraphrase). Why: increases effective data and reduces overfit. Measure: val metric improvement and robustness tests.4. Reduce model capacity (easy to try)- Smaller network or fewer layers. Why: avoids memorization with limited data. Measure: reduced gap and similar/better val performance.5. Increase training data or clean labels (high impact but costly)- Acquire more labeled data or relabel noisy examples. Measure: sustained val improvements and lower variance across splits.6. Learning rate and optimizer tuning (low-to-medium)- Try lower lr, lr schedules, or gradient clipping. Why: high lr can cause poor generalization. Measure: smoother curves, better minima with lower val loss.7. Ensemble or Bayesian techniques (later)- Use ensembles or MC dropout for uncertainty. Measure: improved val metric and calibrated uncertainties.How to measure effect- Use hold-out test set untouched during tuning. Track primary business metric (e.g., F1, AUC) plus loss. Use cross-validation or repeated runs to ensure changes are statistically significant. Log curves, final gap, and per-class metrics.Final note: follow an iterative loop: diagnose → change one thing at a time → measure on validation and hold-out test → revert/keep. Prioritize quick, low-cost fixes first (early stopping, regularization, augment), then invest in data or architecture changes if needed.
EasyTechnical
19 practiced
Describe backpropagation at a conceptual level suitable for an AI engineer explaining to a non-research audience. Cover what gradients represent, why we propagate error backwards, the role of local derivatives and the chain rule, and how weight updates improve model predictions. Emphasize intuition and practical effects on training rather than heavy math.
Sample Answer
Backpropagation is the practical recipe neural networks use to learn from mistakes. Intuitively: the network makes a prediction, we measure how wrong it is (the loss), and backprop tells each parameter (weight) how much it contributed to that error so we can nudge it in the right direction.Gradients represent the slope of the loss with respect to a parameter — they answer: “If I change this weight a little, will the error go up or down and by how much?” A large gradient means that weight strongly affects the error; a near-zero gradient means it has little effect right now.We propagate error backwards because of the network’s layered structure: outputs depend on previous layers, which depend on earlier ones. Starting from the final error, backprop uses local derivatives (how an individual neuron’s output changes with its input) and the chain rule to combine those small sensitivities into a gradient for each weight. The chain rule lets us compute an overall sensitivity by multiplying local sensitivities along the path from a weight to the loss.Weight updates (typically via gradient descent) move each weight a small step opposite its gradient, reducing the loss for similar inputs. Practically, repeated forward passes, backward gradient computations, and incremental updates make the model’s predictions gradually more accurate. Effects you’ll observe: training loss decreases, validation performance improves (if not overfitting), and learned features in early layers become more useful for the task.
Unlock Full Question Bank
Get access to hundreds of Basic Neural Network Concepts interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.