Deep Learning: Neural Networks and Architectures Questions

Direct answer
Vanishing and exploding gradients happen because backpropagation computes a gradient at an early layer as a product of many per-layer Jacobians; if those Jacobians are consistently smaller than 1 in the relevant directions the product shrinks toward zero across many layers, and if they are consistently larger than 1 it grows without bound.

Structured elaboration
Concretely, for a gradient reaching back $k$ layers, $\dfrac{\partial L}{\partial h_k} = \Big(\prod_{t=k+1}^{T} J_t\Big)\dfrac{\partial L}{\partial h_T}$ where each $J_t$ is a layer's (or timestep's) local Jacobian. If the spectral norm (largest singular value) of each $J_t$ is below 1, the product decays roughly exponentially in the number of layers, giving vanishing gradients; if above 1, it grows exponentially, giving exploding gradients. Eigenvalues describe amplification along specific directions in the weight matrix (an eigenvalue with magnitude below 1 shrinks components along its eigenvector, above 1 grows them); for matrices that are not symmetric or normal, singular values (which govern how the matrix scales VECTOR NORM in the worst case) matter more directly than eigenvalues, because transient amplification can occur even when all eigenvalues sit inside the unit circle.

At least four practical mitigations, and why each works:

1. Better weight initialization (e.g. orthogonal or variance-scaled schemes): keeps each layer's Jacobian singular values close to 1 from the start, rather than relying on training to fix a badly-scaled network.
2. Normalization layers (BatchNorm, LayerNorm): rescale activations at every layer, which keeps the effective Jacobian's conditioning stable regardless of how the raw pre-normalization values would have drifted.
3. Residual (skip) connections: add an identity path around each block, so the EFFECTIVE Jacobian of a residual block is close to the identity matrix plus a small correction, which does not compound toward zero or infinity across many stacked blocks the way a pure product of arbitrary Jacobians would.
4. Gated recurrent units (LSTM/GRU): introduce an additive, gated path for the cell state that lets gradients flow across many timesteps without being forced through repeated multiplicative shrinkage.
5. Gradient clipping: caps the gradient's norm (or per-element value) after it is computed, which does not prevent the underlying instability but stops one single bad batch from destabilizing the whole training run; it addresses exploding gradients specifically, not vanishing ones.

Worked example
Consider a toy 50-layer linear network where every layer's weight matrix has been scaled so its largest singular value is 0.9. The product of 50 such Jacobians has spectral norm on the order of $0.9^{50} \approx 0.0052$, meaning a gradient reaching the earliest layer is attenuated to roughly 0.5% of its original magnitude purely from this multiplicative effect, even before considering the activation function's own derivative. If instead each layer's largest singular value were 1.1, the same computation gives $1.1^{50}\approx 117.4$, an over hundred-fold amplification. This is why keeping each layer's effective Jacobian close to a singular-value of 1 (via good initialization, normalization, or residual connections) is the common thread across all of the mitigations above.

Trade-offs & pitfalls
A frequent mistake is treating gradient clipping as a fix for vanishing gradients; it only bounds gradients from ABOVE, so it does nothing when the true problem is a gradient that is shrinking toward zero. Another common gap is applying only one mitigation (say, just clipping) rather than recognizing that in practice these are usually combined (orthogonal-ish initialization, normalization, and residual or gated paths together), because no single technique fully resolves both directions of the problem across a genuinely deep or long-sequence network.

Direct answer
ReLU, sigmoid, tanh, and softmax are the four activation functions that come up in almost every deep-learning interview: ReLU and its variants dominate hidden layers, sigmoid and softmax dominate output layers for binary and multi-class classification respectively, and tanh appears in some hidden layers (notably recurrent networks) where a zero-centered output helps.

Structured elaboration

Worked example
For logits $z = [1000, 1001]$ (values chosen to show the numerical-stability issue): naive softmax computes $e^{1000}$ and $e^{1001}$, both of which overflow in floating point. The stable form subtracts the max first: $z' = z - \max(z) = [-1, 0]$, giving $e^{-1} \approx 0.368$ and $e^{0} = 1$, sum $\approx 1.368$, so softmax $\approx [0.269, 0.731]$, exactly the correct answer computed without overflow.

Trade-offs & pitfalls
A common miscalibration is placing sigmoid or tanh in many stacked hidden layers of a deep network; both saturate for inputs far from zero, and a deep stack of them compounds into vanishing gradients, which is a large part of why ReLU-family activations became the default for hidden layers once networks got deep. A second pitfall is confusing multi-class (mutually exclusive, use softmax) with multi-label (independent per-class probabilities, use one sigmoid per label) and picking the wrong activation/loss pairing for the task.

Direct answer
With only 2,000 labeled medical images, transfer learning from a pretrained CNN backbone is almost always the right starting point over training from scratch, but the real work is in HOW you fine-tune, since naive full fine-tuning on this little data risks both overfitting and inheriting biases from a backbone that never saw medical imagery.

Structured elaboration
Advantages of transfer learning here: a backbone pretrained on a large natural-image dataset already encodes general low-level visual features (edges, textures, simple shapes) that transfer to SOME degree even across a real domain gap, dramatically reducing how much this specific 2,000-image dataset needs to teach the network from scratch, both in convergence speed and in overfitting risk.
Domain-shift pitfalls specific to medical imaging: natural-image pretraining (typically RGB photographs) may poorly match medical imaging modalities (grayscale X-rays, different contrast and texture statistics than natural photos), so the EARLY layers' features may transfer less cleanly than they would for another natural-image task; medical datasets also commonly carry their own label noise and severe class imbalance (rare pathology classes), both of which interact badly with naive full fine-tuning on a small dataset; and strong validation performance on THIS dataset may still fail to generalize to a different hospital, scanner, or patient population, a distinct risk from ordinary overfitting.

Fine-tuning strategy to balance these concerns: replace the final classifier head with a new, randomly-initialized one sized for the target task; FIRST train only this head with the backbone entirely frozen, letting the classifier adapt to the pretrained features without risking damage to those features from noisy early gradients; THEN progressively unfreeze the last block or two of the backbone specifically (not the whole network at once), using a substantially LOWER learning rate for these newly-unfrozen pretrained layers than for the head, since they already encode useful information that a large update could destroy. Pair this with strong, medically-appropriate data augmentation (contrast/intensity jitter, elastic deformations, rotations, being careful that any augmentation preserves the clinical meaning of the image) and either weight decay, dropout, or label smoothing as standard small-data regularizers.

Worked example
A concrete validation protocol that directly addresses the generalization-risk pitfall: use stratified k-fold cross-validation within the available 2,000 images to estimate variance in performance, but ALSO hold out data from a genuinely different source (a different imaging site or scanner, if any is available) specifically to probe whether performance is inflated by overfitting to this one population's imaging characteristics rather than the underlying pathology; a model that performs well on the in-distribution stratified folds but noticeably worse on the held-out different-source data is showing exactly the domain-shift risk this fine-tuning strategy needs to guard against.

Trade-offs & pitfalls
A common mistake specific to this scenario is fully unfreezing and fine-tuning the ENTIRE backbone from the very first epoch on only 2,000 images; this frequently causes the model to overwrite the pretrained backbone's genuinely useful general features with noise fit to this small dataset, before the new head has had any chance to provide a stable, sensible training signal, which the staged freeze-then-progressively-unfreeze approach specifically avoids. If domain shift turns out to be severe even after careful fine-tuning (validation on the different-source holdout remains poor), the next escalation is self-supervised pretraining on whatever UNLABELED medical images from the target domain might be available, rather than continuing to push harder on supervised fine-tuning from a natural-image backbone.

Direct answer
Triplet loss, contrastive loss, and InfoNCE/NT-Xent all pull similar examples together and push dissimilar ones apart in an embedding space, but they differ in how many negatives they compare against at once, InfoNCE's multi-way comparison against many negatives simultaneously is what makes it converge faster and scale better to large, self-supervised training than the older pairwise or triplet forms.

Structured elaboration
Triplet loss: compares one anchor against one positive and one negative at a time, $L=\max(0, d(a,p)-d(a,n)+\text{margin})$; good for fine-grained ranking tasks, but requires explicitly SAMPLING triplets, and convergence is slow without careful (hard or semi-hard) negative mining.
Contrastive loss: operates on labeled PAIRS (same-class or different-class), pulling positive pairs together and pushing negative pairs apart past a margin; simpler to set up than triplet loss, but less directly aligned with a RANKING objective.
InfoNCE/NT-Xent: a SOFTMAX-based objective comparing one positive against MANY negatives simultaneously per anchor, using cosine similarity scaled by a temperature; this multi-way comparison structure is what typically converges fastest and scales best, especially in self-supervised settings where negatives can be drawn from the rest of a large batch or a stored memory bank.

Mining strategies: HARD negatives (closest to the anchor) give the strongest learning signal per example but risk collapse or instability if used too aggressively, especially under label noise; SEMI-HARD negatives (farther than the true positive but still violating the margin) are a more stable middle ground, historically popularized by FaceNet. InfoNCE's IN-BATCH negative structure largely sidesteps explicit mining, since every other example in the batch automatically serves as a negative.

Batch size and temperature: InfoNCE's effective negative pool grows directly with batch size, so LARGER batches generally improve its performance up to hardware memory limits; when batch size is constrained, a MEMORY BANK (as in MoCo, using a momentum-updated encoder plus a queue of recent embeddings) supplies additional negatives without needing an actually larger batch. Temperature $\tau$ controls how sharply the softmax concentrates on the hardest negatives, lower $\tau$ increases the penalty on the closest (hardest) negatives specifically, common values fall roughly in the 0.05 to 0.2 range, tuned against validation, and always paired with L2-NORMALIZED embeddings when using cosine similarity, since temperature's effect is only well-calibrated when the embedding norm itself isn't also varying uncontrolled.

Worked example
A concrete scaling decision to millions of examples: for a fixed, modest GPU budget, a MoCo-style momentum encoder plus a large negative QUEUE (tens of thousands of stored embeddings, refreshed continuously as training proceeds) reaches a comparable effective negative-pool size to a SimCLR-style approach that instead relies on genuinely large batches spread across many GPUs via an all-gather; the memory-bank approach trades a small amount of NEGATIVE STALENESS (queued embeddings were computed by a slightly earlier version of the encoder) for dramatically lower per-step memory and compute requirements.

Trade-offs & pitfalls
A common mistake is increasing batch size (or negative-queue size) as a blanket fix for weak InfoNCE performance without also re-tuning temperature; the two interact, since the effective difficulty of the multi-way classification task InfoNCE solves scales with the number of negatives, and a temperature tuned for a small negative pool can behave quite differently once the pool grows substantially larger. A second common gap is neglecting to monitor for REPRESENTATIONAL COLLAPSE (all embeddings converging toward a single point, trivially satisfying the loss); tracking metrics like the DISTRIBUTION of pairwise embedding similarities, or downstream retrieval accuracy on a small held-out k-NN probe, catches this failure mode long before it would be obvious from the training loss curve alone, since a collapsed representation can still report a deceptively low training loss.

Direct answer
Supervised, contrastive, masked-modeling, and generative pretraining all learn representations from data before any downstream task is specified, but they differ in what SIGNAL drives that learning, labels, invariance to augmentation, local context reconstruction, or the full data distribution, and that difference determines which downstream tasks they transfer to best.

Structured elaboration
Supervised pretraining (e.g. ImageNet classification): learns features that are strongly aligned with a specific classification objective; transfers cleanly to similar CLASSIFICATION tasks with low adaptation cost, but requires labeled data to begin with and can bias features toward whatever the original label taxonomy happened to emphasize.
Contrastive learning (SimCLR, MoCo): learns representations by pulling together different augmented views of the SAME example while pushing apart different examples; well-suited to both images and text (via sentence-embedding variants), excels for retrieval and embedding-similarity downstream tasks, but needs careful augmentation design and enough negative examples (or a large batch/memory bank) to avoid representational collapse.
Masked modeling (BERT for text, MAE for images): learns by reconstructing deliberately hidden portions of the input from the remaining context; naturally suited to TOKEN- or PATCH-level downstream tasks (sequence labeling, dense prediction) since the pretraining objective itself operates at that granularity.
Generative modeling (autoregressive language models, VAEs, diffusion): learns the full data distribution, valuable specifically when GENERATION or calibrated uncertainty is the actual downstream goal, though the objective (matching the full distribution, including low-level detail) does not always concentrate on the most semantically useful features for a discriminative downstream task as directly as the other three objectives do.

Worked example
Two concrete industry cases where representation learning materially reduces the LABELED-data requirement for a downstream task: (1) a company with millions of UNLABELED product images but only a few thousand labeled ones for a new classification task can contrastively pretrain on the full unlabeled catalog first, then fine-tune the downstream classifier on just the small labeled set, reaching accuracy that would otherwise require far more labels trained from scratch; (2) a search or retrieval system can use contrastively-trained embeddings (trained on click or co-occurrence signals, not explicit relevance labels) to power nearest-neighbor retrieval directly, avoiding the need for a large explicitly-labeled relevance dataset at all.

Trade-offs & pitfalls
Downstream adaptation cost differs meaningfully across these four: supervised and masked-modeling pretraining generally need only a small task-specific head added on top (low adaptation cost for a well-matched task); contrastive pretraining is cheapest to adapt for retrieval/embedding tasks specifically but may need a redesigned head for DENSE, per-pixel or per-token tasks; generative pretraining often needs the MOST additional adaptation work to repurpose for a purely discriminative task, since its objective was never directly aligned with classification in the first place. A common mistake is picking a pretraining objective purely by what's currently popular (contrastive methods, say) without checking whether it actually matches the downstream task's own granularity; a per-token downstream task (like named-entity recognition) is generally better served by a masked-modeling pretraining objective, which already operates at that same token granularity, than by a contrastive objective built around whole-example similarity.