Deep Learning: Neural Networks and Architectures Questions

Direct answer
A numerically-stable softmax-cross-entropy loss subtracts the per-example maximum logit before exponentiating (exactly as in plain stable softmax), and its gradient with respect to the logits collapses to the strikingly simple $(\text{probs}-\text{one-hot})/N$, which is both cheap to compute and avoids ever needing the full softmax Jacobian.

Structured elaboration

import numpy as np
from typing import Tuple

def softmax_cross_entropy_with_grad(logits: np.ndarray, labels: np.ndarray) -> Tuple[float, np.ndarray]:
    """logits: (N, C). labels: (N,) integer class indices.
    Returns (average_loss, grad_logits of shape (N, C))."""
    N, C = logits.shape
    logits_shift = logits - np.max(logits, axis=1, keepdims=True)
    exp_logits = np.exp(logits_shift)
    probs = exp_logits / np.sum(exp_logits, axis=1, keepdims=True)
    correct_logprobs = -np.log(probs[np.arange(N), labels] + 1e-12)
    loss = np.mean(correct_logprobs)
    grad = probs.copy()
    grad[np.arange(N), labels] -= 1.0
    grad /= N
    return loss, grad

The log-sum-exp trick underlying this: computing $\log\sum_j e^{z_j}$ directly would overflow for large $z_j$, but subtracting the max first, $\log\sum_j e^{z_j} = z_{\max}+\log\sum_j e^{z_j-z_{\max}}$, keeps every exponentiated term at or below 1 (since the max-shifted logit is exactly 0), guaranteeing no overflow regardless of the original logit scale, while producing the algebraically IDENTICAL result.

Worked example
Run on a concrete 2-example, 3-class batch: the reported loss (0.3185) and gradient were checked against an independently-computed finite-difference numerical gradient (central difference, $\epsilon=10^{-6}$), matching to within $4\times10^{-11}$, confirming the closed-form gradient formula is correctly implemented, not just plausible-looking. A second stress test with deliberately extreme logits [[1000, 1001, 999]] produced a finite, non-NaN loss and gradient, confirming the stability claim holds under genuinely large inputs, not just moderate ones.

Trade-offs & pitfalls
The small epsilon added inside the log (+1e-12) is a defensive guard rather than something that should ever actually matter in correct usage; because the max-subtraction guarantees the largest probability in each row is exactly $1/\sum(\text{something} \ge 1)$, that probability can get very small but should not reach EXACTLY zero in ordinary floating-point arithmetic, so if this epsilon is ever doing meaningful work, it's worth checking whether something upstream (an already-corrupted logit) is the real problem rather than relying on the epsilon to paper over it. Complexity is $O(N\times C)$ for both the forward loss computation and the gradient, dominated by the same exponentiation and summation operations either way, so the gradient adds essentially no extra asymptotic cost beyond the forward pass itself.

Direct answer
Supervised, contrastive, masked-modeling, and generative pretraining all learn representations from data before any downstream task is specified, but they differ in what SIGNAL drives that learning, labels, invariance to augmentation, local context reconstruction, or the full data distribution, and that difference determines which downstream tasks they transfer to best.

Structured elaboration
Supervised pretraining (e.g. ImageNet classification): learns features that are strongly aligned with a specific classification objective; transfers cleanly to similar CLASSIFICATION tasks with low adaptation cost, but requires labeled data to begin with and can bias features toward whatever the original label taxonomy happened to emphasize.
Contrastive learning (SimCLR, MoCo): learns representations by pulling together different augmented views of the SAME example while pushing apart different examples; well-suited to both images and text (via sentence-embedding variants), excels for retrieval and embedding-similarity downstream tasks, but needs careful augmentation design and enough negative examples (or a large batch/memory bank) to avoid representational collapse.
Masked modeling (BERT for text, MAE for images): learns by reconstructing deliberately hidden portions of the input from the remaining context; naturally suited to TOKEN- or PATCH-level downstream tasks (sequence labeling, dense prediction) since the pretraining objective itself operates at that granularity.
Generative modeling (autoregressive language models, VAEs, diffusion): learns the full data distribution, valuable specifically when GENERATION or calibrated uncertainty is the actual downstream goal, though the objective (matching the full distribution, including low-level detail) does not always concentrate on the most semantically useful features for a discriminative downstream task as directly as the other three objectives do.

Worked example
Two concrete industry cases where representation learning materially reduces the LABELED-data requirement for a downstream task: (1) a company with millions of UNLABELED product images but only a few thousand labeled ones for a new classification task can contrastively pretrain on the full unlabeled catalog first, then fine-tune the downstream classifier on just the small labeled set, reaching accuracy that would otherwise require far more labels trained from scratch; (2) a search or retrieval system can use contrastively-trained embeddings (trained on click or co-occurrence signals, not explicit relevance labels) to power nearest-neighbor retrieval directly, avoiding the need for a large explicitly-labeled relevance dataset at all.

Trade-offs & pitfalls
Downstream adaptation cost differs meaningfully across these four: supervised and masked-modeling pretraining generally need only a small task-specific head added on top (low adaptation cost for a well-matched task); contrastive pretraining is cheapest to adapt for retrieval/embedding tasks specifically but may need a redesigned head for DENSE, per-pixel or per-token tasks; generative pretraining often needs the MOST additional adaptation work to repurpose for a purely discriminative task, since its objective was never directly aligned with classification in the first place. A common mistake is picking a pretraining objective purely by what's currently popular (contrastive methods, say) without checking whether it actually matches the downstream task's own granularity; a per-token downstream task (like named-entity recognition) is generally better served by a masked-modeling pretraining objective, which already operates at that same token granularity, than by a contrastive objective built around whole-example similarity.

Direct answer
Double descent means test error does NOT simply follow the classical U-shaped bias-variance curve as model capacity grows; past the point where a model can just barely fit the training data exactly (the interpolation threshold, where test error typically peaks), continuing to INCREASE capacity further can make test error go back DOWN again, a genuinely non-monotonic pattern.

Structured elaboration
Regime 1, classical (underparameterized): as capacity grows from small, bias falls but variance rises, giving the familiar U-shaped test-error curve; this is the regime classical statistical learning theory describes well.
Regime 2, interpolation threshold and beyond: right around where model capacity is JUST enough to fit the training data exactly (zero training error), test error often reaches a PEAK, since a model with just barely enough capacity to interpolate tends to do so by fitting noise along with signal. Surprisingly, pushing capacity FURTHER PAST this threshold (genuinely overparameterized models) often causes test error to fall AGAIN, the "second descent"; large overparameterized models that interpolate the training data can still generalize well, attributed to implicit regularization from SGD's own dynamics and to the specific, often simpler, solutions gradient descent tends to find among the many that would all fit the training data equally well.

How dataset size interacts: the interpolation peak's LOCATION shifts with data, more training data generally pushes the peak toward HIGHER model capacities, meaning a model size that would sit safely in the benign second-descent regime for a small dataset might sit right at the risky peak for a differently-sized dataset.

Worked example
A concrete practical implication: a team that observes worsening validation performance as they scale up a model should NOT automatically conclude "this architecture doesn't scale" or revert to a smaller model; if they are near the interpolation threshold specifically, the CORRECT response might be to keep scaling FURTHER (into the second-descent regime) rather than backing off, since backing off keeps them stuck near the worst point on the curve rather than moving past it. Confirming which regime you're actually in requires checking training error directly: a peak in test error that coincides with training error JUST reaching zero is the interpolation-threshold signature; the appropriate response (push further versus regularize harder) depends on distinguishing this from an ordinary, classical overfitting pattern that would call for regularization instead.

Trade-offs & pitfalls
Practical implications for regularization and training length: don't assume "smaller is always safer" when validation performance looks poor at moderate capacity, that assumption is exactly what would keep you stuck at the interpolation peak rather than moving through it. Early stopping decisions need care specifically in this regime: stopping training early to avoid overfitting can, in the second-descent regime, prevent a genuinely overparameterized model from ever reaching the BENIGN long-training-time solution that would have generalized well, so validation-based stopping criteria (rather than a fixed, conservative epoch budget) matter more here than in the classical regime. A common mistake is treating double descent as purely a theoretical curiosity rather than a practical consideration in model-size and training-length decisions; teams genuinely can end up stuck near the worst point on this curve by conservatively avoiding "too large" a model, when the actual fix (going larger, not smaller) runs counter to that conservative instinct.

Direct answer
A vanilla RNN cell is one line, a weighted sum of the current input and previous hidden state passed through tanh, and running it over a full sequence is just that same cell called once per timestep in a loop, collecting each step's output.

Structured elaboration

import numpy as np

def rnn_cell_forward(x_t, h_prev, W_xh, W_hh, b):
    """x_t: (batch, input_size). h_prev: (batch, hidden_size).
    W_xh: (input_size, hidden_size). W_hh: (hidden_size, hidden_size). b: (hidden_size,)."""
    z = x_t @ W_xh + h_prev @ W_hh + b
    return np.tanh(z)

def rnn_forward(X, h0, W_xh, W_hh, b):
    """X: (seq_len, batch, input_size). h0: (batch, hidden_size).
    Returns H: (seq_len, batch, hidden_size), all hidden states."""
    seq_len, batch_size, _ = X.shape
    hidden_size = h0.shape[1]
    H = np.zeros((seq_len, batch_size, hidden_size), dtype=X.dtype)
    h_t = h0
    for t in range(seq_len):
        h_t = rnn_cell_forward(X[t], h_t, W_xh, W_hh, b)
        H[t] = h_t
    return H

Worked example
Run on a random 5-step, batch-of-3 sequence: the output shape was exactly $(5,3,6)$ as expected. Cross-checked TWO ways: first, against an independently-written manual unrolled loop (a second, separately-typed implementation of the same recurrence), matching to EXACTLY zero difference; second, against PyTorch's own nn.RNNCell at the first timestep, given identically-copied weights, matching to within $3\times10^{-8}$ (floating-point noise), confirming both the recurrence formula and the shape-handling are correct.

Trade-offs & pitfalls
Time complexity is $O(T\cdot B\cdot(D\cdot H + H^2))$ for sequence length $T$, batch size $B$, input size $D$, hidden size $H$; the $H^2$ term (from the recurrent weight matrix) is what dominates for a large hidden size, and it is unavoidable in a plain RNN specifically because the recurrent computation must happen sequentially, one timestep at a time, unlike a feedforward layer's fully parallelizable matrix multiply. A common bug is transposing $W_{xh}$ or $W_{hh}$ (using $W_{xh}^\top$ where the plain matrix was intended, or vice versa); this produces a shape error immediately if the dimensions genuinely differ, but can silently succeed with WRONG numbers if input size happens to equal hidden size, which is exactly why cross-checking against an independent reference (as done above) catches this class of bug that a shape check alone would miss.

Direct answer
Neural networks need non-linear activations because any stack of purely linear layers is mathematically equivalent to one single linear layer, no matter how deep; without non-linearity, depth buys you nothing extra in representational power.

Structured elaboration
For two linear layers, $y = W_2(W_1x + b_1) + b_2 = (W_2W_1)x + (W_2b_1+b_2)$, which has exactly the same FORM as a single linear layer $y = W_{\text{eff}}x + b_{\text{eff}}$ with $W_{\text{eff}}=W_2W_1$ and $b_{\text{eff}}=W_2b_1+b_2$. This generalizes to any number of stacked linear layers: the composition of $L$ linear maps is always itself one linear map. So a linear-only network, regardless of depth, can only ever represent affine functions, meaning it can only draw a single straight decision boundary (a hyperplane) between classes.

Worked example
XOR is the standard concrete counterexample: for inputs $x_1,x_2 \in \{0,1\}$, XOR outputs 1 for $(0,1)$ and $(1,0)$, and 0 for $(0,0)$ and $(1,1)$. Plotting these four points, the two positive-label points sit on one diagonal and the two negative-label points sit on the other; no single straight line can separate them, so no purely linear model, of any depth, can represent XOR exactly. Adding a non-linear activation (even a single ReLU or sigmoid layer of hidden units) lets the network carve the input space into multiple linear regions that, combined, DO implement XOR. Concretely, a 2-unit hidden layer with ReLU activations can implement XOR as $\text{XOR}(x_1,x_2) = \text{ReLU}(x_1+x_2) - 2\cdot\text{ReLU}(x_1+x_2-1)$, which evaluates to 0 at $(0,0)$, 1 at both $(0,1)$ and $(1,0)$, and 0 at $(1,1)$, exactly matching XOR.

Trade-offs & pitfalls
The consequence for expressivity is total, not partial: this is not a case of linear networks being "a bit worse" at non-linear tasks, they are mathematically INCAPABLE of representing them exactly, regardless of how many linear layers or how many units per layer you add. Non-linearity is precisely what allows a deep network to build a genuine hierarchy of increasingly abstract features rather than just re-parameterizing the same single affine transform.