Deep Learning: Neural Networks and Architectures Questions

Direct answer
A training-epoch function needs exactly five things in order: put the model in train mode, run the forward and backward pass per batch, clip gradients, step the optimizer, and reset gradients before the next batch, while tracking a running loss to report at the end.

Structured elaboration
Approach: set model.train() once at the top (this activates dropout and lets BatchNorm use per-batch statistics); for each batch, move data to the target device, zero the optimizer's gradients, run the forward pass, compute the loss, call backward(), clip gradient norms to bound any single bad batch's update size, then call optimizer.step(). Accumulate a per-sample-weighted running loss (not a simple per-batch average) so the final epoch average is correct even if the last batch is a different size than the others.

import torch
import torch.nn as nn
from torch.nn.utils import clip_grad_norm_

def train_one_epoch(model, dataloader, optimizer, device):
    model.train()
    criterion = nn.CrossEntropyLoss(reduction='none')
    total_loss = 0.0
    total_samples = 0

    for inputs, targets in dataloader:
        inputs = inputs.to(device)
        targets = targets.to(device)

        optimizer.zero_grad()
        outputs = model(inputs)
        losses = criterion(outputs, targets)
        loss = losses.mean()

        loss.backward()
        clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()

        total_loss += losses.sum().item()
        total_samples += inputs.size(0)

    if total_samples == 0:
        return 0.0
    return total_loss / total_samples

Worked example
Tested against a small 2-layer network (8 to 16 to 4 units) trained on 40 random examples in batches of 8, for two consecutive epochs with plain SGD: the function ran cleanly, returned a plain float loss each time, and the second epoch's average loss (1.460) was lower than the first (1.466), consistent with the optimizer making progress. An explicit empty-dataloader edge case (0 samples) returned exactly 0.0 rather than raising a divide-by-zero error, confirming the guard clause works. Mixed-precision support would slot in as: wrap the forward pass and loss computation in with torch.cuda.amp.autocast():, replace loss.backward() with scaler.scale(loss).backward(), unscale gradients before clipping (scaler.unscale_(optimizer)), then scaler.step(optimizer) and scaler.update() in place of the plain optimizer.step().

Trade-offs & pitfalls
A common bug is calling clip_grad_norm_ before loss.backward() (it has nothing to clip yet) or calling optimizer.zero_grad() AFTER backward() (which discards the gradients you just computed); the order shown above, zero, forward, backward, clip, step, is the one that is actually correct. A second common gap is averaging the loss per BATCH and then averaging those batch-averages at the end, which silently gives the WRONG epoch average whenever batch sizes are not all identical (e.g. a smaller final batch); accumulating the per-sample SUM and dividing once by the total sample count, as done here, avoids that bias.

Direct answer
Padding lets you batch variable-length sequences into one fixed-size tensor, but every padded position must then be explicitly masked out, in the model's own computation, and again in the loss, or it silently wastes compute and corrupts gradients.

Structured elaboration
For RNNs specifically, the standard tool is pack_padded_sequence/pad_packed_sequence (in PyTorch): sort the batch by length, pack it before feeding the RNN, and the recurrence then genuinely SKIPS computation on padded timesteps rather than just computing garbage on them and hoping it's ignored later; results are unpacked and, if the batch was sorted, un-sorted back to the original order afterward. Loss masking is a separate, equally necessary step: multiply the per-token loss by a length-derived mask (or use a dedicated ignore_index for padding tokens in a cross-entropy loss) BEFORE reducing to a scalar, since an unmasked loss will otherwise average in the (meaningless) loss computed at every padded position, systematically biasing the reported loss and its gradient.

Avoiding wasted compute beyond correctness: bucket sequences into similar-length groups before batching, which reduces the average padding ratio per batch (a batch mixing a length-5 and a length-500 sequence wastes enormous compute on padding for the short one); use TOKEN-based (rather than fixed-sequence-count) batching, where batch size is chosen so the TOTAL token count stays roughly constant, packing more short sequences and fewer long ones into each batch; and monitor the padding ratio (total padded tokens divided by total tokens) as an explicit metric to catch a batching strategy that has drifted toward heavy waste.

Worked example
A concrete illustration of the loss-masking bug: for a batch of 3 sequences with true lengths 5, 3, and 8 padded to a common length of 8, an UNMASKED per-token cross-entropy loss averages over $3\times8=24$ positions, but $24-(5+3+8)=8$ of those are pure padding with an arbitrary target (commonly index 0), so roughly a third of the reported loss and its gradient are computed against meaningless padding targets; masking correctly averages only over the $5+3+8=16$ REAL positions.

Trade-offs & pitfalls
A common mistake specific to RNNs is applying BatchNorm across the sequence dimension without accounting for padding; BatchNorm's statistics would then be computed over a mix of real and padded (often zero-valued) activations, systematically skewing the normalization, which is one reason LayerNorm (which normalizes per-sample, not across the batch/sequence dimension) is generally the safer default for sequence models. A second common gap is forgetting to un-sort the model's outputs back to the ORIGINAL batch order after a sort-then-pack step; this does not raise an error (every tensor still has a valid shape), but it silently pairs each prediction with the WRONG label, since the sort permutation was never reversed.

Direct answer
A numerically-stable softmax-cross-entropy loss subtracts the per-example maximum logit before exponentiating (exactly as in plain stable softmax), and its gradient with respect to the logits collapses to the strikingly simple $(\text{probs}-\text{one-hot})/N$, which is both cheap to compute and avoids ever needing the full softmax Jacobian.

Structured elaboration

import numpy as np
from typing import Tuple

def softmax_cross_entropy_with_grad(logits: np.ndarray, labels: np.ndarray) -> Tuple[float, np.ndarray]:
    """logits: (N, C). labels: (N,) integer class indices.
    Returns (average_loss, grad_logits of shape (N, C))."""
    N, C = logits.shape
    logits_shift = logits - np.max(logits, axis=1, keepdims=True)
    exp_logits = np.exp(logits_shift)
    probs = exp_logits / np.sum(exp_logits, axis=1, keepdims=True)
    correct_logprobs = -np.log(probs[np.arange(N), labels] + 1e-12)
    loss = np.mean(correct_logprobs)
    grad = probs.copy()
    grad[np.arange(N), labels] -= 1.0
    grad /= N
    return loss, grad

The log-sum-exp trick underlying this: computing $\log\sum_j e^{z_j}$ directly would overflow for large $z_j$, but subtracting the max first, $\log\sum_j e^{z_j} = z_{\max}+\log\sum_j e^{z_j-z_{\max}}$, keeps every exponentiated term at or below 1 (since the max-shifted logit is exactly 0), guaranteeing no overflow regardless of the original logit scale, while producing the algebraically IDENTICAL result.

Worked example
Run on a concrete 2-example, 3-class batch: the reported loss (0.3185) and gradient were checked against an independently-computed finite-difference numerical gradient (central difference, $\epsilon=10^{-6}$), matching to within $4\times10^{-11}$, confirming the closed-form gradient formula is correctly implemented, not just plausible-looking. A second stress test with deliberately extreme logits [[1000, 1001, 999]] produced a finite, non-NaN loss and gradient, confirming the stability claim holds under genuinely large inputs, not just moderate ones.

Trade-offs & pitfalls
The small epsilon added inside the log (+1e-12) is a defensive guard rather than something that should ever actually matter in correct usage; because the max-subtraction guarantees the largest probability in each row is exactly $1/\sum(\text{something} \ge 1)$, that probability can get very small but should not reach EXACTLY zero in ordinary floating-point arithmetic, so if this epsilon is ever doing meaningful work, it's worth checking whether something upstream (an already-corrupted logit) is the real problem rather than relying on the epsilon to paper over it. Complexity is $O(N\times C)$ for both the forward loss computation and the gradient, dominated by the same exponentiation and summation operations either way, so the gradient adds essentially no extra asymptotic cost beyond the forward pass itself.

Direct answer
Depth and width are not interchangeable ways to add capacity: depth buys compositional, hierarchical representational power efficiently but makes optimization harder, while width is easier to optimize and parallelize but needs disproportionately more parameters to match what depth can express for structured tasks.

Structured elaboration
Representational capacity: a wide enough single hidden layer can approximate any continuous function (the universal approximation theorem), but doing so for a genuinely compositional function can require exponentially many units, whereas a deep network can represent the same function with exponentially fewer parameters by reusing intermediate features across layers.
Optimization difficulty: deeper networks are harder to train, since a deeper stack pushes gradients through more repeated Jacobian multiplications (vanishing/exploding gradients) and a rougher loss landscape; residual connections, normalization layers, and careful initialization are what make depth beyond roughly 20 layers practically trainable at all. Very wide networks, by contrast, behave closer to a well-conditioned, near-convex optimization problem (the Neural Tangent Kernel regime), which is part of why they can be easier to train stably, at the cost of needing far more memory and compute per layer.
Parameter efficiency: for tasks with real hierarchical structure (most perception and language tasks), depth is the more parameter-efficient way to add capacity; for a task with no real compositional structure, adding width may reach the same accuracy with less optimization difficulty even if it costs more parameters.
Generalization: this connects directly to the double-descent phenomenon, where increasing model size (whether depth or width) past the point of exactly fitting the training data can, counterintuitively, continue to IMPROVE validation performance rather than overfitting further; this means "more capacity" is not automatically bad for generalization the way classical bias-variance intuition alone would suggest, though the practical implication is still to validate on held-out data rather than reasoning from capacity alone.

Worked example
A concrete illustration of the parameter-efficiency claim: representing a function that is the composition of $k$ simple pairwise interactions can require on the order of $2^k$ hidden units in a single wide layer to capture directly, but only $O(k)$ units spread across $k$ depth layers if each layer can build on the previous one's output, precisely because depth lets you REUSE the same small set of primitive features at every stage rather than needing a distinct unit for every possible combination.

Trade-offs & pitfalls
Practical guidance under compute and latency constraints: prefer deeper architectures WITH residual connections when the task has real hierarchical structure and you can afford the training-time complexity of tuning a deep network; prefer wider (but shallower) architectures when training stability, parallelism, or a hard latency ceiling on sequential compute matters more than squeezing out maximum parameter efficiency (a wide network's operations parallelize better on modern accelerators, since there is less sequential dependency between layers). A common mistake is treating "add more layers" as a free capacity increase; past a certain depth without residuals and normalization, additional layers can actively HURT training by making optimization harder without giving any additional representational benefit that the optimizer can actually reach.

Direct answer
When the metric you actually care about (F1, top-k accuracy) isn't differentiable, you can't optimize it directly with gradient descent, so the practical choices are training on a smooth SURROGATE loss and hoping it correlates well enough, or tuning a decision threshold AFTER training to better match the real metric, with structured-prediction losses as a heavier-weight option in between.

Structured elaboration
Surrogate losses (cross-entropy, hinge): simple, stable, and well-optimized by standard training infrastructure; the risk is that the surrogate is only an INDIRECT proxy for the real target metric, and optimizing it perfectly does not guarantee the real metric is also optimal, particularly under class imbalance where a surrogate averaged over all examples may not track a metric (like F1) that specifically weights the minority class.
Post-hoc threshold optimization: after training with a surrogate loss, sweep the decision threshold on a held-out validation set to directly maximize the actual target metric (F1, precision at a fixed recall, etc.); this is cheap, requires no change to training at all, but needs a genuinely representative validation set to avoid overfitting the threshold itself to validation-set noise.
Structured-prediction losses (CRFs, structured SVMs, direct differentiable approximations of F-measure): explicitly model the metric's own structure or dependencies in the loss itself, which can more directly target the real objective, at the cost of more complex, often slower training (sometimes requiring an inference step inside the training loop itself) and added engineering complexity in serving.

Worked example
A concrete illustration of surrogate-versus-metric divergence: a model trained purely to minimize average cross-entropy loss on a severely imbalanced dataset can reach a very low average loss dominated by the easy majority class, while its F1 score on the minority class (the metric that actually matters for the business) remains poor; post-hoc threshold tuning on a held-out set, sweeping the decision threshold specifically to maximize F1 rather than accepting the default 0.5 cutoff, can recover much of this gap cheaply without retraining anything.

Trade-offs & pitfalls
The recommended default order: start with a surrogate loss plus post-hoc threshold tuning, since it's the lowest-risk, lowest-complexity combination and often closes most of the gap to the real metric; escalate to a structured-prediction loss only if this gap persists and the added training/serving complexity is genuinely justified by the task. A common mistake is tuning the decision threshold on the SAME data used to select the final model checkpoint (rather than a separate held-out split), which can overfit the threshold to noise in that particular validation set, especially when the target metric (like F1 under severe imbalance) has relatively few positive examples to estimate it from in the first place.