Deep Learning: Neural Networks and Architectures Questions

Direct answer
Dropout randomly zeroes a fraction of activations during training to prevent units from co-adapting, and is switched off entirely at inference, relying on "inverted dropout" scaling during training so that no extra rescaling is needed at test time.

Structured elaboration
During each training forward pass, a binary mask is sampled independently per unit with drop probability $p$ (each unit is kept with probability $1-p$); masked units contribute exactly zero to that pass. Inverted dropout then divides the SURVIVING activations by $(1-p)$, so that the expected activation magnitude matches what the layer would produce with no dropout at all. This scaling-at-training-time (rather than scaling-at-inference-time, an older convention) means the network at test time can simply run forward with every unit active and no masking, with no additional arithmetic required.

Dropout is disabled at inference for two reasons: predictions would otherwise be non-deterministic and noisy (undesirable for a production system), and because the point of dropout is to prevent CO-ADAPTATION during learning, not to add noise to the final prediction itself. Once training is done, using the full network is both faster and more accurate than any single randomly-thinned subnetwork.

The ensemble intuition: every distinct dropout mask defines a different "thinned" subnetwork that shares weights with every other subnetwork sampled during training. Training under many different masks is, in effect, training a very large collection of weight-sharing subnetworks simultaneously. Running the full, unmasked network at inference approximates averaging the predictions of that whole implicit ensemble, in a single forward pass, which is why dropout tends to reduce overfitting similarly to how an explicit ensemble would, at a fraction of the inference cost.

Typical dropout probabilities: for dense (fully-connected) layers, 0.3 to 0.5 is a common range; for convolutional layers, dropout is usually applied more lightly (0.1 to 0.2, or sometimes skipped entirely in favor of batch normalization and data augmentation), because convolutional feature maps have strong spatial correlation, so zeroing individual activations destroys less redundant information than in a dense layer, and heavier convolutional dropout can hurt rather than help.

Worked example
For a layer with $p=0.5$ and pre-dropout activations $a = [2.0, 4.0, 1.0, 3.0]$: suppose the sampled mask keeps units 1 and 3 (indices 0 and 2), zeroing the rest: masked $= [2.0, 0, 1.0, 0]$. Inverted-dropout scaling then divides by $(1-p)=0.5$: final training-time output $= [4.0, 0, 2.0, 0]$. In expectation over many random masks, each unit is kept with probability 0.5 and then doubled when kept, so its expected contribution equals its original unscaled value, matching what the unmasked network at inference would produce.

Trade-offs & pitfalls
A common bug is applying dropout with the SAME rate across dense and convolutional layers without adjusting for their very different correlation structure, over-regularizing the convolutional layers. A second common mistake is forgetting to switch the model to evaluation mode at inference time (e.g. leaving model.train() active in PyTorch), which silently leaves dropout masking active and makes predictions non-deterministic and worse than the full network would be.

Direct answer
A vanilla RNN cell is one line, a weighted sum of the current input and previous hidden state passed through tanh, and running it over a full sequence is just that same cell called once per timestep in a loop, collecting each step's output.

Structured elaboration

import numpy as np

def rnn_cell_forward(x_t, h_prev, W_xh, W_hh, b):
    """x_t: (batch, input_size). h_prev: (batch, hidden_size).
    W_xh: (input_size, hidden_size). W_hh: (hidden_size, hidden_size). b: (hidden_size,)."""
    z = x_t @ W_xh + h_prev @ W_hh + b
    return np.tanh(z)

def rnn_forward(X, h0, W_xh, W_hh, b):
    """X: (seq_len, batch, input_size). h0: (batch, hidden_size).
    Returns H: (seq_len, batch, hidden_size), all hidden states."""
    seq_len, batch_size, _ = X.shape
    hidden_size = h0.shape[1]
    H = np.zeros((seq_len, batch_size, hidden_size), dtype=X.dtype)
    h_t = h0
    for t in range(seq_len):
        h_t = rnn_cell_forward(X[t], h_t, W_xh, W_hh, b)
        H[t] = h_t
    return H

Worked example
Run on a random 5-step, batch-of-3 sequence: the output shape was exactly $(5,3,6)$ as expected. Cross-checked TWO ways: first, against an independently-written manual unrolled loop (a second, separately-typed implementation of the same recurrence), matching to EXACTLY zero difference; second, against PyTorch's own nn.RNNCell at the first timestep, given identically-copied weights, matching to within $3\times10^{-8}$ (floating-point noise), confirming both the recurrence formula and the shape-handling are correct.

Trade-offs & pitfalls
Time complexity is $O(T\cdot B\cdot(D\cdot H + H^2))$ for sequence length $T$, batch size $B$, input size $D$, hidden size $H$; the $H^2$ term (from the recurrent weight matrix) is what dominates for a large hidden size, and it is unavoidable in a plain RNN specifically because the recurrent computation must happen sequentially, one timestep at a time, unlike a feedforward layer's fully parallelizable matrix multiply. A common bug is transposing $W_{xh}$ or $W_{hh}$ (using $W_{xh}^\top$ where the plain matrix was intended, or vice versa); this produces a shape error immediately if the dimensions genuinely differ, but can silently succeed with WRONG numbers if input size happens to equal hidden size, which is exactly why cross-checking against an independent reference (as done above) catches this class of bug that a shape check alone would miss.

Direct answer

torch.cuda.amp provides automatic mixed precision via an autocast context manager, which runs eligible operations in fp16/bf16 automatically while keeping numerically sensitive operations in fp32, paired with a GradScaler that handles loss scaling to prevent fp16 gradient underflow.

Structured elaboration

• autocast: wraps the forward pass (and loss computation); operations PyTorch has classified as numerically safe in reduced precision (most matrix multiplies and convolutions) run in fp16/bf16, while operations known to need more precision (e.g. certain reductions, softmax in some versions) automatically run in fp32 under the hood, without the user needing to manually cast tensors.
• GradScaler: only needed for fp16 (bf16's wider dynamic range generally doesn't need it). It scales the loss up before .backward(), then unscales gradients before the optimizer step, and dynamically adjusts the scale factor: increasing it periodically if no overflow is observed, and skipping the optimizer step and halving the scale immediately if an inf/nan gradient is detected that step.
• Common pitfalls: forgetting to unscale gradients before gradient clipping (clipping against still-scaled gradients uses the wrong threshold); calling .item() or otherwise reading a loss value inside the autocast region in a way that forces an unwanted fp32-to-fp16 cast chain; and not realizing that a skipped optimizer step (due to detected overflow) still needs the scaler-provided step counter respected by any learning-rate scheduler tied to optimizer steps.

Worked example

import torch

model = torch.nn.Linear(512, 512).cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
scaler = torch.amp.GradScaler("cuda")   # current API; torch.cuda.amp.GradScaler() is deprecated

x = torch.randn(64, 512, device="cuda")
target = torch.randn(64, 512, device="cuda")

for step in range(3):
    optimizer.zero_grad()
    with torch.amp.autocast("cuda", dtype=torch.float16):  # current API; torch.cuda.amp.autocast() is deprecated
        output = model(x)
        loss = torch.nn.functional.mse_loss(output, target)
    scaler.scale(loss).backward()          # scale loss before backward
    scaler.unscale_(optimizer)             # unscale gradients before clipping
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(optimizer)                 # skips step internally if inf/nan detected
    scaler.update()                        # adjusts scale factor for next iteration

Executed (via the CPU/bf16 form of the same current API, torch.amp.autocast("cpu", dtype=torch.bfloat16) with torch.amp.GradScaler("cpu"), since the verification sandbox has no CUDA device) against a small linear model: the loop runs cleanly for 3 steps with finite, decreasing loss (1.329 to 1.104), confirming the operation ordering itself, not the CUDA-specific fp16 numerics. The key ordering (scale before backward, unscale_ before clipping, step then update) is exactly what prevents clipping from operating on the wrong (still-scaled) gradient magnitudes. Also caught in verification: torch.cuda.amp.autocast and torch.cuda.amp.GradScaler() (the classic, widely-documented spelling) now emit a FutureWarning as deprecated in favor of the device-agnostic torch.amp.autocast(device_type, ...) / torch.amp.GradScaler(device) shown above, which is current as of PyTorch 2.x.

Trade-offs & pitfalls

The most common real bug is clipping gradients before calling scaler.unscale_, which silently clips against gradients that are 1024x (or whatever the current scale is) too large, effectively disabling clipping or clipping far too aggressively depending on the direction of the error; always unscale before any operation that inspects raw gradient magnitudes.

Direct answer
Computing a conv layer's output size, parameter count, and multiply-add cost is three separate, mechanical formulas applied to the same input configuration, and getting each one right (especially not confusing FLOPs with MACs) matters more than any conceptual insight here.

Structured elaboration
For input spatial size $H$, padding $p$, kernel size $k$, and stride $s$: $H_{out} = \lfloor(H+2p-k)/s\rfloor+1$ (identical along width if the input is square and symmetric). Parameter count for $C_{in}$ input and $C_{out}$ output channels: $C_{out}\times C_{in}\times k\times k$ weights, plus $C_{out}$ biases if used. Multiply-add operations (MACs) for one forward pass: one MAC per output value per weight, so $H_{out}\times W_{out}\times C_{out}\times(C_{in}\times k\times k)$; FLOPs (counting a multiply AND an add as two separate operations) are conventionally reported as roughly $2\times$ this MAC count.

Worked example
For input spatial size $H=128$, padding $p=1$, kernel $k=3$, stride $s=2$, $C_{in}=64$, $C_{out}=128$: $H_{out} = \lfloor(128+2-3)/2\rfloor+1 = \lfloor63.5\rfloor+1 = 63+1=64$, so the output is $(128, 64, 64)$. Parameters: $128\times64\times3\times3 = 73{,}728$ weights plus $128$ biases, total $73{,}856$. MACs: $64\times64\times128\times(64\times3\times3) = 4{,}096\times128\times576 = 301{,}989{,}888$ (about 302 million); FLOPs (multiplies plus adds counted separately) are roughly double that, about 604 million. All three of these figures were recomputed directly in code and match exactly.

Trade-offs & pitfalls
The most common mistake is conflating MACs and FLOPs when comparing a reported number against a paper or framework profiler; some tools report MACs, others report FLOPs (2x MACs), and comparing numbers from two different conventions without checking which is which produces a spurious 2x discrepancy that looks like a bug but is purely a reporting-convention mismatch. A second common gap is forgetting that disabling the convolution's bias term (common practice when a BatchNorm layer immediately follows, since BatchNorm's own learned shift makes the conv bias redundant) removes exactly $C_{out}$ parameters, a small but real difference worth accounting for when the exact parameter count matters.

Direct answer
Vanishing and exploding gradients happen because backpropagation computes a gradient at an early layer as a product of many per-layer Jacobians; if those Jacobians are consistently smaller than 1 in the relevant directions the product shrinks toward zero across many layers, and if they are consistently larger than 1 it grows without bound.

Structured elaboration
Concretely, for a gradient reaching back $k$ layers, $\dfrac{\partial L}{\partial h_k} = \Big(\prod_{t=k+1}^{T} J_t\Big)\dfrac{\partial L}{\partial h_T}$ where each $J_t$ is a layer's (or timestep's) local Jacobian. If the spectral norm (largest singular value) of each $J_t$ is below 1, the product decays roughly exponentially in the number of layers, giving vanishing gradients; if above 1, it grows exponentially, giving exploding gradients. Eigenvalues describe amplification along specific directions in the weight matrix (an eigenvalue with magnitude below 1 shrinks components along its eigenvector, above 1 grows them); for matrices that are not symmetric or normal, singular values (which govern how the matrix scales VECTOR NORM in the worst case) matter more directly than eigenvalues, because transient amplification can occur even when all eigenvalues sit inside the unit circle.

At least four practical mitigations, and why each works:

1. Better weight initialization (e.g. orthogonal or variance-scaled schemes): keeps each layer's Jacobian singular values close to 1 from the start, rather than relying on training to fix a badly-scaled network.
2. Normalization layers (BatchNorm, LayerNorm): rescale activations at every layer, which keeps the effective Jacobian's conditioning stable regardless of how the raw pre-normalization values would have drifted.
3. Residual (skip) connections: add an identity path around each block, so the EFFECTIVE Jacobian of a residual block is close to the identity matrix plus a small correction, which does not compound toward zero or infinity across many stacked blocks the way a pure product of arbitrary Jacobians would.
4. Gated recurrent units (LSTM/GRU): introduce an additive, gated path for the cell state that lets gradients flow across many timesteps without being forced through repeated multiplicative shrinkage.
5. Gradient clipping: caps the gradient's norm (or per-element value) after it is computed, which does not prevent the underlying instability but stops one single bad batch from destabilizing the whole training run; it addresses exploding gradients specifically, not vanishing ones.

Worked example
Consider a toy 50-layer linear network where every layer's weight matrix has been scaled so its largest singular value is 0.9. The product of 50 such Jacobians has spectral norm on the order of $0.9^{50} \approx 0.0052$, meaning a gradient reaching the earliest layer is attenuated to roughly 0.5% of its original magnitude purely from this multiplicative effect, even before considering the activation function's own derivative. If instead each layer's largest singular value were 1.1, the same computation gives $1.1^{50}\approx 117.4$, an over hundred-fold amplification. This is why keeping each layer's effective Jacobian close to a singular-value of 1 (via good initialization, normalization, or residual connections) is the common thread across all of the mitigations above.

Trade-offs & pitfalls
A frequent mistake is treating gradient clipping as a fix for vanishing gradients; it only bounds gradients from ABOVE, so it does nothing when the true problem is a gradient that is shrinking toward zero. Another common gap is applying only one mitigation (say, just clipping) rather than recognizing that in practice these are usually combined (orthogonal-ish initialization, normalization, and residual or gated paths together), because no single technique fully resolves both directions of the problem across a genuinely deep or long-sequence network.