Deep Learning: Neural Networks and Architectures Questions

Direct answer
Neural networks need non-linear activations because any stack of purely linear layers is mathematically equivalent to one single linear layer, no matter how deep; without non-linearity, depth buys you nothing extra in representational power.

Structured elaboration
For two linear layers, $y = W_2(W_1x + b_1) + b_2 = (W_2W_1)x + (W_2b_1+b_2)$, which has exactly the same FORM as a single linear layer $y = W_{\text{eff}}x + b_{\text{eff}}$ with $W_{\text{eff}}=W_2W_1$ and $b_{\text{eff}}=W_2b_1+b_2$. This generalizes to any number of stacked linear layers: the composition of $L$ linear maps is always itself one linear map. So a linear-only network, regardless of depth, can only ever represent affine functions, meaning it can only draw a single straight decision boundary (a hyperplane) between classes.

Worked example
XOR is the standard concrete counterexample: for inputs $x_1,x_2 \in \{0,1\}$, XOR outputs 1 for $(0,1)$ and $(1,0)$, and 0 for $(0,0)$ and $(1,1)$. Plotting these four points, the two positive-label points sit on one diagonal and the two negative-label points sit on the other; no single straight line can separate them, so no purely linear model, of any depth, can represent XOR exactly. Adding a non-linear activation (even a single ReLU or sigmoid layer of hidden units) lets the network carve the input space into multiple linear regions that, combined, DO implement XOR. Concretely, a 2-unit hidden layer with ReLU activations can implement XOR as $\text{XOR}(x_1,x_2) = \text{ReLU}(x_1+x_2) - 2\cdot\text{ReLU}(x_1+x_2-1)$, which evaluates to 0 at $(0,0)$, 1 at both $(0,1)$ and $(1,0)$, and 0 at $(1,1)$, exactly matching XOR.

Trade-offs & pitfalls
The consequence for expressivity is total, not partial: this is not a case of linear networks being "a bit worse" at non-linear tasks, they are mathematically INCAPABLE of representing them exactly, regardless of how many linear layers or how many units per layer you add. Non-linearity is precisely what allows a deep network to build a genuine hierarchy of increasingly abstract features rather than just re-parameterizing the same single affine transform.

Direct answer
Learning-rate schedules all answer the same question, how should the step size change over training, differently: step and exponential decay shrink it on a fixed schedule, cosine annealing shrinks it smoothly (optionally with periodic restarts), cyclical schedules deliberately move it up and down, and warmup ramps it up from near zero before any of the above kicks in.

Structured elaboration
Step decay multiplies the learning rate by a fixed factor at specific milestones (e.g. divide by 10 at epochs 30, 60, 90); simple, and gives clean, large "phase changes" useful for coarse-to-fine training. Exponential decay applies $\eta_t = \eta_0 e^{-kt}$, a smooth continuous shrink with no manual milestone tuning required. Cosine annealing follows $\eta_t = \eta_{\min} + \tfrac12(\eta_{\max}-\eta_{\min})(1+\cos(\pi t/T))$ over $T$ steps, giving a smooth cool-down that empirically often out-performs step decay's abrupt jumps; with restarts (SGDR), the schedule periodically jumps back up to $\eta_{\max}$ partway through training, deliberately re-injecting exploration to help escape a poor local region before resuming the cool-down. Cyclical learning rates instead deliberately cycle the rate between a lower and upper bound throughout training (not just at restarts), which can reduce how precisely you need to have tuned the base learning rate in the first place, since the cycle itself sweeps through a range of values. Linear warmup ramps the learning rate up from near zero to its target value over the first few hundred or thousand steps, BEFORE any decay schedule takes over.

Why warmup matters for large models: at the very start of training, weights (and for adaptive optimizers, the moment estimates) are uncalibrated, so a full-strength learning rate applied immediately can produce a large, destabilizing update before the network or optimizer state has had any chance to settle; warmup avoids this by keeping early updates small while things calibrate, which becomes especially important as model size, batch size, or the base learning rate grow.

Which schedules pair well with large-batch training: warmup is close to mandatory at large batch sizes (large batches typically use a correspondingly larger learning rate via the linear scaling rule, which makes the destabilizing-early-update risk warmup addresses even more acute); cosine annealing (with or without restarts) is the most common pairing with warmup for large-scale training, since its smooth decay complements a smooth ramp-up, whereas step decay's abrupt drops can feel like a second, smaller version of the same instability warmup was meant to avoid.

Worked example
A standard large-model recipe: linear warmup over the first 5% of total training steps (rate rising linearly from 0 to the target learning rate), followed by cosine annealing down to a small floor value for the remaining 95% of steps, with no restarts; this exact combination is the de facto default for most modern transformer training recipes precisely because both halves are smooth and compose cleanly.

Trade-offs & pitfalls
A common mistake is applying a schedule's decay starting from step 0, INCLUDING the warmup period, which causes the rate to be simultaneously rising (from warmup) and falling (from the decay schedule) and produces a confusing, unintended shape; warmup and the main decay schedule should be sequenced as two distinct phases, not blended from the start. A second common gap is choosing a schedule independent of batch size; a schedule tuned for one batch size (in particular its warmup LENGTH and the total number of decay steps) often needs to be re-tuned, not just have its peak learning rate rescaled, when batch size changes substantially.

Direct answer
Backpropagating through BatchNorm means differentiating through the mean and variance it computes from the CURRENT batch, not just through a simple elementwise transform, and the resulting gradient has a clean compact form that couples every example in the batch together through shared terms.

Structured elaboration
For a mini-batch of size $N$ with inputs $x_i$, $\mu=\frac1N\sum_i x_i$, $\sigma^2=\frac1N\sum_i(x_i-\mu)^2$, $\hat x_i = (x_i-\mu)/\sqrt{\sigma^2+\epsilon}$, output $y_i=\gamma\hat x_i+\beta$. Given the upstream gradient $dy_i=\partial L/\partial y_i$, the full chain-rule derivation (through $\hat x$, then $\sigma^2$, then $\mu$, then finally $x_i$ itself, since $\mu$ and $\sigma^2$ both depend on EVERY $x_i$ in the batch) collapses algebraically into the compact form:
$\frac{\partial L}{\partial x_i} = \frac{\gamma}{s}\left(dy_i - \frac1N\sum_j dy_j - \hat x_i\cdot\frac1N\sum_j(dy_j\hat x_j)\right), \qquad s=\sqrt{\sigma^2+\epsilon}$
This is not a simple elementwise scaling like an ordinary activation function's gradient; the two correction terms (subtracting the mean of $dy$, and subtracting $\hat x_i$ times the mean of $dy\cdot\hat x$) exist specifically because $x_i$ influences EVERY OTHER example's normalized output too, through the shared batch statistics $\mu$ and $\sigma^2$.

Worked example
This compact formula was independently checked against a finite-difference numerical gradient on a random 5-example batch (constructing a scalar loss whose gradient with respect to the output is exactly the given upstream gradient, then perturbing each input individually): the analytic and numerical gradients matched to within $5\times10^{-10}$, confirming the compact form is a correct, exact simplification of the longer chain-rule derivation, not merely an approximation.

Intuition for gradient magnitudes and stability: the $\gamma/s$ factor means the backpropagated gradient is rescaled by roughly $1/s$ (the inverse of the batch's own standard deviation) rather than by whatever arbitrary scale the raw activations happened to have; this decouples gradient magnitude from activation SCALE, which is a large part of why BatchNorm lets you use a larger learning rate safely and converge faster, its effect on OPTIMIZATION CONDITIONING (making the loss surface locally better-behaved for gradient descent), not merely a data-preprocessing convenience.

Trade-offs & pitfalls
Limitations of the "internal covariate shift" explanation specifically: the original justification (that BatchNorm helps by reducing the change in each layer's input distribution over training) is informal and has been challenged by later analyses, which attribute the empirical benefit more to improved LOSS-LANDSCAPE CONDITIONING and a mild regularization effect from batch-to-batch statistical noise than to a precisely reducible "shift" in any rigorously defined sense; regardless of which explanation is more mechanistically correct, the empirical benefits (faster convergence, higher usable learning rates) are well established. A separate, genuinely open limitation: BatchNorm's coupling of every example in a batch through shared $\mu,\sigma^2$ means its behavior (and this very gradient formula) becomes unreliable at very small batch sizes, where $\mu$ and $\sigma^2$ are themselves noisy estimates of the true population statistics, which is exactly why GroupNorm or LayerNorm (whose statistics do not depend on batch composition at all) are the standard substitutes in small-batch regimes.

Direct answer
The hinge loss's subgradient is exactly $-yx$ when the margin is violated, exactly zero when the margin is comfortably satisfied, and any value in between at the exact kink where the two cases meet, a genuine ambiguity that SGD resolves in practice simply by picking one of these valid subgradients according to how the boundary condition happens to be coded.

Structured elaboration
For $L(\theta)=\max(0, 1-y\theta^\top x)$, let $z=1-y\theta^\top x$. If $z>0$ (the margin is genuinely violated, this example is being misclassified or under-confidently classified), $L=z$ directly, so $\partial L/\partial\theta=-yx$. If $z<0$ (the example is classified correctly with margin to spare), $L=0$, so $\partial L/\partial\theta=0$. At exactly $z=0$ (the non-differentiable kink), the SUBDIFFERENTIAL is the entire convex hull between these two limiting values, any vector of the form $-\alpha yx$ for $\alpha\in[0,1]$ is a mathematically valid subgradient at that exact point.

How SGD handles this in practice: a real implementation needs to output ONE specific vector, not a whole set, so it picks based on how the boundary comparison is coded (commonly treating $z\ge0$ as "violated", giving $-yx$, or $z>0$ strictly, giving $0$ at the exact boundary); this ambiguity does not break convergence theory for subgradient methods (which are specifically designed to handle a general subgradient choice at non-smooth points), though an inappropriately large step size can cause slower or oscillatory behavior specifically AROUND this non-smooth boundary.

Worked example
This was verified numerically at two concrete points: with $x=[1,2,-1]$, $y=1$, and $\theta=[0.1,0.1,0.1]$ (giving $z=0.8>0$, a genuinely violated margin), the analytic subgradient $-yx=[-1,-2,1]$ matched a finite-difference numerical gradient EXACTLY. With $\theta=[5,5,5]$ (giving $z=-9<0$, a comfortably satisfied margin), the numerical gradient was exactly $[0,0,0]$, confirming the loss genuinely contributes zero gradient once the margin is safely met, both matching the two-case derivation above precisely.

Trade-offs & pitfalls
Implications for support vectors: only examples with $z\ge0$ (margin violated or exactly met) receive a non-zero subgradient at all, meaning ONLY these examples actually influence any given parameter update; examples comfortably beyond the margin contribute nothing, mathematically IDENTICAL to not being in the training set at all for that particular update. This is exactly the classical "support vector" property: at the optimum of a regularized hinge-loss objective (an SVM), the examples that determine the final decision boundary are precisely those sitting on or inside the margin, every other, comfortably-classified example could be REMOVED from the training set without changing the final solution at all, a genuinely useful sparsity property that a smooth loss like cross-entropy (where every example always contributes SOME nonzero gradient) does not share.

Direct answer
Computing a conv layer's output size, parameter count, and multiply-add cost is three separate, mechanical formulas applied to the same input configuration, and getting each one right (especially not confusing FLOPs with MACs) matters more than any conceptual insight here.

Structured elaboration
For input spatial size $H$, padding $p$, kernel size $k$, and stride $s$: $H_{out} = \lfloor(H+2p-k)/s\rfloor+1$ (identical along width if the input is square and symmetric). Parameter count for $C_{in}$ input and $C_{out}$ output channels: $C_{out}\times C_{in}\times k\times k$ weights, plus $C_{out}$ biases if used. Multiply-add operations (MACs) for one forward pass: one MAC per output value per weight, so $H_{out}\times W_{out}\times C_{out}\times(C_{in}\times k\times k)$; FLOPs (counting a multiply AND an add as two separate operations) are conventionally reported as roughly $2\times$ this MAC count.

Worked example
For input spatial size $H=128$, padding $p=1$, kernel $k=3$, stride $s=2$, $C_{in}=64$, $C_{out}=128$: $H_{out} = \lfloor(128+2-3)/2\rfloor+1 = \lfloor63.5\rfloor+1 = 63+1=64$, so the output is $(128, 64, 64)$. Parameters: $128\times64\times3\times3 = 73{,}728$ weights plus $128$ biases, total $73{,}856$. MACs: $64\times64\times128\times(64\times3\times3) = 4{,}096\times128\times576 = 301{,}989{,}888$ (about 302 million); FLOPs (multiplies plus adds counted separately) are roughly double that, about 604 million. All three of these figures were recomputed directly in code and match exactly.

Trade-offs & pitfalls
The most common mistake is conflating MACs and FLOPs when comparing a reported number against a paper or framework profiler; some tools report MACs, others report FLOPs (2x MACs), and comparing numbers from two different conventions without checking which is which produces a spurious 2x discrepancy that looks like a bug but is purely a reporting-convention mismatch. A second common gap is forgetting that disabling the convolution's bias term (common practice when a BatchNorm layer immediately follows, since BatchNorm's own learned shift makes the conv bias redundant) removes exactly $C_{out}$ parameters, a small but real difference worth accounting for when the exact parameter count matters.