Deep Learning: Neural Networks and Architectures Questions

Direct answer
An embedding layer is just a lookup table, a learned matrix of shape (number of categories, embedding dimension), that converts a discrete category ID into a dense vector the rest of the network can process like any other numeric feature.

Structured elaboration
Representing inputs: map each category to an integer ID in $[0, N)$, reserving one dedicated ID for unknown or missing categories; the embedding layer itself is an $N\times d$ matrix, and looking up an ID simply selects that row.
Choosing dimensionality $d$: a common rule of thumb is $d \approx \min(50, \lceil 6N^{1/4}\rceil)$, growing slowly with the number of distinct categories $N$; in practice, start small (8 to 64) and tune against validation performance, since an oversized embedding for a low-cardinality feature mostly adds overfitting risk and wasted parameters without adding real capacity.
Handling unseen categories at inference: reserve a dedicated "unknown" ID during training (routing sufficiently rare categories to it during training itself, not only at inference, so its embedding is actually learned from real examples rather than starting cold at serving time), or use feature hashing (mapping category values into a fixed number of hash buckets), which guarantees every possible future category maps to SOME existing embedding, at the cost of occasional hash collisions between genuinely different categories.
Integrating with numerical features: normalize numerical features first (their raw scale is generally very different from an embedding's learned scale), concatenate all embedding vectors together with the normalized numeric features into one combined vector, and feed that into the feedforward network's remaining dense layers.

Worked example

import torch

embs = [emb_layer(ids[:, i]) for i, emb_layer in enumerate(emb_layers)]
x_cat = torch.cat(embs, dim=1)      # (batch, sum of embedding dims)
x_num = num_bn(num_tensor)          # normalized numeric features
x = torch.cat([x_cat, x_num], dim=1)
out = mlp(x)

For a categorical feature with $N=1000$ distinct values, the rule of thumb gives $d\approx\min(50, 6\times1000^{0.25})=\min(50, 6\times5.62)=\min(50,33.7)\approx34$, a moderate embedding size that is neither a single scalar (too little capacity to capture 1000 distinct categories' relationships) nor a needlessly huge vector for a feature this size.

Trade-offs & pitfalls
A common mistake is treating the "unknown" bucket as a purely inference-time fallback, never actually exposing the model to it during training; if the model has never seen the unknown-ID embedding receive a real gradient update, it starts from an untrained, effectively random initialization at exactly the moment (an unseen category at inference) when a well-calibrated fallback matters most. A second common gap is skipping normalization of the accompanying numeric features before concatenation; embeddings are typically initialized and trained to occupy a specific, learned numeric range, and unnormalized numeric features with a very different natural scale can dominate or be dominated by the embedding features purely due to scale, not genuine predictive importance.

Direct answer
An LSTM cell adds three gates and a separate cell state on top of a plain recurrent unit, letting the network learn WHEN to remember, overwrite, or expose information, which is what lets gradients (and information) survive across many more timesteps than a vanilla RNN can manage.

Structured elaboration
Given input $x_t$, previous hidden state $h_{t-1}$, and previous cell state $c_{t-1}$:
$f_t = \sigma(W_f[h_{t-1},x_t]+b_f)\quad\text{(forget gate: how much of }c_{t-1}\text{ to keep)}$
$i_t = \sigma(W_i[h_{t-1},x_t]+b_i)\quad\text{(input gate: how much new content to write)}$
$\tilde c_t = \tanh(W_c[h_{t-1},x_t]+b_c)\quad\text{(candidate cell content)}$
$c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t\quad\text{(cell state update)}$
$o_t = \sigma(W_o[h_{t-1},x_t]+b_o)\quad\text{(output gate: how much of }c_t\text{ to expose)}$
$h_t = o_t \odot \tanh(c_t)$
Sigmoid gates produce values in $(0,1)$, acting as soft, learnable on/off switches; tanh keeps candidate content and the exposed hidden state in $(-1,1)$, centered around zero, which helps gradient-based optimization.

Why the gating structure preserves long-range dependencies: the cell-state update is ADDITIVE ($f_t\odot c_{t-1} + i_t\odot\tilde c_t$), not a repeated multiplicative transform of the kind that causes vanishing gradients in a vanilla RNN's hidden-state recurrence. When the forget gate is close to 1, the gradient with respect to $c_{t-1}$ passes through nearly unattenuated (multiplied by approximately 1, not by a small, saturating activation derivative), so error signal can flow back across many timesteps largely intact, as long as the network has learned to keep the relevant forget gates open along the way.

Worked example
Suppose at some timestep $f_t=0.95$ (mostly remember), $i_t=0.1$ (write a little new content), $c_{t-1}=2.0$, $\tilde c_t = 0.5$: $c_t = 0.95(2.0)+0.1(0.5)=1.9+0.05=1.95$, close to the previous cell value, as expected when the forget gate is near 1. Contrast with $f_t=0.1$ (mostly forget), same other values: $c_t = 0.1(2.0)+0.1(0.5)=0.2+0.05=0.25$, showing the cell state genuinely discarding most of its prior content when the network has learned that forgetting is appropriate at this step.

Trade-offs & pitfalls
A common oversimplification is describing the cell state as "immune" to vanishing gradients; it is much MORE resistant than a vanilla RNN's hidden state, but the forget gate itself is still a learned, saturating sigmoid, so if training pushes forget gates toward the extremes (stuck near 0) the additive benefit is undermined. A practical mitigation often used is initializing the forget-gate bias to a small positive value (e.g. +1), biasing the network toward remembering by default early in training, which tends to make long-range dependencies easier to learn from the start rather than relying on training to discover that remembering is useful.

Direct answer
SGD, SGD with momentum, RMSProp, Adam, and AdamW form a spectrum from simple-but-slow to adaptive-but-generalization-risky; AdamW is usually the safe modern default, but SGD with momentum still wins on final generalization for many vision tasks when you have the compute budget to tune it properly.

Structured elaboration

Why AdamW decouples weight decay: naive L2 regularization adds $\lambda w$ into the gradient before the adaptive rescaling by $\sqrt{\hat v}$, so a parameter with a large accumulated second moment gets its decay term shrunk by that same adaptive factor, which means differently-scaled parameters are decayed by inconsistent effective amounts, undermining the regularization's intent. AdamW instead applies the decay directly to the parameter, outside the adaptive-rescaling step, so every parameter is shrunk by the same proportional amount regardless of its gradient history.

When to recommend each: SGD with momentum for large-scale vision training and any setting where final generalization matters more than wall-clock convergence speed, paired with a learning-rate schedule and enough compute budget to tune it. Adam or AdamW for transformers, NLP, and rapid prototyping, where fast, robust initial convergence with less tuning effort matters more, and where AdamW is close to a strict improvement over Adam at essentially no extra cost. RMSProp and plain AdaGrad are mostly of historical interest now (RMSProp still shows up for RNNs and some reinforcement-learning setups); AdaGrad's monotonically-shrinking accumulated second moment is its own known shortcoming since it eventually stalls the effective learning rate to near zero, which is exactly the failure mode Adam's exponential moving average was designed to fix. For very large batch, large-scale distributed training, plain SGD with momentum plus a linear learning-rate scaling rule and warmup remains a common, well-understood choice, in part because its update rule is simpler to reason about and synchronize across many workers.

Worked example
One Adam update step with $\beta_1=0.9$, $\beta_2=0.999$, $\eta=10^{-3}$, $\epsilon=10^{-8}$, starting from $m_0=0$, $v_0=0$, at step $t=1$ with gradient $g=0.5$:
$m_1 = 0.9(0)+0.1(0.5)=0.05$, $v_1=0.999(0)+0.001(0.25)=0.00025$
Bias-corrected: $\hat m_1 = 0.05/(1-0.9^1)=0.5$, $\hat v_1=0.00025/(1-0.999^1)=0.25$
Update: $\theta \leftarrow \theta - 10^{-3}\times 0.5/(\sqrt{0.25}+10^{-8}) = \theta - 10^{-3}\times 1.0 = \theta - 0.001$
The bias correction matters here precisely because at $t=1$ the raw moments are heavily biased toward zero (their initialization); without it, the first update would be far smaller than intended.

Trade-offs & pitfalls
A common mistake is treating "switch to Adam" as a strict upgrade in every setting; on several large vision benchmarks, SGD with momentum plus a well-tuned schedule still reaches better final validation accuracy than Adam, at the cost of more tuning effort. A second pitfall is applying naive L2 weight decay under Adam and expecting it to behave like classical weight decay under SGD; because of the adaptive-rescaling interaction described above, it will not, which is the entire motivation for AdamW.

Direct answer
Dropout, weight decay, data augmentation, early stopping, batch normalization, and label smoothing all reduce overfitting, but each does so by constraining a different part of the system: activations, weight magnitudes, the input distribution, training duration, internal statistics, or target confidence.

Structured elaboration

Worked example
L2 regularization's effect on the loss, made concrete: for a loss $L(\theta) = L_0(\theta) + \lambda\|\theta\|^2$, the gradient becomes $\nabla L = \nabla L_0 + 2\lambda\theta$, so an SGD update becomes $\theta \leftarrow \theta - \eta\nabla L_0 - 2\eta\lambda\theta = (1-2\eta\lambda)\theta - \eta\nabla L_0$. The $(1-2\eta\lambda)$ factor shrinks every weight multiplicatively toward zero on every step, independent of the data-driven gradient, which is exactly the "decay" in weight decay. For $\eta=0.01$, $\lambda=0.01$: each step shrinks weights by a factor of $1-2(0.01)(0.01)=0.9998$, a small but compounding effect over thousands of steps.

Trade-offs & pitfalls
A common mistake is stacking many of these techniques at maximum strength simultaneously and then being unable to tell which one is responsible for a given change in validation performance; ablate one at a time when tuning. A second pitfall specific to L1/L2 under adaptive optimizers like Adam is that naive L2 does not behave like true weight decay once gradients are being adaptively rescaled per-parameter, which is exactly the motivation for AdamW's decoupled weight decay.

Direct answer
When labeled data is scarce, self-supervised pretraining, semi-supervised learning, augmentation, and transfer learning all extract extra signal from the SAME small labeled set (or from unlabeled data you already have), but validating that any of them genuinely helped requires a validation protocol specifically designed to resist confirmation bias, since pseudo-labeling in particular can easily produce numbers that look like progress but aren't.

Structured elaboration
Self-supervised pretraining: learn general representations from UNLABELED data first (contrastive methods like SimCLR/MoCo, or masked-reconstruction methods like MAE), giving the network a strong starting point before it ever sees the scarce labels; a quick way to sanity-check whether this pretraining actually captured something useful is LINEAR PROBING (freezing the pretrained representation and training only a simple linear classifier on top), which is cheap and gives an early signal before committing to full fine-tuning.
Transfer learning: fine-tune (or use as a frozen feature extractor) a checkpoint already pretrained on a large, related dataset; compare a frozen-feature linear probe against full fine-tuning specifically to gauge how much genuine domain shift exists between the pretraining data and your task.
Semi-supervised learning: consistency regularization (enforcing that predictions stay stable under input augmentation, as in Mean Teacher or FixMatch) and pseudo-labeling (assigning high-confidence labels to unlabeled examples and training on them) both use UNLABELED data beyond the scarce labeled set, typically via a teacher-student or exponential-moving-average setup that smooths the targets rather than trusting the model's own raw, possibly-overconfident predictions directly.
Data augmentation: label-preserving transformations (and stronger policies like RandAugment or MixUp/CutMix) synthetically expand the effective diversity of the scarce labeled set.

Using learning curves and active learning to decide if you have enough data: plot validation performance against labeled-dataset SIZE (training repeatedly on increasing subsets); a curve that is still climbing steeply at your current dataset size is direct evidence more labels would help meaningfully, while a curve that has visibly flattened suggests you're near the point of diminishing returns FOR THIS architecture and task. Active learning (having the model flag its most UNCERTAIN unlabeled examples for human labeling next) directs limited labeling budget toward the examples likely to teach the model the most, rather than labeling more data uniformly at random.

Worked example
A concrete anti-confirmation-bias validation protocol for pseudo-labeling specifically: reserve a CLEAN labeled validation set that NEVER participates in generating pseudo-labels and is never touched by the semi-supervised training loop at all; run an explicit ABLATION comparing (a) labels-only baseline, (b) labels plus self-supervised pretraining, and (c) labels plus pretraining plus pseudo-labeling, all evaluated on that SAME untouched clean validation set; and require the reported improvement to be consistent across multiple random seeds (not a single lucky run) before accepting it as real, ideally with a statistical test (a paired comparison or bootstrap confidence interval) rather than a single point-estimate comparison.

Trade-offs & pitfalls
The single most common confirmation-bias trap is evaluating a pseudo-labeling pipeline on a validation set that WAS INVOLVED, even indirectly, in generating the pseudo-labels being trained on; this systematically inflates the reported gain, since the model has effectively been validated against data whose "ground truth" the model itself helped produce. A second common mistake is accepting a pseudo-labeling gain from a SINGLE training run without checking it holds up across seeds or via a proper statistical comparison; semi-supervised methods can show meaningfully higher run-to-run variance than fully-supervised training, and a single favorable run is weak evidence on its own.