Deep Learning: Neural Networks and Architectures Questions

Direct answer
When the metric you actually care about (F1, top-k accuracy) isn't differentiable, you can't optimize it directly with gradient descent, so the practical choices are training on a smooth SURROGATE loss and hoping it correlates well enough, or tuning a decision threshold AFTER training to better match the real metric, with structured-prediction losses as a heavier-weight option in between.

Structured elaboration
Surrogate losses (cross-entropy, hinge): simple, stable, and well-optimized by standard training infrastructure; the risk is that the surrogate is only an INDIRECT proxy for the real target metric, and optimizing it perfectly does not guarantee the real metric is also optimal, particularly under class imbalance where a surrogate averaged over all examples may not track a metric (like F1) that specifically weights the minority class.
Post-hoc threshold optimization: after training with a surrogate loss, sweep the decision threshold on a held-out validation set to directly maximize the actual target metric (F1, precision at a fixed recall, etc.); this is cheap, requires no change to training at all, but needs a genuinely representative validation set to avoid overfitting the threshold itself to validation-set noise.
Structured-prediction losses (CRFs, structured SVMs, direct differentiable approximations of F-measure): explicitly model the metric's own structure or dependencies in the loss itself, which can more directly target the real objective, at the cost of more complex, often slower training (sometimes requiring an inference step inside the training loop itself) and added engineering complexity in serving.

Worked example
A concrete illustration of surrogate-versus-metric divergence: a model trained purely to minimize average cross-entropy loss on a severely imbalanced dataset can reach a very low average loss dominated by the easy majority class, while its F1 score on the minority class (the metric that actually matters for the business) remains poor; post-hoc threshold tuning on a held-out set, sweeping the decision threshold specifically to maximize F1 rather than accepting the default 0.5 cutoff, can recover much of this gap cheaply without retraining anything.

Trade-offs & pitfalls
The recommended default order: start with a surrogate loss plus post-hoc threshold tuning, since it's the lowest-risk, lowest-complexity combination and often closes most of the gap to the real metric; escalate to a structured-prediction loss only if this gap persists and the added training/serving complexity is genuinely justified by the task. A common mistake is tuning the decision threshold on the SAME data used to select the final model checkpoint (rather than a separate held-out split), which can overfit the threshold to noise in that particular validation set, especially when the target metric (like F1 under severe imbalance) has relatively few positive examples to estimate it from in the first place.

Direct answer
A production-grade training loop needs five things working together correctly: the train/eval mode switch, a scheduler that reacts to validation loss, an early-stopping counter, checkpointing of the BEST (not most recent) model, and, for real production hardening, mixed precision, gradient clipping, and a warmup period layered on top.

Structured elaboration

import torch, torch.nn as nn, torch.optim as optim, os

def train_loop(model, train_loader, val_loader, device, epochs=50, lr=1e-3, weight_decay=1e-4,
               checkpoint_path="best_model.pt", patience=3, clip_grad_norm=1.0):
    model.to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=1)
    best_val_loss = float('inf'); best_epoch = -1; epochs_no_improve = 0

    for epoch in range(1, epochs + 1):
        model.train()
        for xb, yb in train_loader:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = criterion(model(xb), yb)
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), clip_grad_norm)
            optimizer.step()

        model.eval(); val_loss = 0.0; total = 0
        with torch.no_grad():
            for xb, yb in val_loader:
                xb, yb = xb.to(device), yb.to(device)
                loss = criterion(model(xb), yb)
                val_loss += loss.item() * xb.size(0); total += xb.size(0)
        val_loss /= total
        scheduler.step(val_loss)

        if val_loss < best_val_loss - 1e-6:
            best_val_loss, best_epoch, epochs_no_improve = val_loss, epoch, 0
            torch.save({'epoch': epoch, 'model_state_dict': model.state_dict(),
                        'optimizer_state_dict': optimizer.state_dict(),
                        'scheduler_state_dict': scheduler.state_dict(), 'val_loss': val_loss}, checkpoint_path)
        else:
            epochs_no_improve += 1
        if epochs_no_improve >= patience:
            break

    ckpt = torch.load(checkpoint_path, map_location=device, weights_only=False)
    model.load_state_dict(ckpt['model_state_dict'])
    return model, best_epoch, best_val_loss

Production hardening layered on top: mixed precision wraps the forward pass and loss in torch.autocast, scales the loss via a GradScaler before backward(), and unscales gradients before clipping; a linear warmup for the first 5% of total steps ramps the learning rate up from near zero before the main schedule takes over, avoiding an unstable first few updates at full learning rate.

Worked example
Run on synthetic data (200 training, 50 validation examples) for up to 30 epochs with patience=3: early stopping genuinely triggered after 7 epochs, correctly identifying epoch 4 as the best (validation loss 1.1423). Critically, the RESTORED model's weights were verified to match the SAVED checkpoint's weights exactly (every parameter tensor compared with torch.allclose), confirming the save/restore round-trip is correct and the function genuinely returns the BEST epoch's weights, not whatever the final epoch happened to leave in the model.

Trade-offs & pitfalls
The single most valuable thing this test caught is exactly the kind of bug that's easy to introduce silently: forgetting to actually RELOAD the best checkpoint before returning (leaving the model in its FINAL, possibly-worse, epoch state) would not raise any error and would look correct in a quick smoke test, since training still "completes" and a checkpoint file still exists on disk; only an explicit weight-by-weight comparison, as done here, catches whether the returned model is really the best one. A second common gap in AMP-hardened versions of this loop is clipping gradients BEFORE unscaling them when mixed precision is enabled; the scaled gradients are artificially large by the scale factor, so clipping against an ordinary threshold before unscaling either clips far too aggressively or not at all, depending on the current scale value.

Direct answer
When labeled data is scarce, self-supervised pretraining, semi-supervised learning, augmentation, and transfer learning all extract extra signal from the SAME small labeled set (or from unlabeled data you already have), but validating that any of them genuinely helped requires a validation protocol specifically designed to resist confirmation bias, since pseudo-labeling in particular can easily produce numbers that look like progress but aren't.

Structured elaboration
Self-supervised pretraining: learn general representations from UNLABELED data first (contrastive methods like SimCLR/MoCo, or masked-reconstruction methods like MAE), giving the network a strong starting point before it ever sees the scarce labels; a quick way to sanity-check whether this pretraining actually captured something useful is LINEAR PROBING (freezing the pretrained representation and training only a simple linear classifier on top), which is cheap and gives an early signal before committing to full fine-tuning.
Transfer learning: fine-tune (or use as a frozen feature extractor) a checkpoint already pretrained on a large, related dataset; compare a frozen-feature linear probe against full fine-tuning specifically to gauge how much genuine domain shift exists between the pretraining data and your task.
Semi-supervised learning: consistency regularization (enforcing that predictions stay stable under input augmentation, as in Mean Teacher or FixMatch) and pseudo-labeling (assigning high-confidence labels to unlabeled examples and training on them) both use UNLABELED data beyond the scarce labeled set, typically via a teacher-student or exponential-moving-average setup that smooths the targets rather than trusting the model's own raw, possibly-overconfident predictions directly.
Data augmentation: label-preserving transformations (and stronger policies like RandAugment or MixUp/CutMix) synthetically expand the effective diversity of the scarce labeled set.

Using learning curves and active learning to decide if you have enough data: plot validation performance against labeled-dataset SIZE (training repeatedly on increasing subsets); a curve that is still climbing steeply at your current dataset size is direct evidence more labels would help meaningfully, while a curve that has visibly flattened suggests you're near the point of diminishing returns FOR THIS architecture and task. Active learning (having the model flag its most UNCERTAIN unlabeled examples for human labeling next) directs limited labeling budget toward the examples likely to teach the model the most, rather than labeling more data uniformly at random.

Worked example
A concrete anti-confirmation-bias validation protocol for pseudo-labeling specifically: reserve a CLEAN labeled validation set that NEVER participates in generating pseudo-labels and is never touched by the semi-supervised training loop at all; run an explicit ABLATION comparing (a) labels-only baseline, (b) labels plus self-supervised pretraining, and (c) labels plus pretraining plus pseudo-labeling, all evaluated on that SAME untouched clean validation set; and require the reported improvement to be consistent across multiple random seeds (not a single lucky run) before accepting it as real, ideally with a statistical test (a paired comparison or bootstrap confidence interval) rather than a single point-estimate comparison.

Trade-offs & pitfalls
The single most common confirmation-bias trap is evaluating a pseudo-labeling pipeline on a validation set that WAS INVOLVED, even indirectly, in generating the pseudo-labels being trained on; this systematically inflates the reported gain, since the model has effectively been validated against data whose "ground truth" the model itself helped produce. A second common mistake is accepting a pseudo-labeling gain from a SINGLE training run without checking it holds up across seeds or via a proper statistical comparison; semi-supervised methods can show meaningfully higher run-to-run variance than fully-supervised training, and a single favorable run is weak evidence on its own.

Direct answer
Choosing a loss function starts from the prediction target's structure, not habit: binary classification pairs a single logit with binary cross-entropy, multi-class pairs softmax logits with categorical cross-entropy, and regression pairs an unconstrained linear output with MSE, MAE, or Huber depending on how you want to weight outliers. Imbalanced or overconfident cases then call for adjustments layered on top of that base choice.

Structured elaboration
Binary classification: binary cross-entropy (BCE) computed directly from logits (e.g. a fused "BCE with logits" op), never from a sigmoid output followed by a separate log, since that two-step form is numerically unstable near 0 and 1; BCE is exactly the negative log likelihood (NLL) of the true label under a Bernoulli model parameterized by the sigmoid output, the same NLL view categorical cross-entropy takes below. Hinge loss is the margin-based alternative for binary classification: $L=\max(0,\,1-y\hat{z})$ for labels $y\in\{-1,+1\}$ and raw score $\hat{z}$, with no sigmoid involved. Unlike BCE, hinge loss does not ask for a calibrated probability at all; it only penalizes predictions that are wrong or inside the margin, which is why it is the standard loss for SVM-style maximum-margin classifiers rather than for a model whose output needs to be read as a probability.
Multi-class classification: categorical cross-entropy, computed from softmax logits together (the fused softmax+cross-entropy op), again for numerical stability and because their combined gradient has the clean closed form $\hat{y}-y$; this is exactly the negative log likelihood (NLL) of the true class under the categorical distribution the softmax defines, so categorical cross-entropy and NLL loss refer to the same quantity here (frameworks sometimes split them into two ops purely for numerical-stability reasons).
Regression: MSE ($L=(\hat y-y)^2$) assumes roughly Gaussian noise and penalizes large errors quadratically, so it is sensitive to outliers; MAE ($L=|\hat y - y|$) is robust to outliers but has a non-smooth gradient at zero error; Huber loss blends the two, quadratic near zero and linear beyond a threshold $\delta$, giving a differentiable loss that is still robust to a handful of extreme outliers.
Class imbalance: start with class weighting (upweight the minority class in the cross-entropy) or resampling; reach for focal loss when imbalance is severe and many easy-negative examples otherwise dominate the gradient, since focal loss's $(1-p_t)^\gamma$ term down-weights already-confident correct predictions.
Overconfidence: label smoothing softens one-hot targets toward a small uniform mass on the wrong classes, which discourages the network from driving logits to extreme values and tends to improve calibration, at some cost to peak accuracy on tasks that genuinely need near-certain predictions.

Worked example
A concrete regression case shows why the loss choice matters numerically. Given residuals $r = [1, 1, 1, 1, 20]$ (four small errors, one outlier):
MSE $= \tfrac{1}{5}(1+1+1+1+400) = \tfrac{404}{5} = 80.8$, dominated entirely by the one outlier.
MAE $= \tfrac{1}{5}(1+1+1+1+20) = \tfrac{24}{5} = 4.8$, far less swayed by the outlier.
Huber with $\delta=1$: for $|r|\le \delta$, loss is $\tfrac12 r^2$; beyond, $\delta(|r|-\tfrac12\delta)$. For the four unit residuals: $\tfrac12(1)^2=0.5$ each. For the outlier ($|r|=20 > \delta$): $1\times(20-0.5)=19.5$. Total $=4\times0.5+19.5=21.5$, average $4.3$, close to MAE's robustness but with a smooth gradient everywhere.

Trade-offs & pitfalls
The single most common numerical-stability mistake is computing softmax (or sigmoid) as a separate step from the log used in cross-entropy; frameworks provide fused ops precisely because the intermediate normalized probabilities can underflow to exactly zero for very confident wrong predictions, making $\log(0)$ produce $-\infty$ or NaN. A second pitfall is defaulting to class weighting or focal loss without first checking whether the evaluation metric itself (plain accuracy) is even meaningful under the imbalance; fixing the loss does not fix a misleading metric.

Direct answer
An embedding layer is just a lookup table, a learned matrix of shape (number of categories, embedding dimension), that converts a discrete category ID into a dense vector the rest of the network can process like any other numeric feature.

Structured elaboration
Representing inputs: map each category to an integer ID in $[0, N)$, reserving one dedicated ID for unknown or missing categories; the embedding layer itself is an $N\times d$ matrix, and looking up an ID simply selects that row.
Choosing dimensionality $d$: a common rule of thumb is $d \approx \min(50, \lceil 6N^{1/4}\rceil)$, growing slowly with the number of distinct categories $N$; in practice, start small (8 to 64) and tune against validation performance, since an oversized embedding for a low-cardinality feature mostly adds overfitting risk and wasted parameters without adding real capacity.
Handling unseen categories at inference: reserve a dedicated "unknown" ID during training (routing sufficiently rare categories to it during training itself, not only at inference, so its embedding is actually learned from real examples rather than starting cold at serving time), or use feature hashing (mapping category values into a fixed number of hash buckets), which guarantees every possible future category maps to SOME existing embedding, at the cost of occasional hash collisions between genuinely different categories.
Integrating with numerical features: normalize numerical features first (their raw scale is generally very different from an embedding's learned scale), concatenate all embedding vectors together with the normalized numeric features into one combined vector, and feed that into the feedforward network's remaining dense layers.

Worked example

import torch

embs = [emb_layer(ids[:, i]) for i, emb_layer in enumerate(emb_layers)]
x_cat = torch.cat(embs, dim=1)      # (batch, sum of embedding dims)
x_num = num_bn(num_tensor)          # normalized numeric features
x = torch.cat([x_cat, x_num], dim=1)
out = mlp(x)

For a categorical feature with $N=1000$ distinct values, the rule of thumb gives $d\approx\min(50, 6\times1000^{0.25})=\min(50, 6\times5.62)=\min(50,33.7)\approx34$, a moderate embedding size that is neither a single scalar (too little capacity to capture 1000 distinct categories' relationships) nor a needlessly huge vector for a feature this size.

Trade-offs & pitfalls
A common mistake is treating the "unknown" bucket as a purely inference-time fallback, never actually exposing the model to it during training; if the model has never seen the unknown-ID embedding receive a real gradient update, it starts from an untrained, effectively random initialization at exactly the moment (an unseen category at inference) when a well-calibrated fallback matters most. A second common gap is skipping normalization of the accompanying numeric features before concatenation; embeddings are typically initialized and trained to occupy a specific, learned numeric range, and unnormalized numeric features with a very different natural scale can dominate or be dominated by the embedding features purely due to scale, not genuine predictive importance.