LLM Fine-Tuning and Alignment Questions

Direct answer: A controlled, repeatable experiment comparing full fine-tuning, LoRA, and adapter-based fine-tuning on the same 2,000-example 5-class classification dataset needs matched dataset splits, an equal hyperparameter-search budget per method, both accuracy and resource metrics, and paired statistical tests across multiple seeds, not a single run per method.

Structured elaboration:

• Dataset splits: a stratified split preserving class balance, for example 70% train (1,400), 15% validation (300), 15% held-out test (300), plus either 5-fold stratified cross-validation on the combined train-and-validation portion, or multiple randomized seeds, to get a genuine variance estimate rather than a single point comparison between methods.
• Metrics: macro F1 as the primary metric (appropriate for a 5-class task, since it does not let a dominant class hide poor performance on the others) plus accuracy; a calibration metric (expected calibration error) since a well-calibrated model matters beyond raw accuracy; and resource metrics for every method, peak GPU memory, total GPU-hours, number of trainable parameters, final checkpoint size on disk, and inference latency/throughput at a fixed batch size, since the whole point of the comparison is cost versus performance, not performance alone.
• Hyperparameter search: give each method the SAME search budget (for example 30 trials) over its own relevant space, learning rate and batch size and weight decay for all three, plus rank and alpha specifically for LoRA, and bottleneck dimension and nonlinearity specifically for adapters, using a method like Bayesian optimization or ASHA to make efficient use of that shared budget rather than a naive grid search.
• Reproducibility and tracking: log every hyperparameter, GPU identifier, peak memory, GPU-hours, wall-clock time, checkpoint, and random seed with an experiment tracker, and pin the software environment (container or environment file) with seed-controlled data loaders so the entire comparison can be rerun and reproduced later.
• Statistical significance: run at least 5 independent repeats per method with different seeds using each method's best validation hyperparameters, report mean and standard error on the held-out test set, and use a PAIRED test (a paired t-test or Wilcoxon signed-rank test, since the same data splits and seeds are shared across methods) rather than an unpaired comparison, along with an effect size and confidence interval, not just a p-value.

Worked example: After running this design, a typical result table might show LoRA reaching macro F1 within half a point of full fine-tuning while using roughly a third of the GPU-hours and a small fraction of the trainable parameters, a genuine case where the practical conclusion (LoRA is the better choice) depends as much on the resource-cost columns as on the raw F1 column; presenting this as a Pareto front of F1 against GPU-hours (rather than only a table of best F1 per method) makes that cost-performance trade-off visible to a decision-maker at a glance, and reporting the paired significance test alongside it confirms whether a half-point F1 difference is a real effect or within the noise of seed variance.

Trade-offs and pitfalls: Comparing only the single best hyperparameter configuration found for each method, without repeating that configuration across multiple seeds, risks reporting a lucky (or unlucky) seed as the method's real performance; and reporting only accuracy or F1 without the resource-cost columns hides the entire point of a parameter-efficiency comparison, a small accuracy edge for full fine-tuning is not automatically worth 3x the GPU-hours, and the experiment's report should make that trade-off explicit rather than declaring a single winner on accuracy alone.

Direct answer: LoRA (Low-Rank Adaptation) freezes the pretrained weight matrix and instead learns a small, low-rank update on top of it, so fine-tuning trains far fewer parameters than updating the full matrix while still adapting the model's behavior.

Structured elaboration: For a weight matrix $W$ of shape $(d_{out}, d_{in})$, LoRA replaces the usual full-matrix update with two much smaller trainable matrices $A$ (shape $r \times d_{in}$) and $B$ (shape $d_{out} \times r$), where the effective weight used at inference is:

$W' = W + \frac{\alpha}{r} \, B A$

Here $r \ll \min(d_{out}, d_{in})$ is the rank, a hyperparameter controlling how expressive the update can be (typical values are 4 to 64 for large language models). $\alpha$ is a scaling factor that controls the effective magnitude of the update relative to the rank; dividing by $r$ keeps the update's scale roughly comparable across different rank choices, so you can change $r$ without having to re-tune $\alpha$ from scratch. $B$ is typically initialized to all zeros and $A$ to small random values, so the adapted model starts out numerically identical to the original pretrained model and then diverges as training updates $A$ and $B$. LoRA adapters are most commonly applied to the attention query and value projection matrices (and sometimes the feed-forward layers), since those dominate a transformer's parameter count and strongly shape its behavior. Per-task adapters are stored as just the pair $(A, B)$ for each layer they are applied to, which is typically megabytes rather than the gigabytes a full fine-tuned checkpoint would need, so serving many task-specific variants of one base model becomes practical.

Worked example: For a single attention projection of shape $(12288, 12288)$ (a dimension similar to a large transformer's hidden size), full fine-tuning of that one projection would train $12288 \times 12288 = 150{,}994{,}944$ parameters. With LoRA at rank $r=8$, the trainable parameter count is $r \times (d_{in} + d_{out}) = 8 \times (12288 + 12288) = 196{,}608$ parameters, about $0.13\%$ of the full count, a roughly 768x reduction for that single projection. This is why LoRA cuts training memory so much: gradients and optimizer state (which for Adam-family optimizers are often 2-3x the raw parameter count) only need to be stored for $A$ and $B$, not for the frozen $12288 \times 12288$ matrix.

Trade-offs and pitfalls: LoRA reduces trainable parameters, but at inference the unmerged form ($W$ plus a separate low-rank multiply) adds a small amount of extra compute; in production this is almost always avoided by merging $BA$ into $W$ once training is done, which restores exactly the same inference cost as the original model with zero runtime overhead, at the cost of losing the ability to instantly swap adapters without reloading weights. Choosing $r$ too small can under-fit a task that genuinely needs a higher-rank update (complex domain shifts, multi-task mixtures), while choosing it unnecessarily large gives up most of LoRA's memory savings without a clear accuracy benefit, so rank is usually tuned empirically starting from a small value like 8.

Direct answer: Evaluating how 4-bit quantization affects alignment requires comparing the FP16 baseline against multiple quantization variants (post-training quantization, quantization-aware fine-tuning, and a hybrid that keeps critical layers at higher precision) on the SAME prompts with the SAME decoding settings, measuring helpfulness and safety with properly powered, paired statistical tests, not just a qualitative before-and-after comparison.

Structured elaboration:

• Experimental groups: the original FP16 model as the control; a post-training-quantized 4-bit variant (the cheapest to produce); a quantization-aware fine-tuned 4-bit variant (recalibrated on instruction data, typically higher quality but more expensive to produce); and a hybrid variant keeping a small number of critical layers at higher precision while quantizing the rest, run with identical decoding settings and seeds across all variants so any observed difference is attributable to quantization itself, not incidental sampling variation.
• Metrics: helpfulness via human-rated scores on a representative, stratified prompt set plus automated metrics for tasks where they apply; safety via a red-team prompt suite scored both by human labelers and an automated toxicity or unsafe-content classifier; calibration (confidence versus accuracy); and operational metrics (latency, memory, throughput), since quantization's whole point is an operational trade-off, that side needs to be measured with the same rigor as the alignment side.
• Sampling and statistical rigor: use a stratified prompt corpus spanning common instructions, factual/knowledge questions, reasoning tasks, and adversarial/safety prompts, sized via a power analysis targeting a meaningful minimum effect (a moderate effect size at 80% power typically implies several hundred human-rated samples per metric); randomize prompt order, blind human raters to which model variant produced each response, use multiple raters and measure their agreement, and analyze with PAIRED tests on the same prompts across model variants (an appropriate paired test for ordinal ratings, an appropriate paired test for binary safety outcomes), reporting effect sizes and confidence intervals with a multiple-comparison correction since several quantization variants are being compared against the same baseline.
• Mitigation if degradation is found: cluster the failure cases to identify the specific pattern (a particular content type, or a specific behavior like hallucination rate rising specifically on longer responses), then apply a targeted fix, quantization-aware fine-tuning on a combined instruction-plus-safety dataset, keeping specific sensitive layers (commonly attention projections) at higher precision, per-channel quantization or learned scaling factors, or a lightweight LoRA fine-tune specifically to restore the lost alignment behavior while keeping most of the quantization's size and speed benefit; re-run the SAME evaluation protocol after any mitigation, since a fix that looks like it worked on a quick spot-check needs the same statistical rigor as the original comparison to actually confirm the regression is closed.

Worked example: A concrete degradation-and-fix cycle: the paired comparison shows the plain post-training-quantized 4-bit variant has a statistically significant increase in unsafe-response rate on the adversarial prompt slice specifically (not on the benign slice), clustering the failed cases reveals they concentrate on a specific attack pattern; a targeted fix, quantization-aware fine-tuning including that specific attack pattern's examples in the recalibration data, is applied and the full evaluation protocol (including the same paired statistical test) is re-run; only if the unsafe-response rate on that same adversarial slice is no longer significantly different from the FP16 baseline is the fix considered validated, a smaller informal spot-check would not have caught whether the fix generalized beyond the specific examples used to construct it.

Trade-offs and pitfalls: Comparing quantized and unquantized models with different decoding settings, seeds, or prompt sets (rather than the same ones) confounds any observed difference with those incidental factors rather than isolating the effect of quantization itself. A second pitfall is treating a spot-check on a handful of examples as sufficient evidence a mitigation worked, without the same statistical rigor (paired tests, effect sizes, correction for multiple comparisons) applied to the post-mitigation evaluation, a fix can look successful on the specific failures it was built to address while a proper re-evaluation would reveal it did not generalize to the broader adversarial slice.

Direct answer: A supervised fine-tuning training loop for a causal language model computes next-token cross-entropy loss shifted by one position, masks out any tokens that should not contribute to the loss (padding, and typically the prompt portion in an instruction-tuning setup), accumulates gradients across several micro-batches before each optimizer step to reach a larger effective batch size than fits in memory at once, and optionally runs under automatic mixed precision.

Structured elaboration: Because the model is autoregressive, the loss compares each position's PREDICTED next-token distribution (the logits at position t) against the ACTUAL token at position t+1, which is why the logits and labels are shifted relative to each other before computing cross-entropy. Using a sentinel label value (commonly -100) at positions that should not contribute, padding tokens, and in an instruction-tuning setup, typically the prompt tokens themselves, lets a single call to a masked cross-entropy function (ignore_index set to that sentinel) skip exactly those positions without any separate masking logic. Mixed precision (autocast) reduces memory and speeds up training on supporting hardware. Gradient accumulation lets several micro-batch forward/backward passes contribute to one optimizer step; the loss from each micro-batch must be scaled down before its backward pass so the accumulated gradient matches what a single large batch would have produced, and that scale factor needs to be the fraction of the TOTAL scored (non-masked) tokens across all micro-batches in the accumulation window, not simply one divided by the number of micro-batches, since micro-batches can have different amounts of valid content after masking.

Worked example (executed):

import torch
import torch.nn.functional as F

def sft_train_step(model, optimizer, micro_batches, scaler, use_amp):
    """micro_batches: list of (input_ids, attention_mask, labels) tuples that together form
    one optimizer update (gradient accumulation). labels use ignore_index=-100 for masked
    positions (padding and, in instruction tuning, the prompt tokens)."""
    optimizer.zero_grad()
    device_type = "cuda" if next(model.parameters()).is_cuda else "cpu"
    total_valid = sum((labels[:, 1:] != -100).sum().item() for _, _, labels in micro_batches)
    total_loss = 0.0
    for input_ids, attention_mask, labels in micro_batches:
        with torch.autocast(device_type=device_type, enabled=use_amp):
            logits = model(input_ids)
            shift_logits = logits[:, :-1, :].contiguous()
            shift_labels = labels[:, 1:].contiguous()
            loss_sum = F.cross_entropy(
                shift_logits.view(-1, shift_logits.size(-1)),
                shift_labels.view(-1),
                ignore_index=-100,
                reduction="sum",
            )
            scaled_loss = loss_sum / total_valid
        if use_amp:
            scaler.scale(scaled_loss).backward()
        else:
            scaled_loss.backward()
        total_loss += loss_sum.item()
    if use_amp:
        scaler.step(optimizer)
        scaler.update()
    else:
        optimizer.step()
    return total_loss / total_valid

I trained a small causal-LM stand-in (embedding plus linear output head, vocabulary of 2,256 tokens) on a fixed set of 4 micro-batches per optimizer step, each with the first 4 positions masked out as "prompt": loss dropped from 7.77 to 0.00034 over 800 optimizer steps, confirming the loop actually learns. I separately verified the accumulation scaling itself is correct, not just plausible, by comparing the accumulated gradients from this micro-batch loop against the gradients from a single equivalent big-batch forward/backward pass (all micro-batches concatenated, mean cross-entropy over every valid position): normalizing each micro-batch's summed loss by the accumulation window's TOTAL valid-token count (as in the code above) reproduced the single-big-batch gradient exactly; normalizing instead by a flat 1/(number of micro-batches), the more commonly seen but subtly wrong convention when micro-batches can differ in how many tokens are masked, did not match. This run used autocast with mixed precision disabled (enabled=False) since this environment has no CUDA device; the exact same code path exercises correctly with enabled=True on GPU hardware, only the numeric precision used internally changes.

Trade-offs and pitfalls: A very easy bug in this pattern is forgetting the shift between logits and labels, computing cross-entropy directly against unshifted labels silently trains the model to predict the CURRENT token from itself rather than the next token, which technically runs without error but never learns anything useful for generation. A second common mistake is masking the prompt tokens in the INPUT (so the model never even sees them) rather than only in the LABELS (so the model still sees the prompt as context but is not scored on predicting it), which is the correct behavior for instruction tuning, where the prompt provides context the response must be conditioned on. A third, easy-to-miss mistake specific to gradient accumulation is normalizing each micro-batch's loss by a flat 1/(number of micro-batches) rather than by the accumulation window's total valid-token count, when micro-batches vary in sequence length or masked-token count (the normal case with padding), that flat normalization silently changes the effective per-token learning rate batch to batch, exactly the class of bug this executed comparison against a single big-batch gradient was designed to catch.

Direct answer: Designing a reward model that captures long-term user satisfaction rather than just immediate preference requires a temporal model that predicts a delayed outcome (retention, repeat engagement, or a lifetime-value proxy) from a user's interaction history, with explicit correction for the exposure bias that comes from only observing outcomes for content the existing system actually served.

Structured elaboration:

• Input signals across three tiers: immediate signals (clicks, dwell time, explicit ratings) that are cheap and abundant but only weakly correlated with what actually matters long-term; delayed signals (days-active retention, repeat sessions, conversion or churn events) that are the actual target but sparse and slow to observe; and contextual signals (interaction history, content embeddings, user segment, recency) that let the model condition its prediction on more than the current interaction alone.
• Model design: a user encoder (a sequence model over past interactions producing a user-state representation) combined with a temporal prediction head that forecasts the delayed target (for example a fixed-horizon retention or engagement proxy) from the current state and action; jointly training this delayed-outcome objective alongside an immediate-reward auxiliary objective tends to stabilize training, since the delayed signal alone is sparse and noisy early in training; and a latent, regularized "stable preference" representation (learned via a variational or contrastive objective) helps the model separate a user's persistent preference from short-term noise in any single interaction.
• Causal correction for exposure bias: because the training data only reflects outcomes for content the CURRENT system actually chose to show, naively fitting a model to this logged data risks learning "what the existing system already favors" rather than genuine long-term value, correcting with inverse-propensity weighting (or a doubly-robust estimator) on the logged exposure probabilities is necessary to avoid this bias compounding.
• Evaluation: correlate the model's predicted long-term score against the ACTUAL observed long-term KPI at multiple time horizons (for example both a 7-day and a 90-day window) rather than only the immediate signal; use offline policy evaluation (importance-sampling-based estimators) to estimate how much a policy optimizing this reward would actually improve real long-term outcomes before ever deploying it live; run ablations removing the debiasing correction or the latent stable-preference component to confirm each piece is actually earning its complexity; and ultimately validate with a real, pre-registered A/B test measuring the true long-term metric over a sufficiently long window, since no offline proxy fully substitutes for that.

Worked example: A concrete ablation-driven validation: train three variants, the full model (temporal head plus debiasing plus latent stable-preference embedding), a version with the propensity-based debiasing removed, and a version using only the immediate-reward signal with no temporal head at all; if the immediate-reward-only variant correlates well with SHORT-horizon engagement but poorly with the actual 90-day retention outcome, while the full model's correlation holds up much better at the 90-day horizon, that gap is the concrete, measured evidence that the temporal modeling and debiasing are earning their added complexity, rather than an assumption.

Trade-offs and pitfalls: The single biggest risk in this design is that immediate signals are so much more abundant and easier to optimize against that a poorly-regularized model can quietly collapse back toward optimizing short-term engagement, which is exactly why the auxiliary immediate-reward objective needs to be balanced against, not allowed to dominate, the delayed-outcome objective during training, and why the evaluation explicitly checks correlation at the LONG horizon, not just the short one. A second pitfall is skipping the propensity-based debiasing step because logged exposure data is convenient to use as-is, doing so risks training the reward model to reinforce whatever the existing system already over-serves, rather than genuinely discovering what drives long-term satisfaction.