LLM Fine-Tuning and Alignment Questions

Direct answer: Domain adaptation is the sub-problem of transfer learning that specifically addresses a shift in data distribution between the source data a model was trained or pretrained on and the target domain it must perform well in, often with little or no labeled target data, whereas general transfer learning more broadly assumes the source and target tasks are similar and simply reuses pretrained representations.

Structured elaboration: Transfer learning's focus is transferring representations or weights across tasks or domains under the assumption that they are reasonably similar; domain adaptation's focus is explicitly modeling and correcting for the distribution difference itself (covariate shift, label shift, or feature shift) so a single task model still generalizes despite that shift, often without any labeled examples from the target domain at all.

Worked example: In computer vision, a common domain-adaptation scenario is training an autonomous-driving perception model on abundant labeled simulated images and deploying it on real camera footage, where the "appearance gap" between synthetic and real images is the domain shift; an initial strategy combines unsupervised adversarial feature alignment (training the model so its internal features cannot be distinguished as coming from simulation versus reality), style-transfer techniques to make source images look more realistic, and self-training that pseudo-labels confident predictions on real target images and folds them back into training. In NLP, a comparable scenario is training a sentiment classifier on long-form product reviews and deploying it on short, noisy tweets, a domain shift in length, vocabulary, and style; an initial strategy fine-tunes a pretrained transformer, continues masked-language-model pretraining on unlabeled tweets specifically to adapt its embeddings to that vocabulary and style, and uses a contrastive or multi-task loss to align source and target sentence representations, again combined with self-training on high-confidence pseudo-labels.

Trade-offs and pitfalls: These approaches all lean on the same small set of pillars, representation alignment, data augmentation or style transfer, and leveraging unlabeled target data through self-training, but each introduces its own risk: adversarial feature alignment can be unstable to train and requires careful tuning to avoid collapsing useful task-relevant information along with the domain-specific signal, and self-training on pseudo-labels can reinforce the model's existing mistakes if the confidence threshold for accepting a pseudo-label is set too low, so pseudo-labels should be validated against a small genuinely-labeled target sample before being trusted at scale.

Direct answer: Alignment drift is when a deployed model's behavior gradually diverges from its intended safety, policy, or utility goals over time, for example starting to produce unsafe, biased, or off-specification outputs even though it passed evaluation initially, and it typically comes from a small set of well-understood causes rather than random decay.

Structured elaboration: Common causes include data drift (the distribution of real user inputs shifting toward new topics or adversarial phrasing the model was not evaluated against), incremental model updates or pipeline changes introducing unintended behavioral side effects, feedback-loop effects (model-generated content getting re-ingested into future training data, reinforcing whatever undesirable patterns already existed), changes in the surrounding system (prompt-template edits, third-party tool or integration changes) that shift the effective input the model sees without the model itself changing, and drift in the evaluation metrics or labeling standards themselves, making an apparently-stable score mask a real underlying change. Three practical mitigation categories address these: continuous monitoring with alerting on safety and quality metrics (using both random and high-risk-targeted sampling so rare but severe drift is not diluted into an aggregate average); robust CI/CD gating for any model or pipeline change (regression tests against an immutable evaluation set, red-team scenarios, and canary rollouts before a change reaches full traffic); and data-governance controls specifically around feedback loops (never blindly retraining on the model's own recent outputs without labeling and vetting them first, since that is exactly how a feedback loop reinforces its own drift).

Worked example: A concrete drift-detection trigger: if a fixed, immutable regression suite (never updated to "adjust for" the new normal) shows a safety-violation rate creeping up 0.5 percentage points per week over a month, even though no explicit model change was deployed in that window, that pattern points toward either upstream data drift (user inputs are shifting) or a feedback-loop effect (model outputs are being re-ingested somewhere upstream), and distinguishing the two requires checking whether the input distribution itself has measurably shifted versus whether recent training data included un-vetted model-generated content.

Trade-offs and pitfalls: The single most common way alignment drift becomes hard to detect is silently updating the evaluation set or metric definitions "to keep them current" without preserving an immutable baseline version, once the yardstick itself moves, a real behavioral regression can be invisible in the metrics. A second common failure is that fixing drift by immediately re-fine-tuning on freshly-observed problematic outputs, without vetting that data first, can itself become the next feedback loop that causes further drift down the line.

Direct answer: Catastrophic forgetting is the sharp loss of a pretrained model's previously-learned capability when it is fine-tuned on new data, because gradient updates that improve the new objective can overwrite the weights that encoded earlier, still-useful behavior.

Structured elaboration: At least five mitigation strategies, in roughly increasing order of engineering complexity, are commonly used: (1) replay/rehearsal, mixing examples from the original task or distribution back into fine-tuning, which is effective when a representative replay set exists but adds data-storage and privacy considerations; (2) Elastic Weight Consolidation and similar regularizers, which penalize changes to parameters estimated to be important for the original task (via something like a Fisher-information estimate), effective for moderate distribution shifts at a moderate compute cost to estimate that importance; (3) simple L2 or knowledge-distillation regularization, constraining the fine-tuned model's weights or outputs to stay close to the original model's, cheap to add and often surprisingly effective; (4) parameter-efficient fine-tuning (adapters or LoRA), which freezes the backbone entirely and trains only a small add-on, very effective at retention precisely because the original weights never move at all; and (5) multi-task or joint fine-tuning, training on the new task alongside the original task's data simultaneously, which prevents forgetting by construction but costs more compute and requires access to the original training distribution.

Worked example: For a model being adapted to a new domain while a small held-out sample of its original general-capability benchmark is available, a concrete test is to measure that benchmark's score before and after fine-tuning: if fine-tuning on the new domain drops the held-out general benchmark score by, say, 15 percentage points while the new-domain metric improves, that gap is the direct, measurable signature of catastrophic forgetting, and re-running the same fine-tuning with a LoRA adapter instead of full-parameter updates (leaving the backbone frozen) is a common way to test whether the drop is attributable to updating the full model versus the domain adaptation itself.

Trade-offs and pitfalls: Regularization-based methods (EWC, L2, distillation) are cheap to add but can under-perform on large distribution shifts, where the new task genuinely conflicts with the old one and no amount of gentle regularization avoids some trade-off. Replay is conceptually robust across a wider range of shift sizes but assumes you still have access to representative original-task data, which is not always true (for example, if the original pretraining data cannot be redistributed for privacy or licensing reasons). Parameter-efficient fine-tuning gives one of the best retention-versus-cost trade-offs specifically because it structurally prevents forgetting (the backbone literally does not change), which is why it is often the first mitigation reached for in modern LLM fine-tuning pipelines, rather than the regularization-heavy approaches that were more common before PEFT was widely adopted.

Direct answer: Instruction tuning is supervised fine-tuning specifically on (instruction, response) pairs so a model learns to follow natural-language instructions directly, and it differs from generic supervised fine-tuning mainly in data shape and breadth, and from RLHF mainly in what kind of feedback it learns from, direct labeled targets versus relative human preferences.

Structured elaboration:

• Data: instruction tuning uses explicit instruction-to-response pairs, often covering a broad and diverse set of task types (summarize, explain, format, answer); generic supervised fine-tuning is the same maximum-likelihood training recipe but is typically narrower in scope, tuned to one specific task like classification or translation rather than general instruction-following; RLHF instead uses preference data, pairwise comparisons or rankings of candidate outputs, not direct target responses at all.
• Objectives: both instruction tuning and generic SFT maximize the likelihood of the target response given the input (a standard cross-entropy objective); RLHF instead optimizes a policy to maximize a reward signal learned from those preferences, typically via a policy-gradient method like PPO, which is a fundamentally different training loop and objective shape.
• Iteration speed: instruction tuning and SFT are fast to iterate, since they are standard supervised training, and it is cheap to add or regenerate instruction-response pairs (including synthetically); RLHF is slower and more expensive, since it requires collecting preference labels, training a reward model, and running an RL loop before you can even measure whether an update helped.
• Data types, sizes, and behavioral improvements: instruction tuning typically needs on the order of thousands to tens of thousands of diverse instruction-response pairs and mainly improves instruction-following breadth and output formatting; RLHF typically needs a comparable or larger volume of preference comparisons and mainly improves nuanced alignment properties (tone, safety, subtle helpfulness judgments) that are hard to specify as a single correct target response.
• Typical failure modes: instruction tuning can overfit to the phrasing of its training instructions, produce fluent but factually wrong answers, or faithfully follow harmful instructions if any slipped into the training data; RLHF's distinct failure modes include reward hacking, RL-specific training instability, and amplification of whatever bias existed in the annotator pool.
• Why pipelines commonly use both SFT and an RLHF/DPO stage rather than either alone: SFT efficiently teaches the model the basic shape of following instructions and producing well-formatted, on-task responses from direct examples, which RLHF alone would be slow and expensive to teach from scratch since it only ever gets a relative preference signal, not a target to imitate; RLHF or DPO then refines nuanced qualities (tone, subtle safety judgments, helpfulness trade-offs) that are hard to write down as a single correct example but are easy for a human to judge comparatively once the model already produces reasonable candidates to compare.

Worked example: A team building a general-purpose assistant that must reliably follow many natural-language task templates before a demo deadline would reach for instruction tuning first: assembling a broad, diverse instruction-response dataset (human-authored plus synthetic examples) and running standard supervised fine-tuning is far faster to iterate on than starting with preference labeling and an RL loop, and gets basic instruction-following working quickly; RLHF or DPO would typically be layered on afterward, once that baseline exists, to refine harder-to-specify qualities like tone and safety.

Trade-offs and pitfalls: A common mistake is treating instruction tuning as a full substitute for an alignment stage, it teaches a model to follow the FORM of instructions well but does not by itself teach the nuanced preference judgments (which of two technically-correct responses is actually better) that RLHF or DPO are specifically designed to capture from comparative human feedback.

Direct answer: A reward model in RLHF is a supervised model, usually a transformer with a scalar output head, that takes a prompt plus one or more candidate completions and predicts a scalar score reflecting how much a human would prefer that completion, so it can act as a differentiable proxy for human judgment during policy optimization.

Structured elaboration:

• Typical inputs: either a single (prompt, completion) pair scored independently, or a prompt paired with multiple candidate completions scored together for ranking; completions are usually concatenated with the prompt and encoded with a separator so the model can attend to both.
• Label formats: pairwise preferences (rater says completion A is better than B) are the most common and are trained with a Bradley-Terry-style pairwise loss; numeric Likert-style scores are trained more like regression; in practice, numeric labels are often converted into pairwise comparisons since relative judgments tend to be more reliable and consistent across annotators than absolute scores.
• Model architecture: a transformer encoder or decoder backbone (frequently initialized from the same pretrained or supervised-fine-tuned checkpoint as the policy, for representation quality) with the final hidden state, at the end-of-sequence position or pooled across tokens, fed into a small linear or MLP head that outputs a single scalar.
• Preprocessing: consistent tokenization with the policy model, truncation or careful cropping so prompt plus completion fit the context window (usually preserving the completion, since that is what is being judged), padding and batching (often length-bucketed to reduce wasted computation on padding tokens), and clear delimiters between prompt and completion sections.
• Why a reward model is necessary: human feedback itself is not differentiable and is far too slow and expensive to query on every training step, so a reward model distills a large batch of human preference judgments into a fast, differentiable scoring function that a policy-optimization algorithm (like PPO) can query millions of times cheaply during training.
• Loss functions: for pairwise preferences, the standard choice is the Bradley-Terry pairwise-logistic loss, $\mathcal{L} = -\log \sigma(s(a) - s(b))$ where $a$ is the preferred completion, which is equivalent to a cross-entropy loss on the model's implied preference probability; for scalar labels, mean-squared error against the numeric rating is used instead. The pairwise loss is numerically stabilized in practice using a log-sigmoid or softplus formulation rather than computing sigmoid and log separately, since that avoids overflow for large score differences.

Worked example: For a batch of preference pairs, if the reward model outputs $s(a) = 2.3$ for the preferred completion and $s(b) = 0.8$ for the rejected one, the pairwise loss is $-\log \sigma(2.3 - 0.8) = -\log \sigma(1.5) \approx -\log(0.818) \approx 0.201$, a small loss because the model already ranks them correctly with a healthy margin. If instead the model scored them the wrong way round, say $s(a)=0.8, s(b)=2.3$, the loss becomes $-\log \sigma(-1.5) \approx -\log(0.182) \approx 1.70$, a much larger loss that pushes the model to correct its ranking. In practice this loss is computed over many thousands of such pairs per training step, not one at a time.

Trade-offs and pitfalls: A reward model trained only on pairwise comparisons has no absolute scale, only relative ordering is meaningful, so its raw scores should not be compared across very different prompt types without care. Regularization and disagreement handling matter: if the training data mixes multiple annotators with different standards without any normalization, the reward model can pick up annotator-specific artifacts (such as a preference for longer responses) rather than genuine quality signal, a precursor to the reward hacking problem seen later in the RLHF pipeline.