LLM Fine-Tuning and Alignment Questions

Direct answer: LoRA (Low-Rank Adaptation) freezes the pretrained weight matrix and instead learns a small, low-rank update on top of it, so fine-tuning trains far fewer parameters than updating the full matrix while still adapting the model's behavior.

Structured elaboration: For a weight matrix $W$ of shape $(d_{out}, d_{in})$, LoRA replaces the usual full-matrix update with two much smaller trainable matrices $A$ (shape $r \times d_{in}$) and $B$ (shape $d_{out} \times r$), where the effective weight used at inference is:

$W' = W + \frac{\alpha}{r} \, B A$

Here $r \ll \min(d_{out}, d_{in})$ is the rank, a hyperparameter controlling how expressive the update can be (typical values are 4 to 64 for large language models). $\alpha$ is a scaling factor that controls the effective magnitude of the update relative to the rank; dividing by $r$ keeps the update's scale roughly comparable across different rank choices, so you can change $r$ without having to re-tune $\alpha$ from scratch. $B$ is typically initialized to all zeros and $A$ to small random values, so the adapted model starts out numerically identical to the original pretrained model and then diverges as training updates $A$ and $B$. LoRA adapters are most commonly applied to the attention query and value projection matrices (and sometimes the feed-forward layers), since those dominate a transformer's parameter count and strongly shape its behavior. Per-task adapters are stored as just the pair $(A, B)$ for each layer they are applied to, which is typically megabytes rather than the gigabytes a full fine-tuned checkpoint would need, so serving many task-specific variants of one base model becomes practical.

Worked example: For a single attention projection of shape $(12288, 12288)$ (a dimension similar to a large transformer's hidden size), full fine-tuning of that one projection would train $12288 \times 12288 = 150{,}994{,}944$ parameters. With LoRA at rank $r=8$, the trainable parameter count is $r \times (d_{in} + d_{out}) = 8 \times (12288 + 12288) = 196{,}608$ parameters, about $0.13\%$ of the full count, a roughly 768x reduction for that single projection. This is why LoRA cuts training memory so much: gradients and optimizer state (which for Adam-family optimizers are often 2-3x the raw parameter count) only need to be stored for $A$ and $B$, not for the frozen $12288 \times 12288$ matrix.

Trade-offs and pitfalls: LoRA reduces trainable parameters, but at inference the unmerged form ($W$ plus a separate low-rank multiply) adds a small amount of extra compute; in production this is almost always avoided by merging $BA$ into $W$ once training is done, which restores exactly the same inference cost as the original model with zero runtime overhead, at the cost of losing the ability to instantly swap adapters without reloading weights. Choosing $r$ too small can under-fit a task that genuinely needs a higher-rank update (complex domain shifts, multi-task mixtures), while choosing it unnecessarily large gives up most of LoRA's memory savings without a clear accuracy benefit, so rank is usually tuned empirically starting from a small value like 8.

Direct answer: A reward model in RLHF is a supervised model, usually a transformer with a scalar output head, that takes a prompt plus one or more candidate completions and predicts a scalar score reflecting how much a human would prefer that completion, so it can act as a differentiable proxy for human judgment during policy optimization.

Structured elaboration:

• Typical inputs: either a single (prompt, completion) pair scored independently, or a prompt paired with multiple candidate completions scored together for ranking; completions are usually concatenated with the prompt and encoded with a separator so the model can attend to both.
• Label formats: pairwise preferences (rater says completion A is better than B) are the most common and are trained with a Bradley-Terry-style pairwise loss; numeric Likert-style scores are trained more like regression; in practice, numeric labels are often converted into pairwise comparisons since relative judgments tend to be more reliable and consistent across annotators than absolute scores.
• Model architecture: a transformer encoder or decoder backbone (frequently initialized from the same pretrained or supervised-fine-tuned checkpoint as the policy, for representation quality) with the final hidden state, at the end-of-sequence position or pooled across tokens, fed into a small linear or MLP head that outputs a single scalar.
• Preprocessing: consistent tokenization with the policy model, truncation or careful cropping so prompt plus completion fit the context window (usually preserving the completion, since that is what is being judged), padding and batching (often length-bucketed to reduce wasted computation on padding tokens), and clear delimiters between prompt and completion sections.
• Why a reward model is necessary: human feedback itself is not differentiable and is far too slow and expensive to query on every training step, so a reward model distills a large batch of human preference judgments into a fast, differentiable scoring function that a policy-optimization algorithm (like PPO) can query millions of times cheaply during training.
• Loss functions: for pairwise preferences, the standard choice is the Bradley-Terry pairwise-logistic loss, $\mathcal{L} = -\log \sigma(s(a) - s(b))$ where $a$ is the preferred completion, which is equivalent to a cross-entropy loss on the model's implied preference probability; for scalar labels, mean-squared error against the numeric rating is used instead. The pairwise loss is numerically stabilized in practice using a log-sigmoid or softplus formulation rather than computing sigmoid and log separately, since that avoids overflow for large score differences.

Worked example: For a batch of preference pairs, if the reward model outputs $s(a) = 2.3$ for the preferred completion and $s(b) = 0.8$ for the rejected one, the pairwise loss is $-\log \sigma(2.3 - 0.8) = -\log \sigma(1.5) \approx -\log(0.818) \approx 0.201$, a small loss because the model already ranks them correctly with a healthy margin. If instead the model scored them the wrong way round, say $s(a)=0.8, s(b)=2.3$, the loss becomes $-\log \sigma(-1.5) \approx -\log(0.182) \approx 1.70$, a much larger loss that pushes the model to correct its ranking. In practice this loss is computed over many thousands of such pairs per training step, not one at a time.

Trade-offs and pitfalls: A reward model trained only on pairwise comparisons has no absolute scale, only relative ordering is meaningful, so its raw scores should not be compared across very different prompt types without care. Regularization and disagreement handling matter: if the training data mixes multiple annotators with different standards without any normalization, the reward model can pick up annotator-specific artifacts (such as a preference for longer responses) rather than genuine quality signal, a precursor to the reward hacking problem seen later in the RLHF pipeline.

Direct answer: Instruction tuning is supervised fine-tuning specifically on (instruction, response) pairs so a model learns to follow natural-language instructions directly, and it differs from generic supervised fine-tuning mainly in data shape and breadth, and from RLHF mainly in what kind of feedback it learns from, direct labeled targets versus relative human preferences.

Structured elaboration:

• Data: instruction tuning uses explicit instruction-to-response pairs, often covering a broad and diverse set of task types (summarize, explain, format, answer); generic supervised fine-tuning is the same maximum-likelihood training recipe but is typically narrower in scope, tuned to one specific task like classification or translation rather than general instruction-following; RLHF instead uses preference data, pairwise comparisons or rankings of candidate outputs, not direct target responses at all.
• Objectives: both instruction tuning and generic SFT maximize the likelihood of the target response given the input (a standard cross-entropy objective); RLHF instead optimizes a policy to maximize a reward signal learned from those preferences, typically via a policy-gradient method like PPO, which is a fundamentally different training loop and objective shape.
• Iteration speed: instruction tuning and SFT are fast to iterate, since they are standard supervised training, and it is cheap to add or regenerate instruction-response pairs (including synthetically); RLHF is slower and more expensive, since it requires collecting preference labels, training a reward model, and running an RL loop before you can even measure whether an update helped.
• Data types, sizes, and behavioral improvements: instruction tuning typically needs on the order of thousands to tens of thousands of diverse instruction-response pairs and mainly improves instruction-following breadth and output formatting; RLHF typically needs a comparable or larger volume of preference comparisons and mainly improves nuanced alignment properties (tone, safety, subtle helpfulness judgments) that are hard to specify as a single correct target response.
• Typical failure modes: instruction tuning can overfit to the phrasing of its training instructions, produce fluent but factually wrong answers, or faithfully follow harmful instructions if any slipped into the training data; RLHF's distinct failure modes include reward hacking, RL-specific training instability, and amplification of whatever bias existed in the annotator pool.
• Why pipelines commonly use both SFT and an RLHF/DPO stage rather than either alone: SFT efficiently teaches the model the basic shape of following instructions and producing well-formatted, on-task responses from direct examples, which RLHF alone would be slow and expensive to teach from scratch since it only ever gets a relative preference signal, not a target to imitate; RLHF or DPO then refines nuanced qualities (tone, subtle safety judgments, helpfulness trade-offs) that are hard to write down as a single correct example but are easy for a human to judge comparatively once the model already produces reasonable candidates to compare.

Worked example: A team building a general-purpose assistant that must reliably follow many natural-language task templates before a demo deadline would reach for instruction tuning first: assembling a broad, diverse instruction-response dataset (human-authored plus synthetic examples) and running standard supervised fine-tuning is far faster to iterate on than starting with preference labeling and an RL loop, and gets basic instruction-following working quickly; RLHF or DPO would typically be layered on afterward, once that baseline exists, to refine harder-to-specify qualities like tone and safety.

Trade-offs and pitfalls: A common mistake is treating instruction tuning as a full substitute for an alignment stage, it teaches a model to follow the FORM of instructions well but does not by itself teach the nuanced preference judgments (which of two technically-correct responses is actually better) that RLHF or DPO are specifically designed to capture from comparative human feedback.

Direct answer: Reinforcement Learning from Human Feedback (RLHF) aligns a large language model by using human preference judgments to train a reward model, then using that reward model as the training signal for a reinforcement-learning algorithm that updates the policy (the LLM itself) to produce outputs humans prefer.

Structured elaboration:

• Core components: the policy (the LLM generating responses, usually already supervised-fine-tuned as a starting point), the reward model (a learned scalar predictor of human preference), human preference data (pairwise comparisons or rankings collected from raters), and an RL algorithm (most commonly PPO) that updates the policy using the reward model's scores, typically regularized by a KL penalty against a frozen reference policy to prevent it drifting too far from reasonable, fluent behavior.
• Typical data flow: a prompt is sent to the policy, which generates one or more candidate responses; human raters compare candidates and record a preference; the accumulated preference data trains the reward model to predict those preferences; the RL loop then samples fresh responses from the (evolving) policy, scores them with the reward model, and updates the policy to increase expected reward while the KL penalty discourages it from moving too far from its starting behavior.
• Where raters fit in: they sit specifically between generation and reward-model training, they never see or influence the RL update directly, their judgments are distilled entirely through the reward model, which is what makes the pipeline scalable (the reward model can be queried far faster and more cheaply than a human).
• Practical outline for improving assistant helpfulness end to end: collect preference data on the assistant's current outputs, train or update the reward model on that data, run PPO (or a similar policy-optimization method) against the reward model with a KL constraint, and add explicit safety constraints (either in the reward shaping or as hard runtime filters) so that optimizing for the measured reward does not come at the cost of safety.
• Common pitfalls: reward hacking, where the policy learns to exploit quirks in the reward model (for example padding responses to trigger a length-based preference bias) rather than genuinely improving quality, is the most cited failure mode, and validating that the reward model actually generalizes to real human preferences, rather than just fitting its own training distribution, is essential before trusting it to drive a large policy update.

Worked example: A single preference-label record that a pipeline might store looks like:

{
  "prompt_id": "p123",
  "prompt": "Summarize the risks of investing in crypto.",
  "response_a": "Crypto is volatile and largely unregulated, with risk of total loss.",
  "response_b": "Investing in crypto can lead to large losses due to price volatility and limited investor protections.",
  "preferred": "b",
  "rater_id": "r42",
  "timestamp": "2026-05-01T12:34:56Z"
}

This single record becomes one training example for the reward model (label: response b scored higher than response a for this prompt); the reward model is trained on many thousands of records like this before it is trusted to score fresh policy rollouts during the RL stage.

Trade-offs and pitfalls: RLHF is more expensive and operationally heavier than supervised fine-tuning alone, since it requires an entire second model (the reward model) plus an RL training loop, both of which introduce their own failure modes (a miscalibrated reward model, or a KL coefficient tuned badly enough to let the policy drift into degenerate behavior). It is usually applied only after supervised fine-tuning has already gotten the model to a reasonable baseline, since starting RLHF from a poorly fine-tuned model tends to be unstable and slow.

Direct answer: Pairwise preference judgments, Likert-scale ratings, binary accept/reject, and full ordinal ranking each collect human feedback differently, and they trade off label quality, cost, annotator cognitive load, and how directly they map onto a training objective.

Structured elaboration:

• Pairwise preference judgments (choose which of two outputs is better): give a strong relative-quality signal and are naturally robust to different annotators using different internal "scales", since a rater only ever has to judge relative, not absolute, quality; the cognitive load is low (comparing two things is easy), and the format maps directly onto the standard pairwise-logistic (Bradley-Terry) loss used to train reward models.
• Likert-scale ratings (for example 1-5): give a finer-grained, absolute signal per item, but different raters tend to anchor the scale differently (one rater's "4" is another's "3"), which introduces scale bias that must be corrected for (typically by normalizing per-annotator, or converting Likert scores into pairwise comparisons before training); cognitive load is moderate since raters must internally calibrate a scale.
• Binary accept/reject: the cheapest and lowest-cognitive-load option, but the resulting signal is coarse (it misses gradations of quality) and, because most outputs from a reasonably capable model tend to be "acceptable", the accept/reject label distribution can become sparse and less informative than a comparison would be.
• Full ordinal ranking of a set of k outputs: extracts the richest information per task, ranking 4 outputs implicitly encodes 6 pairwise comparisons, but cognitive load rises sharply with k (holding many outputs in mind to rank consistently is harder than comparing two), which can reduce consistency for larger k; rankings are typically decomposed into pairwise comparisons for training, so this format is really a more information-dense way to generate the same underlying pairwise training signal.
• Overall trade-off: pairwise comparisons (or rankings converted to pairwise comparisons) are generally preferred for RLHF because they align directly with the standard reward-model training objective and avoid the scale-bias problem inherent to absolute ratings, while Likert and binary formats remain useful when you specifically need an absolute quality signal or a fast, cheap filtering pass.

Worked example: If two annotators rate the same response a "3" and a "4" on a 1-5 Likert scale, that half-point gap could reflect a real quality difference, or it could simply be two different people's internal calibration of the scale, whereas if both annotators are instead asked "which of these two responses is better", a consistent pairwise choice from both raters gives an unambiguous signal that converts cleanly into a single training example, without needing any per-annotator scale correction.

Trade-offs and pitfalls: Choosing full ordinal ranking purely because it "contains more information per task" can backfire if raters cannot reliably rank more than 3-4 items consistently, in which case the extra information is partly noise. Choosing binary accept/reject purely for its low cost can starve the reward model of gradation exactly where it matters most, distinguishing a good response from a great one, which is often the harder and more valuable distinction to learn.