LLM Fine-Tuning and Alignment Questions

Direct answer: Debugging a hallucination increase after fine-tuning on domain-specific QA requires first isolating whether the cause is the DATA, the TRAINING process, or INFERENCE-time behavior, since each points to a different fix, then applying training-level and runtime mitigations in parallel rather than waiting for one full explanation before acting.

Structured elaboration:

• Isolation experiments: an A/B comparison of the base (pre-fine-tune) model against the fine-tuned model on the SAME held-out QA set, with human-evaluated faithfulness labels, quantifies exactly how much the fine-tuning step itself changed the hallucination rate; breaking this down by domain, question type, and question length surfaces WHERE the increase concentrates rather than treating it as uniform; an ablation fine-tuning on subsets of the data (a smaller fraction, or excluding a specific noisy source) tests whether the increase scales with a specific portion of the training data; and running the same prompts at temperature zero isolates whether the behavior is a genuinely learned pattern versus incidental sampling randomness.
• Dataset checks: audit the provenance of training sources specifically for dubious or fabricated content (scraped forum answers, synthetic chat data of uncertain quality), sample-check labels for actual hallucinated or unsupported answers that may have been included as if they were correct targets, check for data contamination (examples that overlap with or leak the evaluation set), and measure distributional drift between the fine-tuning data and the actual intended production query distribution, a training set that looks fine in isolation can still be a poor match for what the model will actually be asked in production.
• Training interventions: prioritize high-quality, evidence-backed examples early in training and downweight or remove low-confidence sources; add an explicit training signal against hallucination, negative examples pairing a question with a plausible-but-false answer labeled as bad, or an auxiliary objective requiring the model to point to supporting evidence for its claims; reduce the learning rate, use fewer epochs, or freeze lower layers to avoid overwriting general world knowledge the base model already had correct; mix in a small fraction of the original clean pretraining or instruction-tuning data to help preserve that general knowledge; and explicitly teach the model to say "no reliable source" or otherwise abstain when evidence is genuinely absent, rather than only ever training it to produce a confident answer.
• Runtime mitigations, deployable faster than a full retrain: retrieval-augmented generation that conditions the answer on retrieved documents and requires citations; a separate verifier model that checks generated claims against retrieved evidence and triggers a fallback or abstention when support is weak; lower-temperature, more conservative decoding specifically for factual answers; and an explicit confidence threshold below which the system returns "insufficient evidence" rather than a confident guess.
• Evaluation and ongoing monitoring: track hallucination rate on a dedicated factual-QA regression suite going forward, and block any future release that regresses beyond a set threshold, treating this the same way a functional regression test would be treated, not as a one-time investigation.

Worked example: A concrete short investigation plan: in the first week, run the A/B comparison plus a source audit on a sample of the training data; if the noise is concentrated in one identifiable source (for example scraped forum content with a much higher rate of unverified claims than the rest of the dataset), remove that source and re-run a small-scale fine-tune to confirm the hallucination rate drops before committing to a full production retrain; in parallel, deploy retrieval-augmented generation plus a verifier as an immediate runtime hotfix, since that mitigates user-facing harm regardless of how long the training-side investigation takes; over the following weeks, run controlled comparisons (clean data only, clean data plus an evidence-pointing objective, clean data plus RLHF specifically for factuality) against the same held-out faithfulness-labeled set to decide which training-level fix is actually worth adopting long-term.

Trade-offs and pitfalls: Waiting for a complete root-cause explanation before deploying ANY mitigation delays reducing real user-facing harm unnecessarily, the runtime mitigations (retrieval augmentation, a verifier, conservative decoding) can and should deploy in parallel with the slower training-level investigation, not after it concludes. A second pitfall is treating a reduced hallucination rate after removing one suspected noisy source as final proof of the diagnosis without re-running the full evaluation suite, a small-scale confirmation run is a good first check, but the production decision should rest on the same rigorous held-out evaluation used throughout this investigation, not an informal spot-check.

Direct answer: Converting raw pairwise preference records into reward-model training examples means turning each (prompt, completion_a, completion_b, preferred) tuple into a single encoded input string plus a binary label, while explicitly handling malformed rows rather than silently mislabeling them.

Structured elaboration: The function packs the prompt and both candidate completions into one input string with clear delimiters (so the model sees both candidates in a single forward pass and can compare them), and converts the preference field into a numeric label. A key design decision is normalizing the preferred field (trimming whitespace, uppercasing) before checking it, and explicitly SKIPPING any row whose preferred value is not recognizably "A" or "B", rather than defaulting it to one label or the other, since silently mislabeling a malformed row would inject incorrect training signal without any visible error.

Worked example (executed):

def build_reward_model_pairs(records):
    """records: list of (prompt, completion_a, completion_b, preferred), preferred in {'A','B'}.
    Returns (examples, skipped) where examples is a list of (input_text, label), label=1 if A preferred."""
    examples = []
    skipped = 0
    for prompt, completion_a, completion_b, preferred in records:
        p = preferred.strip().upper()
        if p not in ("A", "B"):
            skipped += 1
            continue
        input_text = f"PROMPT: {prompt}\nA: {completion_a}\nB: {completion_b}"
        label = 1 if p == "A" else 0
        examples.append((input_text, label))
    return examples, skipped

I verified this against three test records, including one deliberately malformed row with preferred="C": the function correctly produced 2 valid examples (labels 1 and 0 respectively, matching "A" and "b" preferred, confirming case-insensitivity works) and reported 1 skipped row, and the encoded input text matched the exact expected format ("PROMPT: ...\nA: ...\nB: ..."), all assertions passed.

Trade-offs and pitfalls: A common bug in this exact function is defaulting an unrecognized preferred value to a fixed label (for example always 0) instead of skipping it, which silently trains the reward model on incorrect labels for any malformed row rather than surfacing a data-quality problem. A second consideration is that this function assumes prompt and completions are already plain, detokenized strings, if the upstream data contains un-decoded byte sequences or inconsistent whitespace, that should be normalized before this step, not inside it, to keep this function's contract simple and testable.

Direct answer: Applying RLHF to a multi-modal (text-and-image) model requires a reward model that jointly attends across both modalities rather than treating them separately, human feedback collection that includes modality-specific error types like hallucinated visual details, and additional safety considerations (visual grounding, privacy in images, dual-modality attacks) beyond what text-only RLHF already covers.

Structured elaboration:

• Reward-model design: the input is an image representation, the question, and a candidate answer together, processed through cross-attention layers that let the model jointly reason across modalities before pooling into a single representation and predicting a scalar reward; training combines the standard pairwise-preference loss with, where available, explicit scalar labels for factuality and safety, and calibration or uncertainty estimation (an ensemble or similar) helps downweight low-confidence reward-model outputs before they drive a policy update.
• Human feedback collection specific to multi-modal outputs: beyond ordinary pairwise preference, collect targeted error labels specifically distinguishing hallucination from legitimate inference, binary correctness judgments, and, where relevant, region-level grounding annotations tying an answer's claim to a specific part of the image; give annotators tool support (image zoom, detected-object overlays, attention visualization) so they can actually verify visual claims rather than guessing, and include adversarial and edge-case examples (occluded images, composite scenes) deliberately, not just typical clean images.
• Safety considerations unique to multi-modal alignment: vision-specific hallucination (the model confidently describing details that are not actually in the image) needs explicit grounding checks and a trained willingness to say a detail "is not visible" rather than confabulating; privacy risks specific to images (faces, license plates, other identifying visual information) need detection and blocking, with the reward model specifically penalizing attempts to infer someone's identity from an image; dual-modality attacks (a text prompt deliberately crafted to manipulate what the model attends to in the image) need adversarial training using deliberately mismatched image-text pairs; and spurious correlations (the reward model learning a dataset bias tying certain objects to certain demographic assumptions, for example) need explicit balancing and counterfactual examples to avoid the reward model reinforcing them.
• Training and evaluation loop: iterate collecting feedback, training or updating the reward model (often pretraining it on cheaper synthetic labels before refining on real human preferences), fine-tuning the policy with a KL constraint against the base model, and evaluating on held-out factuality and safety benchmarks plus red-team tests; evaluation should go beyond generic text-quality metrics to include a grounding metric (how well an answer's claims are supported by the actual image content) and an abstention rate (how often the model correctly declines rather than confabulating), not just a preference win-rate.

Worked example: A concrete grounding check built into evaluation: for a visual question-answering benchmark where the ground truth is known, measure not just whether the final answer is correct but whether the model's answer references image content that is ACTUALLY present (a grounding metric, for example an intersection-over-union style score against the true supporting image region when region-level annotations exist), a model can produce a correct-sounding answer for the wrong reason (matching a common dataset pattern rather than genuinely reading the image), and this grounding-specific check is what catches that failure mode where a plain accuracy metric would not.

Trade-offs and pitfalls: Collecting richer signals (region-level grounding annotations, explicit hallucination-versus-inference labels) improves safety and interpretability but meaningfully increases annotation cost and complexity relative to plain preference comparisons, which scale more cheaply but carry a noisier signal, most practical pipelines use a mix, cheaper preference data at volume plus a smaller, richer grounding-annotated set for the highest-stakes evaluation. A second, real tension is that a strong KL constraint against the base model preserves general helpfulness but can also limit how much the safety-specific reward signal is able to actually shift behavior, this coefficient needs deliberate validation specific to the multi-modal safety properties that matter most, not just carried over unchanged from a text-only RLHF setup.

Direct answer: RLHF/PPO instability usually comes from a small set of well-known sources, catastrophic forgetting of base capabilities, reward hacking, high-variance gradients, and distributional shift between the rollout policy and the data the reward model was trained on, and the PPO hyperparameters most responsible for turning these into visible instability are the KL coefficient, the clipping epsilon, and advantage normalization.

Structured elaboration:

• KL coefficient too low (or KL penalty absent): the policy drifts far from the reference model, symptom is generation quality visibly degrading (repetitive, incoherent, or off-distribution text) even as the measured reward keeps climbing, a classic reward-hacking signature; the fix is raising the KL coefficient or tightening a hard KL budget, and re-checking that the reference model used for the penalty is actually the intended baseline.
• Clipping epsilon too large: large policy updates are allowed through with little constraint, symptom is reward or KL divergence spiking sharply between consecutive updates; the fix is lowering epsilon (a common default is 0.2) and, if instability persists, reducing the learning rate or number of PPO epochs per rollout batch.
• Advantage normalization missing or misconfigured: without normalizing advantages (typically to zero mean, unit variance per batch), a few outlier high-reward rollouts can dominate the gradient, symptom is erratic, spiky training curves that do not smooth out with more data; the fix is per-batch advantage normalization and clipping extreme advantage values.
• Beyond hyperparameter tuning, more advanced mitigations exist for persistent instability: trust-region-style constraints tighter than PPO's default clipping, ensembling multiple reward models and using their disagreement as an uncertainty signal, curriculum learning (starting from easier prompts before harder ones), and replay buffers that mix in earlier, more stable rollouts; each of these adds engineering complexity and, in the case of reward-model ensembles, extra training and serving cost, so they are typically reached for only after basic hyperparameter tuning has failed to resolve the instability.

Worked example: A concrete symptom-to-fix trace: reward climbs 20% over 500 steps while a held-out human-eval sample shows quality dropping, this points at KL coefficient too low; doubling the KL coefficient and re-running from the last stable checkpoint is the first fix to try before reaching for anything more elaborate like reward-model ensembling. If instead reward and KL both spike sharply at the same update, that points at the clip epsilon or learning rate rather than the KL coefficient, and the fix is tightening the clip range first.

Trade-offs and pitfalls: It is tempting to treat every RLHF instability as a reward-hacking problem needing a reward-model fix, but many cases are simple PPO-hyperparameter misconfigurations that are far cheaper to diagnose and correct first. Conversely, advanced mitigations like reward-model ensembles or curriculum learning are sometimes reached for prematurely when a simpler KL-coefficient or clip-epsilon adjustment would have resolved the same symptom at a fraction of the engineering cost.

Direct answer: Deciding whether to retrain a foundation model from scratch, continue pretraining, or only fine-tune adapters (or stay frozen and use retrieval-augmented prompting instead) should be driven by a structured cost-benefit framework, quantifying the actual domain gap and available data, estimating benefit against compute cost, and weighing forgetting risk and time-to-market, rather than defaulting to whichever option sounds most thorough.

Structured elaboration:

• Quantify the domain gap and data sufficiency first: measure how far the target domain's data actually is from the source distribution (an embedding-shift statistic, or a task-level performance delta on a few-shot evaluation), and count how much labeled and unlabeled target data is actually available, since the right answer depends heavily on both numbers, not on intuition about how "different" the domain feels.
• Estimate benefit versus cost explicitly: expected performance gain tends to scale with both the domain gap and the (diminishing-returns) log of available data volume, while compute cost scales very differently across the three options, retraining from scratch costs roughly the full model's original pretraining compute, continued pretraining costs a fraction of that, and adapters cost comparatively little; putting a real GPU-hour and dollar estimate against each option turns "which is best" into a concrete comparison.
• Weigh risk and operational factors: the likelihood of catastrophic forgetting scales with how orthogonal the new task or domain is to the base model's original capabilities (and is mitigated by replay or multi-task fine-tuning); adapters reach production fastest, continued pretraining next, full retraining slowest; and a full retrain may trigger a costly re-certification or revalidation process in a regulated setting that adapters would not.
• Decision guidelines: prefer staying frozen and relying on retrieval-augmented prompting when the gap is mainly about missing or fast-changing FACTUAL knowledge rather than a behavioral or stylistic shift, when the underlying corpus changes faster than any retraining cadence could track, or when auditability (being able to cite the exact source behind an answer) matters more than baking the knowledge into the weights, since RAG avoids fine-tuning cost and forgetting risk entirely at the price of retrieval-time latency and dependence on a well-maintained index; prefer fine-tuning adapters when the domain gap is small to moderate, data is limited, time pressure is high, and risk tolerance is low; prefer continued pretraining when the gap is moderate, unlabeled data is abundant, and you need broad improvement across many downstream tasks while still preserving base capabilities; reserve retraining from scratch for a very large domain shift (a genuinely new modality or language), a massive high-quality dataset, or a required architectural change, since anything short of that rarely justifies its cost.
• Validate cheaply before committing: run a linear-probe or few-shot evaluation, a short (one to two epoch) continued-pretraining trial, and a quick adapter prototype against your core success metrics before committing to the full-scale version of whichever option the framework points to, these cheap probes catch a wrong framework-based call before it becomes an expensive mistake.

Worked example: For a moderate domain shift (say, adapting a general assistant to a specialized professional vertical) with a large pool of unlabeled in-domain text but only a modest labeled set, the framework points toward continued pretraining on the unlabeled corpus followed by adapters or LoRA fine-tuning on the labeled data, rather than either a frozen-plus-RAG approach (which would under-use the abundant unlabeled data) or a full retrain (which the domain gap and available compute do not justify). Running a cheap 1-2 epoch continued-pretraining probe first and measuring the delta on a held-out task-relevant evaluation confirms whether that recommendation is actually earning its cost before committing the full compute budget to it.

Trade-offs and pitfalls: The most common mistake is skipping the cheap validation probes and committing straight to the framework's recommended option at full scale, if the domain-gap estimate was wrong, that mistake is expensive to discover only after a full-scale continued-pretraining or retraining run has already consumed most of its budget. A second pitfall specific to regulated or safety-critical deployments is underestimating the re-certification cost a full retrain can trigger, which can dominate the total cost comparison even when the raw compute cost looks similar to continued pretraining.