Generative AI and Large Language Models Questions

Direct answer

Nucleus (top-p) sampling picks the smallest set of highest-probability tokens whose cumulative probability reaches a threshold $p$, then samples from just that set, discarding the unreliable long tail while still adapting to how confident or uncertain the model's distribution is at each step.

Structured elaboration

The algorithm has four stages: (1) convert logits to probabilities with a numerically stable softmax, (2) sort the probabilities in descending order and compute their cumulative sum, (3) find the smallest prefix of that sorted list whose cumulative probability is at least $p$ (the "nucleus"), always keeping at least one token even if the very first token alone already exceeds $p$, and (4) renormalize the nucleus's probabilities to sum to 1 and sample from that reduced distribution. The edge cases worth handling explicitly: $p$ very small (effectively picks only the single highest-probability token, i.e., near-greedy behavior) and $p$ very large or equal to 1.0 (the nucleus covers the whole vocabulary, and the method degenerates to plain multinomial sampling from the original distribution).

import numpy as np

def nucleus_sample(logits: np.ndarray, p: float = 0.9, rng: np.random.Generator = None) -> int:
    """Nucleus (top-p) sampling. logits: 1-D array of raw scores (size V).
    Returns the sampled token index."""
    if rng is None:
        rng = np.random.default_rng()
    logits = np.asarray(logits, dtype=np.float64)
    shifted = logits - np.max(logits)          # numerical stability
    probs = np.exp(shifted)
    probs /= probs.sum()

    order = np.argsort(-probs)                  # descending probability order
    sorted_probs = probs[order]
    cum = np.cumsum(sorted_probs)

    cutoff = np.searchsorted(cum, p, side="left") + 1
    cutoff = max(1, min(cutoff, len(sorted_probs)))  # always keep >=1 token

    nucleus_idx = order[:cutoff]
    nucleus_probs = sorted_probs[:cutoff]
    nucleus_probs = nucleus_probs / nucleus_probs.sum()  # renormalize

    choice = rng.choice(len(nucleus_idx), p=nucleus_probs)
    return int(nucleus_idx[choice])

Complexity. Computing the softmax is $O(V)$ where $V$ is the vocabulary size; sorting the probabilities is $O(V \log V)$, which dominates the overall cost; the cumulative sum and searchsorted lookup are $O(V)$ and $O(\log V)$ respectively. So the whole operation is $O(V \log V)$ per sampling step, dominated by the sort.

Worked example

I verified this implementation in a sandbox against known-expected behavior on three constructed distributions:

• A sharply peaked distribution ([10, 0, 0, 0, 0] as logits, so one token has overwhelmingly higher probability): across 2,000 samples at $p=0.9$, all 2,000 selected the dominant token, since its probability alone already exceeds 0.9 and the nucleus collapses to size 1.
• A uniform distribution ([0, 0, 0, 0, 0] logits over 5 tokens) at $p=1.0$: across 5,000 samples, each of the 5 tokens was chosen roughly 1,000 times (973 to 1,042 in the actual run), matching the expected uniform 1/5 probability, confirming the nucleus correctly covers the full vocabulary and samples proportionally when $p=1.0$.
• The tiny-$p$ edge case ($p=10^{-6}$ on logits [1, 5, 2]): all 50 trials picked index 1, the argmax, confirming the "always keep at least one token" guard produces near-greedy behavior rather than crashing or sampling nothing.

Trade-offs & pitfalls

A common bug is forgetting the "keep at least one token" floor, which can crash or misbehave when a single token's probability already exceeds $p$ on its own (searchsorted would return a cutoff of 0 without the max(1, ...) guard). Another subtlety: nucleus sampling is usually combined with temperature scaling in practice (apply temperature to the logits before computing the softmax), since $p$ alone controls how MANY tokens are eligible but temperature controls how sharply probability concentrates among them; using $p$ without temperature can still produce a poorly calibrated nucleus if the underlying logit distribution is itself miscalibrated.

Situation: At a previous company, our conversational support assistant occasionally gave unhelpful suggestions, and we decided to build a human-in-the-loop preference-collection pipeline to fine-tune it toward more helpful, safer responses.

Task: I led the end-to-end deployment: designing the labeling workflow, writing annotator instructions, building the collection tooling, running the fine-tuning, and executing a staged production rollout with monitoring, balancing speed with cost, latency, and quality trade-offs at every step.

Action: I instrumented the assistant to surface pairs of candidate responses for selected queries, routing a sample to in-house annotators and a small trusted external pool, logging the full context, both candidates, and metadata for each comparison. I wrote a concise rubric emphasizing correctness, clarity, tone, and safety, with worked examples and counterexamples, and a short qualification test with feedback before annotators went live. I built a lightweight labeling UI backed by a task queue and object storage, with quality-control checks built in from day one, gold questions, inter-annotator agreement tracking, and periodic random audits. Once labels accumulated, we aggregated them into a reward model, ran PPO-style fine-tuning on our base model in a staging environment, and rolled out via a staged A/B test (5% traffic, then 25%) with automatic rollback wired to latency, safety-violation rate, and user-satisfaction metrics.

Result: Within six weeks, user-reported helpfulness rose 12%, the safety-incident rate dropped 30%, and offline response relevance improved by 0.18 NDCG. Annotator agreement stabilized at a solid 0.78 Cohen's kappa, giving us confidence the underlying labels were reliable enough to trust.

Trade-offs I made deliberately: To move quickly, I limited the annotator pool to existing in-house staff plus a small trusted external group rather than immediately building broader demographic representation, a real trade-off between speed and representativeness that I flagged explicitly to stakeholders rather than treating as a hidden cost. I also chose a compact rubric over an exhaustive one to speed up labeling, accepting some added annotation noise, which the gold-question checks and larger sample sizes were specifically there to control for. Finally, I deliberately staged the compute investment, validating the reward model and running smaller-scale fine-tuning iterations via A/B tests before committing to a larger, more expensive full RLHF run, so we could catch problems cheaply before scaling the cost.

Direct answer: The KL (Kullback-Leibler divergence) penalty in RLHF measures how far the current policy has drifted from a fixed reference policy (usually the supervised-fine-tuned starting point) and subtracts a term proportional to that divergence from the training objective, which keeps the policy from making large, uncontrolled changes purely to chase reward.

Structured elaboration: Without a KL penalty, a policy-optimization algorithm like PPO can, in principle, keep increasing reward by drifting arbitrarily far from reasonable, fluent language, especially if the reward model has any exploitable blind spot, since nothing in the objective directly penalizes that drift. The KL penalty term, typically $\text{reward} - \beta \, D_{KL}(\pi_\theta \| \pi_{ref})$, directly trades off reward against how different the new policy's output distribution is from the reference model's, so it preserves general language quality and previously-learned behavior even while the policy adapts to the reward signal. In practice the coefficient $\beta$ is tuned adaptively in many implementations: if the measured KL divergence for a batch exceeds a target value, $\beta$ is increased for the next update (pulling the policy back toward the reference), and if KL is comfortably below target, $\beta$ is decreased slightly to allow more room for improvement.

Worked example: If a batch's rollouts show a KL divergence of 0.02 against a target of 0.01, a common adaptive scheme (roughly following the original PPO-KL adaptive-penalty approach) would increase $\beta$, for example multiplying it by 1.5, for the next update; if instead the observed KL were 0.002, well under target, $\beta$ might be reduced, for example halved, to let the policy move more freely while it remains far from the drift limit.

Trade-offs and pitfalls: Setting $\beta$ too high effectively freezes the policy close to its starting point, wasting the RLHF stage's ability to improve on the reward signal at all; setting it too low reintroduces the risk of unconstrained drift and reward hacking that the penalty exists to prevent. Because the reference policy is fixed throughout training, if the supervised-fine-tuned starting point itself has quality issues, the KL penalty will actively work against fixing them, since it specifically discourages moving away from that reference.

Direct answer

For a first production release of a chatbot, retrieval plus a reranker is generally the strongest starting choice among supervised fine-tuning, retrieval+reranking, and RLHF, because it gets you grounded, auditable answers with the least amount of training risk and the fastest iteration loop, while the other two approaches each carry a cost or risk profile that's harder to justify before you have real production signal.

Structured elaboration

Option A: supervised fine-tuning on conversation logs. Requires a substantial labeled dataset of good conversations, has real training cost and time, and the resulting model's answers come entirely from what it "absorbed" during fine-tuning, with no built-in mechanism to ground responses in a verifiable source, so factual errors are harder to trace and fix (you'd need to retrain rather than update a document). Response quality can be very good if the training data is high quality, but data quality and coverage become the single point of failure.

Option B: retrieval plus a reranker. A frozen (or lightly prompted) base LLM is given retrieved passages relevant to the query and generates its answer grounded in them. This needs no model training at all, just a good retrieval index, so it's fast to build and iterate on, its answers are traceable back to a specific document (which matters for both quality debugging and end-user trust), and updating the knowledge base is as simple as updating the index, no retraining required. The main risk shifts entirely to retrieval quality: if the retriever misses the right passage, the answer will be wrong or unsupported no matter how good the generator is.

Option C: RL fine-tuning with human feedback (RLHF). Requires collecting preference data, training a reward model, and running a genuinely nontrivial RL training pipeline (with real risk of reward hacking or instability). It's the most expensive and highest-risk of the three to build correctly, and mainly earns its cost when you need to shape subtle behavioral qualities (tone, helpfulness, refusal behavior) that are hard to specify any other way, not for injecting or updating factual knowledge, which RLHF is not well suited for at all.

Why B for a first release. Compute and data cost are lowest (no training run needed beyond building a retrieval index), monitoring is more interpretable (you can inspect exactly which passages fed which answer), and safety is easier to reason about because ungrounded claims are visibly rarer when the model is prompted to answer from retrieved evidence, though not eliminated; the model can still ignore or misread retrieved passages.

Worked example

Consider a customer-support chatbot for a software product with a large, frequently-updated help-center. Fine-tuning (Option A) would require re-training every time the help docs change, a maintenance burden that scales badly. RLHF (Option C) doesn't even address the core need, correctly answering factual product questions, since it shapes tone and behavior rather than knowledge. Retrieval plus reranking (Option B) lets the team ship an assistant that answers directly from the current help docs, update the index the moment docs change with zero retraining, and trace every wrong answer back to either a retrieval miss (fix the index or the query) or a generation error (fix the prompt), which is a dramatically faster iteration loop for a first release than either training-based alternative.

Trade-offs & pitfalls

This doesn't mean fine-tuning and RLHF are wrong forever, only that they're premature for a FIRST release. Once the retrieval-based system is live and you've accumulated real usage data (what users actually ask, where the base model's tone or behavior falls short even with good retrieved evidence, where retrieval quality is a bottleneck), a natural evolution is to add lightweight fine-tuning for domain-specific behavior, then RLHF-style preference optimization for tone and helpfulness, on top of the retrieval foundation rather than instead of it. The common mistake is reaching for the most sophisticated technique (RLHF) first, out of a sense that it's the "state of the art" approach, when the actual bottleneck for a first release is almost always factual grounding and iteration speed, which retrieval addresses far more directly.

Direct answer: Allocating a fixed GPU-hour budget across continued pretraining, supervised fine-tuning, preference collection, reward-model training, and RLHF should maximize expected human-preference gain per GPU-hour, modeled with an explicit (if approximate) marginal-return function per stage, then validated and corrected with cheap real pilot experiments rather than trusted blindly.

Structured elaboration:

• Objective and formalization: define the goal explicitly as maximizing total incremental preference gain across stages divided by the fixed total GPU-hour budget, subject to the constraint that GPU-hours across all stages sum to the budget, which turns "how should we split the budget" into a concrete constrained-optimization problem rather than an intuition call.
• Approximate marginal-return model per stage: continued pretraining and reward-model training both tend to show strong early gains with sharply diminishing returns (a logarithmic-shaped return curve is a reasonable approximation), supervised fine-tuning and RLHF policy optimization tend to saturate faster once enough structure has been learned (an exponential-saturation shape is a reasonable approximation there); and because reward-model quality and RLHF gains both depend on how many labeled preference pairs exist, those two stages' effective returns should be modeled as scaling with the annotation volume, not treated as independent of it.
• Allocation procedure: fit the model's parameters from historical runs or a small pilot grid search, then solve the constrained allocation (a greedy hill-climb that allocates each additional GPU-hour to whichever stage currently shows the highest estimated marginal return, or a more principled Bayesian-optimization search over the full allocation vector), while respecting minimum viable investments (for example, a reward model needs some minimum labeled-pair count before it is trainable at all, regardless of what the marginal-return curve suggests in isolation).
• Validation and iteration: treat the fitted model's predictions as a prior, not ground truth, run small, cheap proxy experiments (a short continued-pretraining snapshot, a small SFT run, a few annotation-budget variants) specifically to estimate the model's parameters empirically; reserve a genuine exploration fraction of the total budget (for example around 10-15%) to test allocations the current model considers suboptimal, since the model's own estimate could be wrong; and update the marginal-return model with observed real results as they come in, re-solving the allocation as better estimates arrive rather than committing the entire budget to the first fitted plan.

Worked example: If the fitted model initially estimates continued pretraining has the highest marginal return per GPU-hour early in the budget, a hill-climbing allocation would spend GPU-hours there first, but only up to the point where its estimated marginal return, now diminished by the logarithmic-return assumption, drops below what supervised fine-tuning or annotation collection would yield for the same GPU-hour, at which point the allocation shifts; the 10-15% exploration reserve specifically tests whether the model's assumed shape is wrong, for example if continued pretraining's real marginal return diminishes FASTER than the fitted logarithmic curve assumed, that would be visible in the exploration results and should correct the next allocation round rather than being ignored.

Trade-offs and pitfalls: Committing the entire GPU-hour budget to a single allocation plan based purely on the fitted marginal-return model, without any real-world exploration budget reserved, risks over-investing in whichever stage the (necessarily approximate) model happens to favor, a mistake that is expensive to discover only after the budget is spent. A second pitfall is treating annotation and reward-model training as independent of each other in the allocation model, they are coupled (reward-model and RLHF gains both scale with how many preference pairs exist), and modeling them as independent risks under-allocating to annotation relative to what the coupled reality actually requires.