Generative AI and Large Language Models Questions

Direct answer

Designing a tokenizer and vocabulary for 500k multilingual sentences means making an explicit trade-off between vocabulary size, sequence length, and fairness of coverage across languages and scripts, then validating it doesn't silently starve any language of good subword representation before training even starts.

Structured elaboration

Vocabulary-size trade-off. A bigger vocabulary (say 100k+ tokens) lets common words in well-represented languages compress to fewer tokens, shortening sequences and cutting inference cost per unit of text, but it grows the embedding and output-projection matrices (some of the largest parameter blocks in the model) and pushes more tokens into the long tail where they're seen too rarely during training to be well-learned. A smaller vocabulary (say 30k-50k) keeps the embedding matrix compact but produces longer sequences for every language, which raises compute cost under the quadratic scaling of attention and can push some inputs uncomfortably close to the context window limit.

Handling rare scripts. Byte-level SentencePiece is the standard choice here because it guarantees every script is representable (it operates on raw bytes/Unicode, not on a hand-curated per-language character set), avoiding the failure mode where a language absent from your training corpus becomes entirely untokenizable.

Code-mixing. Real multilingual text frequently mixes languages within a single sentence or even a single word (e.g., borrowed technical terms). A subword scheme trained on the full multilingual corpus jointly (rather than one tokenizer per language stitched together) handles this naturally, since it learns subword boundaries from the actual mixed data rather than assuming clean per-language segments.

Balancing coverage across languages. If you build the vocabulary by pure frequency over the raw 500k-sentence corpus, whichever language dominates the corpus by volume will also dominate the vocabulary, and lower-resource languages will get merged less aggressively, ending up needing more tokens per word than the high-resource language even for equally common concepts. The standard mitigation is to upsample or reweight lower-resource languages during vocabulary construction (not necessarily during model training itself) so the merge algorithm sees them often enough to build efficient subwords for them too.

Pre-processing. Unicode normalization (so visually identical characters with different byte representations, like different Unicode forms of an accented letter, don't silently become different tokens) and consistent canonicalization (casing, punctuation handling) need to be decided and applied consistently before vocabulary construction, since inconsistencies here permanently bake inefficient or duplicate subwords into the vocabulary.

Trade-offs & pitfalls

The single most common failure in multilingual tokenizer design is treating it as a purely mechanical step and running the vocabulary-construction algorithm on the raw corpus as-is; without deliberate language reweighting, the resulting tokenizer will quietly be much less efficient for underrepresented languages, which then shows up downstream as those languages needing more tokens (and therefore more cost and more context-window budget) per sentence, and often as somewhat worse model quality in those languages since more of their subwords fall into the poorly-trained long tail of the vocabulary.

Direct answer

Hallucination is when a generative model produces output that is fluent and confident-sounding but not actually grounded in truth or in its source material; it happens fundamentally because the model's training objective rewards producing plausible-sounding continuations, not verified-true ones.

Structured elaboration

Concrete forms it can take. A model can state an outright factual error, confidently asserting something false as if it were true. It can produce a plausible-sounding fabrication, inventing a specific detail, citation, statistic, or event that sounds exactly like the kind of specific detail a correct answer would contain, but that simply does not exist. And it can make an incorrect attribution, correctly stating a real fact but attributing it to the wrong source, person, or context.

Why this happens from a training and objective-function perspective. The model is trained to predict the next token that is most probable given everything it has seen, both during pre-training (predicting the next token of real text) and typically further tuned to produce helpful, fluent, confident-sounding responses. Nothing in that objective directly rewards "say I don't know when uncertain" or "only state things you can verify." A fluent, specific, confident-sounding answer is exactly the kind of continuation the training objective rewards, whether or not it is actually true, because fluency and specificity are also what genuinely correct training-data answers look like. The model has no strong internal signal distinguishing "this specific detail is real" from "this specific detail merely has the surface form of a real one."

Categories of mitigation, at a high level. Grounding techniques anchor the model's output in externally verifiable evidence, for example retrieval-augmented generation or requiring the model to cite a source, rather than relying purely on what it recalls from training. Calibration techniques try to make the model's expressed confidence track its actual likelihood of being correct, so it becomes more willing to hedge or decline when genuinely uncertain instead of always answering in the same confident tone regardless of accuracy. Verification techniques check the model's output after the fact, either automatically by cross-referencing claims against a trusted source, or with a human in the loop, before the output reaches the end user.

Worked example

Consider asking an LLM "who won the Nobel Prize in Physics in a given recent year, and for what work." If the model's training data cut off before that year's announcement, a well-behaved model would say it does not know. A hallucinating model instead often produces a specific, plausible-sounding name and a specific, plausible-sounding citation for the discovery, both of which read exactly like a correct answer would, because the model is pattern-completing "Nobel Prize announcements have this shape" rather than retrieving a verified fact it does not have. Grounding this query in a live retrieval step (searching for the actual announcement and feeding the result into the prompt) directly fixes this specific failure mode, because the model is now completing a pattern anchored in real retrieved text rather than free-associating from a gap in its training data.

Trade-offs & pitfalls

No single mitigation category eliminates hallucination outright. Grounding only helps to the extent the retrieved evidence is itself relevant and correctly used, since the model can still ignore or misread grounded evidence. Calibration is imperfect, and models can still be confidently wrong. Verification adds latency and cost and does not scale to checking every single output in a high-volume product. In practice, teams typically combine categories, for example retrieval-grounded generation plus a lighter-weight automated verification pass, rather than expecting any one technique alone to solve the problem, and treat hallucination rate as a metric to continuously monitor and reduce rather than a bug to be fixed once.

Direct answer: The KL (Kullback-Leibler divergence) penalty in RLHF measures how far the current policy has drifted from a fixed reference policy (usually the supervised-fine-tuned starting point) and subtracts a term proportional to that divergence from the training objective, which keeps the policy from making large, uncontrolled changes purely to chase reward.

Structured elaboration: Without a KL penalty, a policy-optimization algorithm like PPO can, in principle, keep increasing reward by drifting arbitrarily far from reasonable, fluent language, especially if the reward model has any exploitable blind spot, since nothing in the objective directly penalizes that drift. The KL penalty term, typically $\text{reward} - \beta \, D_{KL}(\pi_\theta \| \pi_{ref})$, directly trades off reward against how different the new policy's output distribution is from the reference model's, so it preserves general language quality and previously-learned behavior even while the policy adapts to the reward signal. In practice the coefficient $\beta$ is tuned adaptively in many implementations: if the measured KL divergence for a batch exceeds a target value, $\beta$ is increased for the next update (pulling the policy back toward the reference), and if KL is comfortably below target, $\beta$ is decreased slightly to allow more room for improvement.

Worked example: If a batch's rollouts show a KL divergence of 0.02 against a target of 0.01, a common adaptive scheme (roughly following the original PPO-KL adaptive-penalty approach) would increase $\beta$, for example multiplying it by 1.5, for the next update; if instead the observed KL were 0.002, well under target, $\beta$ might be reduced, for example halved, to let the policy move more freely while it remains far from the drift limit.

Trade-offs and pitfalls: Setting $\beta$ too high effectively freezes the policy close to its starting point, wasting the RLHF stage's ability to improve on the reward signal at all; setting it too low reintroduces the risk of unconstrained drift and reward hacking that the penalty exists to prevent. Because the reference policy is fixed throughout training, if the supervised-fine-tuned starting point itself has quality issues, the KL penalty will actively work against fixing them, since it specifically discourages moving away from that reference.

Direct answer: A good preference-data-collection pipeline pairs a well-designed rater interface (clear side-by-side comparison, a tie option, and calibration checks) with a sampling strategy that prioritizes informative pairs, and layers quality control (gold examples, agreement thresholds) on top so noisy labels do not silently degrade the reward model.

Structured elaboration:

• UI elements: present the prompt/context once, then the two candidate outputs side by side with a prominent choice control and keyboard shortcuts for speed, include an explicit tie or "no preference" option (forcing a choice when raters are genuinely uncertain injects noise), show progress indicators and keep each task batch short (roughly 3-5 pairs) to limit fatigue, and interleave a small number of known gold-labeled pairs so raters get real-time calibration feedback.
• Sampling strategy: start with a stratified baseline across prompt types, model families, and difficulty levels so the dataset has broad coverage, then layer active sampling on top, prioritizing pairs where the current reward model's predicted scores for the two candidates are close together (high uncertainty) or where different candidate-generating models disagree strongly, since these pairs carry the most information per label; keep a deliberate slice (for example around 20%) of purely random sampling so the active-learning loop does not create blind spots by only ever labeling near its own current decision boundary.
• Quality control: insert gold-labeled examples into a percentage of every annotator's task stream (rotated so they cannot be memorized) to calibrate and catch quality drops in real time; use redundancy, multiple independent raters per pair, escalating from a base level (for example 3 raters) to more raters or expert adjudication specifically for pairs where initial agreement is low, rather than applying high redundancy uniformly and wasting labeling budget on easy, unambiguous pairs; require a minimum agreement threshold (for example majority agreement, or Cohen's kappa above a set bar) before a labeled pair is included in reward-model training, with a stricter bar for anything safety-relevant.
• Storage: store each annotation event as an immutable record capturing the prompt, both candidates, the annotator's choice, a tie flag, timing information, which gold/calibration checks applied, and the model versions that generated the candidates, so every training example is fully traceable back to its provenance for later audits or dataset-version comparisons.

Worked example: A concrete cost-quality trade-off: if simple majority-of-3 agreement is only around 60% on a given pair, escalating that specific pair to 5 raters (or an expert adjudicator) rather than escalating the ENTIRE dataset to 5-rater redundancy keeps the average cost per label close to the 3-rater baseline while still resolving the genuinely ambiguous pairs correctly, this adaptive-redundancy approach typically costs far less than uniform high redundancy while catching most of the same disagreement cases, since most pairs are not actually ambiguous.

Trade-offs and pitfalls: Higher uniform redundancy increases label reliability but multiplies cost linearly with every additional rater per pair, which is why adaptive (uncertainty-triggered) redundancy is usually the better default; conversely, setting the agreement threshold for inclusion too low lets noisy, low-consensus labels leak into reward-model training, which is a direct path to the annotator-bias and reward-hacking problems discussed elsewhere in this pipeline. UI choices themselves can introduce bias, for example if candidate A always appears on the left, position bias can creep into the labels, so randomizing left/right placement is not a cosmetic detail but a genuine label-quality control.

Direct answer

Greedy decoding always picks the single highest-probability next token; beam search tracks several candidate sequences in parallel and keeps the best-scoring ones; and sampling methods (temperature, top-k, top-p/nucleus) draw the next token stochastically from the model's probability distribution, with each method shaping that distribution differently. Greedy and beam search are deterministic and favor coherence and correctness; sampling methods trade some coherence for diversity, and are generally preferred whenever you want varied, human-feeling output rather than a single "best" answer.

Structured elaboration

Greedy decoding. At every step, take $\arg\max$ over the next-token distribution. It's the cheapest and fastest method, but it's myopic: a locally best token can lead down a path that turns out worse overall than a path that started with a slightly lower-probability token.

Beam search. Maintain the top-$k$ partial sequences (the "beam") at every step by cumulative log-probability, expanding each and keeping only the best $k$ overall after each step, rather than committing to one token at a time. This finds higher total-probability sequences than greedy in general (it explores more of the search space), at the cost of $k\times$ more compute per step and a tendency to produce generic, "safe" text since it is still hunting for the single most probable sequence.

Temperature scaling. Divide the logits by a temperature $T$ before the softmax. $T < 1$ sharpens the distribution toward the most likely tokens (closer to greedy, more deterministic, less diverse); $T > 1$ flattens it (more diverse, more likely to wander or make mistakes); $T = 1$ leaves the distribution unchanged.

Top-k sampling. Restrict sampling to only the $k$ highest-probability tokens at each step, renormalize, then sample. This prevents sampling an absurdly unlikely token from the long tail, but a fixed $k$ is either too restrictive when the distribution is genuinely flat (many plausible tokens) or too permissive when it's genuinely peaked (few plausible tokens), since $k$ doesn't adapt to how confident the model actually is at each step.

Top-p (nucleus) sampling. Instead of a fixed token count, take the smallest set of tokens whose cumulative probability exceeds $p$, then sample from just that set. This adapts automatically: a peaked distribution yields a small nucleus (close to greedy), a flat distribution yields a larger nucleus (more genuine diversity), which is why nucleus sampling is the more commonly preferred default over top-k in modern generation stacks.

Effects on diversity, determinism, hallucination, repetition. Greedy and low-temperature/low-p decoding produce the most repetitive, deterministic, and "safe" output, and are somewhat less prone to wandering into unsupported claims since the model always follows its most confident path; but low diversity itself can look unnatural or repetitive in open-ended generation. Higher-temperature or higher-p sampling produces more varied, natural-sounding output, at the cost of occasionally sampling a lower-probability, less-supported continuation, which is one of several contributors to hallucination risk.

Rule of thumb. Prefer beam search (or greedy) for tasks with one clear correct answer where you want the single best sequence, such as translation or code generation from a well-specified spec. Prefer top-p/temperature sampling for open-ended, creative, or conversational generation where some variability is desirable and there is no single "correct" continuation.

Trade-offs & pitfalls

A common mistake is using beam search for open-ended chat generation; because beam search hunts for the single highest-probability sequence, it systematically favors bland, generic, high-frequency phrasing over natural, varied language, producing output that reads as repetitive or oddly stilted precisely BECAUSE it's optimizing too hard for likelihood rather than naturalness. The opposite mistake, high-temperature sampling on a task with one correct answer (code generation, math, structured extraction), introduces unnecessary variance and errors into a setting where determinism was actually the right choice.