Generative AI and Large Language Models Questions

Direct answer

The three common LLM pre-training objectives are next-token prediction (causal language modeling), masked language modeling, and denoising/span-corruption objectives. Each defines a different self-supervised task on the same unlabeled text, and the choice of objective shapes what the resulting model is naturally good at. For a model meant for free-form generation and chat, causal next-token prediction is the right choice because it is the only one of the three that matches how the model will actually be used at inference: producing text one token at a time without seeing the future.

Structured elaboration

Next-token prediction (causal LM). The model sees tokens $1..t-1$ and predicts token $t$, trained with a causal mask so it never sees future tokens. This directly trains the model on the exact task it performs at inference time in a chat setting: generate the next token given everything so far. Weakness: it only ever sees left context during training, so it is somewhat less data-efficient at extracting bidirectional signal than masked objectives.

Masked language modeling (MLM). Random tokens in the input are replaced with a mask token, and the model predicts the missing tokens using context from BOTH directions. This is excellent for building rich representations for understanding tasks (classification, extraction) because every prediction can use the full sentence. Weakness: there's a training/inference mismatch for generation, since MLM never trains the model to produce a sequence left-to-right without seeing it in full first, so MLM-trained models are not naturally suited to open-ended generation.

Denoising / span-corruption (seq2seq). A contiguous span of text is corrupted (masked, shuffled, or dropped) and the model must reconstruct or generate the correct span, typically in an encoder-decoder setup (T5-style). This sits between the other two: it teaches the model to produce coherent, possibly multi-token spans, so it transfers reasonably well to generation and to structured input-to-output tasks (summarization, translation). Weakness: it is architecturally more complex to serve (encoder-decoder rather than a single stack) and less of a drop-in match for simple autoregressive chat.

Why causal LM wins for chat. A chat assistant must generate a reply token by token without access to future tokens (they don't exist yet), and it must be able to keep going indefinitely. Causal next-token prediction trains the model on precisely this generation pattern from day one, so there is no train/inference mismatch; MLM and span-corruption both introduce some mismatch between what the model was trained to do and what it will be asked to do at serving time.

Trade-offs & pitfalls

It is tempting to think MLM's bidirectional context makes it "strictly better" because it uses more information per prediction; that is true for understanding tasks but irrelevant, even actively unhelpful, for generation, where the model must learn to work with incomplete (left-only) context because that is genuinely all it will have at inference. The pitfall is picking a pre-training objective based on which one "uses more information" rather than which one matches the actual downstream usage pattern; a masked-LM-pretrained model asked to generate open-ended text will behave worse than a causally-pretrained one of the same size, because it never practiced generating under the constraint it will face in production.

Direct answer

Prompting an LLM means steering its behavior at inference time through instructions and examples in the input, with no change to the model's weights; fine-tuning changes the weights themselves through additional training. Prompting is the right first approach whenever the base model's existing knowledge and capabilities are already sufficient and you just need to steer HOW it uses them; fine-tuning becomes necessary when the required behavior can't reliably be specified through instructions alone, either because it needs knowledge the model doesn't have, or because it needs a level of consistency prompting can't guarantee.

Structured elaboration

What prompting can and can't do. A prompt can specify style, format, persona, and task framing, and can supply new information directly in context (few-shot examples, retrieved documents). What it cannot do is permanently change what the model "knows" or reliably guarantee a specific behavior across every possible input, since prompt-following is itself a learned, imperfect capability; a sufficiently unusual or adversarial input can still cause the model to ignore or misapply prompt instructions.

When prompting suffices. Style and format control ("respond in JSON," "use a formal tone"), task framing for capabilities the base model already broadly has (summarization, translation, general Q&A), and situations where you can supply the needed facts directly in the prompt (via retrieval) rather than needing them baked into the weights.

When it becomes insufficient. Three concrete triggers: (1) the task requires domain knowledge that's absent or too sparse in the model's pre-training data and too voluminous to supply per-request via retrieval or examples; (2) the required consistency is higher than prompting reliably delivers, e.g., a classification task that must NEVER misfire on adversarial inputs; (3) the per-request cost of achieving the behavior via prompting (long instructions, many few-shot examples) becomes larger than the one-time cost of fine-tuning would be, once volume is high enough.

Trade-offs & pitfalls

The common mistake is reaching for fine-tuning as a default "make the model better at X" move before actually testing whether a well-designed prompt (possibly with retrieval or few-shot examples) already solves the problem; fine-tuning has real fixed costs (data collection, training compute, evaluation, and a slower iteration loop since every behavior change requires retraining) that prompting doesn't. The inverse mistake also happens: sticking with an increasingly baroque, ever-longer prompt to force consistent behavior on a task that has clearly outgrown what prompting can reliably deliver, when a modest fine-tuning investment would produce a shorter, cheaper, more reliable system. The practical discipline is to start with prompting, measure where and how often it fails, and let that concrete failure pattern justify the fine-tuning investment rather than assuming it up front.

Direct answer

Few-shot in-context learning has four well-documented practical limitations: it is bounded by the model's context length, it is sensitive to the order the examples are presented in, it can leak the wrong label pattern if examples are poorly chosen (label leakage / spurious correlation), and it can fail when the distribution of the demonstrations doesn't match the distribution of real test inputs. Each of these has a concrete production mitigation.

Structured elaboration

Context-length constraints. Every example consumes tokens, so there's a hard ceiling on how many demonstrations you can include, and long examples crowd out either more examples or the actual query and instructions. Mitigation: keep demonstrations short and information-dense, or retrieve only the most relevant few examples per query dynamically instead of stuffing a fixed large set into every prompt.

Sensitivity to example order. The same set of examples in a different order can meaningfully change model accuracy, sometimes by double-digit percentage points, because the model is still, fundamentally, predicting the next token conditioned on everything that came before, so recency and position matter more than they intuitively should. Mitigation: test multiple orderings during prompt development and pick (or ensemble across) the ones that are stable, and avoid ordering examples in a way that correlates trivially with the label (e.g., all "positive" examples first).

Label leakage. If the demonstrations have a spurious surface pattern that correlates with the label (e.g., all positive-sentiment examples happen to be longer sentences), the model can learn that surface shortcut instead of the actual task, and perform poorly the moment a real input breaks the pattern. Mitigation: deliberately balance and randomize examples so no superficial feature (length, punctuation, which examples appear first) correlates with the label.

Distributional mismatch between demonstrations and test inputs. If your example set doesn't represent the actual variety of inputs you'll see in production (different phrasing, edge cases, adversarial inputs), the model's in-context "understanding" of the task will be narrower than what production actually demands. Mitigation: continuously sample real production inputs (including near-misses and failures) back into your example pool, rather than freezing a hand-picked set once at launch.

Trade-offs & pitfalls

These four limitations compound: a team that picks a fixed, small, hand-curated example set and never revisits it is simultaneously exposed to order sensitivity (never re-tested), label leakage (never audited for spurious patterns), and distributional mismatch (never refreshed against real traffic). The practical discipline that addresses all four at once is treating your few-shot example set as a live artifact, versioned, periodically re-evaluated against a held-out sample of real queries, and refreshed as production traffic reveals gaps, rather than a one-time prompt-engineering decision made at launch and left alone.

Direct answer

Beam search keeps the top-$k$ highest cumulative-log-probability partial sequences alive at every decoding step (instead of committing to one token at a time like greedy decoding), which lets it recover from a locally suboptimal choice that greedy would be permanently stuck with; length normalization then corrects the length bias that raw cumulative log-probability introduces against longer sequences.

Structured elaboration

Why raw cumulative log-probability is length-biased. Every additional generated token multiplies the sequence's probability by another factor less than 1 (log-probabilities are always $\le 0$ and keep decreasing as you add tokens), so a raw score comparison systematically favors shorter sequences regardless of quality. Length normalization corrects this by dividing the raw score by a length-dependent penalty term, most commonly the GNMT-style form $\text{lp}(Y) = \frac{(5 + |Y|)^\alpha}{(5 + 1)^\alpha}$, where $\alpha$ (the length_penalty parameter) controls how strongly longer sequences are favored: $\alpha = 0$ removes normalization entirely (raw score wins), and larger $\alpha$ increasingly favors longer completions.

Algorithm. At each step, expand every live beam by its top candidate next tokens, score each resulting candidate sequence by its length-normalized cumulative log-probability, then keep only the global top beam_width candidates across ALL beams (not top-beam_width per beam), pruning the rest. Finished sequences (those that produced an end-of-sequence token) are carried forward as-is rather than expanded further, so they can still be beaten by an unfinished beam that later produces a better completion.

def beam_search(score_fn, beam_width, max_len, length_penalty=1.0, start_token=0):
    """score_fn(prefix_tokens) -> list of log-probs for the next token, one per vocab id.
    Returns (best_sequence, normalized_score)."""
    beams = [([start_token], 0.0, False)]  # (sequence, raw cumulative log-prob, finished)

    def length_norm(raw_score, length):
        lp = ((5 + length) / 6) ** length_penalty
        return raw_score / lp

    for _ in range(max_len):
        if all(finished for _, _, finished in beams):
            break
        candidates = []
        for seq, raw_score, finished in beams:
            if finished:
                candidates.append((seq, raw_score, True))
                continue
            next_logprobs = score_fn(seq)
            top_ids = sorted(range(len(next_logprobs)), key=lambda i: -next_logprobs[i])[:beam_width]
            for tid in top_ids:
                candidates.append((seq + [tid], raw_score + next_logprobs[tid], tid == EOS))
        candidates.sort(key=lambda c: -length_norm(c[1], len(c[0])))
        beams = candidates[:beam_width]

    beams.sort(key=lambda c: -length_norm(c[1], len(c[0])))
    best_seq, best_raw, _ = beams[0]
    return best_seq, length_norm(best_raw, len(best_seq))

Memory for large beams. Memory grows linearly with beam_width (you're keeping that many full candidate sequences and their scores alive simultaneously) and, per step, you generate beam_width x branching_factor candidates before pruning back down, so very large beams on very large vocabularies can become the actual memory bottleneck; production systems bound this by only considering the top few next-token candidates per beam (as the implementation above does) rather than the full vocabulary, and by capping max_len to avoid runaway generation.

Worked example

I verified the escape-the-local-optimum behavior in a sandbox with a small constructed toy language model where the correct global optimum requires NOT taking the locally best first token. After token 0, the toy model scores token 1 at log-prob $-0.1$ (locally best) and token 2 at $-1.0$ (locally worse), but token 1's only continuation is forced into a low-probability end-of-sequence ($-3.0$), giving total $-0.1 + -3.0 = -3.1$, while token 2 opens a much better continuation ending at total $-1.0 + -0.05 + -0.05 = -1.1$.

• Greedy (beam_width=1) can only ever keep the single locally-best token alive, so it is permanently stuck on the $-3.1$ path: [0, 1, EOS].
• Beam search (beam_width=2, length_penalty=0) keeps both token 1 and token 2 alive after the first step, discovers token 2's branch has a better TOTAL score once fully expanded, and correctly returns [0, 2, 1, EOS] with score $-1.1$, strictly better than greedy's $-3.1$.
  Both scores were confirmed by running the implementation and comparing against the hand-computed totals above; they matched exactly.

Trade-offs & pitfalls

A high length-penalty $\alpha$ can overcorrect and start favoring longer, more rambling completions purely because they're longer, not because they're actually better, so $\alpha$ needs to be tuned (typically in the 0.6 to 1.0 range in practice) rather than left at its default. Beam search is also strictly more expensive than greedy (roughly beam_width times the compute and memory per step) for a gain that matters most on tasks with a genuinely correct target sequence (translation, code generation from spec); for open-ended generation, beam search's tendency to find the single highest-probability sequence often produces blander, more repetitive text than sampling-based decoding would, which is why it's rarely used for chat.

Direct answer

For a first production release of a chatbot, retrieval plus a reranker is generally the strongest starting choice among supervised fine-tuning, retrieval+reranking, and RLHF, because it gets you grounded, auditable answers with the least amount of training risk and the fastest iteration loop, while the other two approaches each carry a cost or risk profile that's harder to justify before you have real production signal.

Structured elaboration

Option A: supervised fine-tuning on conversation logs. Requires a substantial labeled dataset of good conversations, has real training cost and time, and the resulting model's answers come entirely from what it "absorbed" during fine-tuning, with no built-in mechanism to ground responses in a verifiable source, so factual errors are harder to trace and fix (you'd need to retrain rather than update a document). Response quality can be very good if the training data is high quality, but data quality and coverage become the single point of failure.

Option B: retrieval plus a reranker. A frozen (or lightly prompted) base LLM is given retrieved passages relevant to the query and generates its answer grounded in them. This needs no model training at all, just a good retrieval index, so it's fast to build and iterate on, its answers are traceable back to a specific document (which matters for both quality debugging and end-user trust), and updating the knowledge base is as simple as updating the index, no retraining required. The main risk shifts entirely to retrieval quality: if the retriever misses the right passage, the answer will be wrong or unsupported no matter how good the generator is.

Option C: RL fine-tuning with human feedback (RLHF). Requires collecting preference data, training a reward model, and running a genuinely nontrivial RL training pipeline (with real risk of reward hacking or instability). It's the most expensive and highest-risk of the three to build correctly, and mainly earns its cost when you need to shape subtle behavioral qualities (tone, helpfulness, refusal behavior) that are hard to specify any other way, not for injecting or updating factual knowledge, which RLHF is not well suited for at all.

Why B for a first release. Compute and data cost are lowest (no training run needed beyond building a retrieval index), monitoring is more interpretable (you can inspect exactly which passages fed which answer), and safety is easier to reason about because ungrounded claims are visibly rarer when the model is prompted to answer from retrieved evidence, though not eliminated; the model can still ignore or misread retrieved passages.

Worked example

Consider a customer-support chatbot for a software product with a large, frequently-updated help-center. Fine-tuning (Option A) would require re-training every time the help docs change, a maintenance burden that scales badly. RLHF (Option C) doesn't even address the core need, correctly answering factual product questions, since it shapes tone and behavior rather than knowledge. Retrieval plus reranking (Option B) lets the team ship an assistant that answers directly from the current help docs, update the index the moment docs change with zero retraining, and trace every wrong answer back to either a retrieval miss (fix the index or the query) or a generation error (fix the prompt), which is a dramatically faster iteration loop for a first release than either training-based alternative.

Trade-offs & pitfalls

This doesn't mean fine-tuning and RLHF are wrong forever, only that they're premature for a FIRST release. Once the retrieval-based system is live and you've accumulated real usage data (what users actually ask, where the base model's tone or behavior falls short even with good retrieved evidence, where retrieval quality is a bottleneck), a natural evolution is to add lightweight fine-tuning for domain-specific behavior, then RLHF-style preference optimization for tone and helpfulness, on top of the retrieval foundation rather than instead of it. The common mistake is reaching for the most sophisticated technique (RLHF) first, out of a sense that it's the "state of the art" approach, when the actual bottleneck for a first release is almost always factual grounding and iteration speed, which retrieval addresses far more directly.