Generative AI and Large Language Models Questions

Direct answer

Chain-of-thought (CoT) prompting asks the model to produce intermediate reasoning steps before its final answer, rather than jumping straight to a conclusion. It reliably improves accuracy on multi-step problems (arithmetic, logic, multi-hop question answering) because it gives the model more forward computation and more opportunity to catch its own errors along the way, but it comes at a real latency and cost tax, and can occasionally hurt performance or expose incorrect intermediate steps to the user.

Structured elaboration

Example. For "A store has 12 packs of 4 pencils and sells 7 packs. How many pencils remain?", a direct-answer prompt might just output "20". A CoT prompt (e.g., appending "Let's think step by step") elicits something like: "Total pencils = 12 x 4 = 48. Packs sold = 7 packs x 4 pencils = 28 pencils sold. Remaining = 48 - 28 = 20." Both give the right final answer here, but the CoT version is far more likely to catch itself on a harder or more error-prone multi-step problem.

Why it usually improves accuracy. A transformer produces its output token by token, with a fixed amount of computation per token. Skipping straight to a final answer forces the model to do all the reasoning implicitly, in one shot, with no chance to condition later steps on earlier intermediate results. Writing the intermediate steps out gives the model a place to "show its work" that it can then condition on, effectively increasing the amount of sequential computation available for the problem, which especially helps with tasks that genuinely require several dependent steps (arithmetic, multi-hop reasoning).

When it may degrade performance or increase cost. For genuinely simple, single-step tasks, CoT adds tokens (and therefore latency and dollar cost) without adding accuracy, since there's no multi-step reasoning to unpack. It can also occasionally hurt: a model can talk itself into a wrong path during a long reasoning chain and then confidently continue down it, producing a worse answer than a terse, direct one would have. And every CoT token is a token the user (or a downstream system) can read, so an incorrect intermediate step is now visible where a direct wrong answer alone would not have exposed the flawed reasoning.

Trade-offs & pitfalls

The practical discipline is to reserve CoT for tasks that plausibly need multiple dependent reasoning steps, and skip it for tasks that don't, rather than applying it universally as a "make it smarter" default. In production, exposing the full chain of thought to end users also has a UX and trust cost: it lets users see reasoning that might be wrong, verbose, or reveal how the system arrived at a sensitive conclusion, so many products deliberately generate the reasoning but only display the final answer, or summarize the reasoning rather than showing it verbatim.

Direct answer

Beam search keeps the top-$k$ highest cumulative-log-probability partial sequences alive at every decoding step (instead of committing to one token at a time like greedy decoding), which lets it recover from a locally suboptimal choice that greedy would be permanently stuck with; length normalization then corrects the length bias that raw cumulative log-probability introduces against longer sequences.

Structured elaboration

Why raw cumulative log-probability is length-biased. Every additional generated token multiplies the sequence's probability by another factor less than 1 (log-probabilities are always $\le 0$ and keep decreasing as you add tokens), so a raw score comparison systematically favors shorter sequences regardless of quality. Length normalization corrects this by dividing the raw score by a length-dependent penalty term, most commonly the GNMT-style form $\text{lp}(Y) = \frac{(5 + |Y|)^\alpha}{(5 + 1)^\alpha}$, where $\alpha$ (the length_penalty parameter) controls how strongly longer sequences are favored: $\alpha = 0$ removes normalization entirely (raw score wins), and larger $\alpha$ increasingly favors longer completions.

Algorithm. At each step, expand every live beam by its top candidate next tokens, score each resulting candidate sequence by its length-normalized cumulative log-probability, then keep only the global top beam_width candidates across ALL beams (not top-beam_width per beam), pruning the rest. Finished sequences (those that produced an end-of-sequence token) are carried forward as-is rather than expanded further, so they can still be beaten by an unfinished beam that later produces a better completion.

def beam_search(score_fn, beam_width, max_len, length_penalty=1.0, start_token=0):
    """score_fn(prefix_tokens) -> list of log-probs for the next token, one per vocab id.
    Returns (best_sequence, normalized_score)."""
    beams = [([start_token], 0.0, False)]  # (sequence, raw cumulative log-prob, finished)

    def length_norm(raw_score, length):
        lp = ((5 + length) / 6) ** length_penalty
        return raw_score / lp

    for _ in range(max_len):
        if all(finished for _, _, finished in beams):
            break
        candidates = []
        for seq, raw_score, finished in beams:
            if finished:
                candidates.append((seq, raw_score, True))
                continue
            next_logprobs = score_fn(seq)
            top_ids = sorted(range(len(next_logprobs)), key=lambda i: -next_logprobs[i])[:beam_width]
            for tid in top_ids:
                candidates.append((seq + [tid], raw_score + next_logprobs[tid], tid == EOS))
        candidates.sort(key=lambda c: -length_norm(c[1], len(c[0])))
        beams = candidates[:beam_width]

    beams.sort(key=lambda c: -length_norm(c[1], len(c[0])))
    best_seq, best_raw, _ = beams[0]
    return best_seq, length_norm(best_raw, len(best_seq))

Memory for large beams. Memory grows linearly with beam_width (you're keeping that many full candidate sequences and their scores alive simultaneously) and, per step, you generate beam_width x branching_factor candidates before pruning back down, so very large beams on very large vocabularies can become the actual memory bottleneck; production systems bound this by only considering the top few next-token candidates per beam (as the implementation above does) rather than the full vocabulary, and by capping max_len to avoid runaway generation.

Worked example

I verified the escape-the-local-optimum behavior in a sandbox with a small constructed toy language model where the correct global optimum requires NOT taking the locally best first token. After token 0, the toy model scores token 1 at log-prob $-0.1$ (locally best) and token 2 at $-1.0$ (locally worse), but token 1's only continuation is forced into a low-probability end-of-sequence ($-3.0$), giving total $-0.1 + -3.0 = -3.1$, while token 2 opens a much better continuation ending at total $-1.0 + -0.05 + -0.05 = -1.1$.

• Greedy (beam_width=1) can only ever keep the single locally-best token alive, so it is permanently stuck on the $-3.1$ path: [0, 1, EOS].
• Beam search (beam_width=2, length_penalty=0) keeps both token 1 and token 2 alive after the first step, discovers token 2's branch has a better TOTAL score once fully expanded, and correctly returns [0, 2, 1, EOS] with score $-1.1$, strictly better than greedy's $-3.1$.
  Both scores were confirmed by running the implementation and comparing against the hand-computed totals above; they matched exactly.

Trade-offs & pitfalls

A high length-penalty $\alpha$ can overcorrect and start favoring longer, more rambling completions purely because they're longer, not because they're actually better, so $\alpha$ needs to be tuned (typically in the 0.6 to 1.0 range in practice) rather than left at its default. Beam search is also strictly more expensive than greedy (roughly beam_width times the compute and memory per step) for a gain that matters most on tasks with a genuinely correct target sequence (translation, code generation from spec); for open-ended generation, beam search's tendency to find the single highest-probability sequence often produces blander, more repetitive text than sampling-based decoding would, which is why it's rarely used for chat.

Direct answer: Adapter modules are small bottleneck feed-forward blocks (a down-projection, a nonlinearity, and an up-projection with a residual connection) inserted into each transformer layer, most commonly right after the attention output and after the feed-forward network, so that only the adapters (and possibly the final head) need to be trained while the pretrained backbone stays frozen.

Structured elaboration:

• Insertion points: the two most common locations are between the attention block's output and its residual add-and-normalize step, and between the feed-forward network's output and its own residual add-and-normalize step, giving a typical pattern of layer output flowing through the adapter, then a residual add, before the next layer; inserting adapters inside the feed-forward network itself (between its two dense layers) is less common.
• Parameter budget: for hidden size $H$ and adapter bottleneck dimension $r$, one adapter costs roughly $2 H r$ parameters (a down-projection of size $H \times r$ plus an up-projection of size $r \times H$, biases are negligible); with one adapter per layer this scales as $2Hr \times L$ across $L$ layers, or double that with two adapters per layer (after attention and after the feed-forward network). Relative to full fine-tuning, which updates essentially 100% of the model's parameters, adapters typically add only a small fraction of extra parameters while training well under 5% of the model's total parameter count.
• Advantages relative to full fine-tuning: far cheaper storage per task (one small adapter file instead of a full checkpoint), faster training with a smaller memory footprint (far fewer gradients and optimizer states), and structurally safer against catastrophic forgetting since the backbone's weights never move, which also makes multi-task or multi-tenant setups practical by keeping one shared frozen backbone with many swappable adapters.
• Drawbacks: adapters can trail full fine-tuning's peak performance on tasks that genuinely require large representational change, they add a small but real per-layer inference cost (since, unlike LoRA, the nonlinearity between projections generally prevents merging the adapter back into the base weights), the insertion points and bottleneck size need tuning, and stacking multiple adapters (for multi-task use) can introduce interaction effects that need to be validated rather than assumed benign.

Worked example: A lightweight adapter design for sequence classification on a 1.5B-parameter base model: place one adapter after the feed-forward output of each layer (a lighter-weight choice than adapters at both attention and feed-forward positions), with a down-projection to a bottleneck dimension $r$, a ReLU or GeLU nonlinearity, an up-projection back to hidden size $H$, and a residual connection, plus a small task-specific classification head on top of the pooled output. Assuming a hidden size $H \approx 2048$ and $L=24$ layers (typical dimensions for a model in this parameter range): with a lightweight bottleneck of $r=64$ and one adapter per layer, the parameter cost is $2 \times 2048 \times 64 = 262{,}144$ per adapter, so $262{,}144 \times 24 \approx 6.3\text{M}$ total, about $0.42\%$ of the 1.5B base model. With a moderate bottleneck of $r=256$ and two adapters per layer, the cost per layer is $2 \times (2 \times 2048 \times 256) \approx 2.1\text{M}$, so across 24 layers the total is roughly $50.4\text{M}$, about $3.4\%$ of the base model, both estimates verified directly by the arithmetic above.

Trade-offs and pitfalls: Choosing a bottleneck dimension that is too small can under-fit tasks needing more representational change, while choosing it unnecessarily large gives up much of the parameter-efficiency benefit without a clear performance gain, so the practical approach is to start small (for example $r=64$) and increase only if validation performance clearly justifies it. Unlike LoRA, most adapter designs cannot be merged back into the frozen backbone before serving because of the nonlinearity between the two projections, so their small extra inference cost is a permanent, not one-time, overhead, worth weighing against LoRA when the deployment constraint is strict per-request latency.

Direct answer

Prompting an LLM means steering its behavior at inference time through instructions and examples in the input, with no change to the model's weights; fine-tuning changes the weights themselves through additional training. Prompting is the right first approach whenever the base model's existing knowledge and capabilities are already sufficient and you just need to steer HOW it uses them; fine-tuning becomes necessary when the required behavior can't reliably be specified through instructions alone, either because it needs knowledge the model doesn't have, or because it needs a level of consistency prompting can't guarantee.

Structured elaboration

What prompting can and can't do. A prompt can specify style, format, persona, and task framing, and can supply new information directly in context (few-shot examples, retrieved documents). What it cannot do is permanently change what the model "knows" or reliably guarantee a specific behavior across every possible input, since prompt-following is itself a learned, imperfect capability; a sufficiently unusual or adversarial input can still cause the model to ignore or misapply prompt instructions.

When prompting suffices. Style and format control ("respond in JSON," "use a formal tone"), task framing for capabilities the base model already broadly has (summarization, translation, general Q&A), and situations where you can supply the needed facts directly in the prompt (via retrieval) rather than needing them baked into the weights.

When it becomes insufficient. Three concrete triggers: (1) the task requires domain knowledge that's absent or too sparse in the model's pre-training data and too voluminous to supply per-request via retrieval or examples; (2) the required consistency is higher than prompting reliably delivers, e.g., a classification task that must NEVER misfire on adversarial inputs; (3) the per-request cost of achieving the behavior via prompting (long instructions, many few-shot examples) becomes larger than the one-time cost of fine-tuning would be, once volume is high enough.

Trade-offs & pitfalls

The common mistake is reaching for fine-tuning as a default "make the model better at X" move before actually testing whether a well-designed prompt (possibly with retrieval or few-shot examples) already solves the problem; fine-tuning has real fixed costs (data collection, training compute, evaluation, and a slower iteration loop since every behavior change requires retraining) that prompting doesn't. The inverse mistake also happens: sticking with an increasingly baroque, ever-longer prompt to force consistent behavior on a task that has clearly outgrown what prompting can reliably deliver, when a modest fine-tuning investment would produce a shorter, cheaper, more reliable system. The practical discipline is to start with prompting, measure where and how often it fails, and let that concrete failure pattern justify the fine-tuning investment rather than assuming it up front.

Direct answer

Perplexity is a standard intrinsic measure of how well a language model predicts a sequence of text: it is the exponentiated average negative log-likelihood the model assigns to the correct tokens, so lower perplexity means the model was, on average, less "surprised" by the actual text. As an evaluation metric it has real limitations for open-ended generation, since a model can achieve low perplexity while still producing outputs that are repetitive, factually wrong, or simply not what a person actually wants.

Structured elaboration

The relationship to negative log-likelihood. For a sequence of $N$ tokens, the average negative log-likelihood (NLL) per token is $\text{NLL} = -\frac{1}{N}\sum_{i=1}^{N} \log P(x_i \mid x_{<i})$, the average of the log-probability the model assigned to each actual next token, negated so that a confidently correct prediction contributes a small (near-zero) value and a confidently wrong prediction contributes a large value. Perplexity is simply $\text{Perplexity} = e^{\text{NLL}}$ (exponentiating the average NLL), which converts the metric back onto a more intuitive scale: a perplexity of $k$ can be read as "the model was, on average, about as uncertain as if it were choosing uniformly among $k$ equally likely options at each step."

Limitations as an evaluation metric for open-ended generation. Perplexity only measures how well the model predicts a GIVEN reference text; it says nothing about the quality, correctness, or diversity of text the model would actually GENERATE on its own, since open-ended generation has no single correct continuation. A model can achieve excellent perplexity on held-out text while still being prone to hallucination, repetition, or dull generic phrasing at generation time, none of which perplexity directly penalizes. Perplexity is also sensitive to tokenization and vocabulary choices, so raw perplexity scores are not directly comparable across models that use different tokenizers.

Complementary metrics and human evaluation. Because perplexity says nothing about generation-time behavior, it is typically paired with task-specific automatic metrics (exact-match or F1 for question answering, BLEU or ROUGE for summarization or translation, which compare generated text against a reference) and, for genuinely open-ended tasks with no single correct answer, human evaluation is often essential, since no automatic metric fully captures qualities like helpfulness, coherence, or factual correctness that people actually care about.

Worked example

I verified the perplexity relationship in a sandbox with three constructed cases against known-expected outputs.

• A "perfect" model that puts overwhelming probability mass on the correct target token at every position of a 4-token sequence: the computed negative log-likelihood was 0 and perplexity was exactly 1.0, matching the definition, since a model that is always certain and always correct has, by construction, zero average surprise.
• A uniform model over a vocabulary of 8 tokens (softmax of all-zero logits, so every token is equally likely): perplexity of a uniform distribution over $V$ outcomes should analytically be exactly $V$. The computed perplexity was exactly 8.0 for both sequences in the batch, matching the analytical value exactly.
• A masking test: a padded position was deliberately given a badly WRONG prediction (a huge logit on the wrong token), but the position was excluded via the mask; the computed perplexity ignored it entirely and returned 1.0, matching only the two correctly-predicted, unmasked positions, while the SAME batch computed WITHOUT masking blew up to an enormous perplexity (roughly $9\times10^{28}$), confirming the masking logic is actually load-bearing and not a no-op.

import numpy as np

def sequence_perplexity(logits, target_ids, mask=None):
    B, T, V = logits.shape
    if mask is None:
        mask = np.ones((B, T), dtype=np.float64)
    mask = mask.astype(np.float64)
    shifted = logits - logits.max(axis=-1, keepdims=True)
    logsumexp = np.log(np.exp(shifted).sum(axis=-1, keepdims=True))
    logprobs = shifted - logsumexp
    b_idx = np.arange(B)[:, None]; t_idx = np.arange(T)[None, :]
    target_logprobs = logprobs[b_idx, t_idx, target_ids] * mask
    nll_per_seq = -target_logprobs.sum(axis=1)
    token_counts = np.maximum(mask.sum(axis=1), 1e-9)
    avg_nll = nll_per_seq / token_counts
    return nll_per_seq, np.exp(avg_nll)

Trade-offs & pitfalls

A common mistake is treating perplexity as a general-purpose quality score and using it to compare two models that use different tokenizers; because perplexity is computed per TOKEN, and different tokenizers segment the same text into different numbers of tokens, the same underlying text quality can produce different perplexity numbers purely due to tokenization differences, not genuine quality differences. Perplexity remains genuinely useful as a fast, cheap, reference-free signal during pre-training and for comparing checkpoints of the SAME model family with the SAME tokenizer over training, which is exactly what it was designed for, but it should not be the sole metric used to judge or compare generation quality across different model families or against human expectations.