Generative AI and Large Language Models Questions

Direct answer

Greedy decoding always picks the single highest-probability next token; beam search tracks several candidate sequences in parallel and keeps the best-scoring ones; and sampling methods (temperature, top-k, top-p/nucleus) draw the next token stochastically from the model's probability distribution, with each method shaping that distribution differently. Greedy and beam search are deterministic and favor coherence and correctness; sampling methods trade some coherence for diversity, and are generally preferred whenever you want varied, human-feeling output rather than a single "best" answer.

Structured elaboration

Greedy decoding. At every step, take $\arg\max$ over the next-token distribution. It's the cheapest and fastest method, but it's myopic: a locally best token can lead down a path that turns out worse overall than a path that started with a slightly lower-probability token.

Beam search. Maintain the top-$k$ partial sequences (the "beam") at every step by cumulative log-probability, expanding each and keeping only the best $k$ overall after each step, rather than committing to one token at a time. This finds higher total-probability sequences than greedy in general (it explores more of the search space), at the cost of $k\times$ more compute per step and a tendency to produce generic, "safe" text since it is still hunting for the single most probable sequence.

Temperature scaling. Divide the logits by a temperature $T$ before the softmax. $T < 1$ sharpens the distribution toward the most likely tokens (closer to greedy, more deterministic, less diverse); $T > 1$ flattens it (more diverse, more likely to wander or make mistakes); $T = 1$ leaves the distribution unchanged.

Top-k sampling. Restrict sampling to only the $k$ highest-probability tokens at each step, renormalize, then sample. This prevents sampling an absurdly unlikely token from the long tail, but a fixed $k$ is either too restrictive when the distribution is genuinely flat (many plausible tokens) or too permissive when it's genuinely peaked (few plausible tokens), since $k$ doesn't adapt to how confident the model actually is at each step.

Top-p (nucleus) sampling. Instead of a fixed token count, take the smallest set of tokens whose cumulative probability exceeds $p$, then sample from just that set. This adapts automatically: a peaked distribution yields a small nucleus (close to greedy), a flat distribution yields a larger nucleus (more genuine diversity), which is why nucleus sampling is the more commonly preferred default over top-k in modern generation stacks.

Effects on diversity, determinism, hallucination, repetition. Greedy and low-temperature/low-p decoding produce the most repetitive, deterministic, and "safe" output, and are somewhat less prone to wandering into unsupported claims since the model always follows its most confident path; but low diversity itself can look unnatural or repetitive in open-ended generation. Higher-temperature or higher-p sampling produces more varied, natural-sounding output, at the cost of occasionally sampling a lower-probability, less-supported continuation, which is one of several contributors to hallucination risk.

Rule of thumb. Prefer beam search (or greedy) for tasks with one clear correct answer where you want the single best sequence, such as translation or code generation from a well-specified spec. Prefer top-p/temperature sampling for open-ended, creative, or conversational generation where some variability is desirable and there is no single "correct" continuation.

Trade-offs & pitfalls

A common mistake is using beam search for open-ended chat generation; because beam search hunts for the single highest-probability sequence, it systematically favors bland, generic, high-frequency phrasing over natural, varied language, producing output that reads as repetitive or oddly stilted precisely BECAUSE it's optimizing too hard for likelihood rather than naturalness. The opposite mistake, high-temperature sampling on a task with one correct answer (code generation, math, structured extraction), introduces unnecessary variance and errors into a setting where determinism was actually the right choice.

Direct answer

A discriminative model learns the conditional distribution $P(y \mid x)$ directly: it maps an input straight to a label or decision boundary. A generative model learns (explicitly or implicitly) how the data itself was produced, either the joint $P(x, y)$ or the marginal $P(x)$. Because it models how data is generated, a generative model can also sample new data, something a discriminative model cannot do at all.

Structured elaboration

What each optimizes. A discriminative classifier like logistic regression or an SVM fits a boundary that separates classes as well as possible, using only the conditional likelihood of the label given the input. A generative model like a naive Bayes classifier, an autoregressive language model, a GAN, or a VAE instead tries to capture the full data-generating process, then derives $P(y\mid x)$ (if it needs to classify at all) via Bayes' rule from $P(x\mid y)$ and $P(y)$.

Concrete examples.

• Discriminative: logistic regression for spam detection, an SVM for image classification, a discriminative fine-tuned BERT for sentiment classification.
• Generative: an autoregressive LLM producing text token by token, a GAN or diffusion model producing images, a VAE reconstructing and sampling from a learned latent space.

Three practical implications of the choice.

1. Only a generative model can produce new content. If the product need is "write an email" or "draw an image," you need a generative model; a classifier cannot do this by construction.
2. Discriminative models are usually more sample-efficient and accurate for pure classification. They spend their entire capacity on the decision boundary instead of also modeling the full input distribution, so with a fixed dataset size a discriminative model typically classifies better.
3. Generative models carry model-misspecification risk that discriminative models mostly avoid. If your assumptions about how the data is generated are wrong (e.g., a bad choice of prior or emission distribution), a generative model's classification accuracy degrades along with its generative quality. A discriminative model that never modeled the generative process has no such assumption to get wrong.

Worked example

Take spam detection. A discriminative logistic regression sees an email's features (word counts, sender domain, links) and directly outputs $P(\text{spam}\mid \text{features})$; it has no notion of what a "typical" spam email looks like beyond the boundary it fit. A generative naive Bayes classifier instead estimates $P(\text{word}\mid \text{spam})$ and $P(\text{word}\mid \text{not spam})$ for every word, then combines them with Bayes' rule to classify a new email. Because it explicitly modeled $P(x\mid y)$, that same naive Bayes model could (crudely) generate a plausible spam-sounding word distribution, while the logistic regression has no way to do so; it only ever learned where the boundary sits.

Trade-offs & pitfalls

The two are not always in tension: a generative classifier CAN outperform a discriminative one when labeled data is scarce, because it borrows statistical strength from modeling the unlabeled input distribution too. But when you have plenty of labeled data and only need a decision, the extra generative modeling is often wasted compute and can hurt calibration if the generative assumptions are even slightly wrong. The common pitfall is picking a generative model "because it's more powerful" for a task that is purely classification; if the product only ever needs a label, a well-tuned discriminative model is usually cheaper to train, faster to serve, and more accurate.

Direct answer

Chain-of-thought (CoT) prompting asks the model to produce intermediate reasoning steps before its final answer, rather than jumping straight to a conclusion. It reliably improves accuracy on multi-step problems (arithmetic, logic, multi-hop question answering) because it gives the model more forward computation and more opportunity to catch its own errors along the way, but it comes at a real latency and cost tax, and can occasionally hurt performance or expose incorrect intermediate steps to the user.

Structured elaboration

Example. For "A store has 12 packs of 4 pencils and sells 7 packs. How many pencils remain?", a direct-answer prompt might just output "20". A CoT prompt (e.g., appending "Let's think step by step") elicits something like: "Total pencils = 12 x 4 = 48. Packs sold = 7 packs x 4 pencils = 28 pencils sold. Remaining = 48 - 28 = 20." Both give the right final answer here, but the CoT version is far more likely to catch itself on a harder or more error-prone multi-step problem.

Why it usually improves accuracy. A transformer produces its output token by token, with a fixed amount of computation per token. Skipping straight to a final answer forces the model to do all the reasoning implicitly, in one shot, with no chance to condition later steps on earlier intermediate results. Writing the intermediate steps out gives the model a place to "show its work" that it can then condition on, effectively increasing the amount of sequential computation available for the problem, which especially helps with tasks that genuinely require several dependent steps (arithmetic, multi-hop reasoning).

When it may degrade performance or increase cost. For genuinely simple, single-step tasks, CoT adds tokens (and therefore latency and dollar cost) without adding accuracy, since there's no multi-step reasoning to unpack. It can also occasionally hurt: a model can talk itself into a wrong path during a long reasoning chain and then confidently continue down it, producing a worse answer than a terse, direct one would have. And every CoT token is a token the user (or a downstream system) can read, so an incorrect intermediate step is now visible where a direct wrong answer alone would not have exposed the flawed reasoning.

Trade-offs & pitfalls

The practical discipline is to reserve CoT for tasks that plausibly need multiple dependent reasoning steps, and skip it for tasks that don't, rather than applying it universally as a "make it smarter" default. In production, exposing the full chain of thought to end users also has a UX and trust cost: it lets users see reasoning that might be wrong, verbose, or reveal how the system arrived at a sensitive conclusion, so many products deliberately generate the reasoning but only display the final answer, or summarize the reasoning rather than showing it verbatim.