Model Selection, Tuning, and Generalization Questions

Direct answer
Overfitting is when a model learns patterns specific to its training data (including noise) that don't generalize; underfitting is when a model is too simple to capture the real signal even in the training data. Model capacity is the dial: too much relative to the data and signal available leads to overfitting, too little leads to underfitting.

Structured elaboration
Three observable symptoms of overfitting: (1) training error is very low while validation error is noticeably higher, a persistent gap that doesn't close as training continues; (2) small changes to the training set (different random seed, slightly different sample) produce visibly different model behavior or predictions; (3) the model performs suspiciously well on patterns that shouldn't be learnable from the amount of signal present, e.g. near-perfect accuracy on a genuinely noisy real-world label.
Three observable symptoms of underfitting: (1) both training and validation error are high and close together; (2) the model's predictions look "too smooth" or generic, missing patterns that are visibly present even by eye in the training data; (3) adding capacity (a slightly deeper tree, one more layer, dropping a regularization penalty) meaningfully improves BOTH training and validation performance together, which wouldn't happen if you'd already captured the signal.
Model capacity relates to which one you're facing directly: too much capacity relative to the available data and true signal complexity gives the model room to memorize training-set-specific noise (overfitting); too little capacity means the model literally cannot represent the true relationship regardless of how well it's trained (underfitting).

Worked example
Fit a decision tree with no depth limit to a small tabular dataset: you'll likely see 100% training accuracy and a meaningfully lower validation accuracy, the textbook overfitting signature. Constrain that same tree to max_depth=2 and you may see training accuracy drop to something mediocre while validation accuracy stays close to it, the underfitting signature; the tree is now too simple to represent the interactions that actually matter for this task.

Trade-offs & pitfalls
A single number for training and validation error can be misleading if the validation set is small; always sanity-check with cross-validation before concluding you're clearly in one regime, especially when the gap is small or the dataset itself is small enough that a single split carries meaningful sampling noise.

Direct answer
A strong answer centers on a concrete case where the marginal value of additional tuning experiments had clearly diminished (measured, not assumed), and where the compute/time cost of continuing was better spent elsewhere; the persuasion typically works by making that diminishing-returns evidence visible rather than relying on authority or intuition alone.

Structured elaboration
The story should cover: the initial pressure (a team wanting to keep running more trials, often because "more tuning can only help"), the evidence gathered (a plot of best-score-so-far versus trials run, showing a clear plateau; or a rough cost-per-marginal-improvement calculation), the recommendation made (redirect the compute/time budget elsewhere, e.g. toward data quality or a different modeling approach), and the pushback received (often "we've already invested this much, let's keep going" or a fear of leaving performance on the table).

Worked example
"After 40 hyperparameter trials for a ranking model, the best-score-so-far curve had been flat for the last 15 trials, each new trial's marginal improvement was well within the noise band we'd measured from repeated runs at the same configuration. I proposed redirecting the remaining week's compute budget toward a data-quality investigation instead. The pushback was sunk-cost reasoning: we'd already budgeted the compute, why not use it. I addressed it by showing the plateau plot directly and framing the choice as 'spend this budget where it can still move the needle,' not as wasting what had already been spent."

Trade-offs & pitfalls
Avoid a story where the recommendation to stop was based purely on a hunch that "we've probably found the best config by now"; the interviewer wants to hear about a concrete, visible signal (a plateau, a noise-band comparison) that made the diminishing-returns argument convincing rather than just asserted.

Direct answer
Multi-fidelity methods spend a small, cheap "fidelity" (a few epochs, a data subsample) evaluating many configurations, then progressively promote only the top fraction to the next, more expensive rung; Hyperband/ASHA (Asynchronous Successive Halving) schedule this via successive halving with a chosen halving rate, and BOHB (Bayesian Optimization HyperBand) layers Bayesian optimization on top to make smarter choices about WHICH configurations to try in the first place, rather than choosing them randomly.

Structured elaboration
Successive Halving: start with N configurations at a small resource budget (say 1 epoch each); keep only the top 1/eta fraction (commonly eta=2 or 3), double or triple their resource, repeat, until one (or a small number) survives at the full resource budget. Hyperband runs several such brackets with DIFFERENT starting points on the aggressiveness spectrum (some brackets start with more configurations at lower fidelity, others with fewer configurations at higher fidelity), hedging against not knowing in advance the right balance for a given problem. ASHA (Asynchronous Successive Halving) removes the need to wait for an entire rung to finish before promoting, workers asynchronously promote configurations as soon as enough peers at the current rung have reported, which matters a lot for parallel, heterogeneous compute where some trials finish much slower than others. BOHB replaces Hyperband's random initial configuration sampling with a Bayesian-optimization-guided choice, using the growing pool of (configuration, partial-fidelity result) pairs to propose smarter candidates for the next bracket.
The trade-off: aggressive early pruning (a low eta, meaning you keep only a small fraction each rung) saves the most compute but has the highest risk of discarding a configuration whose learning curve starts slow but eventually overtakes the early leaders (a real phenomenon: some architectures or learning-rate schedules have a slow warm-up before they pull ahead). A less aggressive schedule (higher eta, keeping more configurations longer) hedges against that risk at the cost of spending more compute on ultimately-losing configurations.

Worked example
Tuning 64 configurations with a max budget of 32 epochs and eta=2: successive halving would evaluate all 64 at 1 epoch, keep the top 32, evaluate at 2 epochs, keep the top 16, and so on, reaching a single survivor around 32 epochs, having spent roughly 64x1 + 32x2 + 16x4 + ... epoch-equivalents total, dramatically less than 64x32 if every configuration had been run to completion.

Trade-offs & pitfalls
This mechanism assumes early-fidelity performance is at least weakly predictive of final performance, an assumption that can genuinely fail (e.g. a learning-rate schedule with a long warm-up period, or a regularization technique whose benefit only shows up after many epochs); when you suspect that failure mode for your specific problem, either raise eta (less aggressive pruning) or fall back to Bayesian optimization at full fidelity for the final refinement.

Direct answer
Look at the shape of the gap: if the gap has been shrinking as you've already added data (or a learning curve shows the validation curve still trending down), more data is the better bet; if the gap has been stable in size regardless of data added so far, the problem is more likely capacity, and increasing it is the better bet.

Structured elaboration
The forced-choice framing is really asking you to read the same signal (a learning curve, if you have historical checkpoints at different data sizes) that distinguishes bias from variance. Concretely: plot validation error against the training sizes you've already tried; if it's still falling with a visible slope at your current size, extrapolate, more data likely helps further. If it has flattened, that's an underfitting signature: adding data won't move a curve that's already stopped moving, but adding capacity will let the model represent patterns it currently can't. If you don't have historical checkpoints, a cheap proxy is to train on a random 50% and 100% subsample of your current data right now and compare validation error; a meaningful improvement from the smaller to the larger subsample suggests you're still on the data-hungry part of the curve.

Worked example
If validation error at 33%, 66%, and 100% of your current data (retrospectively, using what you already have) is 0.30, 0.24, 0.21, the still-declining slope says more data is likely to keep helping. If instead it's 0.21, 0.205, 0.20, essentially flat, that's the signal to try more capacity instead.

Trade-offs & pitfalls
This subsampling proxy only tells you about the range of data sizes you've already explored; it can't promise that 10x more data continues the same trend indefinitely, since learning curves themselves eventually flatten (there's a floor set by irreducible noise and how much true signal exists).

Direct answer
Not automatically; a 0.8% aggregate improvement that masks meaningfully worse performance on specific segments is a real risk, especially if those segments matter disproportionately (revenue, fairness, or a vocal user base), and the decision should hinge on segment-level evidence, not just the headline number.

Structured elaboration
The right next step is to break down performance by segment for BOTH the old and new model and look specifically at which segments got better and which got worse, and by how much. If the regressions are concentrated in a small, low-stakes segment while gains are broad, shipping is probably fine. If regressions land on a segment that's disproportionately important (high-value customers, a protected group, a segment already underperforming), the aggregate 0.8% gain likely isn't worth it even though it's a real number.
Additional evidence that would change the answer: the absolute size of the segment-level regression (a 0.1% dip on a segment is very different from a 5% dip), whether the regressing segment is small enough that the aggregate metric barely reflects it (meaning the aggregate gain is coming almost entirely from a different, larger segment), and whether there's a mitigation available (e.g. keep the new model for most segments but fall back to the old one specifically for the regressing segment, if that's operationally feasible).

Worked example
If the 0.8% aggregate gain comes from a 3% improvement on your largest segment (60% of traffic) while a smaller segment (8% of traffic, but your highest-value customers) regresses by 2%, that's a case where the aggregate number is actively hiding a bad trade for the business, and shipping without addressing it would be a mistake even though "accuracy went up."

Trade-offs & pitfalls
Segment-level analysis after the fact is necessary but not sufficient; ideally this kind of subgroup check should be a STANDARD part of your model-comparison process before any ship decision, not something you only think to run when something already feels off.