Model Selection, Tuning, and Generalization Questions

Direct answer
Pure L1 tends to pick one feature arbitrarily out of a correlated group and zero out the rest, which is unstable when you genuinely need several correlated-but-relevant features to stay in the model; Elastic Net's L2 component spreads weight across the correlated group instead, while its L1 component still zeroes out the roughly 9,950 irrelevant features, giving you sparsity where you want it and stability where you need it.

Structured elaboration
With 10,000 features and only ~50 truly predictive, you want aggressive sparsity, which argues for L1. But if some of those 50 are correlated with each other (a common real-world situation: several engineered features capturing overlapping signal), pure L1's tendency to arbitrarily select one representative from a correlated cluster and zero the rest can make your selected feature set unstable across resamples or CV folds, one run keeps feature A, another keeps its near-duplicate feature B. Elastic Net's mixing parameter (l1_ratio) lets you dial in mostly-L1 behavior (aggressive sparsity for the ~9,950 irrelevant features) while retaining enough L2 to keep correlated relevant features together rather than arbitrarily dropping some of them.

Worked example
Consider 5 near-duplicate engineered features all correlated at 0.9+ with each other, all genuinely predictive. Pure L1 (l1_ratio=1) might keep just one of the five and zero the other four in any given fit, and which one survives can flip between CV folds. Elastic Net with l1_ratio=0.5-0.7 tends to keep all five with smaller, comparable weights, giving a more stable and arguably more honest picture of "this whole cluster of features matters."

Trade-offs & pitfalls
Elastic Net adds a second hyperparameter (l1_ratio) to tune alongside the overall strength (alpha), roughly doubling your search space compared to pure L1 or L2; for a first pass, a coarse grid over l1_ratio in {0.1, 0.5, 0.9} combined with a log-scale sweep over alpha usually finds a reasonable region before finer tuning.

Direct answer
Set up a synthetic dataset with a known data-generating process (so you can measure a TRUE held-out error on a genuinely independent large sample), then compare that ground truth against (a) a naive approach that tunes on one train/val split and reports the val score as if it were the generalization estimate, and (b) nested cross-validation on the same tuning data; the naive approach's reported score comes out noticeably more optimistic than its own true error, while nested CV tracks the true error much more closely.

Structured elaboration
The key experimental design choice: generate MORE data than you'll actually use for tuning, and hold a large independent slice aside purely as ground truth, never touched by either method's tuning process. Both methods only ever see the smaller "development" pool; the large held-out slice exists purely so you, the experimenter, can measure each method's TRUE error and quantify the gap between what each method reported and what was actually true.

Worked example (executed)
On a synthetic regression task (1,200 samples total; 400 held out as ground truth, 800 used as the "development" pool both methods tune against): the naive single-split approach reported a validation MSE of 453.7 as its generalization estimate, while its chosen configuration's TRUE error against the independent 400-sample ground-truth set was 466.7, an optimism gap of about 13 MSE points (roughly 3%). Nested CV run on the same 800-sample development pool reported a generalization estimate of 439.7, which is much closer in absolute terms to being a defensible estimate of the same ~466.7 true error, without having peeked at the ground-truth set at all. Note the naive method's gap here is modest because this particular synthetic setup and grid are not adversarially large, in a real project with many more candidate configurations searched, the optimism gap tends to grow.

Trade-offs & pitfalls
The SIZE of the optimism gap is itself dataset- and search-space-dependent, this experiment demonstrates the DIRECTION and mechanism reliably (naive tuning is optimistic, nested CV corrects it), but don't quote the specific 13-point gap as a universal number; re-run the simulation with your own data's characteristics and grid size if you need a project-specific estimate of how much the naive approach would mislead you.

Direct answer
Both the inner and outer splits need to respect time ordering (never train on data that comes after what you're validating on) AND keep all rows from the same group together on one side of any split; a straightforward nested-CV implementation that shuffles randomly or splits by row breaks both guarantees at once.

Structured elaboration
For the OUTER loop: use a time-based split (e.g. expanding-window or rolling-origin splits) so each outer-test fold contains only data from AFTER its corresponding outer-training data, never randomly interleaved; within that, any group (user, session) that straddles the time boundary needs a rule (commonly: assign the whole group to whichever side contains the majority of its timestamps, or exclude straddling groups entirely near the boundary) so no group appears partially on both sides.
For the INNER loop: the same two constraints apply recursively within the outer-training data, the inner splits used for hyperparameter search must also respect time ordering (so the inner "validation" data used for tuning is genuinely later than the inner "training" data) and group integrity. This means you generally can't use a plain grouped K-fold OR a plain time-series split alone inside a nested CV for this kind of data, you need a splitter that enforces both constraints simultaneously.
A practical approach: bucket by time first into coarse periods, then within any period that would otherwise be split, resolve based on group membership, effectively a "grouped, time-ordered" splitter, which most standard library splitters don't provide out of the box and often needs custom logic.

Worked example
A fraud-detection dataset with millions of transactions grouped by user and spanning 18 months: the outer loop might use 6 rolling 3-month windows (train on months 1-3, test on 4; train 1-6, test 7; etc.), and within each outer-training window, the inner loop for hyperparameter tuning further splits by an earlier/later sub-period, with any user whose transactions span a split boundary assigned entirely to the earlier side to avoid leaking a fraudster's later, easier-to-catch behavior into the training data for detecting their earlier behavior.

Trade-offs & pitfalls
This compounding of constraints usually shrinks your usable data for both the inner search and outer evaluation compared to unconstrained nested CV, since you're discarding or reassigning boundary-straddling groups at every split; budget for that data loss when deciding how many outer and inner folds you can realistically afford.

Direct answer
Beyond the single best trial, look at the marginal distribution of scores versus each hyperparameter individually (which ones show a clear trend, which look like noise), check for interaction effects between pairs of hyperparameters (whether one hyperparameter's effect depends on another's value), and examine the top-N trials collectively (not just the single winner) to see whether they cluster in a consistent region of the space or are scattered, which tells you how confidently you can trust the "best" point.

Structured elaboration
A per-hyperparameter marginal plot (score on the y-axis, hyperparameter value on the x-axis, one point per trial) quickly reveals which hyperparameters have a real, visible trend (score clearly improves or degrades as the value changes) versus which look like pure scatter (that hyperparameter probably didn't matter much across the range explored). For hyperparameters showing a real trend, check whether the trend has already plateaued within your explored range (no need to widen the search) or is still improving at the edge of the range you tested (a signal to extend the range in a follow-up search).
Interaction effects matter because a marginal plot can be misleading on its own: two hyperparameters can each look flat individually while their COMBINATION strongly affects the score, or a hyperparameter's apparent trend can reverse depending on the value of a second one (a classic case is learning rate and batch size, where the "best" learning rate shifts depending on batch size, so a marginal plot of learning rate alone, pooling across all batch sizes, can look noisier or flatter than it really is conditional on batch size). Check this with a 2D heatmap or scatter plot for suspected pairs (color or facet by the second hyperparameter's value), or, for a more systematic scan across many pairs, look at how much of the score variance a simple model (like a shallow decision tree or ANOVA-style variance decomposition) attributes to pairwise interaction terms versus main effects alone.
Looking at the top-10 or top-20 trials collectively, rather than just the single best: if they all cluster in a similar region of the space, that's reassuring evidence the "best" point is a real, findable region, not a lucky outlier; if the top trials are scattered across very different configurations with similar scores, that suggests the objective is fairly flat/noisy in this space and the single best trial's exact values shouldn't be over-trusted.

Worked example
Analyzing 300 Bayesian-optimization trials for a gradient-boosted model: a marginal plot shows learning_rate has a clear, strong trend (score rises then falls sharply outside a narrow band), while colsample_bytree shows essentially flat scatter across its whole tested range on its own. But faceting the learning_rate plot by max_depth reveals the "optimal" learning_rate band shifts noticeably between shallow and deep trees, an interaction the marginal plot alone hid; this tells you the two hyperparameters shouldn't be tuned as if independent, and a follow-up search should treat them as a coupled pair rather than sweeping each separately.

Trade-offs & pitfalls
This kind of post-hoc analysis is standard practice but easy to skip when you're just trying to grab the winning configuration and move on; the analysis is cheap (it's just examining data you already have) and its payoff (knowing where future tuning budget is worth spending) compounds over every future round of tuning on similar models. Interaction checks specifically are easy to under-invest in since the number of pairs grows quadratically with the hyperparameter count, prioritize checking pairs you already have some mechanistic reason to suspect interact (like learning rate and batch size, or tree depth and regularization strength) rather than exhaustively screening every possible pair.

Direct answer
Select a stratified sample of representative cases FROM EACH business unit rather than a single pooled dataset, weighting or reporting results separately per unit so a comparison isn't dominated by whichever unit happens to contribute the most data, and explicitly validate that the benchmark's difficulty is comparable across units before trusting cross-unit comparisons.

Structured elaboration
Business units often differ in data volume, label quality, and task difficulty; a naive pooled benchmark (just combine everyone's data and report one aggregate score) risks being dominated by whichever unit contributed the most examples, silently underweighting smaller units' performance in the aggregate number. A fairer design reports per-unit metrics alongside any aggregate, and where an aggregate is needed, weights units by something meaningful to the comparison's purpose (equal per-unit weight if you care about "does this generalize across units," volume-weighted if you care about "aggregate business impact") rather than defaulting to whichever weighting the raw data happens to produce.
Validate comparability: check that each unit's benchmark slice has a similar DIFFICULTY profile (similar class balance, similar noise level) where possible; if one unit's task is inherently harder (more class imbalance, noisier labels), a lower score there doesn't necessarily mean a worse model for that unit, and presenting it as if it does would be misleading.

Worked example
Comparing a candidate model across 4 business units with wildly different data volumes (10,000 vs. 500,000 rows): a naive pooled benchmark would be almost entirely determined by the largest unit's performance; reporting per-unit scores plus an equally-weighted-across-units aggregate gives a much more honest picture of whether the model generalizes, rather than one unit's characteristics masquerading as the "overall" result.

Trade-offs & pitfalls
Getting buy-in on the weighting scheme itself (equal-per-unit vs. volume-weighted) is often a genuine organizational negotiation, not just a technical choice, since different units will naturally prefer whichever weighting favors their own results; make the weighting explicit and agreed upfront, the same discipline as making objective trade-offs explicit in a multi-objective selection.