Classical Machine Learning Algorithms Questions

Direct answer

The bias-variance tradeoff describes how a model's expected prediction error splits into three parts: bias (systematic error from a model that's too simple to represent the true relationship), variance (instability from a model that's sensitive to exactly which training sample it saw), and irreducible noise (error you can't remove no matter what you do). Increasing model complexity typically trades bias for variance, and the goal is to land at the complexity where their sum is smallest.

Structured elaboration

Formally, for a query point $x$, $E\big[(y - \hat f(x))^2\big] = \underbrace{\big(\bar f(x) - f(x)\big)^2}_{\text{bias}^2} + \underbrace{E\big[(\hat f(x) - \bar f(x))^2\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{irreducible noise}}$, where $\bar f(x)$ is the average prediction you'd get if you retrained the model over infinitely many resampled training sets. Bias measures how far that average prediction is from the truth; variance measures how much any single fit swings around that average.

Polynomial regression as the running example. Fit a degree-$d$ polynomial to noisy data generated from some smooth true function:

• Degree 1 (a straight line) on a genuinely curved relationship: every resample of the training data still produces roughly the same, wrong straight line. That's high bias, low variance, the classic underfitting signature: training error is high and validation error is close to it.
• Degree 15 on the same data: the fit bends to chase every data point, including the noise. Resample the training data and you get a visibly different curve each time. That's low bias, high variance, the classic overfitting signature: training error is very low but validation error is much higher and jumps around across resamples.
• An intermediate degree (say 5, matching the true function's shape) minimizes the sum of the two terms.

Practical steps that reduce each:

• To reduce high bias (underfitting): increase model capacity (higher polynomial degree, deeper tree, more features/interactions), reduce regularization strength, or engineer better features.
• To reduce high variance (overfitting): add regularization (L1/L2, pruning, dropout), collect more training data, simplify the model, or use ensembling (bagging averages away variance without changing bias).

Worked example

Fit four polynomial degrees (1, 5, and 15) to the same noisy sine wave with a fixed true noise variance $\sigma^2 = 0.25$. By construction of the decomposition, whatever bias and variance each fit turns out to have, expected error must equal $\text{bias}^2 + \text{variance} + 0.25$ exactly, since that's an algebraic identity. Suppose the degree-1 fit measures bias$^2=0.40$, variance$=0.02$ (stable but wrong): expected error $=0.40+0.02+0.25=0.67$. Suppose the degree-15 fit measures bias$^2=0.01$, variance$=0.55$ (nearly unbiased but wildly sample-dependent): expected error $=0.01+0.55+0.25=0.81$, worse overall despite the much lower bias. A degree-5 fit sitting between them, with bias$^2=0.08$, variance$=0.15$, gives $0.08+0.15+0.25=0.48$, the best of the three, exactly the intermediate-complexity sweet spot the theory predicts should exist.

Trade-offs & pitfalls

• The decomposition is over repeated resampling of the training set, not over repeated queries at different $x$; that's the most common point of confusion when first learning it.
• A single train/validation split gives you one realization of $\hat f$, not the average $\bar f$; cross-validation's fold-to-fold variance in the score is itself an empirical proxy for the model's variance term.
• Regularization and more data both attack variance, but they aren't interchangeable: regularization trades a bit of bias for less variance (a real cost), while more data reduces variance essentially for free.
• Techniques like bagging specifically target the variance term (by averaging over resampled fits) while leaving bias essentially unchanged, which is why bagging helps a lot with high-variance base learners (deep trees) and does very little for a high-bias one (a linear model).

Direct answer

The intercept is the expected outcome for an observation at the reference (baseline) level of every categorical variable and zero on all continuous predictors; each dummy coefficient is the average shift relative to that baseline, holding everything else fixed. You avoid the dummy-variable trap by dropping one level per categorical variable, encoding a K-level category with K-1 dummy columns so the design matrix stays full rank.

Structured elaboration

Encoding: for a K-level categorical variable, create K-1 dummy columns and drop one level (the reference).

Intercept: if continuous predictors aren't centered, the intercept is technically the prediction at continuous = 0, which may be outside the realistic data range. Centering continuous predictors at their sample mean makes the intercept read as "typical prediction for the baseline category" instead of a mathematical extrapolation.

Dummy coefficients: each one is interpreted as "holding the continuous predictors fixed, this category's prediction is [coefficient] higher or lower than the baseline category's prediction."

The dummy-variable trap: including all K dummy columns plus an intercept makes the K dummy columns sum to the constant intercept column, a perfect linear dependency. This makes X^T X singular, so OLS either has no unique solution or the solver silently drops a column / produces unstable coefficients depending on the library. Fixes: drop one level (reference coding, the standard choice), or drop the intercept and keep all K dummies (each coefficient then directly estimates that group's mean), or apply regularization if you specifically want all levels represented (ridge tolerates the redundancy but the individual coefficients lose the clean "versus baseline" reading).

Worked example

Predictors: sqft (continuous) and a 3-level region variable (North = baseline, South, West). Suppose the fitted model is (illustrative coefficients, in $000s):

$\widehat{\text{price}} = 50 + 0.20\,\text{sqft} + 20\,\text{South} + 35\,\text{West}$

The intercept, 50, is the predicted price for a North home at sqft = 0 (a mathematical anchor, not a realistic home). For a South home at sqft = 1500: 50 + 0.20(1500) + 20 = 50 + 300 + 20 = 370 ($000). For a West home at the same sqft: 50 + 300 + 35 = 385 ($000). So, holding square footage fixed, a West home is predicted $15,000 more than an equivalent South home (385 - 370) and $35,000 more than an equivalent North home, which is exactly what the West coefficient of 35 says on its own.

Trade-offs & pitfalls

Forgetting to drop a level triggers the trap; whether that fails loudly or silently depends on the tool (pandas get_dummies keeps all levels unless drop_first=True is passed explicitly; scikit-learn's OneHotEncoder needs drop='first'). The choice of reference level is statistically arbitrary but affects readability, pick a meaningful baseline (most common category, or a natural "control" group) for stakeholder-facing reports. Rare categories can have dummy coefficients with very large standard errors since few observations inform them; grouping rare levels into an "other" bucket is a common practical fix.

Direct answer

Explain it in two layers: first, a plain-language sentence about direction and rough size ("holding everything else equal, one more support ticket in the last 30 days roughly multiplies a customer's odds of churning by 1.5"), then, if they want more, the odds-ratio mechanics behind that sentence. The key translation step a PM needs is going from a coefficient (which lives on the log-odds scale, not intuitive to anyone) to an odds ratio (a multiplier, more intuitive) to an approximate change in probability at their specific baseline rate (the only thing that actually maps to "how big a deal is this").

Structured elaboration

Why coefficients aren't directly readable. Logistic regression models $\log\!\big(\frac{p}{1-p}\big) = \beta_0 + \beta_1 x_1 + \dots + \beta_n x_n$, the log-odds, as a linear function of the features. A coefficient $\beta_j$ is the change in log-odds per one-unit increase in $x_j$, holding other features fixed. Nobody, including most data scientists on first encounter, has good intuition for what "log-odds increased by 0.4" feels like, so the raw coefficient is the wrong unit to hand a PM.

Step 1: convert to an odds ratio. Exponentiating gives $\mathrm{OR}_j = e^{\beta_j}$, the odds ratio: exactly how much the odds (not the probability) get multiplied by a one-unit increase in $x_j$. This is already more intuitive ("odds go up by 50%") but "odds" is still not the same thing as "probability", and conflating them is the single most common mistake in explaining this to a non-technical audience.

Step 2: convert odds ratio to an approximate probability change at a specific baseline. Odds and probability relate by $\text{odds} = \frac{p}{1-p}$, so the same odds ratio produces a very different probability change depending on the baseline rate: a 50% odds increase moves an 8% baseline probability to roughly 11.5%, but the same 50% odds increase would move a 50% baseline probability all the way to 60%. This baseline-dependence is exactly why you should never say "50% higher odds" and let a stakeholder silently substitute "50% higher probability", the two are only close to each other when the baseline probability is very small.

A concrete script for the PM conversation: "Right now, about 8 out of 100 customers churn. For every extra support ticket in the last 30 days, a customer's chance of churning goes up to roughly 11 or 12 out of 100, holding everything else about them the same. It's not a huge jump for one ticket, but it compounds: several tickets in a row push that number up faster than you'd guess from a straight line, because we're modeling odds, not probability, directly."

Worked example

Suppose $\beta_j = 0.4$ for "number of support tickets in the last 30 days", and the baseline churn probability at the average customer's other feature values is $p_0 = 0.08$.

• Odds ratio: $e^{0.4} \approx 1.492$.
• Baseline odds: $\frac{0.08}{1-0.08} = 0.08696$.
• New odds after a one-unit increase: $0.08696 \times 1.492 \approx 0.12972$.
• New probability: $\frac{0.12972}{1+0.12972} \approx 0.1148$, i.e. about 11.5%.

So one extra ticket moves this customer from an 8.0% to an 11.5% chance of churning, a 3.5 percentage-point increase (verified by computing it two independent ways: through the odds transformation above, and directly by shifting the logit by 0.4 and re-applying the sigmoid, both give 0.1148). Note this is a local approximation around this specific baseline; the same coefficient produces a different percentage-point shift for a customer starting at, say, a 40% baseline churn probability, which is exactly why "the effect" has to be quoted relative to a stated starting point rather than as one universal number.

Trade-offs & pitfalls

• Never say "the odds went up 50%, so the probability went up 50%"; that's only approximately true when the baseline probability is very small (well under 10%), and it's flatly wrong otherwise.
• Always anchor the explanation to a real baseline rate the PM already has intuition for ("our typical customer", "our highest-risk segment") rather than quoting the odds ratio in the abstract.
• "Holding everything else equal" is doing real work in this sentence and is worth saying out loud: if support tickets are correlated with tenure or plan type, a PM might otherwise misread the coefficient as the total observed effect of tickets in the raw data, not the effect isolated from those other features.
• Coefficients only describe correlational structure inside a fitted model, not causation; if the eventual decision is "let's proactively call customers with many tickets to prevent churn", that's a causal claim the model alone doesn't license, worth flagging explicitly before a business decision is made on it.

Direct answer

L1 (Lasso) and L2 (Ridge) both shrink coefficients to fight overfitting, but they shrink differently: L2 pulls every coefficient smoothly toward zero without ever reaching it, while L1 pushes some coefficients to exactly zero, performing feature selection as a side effect. ElasticNet blends the two penalties so you get sparsity where the data supports it and stability where predictors are correlated.

Structured elaboration

The objectives. Ridge adds $\lambda \lVert w \rVert_2^2 = \lambda \sum_j w_j^2$ to the squared-error loss; Lasso adds $\lambda \lVert w \rVert_1 = \lambda \sum_j |w_j|$; ElasticNet adds $\lambda \big(\alpha \lVert w \rVert_1 + \tfrac{1-\alpha}{2}\lVert w \rVert_2^2\big)$, mixing the two via $\alpha \in [0,1]$.

Why the geometry differs. Picture the constrained-optimization view: minimize the loss subject to the penalty term staying under some budget. The L2 budget region is a smooth ball, so the loss contours touch it at a generic point where no coordinate is forced to zero. The L1 budget region is a diamond (a cross-polytope in higher dimensions) with sharp corners sitting exactly on the coordinate axes; the loss contours are much more likely to first touch the constraint region at one of those corners, which is precisely where one or more coordinates are zero. That's the entire mechanical reason L1 gives sparsity and L2 doesn't; it isn't a separate rule, it falls out of the shape of the constraint set.

Correlated features. With two highly correlated predictors, Ridge tends to split the coefficient roughly evenly between them (it prefers many small weights over one large one, because the squared penalty punishes a single big coefficient more than two smaller ones summing to the same total effect). Lasso, by contrast, tends to arbitrarily pick one of the correlated pair and zero out the other, which is unstable: a small change in the data can flip which one survives. This instability under correlation is Lasso's most cited practical weakness.

When ElasticNet earns its extra hyperparameter. ElasticNet's L2 component brings back the "grouping effect" Ridge has (correlated features get similar, non-zero coefficients together) while its L1 component still zeroes out genuinely irrelevant features. It's the right default whenever you have many correlated predictors and still want sparsity, e.g. one-hot-encoded categories, engineered features that are near-duplicates of each other, or genomic/text features with block correlation structure. Pure Lasso is preferable when you're confident the true signal is sparse and predictors are close to independent; pure Ridge is preferable when you believe most predictors carry a little bit of signal and you don't need automatic feature selection at all.

Worked example

Take three predictors: $x_1$ and $x_2$ are near-duplicate columns (correlation 0.99996), and $x_3$ is independent and irrelevant. Fit Ridge, Lasso, and ElasticNet on data where the true effect is split evenly between $x_1$ and $x_2$ and $x_3$ has none:

import numpy as np
from sklearn.linear_model import Ridge, Lasso, ElasticNet

np.random.seed(2)
n = 200
x1 = np.random.randn(n)
x2 = x1 + 0.01 * np.random.randn(n)   # near-duplicate of x1, corr ~ 0.99996
x3 = np.random.randn(n)               # independent, irrelevant
X = np.column_stack([x1, x2, x3])
y = 0.65*x1 + 0.65*x2 + 0.0*x3 + 0.3*np.random.randn(n)

ridge = Ridge(alpha=1.0).fit(X, y)
lasso = Lasso(alpha=0.1, max_iter=50000).fit(X, y)
en = ElasticNet(alpha=0.15, l1_ratio=0.5, max_iter=50000).fit(X, y)

This runs (verified) to:

• Ridge: $w = (0.648, 0.656, 0.035)$, distributing the shared signal almost evenly across the correlated pair, with $x_3$'s coefficient pushed toward, but not to, zero.
• Lasso: $w = (1.224, 0.000, 0.000)$, collapsing the correlated pair down to a single surviving feature and zeroing the irrelevant one.
• ElasticNet: $w = (0.604, 0.603, 0.000)$, keeping the grouping behavior on the correlated pair (like Ridge) while still zeroing the irrelevant feature (like Lasso).

Refitting Lasso at the same $\alpha=0.1$ across 20 different noise draws (only the random seed generating $y$'s noise term changes, $X$ stays fixed) confirms the instability claim directly: $x_1$ survives and $x_2$ is zeroed in 14 of the 20 refits, and the reverse (x_2 survives, x_1 zeroed) happens in the other 6, with the loser's coefficient landing at exactly 0.000 every time, never a partial split between them. Which one wins is a property of the noise draw, not of the features themselves, since $x_1$ and $x_2$ are close to interchangeable by construction.

Trade-offs & pitfalls

• Standardize features before applying any of these penalties. Because the penalty is applied to the raw coefficient magnitudes, an unscaled feature with a naturally large numeric range gets penalized less per unit of actual predictive contribution than one on a small scale, which silently biases which features survive.
• Lasso's feature selection is a selection event, not a stable ranking; don't over-interpret "Lasso dropped this feature" as "this feature has zero true effect," especially under correlation.
• ElasticNet costs you a second hyperparameter ($\alpha$) to tune on top of $\lambda$, which roughly doubles your cross-validation grid; that cost is worth it when correlated predictors are actually present, and wasted effort when they aren't.
• None of these penalties handle non-linearity; they only regularize the linear coefficients. Multicollinearity that's actually a symptom of a missing interaction term won't be fixed by any of the three, it needs a feature-engineering fix instead.

Direct answer

When missing fraud is much more expensive than a false alarm, optimize for recall while tracking precision as the cost of that recall, and pick the operating threshold by turning the business's cost asymmetry into an explicit expected-cost calculation rather than defaulting to 0.5. Precision-recall AUC (not ROC-AUC) is the right curve to compare models on, since it's sensitive to exactly the trade-off the business cares about.

Metrics and model selection

• Precision-recall curve, not ROC. With fraud rates typically well under 5%, ROC-AUC can look strong even for a mediocre model because the false-positive-rate denominator (all legitimate transactions) is huge. Precision and recall are both computed against the small positive class, so the PR curve directly shows the trade-off you're navigating.
• $F_\beta$ score with $\beta > 1$ if you want a single tuning number that weights recall above precision; $\beta$ can be derived from the cost ratio (shown below), rather than guessed.
• Report an operational number alongside the statistical one: at your chosen threshold, how many transactions per day get flagged for review, since that's a capacity constraint the fraud team actually lives with, not just a modeling detail.
• Calibrate probabilities before treating the model's output as a probability for cost calculations: a raw model score can rank transactions correctly (higher score means more likely fraud) while still being a bad probability estimate, so you fit a small correction on top of it, either Platt scaling (fits a logistic curve mapping raw scores to calibrated probabilities) or isotonic regression (fits a non-decreasing step function instead, more flexible but needs more data). An uncalibrated score can still rank well while being useless for "expected cost" math.

Choosing the operating threshold from cost

Define $C_{fn}$ as the cost of a missed fraud and $C_{fp}$ as the cost of investigating a false alarm. For a given threshold $t$, the expected cost on a validation set is:

Sweep $t$ across the range of predicted scores, compute $FN(t)$ and $FP(t)$ at each point from the confusion matrix, and pick the $t^\ast$ that minimizes expected cost. If the business instead states a hard floor ("we must catch at least 90% of fraud"), find the lowest threshold that meets that recall and, among thresholds satisfying it, minimize false positives or expected cost. The two approaches usually point to a similar region of the curve; the cost-floor version is often easier to defend to stakeholders because it's phrased as their constraint, not a modeling choice.

Worked example: two thresholds, real costs

Take a validation set of 1,000 transactions with 20 true fraud cases (2%), and set $C_{fn} = 500$ (average loss per missed fraud) and $C_{fp} = 5$ (cost of a wasted investigation).

Threshold A (looser): 15 true positives, 50 false positives, 5 false negatives.

Threshold B (stricter): 10 true positives, 10 false positives, 10 false negatives.

Even though threshold B looks better by the usual "balanced" precision/recall instinct (both at 0.50), it is nearly twice as expensive in real terms, because at this cost ratio ($C_{fn}/C_{fp}=100$) each additional missed fraud is far more expensive than dozens of false alarms. Threshold A, the looser one, is the better operating point given these costs. This is the concrete argument for computing expected cost directly instead of eyeballing precision and recall.

Trade-offs and pitfalls

• Getting $C_{fn}$ and $C_{fp}$ from stakeholders is the actual hard part; a threshold derived from made-up costs is precise-looking but not more correct than a guess, so treat those numbers as an explicit, revisitable assumption, not a constant.
• Investigator capacity is a real constraint the expected-cost formula ignores: a threshold that's optimal on paper but floods the fraud team with more alerts than they can review in a day isn't actually deployable; incorporate a capacity constraint (e.g. review at most $k$ cases/day, rank by score) rather than a pure threshold.
• Fraud labels are frequently delayed (a chargeback shows up weeks later) or incomplete (undetected fraud is recorded as legitimate); training and evaluating against those labels understates the true false-negative rate, so periodically revisit performance as labels mature.
• Thresholds drift: as fraud patterns and the legitimate-transaction mix change over time, a fixed threshold's expected cost changes with it. Recompute periodically rather than treating threshold selection as a one-time decision.