Classical Machine Learning Algorithms Questions

Direct answer

When missing fraud is much more expensive than a false alarm, optimize for recall while tracking precision as the cost of that recall, and pick the operating threshold by turning the business's cost asymmetry into an explicit expected-cost calculation rather than defaulting to 0.5. Precision-recall AUC (not ROC-AUC) is the right curve to compare models on, since it's sensitive to exactly the trade-off the business cares about.

Metrics and model selection

• Precision-recall curve, not ROC. With fraud rates typically well under 5%, ROC-AUC can look strong even for a mediocre model because the false-positive-rate denominator (all legitimate transactions) is huge. Precision and recall are both computed against the small positive class, so the PR curve directly shows the trade-off you're navigating.
• $F_\beta$ score with $\beta > 1$ if you want a single tuning number that weights recall above precision; $\beta$ can be derived from the cost ratio (shown below), rather than guessed.
• Report an operational number alongside the statistical one: at your chosen threshold, how many transactions per day get flagged for review, since that's a capacity constraint the fraud team actually lives with, not just a modeling detail.
• Calibrate probabilities before treating the model's output as a probability for cost calculations: a raw model score can rank transactions correctly (higher score means more likely fraud) while still being a bad probability estimate, so you fit a small correction on top of it, either Platt scaling (fits a logistic curve mapping raw scores to calibrated probabilities) or isotonic regression (fits a non-decreasing step function instead, more flexible but needs more data). An uncalibrated score can still rank well while being useless for "expected cost" math.

Choosing the operating threshold from cost

Define $C_{fn}$ as the cost of a missed fraud and $C_{fp}$ as the cost of investigating a false alarm. For a given threshold $t$, the expected cost on a validation set is:

$$\text{ExpectedCost}(t) = C_{fn} \cdot FN(t) + C_{fp} \cdot FP(t)$$

Sweep $t$ across the range of predicted scores, compute $FN(t)$ and $FP(t)$ at each point from the confusion matrix, and pick the $t^\ast$ that minimizes expected cost. If the business instead states a hard floor ("we must catch at least 90% of fraud"), find the lowest threshold that meets that recall and, among thresholds satisfying it, minimize false positives or expected cost. The two approaches usually point to a similar region of the curve; the cost-floor version is often easier to defend to stakeholders because it's phrased as their constraint, not a modeling choice.

Worked example: two thresholds, real costs

Take a validation set of 1,000 transactions with 20 true fraud cases (2%), and set $C_{fn} = 500$ (average loss per missed fraud) and $C_{fp} = 5$ (cost of a wasted investigation).

Threshold A (looser): 15 true positives, 50 false positives, 5 false negatives.

$$\text{recall} = \frac{15}{20} = 0.75, \qquad \text{precision} = \frac{15}{65} = 0.231$$ $$\text{ExpectedCost}_A = 500(5) + 5(50) = 2{,}500 + 250 = 2{,}750$$

Threshold B (stricter): 10 true positives, 10 false positives, 10 false negatives.

$$\text{recall} = \frac{10}{20} = 0.50, \qquad \text{precision} = \frac{10}{20} = 0.50$$ $$\text{ExpectedCost}_B = 500(10) + 5(10) = 5{,}000 + 50 = 5{,}050$$

Even though threshold B looks better by the usual "balanced" precision/recall instinct (both at 0.50), it is nearly twice as expensive in real terms, because at this cost ratio ($C_{fn}/C_{fp}=100$) each additional missed fraud is far more expensive than dozens of false alarms. Threshold A, the looser one, is the better operating point given these costs. This is the concrete argument for computing expected cost directly instead of eyeballing precision and recall.

Trade-offs and pitfalls

• Getting $C_{fn}$ and $C_{fp}$ from stakeholders is the actual hard part; a threshold derived from made-up costs is precise-looking but not more correct than a guess, so treat those numbers as an explicit, revisitable assumption, not a constant.
• Investigator capacity is a real constraint the expected-cost formula ignores: a threshold that's optimal on paper but floods the fraud team with more alerts than they can review in a day isn't actually deployable; incorporate a capacity constraint (e.g. review at most $k$ cases/day, rank by score) rather than a pure threshold.
• Fraud labels are frequently delayed (a chargeback shows up weeks later) or incomplete (undetected fraud is recorded as legitimate); training and evaluating against those labels understates the true false-negative rate, so periodically revisit performance as labels mature.
• Thresholds drift: as fraud patterns and the legitimate-transaction mix change over time, a fixed threshold's expected cost changes with it. Recompute periodically rather than treating threshold selection as a one-time decision.

Direct answer

Gradient boosting builds its ensemble one small step at a time: start from a simple baseline prediction, look at where that prediction is still wrong (the residual, or more generally the gradient of the loss), fit a new weak learner to predict that error, and add a shrunk version of it to the running prediction. Repeat, and each new learner nudges the ensemble a little closer to the targets. learning_rate controls the size of each nudge, n_estimators controls how many nudges you take, and base-learner complexity controls how much each individual nudge is allowed to represent.

Structured elaboration

The sequential fitting process.

1. Initialize the ensemble with a simple constant prediction, typically the mean of the target for regression with squared-error loss.
2. Compute how wrong the current ensemble is for every training example: for squared-error loss this is literally the residual $y_i - F(x_i)$; for other losses it's the negative gradient of the loss with respect to the current prediction (a "pseudo-residual"), which reduces to the ordinary residual as a special case for squared error.
3. Fit a new weak learner (almost always a shallow decision tree) to predict those residuals/gradients from the input features.
4. Scale the new learner's output by learning_rate and add it to the running prediction: $F_{m}(x) = F_{m-1}(x) + \text{learning\_rate} \times h_m(x)$.
5. Repeat steps 2 to 4 for n_estimators rounds.

What each hyperparameter controls.

• learning_rate scales how much each tree's correction actually moves the ensemble. A smaller learning rate (e.g., 0.05) means each step is conservative, so mistakes get corrected more gradually and the model is less likely to overreact to any single tree's idiosyncrasies, at the cost of needing more trees to reach the same overall fit.
• n_estimators is how many correction rounds you run. With a small learning rate you need more rounds to converge; with a large learning rate, few rounds can already overfit, since large corrections applied repeatedly can chase noise in the residuals.
• Base-learner complexity (tree depth, or number of leaves) controls how much structure each individual correction step is allowed to capture. Very shallow trees (even single-split "stumps") are common because they're weak enough that many small, well-regularized corrections tend to generalize better than a few large, complex ones; deeper trees per step converge in fewer rounds but risk overfitting each individual correction to training-set noise.

These three interact: a smaller learning rate paired with shallower trees and more estimators is the classic "slow and steady" regularized recipe; the total model capacity is roughly the product of how many trees you fit, how expressive each one is, and how much each one is allowed to contribute.

Worked example

Toy regression with three training targets $y = [3, 5, 9]$. Initialize $F_0$ as the mean: $F_0 = (3+5+9)/3 = 17/3 \approx 5.667$ for every example. The residuals against this baseline:

$$r = y - F_0 = [3 - 5.667,\ 5 - 5.667,\ 9 - 5.667] = [-2.667,\ -0.667,\ 3.333]$$

Suppose the first weak learner, fit to these residuals, predicts exactly $h_1 = [-2.667, -0.667, 3.333]$ (a tree that perfectly memorizes 3 training points, for illustration). With learning_rate = 0.1, the update is:

$$F_1 = F_0 + 0.1 \times h_1 = [5.667 - 0.2667,\ 5.667 - 0.0667,\ 5.667 + 0.3333] = [5.400,\ 5.600,\ 6.000]$$

Compare to the true targets $[3, 5, 9]$: the squared error before this step was $(-2.667)^2 + (-0.667)^2 + (3.333)^2 = 7.11 + 0.44 + 11.11 = 18.66$; after this one small step it's $(3-5.400)^2 + (5-5.600)^2 + (9-6.000)^2 = 5.76 + 0.36 + 9.00 = 15.12$, a modest improvement from a single conservative step, exactly the "many small nudges" behavior the learning rate is designed to produce. A larger learning rate of 1.0 applied to the same $h_1$ would move $F_1$ all the way to $[3, 5, 9]$ (perfect fit) in a single step, which is precisely why aggressive learning rates with expressive base learners overfit fast: the model has no room left to generalize beyond memorizing this exact training set.

Trade-offs & pitfalls

• Fitting sequentially to residuals means each tree depends on every tree before it; there's no natural row/column-level parallelism across boosting rounds the way there is within bagging, though modern implementations parallelize the split-finding within each individual tree.
• Too large a learning rate with too many estimators overfits fast, since large corrections applied repeatedly chase noise; too small a learning rate with too few estimators underfits, since the ensemble never fully closes the gap to the targets. Early stopping on a validation set is the standard practical guardrail against picking a bad n_estimators for a given learning rate.
• Shallow base learners (stumps or depth 2-3 trees) are usually preferred over deep ones specifically because gradient boosting already gets its expressiveness from stacking many weak learners; giving each individual learner too much capacity removes the regularizing benefit of the "many small steps" approach.
• This description covers first-order gradient boosting; modern implementations like XGBoost also use second-order (Hessian) information to compute better per-leaf updates, which changes the details of step 3 and 4 but not the overall sequential structure described here.

Direct answer

The bias-variance tradeoff describes how a model's expected prediction error splits into three parts: bias (systematic error from a model that's too simple to represent the true relationship), variance (instability from a model that's sensitive to exactly which training sample it saw), and irreducible noise (error you can't remove no matter what you do). Increasing model complexity typically trades bias for variance, and the goal is to land at the complexity where their sum is smallest.

Structured elaboration

Formally, for a query point $x$, $E\big[(y - \hat f(x))^2\big] = \underbrace{\big(\bar f(x) - f(x)\big)^2}_{\text{bias}^2} + \underbrace{E\big[(\hat f(x) - \bar f(x))^2\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{irreducible noise}}$, where $\bar f(x)$ is the average prediction you'd get if you retrained the model over infinitely many resampled training sets. Bias measures how far that average prediction is from the truth; variance measures how much any single fit swings around that average.

Polynomial regression as the running example. Fit a degree-$d$ polynomial to noisy data generated from some smooth true function:

• Degree 1 (a straight line) on a genuinely curved relationship: every resample of the training data still produces roughly the same, wrong straight line. That's high bias, low variance, the classic underfitting signature: training error is high and validation error is close to it.
• Degree 15 on the same data: the fit bends to chase every data point, including the noise. Resample the training data and you get a visibly different curve each time. That's low bias, high variance, the classic overfitting signature: training error is very low but validation error is much higher and jumps around across resamples.
• An intermediate degree (say 5, matching the true function's shape) minimizes the sum of the two terms.

Practical steps that reduce each:

• To reduce high bias (underfitting): increase model capacity (higher polynomial degree, deeper tree, more features/interactions), reduce regularization strength, or engineer better features.
• To reduce high variance (overfitting): add regularization (L1/L2, pruning, dropout), collect more training data, simplify the model, or use ensembling (bagging averages away variance without changing bias).

Worked example

Fit four polynomial degrees (1, 5, and 15) to the same noisy sine wave with a fixed true noise variance $\sigma^2 = 0.25$. By construction of the decomposition, whatever bias and variance each fit turns out to have, expected error must equal $\text{bias}^2 + \text{variance} + 0.25$ exactly, since that's an algebraic identity. Suppose the degree-1 fit measures bias$^2=0.40$, variance$=0.02$ (stable but wrong): expected error $=0.40+0.02+0.25=0.67$. Suppose the degree-15 fit measures bias$^2=0.01$, variance$=0.55$ (nearly unbiased but wildly sample-dependent): expected error $=0.01+0.55+0.25=0.81$, worse overall despite the much lower bias. A degree-5 fit sitting between them, with bias$^2=0.08$, variance$=0.15$, gives $0.08+0.15+0.25=0.48$, the best of the three, exactly the intermediate-complexity sweet spot the theory predicts should exist.

Trade-offs & pitfalls

• The decomposition is over repeated resampling of the training set, not over repeated queries at different $x$; that's the most common point of confusion when first learning it.
• A single train/validation split gives you one realization of $\hat f$, not the average $\bar f$; cross-validation's fold-to-fold variance in the score is itself an empirical proxy for the model's variance term.
• Regularization and more data both attack variance, but they aren't interchangeable: regularization trades a bit of bias for less variance (a real cost), while more data reduces variance essentially for free.
• Techniques like bagging specifically target the variance term (by averaging over resampled fits) while leaving bias essentially unchanged, which is why bagging helps a lot with high-variance base learners (deep trees) and does very little for a high-bias one (a linear model).

Direct answer

For a medium tabular dataset (tens of thousands of rows, tens of features), I would reach for an RBF-kernel SVM when the feature count is low-to-moderate, the decision boundary is expected to be smooth, and I want a model with only two real hyperparameters (C and gamma) to tune. I would reach for a small feedforward network when I need the training and inference cost to scale cleanly with data size, or when I plan to fold in categorical embeddings and want the flexibility to grow the model later.

Structured elaboration

Practical rule of thumb: start from a strong tabular baseline (gradient-boosted trees) before either of these. Between the two, favor the SVM when you can afford the quadratic training cost and want a smooth decision boundary with minimal tuning; favor the small net when you expect to scale the dataset further, need GPU throughput, or want to integrate embeddings for high-cardinality categoricals.

Worked example

Take n = 50,000 rows and d = 20 numeric features, a realistic "medium" tabular size.

RBF-kernel SVM, kernel matrix memory:

$n = 50{,}000 \Rightarrow n^2 = 2.5\times10^{9} \text{ entries}$
$2.5\times10^{9} \text{ entries} \times 4\text{ bytes (float32)} = 10^{10}\text{ bytes} = 10\text{ GB}$

A full dense kernel matrix for this dataset alone needs 10 GB of memory, before the solver even starts iterating, which is why kernel SVMs at this scale typically require an approximation (Nystrom, random Fourier features) or a specialized low-rank/streaming solver rather than the textbook QP.

Small feedforward net, parameter count: two hidden layers of 64 units each, input dimension 20, single output.

$\#\text{params} = (d \cdot h_1 + h_1) + (h_1 \cdot h_2 + h_2) + (h_2 \cdot 1 + 1)$
$= (20 \cdot 64 + 64) + (64 \cdot 64 + 64) + (64 \cdot 1 + 1) = 1344 + 4160 + 65 = 5569$

5,569 parameters is small enough to train in a handful of epochs over 50,000 rows with ordinary mini-batch SGD, and the memory footprint (a few tens of KB for the weights) is negligible next to the SVM's 10 GB kernel matrix at this same n.

Trade-offs & pitfalls

• The SVM's cost curve is the real constraint, not its accuracy. As n grows past the tens-of-thousands range, the quadratic kernel matrix and cubic-ish solver cost make plain kernel SVM impractical without approximation.
• The net's extra hyperparameters are a real tax, not just a formality. A poorly tuned learning rate or missing regularization on a small net can underperform a default-tuned SVM on the same data.
• Neither model gives calibrated probabilities out of the box: SVM scores need Platt scaling (fitting a small logistic curve on top of the raw score to turn it into a probability), and a net's raw output needs a proper loss (e.g., cross-entropy) and possibly temperature scaling (dividing the pre-probability outputs by a single learned constant to soften overconfident predictions) to be trustworthy as a probability.
• Pitfall: picking the net purely because "neural networks are more powerful" without checking whether the smaller, cheaper SVM already solves the problem at this data size, tree-based models are still usually the strongest baseline for tabular data and should be checked first.

Direct answer

Linear regression models a continuous numeric target as a linear function of the features and is evaluated with error-magnitude metrics like RMSE or MAE and R-squared. Logistic regression models the probability of a binary (or multiclass) outcome via a sigmoid applied to a linear function, and is evaluated with classification metrics like AUC, precision and recall, and calibration measures, not R-squared.

Structured elaboration

What linear in the log-odds means concretely: logistic regression is still a linear model, just linear in log-odds space rather than in probability space; the sigmoid squashes that unbounded linear combination into a valid [0,1] probability.

Worked example

Suppose a fitted logistic regression's linear combination (log-odds) at a particular input works out to 0.5. Converting to a probability via the sigmoid:

$P(y=1\mid x) = \frac{1}{1+e^{-0.5}} = 0.6225$

A linear regression producing that same raw score of 0.5 would report it directly as the predicted value, no transform needed, valid across the whole real line and directly comparable to the actual target's units (dollars, degrees, and so on). The logistic regression's 0.5 log-odds, by contrast, isn't itself a probability, it's not even bounded to [0,1], it has to pass through the sigmoid to become the interpretable 0.6225 probability of the positive class.

Trade-offs & pitfalls

R-squared isn't meaningful for logistic regression the way it is for linear regression, some pseudo-R-squared variants exist but don't carry the same fraction-of-variance-explained interpretation, reporting plain R-squared for a classifier is a common tell that someone copy-pasted a linear-regression evaluation template onto a classification problem. Conversely, using a plain linear regression to predict a 0/1 target directly (the linear probability model) can produce predicted probabilities outside [0,1] and violates the constant-variance assumption by construction, since the variance of a Bernoulli outcome depends on its mean, logistic regression exists specifically to fix both problems.