Linear and Logistic Regression Implementation Questions

Covers the fundamentals and implementation details of linear regression for continuous prediction and logistic regression for binary or multiclass classification. Candidates should understand model formulation, hypothesis functions, and the intuition behind fitting a line or hyperplane for regression and using a sigmoid or softmax function for classification. Include loss functions such as mean squared error for regression and cross entropy loss for classification, optimization methods including gradient descent and variants, regularization techniques, feature engineering and scaling, metrics for evaluation such as mean absolute error and accuracy and area under curve, and hyperparameter selection and validation strategies. Expect discussion of practical implementation using numerical libraries and machine learning toolkits, trade offs and limitations of each approach, numerical stability, and common pitfalls such as underfitting and overfitting.

EasyTechnical

44 practiced

List and explain evaluation metrics appropriate for regression (MAE, MSE, RMSE, R^2) and for classification (accuracy, precision, recall, F1, ROC-AUC, PR-AUC). For each metric, describe what aspect of performance it emphasizes and give one scenario where that metric should be preferred.

Sample Answer

Regression metrics

MAE (Mean Absolute Error)- Definition: average absolute difference between predictions and targets.- Emphasis: overall average magnitude of errors; robust to outliers (linear penalty).- Use when: you want interpretable error in original units and equal weighting of errors (e.g., forecasting daily demand where each unit error has equal cost).

MSE (Mean Squared Error)- Definition: average squared error.- Emphasis: penalizes larger errors more heavily (quadratic penalty), smooth for optimization.- Use when: large errors are particularly harmful (e.g., training models where large deviations cause safety risks) or when optimizing with gradient-based methods benefits from smooth differentiable loss.

RMSE (Root Mean Squared Error)- Definition: square root of MSE (same units as target).- Emphasis: like MSE but easier to interpret in original units; still sensitive to large errors.- Use when: you want interpretable scale while penalizing large errors (e.g., model selection for regression in energy consumption forecasting).

R² (Coefficient of Determination)- Definition: proportion of variance explained by the model (1 − SSE/SST).- Emphasis: relative fit compared to baseline mean predictor; useful for comparing models on same data.- Use when: measuring explanatory power (e.g., assessing how much variance in a continuous outcome the features explain).

Classification metrics

Accuracy- Definition: fraction of correct predictions.- Emphasis: overall correctness.- Use when: classes are balanced and false positives/negatives have similar cost (e.g., digit recognition with balanced classes).

Precision- Definition: TP / (TP + FP).- Emphasis: correctness among positive predictions (trustworthiness of positive labels).- Use when: false positives are costly (e.g., spam filter where marking a legitimate email as spam is bad).

Recall (Sensitivity)- Definition: TP / (TP + FN).- Emphasis: ability to find positive instances (coverage).- Use when: missing positives is costly (e.g., disease screening where undetected cases are dangerous).

F1 score- Definition: harmonic mean of precision and recall.- Emphasis: balance between precision and recall.- Use when: class imbalance exists and you need a compromise (e.g., information extraction where both errors matter).

ROC-AUC- Definition: area under ROC (TPR vs FPR across thresholds).- Emphasis: ranking ability across classes, insensitive to threshold and class imbalance to some extent.- Use when: evaluating model’s discriminative power regardless of threshold (e.g., medical triage model ranking risk).

PR-AUC- Definition: area under precision-recall curve.- Emphasis: performance on the positive class, especially with class imbalance.- Use when: positive class is rare and you care about precision/recall trade-offs (e.g., fraud detection).

Notes: choose metrics aligned with business costs, class balance, and optimization objectives; report multiple metrics for a full picture.

MediumTechnical

55 practiced

Implement early stopping in a logistic regression training loop. Your implementation should accept patience (number of epochs to wait for improvement), min_delta for minimal improvement, and a flag to restore the best parameters observed on the validation set. Show how you'd integrate it into an SGD training loop.

Sample Answer

Approach: implement an EarlyStopping helper that tracks best validation loss, counts epochs with no sufficient improvement (min_delta), supports patience, and optionally restores best model params. Integrate into an SGD training loop for a logistic regression (numpy) showing checkpointing of weights.

python

import numpy as np

class EarlyStopping:
    def __init__(self, patience=5, min_delta=0.0, restore_best=False):
        self.patience = patience
        self.min_delta = min_delta
        self.restore_best = restore_best
        self.best_loss = np.inf
        self.best_params = None
        self.wait = 0
        self.stopped_epoch = None

    def step(self, current_loss, params, epoch):
        improved = (self.best_loss - current_loss) > self.min_delta
        if improved:
            self.best_loss = current_loss
            self.best_params = {k: v.copy() for k, v in params.items()}
            self.wait = 0
        else:
            self.wait += 1

        if self.wait >= self.patience:
            self.stopped_epoch = epoch
            return True  # stop
        return False  # continue

    def on_train_end(self, params):
        if self.restore_best and self.best_params is not None:
            for k in params:
                params[k][...] = self.best_params[k]

# Simple logistic regression helper functions
def sigmoid(z): return 1 / (1 + np.exp(-z))

def compute_loss(X, y, w, b):
    m = X.shape[0]
    logits = X.dot(w) + b
    p = sigmoid(logits)
    # binary cross-entropy
    eps = 1e-12
    return -np.mean(y * np.log(p+eps) + (1-y)*np.log(1-p+eps))

# SGD training loop with early stopping
def train_sgd(X_train, y_train, X_val, y_val, lr=0.1, epochs=100, batch_size=32,
              patience=5, min_delta=1e-4, restore_best=True):

    n_features = X_train.shape[1]
    w = np.zeros(n_features)
    b = 0.0
    params = {'w': w, 'b': np.array(b)}  # b wrapped to allow .copy()

    es = EarlyStopping(patience=patience, min_delta=min_delta, restore_best=restore_best)

    for epoch in range(1, epochs+1):
        # Shuffle
        perm = np.random.permutation(X_train.shape[0])
        X_sh, y_sh = X_train[perm], y_train[perm]
        for i in range(0, X_train.shape[0], batch_size):
            xb = X_sh[i:i+batch_size]; yb = y_sh[i:i+batch_size]
            preds = sigmoid(xb.dot(w) + b)
            grad_w = xb.T.dot(preds - yb) / xb.shape[0]
            grad_b = np.mean(preds - yb)
            w -= lr * grad_w
            b -= lr * grad_b
            params['w'] = w
            params['b'][...] = b

        val_loss = compute_loss(X_val, y_val, w, b)
        if es.step(val_loss, params, epoch):
            print(f"Early stopping at epoch {epoch}. Best val_loss={es.best_loss:.6f}")
            break

    es.on_train_end(params)
    return params['w'], float(params['b']), es.best_loss, es.stopped_epoch

# Usage: w, b, best_loss, stop_epoch = train_sgd(X_train, y_train, X_val, y_val)

Key points:- We track validation loss, require improvement > min_delta to reset wait.- restore_best replaces parameters with the best seen on validation.- Checkpointing uses copies to avoid mutation during training.

Complexity:- Per epoch: O(N * d) where N is dataset size and d features (same as normal SGD).Edge cases:- Validation loss NaN/inf (handle by clipping or skipping step).- Very small min_delta or patience=0 (stop immediately if no improvement).Alternatives:- Monitor validation accuracy instead of loss, use moving-average smoothing, or implement exponential moving best to reduce sensitivity to noisy val metrics.

MediumTechnical

56 practiced

Explain multicollinearity and how it affects coefficient estimates in linear regression. Describe detection methods (variance inflation factor (VIF), eigenvalues/condition number), and remedies such as ridge regularization, PCA, or dropping correlated predictors. Discuss interpretability trade-offs for each remedy.

Sample Answer

Multicollinearity occurs when two or more predictor variables in a linear regression are highly linearly correlated, so the design matrix X has near-dependent columns. It doesn't bias OLS coefficients but inflates their variance, making estimates unstable, coefficients sensitive to small data changes, t‑tests unreliable, and interpretations of individual coefficients meaningless.

Detection:- Variance Inflation Factor (VIF): VIF_j = 1 / (1 − R_j^2). Values > 5–10 indicate problematic multicollinearity for feature j.- Eigenvalues / condition number: compute eigenvalues of X^T X. Very small eigenvalues or a large condition number (sqrt(λ_max/λ_min) ≫ 30) signal near-collinearity and numerical instability.

Remedies and interpretability trade-offs:- Ridge regression (L2): shrinks coefficients by penalizing large weights, dramatically reduces variance and improves prediction. Trade-off: biased coefficients and harder to ascribe causal effect to any single predictor, but preserves all features (useful in prediction-focused tasks).- PCA (principal component regression): transform correlated features into orthogonal components and regress on top components. Trade-off: excellent variance reduction and numerical stability, but components are linear combinations—interpretability of original features is lost unless you back-project or examine loadings.- Drop correlated predictors / feature selection: remove or combine redundant variables (e.g., averaging, domain-driven choice). Trade-off: retains interpretability for remaining variables but risks omitting useful signal and requires domain knowledge to choose which to drop.- Other: partial least squares, elastic net (combines L1/L2) — elastic net can select groups of correlated predictors while regularizing.

Practical guidance for AI engineers: prioritize remedies based on objective—if goal is best predictive performance use ridge/elastic net or PCA; if you need interpretable coefficients, prefer careful selection or domain-driven aggregation and report multicollinearity diagnostics. Always standardize features before VIF, PCA, or regularization and report uncertainty/robustness analyses when interpreting coefficients.

MediumTechnical

57 practiced

Explain probability calibration for classifiers. Compare Platt scaling and isotonic regression: how they work, strengths and weaknesses, and how to evaluate calibration (calibration curve, Brier score). When would you prefer a parametric method (Platt) versus a non-parametric (isotonic) approach?

Sample Answer

Probability calibration ensures a classifier's predicted probabilities match observed frequencies: e.g., among samples predicted 0.8, about 80% should be positive. Good calibration is critical for decision-making, risk estimates, ensembling and downstream probabilistic reasoning.

Platt scaling- How: Fits a sigmoid (logistic) function to map model scores (typically logits) to calibrated probabilities by optimizing log-loss on a validation set: p=1/(1+exp(A·score + B)).- Strengths: Parametric (only two parameters), robust with limited data, smooth monotonic mapping, easy to regularize and fast to fit.- Weaknesses: Assumes sigmoid shape; fails if calibration relationship is complex or non-monotonic.

Isotonic regression- How: Fits a non-parametric, monotonic piecewise-constant mapping from scores to probabilities using pooled adjacent violators (PAV) algorithm on calibration data.- Strengths: Flexible, can model arbitrary monotone calibration curves; can correct complex miscalibration.- Weaknesses: Needs more data to avoid overfitting, produces step functions (less smooth), can be unstable with few calibration points.

Evaluation- Calibration curve (reliability diagram): Bin predicted probabilities (e.g., 10 bins), plot mean predicted vs empirical frequency. Perfect calibration lies on diagonal. Use small bins carefully to avoid high variance.- Brier score: Mean squared error between predicted probability and true label (0/1). Decomposes into calibration, refinement, and uncertainty; lower is better. Complement with log-loss for sensitivity to confidence in extremes.- Also use Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) for summary metrics (note ECE depends on binning).

When to prefer which- Prefer Platt scaling when calibration dataset is small, model scores are roughly sigmoid-shaped (SVMs, some neural logits after temperature scaling), or you need a simple, low-variance solution.- Prefer isotonic regression when you have a reasonably large, representative calibration set and suspect complex, monotonic miscalibration that a sigmoid can't capture.- Practical note: For modern deep nets, temperature scaling (a single-parameter variant of Platt applied to logits) often works well; combine cross-validation and hold-out calibration sets and always evaluate with reliability diagrams and Brier/log-loss to choose method.

HardTechnical

79 practiced

Using PyTorch, implement a single-layer neural network equivalent to sklearn's logistic regression (binary). Show the model definition, training loop using BCEWithLogitsLoss, how to apply L2 regularization via optimizer weight_decay, and how to incorporate class weights. Run a small experiment to show that trained weights are similar to sklearn's LogisticRegression under matching hyperparameters.

Sample Answer

Approach: build a single linear layer with sigmoid via BCEWithLogitsLoss (which expects logits). Use optimizer weight_decay to match sklearn's L2 penalty and pass class weights via BCEWithLogitsLoss's pos_weight for imbalance. Train on a small synthetic binary dataset and compare learned coefficients and intercept to sklearn's LogisticRegression with same C (inverse of regularization) and class_weight.

python

import numpy as np
import torch
from torch import nn, optim
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Prepare synthetic data
X, y = make_classification(n_samples=2000, n_features=5, n_informative=3,
                           weights=[0.7,0.3], flip_y=0.01, random_state=0)
scaler = StandardScaler().fit(X)
Xs = scaler.transform(X)

# sklearn baseline
C = 1.0  # inverse regularization strength -> weight_decay = 1/(2*C*N) mapping explained below
clf = LogisticRegression(penalty='l2', C=C, class_weight={0:1.0,1:2.0}, solver='lbfgs', fit_intercept=True)
clf.fit(Xs, y)

# PyTorch model
device = torch.device("cpu")
X_t = torch.tensor(Xs, dtype=torch.float32, device=device)
y_t = torch.tensor(y.reshape(-1,1), dtype=torch.float32, device=device)

class LinearLogistic(nn.Module):
    def __init__(self, n_features):
        super().__init__()
        self.linear = nn.Linear(n_features, 1, bias=True)
    def forward(self, x):
        return self.linear(x)  # logits

model = LinearLogistic(Xs.shape[1]).to(device)

# Map sklearn's C to PyTorch weight_decay roughly: sklearn minimizes 0.5*alpha*||w||^2 where alpha=1/C.
# PyTorch's weight_decay adds (wd/2)*||w||^2 per parameter per update; mapping depends on lr and batch.
# For simplicity use small weight_decay = 1.0/C / X.shape[0]
weight_decay = 1.0 / C / X.shape[0]

# class weights: use pos_weight in BCEWithLogitsLoss = weight_for_positive / weight_for_negative
pos_weight = torch.tensor(2.0/1.0, dtype=torch.float32, device=device)
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

opt = optim.SGD(model.parameters(), lr=1.0, weight_decay=weight_decay)

# Training loop
batch_size = 256
for epoch in range(200):
    perm = torch.randperm(len(X_t))
    for i in range(0, len(X_t), batch_size):
        idx = perm[i:i+batch_size]
        xb, yb = X_t[idx], y_t[idx]
        logits = model(xb)
        loss = criterion(logits, yb)
        opt.zero_grad()
        loss.backward()
        opt.step()

# Compare coefficients (adjust for scaler)
w_t = model.linear.weight.detach().cpu().numpy().reshape(-1)
b_t = model.linear.bias.item()
# sklearn coef corresponds to scaled features; direct compare:
print("sklearn coef:", clf.coef_.ravel(), "intercept:", clf.intercept_)
print("torch coef:", w_t, "intercept:", b_t)

Key points / reasoning:- BCEWithLogitsLoss expects logits; use pos_weight to reflect class_weight for positives.- L2: sklearn's objective differs by constants; exact mapping between weight_decay and sklearn's C depends on optimizer steps/batch size and loss scaling. Using weight_decay ≈ 1/(C*N) gives similar magnitude; tune if needed.- Verify coefficients after training—values should be close qualitatively; exact match requires matching solver (LBFGS) and regularization constant scaling exactly.

Unlock Full Question Bank

Get access to hundreds of Linear and Logistic Regression Implementation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.