Linear and Logistic Regression Implementation Questions

Covers the fundamentals and implementation details of linear regression for continuous prediction and logistic regression for binary or multiclass classification. Candidates should understand model formulation, hypothesis functions, and the intuition behind fitting a line or hyperplane for regression and using a sigmoid or softmax function for classification. Include loss functions such as mean squared error for regression and cross entropy loss for classification, optimization methods including gradient descent and variants, regularization techniques, feature engineering and scaling, metrics for evaluation such as mean absolute error and accuracy and area under curve, and hyperparameter selection and validation strategies. Expect discussion of practical implementation using numerical libraries and machine learning toolkits, trade offs and limitations of each approach, numerical stability, and common pitfalls such as underfitting and overfitting.

HardTechnical

49 practiced

A logistic regression model for 90-day customer retention shows degrading performance due to temporal drift. Propose step-by-step remediation covering data partitioning for time-aware validation, feature engineering (lags, rolling statistics, exponential decay), retraining frequency, and how to evaluate whether retraining improved production performance.

Sample Answer

Step 1 — Clarify scope & metrics- Define target (90-day retention), business SLA, success metric (AUC, PR-AUC, calibration, expected vs actual retention), and acceptable degradation threshold.

Step 2 — Time-aware data partitioning- Use a rolling-origin (time-series cross-validation). Example: train on months [T0..T6], validate on T7; then train [T1..T7], validate T8, etc. Ensure no leakage: features only use data available at prediction time.- Hold out a chronologically later test set (e.g., most recent 3 months) for final evaluation.

Step 3 — Feature engineering for temporal drift- Add lags: include behavior/features at t-1, t-7, t-30 depending on resolution.- Rolling statistics: mean, std, min, max, count over windows (7/30/90 days).- Exponential decay features: weighted sums where weight = alpha*(1-alpha)^(age) to emphasize recent activity.- Time-since events: days since last purchase/login.- Interaction with calendar features (seasonality, promotions).- Monitor feature distributions (KS, population stability index) to detect drift.

Step 4 — Retraining strategy & frequency- Start with periodic retraining: weekly or biweekly if user behavior changes fast; monthly if slower. Use validation curve from rolling CV to choose cadence.- Consider incremental updates: warm-start coefficients or online learning for logistic regression (e.g., SGD with partial_fit).- Maintain model registry with versioning and metadata (training window, data stats).

Step 5 — Validation & deployment checks- Before deployment, compare rolling-CV performance vs current prod: uplift in AUC and calibration.- Backtest: simulate production by scoring holdout period with model trained on prior window; compare predicted vs observed retention.

Step 6 — Production monitoring & evaluation- Real-time: track model-level metrics (AUC on labeled streaming sample, calibration, predicted retention vs realized retention), feature distribution drift alerts (PSI), and business KPIs (actual 90-day retention).- A/B test new model vs incumbent (canary): randomize users, compare uplift in retention, lift in retention rate, and secondary metrics (LTV, churn).- Statistical test: use survival analysis or proportion tests to confirm significant improvement; check calibration plots.

Step 7 — Post-retraining analysis & guardrails- If retrained model improves rolling-CV but not prod, investigate label delays, sampling bias, or feature pipeline mismatch.- Automate retrain triggers based on drift thresholds and hold manual review for large shifts.

This sequence ensures time-aware validation, temporally robust features, an appropriate retrain cadence, and rigorous evaluation tying model metrics to real production retention.

MediumTechnical

54 practiced

Explain numerical issues when computing softmax and cross-entropy for multiclass logistic regression. Implement a numerically stable softmax function and show how to compute log-softmax and cross-entropy loss with the log-sum-exp trick in Python/NumPy to avoid overflow and loss of precision.

Sample Answer

Computing softmax and cross-entropy naively causes numerical issues: exponentials of large logits overflow, very negative logits underflow to 0, and subtracting nearly equal large numbers causes loss of precision. These lead to NaNs/Inf in loss and unstable gradients. The standard fix is the log-sum-exp trick: subtract the maximum logit per example before exponentiating, compute log-sum-exp in a stable way, and use log-softmax for loss.

Approach:1. Stable softmax: subtract max per row, exponentiate, normalize.2. Log-softmax: compute logits - max - log(sum(exp(...))).3. Cross-entropy: use negative log-softmax of the true class (works for one-hot or class indices).

python

import numpy as np

def stable_softmax(logits):
    """
    logits: shape (N, C) or (C,) for one sample
    returns: shape (N, C) softmax probabilities
    """
    logits = np.atleast_2d(logits)  # ensure 2D
    # subtract max per row for numerical stability
    max_logits = np.max(logits, axis=1, keepdims=True)
    exps = np.exp(logits - max_logits)
    probs = exps / np.sum(exps, axis=1, keepdims=True)
    return probs if probs.shape[0] > 1 else probs[0]

def log_softmax(logits):
    """
    Returns stable log-softmax: log(softmax(logits))
    """
    logits = np.atleast_2d(logits)
    max_logits = np.max(logits, axis=1, keepdims=True)
    shifted = logits - max_logits
    # log-sum-exp
    lse = np.log(np.sum(np.exp(shifted), axis=1, keepdims=True))
    log_probs = shifted - lse
    return log_probs if log_probs.shape[0] > 1 else log_probs[0]

def cross_entropy_loss(logits, targets):
    """
    logits: shape (N, C)
    targets: either shape (N,) with integer class indices or (N, C) one-hot
    returns: scalar average loss
    """
    logits = np.atleast_2d(logits)
    log_probs = log_softmax(logits)  # stable
    if targets.ndim == 1:
        # class indices
        N = logits.shape[0]
        loss = -np.mean(log_probs[np.arange(N), targets])
    else:
        # one-hot targets
        loss = -np.mean(np.sum(targets * log_probs, axis=1))
    return loss

Key points:- Subtracting max(logits) prevents large exponentials.- log_softmax yields stable log-probs for direct use in cross-entropy.- Time complexity: O(N*C). Space: O(N*C) for temporary arrays.Edge cases:- Single-sample inputs (handled by atleast_2d and returning 1D).- Very large/small logits handled by subtraction.- Avoid computing softmax then log when computing loss — use log-softmax to prevent precision loss.

MediumTechnical

47 practiced

Describe residual analysis for a fitted linear regression model: how to use residual vs fitted plots, QQ-plots, Cook's distance, and leverage to detect non-linearity, non-normality, heteroscedasticity, and influential observations. Recommend remediation steps for common issues.

Sample Answer

Residual analysis is how you check whether a fitted linear regression meets assumptions (linearity, independence, homoscedasticity, normality of errors) and to find influential points. Key diagnostic plots and measures:

Residuals vs Fitted plot- What to look for: scatter of residuals around zero with no pattern.- Detects: non-linearity (systematic curve), heteroscedasticity (funnel shape).- Remediation: if non-linearity, add transformed predictors (log, polynomial, splines) or use generalized additive models; if heteroscedasticity, try variance-stabilizing transforms (log/y-root), weighted least squares, or robust standard errors.

QQ-plot of residuals- What to look for: residuals should fall roughly on the 45° line.- Detects: non-normality (heavy tails, skewness).- Remediation: consider transforming response, use robust regression or bootstrap confidence intervals; for large samples, non-normality mainly affects small-sample inference.

Leverage and Cook’s distance- Leverage: measures how far an observation’s predictor values are from the mean predictor cloud (high leverage can unduly influence fit).- Cook’s distance: combines leverage and residual size to quantify influence.- What to look for: points with high leverage and large Cook’s D (rule of thumb: Cook’s D > 4/n).- Remediation: inspect data entry/measurement errors, consider re-fitting without the point to assess impact, use robust regression (e.g., Huber, RANSAC), or model interactions/heterogeneous effects if point is legitimate.

Other checks and best practices- Scale-location plot (square-root standardized residuals vs fitted) for heteroscedasticity.- Residual autocorrelation (Durbin-Watson, ACF) for time series—use AR errors or GLS.- Report diagnostics and sensitivity analyses to stakeholders; prefer approaches that preserve valid inference (robust SEs, WLS) rather than deleting data without justification.

Always combine visual diagnostics, statistical tests, and domain knowledge before changing the model.

MediumTechnical

63 practiced

Implement mini-batch stochastic gradient descent for binary logistic regression in Python/NumPy. Include shuffling each epoch, L2 regularization (excluding bias), and a decaying learning rate schedule. Provide code outline and explain how you would vectorize updates for mini-batches.

Sample Answer

Approach: Use NumPy to implement mini-batch SGD for binary logistic regression. Each epoch: shuffle data, split into mini-batches, compute vectorized predictions and gradients for each batch, apply L2 regularization only to weights (not bias), and update parameters with a decaying learning rate (e.g., lr0 / (1 + decay * epoch)).

python

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def mini_batch_sgd(X, y, epochs=50, batch_size=64, lr0=0.1, decay=0.01, l2=1e-4, seed=None):
    """
    X: (n_samples, n_features)
    y: (n_samples,) with values 0 or 1
    """
    rng = np.random.default_rng(seed)
    n, d = X.shape
    # initialize weights and bias
    w = np.zeros(d)
    b = 0.0

    for epoch in range(epochs):
        # decaying learning rate
        lr = lr0 / (1 + decay * epoch)

        # shuffle
        perm = rng.permutation(n)
        X_shuffled = X[perm]
        y_shuffled = y[perm]

        # process mini-batches
        for i in range(0, n, batch_size):
            Xb = X_shuffled[i:i+batch_size]           # (m, d)
            yb = y_shuffled[i:i+batch_size]           # (m,)

            m = Xb.shape[0]
            # vectorized forward pass
            preds = sigmoid(Xb.dot(w) + b)            # (m,)
            # gradient of loss wrt predictions (binary cross-entropy)
            error = preds - yb                        # (m,)
            # vectorized gradients
            grad_w = (Xb.T.dot(error)) / m + l2 * w   # (d,)
            grad_b = error.mean()                     # scalar (no regularization on bias)

            # parameter update
            w -= lr * grad_w
            b -= lr * grad_b

    return w, b

Key points / Vectorization:- Predictions computed as sigmoid(Xb.dot(w) + b) avoids loops.- Weight gradient uses Xb.T.dot(error) (sum over examples) then averaged — this vectorizes computation over the mini-batch.- L2 regularization added as l2 * w; bias excluded.- Shuffling via a random permutation each epoch ensures samples seen in random order.

Complexity:- Per epoch cost O(n * d). Memory O(d) besides data.Edge cases:- Very small batch_size (1) yields true SGD; batch_size >= n is full-batch gradient descent.- Watch for numerical overflow in sigmoid; consider clipping z.

HardTechnical

49 practiced

Explain logistic regression as a generalized linear model (GLM) with canonical logit link. Discuss the score function, Fisher information matrix, and how deviance is used as a goodness-of-fit measure. Contrast likelihood-based inference in GLMs with penalized maximum likelihood used in machine learning practice.

Unlock Full Question Bank

Get access to hundreds of Linear and Logistic Regression Implementation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.