Linear and Logistic Regression Implementation Questions

Covers the fundamentals and implementation details of linear regression for continuous prediction and logistic regression for binary or multiclass classification. Candidates should understand model formulation, hypothesis functions, and the intuition behind fitting a line or hyperplane for regression and using a sigmoid or softmax function for classification. Include loss functions such as mean squared error for regression and cross entropy loss for classification, optimization methods including gradient descent and variants, regularization techniques, feature engineering and scaling, metrics for evaluation such as mean absolute error and accuracy and area under curve, and hyperparameter selection and validation strategies. Expect discussion of practical implementation using numerical libraries and machine learning toolkits, trade offs and limitations of each approach, numerical stability, and common pitfalls such as underfitting and overfitting.

MediumTechnical

54 practiced

You have a binary classification problem with 0.5% positive class. Discuss strategies for training and evaluating logistic regression under severe class imbalance: resampling (oversample, undersample, SMOTE), class weighting in loss, threshold adjustment, metrics to optimize, and specialized loss functions like focal loss. Explain operational trade-offs for each approach.

Sample Answer

Start by framing the problem: positives are 0.5% — extremely rare — so naive accuracy is meaningless and model must focus on recall/precision trade-offs and operational cost of false positives vs false negatives.

Resampling- Oversample positives (random or SMOTE): increases signal and class balance for training; SMOTE generates synthetic examples to reduce overfitting vs simple duplication. Risk: synthetic points may be unrealistic; can leak when features include IDs/time.- Undersample negatives: speeds training and reduces class skew but risks discarding useful negative variety and hurting generalization. Best for very large negative pools where negatives are redundant.Operational trade-off: resampling changes training distribution — always validate on realistic (original) distribution.

Class weighting in loss- Weight positive class higher (inverse frequency) so logistic regression penalizes missed positives more. Simple, preserves original sample distribution, works well with scikit-learn’s class_weight or custom loss.Trade-off: can increase false positives; tuning required to reflect real costs.

Threshold adjustment- Train for calibrated probabilities; then choose decision threshold based on target metric (e.g., maximize F1, or set threshold to meet target recall while controlling precision). Use ROC/PR curves and precision@k. Operational: keeps probabilities for downstream ranking and supports different business constraints without retraining.

Metrics to optimize / evaluate- Prefer Precision-Recall curve, average precision, precision@k, and recall at acceptable precision. Use confusion-matrix derived costs or expected value (profit) to choose threshold. Report calibration (Brier), and use stratified cross-validation with preserved class ratios or time-based splits.

Specialized loss functions (focal loss)- Focal loss down-weights easy negatives and focuses gradient on rare hard positives; useful when positives are extremely rare and many negatives are trivial. More common with deep nets than logistic regression, but can be adapted.Trade-offs: introduces hyperparameters (gamma), may complicate optimization and probability calibration.

Practical recipe1. Start with class-weighted logistic regression + proper calibration (Platt/ isotonic).2. Evaluate using PR curve, precision@k, and business-cost function on holdout with original imbalance.3. If model underfits positives, try SMOTE or targeted oversampling of hard positives (not random).4. For more complex models, consider focal loss or ensemble methods (stacking, anomaly detection for candidate generation).5. Monitor in production for data drift; maintain sampling for periodic re-training and maintain operating point via thresholding.

Key principle: keep training interventions to improve signal but always evaluate decisions on the real-world class distribution and business cost function.

MediumTechnical

64 practiced

You have a loop-based Python implementation of logistic regression that is too slow on 1M samples. Describe how you would profile the code, identify hotspots, and optimize it for speed. Include concrete optimizations: vectorization, memory layout, using BLAS-backed operations, dtype choices, batching, and when to move to GPU or PyTorch for further performance gains.

Sample Answer

Start by profiling to find hotspots (not guessing):- Run a statistical profiler: cProfile + pstats to see function-level cost.- Use line_profiler (@profile) for per-line timing inside the training loop.- Use memory_profiler if swapping or memory allocs are suspected.- Check system metrics (top, iostat) to see CPU vs I/O bottleneck.

Typical findings: Python loops (per-sample sigmoid, dot products, weight updates) and frequent allocations.

Concrete optimizations (practical steps and why):

1) Vectorize with NumPy (use BLAS-backed ops)- Replace per-sample loop with matrix ops so work is done in C/Fortran (GEMM).- Use np.dot / @ for X.T @ (sigmoid - y); these call optimized BLAS/LAPACK.

python

# Vectorized epoch (batch or full-batch)
def sigmoid(z): return 1.0 / (1.0 + np.exp(-z))
for epoch in range(epochs):
    logits = X @ w + b           # single big matmul (BLAS)
    probs = sigmoid(logits)      # elementwise in C
    grad_w = X.T @ (probs - y) / n
    w -= lr * grad_w

2) Memory layout and dtypes- Ensure X is np.ascontiguousarray(X, dtype=np.float32) to reduce memory and increase throughput.- float32 often sufficient; halves memory, improves cache and BLAS throughput on modern CPUs/GPUs.- Avoid repeated temporaries: reuse buffers or in-place ops (e.g., probs -= y).

3) Mini-batching- If data doesn't fit or to improve cache behavior, process in large batches (e.g., 16k–64k rows) rather than per-sample.- Use reshaped blocks to leverage GEMM.

4) Reduce Python overhead- Move inner loops entirely into NumPy; if non-vectorizable logic remains, use Numba @njit or Cython for compiled speed.

5) Parallelism & BLAS tuning- Ensure NumPy links to an optimized BLAS (OpenBLAS, MKL). Set OMP_NUM_THREADS appropriately.- Use threadpoolctl to set BLAS threads to avoid oversubscription.

6) When to move to GPU / PyTorch- If matrix sizes are large and BLAS CPU performance plateaued, move model to PyTorch/TensorFlow to leverage cuBLAS/cuDNN.- Use PyTorch DataLoader for efficient batching, pin_memory, and mixed precision (AMP) to speed up training.- GPU beneficial when matrix ops dominate and data transfer overhead is amortized by large batches.

7) Additional tips- Profile after each change. Microbenchmarks: timeit, nvprof for GPU.- Monitor numerical stability (use log-sum-exp where needed).- For inference, consider quantization or ONNX runtime for production speed.

This approach—measure, vectorize, optimize memory and dtype, use optimized BLAS, then scale to GPU/accelerated frameworks—typically reduces training time from loop-bound seconds-per-sample to minutes for 1M samples.

MediumTechnical

53 practiced

Derive the gradient of the cross entropy loss for the softmax multinomial logistic regression model. Show the derivation from the log-likelihood and express the result in vectorized form for a batch: how the gradient reduces to X^T (P - Y) where P are predicted softmax probabilities and Y are one-hot labels.

HardTechnical

52 practiced

Prove that the logistic (cross-entropy) loss for binary classification is convex in the model parameters. Provide the Hessian form and show it is positive semidefinite, and discuss the practical implications of convexity for optimization and global optimality. Also explain how adding L2 regularization affects convexity.

EasyTechnical

61 practiced

Implement a numerically stable sigmoid function in Python accepting NumPy arrays. Your implementation should avoid overflow for large positive or negative inputs and return probabilities in [0,1]. Include quick checks for inputs like 1000 and -1000 to demonstrate stability.

Sample Answer

A numerically stable sigmoid computes 1 / (1 + exp(-x)) but avoids overflow when x is very large (exp(-x) underflows) or very negative (exp(x) overflows). The usual stable trick is branch on sign of x:

- if x >= 0: sigmoid = 1 / (1 + exp(-x)) (safe because -x <= 0)- else: sigmoid = exp(x) / (1 + exp(x)) (safe because x <= 0)

This is vectorizable with NumPy using np.where.

python

import numpy as np

def sigmoid(x):
    """
    Numerically stable sigmoid for NumPy arrays.
    Returns values in (0,1) for finite inputs; preserves NaN and inf semantics.
    """
    x = np.asarray(x, dtype=np.float64)
    positive = x >= 0
    out = np.empty_like(x)
    # For positive x: 1 / (1 + exp(-x))
    out[positive] = 1.0 / (1.0 + np.exp(-x[positive]))
    # For negative x: exp(x) / (1 + exp(x))
    exp_x = np.exp(x[~positive])
    out[~positive] = exp_x / (1.0 + exp_x)
    return out

Key points:- Vectorized and uses float64 for precision.- Time complexity O(n), space O(n) for output.- Handles large magnitudes without overflow.

Quick checks:

python

print(sigmoid(np.array([1000.0])))   # -> array([1.0]) (no overflow)
print(sigmoid(np.array([-1000.0])))  # -> array([0.0]) (no underflow to -inf)
print(sigmoid(np.array([0.0, 1.0, -1.0, np.nan, np.inf, -np.inf])))

Edge cases:- NaN remains NaN; positive infinity yields 1.0, negative infinity yields 0.0.- If lower precision is required, cast accordingly but watch dynamic range.

Unlock Full Question Bank

Get access to hundreds of Linear and Logistic Regression Implementation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.