Classification and Regression Fundamentals Questions

Covers the core concepts and distinctions between classification and regression in supervised learning. Classification predicts discrete categories, either binary or multi class, while regression predicts continuous numerical values. Candidates should understand how to format and encode target variables for each task, common algorithms for each family, and the theoretical foundations of representative models such as linear regression and logistic regression. For regression, know least squares estimation, coefficients interpretation, residual analysis, assumptions of the linear model, R squared, and common loss and error measures including mean squared error, root mean squared error, and mean absolute error. For classification, know logistic regression with its sigmoid transformation and probability interpretation, decision trees, k nearest neighbors, and other basic classifiers; understand loss functions such as cross entropy and evaluation metrics including accuracy, precision, recall, F one score, and area under the receiver operating characteristic curve. Also be prepared to discuss model selection, regularization techniques such as L one and L two regularization, handling class imbalance, calibration and probability outputs, feature preprocessing and encoding for targets and inputs, and trade offs when choosing approaches based on problem constraints and data characteristics.

EasyTechnical

29 practiced

Compare using mean squared error (MSE) and cross-entropy (log-loss) for classification tasks. Explain why cross-entropy is typically preferred for probabilistic classifiers, and when MSE might still be used or cause issues (e.g., saturation, poor gradients).

HardSystem Design

22 practiced

Design a scalable training and inference approach for a large regression model trained on 100M rows and 1,000 features (e.g., GBM). Discuss distributed training alternatives (XGBoost, LightGBM, Dask, Spark), feature storage strategies, incremental training, hyperparameter search at scale, and low-latency inference options (sharded serving, model quantization, caching).

Sample Answer

Requirements & constraints- Train GBM on 100M rows × 1,000 features; production needs low-latency inference (<= tens of ms), scalable hyperparameter search, incremental updates, and operational reliability.

High-level architecture- Offline data lake (Parquet/Delta Lake) for historic data + online feature store (Feast/Redis) for serving features.- Distributed trainer cluster (K8s) with compute backends (CPU/GPU) and a tuning orchestration layer (Ray/MLflow).- Model packaging via Treelite/ONNX and a sharded, autoscaled serving fleet with caching and monitoring.

Distributed training alternatives and choice guidance- XGBoost (Rabit or Dask/Ray integration): proven for large GBMs, supports GPU; good for moderate cluster sizes and deterministic training.- LightGBM: faster histogram-based training, supports MPI/Dask and GPU; efficient memory use for high-dimensional features.- Spark MLlib: good if your pipeline is Spark-native and data locality matters, but less performant than XGBoost/LightGBM for GBM accuracy/speed.- Dask: flexible orchestration layer to distribute LightGBM/XGBoost and integrate with Python ecosystem.Recommendation: use XGBoost/LightGBM on Dask/Ray for flexibility, with GPU acceleration if available. Use columnar input (Parquet) and tune worker-memory to fit histogram buckets.

Feature storage & serving- Offline store: partitioned Parquet/Delta for bulk training; materialize feature joins as precomputed wide tables to reduce ETL time.- Online store: low-latency key-value store (Redis or DynamoDB) populated from streaming pipelines (Kafka + Spark/Flink), served through Feast for consistency and lineage.- Keep feature derivation code in the store to reproduce training-serving parity; store feature metadata, statistics, and drift signals.

Incremental training strategy- Warm-start boosting: keep previous trees and continue training on new data (supported by LightGBM/XGBoost with booster.load/continued training) for frequent updates.- Mini-batch streaming retrain: accumulate daily deltas, retrain on sliding window (e.g., last N months) with importance sampling to keep dataset size manageable.- Periodic full retrain (weekly/monthly) to avoid drift accumulation; use validation windows and canary rollout.- Use incremental feature materialization (compute only changed partitions) to speed data prep.

Hyperparameter search at scale- Use distributed tuning frameworks: Ray Tune, Optuna with Dask/Ray or MLflow to parallelize trials.- Use multi-fidelity strategies: successive halving / ASHA / HyperBand to stop poor trials early; combine Bayesian optimization for sample efficiency.- Use smaller subsets and lower-precision (float16 / fewer rows / fewer trees) for early-stage search, then fine-tune best configs on full data.- Cache preprocessed datasets and use warm-start of models to reduce per-trial cost.

Low-latency inference options- Compile model with Treelite to produce a lightweight C/C++ runtime or export to ONNX and use an optimized runtime; both reduce latency vs Python predict.- Model quantization: convert leaf outputs/features to int8/float16 where safe to reduce model size and improve CPU inference.- Sharded serving: horizontally shard model instances by key-range with local feature caches; colocate Redis client to reduce network round-trip.- Batching & async: micro-batching for throughput if latency budget allows; use adaptive batching.- Caching: LRU cache for frequent keys; pre-warm cache for hot users.- Edge vs centralized: for ultra-low latency, embed compiled model in edge services; otherwise use regional serving with < RTT requirements.

Operational considerations & trade-offs- Monitoring: latency, throughput, feature drift, label drift, prediction distributions, and shadow canary tests.- Reproducibility: log training datasets & feature versions (with feature store), seed RNGs, model artifacts.- Cost-vs-latency: GPUs speed training and tuning but increase cost; Treelite+quantization reduces inference infra cost.- Complexity: Dask/Ray + distributed tuning adds orchestration burden; prefer managed services when available.

Example flow- Precompute features into Parquet partitions; train LightGBM on Dask cluster with GPU workers; use Ray Tune + ASHA to find hyperparams using successive halving (start with 10% data, escalate top trials).- Export best booster to Treelite, compile to shared lib, deploy to K8s as sharded gRPC service that looks up features from Redis/Feast with local LRU cache; use canary rollout and realtime monitors.

This design balances throughput for 100M×1k training, efficient hyperparameter search, frequent incremental updates, and sub-50ms production inference through model compilation, sharding, caching, and online feature stores.

MediumTechnical

22 practiced

Implement functions in Python to compute precision, recall, and F1 score for multi-class classification with options for micro, macro, and weighted averaging. Specify the function signature and input assumptions (lists/numpy arrays of integer labels). Explain complexity and how to handle zero-division when a class has no predicted samples.

Sample Answer

Approach: compute per-class true positives (TP), false positives (FP), false negatives (FN) from integer label arrays, then produce precision/recall/F1 per class and aggregate by micro/macro/weighted. Handle zero-division by defining precision/recall as 0.0 when denominator is 0 (common practice) and allow an optional parameter to set that fallback.

python

import numpy as np
from typing import Tuple, Iterable, Dict

def precision_recall_f1(
    y_true: Iterable[int],
    y_pred: Iterable[int],
    average: str = "macro",
    zero_division: float = 0.0
) -> Tuple[float, float, float]:
    """
    Compute precision, recall, f1 for multi-class labels.

    Inputs:
    - y_true, y_pred: lists or numpy arrays of integer labels (same length)
    - average: "micro", "macro", or "weighted"
    - zero_division: value to use when a division by zero occurs (e.g., 0.0 or 1.0)

    Returns: (precision, recall, f1)
    """
    y_true = np.asarray(y_true)
    y_pred = np.asarray(y_pred)
    if y_true.shape != y_pred.shape:
        raise ValueError("y_true and y_pred must have same shape")

    labels = np.union1d(y_true, y_pred)  # include classes seen only in preds or truths
    tp = {l: 0 for l in labels}
    fp = {l: 0 for l in labels}
    fn = {l: 0 for l in labels}

    for t, p in zip(y_true, y_pred):
        if t == p:
            tp[t] += 1
        else:
            fp[p] += 1
            fn[t] += 1

    # per-class metrics
    prec = {}
    rec = {}
    support = {}  # number of true instances per class
    for l in labels:
        support[l] = tp[l] + fn[l]
        denom_p = tp[l] + fp[l]
        denom_r = tp[l] + fn[l]
        prec[l] = (tp[l] / denom_p) if denom_p != 0 else zero_division
        rec[l] = (tp[l] / denom_r) if denom_r != 0 else zero_division

    if average == "micro":
        total_tp = sum(tp.values())
        total_fp = sum(fp.values())
        total_fn = sum(fn.values())
        precision = total_tp / (total_tp + total_fp) if (total_tp + total_fp) != 0 else zero_division
        recall = total_tp / (total_tp + total_fn) if (total_tp + total_fn) != 0 else zero_division
        f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) != 0 else zero_division
        return precision, recall, f1

    # macro or weighted
    if average == "macro":
        precision = float(np.mean(list(prec.values())))
        recall = float(np.mean(list(rec.values())))
    elif average == "weighted":
        weights = np.array([support[l] for l in labels], dtype=float)
        total = weights.sum()
        if total == 0:
            precision = recall = zero_division
        else:
            precision = float(np.sum([prec[l] * support[l] for l in labels]) / total)
            recall = float(np.sum([rec[l] * support[l] for l in labels]) / total)
    else:
        raise ValueError("average must be one of {'micro','macro','weighted'}")

    f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) != 0 else zero_division
    return precision, recall, f1

Key concepts:- Micro averages aggregate TP/FP/FN across classes (suitable when class imbalance and you care about overall instance-level performance).- Macro averages compute unweighted mean of per-class metrics (treats all classes equally).- Weighted uses class support to weight per-class metrics.

Complexity:- Time: O(n + k) where n = number of samples, k = number of distinct labels (dominant term O(n)).- Space: O(k) for per-class counters.

Zero-division handling:- If a class has no predicted samples, denom for precision is zero — we return zero_division (default 0.0). This mirrors scikit-learn behavior and avoids exceptions; optionally set zero_division=1.0 if you prefer a different convention.

MediumTechnical

41 practiced

List and explain key residual diagnostics for linear regression: residual vs fitted plot, QQ-plot for normality, Cook's distance for influential points, leverage, Durbin-Watson for autocorrelation, and Breusch-Pagan for heteroscedasticity. For each, describe what you would do if a problem is detected.

Sample Answer

Residual vs Fitted plot- What: Scatter of residuals (y - yhat) vs predicted values. Checks linearity, nonlinearity, heteroscedasticity, and model misspecification.- What to look for: Random cloud centered at 0. Pattern (curvature) → nonlinearity. Funnel shape → heteroscedasticity.- Fixes: If nonlinearity, add nonlinear terms (polynomials, splines) or use feature transforms (log, sqrt) or change model (tree-based, GAM). For heteroscedasticity, try variance-stabilizing transform of target or weighted least squares; use robust standard errors.

QQ-plot for normality of residuals- What: Plots standardized residual quantiles vs theoretical normal quantiles. Tests normality assumption for inference and CIs.- What to look for: Systematic deviations at tails → heavy/light tails; S-shaped → skewness.- Fixes: If non-normality, consider transforming target, using robust regression (Huber, RANSAC), or bootstrap CIs; for large samples, central limit theorem often mitigates concerns for coefficient estimates.

Cook’s distance (influential points)- What: Measures each point’s influence on fitted values/coefficients (combines residual and leverage).- What to look for: Points with Cook’s D substantially larger than others (rule of thumb >4/n).- Fixes: Inspect points for data errors; if valid but influential, consider robust regression, down-weighting, adding interaction/feature to explain, or report sensitivity analyses (fit with/without point).

Leverage (hat values)- What: Measures how far an observation’s predictors are from the mean of predictors; high leverage can disproportionately affect fit.- What to look for: Hat values > 2p/n or 3p/n (p = #params).- Fixes: Check for data entry or rare categorical levels; add features that explain the leverage; consider regularization or robust methods; if legitimate, document impact and run sensitivity checks.

Durbin–Watson for autocorrelation- What: Tests residual autocorrelation (esp. first-order) for ordered data/time series.- What to look for: DW ≈2 no autocorrelation; <2 positive autocorrelation; >2 negative.- Fixes: For autocorrelation, include lagged variables, use time-series models (ARIMA, GLS), use Newey-West (HAC) standard errors, or model temporal structure (state-space, recurrent nets).

Breusch–Pagan for heteroscedasticity- What: Tests whether residual variance depends on predictors.- What to look for: Significant p-value → heteroscedasticity.- Fixes: Use variance-stabilizing transforms (log), weighted least squares, heteroscedasticity-consistent (HC) standard errors, or model conditional variance explicitly (GAM, heteroscedastic regression).

General workflow and practical tips- Always visualize residuals before tests. Prioritize fixes based on goal: prediction (use flexible/robust models, cross-validate) vs inference (transformations, WLS, HC SEs). Document decisions and run sensitivity checks (re-fit after fixes, compare metrics). Automate diagnostics in pipelines and monitor in production for distribution shifts.

MediumTechnical

26 practiced

You must choose between a linear regression and a tree-based model (e.g., gradient boosted trees) for a regression problem with heavy-tailed errors and obvious nonlinearity in feature relationships. Describe factors that will influence your choice (interpretability, robustness to outliers, sample size, feature interactions) and propose data transformations or alternative losses you might try.

Sample Answer

Start by listing the problem signals: heavy‑tailed errors and clear nonlinearity. Those push toward nonparametric/robust methods, but final choice depends on tradeoffs.

Factors that influence choice- Predictive flexibility & interactions: Gradient boosted trees (GBMs) naturally capture nonlinearities and feature interactions without feature engineering; linear regression requires engineered interaction/polynomial terms.- Robustness to outliers: Trees are generally more robust to extreme y values and extreme x values (local splits). Linear regression with OLS is sensitive to outliers unless you change loss or preprocess.- Interpretability: Linear models win for global interpretability (coefficients), which matters if stakeholders need simple explanations. GBMs can be explained with SHAP/partial dependence but are more complex.- Sample size & overfitting: With small data, a simpler/regularized linear model may generalize better. GBMs need careful tuning (early stopping, regularization) but work well with moderate-to-large datasets.- Computational & deployment constraints: Linear models are cheaper to train/serve. GBMs are heavier but still production-friendly (LightGBM/XGBoost/CatBoost).

Practical transformations and alternatives- Target transforms: log(y) or Box‑Cox to compress heavy tails (be mindful of zero/negative values). Back-transform predictions carefully and correct for bias (use smearing / Jensen correction).- Robust losses for linear models: use Huber or Tukey’s biweight, or quantile regression if asymmetric errors matter. Use L1 (least absolute deviations) for median regression.- Robust estimation: RANSAC or M‑estimators can reduce influence of extreme points.- For GBMs: choose loss functions appropriate for heavy tails (Huber loss in XGBoost/LightGBM, Tweedie for compound Poisson–Gamma outcomes). Consider setting objective=quantile for conditional quantiles.- Outlier handling: winsorization, trimming, or model-based anomaly detection to downweight extreme examples rather than outright removal.- Feature engineering: add spline or polynomial terms if preferring linear model; for trees, ensure categorical encoding is appropriate (target encoding with regularization).- Evaluation: use robust metrics (MAE, median absolute error, quantile loss) and cross‑validation stratified by heavy-tail structure; inspect residuals, conditional heteroskedasticity.

Recommendation workflow1. Baseline: fit a regularized linear model (Ridge/Lasso) and robust linear (Huber or LAD) on transformed target to get interpretable baseline.2. Compare to tuned GBM (with robust loss / Tweedie or Huber) using CV and robust metrics (MAE, quantile losses).3. Use residual analysis and SHAP/PDP to verify patterns; if GBM substantially improves and stakeholder acceptability is OK, deploy GBM with monitoring. If interpretability or small-data risk dominates, stick with robust/transformed linear model.

This approach balances predictive performance, robustness, interpretability, and operational constraints.

Unlock Full Question Bank

Get access to hundreds of Classification and Regression Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.