Scikit Learn, Pandas, and NumPy Usage Questions

Practical proficiency with these core libraries. Pandas: DataFrames, data manipulation, handling missing values. NumPy: arrays, vectorized operations, mathematical functions. Scikit-learn: preprocessing, model fitting, evaluation metrics, pipelines. Knowing standard patterns and APIs. Writing efficient, readable code using these libraries.

HardTechnical

111 practiced

You have a long-running pandas preprocessing step that's the bottleneck in a pipeline. Explain how you would profile the code to find hotspots (CPU and memory), and give concrete optimizations such as using vectorized operations, replacing apply with Cython/numba, or using chunked processing with generators. Mention tools like %timeit, line_profiler, and memory_profiler.

Sample Answer

Start by measuring where time and memory go, then iterate with targeted optimizations.

Profiling (CPU):- Quick microbench: %timeit for small-snippet comparisons in a notebook.- Line-level: line_profiler (kernprof -l) to find expensive lines inside functions.- Whole-program: cProfile + snakeviz for call-level hotspots.Example:

python

# in notebook
%timeit preprocess(df)
# or
!kernprof -l -v script.py  # uses @profile from line_profiler

Profiling (memory):- memory_profiler (@profile decorator, mprof) for per-line memory.- tracemalloc for allocation tracing.- pandas: df.info(memory_usage='deep'), df.memory_usage(deep=True).

Common hotspots & concrete fixes:1. Replace apply/iterrows:- Vectorize with pandas/numpy (broadcast ops) whenever possible.- Use .loc/.at for targeted assignments.- Example: avoid df['x'] = df.apply(lambda r: r.a + r.b, axis=1) → df['x'] = df.a + df.b

2. Optimize dtypes / memory:- Downcast ints/floats: pd.to_numeric(..., downcast='integer')- Convert low-cardinality strings to categorical.

3. Avoid copies and chained indexing:- Use inplace where safe, or reassign a single new df.- Use df = df.drop(columns=...) rather than many intermediate copies.

4. Replace Python loops / heavy apply with compiled code:- Numba JIT for numerical row-wise ops (fast, minimal code change).- Cython for complex kernels if numba can't handle types.Example numba:

python

from numba import njit
@njit
def sum_arrays(a, b):
    out = np.empty_like(a)
    for i in range(a.size):
        out[i] = a[i] + b[i]
    return out
df['x'] = sum_arrays(df['a'].values, df['b'].values)

5. Chunked processing / streaming:- Read/process in chunks: pd.read_csv(..., chunksize=100_000) with generator pattern to reduce peak memory and allow pipelined transformations.- Write intermediate results to parquet or use Dask for out-of-core parallelism.

6. Parallelism and tooling:- Dask / modin / vaex for larger-than-memory or easy parallelism.- Multiprocessing for independent groups (groupby.apply parallel), mindful of serialization cost.- Use swifter/pandarallel for easy parallel apply (benchmark first).

Workflow example:- Profile to find offending line.- Try vectorized rewrite; if not possible, implement numba/Cython kernel.- Reduce dtype sizes + chunking to lower memory.- Re-profile to confirm gains; iterate.

Edge considerations: correctness with NaNs/categoricals, I/O overhead, serialization when parallelizing, and reproducible benchmarks (warm-up runs, same data).

HardTechnical

65 practiced

Explain how feature selection inside cross-validation can leak information if done incorrectly. Show how to use sklearn Pipeline to include feature selection (e.g., SelectKBest or a model-based selector) safely inside cross-validation so the selection is performed within each fold only.

Sample Answer

Feature selection must be treated as part of the model training process. If you select features on the whole dataset before cross-validation (or test set), you leak information: selection statistics (e.g., correlations, ANOVA F-scores) were computed using data that the CV fold should consider "unseen", so downstream metrics are optimistically biased.

Correct approach: include feature selection inside the CV loop. In scikit-learn this is done by composing a Pipeline that contains the selector and the estimator; then run cross_val_score or GridSearchCV on the pipeline. That ensures for each fold the selector is fit only on that fold's training split and applied to the fold's validation split.

Example with SelectKBest:

python

from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np

pipe = Pipeline([
    ('selector', SelectKBest(score_func=f_classif, k=10)),
    ('clf', RandomForestClassifier(n_estimators=100, random_state=0))
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipe, X, y, cv=cv, scoring='accuracy')
print(np.mean(scores), np.std(scores))

For model-based selection use SelectFromModel inside the pipeline:

python

from sklearn.feature_selection import SelectFromModel
pipe = Pipeline([
    ('sfm', SelectFromModel(RandomForestClassifier(n_estimators=200))),
    ('clf', RandomForestClassifier(n_estimators=100))
])

Notes and best practices:- Never fit SelectKBest or scaler on full data before CV.- Use ColumnTransformer inside Pipeline if you need per-column transforms.- When tuning k or selector hyperparameters, nest selection in GridSearchCV that wraps the pipeline to avoid double-dipping.- Validate final performance on a held-out test set not used during CV.

MediumTechnical

64 practiced

Describe how to use sklearn.feature_extraction.text.HashingVectorizer vs TfidfVectorizer. Explain trade-offs with respect to memory, collision risk, ability to inverse-transform, and suitability for production pipelines that need to remain constant between train and inference.

Sample Answer

Definition & primary difference:- HashingVectorizer maps tokens to a fixed-size integer feature index via a hash function (no vocabulary stored). TfidfVectorizer builds an explicit vocabulary and computes TF-IDF weights using token counts and IDF statistics.

Memory:- HashingVectorizer: constant, small memory footprint because you only store the fixed hash dimension (e.g., n_features=2^20). Good for high-dimensional streaming or very large corpora.- TfidfVectorizer: memory grows with vocabulary size and requires storing the vocabulary and IDF vector.

Collision risk:- HashingVectorizer: collisions inevitable — different tokens can map to same index. Collision rate decreases with larger n_features but never zero. Collisions can inject noise and reduce interpretability or performance on sparse, small-signal features.- TfidfVectorizer: no collisions (each token has unique index).

Inverse-transform & interpretability:- HashingVectorizer: cannot reliably inverse-transform because original tokens are lost (unless you maintain external maps). Limited feature interpretability.- TfidfVectorizer: supports inverse-transform via stored vocabulary and allows feature inspection, useful for debugging and explainability.

Suitability for production pipelines (train vs inference constancy):- HashingVectorizer: deterministic given same hash function and n_features, so it yields consistent feature mapping between train and inference without shipping a vocabulary — good for lightweight deployments. But collisions can differ in effect if token distributions shift.- TfidfVectorizer: to ensure exact train/inference parity you must serialize and deploy the fitted vectorizer (vocabulary + IDF). This guarantees no mapping drift and preserves IDF scaling; recommended when explainability and reproducibility are required.

When to use which:- Use HashingVectorizer when memory/latency constraints, huge or streaming vocabularies, and you accept some collision noise.- Use TfidfVectorizer when you need interpretability, no collisions, or IDF-based weighting consistent with train data — and you can persist the fitted transformer.

Best practices:- If using HashingVectorizer, choose a large n_features, consider signed hashing (to reduce bias), and possibly pair with a FeatureHasher + downstream normalization. Monitor for feature drift.- If using TfidfVectorizer in production, persist the fitted object (pickle or MLflow) and include it in model versioning/deployment to keep train/inference behavior identical.

HardTechnical

66 practiced

Demonstrate how to use NumPy advanced indexing and broadcasting to normalize rows of a 2D array so each row has zero mean and unit variance (per-row standardization), without using explicit Python loops. Show code and explain in-place vs. copy behavior and memory implications.

Sample Answer

Approach: compute per-row mean and std with keepdims so shapes align (n_rows, 1), then use vectorized broadcasting to subtract and divide. Avoid explicit Python loops. Handle zero-std rows to prevent division by zero. Decide between creating a normalized copy (safe, simpler) or normalizing in-place to save memory and avoid temporaries.

python

import numpy as np

def standardize_rows(X, eps=1e-8, inplace=False):
    """
    Per-row standardization: each row -> zero mean, unit variance.
    X: 2D numpy array (n_rows, n_cols)
    eps: small value or used to replace zero std
    inplace: if True, modify X in-place and return X; else return a new array
    """
    if not isinstance(X, np.ndarray) or X.ndim != 2:
        raise ValueError("X must be a 2D numpy array")

    # compute row means and stds as column vectors (n,1) to broadcast across columns
    means = X.mean(axis=1, keepdims=True)            # shape (n_rows, 1)
    stds = X.std(axis=1, keepdims=True)              # shape (n_rows, 1)

    # avoid divide-by-zero: set zeros to 1 (or add eps)
    zero_mask = (stds == 0)
    if np.any(zero_mask):
        stds = np.where(zero_mask, 1.0, stds)

    if inplace:
        # subtract means in-place (broadcasting of (n,1) across columns)
        X -= means               # no full (n,m) temporary for means; writes directly into X
        X /= stds                # divides rows in-place; avoids creating full copy
        return X
    else:
        # this creates a temporary full (n,m) array for the result
        return (X - means) / stds

Key points / reasoning:- keepdims=True makes means and stds shape (n_rows,1), so broadcasting across columns is automatic and explicit.- Non-inplace (copy) version uses (X - means) / stds, which allocates a full temporary result of size n*m.- Inplace operations (X -= means; X /= stds) avoid allocating the full output temporary and write into X directly. Broadcasting of the small (n,1) arrays does not expand into a full copy; NumPy applies the broadcasted RHS while writing into X, so memory cost is lower.- Caveat: some NumPy ufuncs may create temporaries for complex dtypes or when strides require it; in-place is best for memory but mutates original data (dangerous if original needed later).- Time complexity: O(n*m). Memory: non-inplace uses O(n*m) extra for result; inplace uses O(n + m?) negligible extra (means/std arrays of size n). For extremely large arrays prefer inplace or chunked processing.- Edge cases: rows with constant values (std=0) handled by replacing std with 1 to yield zeroed row; consider using eps or masking if different behavior desired.

HardTechnical

62 practiced

Describe practical approaches to time-series cross-validation using pandas and scikit-learn. Explain TimeSeriesSplit behavior, how to include expanding windows or rolling windows, and how to incorporate feature engineering steps (lag features) without leaking future information.

Sample Answer

Brief approach: use scikit-learn's TimeSeriesSplit for walk-forward validation, and implement expanding or rolling windows by controlling the train slice per split. Crucially, create lag/rolling features inside a transformer that only uses training data to avoid leakage — store required history on fit and use it to compute test features.

Example: TimeSeriesSplit (expanding) and a custom LagFeatureTransformer that preserves trailing history.

python

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit
from sklearn.base import BaseEstimator, TransformerMixin

class LagFeatureTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, target_col, lags=3):
        self.target_col = target_col
        self.lags = lags
        self.tail_ = None  # will hold last (lags) rows of training series

    def fit(self, X, y=None):
        s = X[self.target_col].astype(float)
        self.tail_ = s.iloc[-self.lags:].copy() if len(s) >= self.lags else s.copy()
        return self

    def transform(self, X):
        s = X[self.target_col].astype(float).copy()
        # prepend training tail so shifts have prior values, then drop prepended rows
        pad = pd.concat([self.tail_, s])
        df_lags = pd.concat({f"lag_{i}": pad.shift(i) for i in range(1, self.lags+1)}, axis=1)
        df_lags = df_lags.iloc[len(self.tail_):].reset_index(drop=True)
        out = pd.concat([s.reset_index(drop=True), df_lags], axis=1)
        return out.drop(columns=[self.target_col])

# usage: perform manual walk-forward using TimeSeriesSplit indices
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(df):
    X_train, X_test = df.iloc[train_idx], df.iloc[test_idx]
    lagger = LagFeatureTransformer(target_col='y', lags=3).fit(X_train)
    X_train_lags = lagger.transform(X_train)
    X_test_lags  = lagger.transform(X_test)
    # train model on X_train_lags, evaluate on X_test_lags

Key points:- TimeSeriesSplit yields expanding train windows by default; set max_train_size to create rolling (fixed-size) training windows.- Avoid computing lags on the full dataset before splitting — that leaks future information.- Your transformer should capture only training history (tail) on fit and use it to compute test lags.- For other featurizations (rolling stats, target encodings), compute aggregations during fit on training data and apply to test; store necessary aggregates or trailing windows.- Consider pipeline + custom transformers so grid-search respects temporal separation; do not shuffle.- Evaluate with proper metrics over splits and inspect stability (bias/variance) across folds.

Unlock Full Question Bank

Get access to hundreds of Scikit Learn, Pandas, and NumPy Usage interview questions and detailed answers.

Join thousands of developers preparing for their dream job.