Scikit Learn, Pandas, and NumPy Usage Questions

Practical proficiency with these core libraries. Pandas: DataFrames, data manipulation, handling missing values. NumPy: arrays, vectorized operations, mathematical functions. Scikit-learn: preprocessing, model fitting, evaluation metrics, pipelines. Knowing standard patterns and APIs. Writing efficient, readable code using these libraries.

MediumTechnical

62 practiced

For an imbalanced binary classification problem where the positive class is 1% of samples, explain why ROC AUC can be misleading. Using scikit-learn, write code to compute the precision-recall curve and average precision score from predicted probabilities and explain when PR AUC is preferred over ROC AUC.

Sample Answer

ROC AUC can be misleading with extreme class imbalance because it measures the trade-off between true positive rate (TPR) and false positive rate (FPR). When positives are only 1%, FPR changes are computed over a very large number of negatives, so a model that produces many false positives can still have a low FPR (and thus high ROC AUC) while precision (positive predictive value) is poor. ROC AUC therefore can hide that most predicted positives are wrong.

Use the precision-recall (PR) curve and average precision (AP) to focus on the performance for the positive class:

python

from sklearn.metrics import precision_recall_curve, average_precision_score
import matplotlib.pyplot as plt
import numpy as np

# y_true: binary ground truth (0/1), y_scores: predicted probabilities for positive class
# Example synthetic
rng = np.random.RandomState(0)
y_true = np.concatenate([np.ones(100), np.zeros(9900)])          # 1% positives
y_scores = np.concatenate([rng.uniform(0.6,1.0,100), rng.uniform(0,0.8,9900)])

precision, recall, thresholds = precision_recall_curve(y_true, y_scores)
avg_prec = average_precision_score(y_true, y_scores)

plt.plot(recall, precision, label=f'AP={avg_prec:.3f}')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall curve')
plt.legend()
plt.show()

print('Average precision:', avg_prec)

When to prefer PR AUC:- If the positive class is rare and the application cares about correctness of positive predictions (high precision) or recall (detecting positives).- When false positives are costly or when you must prioritize ranking positives higher.PR AUC emphasizes model performance on the positive class and better reflects practical usefulness under imbalance.

HardTechnical

61 practiced

Explain numpy.memmap and np.lib.stride_tricks.sliding_window_view. Given a very large 1D signal stored on disk, show how to compute sliding window features of width 100 without loading the entire signal into memory using memmap and sliding_window_view. Provide code and list caveats such as writeability and alignment.

Sample Answer

Approach: memory-map the large 1D file with numpy.memmap to avoid loading it fully, then create sliding windows via np.lib.stride_tricks.sliding_window_view. Because sliding_window_view returns a strided view (no copy), you must be careful about total output size, alignment/contiguity, and writeability. For very large signals, produce windows in manageable chunks.

Code (reads binary float32 signal on disk; computes mean and std per window in streaming chunks):

python

import numpy as np
from numpy.lib.stride_tricks import sliding_window_view

# parameters
path = "large_signal.bin"
dtype = np.float32
window = 100
chunk_windows = 1_000_000  # max windows to materialize at once

# memory-map the file (read-only)
sig = np.memmap(path, dtype=dtype, mode="r")  # 1D array

n = sig.shape[0]
if n < window:
    raise ValueError("signal shorter than window")

total_windows = n - window + 1

results_means = np.empty(total_windows, dtype=np.float64)
results_stds = np.empty(total_windows, dtype=np.float64)

i = 0
while i < total_windows:
    # compute how many windows to create in this chunk
    w = min(chunk_windows, total_windows - i)
    # we need a slice of the underlying signal that covers these windows:
    # slice start = i, slice stop = i + w + (window - 1)
    start = i
    stop = i + w + window - 1
    sig_slice = sig[start:stop+1]  # memmap slice (still a view)
    # sliding_window_view on the slice creates (w, window) view without copy
    win_view = sliding_window_view(sig_slice, window_shape=window)
    # compute features across axis 1
    results_means[i:i+w] = win_view.mean(axis=1)
    results_stds[i:i+w] = win_view.std(axis=1)
    i += w

Key points and caveats:- memmap returns ndarray-like object backed by file; mode="r" is read-only, "r+" allows writes which flush to disk.- sliding_window_view returns a view using strides (no copy). It relies on the underlying array being contiguous for intuitive behavior; slicing a memmap as above keeps correct strides.- The view may be non-writeable if the memmap is read-only; modifying the view will raise an error or attempt writes to disk when writable.- The total number of windows can be huge: avoid creating the full (n-window+1, window) array in memory — process in chunks as shown.- Alignment/stride pitfalls: if the file was created with a different dtype/byte order or you use structured dtypes, ensure dtype and endianess match.- On some platforms, memmap-backed arrays may not support all NumPy operations that require contiguous writable buffers; copying may be needed for such ops.- Performance: sliding_window_view is very memory-efficient and fast since it avoids copies, but heavy vectorized ops over huge views can still be limited by RAM for temporary arrays (e.g., calculating std creates temporaries). Use chunking to control peak memory.

MediumTechnical

53 practiced

You receive a DataFrame of 20 million rows where numeric columns are float64 and many string columns repeat values. Describe and show pandas code to reduce memory footprint: downcast numeric types, convert repeated strings to categorical dtype, and use sparse dtypes where appropriate. Include how to compute and report the estimated memory saving.

Sample Answer

Approach: measure current memory, downcast numeric columns (float64→float32 or smaller; ints to smallest int), convert high-cardinality repeated object/string columns to category, and convert very sparse numeric columns (many zeros/NaNs) to pandas sparse dtypes. Finally report estimated savings.

python

import pandas as pd
import numpy as np

def optimize_memory(df, verbose=True, sparse_threshold=0.9):
    start_mem = df.memory_usage(deep=True).sum()
    df_opt = df.copy()

    # 1) Downcast numeric types
    for col in df_opt.select_dtypes(include=['float64','int64']).columns:
        if pd.api.types.is_float_dtype(df_opt[col]):
            df_opt[col] = pd.to_numeric(df_opt[col], downcast='float')
        else:
            df_opt[col] = pd.to_numeric(df_opt[col], downcast='integer')

    # 2) Convert object (string) columns with many repeats to category
    for col in df_opt.select_dtypes(include=['object']).columns:
        n_unique = df_opt[col].nunique(dropna=False)
        n_total = len(df_opt[col])
        if n_unique / n_total < 0.5:  # tweak threshold as needed
            df_opt[col] = df_opt[col].astype('category')

    # 3) Convert very sparse numeric columns to SparseDtype
    for col in df_opt.select_dtypes(include=['number']).columns:
        ser = df_opt[col]
        # fraction of zeros or NaNs
        sparse_frac = ((ser == 0) | ser.isna()).sum() / len(ser)
        if sparse_frac >= sparse_threshold:
            # choose underlying dtype from current dtype
            underlying = ser.dtype
            df_opt[col] = pd.arrays.SparseArray(ser, fill_value=0).astype(pd.SparseDtype(underlying, 0))

    end_mem = df_opt.memory_usage(deep=True).sum()
    if verbose:
        print(f"Start memory: {start_mem/1024**2:.2f} MB")
        print(f"End memory:   {end_mem/1024**2:.2f} MB")
        print(f"Reduced by:   {(start_mem-end_mem)/start_mem*100:.1f}% ({(start_mem-end_mem)/1024**2:.2f} MB)")
    return df_opt

# Usage:
# df_optimized = optimize_memory(df)

Key points:- Use downcast='float' => float32, reduces ~50% for floats; integers similarly.- Converting repeated strings to category stores codes + small lookup, huge savings when repeats are common.- SparseDtype works well when a large fraction are zeros or a single fill value.- Compute memory via memory_usage(deep=True).sum() before and after to quantify savings.- Test thresholds (unique ratio, sparse_frac) on a sample of your data first.

EasyTechnical

54 practiced

Write Python code using scikit-learn to construct a Pipeline that scales numeric features with StandardScaler, fits a LogisticRegression with L2 penalty, and uses cross_val_score to evaluate ROC AUC. Explain why using a Pipeline is important when performing cross-validation and how it prevents data leakage.

Sample Answer

To build a safe ML workflow with scaling and logistic regression and evaluate with ROC AUC, use a scikit-learn Pipeline so all preprocessing is applied inside each CV fold (prevents data leakage).

python

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

# Example data (replace with your dataset)
X = np.random.randn(500, 10)
y = np.random.randint(0, 2, size=500)

# Pipeline: scaler -> logistic regression with L2 penalty (default)
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(penalty="l2", solver="liblinear", max_iter=1000))
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="roc_auc")
print("ROC AUC per fold:", scores)
print("Mean ROC AUC:", scores.mean())

Why Pipeline matters:- If you scale before cross-validation (fit scaler on full data), info from test folds leaks into training—optimistic, biased scores.- Pipeline ensures StandardScaler.fit is executed only on each training fold and then applied to that fold's validation data, preserving proper train/validation separation.- It centralizes preprocessing, reduces bugs, and makes hyperparameter tuning with GridSearchCV safe because transformations are included in the CV loop.

MediumTechnical

60 practiced

Compare SimpleImputer (mean, median, most_frequent), KNNImputer, and IterativeImputer in scikit-learn. For a dataset with mixed numeric and categorical features and approximately 10% MCAR missingness, which strategy would you choose and why? Include a short code snippet showing how to apply KNNImputer on numeric columns within a pipeline.

Sample Answer

Short comparison- SimpleImputer (mean/median/most_frequent): deterministic, very fast, good baseline. Mean/median preserve central tendency for numerics but ignore multivariate relationships; most_frequent works for categoricals but can distort distribution if missingness is large.- KNNImputer: fills missing numeric values using distance-weighted average of neighbors. Captures local multivariate structure (correlations) without fitting a parametric model. Sensitive to scaling and high-dimensional sparsity; slower than SimpleImputer but often better than univariate methods for moderate missingness.- IterativeImputer (MICE-like): fits a model per feature conditioning on others (e.g., BayesianRidge). Usually produces statistically coherent imputations and uncertainty if multiple imputations used, but is computationally heavier and requires careful model choice and convergence checks.

Which to choose for mixed features, ~10% MCAR- Strategy: use most_frequent SimpleImputer for categorical features and KNNImputer or IterativeImputer for numeric features.- Recommendation: KNNImputer is a pragmatic choice here: 10% MCAR is low, so KNN will leverage correlations without heavy modeling; ensure numeric features are scaled and include only relevant numeric predictors. Use IterativeImputer if you need better statistical fidelity or plan to perform multiple imputations for uncertainty quantification (at cost of compute).

Practical pipeline example (apply KNNImputer only to numeric columns)

python

from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import pandas as pd

numeric_cols = ['age','income','score']
cat_cols = ['gender','region']

numeric_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('knn_impute', KNNImputer(n_neighbors=5))
])

cat_pipeline = Pipeline([
    ('impute_freq', SimpleImputer(strategy='most_frequent'))
])

preprocessor = ColumnTransformer([
    ('num', numeric_pipeline, numeric_cols),
    ('cat', cat_pipeline, cat_cols)
])

# fit_transform on training data
df = pd.DataFrame(...)  # your data
X_processed = preprocessor.fit_transform(df)

Key practical notes- Always scale numerics before KNNImputer.- Encode categoricals after imputation (or use SimpleImputer then OneHotEncoder in pipeline).- Check post-imputation distributions and downstream model performance; run sensitivity checks with IterativeImputer if results differ.

Unlock Full Question Bank

Get access to hundreds of Scikit Learn, Pandas, and NumPy Usage interview questions and detailed answers.

Join thousands of developers preparing for their dream job.