Scikit Learn, Pandas, and NumPy Usage Questions

Practical proficiency with these core libraries. Pandas: DataFrames, data manipulation, handling missing values. NumPy: arrays, vectorized operations, mathematical functions. Scikit-learn: preprocessing, model fitting, evaluation metrics, pipelines. Knowing standard patterns and APIs. Writing efficient, readable code using these libraries.

MediumTechnical

72 practiced

Using RandomizedSearchCV, show how to tune hyperparameters of a pipeline consisting of TfidfVectorizer (for text) and LogisticRegression. Provide code that sets up param_distributions for vectorizer parameters like max_df and ngram_range, for logistic C, and explain how n_iter and scoring choices influence the search.

Sample Answer

Approach: build a scikit-learn Pipeline with TfidfVectorizer and LogisticRegression, define a param_distributions dict for RandomizedSearchCV (using sensible ranges and distributions), run RandomizedSearchCV with n_iter samples and explain how n_iter and scoring affect the search.

python

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, expon, loguniform

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer()),
    ("clf", LogisticRegression(solver="saga", max_iter=10000, random_state=42))
])

param_distributions = {
    # Tfidf: percent of docs to ignore (0.0-1.0), try ints for max_features and ngram ranges
    "tfidf__max_df": loguniform(0.5, 1.0),         # continuous between 0.5 and 1.0
    "tfidf__min_df": randint(1, 5),                # ignore terms that appear in fewer than k docs
    "tfidf__ngram_range": [(1,1), (1,2), (1,3)],   # tried as categorical choices
    "tfidf__max_features": randint(5000, 30000),   # limit feature size
    # LogisticRegression regularization strength C (log-uniform search)
    "clf__C": loguniform(1e-4, 1e2),
    "clf__penalty": ["l2"],                        # saga supports l1/l2; pick appropriate solver if adding l1
}

rs = RandomizedSearchCV(
    pipeline,
    param_distributions=param_distributions,
    n_iter=60,                 # number of random hyperparameter sets to try
    scoring="roc_auc",         # choose metric aligned with the problem
    cv=5,
    verbose=2,
    n_jobs=-1,
    random_state=42
)

# Fit on training data (X_text: list/series of strings, y: labels)
rs.fit(X_text, y)
print("Best params:", rs.best_params_)
print("Best score:", rs.best_score_)

Key explanations:- n_iter controls budget: larger n_iter explores more combinations and increases chance of finding a strong config; set according to compute/time constraints.- Use loguniform for C because C spans orders of magnitude; sampling uniformly would bias to one scale.- scoring should match business/ML objective: accuracy for balanced classification, roc_auc for imbalanced or probabilistic ranking, f1 for class-imbalanced single-threshold performance.- RandomizedSearchCV is more efficient than GridSearch when search space is large; include sensible distributions and some categorical choices.Edge cases: if using l1 penalty change solver to "saga" and ensure stability for very small C values; restrict ngram_range if memory is limited; consider using Pipeline + FeatureUnion or hashing for extreme scale.

HardTechnical

68 practiced

Given irregular timestamps per entity, compute for each timestamp the exponentially weighted mean (EWMA) for that entity with half-life of 7 days. Implement this in pandas using groupby and ewm or a manual approach that handles irregular sampling and preserves alignment to original timestamps. Explain numerical stability considerations.

Sample Answer

Approach: compute a per-row time-decay alpha from the elapsed time since the previous observation using continuous-time EWMA:alpha = 1 - exp(-ln(2) * delta / half_life). Then apply the recursive update s_t = alpha * x_t + (1 - alpha) * s_prev per entity in timestamp order. This preserves original timestamps and handles irregular sampling.

python

import numpy as np
import pandas as pd

HALF_LIFE_DAYS = 7.0
LN2 = np.log(2.0)

def ewma_irregular(group, value_col='value', ts_col='ts'):
    g = group.sort_values(ts_col).copy()
    ts = pd.to_datetime(g[ts_col])
    # delta in days (float); first delta = inf -> set alpha=1 to initialize to first value
    delta_days = ts.diff().dt.total_seconds().div(86400).fillna(np.inf).to_numpy()
    # compute alpha per interval: alpha = 1 - exp(-ln2 * delta / half_life)
    # for very large delta, exp(...) underflows to 0 -> alpha -> 1 (good)
    exponent = -LN2 * (delta_days / HALF_LIFE_DAYS)
    # guard numerical stability: clip exponent to avoid +inf or too-large positive
    exponent = np.clip(exponent, -700, 0.0)  # exp(-700) ~ 5e-305 safe in float64
    alpha = 1.0 - np.exp(exponent)
    vals = g[value_col].to_numpy(dtype='float64')
    out = np.empty_like(vals, dtype='float64')
    s = np.nan
    for i, (a, x) in enumerate(zip(alpha, vals)):
        if i == 0 or np.isinf(delta_days[i]):  # initialize on first observation
            s = x
        else:
            # recursive stable update
            s = a * x + (1.0 - a) * s
        out[i] = s
    g[value_col + '_ewma'] = out
    return g

# Usage:
# df with columns ['entity','ts','value']
result = df.groupby('entity', group_keys=False).apply(ewma_irregular)

Key points and numerical stability:- Use continuous-time decay so irregular intervals are correctly handled.- Compute alpha = 1 - exp(-k * delta); clamp exponent to [-700,0] to avoid exp underflow/overflow.- Use float64 to reduce rounding error; recursive formula avoids large cancellations.- Initialize EWMA to the first observed value (alternatively use prior or NaN).- Complexity: O(N) time, O(1) extra space per group (in-place arrays).Edge cases:- Missing values: skip or treat x as NaN (propagate or impute before EWMA).- Zero or tiny deltas: alpha≈0 -> preserves previous state; large deltas -> alpha→1 resets to current value.Alternative: vectorized numba implementation for very large datasets for faster loops.

EasyTechnical

118 practiced

Describe the difference between the legacy numpy.random.seed approach and the new numpy.random.Generator API. Show code to create a reproducible Generator, sample normal variates, and explain why the new API is preferred for modern code and parallel workflows.

Sample Answer

Legacy approach (pre-NumPy 1.17) used global state via numpy.random.seed and the legacy RandomState. That pattern mutates a global RNG, making reproducibility fragile in larger codebases and unsafe for parallelism. The new API provides numpy.random.default_rng which returns a Generator with explicit, local state (PCG64 by default).

Example — create reproducible Generator and sample normals:

python

import numpy as np

# Create reproducible Generator from an integer seed
rng = np.random.default_rng(42)  # returns a Generator (PCG64)

# Sample 5 standard normal variates
samples = rng.normal(loc=0.0, scale=1.0, size=5)
print(samples)

Why the new API is preferred:- Explicit state: Generator objects encapsulate state; functions don't rely on a hidden global, so results are reproducible when you pass or persist the Generator.- Better algorithms: Generator uses modern bit generators (PCG64, etc.) with improved statistical properties and performance.- Thread- and process-safe parallelism: Create independent streams by instantiating separate Generators with different seeds or by using BitGenerator.jumped/advance, avoiding correlations across parallel workers.- Cleaner API: More methods, consistent behavior (e.g., default_rng replaces many convenience functions), and better forward compatibility.

Best practice: create and pass a Generator into functions that need randomness rather than calling global numpy.random.seed.

EasyTechnical

55 practiced

Explain the difference between a NumPy view and a copy. Give short Python examples showing slicing that returns a view and when operations produce a copy. Explain memory implications and how to detect whether an array owns its data.

Sample Answer

A NumPy view is a new ndarray object that references the same underlying data buffer as the original array (no data copied). A copy allocates a new buffer and duplicates data. Views are cheap (no extra memory) but modifying either array affects the other; copies are independent.

Example: slicing returns a view

python

import numpy as np
a = np.arange(10)
v = a[2:7]        # view: shares data
v[0] = 99
print(a)          # a[2] is now 99

Examples that produce copies

python

b = a[:].copy()   # explicit copy
c = a[[1,3,5]]    # fancy indexing produces a copy
d = a + 0         # arithmetic produces a new array (copy)

Memory implications- Views: low memory overhead, faster to create, but risk of unintended side effects and keeping large buffers alive if a small view references a large array.- Copies: use more memory and time but give safety/isolation.

How to detect ownership / sharing- array.base is None => array owns its data. If not None, it's a view (or references another array).- array.flags['OWNDATA'] is True => owns data.- np.may_share_memory(a, b) checks if two arrays might share the same memory buffer.

Examples:

python

v.base is a        # True for the slice view
a.flags['OWNDATA'] # True
v.flags['OWNDATA'] # False
np.may_share_memory(a, v)  # True

HardTechnical

59 practiced

Explain how to convert a scikit-learn pipeline containing a TfidfVectorizer and LogisticRegression to ONNX using skl2onnx. Provide the conversion commands, initial type declarations, and describe common pitfalls such as unsupported transformers, sparse inputs, and handling unknown categories. Show how to validate ONNX outputs with onnxruntime against sklearn predictions.

Sample Answer

Approach summary: export the fitted scikit-learn Pipeline (TfidfVectorizer -> LogisticRegression) with skl2onnx by declaring the pipeline's input type (text = string tensor), converting to ONNX, handling sparse outputs, and validating predictions with onnxruntime.

Conversion (Python):

python

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import StringTensorType
import joblib

# load fitted pipeline: pipeline = TfidfVectorizer -> LogisticRegression
pipeline = joblib.load("pipeline.pkl")

initial_type = [("text_input", StringTensorType([None, 1]))]  # batch x 1 string

onnx_model = convert_sklearn(
    pipeline,
    initial_types=initial_type,
    target_opset=18,            # pick appropriate opset
    options={id(pipeline.named_steps['tfidf']): {'zipmap': False}}
)

with open("pipeline.onnx", "wb") as f:
    f.write(onnx_model.SerializeToString())

Key details and pitfalls- Initial types: for text use StringTensorType([None, 1]) or StringTensorType([None]) depending on pipeline shape. Many examples use [None, 1] when passing column vectors.- Sparse inputs/outputs: TfidfVectorizer produces sparse matrices. skl2onnx converts to sparse tensors; some runtimes (older onnxruntime) need to support sparse tensors. If you get unsupported sparse errors, convert TfidfVectorizer output to dense before logistic regression (wrap with FunctionTransformer(lambda x: x.toarray())) or set options to force dense conversion when possible.- Unsupported transformers: custom transformers or lambda-based FunctionTransformer may not have converters. Provide a custom converter for skl2onnx or replace with supported equivalents. Avoid Python-only preprocessing in the pipeline; do preprocessing in sklearn-compatible transformers.- Unknown categories (relevant for OneHotEncoder, OrdinalEncoder): set handle_unknown='ignore' so ONNX won't fail on unseen categories.- Opset compatibility: ensure target_opset is supported by your onnxruntime version.

Validate ONNX outputs with onnxruntime:

python

import numpy as np
import onnxruntime as ort

# prepare sample inputs (batch of strings)
X = np.array([["sample text 1"], ["another example"]], dtype=object)

# sklearn predictions
sk_preds = pipeline.predict_proba(X.ravel())  # or pipeline.predict_proba(X) depending on shape

# onnxruntime
sess = ort.InferenceSession("pipeline.onnx")
input_name = sess.get_inputs()[0].name
ort_inputs = {input_name: X}
ort_outs = sess.run(None, ort_inputs)
# if you set zipmap False, probability output is typically ort_outs[1] or check output names
print("sklearn:", sk_preds)
print("onnx:", ort_outs)

Validation tips:- Match input shapes: if initial_type used [None,1], pass shape (N,1); if [None], pass (N,) strings.- Compare predict_proba outputs with a tolerance (e.g., np.allclose(sk_preds, ort_probs, atol=1e-6)).- If you get different numeric results, check opset mismatch, float32 vs float64 casting (ONNX uses float32); cast sklearn outputs to float32 before compare.

Alternative workarounds- If conversion fails due to sparse support, explicitly chain a transformer that densifies before the classifier.- For complex preprocessing (tokenizers, custom text cleaning) perform it outside the pipeline and convert only model & vectorizer or reimplement preprocessing in ONNX via available ops.

This process yields a portable ONNX model validated against sklearn using onnxruntime; handle sparse tensors, unsupported transformers, and unknown categories proactively to avoid runtime conversion/serving errors.

Unlock Full Question Bank

Get access to hundreds of Scikit Learn, Pandas, and NumPy Usage interview questions and detailed answers.

Join thousands of developers preparing for their dream job.