Python Programming & ML Libraries Questions

Python programming language fundamentals (syntax, data structures, control flow, error handling) with practical usage of machine learning libraries such as NumPy, pandas, scikit-learn, TensorFlow, and PyTorch for data manipulation, model development, training, evaluation, and lightweight ML tasks.

EasyTechnical

21 practiced

Given a pandas DataFrame `df` with columns ['user_id','event','value'], write code to compute for each user the average `value` per event type and return a pivoted DataFrame where rows are user_id and columns are event types. Include handling for missing event types and show example input and output behavior.

Sample Answer

Approach: group by user_id and event, compute mean(value), then pivot so rows are user_id and columns event types. Fill missing event types with 0 or NaN depending on preference; I'll show both.

python

import pandas as pd

# Example input
df = pd.DataFrame([
    {'user_id': 1, 'event': 'click', 'value': 10},
    {'user_id': 1, 'event': 'click', 'value': 30},
    {'user_id': 1, 'event': 'view',  'value': 5},
    {'user_id': 2, 'event': 'view',  'value': 7},
    {'user_id': 3, 'event': 'purchase','value': 100},
])

# Compute mean value per user_id and event, then pivot
mean_per_event = df.groupby(['user_id', 'event'], observed=True)['value'].mean().reset_index()
pivoted = mean_per_event.pivot(index='user_id', columns='event', values='value')

# Option 1: keep NaN for missing events
pivot_na = pivoted.copy()

# Option 2: fill missing events with 0 (or another sentinel)
pivot_filled = pivoted.fillna(0)

# Optional: ensure column order (all possible events)
all_events = ['click', 'view', 'purchase']
pivot_final = pivot_filled.reindex(columns=all_events, fill_value=0)

print("Pivot with NaN:\n", pivot_na)
print("\nPivot with zeros and ordered columns:\n", pivot_final)

Key points:- groupby + mean gives per-user-per-event averages.- pivot (or unstack) reshapes to wide format.- fillna controls how missing event types are handled.Complexity: O(n) time to aggregate, O(k*m) memory for k users × m events.Edge cases: empty df, single user, events with all-NaN values. If you need counts too, include .agg(['mean','count']).

MediumTechnical

23 practiced

A NumPy-based function you wrote runs slower than expected because of unnecessary copies and temporary arrays. Describe common causes for NumPy performance issues (copying, non-contiguous arrays, Python loops) and give specific code-level fixes for each cause.

Sample Answer

Common causes and specific fixes:

1) Unnecessary copies / temporaries- Cause: chaining operations creates temporaries (a+b+c produces two temporaries).- Fix: use in-place ops or combine with ufuncs where possible.

python

# bad: creates temporaries
y = (a + b) * c

# better: use out= to avoid temporaries
tmp = np.add(a, b, out=np.empty_like(a))
y = np.multiply(tmp, c, out=np.empty_like(a))

# best when safe: reuse buffer
np.add(a, b, out=a)        # only if a can be overwritten
np.multiply(a, c, out=a)

2) Non-contiguous arrays- Cause: views from transpose/slice are non-C (or F) contiguous; many ops copy.- Fix: ensure appropriate memory layout with np.ascontiguousarray or operate on strides-aware routines.

python

x = large_matrix.T        # not contiguous
x_c = np.ascontiguousarray(x)  # one copy but avoids repeated copies later

3) Python loops (lack of vectorization)- Cause: elementwise Python loops are slow.- Fix: vectorize with NumPy ufuncs, broadcasting, or use numba for complex logic.

python

# slow
out = np.empty_like(a)
for i in range(a.size):
    out[i] = func(a[i])

# fast
out = func(a)  # where func is ufunc-compatible or use np.where, np.maximum, etc.

# or numba
from numba import njit
@njit
def compute(a):
    out = np.empty_like(a)
    for i in range(a.size):
        out[i] = complex_logic(a[i])
    return out

Profiling tips: use %timeit, np.profiler, or line_profiler to find hotspots. Measure memory with tracemalloc or psutil. Trade-offs: in-place saves memory but mutates inputs; copying once up-front can be better than many tiny copies.

HardSystem Design

24 practiced

Design an end-to-end training pipeline for a supervised learning model that will be retrained weekly: include data ingestion, validation, feature store, experiment tracking, resource scheduling, and deployment steps. Specify scale assumptions (100M rows/day, models updated weekly) and describe trade-offs for retraining frequency and storage.

Sample Answer

Requirements & assumptions- Ingest 100M rows/day (structured), labels available with small delay. Models retrained weekly. SLOs: inference <100ms, 99th percentile; training window uses last 30–90 days of data. Expect high cardinality features.

High-level architecture1. Data ingestion: producers → Kafka (or Kinesis) for streaming; raw writes to immutable object store (S3/GS) as partitioned parquet/delta (date/hour/partition-key). Stream consumers write to a staging Delta Lake table.2. Validation & monitoring: Streaming data validator (Deequ/Great Expectations) checks schema, null rates, distributions; anomalies alert to Slack/PagerDuty and pause pipeline.3. Feature store: Feast + offline store on Delta/Snowflake, online store on Redis/Bigtable for low-latency serving. Features materialized daily (or near-real-time for online-only features).4. Experiment tracking: MLflow or Weights & Biases to log experiments, params, metrics, artifacts, dataset hashes, and model lineage.5. Training pipeline & scheduling: Orchestrate with Airflow/Kubeflow Pipelines: - Weekly scheduled DAG: snapshot labeled training set (SQL on Delta), run feature joins, split, train with distributed Spark/TF on GPU nodes or AWS SageMaker training jobs. - Support incremental/online updates (warm-start) to reduce compute.6. Validation & CI: Post-train evaluation (holdout, backtesting, data drift tests). If metrics exceed thresholds, register model in model registry; otherwise, fail and notify.7. Deployment: Canary rollout to subset of traffic via inference service (KFServing/Seldon) with autoscaling (K8s HPA + CPU/GPU). A/B testing and shadow traffic. Rollback automated on performance regressions.8. Monitoring & feedback: Real-time inference logs → feature/label join to compute live metrics (accuracy, calibration), concept drift detectors. Retraining triggered early if drift high.

Scale & implementation details- Storage: partitioned Delta Lake on S3 with compaction; use columnar format (parquet) and compression. Retain raw data for 90–365 days depending on compliance, archive older to Glacier. Offline feature store stores precomputed feature vectors for last N days to speed training.- Compute: Spark for ETL and feature materialization; distributed training for large datasets, sample for hyperparameter search. Use spot instances to reduce cost.- Throughput: Kafka topics sharded by customer/account id; consumers horizontally scalable. Online store sharded and cached for hot keys.

Trade-offs- Retraining frequency: - Weekly: balances staleness vs compute cost—suitable when label lag or business cycles are weekly. Pros: stable, reproducible; Cons: may miss sudden drift. - More frequent (daily/real-time): reduces staleness and adapts quickly to drift but increases compute and validation overhead, risk of noisy models. - Less frequent (monthly): cheaper but risks degraded performance.- Storage: - Keep raw + processed: higher cost but required for reprocessing, audits, and reproducibility. - Materialize features vs compute on-the-fly: materialization speeds training/inference at storage cost; on-the-fly reduces storage but increases latency and compute at inference. - Retention window trade-off: longer retention improves training signal but costs more; use tiered storage (hot for recent N days, cold/archived older).

Operational best practices- Use dataset versioning (hashes), automated tests for data and model, canary + automatic rollback, and cost-aware scheduling (spot instances, incremental training). Maintain runbooks and SLA for retraining and drift response.

MediumTechnical

25 practiced

You have a pandas groupby transform that runs slowly on a DataFrame with 10M rows. Describe 5 practical optimizations to speed up groupby aggregations or replace them with faster approaches, considering dtypes, indexing, chunking, categorical encoding, and third-party tools.

Sample Answer

1) Tighten dtypes — reduce memory so groupby fits in RAM and operations use faster numeric code. Convert ints/floats to smallest safe type and datetimes to datetime64[ns]:

python

df['id'] = df['id'].astype('int32')
df['value'] = df['value'].astype('float32')
df['ts'] = pd.to_datetime(df['ts'])

Why: smaller memory => fewer cache misses and faster sorting/hashing.

2) Use categorical keys for grouping:

python

df['category'] = df['category'].astype('category')
res = df.groupby('category')['value'].transform('mean')

Why: grouping by categorical uses integer codes instead of hashing strings.

3) Avoid full-row operations; project only necessary columns and set index/sort on group keys:- Do df = df[['key','value']] before groupby.- If you're doing rolling per-group, sort by key+time and use groupby(level=0) on a MultiIndex for faster locality.

4) Chunking / algorithmic alternatives:- Process in chunks (by key or by row) and aggregate partial results, then combine. Useful when memory-bound.- Replace transform with merge-from-aggregates when appropriate:

python

agg = df.groupby('key')['value'].mean().reset_index()
df = df.merge(agg, on='key', suffixes=('','_mean'))

This avoids repeated work and can be faster than transform in some cases.

5) Use faster engines: Dask, Polars, or Vaex for out-of-core or SIMD-optimized groupbys.- Dask: parallel, familiar API for >RAM datasets.- Polars: Rust-backed, much faster single-node groupby/transform.Example Polars:

python

import polars as pl
pl_df = pl.from_pandas(df)
pl_df.groupby('category').agg(pl.col('value').mean())

Trade-offs: third-party brings dependency and sometimes API changes, but huge speedups for large data.

Other tips: avoid python-level apply, prefer built-in aggregations, pre-compute and cache group keys, and profile with pandas’ .info(), memory_usage, and line_profiler to target hotspots.

HardTechnical

22 practiced

You must train scikit-learn models on a dataset that is larger than memory. Propose and implement an approach using Dask or `partial_fit` with chunks, explain when each is appropriate, and discuss how to evaluate and checkpoint models during a long-running out-of-core training job.

Sample Answer

Approach summary:- Two practical ways: (A) Use scikit-learn’s partial_fit with chunked data (pandas read_csv/chunksize or iterators) for models that implement partial_fit (SGDClassifier/Regressor, PassiveAggressive, MiniBatchKMeans, Perceptron). (B) Use Dask (dask.dataframe + dask-ml or dask-ml estimators / joblib backend) when you need parallelism across a cluster or when data lives on multiple machines / distributed filesystems.

Code: partial_fit with chunks (example: incremental logistic regression)

python

import pandas as pd
from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import log_loss
import joblib

model = SGDClassifier(loss='log', max_iter=1, tol=None, warm_start=True)
scaler = StandardScaler()
first_pass = True
checkpoint_every = 5  # epochs
chunksize = 100_000
train_path = "data/large_train.csv"
val_path = "data/val_sample.csv"
val = pd.read_csv(val_path)  # small enough to fit memory

for epoch in range(10):
    reader = pd.read_csv(train_path, chunksize=chunksize)
    for i, chunk in enumerate(reader):
        X_chunk = chunk.drop("y", axis=1)
        y_chunk = chunk["y"]
        if first_pass:
            scaler.partial_fit(X_chunk)  # if using StandardScaler with partial_fit support via incremental implementation
            model.partial_fit(scaler.transform(X_chunk), y_chunk, classes=[0,1])
            first_pass = False
        else:
            Xs = scaler.transform(X_chunk)
            model.partial_fit(Xs, y_chunk)
        if i % 50 == 0:
            # quick checkpoint
            joblib.dump({'model': model, 'scaler': scaler}, f"chkpt_epoch{epoch}_chunk{i}.pkl")
    # end epoch: evaluate on validation set
    Xv = scaler.transform(val.drop("y", axis=1))
    yv = val["y"]
    preds = model.predict_proba(Xv)[:,1]
    print(epoch, "val loss:", log_loss(yv, preds))
    if epoch % checkpoint_every == 0:
        joblib.dump({'model': model, 'scaler': scaler}, f"chkpt_epoch{epoch}.pkl")

Code: Dask approach (distributed, parallel)

python

from dask.distributed import Client
import dask.dataframe as dd
from dask_ml.linear_model import LogisticRegression
from dask_ml.model_selection import train_test_split

client = Client("scheduler-address:8786")
ddf = dd.read_csv("s3://bucket/large-*.csv")
X = ddf.drop("y", axis=1)
y = ddf["y"]
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.05)
model = LogisticRegression()  # dask-ml wraps incremental solvers
model.fit(X_train, y_train)
print(model.score(X_val, y_val))

When to use each:- partial_fit with chunks: simple, low infra cost, ideal when single-machine streaming from disk, models support incremental updates, and you need tight control over memory. Good for moderate-scale datasets.- Dask: use when dataset spans cluster storage (S3/HDFS), you need parallel I/O/compute, or when preprocessing/feature engineering benefits from distributed compute. Dask-ML also supports out-of-core and distributed training of some estimators.

Evaluation and checkpointing strategy:- Keep a small hold-out validation set (fits in memory) to compute metrics after each epoch or after N chunks.- Compute streaming metrics: maintain running aggregates (log loss, confusion matrix) to monitor convergence without storing all predictions.- Checkpoint frequently (per N chunks or per epoch) using joblib (or cloud storage). Save model + preprocessing pipeline + optimizer state.- Use atomic saves (write to temp then move) to avoid corrupted checkpoints.- Implement early stopping based on validation metric with patience (restore best checkpoint).- Log metrics to experiment tracking (MLflow, Weights & Biases) and the cluster scheduler logs.- For Dask, use client.save_state or persist model checkpoints to distributed filesystem; use dask-ml incremental estimators’ partial_fit for custom loops.

Additional considerations:- Feature transformations: use incremental or stateless transforms (HashingVectorizer) or compute scalers incrementally.- Class imbalance: track class distribution across chunks and adjust class_weight or sample weights incrementally.- Concept drift: periodically evaluate on recent validation slice, consider windowed training or learning rate schedules.- Fault tolerance: smaller checkpoints lower rework on failure.

Complexity:- Time ~ O(N) per epoch (I/O+model update). I/O often dominates for very large data.- Memory: controlled by chunk size for partial_fit; Dask memory controlled by partitions and workers.

This approach provides reproducible, monitorable out-of-core training with clear trade-offs between simplicity and distributed scale.

Unlock Full Question Bank

Get access to hundreds of Python Programming & ML Libraries interview questions and detailed answers.

Join thousands of developers preparing for their dream job.