Scikit Learn, Pandas, and NumPy Usage Questions

Practical proficiency with these core libraries. Pandas: DataFrames, data manipulation, handling missing values. NumPy: arrays, vectorized operations, mathematical functions. Scikit-learn: preprocessing, model fitting, evaluation metrics, pipelines. Knowing standard patterns and APIs. Writing efficient, readable code using these libraries.

MediumTechnical

59 practiced

Using NumPy, implement an efficient method to compute pairwise squared Euclidean distances between two arrays A (shape 10000x128) and B (shape 500x128) without explicit Python loops. Provide code and explain memory and computational complexity; suggest batching if necessary.

Sample Answer

Approach: Use the identity ||a - b||^2 = ||a||^2 + ||b||^2 - 2 a·b to compute all pairwise squared distances via matrix operations and broadcasting. This avoids Python loops and uses efficient BLAS for the dot product. Convert to float32 if precision allows to halve memory.

python

import numpy as np

def pairwise_sq_dists(A, B, batch_size=None, dtype=np.float32):
    """
    Compute pairwise squared Euclidean distances between A (N x D) and B (M x D).
    If batch_size is provided, process A in chunks to reduce peak memory.
    Returns an (N x M) array of distances.
    """
    A = np.asarray(A, dtype=dtype)
    B = np.asarray(B, dtype=dtype)
    N, D = A.shape
    M = B.shape[0]

    # Precompute norms of B
    B_norm2 = np.sum(B * B, axis=1)  # (M,)

    if batch_size is None:
        # Single-shot computation
        A_norm2 = np.sum(A * A, axis=1)[:, None]  # (N,1)
        # (N x M) = (N x D) dot (D x M)
        cross = A.dot(B.T)  # (N, M)
        dists = A_norm2 + B_norm2[None, :] - 2.0 * cross
        # Numerical safety: small negatives -> 0
        np.maximum(dists, 0.0, out=dists)
        return dists
    else:
        # Batched over A
        dists = np.empty((N, M), dtype=dtype)
        for i in range(0, N, batch_size):
            a_chunk = A[i:i+batch_size]
            a_norm2 = np.sum(a_chunk * a_chunk, axis=1)[:, None]  # (bs,1)
            cross = a_chunk.dot(B.T)  # (bs, M)
            chunk_dists = a_norm2 + B_norm2[None, :] - 2.0 * cross
            np.maximum(chunk_dists, 0.0, out=chunk_dists)
            dists[i:i+batch_size] = chunk_dists
        return dists

Key points:- Time complexity: O(N * M * D) dominated by the matrix multiplication.- Memory: output is O(N * M). With A=10000, B=500 => 5e6 floats. float64 ~40 MB, float32 ~20 MB. Intermediate cross product is also O(N*M) unless batched.- Use batching to limit peak memory: choose batch_size so (batch_size * M) fits comfortably (e.g., batch_size = 1000 → 1000*500 = 500k floats ~2 MB float32).- Numerical note: subtracting can produce tiny negative values; clamp with maximum(,0).- Alternative optimizations: use GPU (CuPy/PyTorch), memory-mapped arrays, or incremental nearest-neighbor algorithms if you only need k-NN instead of full matrix.

EasyTechnical

66 practiced

Given a transactions DataFrame with columns ['user_id', 'transaction_id', 'amount', 'transaction_date'], write pandas code to compute for each user:- total_amount- transaction_count- average_amount- most_recent_transaction_dateReturn a DataFrame indexed by user_id with these columns. Provide an explanation of any aggregation choices.

Sample Answer

Approach: convert dates to datetime (if needed), group by user_id and compute aggregations with named aggregations so output columns are clear. Use sum/count/mean for amounts and max for most recent date.

python

import pandas as pd

# assume df is the input DataFrame
df = df.copy()

# ensure transaction_date is datetime
df['transaction_date'] = pd.to_datetime(df['transaction_date'], errors='coerce')

# group and aggregate with named aggregations
result = (
    df.groupby('user_id', as_index=True)
      .agg(
          total_amount=pd.NamedAgg(column='amount', aggfunc='sum'),
          transaction_count=pd.NamedAgg(column='transaction_id', aggfunc='count'),
          average_amount=pd.NamedAgg(column='amount', aggfunc='mean'),
          most_recent_transaction_date=pd.NamedAgg(column='transaction_date', aggfunc='max')
      )
)

# optional: sort by total_amount descending
# result = result.sort_values('total_amount', ascending=False)

Key choices:- sum for total_amount and mean for average_amount are standard for monetary aggregates.- count(transaction_id) counts transactions; use .nunique('transaction_id') if duplicates possible.- max on transaction_date gives the most recent date.- to_datetime with errors='coerce' handles bad date strings (will become NaT). Handle NaNs later as needed.

Time complexity: O(n) for a single pass groupby. Space: O(k) where k = number of users. Edge cases: empty df, null amounts, duplicate transaction_ids, or all-NaT dates—handle with dropna or filling as business rules require.

MediumTechnical

69 practiced

You join sales and customers tables and find inconsistent timestamps and duplicated customer IDs across sources. Outline a step-by-step data-cleaning plan using pandas to deduplicate records and reconcile timestamps, including code snippets for deduplication criteria and timestamp alignment. Consider business keys and auditability.

Sample Answer

Step-by-step plan (business-key first, reproducible, auditable):

1) Understand keys & schema- Identify business keys (e.g., customer_id, sale_id, order_number) and source_id, source_ts, ingestion_ts.- Decide canonical source for each field (e.g., sales table authoritative for sale_amount).

2) Ingest with provenance columns- Read both sources adding source name and ingestion timestamp.

python

import pandas as pd, datetime as dt
sales = pd.read_csv('sales.csv')
customers = pd.read_csv('customers.csv')
for df, name in [(sales,'sales'),(customers,'customers')]:
    df['__source'] = name
    df['__ingest_ts'] = pd.Timestamp.utcnow()

3) Normalize types and parse timestamps

python

for col in ['sale_ts','customer_updated_ts']:
    df = sales if col=='sale_ts' else customers
    df[col] = pd.to_datetime(df[col], utc=True, errors='coerce')

4) Deduplicate within each source using explicit criteria (audit retained)- Keep latest by (business key + priority field + ingestion/source timestamp)

python

def dedupe(df, keys, prefer_col=None):
    df = df.sort_values(by=keys + ([prefer_col] if prefer_col else []) + ['__ingest_ts'], ascending=False)
    df['_keep'] = ~df.duplicated(subset=keys)
    kept = df[df['_keep']].drop(columns=['_keep'])
    dropped = df[~df['_keep']]
    return kept, dropped

cust_kept, cust_dropped = dedupe(customers, ['customer_id'], prefer_col='customer_updated_ts')
sales_kept, sales_dropped = dedupe(sales, ['sale_id','customer_id'], prefer_col='sale_ts')

- Persist dropped rows to audit tables.

5) Reconcile duplicates across sources (survivorship rules)- Define rules table: prefer source X, or latest update_ts, or non-null fields.

python

merged = cust_kept.merge(sales_kept, on='customer_id', how='outer', suffixes=('_cust','_sale'))
# Example field-level survivorship
def choose(a,b, prefer='cust'):
    if prefer=='cust': return a.combine_first(b)
merged['email'] = merged['email_cust'].combine_first(merged['email_sale'])

6) Timestamp alignment & clock skew handling- Convert to UTC, detect large offsets per source, and correct using median offset vs reference (e.g., DB ingestion time)

python

# compute offset = ingestion_ts - source_ts, take median per __source
merged['offset'] = merged['__ingest_ts'] - merged['sale_ts']
offsets = merged.groupby('__source')['offset'].median()
# apply correction
merged['sale_ts_aligned'] = merged.apply(lambda r: r['sale_ts'] + offsets.loc[r['__source']], axis=1)

- Flag records with null or implausible timestamps for manual review.

7) Validation & metrics- Row counts before/after, unique counts per business key, number of resolved conflicts, percent nulls.- Save audit logs: dropped rows, conflict resolution decisions, timestamp corrections (store offsets and rationale).

8) Automation & testing- Wrap steps into functions, add unit tests for dedupe logic, run daily pipeline, produce summary report for stakeholders.

Key points:- Use explicit business keys and documented survivorship rules- Retain provenance and audit tables for traceability- Prefer deterministic rules (source priority, latest timestamp, non-null) and flag ambiguous cases for manual review.

MediumTechnical

53 practiced

You must process a 50GB CSV file that won't fit into memory. Describe with code how you'd use pandas.read_csv with chunksize to compute per-user total transaction amount, returning the final aggregated Series or DataFrame. Explain how to combine chunk results efficiently.

Sample Answer

Approach: iterate the CSV in chunks with pandas.read_csv(..., chunksize=...), compute per-chunk user sums (groupby 'user_id'), and incrementally merge into a running aggregation using Series.add(fill_value=0) — this avoids holding all rows or many intermediate DataFrames in memory.

python

import pandas as pd

def aggregate_user_totals(path, chunksize=10_000_00):
    # running totals as a Series: index=user_id, value=total_amount
    totals = pd.Series(dtype='float64')

    # read only necessary columns, set dtypes for memory efficiency
    for chunk in pd.read_csv(
        path,
        usecols=['user_id', 'amount'],
        dtype={'user_id': 'int64', 'amount': 'float64'},
        chunksize=chunksize
    ):
        # group within chunk
        chunk_totals = chunk.groupby('user_id', sort=False)['amount'].sum()
        # merge into running totals; add handles differing indices
        totals = totals.add(chunk_totals, fill_value=0)

    # optional: sort or convert to DataFrame
    totals = totals.sort_values(ascending=False)
    return totals  # Series indexed by user_id

# example usage:
# result = aggregate_user_totals('transactions_large.csv')

Why this is efficient:- Each chunk is small enough to fit memory.- Grouping per-chunk reduces intermediate size.- Series.add(fill_value=0) merges in O(k) where k = unique users in chunk; overall cost proportional to total unique users times number of chunks they appear in.- Usecols and explicit dtypes reduce memory and parsing time.

Edge cases / improvements:- If user_id is string, use dtype='category' if many repeats.- If millions of unique users, consider storing intermediate aggregates in a key-value store (SQLite, Redis) or using Dask.- Validate and handle NaNs in amount before grouping (e.g., chunk['amount']=chunk['amount'].fillna(0)).

MediumTechnical

67 practiced

Discuss trade-offs between one-hot encoding and target encoding for a categorical feature with high cardinality (~10,000 categories). In what situations is target encoding appropriate? How do you avoid target leakage when applying target encoding in cross-validation or production?

Sample Answer

One-hot vs target encoding (10k categories)- One-hot: creates ~10k binary columns → extremely sparse, high memory/compute cost, can blow up models that don’t handle high dimensionality well (linear/logistic may suffer multicollinearity). It’s interpretable (each category explicit) and safe from leakage.- Target encoding (mean encoding): replaces category with summary statistic of target (e.g., mean conversion). Highly compact, can capture signal when categories correlate with target and when frequency varies. Works well with tree-based models and when you want fewer features.

When target encoding is appropriate- High-cardinality categorical with many rare levels (10k) where one-hot is infeasible.- Sufficient historical data per category (or you apply smoothing).- Predictive relationship exists between category and target.- You accept some loss of per-category interpretability in exchange for performance.

Risks and how to avoid target leakage- Directly using global target means leaks label information → overfitting.- Use only training-set statistics when encoding validation/test.

Practical safe strategies1. K-fold (out-of-fold) target encoding: for each fold, compute category means on other folds and apply to held-out fold. For final model train, compute on full train; apply to test using those train stats.2. Leave-one-out or additive smoothing: for each row, use (sum_y - y_i)/(count - 1) or regularized mean: (sum_y + prior * m)/(count + m) to shrink rare categories toward global prior.3. Nested CV / pipeline: wrap encoding inside cross-validation or scikit-learn Pipeline so transforms fit only on train splits.4. Add noise/randomization during training to reduce memorization (small Gaussian noise).5. For production: persist mapping of category → encoded value computed from training data; for unseen categories use global prior or frequency-based backoff; consider online updates with decayed counts if distribution shifts.

Monitoring and interpretability- Monitor performance drift and category population changes.- Report approximate effect sizes by aggregated groups (top N categories) for stakeholders.

Summary: For ~10k categories, target encoding is generally preferable when predictive signal exists and you rigorously prevent leakage via out-of-fold encoding, smoothing/prior regularization, and careful production mapping. One-hot remains a safer, more interpretable but often impractical baseline.

Unlock Full Question Bank

Get access to hundreds of Scikit Learn, Pandas, and NumPy Usage interview questions and detailed answers.

Join thousands of developers preparing for their dream job.