Python and Data Manipulation Questions

Demonstrate practical proficiency in Python for data exploration and preprocessing. Expect to perform data cleaning, joins, group by aggregations, pivots and reshaping, vectorized operations, missing value handling, and basic performance tuning using libraries such as NumPy and Pandas. Show how to write readable, testable, and efficient code for sampling, feature extraction, and quick prototyping, and how to scale to larger data sets using chunking or streaming approaches.

MediumTechnical

40 practiced

You are prototyping a model that uses both structured columns and free-text customer comments. In pandas, how would you extract a small, stable set of text features and combine them with the structured features without making the pipeline fragile or impossible to reproduce?

Sample Answer

**Approach**I would keep the text feature set small, explicit, and versioned so it stays stable across retrains. In pandas, I usually normalize the comments, derive a few interpretable features like length, word count, exclamation count, and a fixed set of keyword flags, then combine them with the structured columns using `concat`. If I need learned text features, I fit that step on training data only and persist the fitted vocabulary so the column set does not drift.

Example: the comment `Late delivery, but resolved!` becomes word_count = 4, has_late = 1, and exclaim_count = 1.

python

keywords = ['late', 'refund', 'cancel']
text = df['comment'].fillna('').str.lower()

text_feats = pd.DataFrame({
    'comment_len': text.str.len(),
    'word_count': text.str.split().str.len(),
    'exclaim_count': text.str.count('!'),
})

for word in keywords:
    text_feats[f'has_{word}'] = text.str.contains(word, regex=False).astype('int8')

X = pd.concat([df[structured_cols].reset_index(drop=True), text_feats], axis=1)

This is reproducible because the feature definitions are fixed, easy to test, and do not depend on whatever words happen to appear in a new batch.

HardTechnical

52 practiced

You need to process a 50 GB CSV of transactions on a machine that cannot hold the full file in memory. Write a Python function that reads the file in chunks, filters invalid rows, and returns per-customer total spend and transaction count. The function should be restartable and should not require loading all rows at once.

Sample Answer

To handle a 50 GB CSV, I would stream it in chunks, validate each chunk, and keep only running aggregates in memory. A chunk is a small batch of rows read at once, so we never load the full file.

python

import json, os
from collections import defaultdict
import pandas as pd

def process_transactions_csv(path, chunksize=100_000, checkpoint_path='checkpoint.json'):
    totals = defaultdict(float)
    counts = defaultdict(int)
    start = 0

    if checkpoint_path and os.path.exists(checkpoint_path):
        ckpt = json.load(open(checkpoint_path))
        start = ckpt.get('processed_chunks', 0)
        totals.update({k: float(v) for k, v in ckpt.get('totals', {}).items()})
        counts.update({k: int(v) for k, v in ckpt.get('counts', {}).items()})

    for i, chunk in enumerate(pd.read_csv(path, chunksize=chunksize), start=1):
        if i <= start:
            continue
        need = {'customer_id', 'amount'}
        if not need.issubset(chunk.columns):
            raise ValueError(f'Missing columns: {need - set(chunk.columns)}')

        valid = chunk[['customer_id', 'amount']].copy()
        valid['customer_id'] = valid['customer_id'].astype('string').str.strip()
        valid['amount'] = pd.to_numeric(valid['amount'], errors='coerce')
        valid = valid[valid['customer_id'].notna() & valid['customer_id'].ne('') & valid['amount'].notna()]

        agg = valid.groupby('customer_id', as_index=True)['amount'].agg(['sum', 'count'])
        for cid, row in agg.iterrows():
            totals[cid] += float(row['sum'])
            counts[cid] += int(row['count'])

        if checkpoint_path:
            json.dump({'processed_chunks': i, 'totals': dict(totals), 'counts': dict(counts)}, open(checkpoint_path, 'w'))

    return pd.DataFrame([
        {'customer_id': cid, 'total_spend': totals[cid], 'transaction_count': counts[cid]}
        for cid in totals
    ]).sort_values('customer_id').reset_index(drop=True)

Example: if one chunk has Alice 10, Bob invalid, Alice 5, the result for Alice is total_spend 15 and transaction_count 2.

Time complexity is O(n) over rows. Space is O(k) for k distinct customers, plus one chunk. Edge cases: empty file, missing columns, bad amounts, and restarting on the same immutable source file. If customer cardinality is huge, I would persist aggregates to SQLite or Parquet instead of holding them in a dict.

MediumTechnical

43 practiced

A teammate hands you a pandas notebook that builds weekly reporting tables for leadership. What checks would you add so the notebook fails fast when an input column changes type, a join stops matching, or an aggregate quietly drifts from expected values?

MediumTechnical

46 practiced

Given an event table with user_id, event_ts, and spend, write a pandas function that returns one row per user with first_event_ts, last_event_ts, active_day_count, and average spend per active day. Assume timestamps are strings that may need parsing and some users have repeated events on the same day.

Sample Answer

To solve this cleanly, I would parse the timestamps once, create a per-row event day, then aggregate by user. `active_day_count` should count unique calendar days, so repeated events on the same day do not inflate it. The average spend per active day is total spend divided by active days.

Example: if user 7 has two events on 2024-01-03 with spend 30 and 20, plus one event on 2024-01-05 with spend 50, then first_event_ts is 2024-01-03, last_event_ts is 2024-01-05, active_day_count is 2, and average spend per active day is 100 / 2 = 50.

python

import numpy as np
import pandas as pd

def summarize_users(events: pd.DataFrame) -> pd.DataFrame:
    df = events.copy()
    df['event_ts'] = pd.to_datetime(df['event_ts'], errors='coerce')
    df['event_day'] = df['event_ts'].dt.normalize()

    out = (
        df.groupby('user_id', as_index=False)
          .agg(
              first_event_ts=('event_ts', 'min'),
              last_event_ts=('event_ts', 'max'),
              active_day_count=('event_day', 'nunique'),
              total_spend=('spend', 'sum'),
          )
    )
    out['avg_spend_per_active_day'] = out['total_spend'] / out['active_day_count'].replace(0, np.nan)
    return out.drop(columns='total_spend').sort_values('user_id').reset_index(drop=True)

Time complexity is O(n) for the parse and groupby. Space is O(n) for the helper day column. If event_ts is invalid, it becomes NaT and will not count as an active day, so I would clean bad timestamps upstream if that matters.

EasyTechnical

50 practiced

You are given a pandas DataFrame named df with columns customer_id, signup_date, country, age, and spend. Write a Python function clean_customers(df) that returns a cleaned DataFrame with parsed dates, standardized country values, numeric age and spend columns, and exact duplicate rows removed. If age is missing, fill it with the median age for that country when possible, otherwise use the overall median. Keep the original row order as much as practical.

Sample Answer

I would remove exact duplicate rows first, then clean types and impute age in a way that preserves row order as much as possible. For country, I would standardize obvious variants with strip, upper, and a small mapping for known aliases. For age, I would use the country median when available, otherwise the overall median.

Example: if a US customer has missing age and the US median is 41, I fill 41. If the country is missing or has no usable ages, I fall back to the overall median, say 38.

python

import numpy as np
import pandas as pd

def clean_customers(df: pd.DataFrame) -> pd.DataFrame:
    out = df.drop_duplicates().copy()
    out['_row_order'] = np.arange(len(out))

    out['signup_date'] = pd.to_datetime(out['signup_date'], errors='coerce')
    out['country'] = (
        out['country'].astype('string')
        .str.strip()
        .str.upper()
        .replace({'U.S.A.': 'US', 'UNITED STATES': 'US', 'UK': 'GB'})
    )
    out['age'] = pd.to_numeric(out['age'], errors='coerce')
    out['spend'] = pd.to_numeric(out['spend'], errors='coerce')

    overall_median = out['age'].median()
    country_median = out.groupby('country')['age'].transform('median')
    out['age'] = out['age'].fillna(country_median).fillna(overall_median)

    return out.sort_values('_row_order').drop(columns='_row_order').reset_index(drop=True)

This keeps the original order, uses pandas-native conversions, and avoids chained assignment problems.

Unlock Full Question Bank

Get access to hundreds of Python and Data Manipulation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.