Data Processing and Transformation Questions
Focuses on algorithmic and engineering approaches to transform and clean data at scale. Includes deduplication strategies, parsing and normalizing unstructured or semi structured data, handling missing or inconsistent values, incremental and chunked processing for large datasets, batch versus streaming trade offs, state management, efficient memory and compute usage, idempotency and error handling, and techniques for scaling and parallelizing transformation pipelines. Interviewers may assess problem solving, choice of algorithms and data structures, and pragmatic design for reliability and performance.
Sample Answer
Sample Answer
{
"user_id": 123,
"contact": { "email": "a@example.com" },
"name": { "first": "A", "last": "B" },
"address": { "city": "Seattle" },
"signup": "2024-03-01T12:00:00Z"
}Sample Answer
import json
import pandas as pd
from typing import List, Dict, Iterable
def normalize_profiles(profiles: Iterable[Dict]) -> pd.DataFrame:
"""
profiles: iterable of dicts (can be list or generator)
Returns DataFrame with columns: user_id, email, first_name, last_name, city, signup_date
"""
# json_normalize handles nested keys using dot notation
df = pd.json_normalize(profiles)
# extract/rename columns, using .get-like access with fallback to NaN
out = pd.DataFrame({
"user_id": df.get("user_id"),
"email": df.get("contact.email"),
"first_name": df.get("name.first"),
"last_name": df.get("name.last"),
"city": df.get("address.city"),
"signup_date": df.get("signup")
})
# parse dates robustly; invalid parsing -> NaT
out["signup_date"] = pd.to_datetime(out["signup_date"], utc=True, errors="coerce")
# Optional: enforce types, handle missing user_id as NaN or drop
# out["user_id"] = pd.to_numeric(out["user_id"], errors="coerce")
return out[["user_id","email","first_name","last_name","city","signup_date"]]Sample Answer
Sample Answer
Unlock Full Question Bank
Get access to hundreds of Data Processing and Transformation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.