InterviewStack.io LogoInterviewStack.io

Data Processing and Transformation Questions

Focuses on algorithmic and engineering approaches to transform and clean data at scale. Includes deduplication strategies, parsing and normalizing unstructured or semi structured data, handling missing or inconsistent values, incremental and chunked processing for large datasets, batch versus streaming trade offs, state management, efficient memory and compute usage, idempotency and error handling, and techniques for scaling and parallelizing transformation pipelines. Interviewers may assess problem solving, choice of algorithms and data structures, and pragmatic design for reliability and performance.

HardTechnical
32 practiced
Production shows a 15% drop in aggregated daily sales counts, but no alerts fired. Describe a forensic investigation plan across ingestion (Kafka), processing (Spark/Flink), storage (S3/warehouse), and visualization (BI). Include specific queries or checks you would run, metric correlations, and steps to locate where data was lost or filtered.
EasyTechnical
39 practiced
You must decide between batch and streaming for two tasks: (A) daily aggregated revenue reports consumed by analysts, and (B) real-time fraud alerts that require <10s detection. Describe the trade-offs (latency, complexity, cost, data completeness, operational burden), and justify which approach you'd choose for each task and why.
EasyTechnical
53 practiced
Write a Python function (Python 3.x, pandas allowed) that normalizes a list of nested JSON user profiles into a DataFrame with columns: user_id, email, first_name, last_name, city, signup_date. Input example:
{
  "user_id": 123,
  "contact": { "email": "a@example.com" },
  "name": { "first": "A", "last": "B" },
  "address": { "city": "Seattle" },
  "signup": "2024-03-01T12:00:00Z"
}
Explain how you handle missing fields, nested objects, and parsing dates. Also note how you'd adapt this for datasets that don't fit into memory.
HardTechnical
39 practiced
Case study: Build a near-real-time attribution pipeline that joins clickstream events with ad impressions and conversion events to compute last-touch attribution per user per campaign. Requirements: deduplicate impressions, support PII hashing/anonymization, handle late conversions up to 7 days, provide streaming metrics for dashboards, and support batch reprocessing for accuracy. Design the end-to-end architecture, storage formats, windowing strategy, and correctness guarantees.
MediumSystem Design
35 practiced
Design an incremental ETL pipeline that streams CDC (change data capture) from Postgres into BigQuery (or Snowflake) to achieve near real-time replication (<5s latency). Address ordering, deduplication, schema evolution, transactional consistency, idempotency, and tooling choices (e.g., Debezium, Kafka, connectors).

Unlock Full Question Bank

Get access to hundreds of Data Processing and Transformation interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.