Data Processing and Transformation Questions
Focuses on algorithmic and engineering approaches to transform and clean data at scale. Includes deduplication strategies, parsing and normalizing unstructured or semi structured data, handling missing or inconsistent values, incremental and chunked processing for large datasets, batch versus streaming trade offs, state management, efficient memory and compute usage, idempotency and error handling, and techniques for scaling and parallelizing transformation pipelines. Interviewers may assess problem solving, choice of algorithms and data structures, and pragmatic design for reliability and performance.
HardSystem Design
35 practiced
Design a global feature store that supports multi-region ML serving with low-latency online lookups and cross-region replication. Requirements: 99.99% availability for online lookups, configurable freshness SLAs per feature, and efficient tooling for point-in-time correct offline joins. Discuss storage choices, replication strategy (sync vs async), conflict resolution, and developer APIs.
Sample Answer
Requirements clarification:- Online low-latency lookups (<ms-10s), 99.99% availability globally.- Configurable freshness SLA per feature (e.g., 1s, 1h, 24h).- Point-in-time correct offline joins for training.- Cross-region replication with failure tolerance.High-level architecture:- Region-local serving tier (fast key-value store + edge cache) per region.- Global control plane for metadata, schema, feature registry, and replication policy.- Event streaming + durable cold storage for historical features and point-in-time joins.Storage choices:- Online store (per-region): strongly-consistent, low-latency KV (e.g., DynamoDB global tables with DAX, CockroachDB, or Redis Enterprise Active-Active). Use SSD-backed stores to meet latency.- Write-ahead/event log: Kafka or cloud Pub/Sub for change streams and audit.- Offline/historical store: Columnar object storage (Parquet on S3/GCS) partitioned by entity and time for efficient point-in-time joins.- Cache layer: local LRU TTL caches or CDN-like edge caches for sub-ms reads.Replication strategy:- Hybrid: synchronous replication within region (multiple AZs) to achieve single-region durability; async geo-replication across regions.- For features with strict freshness SLA (<s), optionally enable cross-region sync (synchronous or consensus-based) at registration time; trade-off: higher write latency.- Default: async replication from primary region using CDC (change-data-capture) -> event log -> apply in target regions with near-real-time pipelines.Conflict resolution:- Use a single source-of-truth per feature by default (primary region) to avoid conflicts.- For active-active writes (when enabled), use CRDTs for aggregate features (counters, sets) or last-write-wins with monotonic version vectors/timestamps (Lamport or hybrid logical clocks) for scalar features.- Provide per-feature conflict policy (primary, CRDT, LWW-with-timestamp, merge-fn).Freshness SLAs:- Feature metadata includes freshness SLA. Serving tier exposes current staleness metric computed from last-applied timestamp per key.- Routes reads: if local replica staleness > SLA, either (a) fallback to querying primary (if latency budget allows), or (b) return cached but flag staleness and let caller decide. Offer policy per feature.Point-in-time correct offline joins:- Persist all update events in immutable event log + snapshotting to Parquet with time-partitioning. Maintain a feature timestamp watermark.- Provide a point-in-time join service that, given a reference time t, performs joins by reading feature values whose event timestamp <= t, using indexed Parquet files and pre-computed materialized snapshots to speed queries.- Provide APIs to export training datasets that guarantee point-in-time correctness (querying historical store + event log replay for late-arriving data).Developer APIs:- Feature registration API: registerFeature(name, entity, type, primaryRegion, conflictPolicy, freshnessSLA, ttl, retention).- Write API: put(feature, entityId, value, timestamp, region) with optional consistency flag (sync/async).- Read API: get(featureSet, entityId, options) -> {value, timestamp, staleness, regionSource, consistency}.- Batch/export API: exportTrainingDataset(featureSet, entityIds, asOfTime) -> dataset URI (point-in-time safe).- Observability APIs: getStalenessMetrics(feature, region), replicationLag(feature, region), SLAViolationAlerts.- SDKs for languages (Python/Java) that handle retries, fallbacks, and staleness awareness.Operational considerations:- Monitor replication lag, error budgets, and per-feature traffic. Auto-scale regional serving stores.- Backfill and idempotent apply logic for late events. Maintain retention and compaction policy for historical data.- Provide feature versioning and immutability guarantees for training reproducibility.Trade-offs:- Strong cross-region sync gives fresher reads but higher write latency and lower throughput.- Async replication improves write latency and availability but requires careful conflict handling and SLA-based fallbacks.- Use configurable per-feature policy so teams choose the best trade-off for their model needs.
EasyTechnical
39 practiced
Compare JSON, Avro, Parquet, and TFRecord for storing preprocessed training data for ML. Discuss schema evolution support, compression and storage efficiency, read performance (random vs columnar), splittability for parallel processing, and compatibility with frameworks like TensorFlow and PyTorch.
Sample Answer
High-level summary: JSON is simple and human-readable but inefficient; Avro is row-oriented with strong schema and good evolution support; Parquet is columnar and best for analytics and storage efficiency; TFRecord is TensorFlow’s binary row-oriented format optimized for sequential reads during training.Schema evolution:- JSON: no enforced schema; flexible but brittle—consumer must handle missing/extra fields manually.- Avro: explicit schema stored with data; good forward/backward compatibility and safe evolution.- Parquet: stores schema and supports evolution (adding columns) but stricter than Avro for complex changes.- TFRecord: no built-in schema; conventionally paired with a tf.Example spec—evolution must be managed by code.Compression & storage efficiency:- JSON: large (text), compressible but still bulky.- Avro: binary + block compression → compact for row data.- Parquet: excellent compression due to columnar layout and encoding (dictionary, RLE) → best for numeric features and wide tables.- TFRecord: compact binary; combined with compressed TFRecord (e.g., GZIP, ZSTD) is efficient for sequential training.Read performance (random vs columnar):- JSON/Avro/TFRecord (row formats): good for row-wise/random access, example-by-example training.- Parquet (columnar): excellent for selective column reads, analytics, feature selection; poor for reading full examples unless columns are co-located or read together.Splittability / parallel processing:- JSON: splittable only if newline-delimited JSON (NDJSON); large files can be split but parsing overhead.- Avro: splittable at block boundaries; good for distributed processing.- Parquet: splittable by row groups; designed for parallel reads.- TFRecord: splittable if using record offsets or sharded TFRecords; TensorFlow data pipeline handles sharding well.Compatibility with frameworks:- TensorFlow: TFRecord + tf.data is native and high-performance; TensorFlow also reads TFRecord via Feature protos.- PyTorch: no native TFRecord; common to use Parquet/Avro/NDJSON + pandas or custom Dataset. Parquet + pyarrow is popular for efficient column access and conversion to tensors.- General: Avro/Parquet integrate well with Spark, Beam, BigQuery and cloud storage; JSON is universal but less efficient.Recommendation:- For large-scale training with many numeric features and analytics: store raw features in Parquet, materialize TFRecord shards for high-throughput TensorFlow training.- For rapid iteration or small datasets: NDJSON or Avro (if you need schema guarantees).- For TensorFlow production pipelines: TFRecord for training performance; for PyTorch workflows prefer Parquet or sharded binaries/LMDB and convert on-the-fly.
HardTechnical
35 practiced
Design an algorithm and practical pipeline for joining a 1TB user profile table to a 10TB events table for nightly feature computation on a cluster with limited shuffle capacity. Discuss bloom filters, map-side joins, repartition heuristics, multi-stage joins, and how to minimize network shuffle while preserving correctness.
Sample Answer
Situation/Goal: nightly feature compute joining 1TB user_profile (UP) to 10TB events (EV) on a cluster with constrained shuffle capacity. Objective: minimize network shuffle while preserving correctness and reasonable run-time.Design overview (multi-stage, shuffle-minimizing):1. Partition-aligned map-side join where possible- Pre-partition UP by user_id (hash) into P partitions sized to fit per-worker memory/disk budget (target partition ~1–5 GB). Persist to distributed storage with partition ids in filenames so executors can read corresponding UP partition locally when computing features for that partition of EV.- Partition EV by same hash function so most joins are map-side (no shuffle) if partitions are co-located or read together on the same executor.2. Bloom-filter pre-filtering to reduce EV volume- Build a Bloom filter over UP user_ids (or per-UP-partition Bloom filters) and broadcast the filter(s) to executors. For 1TB UP, per-partition Bloom filters are small (size tunable to desired false-positive rate, e.g. 1%).- Filter EV: drop events whose user_id is definitely absent in UP. Bloom false positives allowed (they increase join work), but false negatives impossible => correctness preserved.3. Two-tier strategy for skew and capacity limits- Sample key frequency to detect heavy hitters. For heavy users, handle separately: replicate heavy-UP keys (small set) to executors and perform local joins for their events (map-side), or aggregate their events first to reduce volume.- For remaining keys, perform partitioned map-side join. If cluster cannot co-locate partitions, read partitioned UP from object store in each task and stream-join with its EV partition to avoid a wide shuffle.4. Fallback shuffle-join for leftover mismatches- Some partitions may still be too large. For those, do a controlled repartition (shuffle) with heuristics: split by key salt to increase parallelism and keep per-task size below threshold; set spark.sql.shuffle.partitions to match P and tune spark.reducer.maxSizeInFlight and memory configs.Practical Spark pipeline (steps):1. Sample EV to estimate cardinalities, detect skew, choose P = ceil((size_EV_filtered / desired_partition_size)).2. Create UP partition files by hash(user_id, P). For each UP partition also store a Bloom filter summarizing its keys.3. Broadcast per-partition Bloom filters (or a compact global Bloom if memory allows).4. Map stage: read EV, for each event compute partition_id; consult Bloom for that partition; if negative -> drop; if positive -> write event to local per-partition intermediate file (no shuffle) or keep in-memory buffer.5. Reduce/task stage per partition: read UP partition file locally (or from HDFS/S3), stream join with corresponding filtered EV. Use sort-merge within partition or hash-join if memory fits.6. For partitions where EV or UP exceed resource budget, split by salting and trigger a controlled shuffle-join only for those slices.Repartition heuristics and tuning:- Target partition size: choose based on executor memory and acceptable spill; e.g., 1–5 GB raw input per task.- Compute P = max(minPartitions, ceil(filtered_EV_size / target_size)).- Use sampling to refine P; auto-scale Bloom size: m = -n ln(p) / (ln2)^2 for desired false positive p.- For skew: top-k keys replicated; salt remaining keys into S subpartitions to avoid reducers > target_size.- Limit concurrent shuffle with spark.dynamicAllocation and cap spark.sql.shuffle.partitions to cluster shuffle bandwidth.Correctness and trade-offs:- Bloom filters only remove items that are definitely absent -> no false negatives; only false positives lead to extra join cost, not incorrect results.- Map-side joins preserve correctness if partitioning function identical; ensure deterministic hashing and consistent number of partitions.- Replicating heavy keys trades network for compute but avoids stranding a single reducer.- Controlled shuffles applied only to problematic partitions keep global shuffle low.Monitoring & operational points:- Log per-partition sizes; fail fast if a partition exceeds threshold and auto-split.- Tune Bloom false positive rate to balance network vs compute (lower p => larger Bloom broadcast).- Use compressed intermediate files and streaming joins to reduce memory pressure.- Validate nightly by sampling joined outputs and cross-checking counts against a full (but expensive) baseline run.This pipeline minimizes cluster-wide shuffle by using Bloom filters and map-side or local partition reads for the common case, handles skew with replication/salting, and resorts to controlled shuffles only where necessary, preserving correctness while respecting limited shuffle capacity.
MediumTechnical
32 practiced
Your pandas-based feature transformation pipeline runs slowly and OOMs on a 32GB worker. Describe a structured approach to optimize it: profiling steps, vectorization opportunities, reducing intermediate copies, memory layout (dtypes, categorical encoding), and when to migrate to Dask or Spark. Provide concrete examples.
Sample Answer
Start with profiling and reproducible failure:- Reproduce on a sample worker with the same data subset. Use memory/profiler tools: memory_profiler (mprof), tracemalloc, pandas_profiling for column stats, and line_profiler to find hotlines.- Example: mprof run -o run.dat python transform.py; mprof plot.Optimize algorithm & vectorize:- Replace row-wise apply/lambdas with vectorized ops or NumPy. Example:- Use boolean masking and .loc instead of iterating.Reduce intermediate copies:- Chain operations carefully and reassign to same column to avoid temporaries. Use inplace where safe (note pandas deprecation caveats).- Example: avoid df = df.merge(...).assign(...).pipe(...) on huge tables; do merges selectively and drop unneeded columns immediately: del df['tmp'] ; gc.collect().Memory layout & dtypes:- Downcast numerics: df['x']=pd.to_numeric(df['x'], downcast='unsigned')- Convert strings with low cardinality to category:- Use nullable dtypes only when needed. For timestamps, use datetime64[ns] and consider int64 epoch if faster.- Use memory usage check: df.memory_usage(deep=True).sort_values(ascending=False)Sparse and chunking strategies:- Use pd.read_csv(..., dtype=..., usecols=..., chunksize=1_000_000) to stream and transform per chunk, aggregating results.When to migrate to Dask/Spark:- Stay in pandas if dataset fits comfortably in ~2-3x RAM (after optimizations). Migrate when: - Single-node memory can't handle peak working set even after downcasting/chunking. - You need parallel processing across cores for I/O/CPU bound transforms. - You require distributed storage/compute or integration with cluster resources.- Dask: minimal API changes, good for scaling pandas code on a single cluster; example: import dask.dataframe as dd; ddf = dd.read_parquet(...); use ddf.map_partitions for custom ops.- Spark: choose when you need robust fault tolerance, heavy shuffles, or integrate with Hadoop/S3 at scale. Reimplement heavy UDFs in native Spark SQL or use pandas UDFs (PySpark) carefully.Final checklist:- Profile -> vectorize hot paths -> reduce copies -> tune dtypes & categories -> stream/chunk -> pick Dask for pandas-like scaling, Spark for large distributed ETL and production pipelines.
python
# slow
df['len'] = df['text'].apply(len)
# fast
df['len'] = df['text'].str.len()python
df['cat']=df['cat'].astype('category')HardTechnical
42 practiced
Describe a robust checkpointing and recovery strategy for a long-running streaming feature transformation job that must support operator upgrades and schema evolution with minimal data loss. Include how to version state, migrate state when the operator's schema changes, and validate restored state post-upgrade.
Sample Answer
Requirements and constraints:- Continuous low-latency stream processing with durable checkpoints, support rolling operator upgrades and schema evolution, minimal data loss, verifiable correctness after restore.Design overview:- Use external durable checkpoint store (e.g., object store with strong consistency like S3 + manifest, or RocksDB-backed state with write-ahead logs) and a streaming engine that supports consistent snapshots (Flink/Beam/Streaming SQL).- Each checkpoint includes: operator_id, state_version, state_schema_id, checkpoint_offset for source(s), and checksum/signature.Versioning state:- Every operator state carries metadata: operator_id, semantic_version (major.minor.patch), schema_id (hash of protobuf/Avro/JSON schema), migration_plan_id, and creation_timestamp.- Persist state blobs tagged by (operator_id, schema_id, version). Maintain registry of schema definitions and migration functions.State migration strategy:- Prefer backward-compatible schema changes. For non-compatible changes, implement explicit migration functions: - Online lazy migration for large state: keep state in original shape but add an on-read adapter that transforms to new schema lazily; write-back in new format on first update. - Bulk migration for critical structural changes: run a one-time migration job that reads snapshots, applies deterministic transformation, writes new-versioned snapshots; lock operator checkpoints or quiesce input to ensure consistency.- Provide deterministic idempotent migrations and record migration provenance in checkpoint metadata.Operator upgrade workflow:1. Deploy operator code with supported old_version compatibility flag.2. Start new operator in dual-mode (read-old, write-new) if supported for seamless handoff.3. Take a consistent checkpoint (quiesce sources if needed).4. Promote new operator to active; if migration required, either lazy-migrate during processing or trigger bulk migration using the snapshot.Validating restored state:- After restore, run built-in sanity checks: - Schema conformance: validate state against expected schema_id and checksum. - Row counts / key counts: compare pre-upgrade snapshot metrics. - Replay a short deterministic replay of sample inputs (canary inputs) to validate outputs match expected golden outputs. - End-to-end shadow run: route a percentage of traffic to restored job and monitor metrics (latency, loss, model scores) against baseline.- Maintain automated rollback path: keep previous checkpoints and allow restoring to previous operator version within SLA.Operational best practices:- Keep migration code in version control with tests (unit + property-based). Use protobuf/Avro with compatibility rules when possible.- Monitor checkpoint durations, lag, and failed migrations; alert on checksum/state-mismatch.- Automate migration orchestration and verification in CI/CD pipelines and provide documentation for manual emergency migrations.Why this works:- Metadata versioning + idempotent migrations enable safe upgrades.- Lazy migration reduces downtime for large states; bulk migration offers deterministic, auditable transformations.- Validation steps provide high confidence and a clear rollback path minimizing data loss.
Unlock Full Question Bank
Get access to hundreds of Data Processing and Transformation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.