Break complex problems into smaller, manageable subproblems and solution components. Demonstrate how to identify the root problem, extract core patterns, choose appropriate approaches for each subproblem, sequence work, and integrate partial solutions into a coherent whole. For technical roles this includes recognizing algorithmic patterns, scaling considerations, edge cases, and trade offs. For non technical transformation work it includes logical framing, hypothesis driven decomposition, and measurable success criteria for each subcomponent.
EasyTechnical
67 practiced
List and explain three common decomposition patterns used in ETL pipeline design (for example: extract-transform-load split, map-reduce style, modular operator pipelines). For each pattern describe typical components, parallelism characteristics, failure domains, and at least one concrete technology example (e.g., Spark, Flink, Airflow).
Sample Answer
1) Extract → Transform → Load (ETL split)- Typical components: source connectors (ingest), staging layer, transformation engine, target loader, orchestration/scheduling.- Parallelism: coarse-grained — extracts and loads often run per-source or per-table in parallel; transformations can be parallelized by partition/key (e.g., partitioned SQL or Spark jobs).- Failure domains: per-task/job level (failed extract, corrupt staging data, transform job crash). Isolation via retries, idempotent loaders, and checkpointed staging reduces blast radius.- Example tech: Airflow (orchestration) + Spark for transforms + S3/Redshift for staging/warehouse.2) Map-Reduce / Parallel Batch- Typical components: distributed mapper tasks (local transform), shuffle/sort, reducer tasks (aggregation/writes), job manager.- Parallelism: fine-grained data parallelism — split input into many map tasks processed concurrently; shuffle causes network/IO bottlenecks.- Failure domains: node/task failure during map or reduce; shuffle/network hotspots; can re-run failed tasks but expensive for long jobs.- Example tech: Hadoop MapReduce, Apache Spark (RDD/DataFrame jobs using map/reduceByKey).3) Modular Operator / Stream Processing Pipeline- Typical components: event source, stream processors/operators (filter, enrich, windowed aggregations), state store, sink, stream orchestrator.- Parallelism: operator-level parallelism with partitioned streams; scale by increasing task/slot count per operator; backpressure handling.- Failure domains: operator instance or partition-level failure; state loss without durable checkpoints; late/event-time handling complexity.- Example tech: Apache Flink (exactly-once stateful processing) or Kafka Streams.For each pattern choose monitoring, idempotency, and checkpointing strategies to limit failure impact and enable safe retries.
MediumSystem Design
73 practiced
You are migrating batch ETL to near-real-time analytics using Kafka + Spark Structured Streaming. Decompose the migration into phases: discovery, dual-run (batch + streaming), schema compatibility, backfill strategy, consumer updates, testing, and cutover. For each phase, list artifacts, success metrics, and rollback strategy.
Sample Answer
Discovery- Artifacts: inventory of current batch jobs, data schemas, SLAs (latency/throughput), data volume/time distribution, dependency graph, Kafka cluster sizing plan, cost estimate.- Success metrics: complete inventory, identified 100% of upstream/downstream dependencies, latency target defined (e.g., <60s), capacity plan validated.- Rollback: keep batch runbook and schedule unchanged; mark migration as advisory until dual-run phase.Dual-run (batch + streaming)- Artifacts: streaming job prototypes (Spark Structured Streaming), connector configs (Kafka producers/consumers), orchestration updates, monitoring dashboards showing both outputs.- Success metrics: streaming output parity with batch within error bounds (e.g., ≤0.5% diff), sustained throughput for peak load for 72h, resource utilization within budget.- Rollback: stop streaming consumers, continue relying on batch; snapshot streaming code/config for troubleshooting.Schema compatibility- Artifacts: schema registry entries (Avro/Protobuf/JSON Schema), compatibility rules (backward/forward/frozen), migration plan for field renames/nullable changes, transformation mapping.- Success metrics: all producers/consumers validated against registry, no schema violation errors in prod pre-run, automated tests passing.- Rollback: enforce producer-side validation to previous schema; if incompatible changes deployed, revert producer config or use compatibility mode in registry.Backfill strategy- Artifacts: backfill plan (range, throttling, deduplication logic), idempotent write strategy to downstream stores, tooling for historical replay (batch job reading historical source → Kafka topics).- Success metrics: complete backfill within maintenance window or steady-state progress metrics, no duplicates or gaps, downstream queries return expected results.- Rollback: stop backfill, truncate/restore downstream partitions from pre-backfill snapshot if unacceptable; use tombstone messages or compensating transactions if needed.Consumer updates- Artifacts: consumer migration guides, feature-flagged consumers supporting both batch and streaming inputs, client libraries, testing harness.- Success metrics: 100% of critical consumers successfully read streaming data in test env, graceful fallback to batch, consumer lag < defined SLA.- Rollback: flip feature flag back to batch-only; retain both endpoints until cutover validated.Testing- Artifacts: unit/integration tests, end-to-end QA environment, data parity checks, performance/load tests, monitoring/alerting playbooks.- Success metrics: automated tests coverage, parity metrics within tolerance, performance tests meet latency/throughput targets, runbook validated in staged failover.- Rollback: use test failures to halt rollout; revert configs, restore test datasets.Cutover- Artifacts: cutover checklist, final runbook, stakeholder communication, monitoring dashboards, post-cut metrics plan.- Success metrics: sustained streaming correctness and latency for defined observation window (e.g., 48h), incident rate ≤ baseline, cost within projection.- Rollback: execute cutover rollback plan—disable streaming consumers, re-enable batch schedules, restore any impacted downstream state from snapshots, run post-mortem and iterate.Cross-cutting considerations- Observability artifacts: end-to-end lineage, DLQ for malformed messages, metrics (lag, throughput, error rates), SLAs.- Security/compliance artifacts: access controls, encryption configs.- Rollback strategies emphasize safe toggles (feature flags), immutable snapshots, and ability to resume batch processing with zero data-loss.
HardSystem Design
62 practiced
Design a comprehensive testing decomposition for data pipelines: unit tests for transforms, integration tests for connectors, contract tests for producers/consumers, schema tests, end-to-end staging, and CI gating. Provide examples of test inputs, expected outputs, and a strategy for non-deterministic streaming transforms.
Sample Answer
Requirements & constraints:- Validate correctness, schema, contracts, and resilience for batch & streaming (Kafka), support deterministic CI, and allow reproducible staging E2E tests.Testing decomposition (high level):1. Unit tests for transforms- Test function-level logic with small fixtures. Use pytest + hypothesis for property-based checks.- Example input (JSON row): {"user_id": "u1","ts":"2025-01-02T10:00:00Z","action":"click","value":"10"}- Expected output (dict): {"user_id":"u1","action":"click","value":10,"is_valid":True}- Use in-memory runners (pandas, Spark local mode) and assert edge cases (nulls, type coercion, rounding).2. Integration tests for connectors- Run connector code against ephemeral test services (Testcontainers / LocalStack / Embedded Kafka). Validate read/write semantics and retries.- Example: push 10 Avro messages to Kafka topic; expect connector to write 10 rows to staging table with same keys and offsets committed.3. Contract tests (producer/consumer)- Use Pact or a schema-based contract approach (Avro/Protobuf + compatibility rules). Consumers publish expected fields & types; producers run consumer contract verification.- Example contract: topic "events" schema v1: required fields user_id:string, event_ts:long. CI fails if producer writes incompatible schema.4. Schema tests & data quality- Use Great Expectations or custom validators to check schema, ranges, uniqueness, foreign-key-like references.- Example expectation: event_ts within last 90 days; user_id regex ^[a-z0-9_]+$; value >= 0.5. End-to-end staging- Deploy pipeline to staging with seeded datasets; run full ingestion -> transform -> materialized view. Use replayable fixtures (fixed timestamps, deterministic partitioning).- Validate row counts, sample golden rows, data lineage (source offset -> output).6. CI gating & observability- Gate merges on: all unit tests pass, integration smoke tests (connectors) pass in ephemeral environment, contract verification green, schema & GE checks pass on sample data, and e2e staged smoke succeeds.- Require provenance metrics and test coverage thresholds for transforms.Non-deterministic streaming transforms strategy:- Make transforms idempotent and functionally pure where possible.- For stateful ops (session windows, aggregations): - Use deterministic watermarks/timestamps in tests; seed random generators; fix partitioning keys. - Validate statistical properties rather than exact rows: e.g., counts within tolerance, distribution sketches (t-digest/HyperLogLog) equals expected. - Use time-advance simulation (Beam TestStream / Kafka replay with controlled timestamps) to exercise lateness and windowing. - Golden-approx tests: compare deterministic fields exactly and allow tolerances for non-deterministic metrics with assertions (abs/relative error).Example non-deterministic test:- Feed 1000 events with controlled skewed timestamps; assert that 1) total count ==1000, 2) per-window counts within ±2% of expected, 3) top-5 keys deterministic by sorting keys then comparing.Practical tooling & patterns:- Pytest + Faker/Hypothesis, Spark local or Beam DirectRunner, Testcontainers/LocalStack, Confluent Schema Registry (Avro), Great Expectations, Pact for contracts, CI (GitHub Actions/GitLab) orchestration to spin ephemeral infra, and feature flags to run long E2E nightly while gating quick checks on PRs.Trade-offs:- Full E2E each PR is slow — use fast unit+contract+connector smoke in PRs, schedule full staged E2E nightly or on release branch.- Statistical assertions require careful thresholds to avoid flakiness; tune with historical variance.This decomposition yields fast feedback on logic, safety on schema/contract evolution, reliable connector behavior, and robust validation for non-deterministic streaming behavior.
MediumTechnical
66 practiced
How do you decompose compute and storage trade-offs when choosing between storing denormalized, partitioned Parquet files in a data lake versus loading data into a columnar OLAP store (e.g., Redshift/BigQuery)? Provide a decision matrix that includes query latency, concurrency, cost, maintainability, and freshness.
Sample Answer
Start by framing the two options in terms of where work happens: data-lake (denormalized, partitioned Parquet) pushes work to compute at query time (read + scan + CPU for predicate/filter/projection), whereas a columnar OLAP store pushes work to a managed engine (pre-sorted/encoded/clustered) so queries hit optimized storage formats and indexes. Decompose trade-offs across the requested dimensions:Decision matrix (Parquet in data lake vs Columnar OLAP):- Query latency - Parquet: higher and variable — depends on scan size, partition pruning, and compute cluster startup. Good for large analytical scans but slower for low-latency ad-hoc. - OLAP: low and consistent — pre-optimized, vectorized execution and cached metadata.- Concurrency - Parquet: limited — concurrent users require spinning many compute clusters or a scalable query engine (costly). - OLAP: high — built for many concurrent BI users with queuing and slots.- Cost - Parquet: lower storage cost (cheap object storage). Compute cost shifts to on-demand clusters; cost efficient for infrequent large batch queries. - OLAP: higher storage and ongoing compute costs (storage + reserved/query slots), but better price/per-query at high concurrency/low latency.- Maintainability - Parquet: simpler pipeline, schema-on-read flexibility, but needs well-designed partitioning, compaction, and ETL for denorm’d tables. - OLAP: more operational (schema migrations, vacuuming, distribution keys), but offers managed optimizations reducing ad-hoc tuning.- Freshness - Parquet: depends on ingestion cadence; near-real-time requires micro-batches / streaming compaction (complex). - OLAP: easier for near-real-time if you have streaming ingestion or CDC connectors; supports incremental loads with lower query impact.Guidance: Prefer Parquet for cost-sensitive, large-scale batch analytics where query concurrency/latency are low and storage is dominant. Prefer OLAP for interactive BI, high concurrency, strict SLAs, and predictable performance despite higher ongoing cost. Hybrid: store raw/denormalized Parquet for archival and ETL; ETL into OLAP for curated, high-use datasets. Include lifecycle: monitor query patterns, cost per query, and move hot tables into OLAP.
MediumTechnical
72 practiced
Decompose the cost, complexity, and engineering tasks required to support incremental recomputation in a large batch pipeline so you only recompute partitions with changes: identify idempotent stages, persist lineage or input checksums, compute change sets (diffs), and apply updates to target partitions. Recommend storage layouts and orchestration patterns to support selective reprocessing.
Sample Answer
Situation: You’re asked to add incremental recompute to a large batch pipeline so only partitions with input changes are reprocessed.Decomposition — cost, complexity, engineering tasks- Cost: storage for lineage/checksums/manifests (~small relative to data), compute for diffing and selective runs (saves downstream compute), engineering time to instrument idempotency, testing, and rollout.- Complexity: medium — requires dataset/catalog changes, deterministic transforms, and safe mutation of target partitions; increases operational surface (metadata, retries).Engineering tasks (concrete)1. Identify idempotent stages - Audit pipeline: mark pure-transform stages (map/filter/aggregate without external side effects) as idempotent. - Refactor any stateful/side-effect stages into separate append-only sinks or make them compensatable.2. Persist lineage / input checksums - For each input partition (date/hour/source-file), compute a checksum (e.g., hash of file list + sizes or per-file MD5) and store in a metadata table/manifest (Hive metastore / Glue / Delta/Iceberg table / DynamoDB). - Store last-successful-run manifest per target partition.3. Compute change sets (diffs) - Diff current input manifest vs last manifest to get changed partitions/files. - Optionally use CDC (Debezium) or cloud object storage events to reduce scan cost.4. Selective reprocessing and applying updates - For changed partitions, launch jobs to recompute only affected partitions. - Apply updates atomically: write to a staging area and perform atomic swap/commit (use table formats that support atomic commits: Delta Lake, Iceberg, Hudi). - Ensure downstream consumers see consistent state via versioned snapshots.Storage layouts (recommendations)- Partition by natural keys with manageable cardinality (date / hour / region). Keep partition depth reasonable.- Use columnar formats (Parquet/ORC) for compute efficiency.- Prefer ACID table formats (Delta / Iceberg / Hudi) to simplify atomic replace/merge and time-travel.- Store manifests and checksums in a small, fast metadata store (Glue Catalog / Hive / relational DB / DynamoDB).Orchestration patterns- Orchestrator: Airflow/Dagster/Argo Workflows controlling DAGs per partition or per changed-partition batch.- Pattern: controller job that: 1. Generates diffs 2. Enqueues per-partition tasks (Spark jobs or worker pods) 3. Aggregates results and commits manifests- Use idempotent task retries, backoff, and a transactional commit step.- For high-throughput, use a fan-out/fan-in pattern with autoscaling (Kubernetes or EMR autoscaling).- Add observability: lineage traces, per-partition metrics, SLA checks.Trade-offs- Full snapshot vs incremental: snapshots simpler but costlier. Incremental saves compute but requires metadata rigor.- CDC reduces scanning but demands upstream support.Key best practices- Make transforms deterministic and unit-testable.- Use ACID table formats for safe updates.- Keep metadata small and authoritative.- Build a small “recompute controller” service to orchestrate diffs, idempotent tasks, and atomic commits.
Unlock Full Question Bank
Get access to hundreds of Problem Decomposition interview questions and detailed answers.
