Design and operate data pipelines and stream processing systems to guarantee correctness, durability, and predictable recovery under partial failures, network partitions, and node crashes. Topics include delivery semantics such as at most once, at least once, and exactly once and the trade offs among latency, throughput, and complexity. Candidates should understand idempotent processing, deduplication techniques using unique identifiers or sequence numbers, transactional and atomic write strategies, and coordinator based or two phase commit approaches when appropriate. State management topics include checkpointing, snapshotting, write ahead logs, consistent snapshots for aggregations and joins, recovery of operator state, and handling out of order events. Operational practices include safe retries, retry and circuit breaker patterns for downstream dependencies, dead letter queues and reconciliation processes, strategies for replay and backfill, runbooks and automation for incident response, and failure mode testing and chaos experiments. Data correctness topics include validation and data quality checks, schema evolution and compatibility strategies, lineage and provenance, and approaches to detect and remediate data corruption and schema drift. Observability topics cover metrics, logs, tracing, alerting for pipeline health and state integrity, and designing alerts and dashboards to detect and diagnose processing errors. The topic also includes reasoning about when exactly once semantics are achievable versus when at least once with compensating actions or idempotent sinks is preferable given operational and performance trade offs.
HardTechnical
30 practiced
Compare coordinator-based two-phase commit (2PC) and eventual-consistency with compensating transactions for coordinating writes to multiple heterogeneous sinks. For each approach discuss failure modes, impact on throughput/latency, operational complexity, and provide examples where 2PC is justified versus where compensation is preferable.
Sample Answer
High-level summary: coordinator-based 2PC enforces atomic, synchronous commits across participants; eventual consistency with compensating transactions (sagas) accepts temporary divergence and resolves it later with compensating actions. For data engineers working with heterogeneous sinks (RDBMS, data warehouses, object stores, pub/sub, third-party APIs), the choice hinges on consistency guarantees required, latency/throughput targets, and operational cost.Failure modes- 2PC: - Coordinator crash after PREPARE: participants may block holding locks until coordinator recovers (blocking). - Participant crash after committing: coordinator and others may be inconsistent until recovery. - Network partitions can leave participants in uncertain state; manual intervention sometimes required. - Global deadlocks and lock contention across heterogeneous systems.- Compensation: - Compensating action may fail or be non-idempotent; full undo may be impossible (side-effects, irrevocable external actions). - Long windows of inconsistency create correctness and business-risk exposure. - Complexity in defining and testing compensation logic; eventual reconciliation may be imperfect (semantic drift).Impact on throughput and latency- 2PC: - High write latency: commit requires round-trips (prepare + commit) to all participants synchronously. - Throughput reduced due to locks and serialized coordination; participants may hold resources, reducing concurrency. - Scalability limited across many heterogeneous sinks.- Compensation: - Lower write latency and higher throughput because commits are local/async; system accepts the write and applies changes to other sinks asynchronously. - Backpressure can occur in the async pipeline but is often easier to scale (parallel workers, batching). - Tail-latency moves from write path to reconciliation/compensation path.Operational complexity- 2PC: - Needs distributed transaction manager (XA, two-phase commit coordinator), drivers that support transactional protocol for each sink. - Hard to integrate heterogeneous systems (data warehouses, S3, external APIs rarely support XA). - Recovery procedures, monitoring for blocked transactions, and coordinated upgrades add ops burden.- Compensation: - Need to design, implement, and test compensating flows for every failure scenario. - Requires robust idempotency, deduplication, visibility into pending/in-flight operations, and reconciliation tooling (dashboards, automated retries). - More code surface and business-logic ownership: compensation semantics often domain-specific.Examples where 2PC is justified- Financial core systems where atomicity is legally/morally required and participants support XA: e.g., ledger transfers across core banking databases that must not split.- Low-latency, small-scale OLTP where participants are few, homogeneous, and under your control (same DB cluster or tightly-coupled microservices with transactional middleware).- When regulatory/compliance demands strict synchronous consistency and manual reconciliation is unacceptable.Examples where compensation is preferable- Data pipelines writing to heterogeneous sinks (data warehouse, object store, search index, analytics DB, third-party API): many sinks lack distributed transaction support.- High-throughput ingestion where write latency matters and eventual correctness is acceptable (analytics, metrics, telemetry).- Multi-step business workflows spanning services where each step has its own local transaction and compensations are well-defined (order fulfillment, inventory reservations).- Integrations with external vendors where rollback is not possible; instead issue compensating refunds or adjustments later.Practical patterns and hybrids- Transactional outbox + CDC: write an event to a local transactional outbox within the primary DB transaction (local atomicity), then asynchronously deliver to sinks. Achieves durability + eventual consistency without 2PC.- Idempotent consumers, deduplication, exactly-once sinks where possible (cloud streaming with exactly-once semantics).- Compensation + reconciliation jobs and strong observability: automated retries, human-in-the-loop dashboards for unresolved cases.- Limit 2PC to narrow, critical domains; use eventual consistency for broad, heterogeneous pipeline sinks.Recommendation for data engineers- Default to eventual consistency + outbox/CDC for heterogeneous sinks: simpler integration, better throughput, and realistic for analytics workloads.- Reserve 2PC only when strict, synchronous atomicity across participating systems is mandatory and all participants support it.- Invest in good monitoring, idempotent design, well-tested compensations, and reconciliation processes to make eventual consistency reliable.
MediumTechnical
33 practiced
Design a high-throughput streaming deduplication mechanism for messages that include a unique sequence number per entity (aggregate id). Discuss state-store choices (in-memory, RocksDB, external KV like Redis), TTL strategies to bound state, checkpointing considerations, handling partition rebalances, and how to reconcile dedupe state after consumer failures.
Sample Answer
Requirements & approach:- Goal: high-throughput streaming dedupe per aggregate id using sequence numbers; tolerate rebalances and consumer failures; bound state size and ensure correctness (at-least-once or exactly-once semantics as required).State-store options (trade-offs):- In-memory: lowest latency, highest throughput but volatile and limited size; suitable for short TTLs and ephemeral caches on single instance.- RocksDB (embedded): durable, local disk-backed, supports large state per partition, good for Flink/Kafka Streams; moderate latency, fast local reads/writes.- External KV (Redis/Cloud KVS): centralizes state, can scale independently, supports TTLs and replication. Adds network latency and becomes an operational dependency; requires careful sharding to match stream partitioning.TTL strategies to bound state:- Use per-aggregate sliding TTL based on max allowed out-of-order/window duration (e.g., keep highest seen sequence + watermark lag). Evict aggregates after inactivity window (max event lag + business retention).- For sequence-based dedupe, store only highest-seen seq and a small bitmap/window for recent gaps; TTL on aggregate resets on activity.- Periodic compaction: background job scans and removes state older than retention.Checkpointing & consistency:- Prefer framework-managed state checkpoints (e.g., Flink Checkpoints or Kafka Streams changelog) to persist RocksDB/embedded state atomically with offsets. This enables exactly-once recovery.- If using external KV, coordinate writes with offset commit (two-phase or idempotent writes) or log state changes to a changelog topic so you can restore state deterministically.Partition rebalances & failure handling:- Ensure state is partition-aligned: one consumer instance owns set of partitions and corresponding local state. On rebalance, use framework hooks to snapshot and transfer state.- With RocksDB + changelog/checkpoints: after rebalance the new instance restores state from checkpoint/changelog before processing assigned partitions.- With external KV: no local state transfer needed; new consumer can read KV for partition keys. Still must ensure no double-processing: commit offsets only after state update (or use idempotent state updates).Reconciling dedupe after failures:- On restart, restore state from the checkpoint/changelog; start processing from the last committed offsets.- If using external KV without coordinated commits, run a reconciliation pass: for each partition/aggregate, compare max-seen sequence in KV vs incoming events; drop events <= stored seq. For gaps (missed events), either accept or request replay upstream depending on business SLA.- When exact ordering matters, use watermarking and hold processing until reorder window passes; use sequence ranges to detect duplicates and gaps, and optionally emit tombstones or metrics for manual reconciliation.Operational considerations:- Monitor state size, eviction rates, duplicate rate, and lag metrics.- Tune retention to balance memory/disk and correctness.- Prefer embedded stores + framework checkpoints for strongest correctness with minimal network overhead; use external KV when cross-consumer reads or very large shared state required.
EasyTechnical
35 practiced
Differentiate between a Dead Letter Queue, a poison pill message, and transient failure retries in a streaming system. Provide an operational decision flow: when to retry, when to DLQ, when to alert an engineer, and how many retries/backoff strategy you'd apply for transient errors.
Sample Answer
Dead Letter Queue (DLQ), poison pill, and transient retries — definitions and operational flow for a streaming data pipeline:Definitions:- Dead Letter Queue (DLQ): a side queue to store messages that cannot be processed after defined retry attempts or that fail schema/validation checks. DLQ preserves payload + metadata for later inspection/reprocessing.- Poison pill message: a specific message that consistently breaks processing (e.g., malformed JSON, corrupt payload, or triggers deterministic exception). It causes repeated failures and blocks progress if not isolated.- Transient failure retries: temporary errors (network timeouts, downstream rate limits, transient DB locks) that typically succeed on retry.Operational decision flow:1. Attempt immediate processing.2. If error => classify: - Transient (timeouts, 5xx, network): retry with backoff. - Deterministic/data error (schema, validation, parse): treat as poison pill → send to DLQ immediately (no retry). - Unknown intermittent: retry a few times, then DLQ if still failing.3. After DLQ, generate alert if: - DLQ rate spikes above threshold or same message repeatedly arrives (indicates upstream systemic issue). - Number of DLQ entries for a job in short window exceeds SLA.Retry/backoff policy (practical defaults):- Retries: 3 attempts (initial + 2 retries) for many data pipelines. For critical downstreams, up to 5 with caution.- Backoff: exponential backoff with jitter, e.g., base 1s: 1s, 2s, 4s (+ random ±25% jitter).- For rate-limited responses use "Retry-After" header if available.Operational notes:- Idempotency: ensure consumers are idempotent to safely retry.- Monitoring: track retry counts, DLQ size, and poison-pill frequency; alert on anomalies.- Reprocessing: provide tooling to inspect, fix, and replay DLQ messages (with audit trail).This approach minimizes lost data, avoids blocking pipelines on poison pills, and limits noisy retries for transient issues while surfacing real problems to engineers.
MediumSystem Design
42 practiced
You must replay 90 days of events into a partitioned Parquet lake to fix a bug. Propose a backfill plan that guarantees no duplicates, maintains partitioning (date/hour), is efficient for large volumes, includes validation steps, and provides a rollback strategy. Mention tools (Spark/Beam), staging patterns, and verification checks.
Sample Answer
Requirements & constraints:- Replay 90 days of events into existing partitioned Parquet lake (partitioned by date/hour)- No duplicates -> idempotent- Maintain partitioning (date/hour)- Efficient at large volume- Include validation & rollbackPlan (high level):1. Clarify scope: exact event source, unique key (event_id), authoritative timestamp, data schema, retention, SLAs.2. Tooling: Apache Spark (Spark Structured Streaming / batch) or Apache Beam on Dataflow for parallel, scalable transforms. Use cloud object storage (S3/GCS/ADLS) with atomic rename semantics or metastore-managed partitions (Hive/Glue).Backfill pattern:- Process per partition (date/hour) in parallel batches (e.g., day-level parallelism, hour sub-partitions) to maximize throughput without overwhelming HDFS/S3 listing.- For each target partition: a. Read raw events for that partition from source. b. Apply deterministic transforms and compute a partition key, event_id, and event_ts. c. Deduplicate in memory using keyed aggregation: keep latest event per event_id (e.g., max(event_ts) or last_write_wins). d. Write results to a staging location: /lake/staging/YYYY-MM-DD/HH/_tmp-<jobid>/ as Parquet with target partition layout. Use parquet file sizes ~128–512MB and enable vectorized writes. e. Produce a manifest/metadata file for the partition (record counts, checksum, min/max ts, sample hashes).Atomic publish:- Validate staging (see next). If OK, perform atomic swap: - Option A (metastore): use ALTER TABLE REPLACE PARTITION or IMPORT partition metadata pointing to new path and mark old partition archived. - Option B (object store): move staging folder to final path via single rename (or copy+delete when rename unsupported). Update Hive/Glue partition location atomically.- Keep previous partition copy at /lake/archive/YYYY-MM-DD/HH/ for rollback (or snapshot via versioned bucket).Efficiency considerations:- Parallelize by partition range; use cluster autoscaling.- Use predicate pushdown, column projection, and only read necessary columns.- Use map-side aggregation for dedupe to reduce shuffle.- Compact small files after publish if needed via a background compaction job.Validation steps (per partition):- Record-level checks: - Count match between source read and final written after dedupe (expected final_count <= source_count). - Unique(event_id) enforced. - Min/max timestamps within partition bounds. - Row-level sample hash: compute hash of deterministic sort of keys and compare against expected hash from staging.- End-to-end checks: - Compare pre-backfill aggregates (e.g., counts by user, metric sums) to post-backfill to ensure intended delta. - If there is an upstream canonical store (DB), cross-check random samples.- Monitoring: emit metrics to Prometheus/Cloud Monitoring: processed partitions, rows in/out, validation failures.Rollback strategy:- Never overwrite live partition in-place. Always publish with atomic swap and keep archived copy (move old data to /lake/archive/YYYY-MM-DD/HH/ with timestamp).- If validation fails after publish, re-validate; if irrecoverable, revert by moving archived copy back into place and updating metastore.- Keep logs, manifests, and job ids for traceability. Automate rollback with playbooks and a single CLI command per partition.Safety & orchestration:- Use an orchestrator (Airflow, Prefect) to schedule per-partition jobs, enforce dependency graph (process older partitions first or in controlled parallelism), and enable safe retries.- Limit parallelism to not overwhelm catalog or small-file explosion.- Run a dry-run on a sample day and a pilot on last 7 days before full 90-day run.Example Spark sketch (conceptual):
Final notes:- Emphasize idempotency (dedupe by event_id), atomic publish, per-partition staging and archiving, automated validation, and an executable rollback path. This minimizes risk while keeping the backfill efficient and auditable.
HardTechnical
33 practiced
Explain the Chandy-Lamport distributed snapshot algorithm. Describe how you would adapt it to implement consistent checkpoints across cooperating microservices that exchange events (over Kafka or message queues), and discuss practical challenges such as capturing in-flight messages and integrating with persisted broker logs.
Sample Answer
Brief explanation of Chandy‑Lamport:Chandy‑Lamport produces a consistent global snapshot of a distributed system without stopping it. A process that initiates sends a special marker on all outgoing channels, records its local state, and then records the state of each incoming channel as the sequence of messages that arrive after recording local state and before receiving a marker on that channel. When a process receives a marker the first time it records its local state and sends markers on its outgoing channels; subsequent markers delimit the in‑flight messages for that channel. The result is a set of local states + channel histories that together form a consistent cut.Adapting to cooperating microservices over Kafka/message queues:- Use the marker concept mapped to checkpoint barriers: a centralized coordinator (or leader) issues a checkpoint id. Each service: - Records its local state (pending timers, in‑memory buffers, application offsets). - Captures and persistently records current consumer offsets per topic-partition it reads. - Sends a marker event (or commits a special transactional message) to outgoing topics so downstream services learn the checkpoint boundary. - Optionally pause processing until checkpoint metadata is durably stored (or continue with careful offset management).- For Kafka specifically, leverage Kafka transactional API and consumer offsets: - Producers write output within a transaction tagged with checkpoint id; commitTransaction ensures downstream consumers won't see partial outputs. - Consumers atomically commit offsets for the checkpoint (using group offsets or an external offset store) only after local state is persisted. - Persist checkpoint metadata: service local state snapshot + committed offsets + checkpoint id in durable storage (S3, DB, or Kafka-compacted topic).Practical challenges and mitigations:- Capturing in‑flight messages: queues decouple sender/receiver; in‑flight is represented by offsets plus uncommitted transactional writes. Record consumer offsets and transactional producer states. Using Kafka transactions converts "in‑flight" to either committed or aborted — simplifies snapshot correctness.- Exactly‑once vs at‑least‑once: without transactions you must make operations idempotent and handle duplicates on restore. Prefer Kafka transactions + idempotent producers where possible.- Coordinating many services: coordinator must be fault tolerant. Use leader election and retries; include timeout/backoff to avoid blocking healthy pipelines.- Performance and availability: synchronous global checkpoints can add latency. Use asynchronous, incremental, or layered checkpoints (local frequent, global less frequent) and incremental state snapshots (change logs).- Integration with persisted broker logs: checkpoint should store per‑partition offsets and producer transaction markers; on restore, consumers seek to those offsets and producers resume with new producer ids or recover transactions if supported.- External side effects (databases, third‑party APIs): require two‑phase commit or compensating actions; prefer storing side effects in durable logs and reapplying idempotently.- Distributed failures: ensure checkpoints are atomic—only promote a checkpoint as valid when all participants acknowledged; support rollback and replay by seeking consumers to recorded offsets and replaying producer outputs from committed logs.Trade‑offs:- Strong consistency with global synchronous checkpoints increases latency and complexity; eventual consistent, offset‑based checkpoints with idempotent processing is often more practical for high‑throughput data pipelines.- Use Kafka’s built‑in primitives (transactions, compacted topics for checkpoint metadata) when available to reduce complexity and achieve near‑exactly‑once semantics.This approach delivers consistent checkpoints by combining Chandy‑Lamport’s logical markers with broker features (offsets, transactions) and durable metadata storage, while acknowledging operational trade‑offs around latency, complexity, and external side effects.
Unlock Full Question Bank
Get access to hundreds of Data Reliability and Fault Tolerance interview questions and detailed answers.
