Design and operate data pipelines and stream processing systems to guarantee correctness, durability, and predictable recovery under partial failures, network partitions, and node crashes. Topics include delivery semantics such as at most once, at least once, and exactly once and the trade offs among latency, throughput, and complexity. Candidates should understand idempotent processing, deduplication techniques using unique identifiers or sequence numbers, transactional and atomic write strategies, and coordinator based or two phase commit approaches when appropriate. State management topics include checkpointing, snapshotting, write ahead logs, consistent snapshots for aggregations and joins, recovery of operator state, and handling out of order events. Operational practices include safe retries, retry and circuit breaker patterns for downstream dependencies, dead letter queues and reconciliation processes, strategies for replay and backfill, runbooks and automation for incident response, and failure mode testing and chaos experiments. Data correctness topics include validation and data quality checks, schema evolution and compatibility strategies, lineage and provenance, and approaches to detect and remediate data corruption and schema drift. Observability topics cover metrics, logs, tracing, alerting for pipeline health and state integrity, and designing alerts and dashboards to detect and diagnose processing errors. The topic also includes reasoning about when exactly once semantics are achievable versus when at least once with compensating actions or idempotent sinks is preferable given operational and performance trade offs.
MediumSystem Design
40 practiced
Design an observability plan for stateful streaming operators focused on operator lag, checkpoint duration, state restore time, and silent state corruption. Propose metrics, log entries, tracing spans, example dashboard panels, and three alert thresholds with remediation steps.
Sample Answer
Requirements & goals:- Detect operator lag, slow/failed checkpoints, slow state restores, and silent state corruption quickly and with actionable context.- Support streaming framework (e.g., Flink-style) with per-operator, per-job, per-task granularity.Metrics (Prometheus-style):- operator_processing_lag_seconds{job,operator,partition} gauge (current event-time lag)- operator_max_lag_seconds{...} gauge (max across subtasks)- checkpoint_duration_seconds{job,checkpoint_id} histogram (buckets + p50/p95/p99)- checkpoint_failure_total{job,reason} counter- state_size_bytes{job,operator} gauge- state_restore_duration_seconds{job,operator} histogram- state_digest_mismatch_total{job,operator} counter (detects corruption)- records_processed_total{job,operator} counter- input_backpressure_ratio{job,operator} gauge (0..1)Log entries (structured JSON):- INFO checkpoint.started {"job":"orders","checkpoint_id":123,"operator":"agg","start_ts":...}- ERROR checkpoint.failed {"job":"orders","checkpoint_id":123,"operator":"agg","reason":"timeout","duration_ms":60000}- WARN state.restore.slow {"job":"orders","operator":"join","duration_ms":45000,"expected_ms":5000}- ERROR state.digest.mismatch {"job":"orders","operator":"agg","snapshot_id":45,"expected_hash":"abc","actual_hash":"abd"}Tracing spans (distributed tracing):- Span: "operator.process(event)" tags: job, operator, subtask, event_timestamp, processing_time_ms- Span: "checkpoint.snapshot" tags: checkpoint_id, stage (serialize, write, ack), bytes_written, duration_ms- Span: "state.restore" tags: snapshot_id, operator, duration_ms, state_size_bytes- Propagate trace-id from ingestion -> operator -> sink to correlate lag with upstream.Dashboard panels:1) Operator Lag Heatmap: operator vs time, color by operator_processing_lag_seconds; filter by job2) Checkpoint Latency Overview: p50/p95/p99 of checkpoint_duration_seconds, plus failure rate3) State Restore Duration: per-operator histogram and recent slow restores list4) State Size & Growth: state_size_bytes trend + records_processed_total5) Corruption & Integrity: state_digest_mismatch_total events + linked logs/traces6) Backpressure & Throughput: input_backpressure_ratio + records/secAlerting (with runbooks):1) Critical — Checkpoint failures spike- Condition: checkpoint_failure_total increases by >5 in 5m or checkpoint_duration_seconds p99 > 2× SLO for 3 consecutive checkpoints- Severity: P0- Remediation: - Immediately pause upgrades/rollouts for this job. - Inspect recent checkpoint logs and traces (click dashboard -> checkpoint spans). - Check storage (disk, S3) availability and permissions. - Restart affected task manager(s) if serialization writers stuck; escalate to infra if storage issues persist.2) High — Operator processing lag sustained- Condition: operator_max_lag_seconds > alert_threshold (e.g., 60s) for >10m and increasing trend- Severity: P1- Remediation: - Identify hotspot operator via heatmap and trace top spans. - Scale out parallelism for that operator or throttle upstream. - Check for GC pauses or backpressure in task logs; if GC, tune heap/GC or restart node. - If state size per partition large, consider state compaction or rebalance.3) Warning — Silent state corruption detected- Condition: state_digest_mismatch_total >=1- Severity: P0 (data integrity)- Remediation: - Quarantine job: pause sinks to prevent bad outputs. - Retrieve failing snapshot ID and compare hashes; run recovery from previous known-good checkpoint. - Restore state to a safe checkpoint and replay from upstream offset; open incident and notify data owners. - Add stronger checksum (e.g., CRC64) and periodic snapshot verification; if recurring, investigate storage durability.Notes on instrumentation & reliability:- Expose histogram buckets for checkpoint and restore durations to avoid skews.- Emit logs correlated with trace_id and checkpoint_id for fast triage.- Regularly run synthetic verification job that writes known patterns, verifies digests to catch silent corruption proactively.- Alert cooldowns and runbooks stored in pagerDuty/Confluence with links in alerts.
EasyTechnical
36 practiced
Differentiate between a dead-letter queue (DLQ), a poison message, and a retry policy. Provide a rule set that avoids infinite retry loops and describe how you would surface poison messages for engineering triage.
Sample Answer
Dead-letter queue (DLQ), poison message, and retry policy — definitions and operational rules:Definitions- Dead-letter queue (DLQ): A separate, durable queue/topic where messages that cannot be processed successfully after defined attempts are moved for inspection. Acts as a safety net so processing pipelines can continue.- Poison message: A specific message that consistently fails processing (e.g., malformed payload, missing required fields, or causes deterministic exceptions) and would otherwise block progress if retried indefinitely.- Retry policy: The rules that control how and when a failed message is retried (count, backoff, jitter, escalation).Rule set to avoid infinite retry loops1. Max attempts: Fail fast to DLQ after a finite number of attempts (e.g., 5).2. Backoff strategy: Use exponential backoff with jitter to reduce contention (e.g., base 1s, factor 2, jitter ±20%).3. Error classification: Retry only for transient errors (timeouts, rate limits). For deterministic errors (validation, schema mismatch), immediately route to DLQ.4. Visibility/increment: Track attempt count in message metadata; increment atomically so retries honor limits.5. Circuit breaker: If aggregate failure rate for a message type exceeds threshold, pause automatic retries and alert.6. Time-to-live: Optionally set a total retry window (e.g., 1 hour); after expiry send to DLQ.How to surface poison messages for engineering triage- Enrich DLQ entries with metadata: original headers, schema version, attempt count, timestamps, consumer id, stack trace, and sample payload.- Monitoring & alerts: Emit metrics (DLQ rate, top error types). Create alerts when DLQ rate or specific error spikes above thresholds.- Dashboards and search: Provide a searchable DLQ dashboard (message preview, filters by error, age, consumer). Integrate with logging (correlate message ID → logs/trace).- Triage workflow: Wire DLQ → ticketing system or runbook (template: reproduce steps, impact, suggested remediation). Allow safe replay: sandboxed replay tooling that can reprocess a DLQ message after fixes with dry-run option.- Ownership and SLAs: Assign team(s) responsible for DLQ topics and define SLA for triage and remediation.This approach keeps the pipeline resilient, prevents infinite loops, and makes poison messages visible and actionable for engineers.
EasyTechnical
32 practiced
Explain watermarks and how streaming systems handle out-of-order events. Describe how allowed lateness, watermark advancement strategies, and window eviction policies affect correctness of aggregations and joins under network delays.
Sample Answer
Watermarks are a streaming-system mechanism that estimate event-time progress: a watermark t means “the system believes no more on-time events with event-time ≤ t will arrive.” They let engines trigger time-based operations (window close, join emit) even with out-of-order arrivals.Handling out-of-order events:- Buffering by event-time and using watermarks to decide when to emit results.- Mark late arrivals (event-time < current watermark) for special handling: drop, side-output, or update state if allowed.Key knobs and their correctness effects:- Allowed lateness: a configured grace period after a window’s end during which late events still update the window. Larger allowed-lateness increases correctness (fewer missed updates) but raises state retention and latency/cost.- Watermark advancement strategies: - Periodic heuristic (max observed timestamp − delta): simple but can lag under skew, causing late marks. - Source-driven punctuations: explicit watermarks from producers are more accurate. - Hybrid: combine heuristics with source punctuations for robustness. Fast-forwarding watermarks too aggressively causes incorrectly finalized windows/joins; too slow delays outputs.- Window eviction policies: when state for a window is discarded. Evicting at watermark (plus allowed lateness) ensures correctness; premature eviction loses late events, causing wrong aggregates/joins; overly delayed eviction increases memory and GC pressure.Example: for a 1-minute tumbling window with 30s allowed lateness and source punctuations every 10s, watermarks ensure almost-correct aggregations under typical network delay; but spikes beyond 30s will be emitted as late (side-output) or dropped depending on policy.Best practices: set allowed-lateness based on observed network skew, prefer source punctuations when possible, monitor late-event rates, and use side-outputs to reconcile late updates if strict correctness is required.
MediumSystem Design
30 practiced
Design a DLQ and retry strategy for a pipeline that writes to a flaky external HTTP API: include error classification, exponential backoff with jitter, circuit breaker thresholds, DLQ schema, replay tooling, and how to avoid dropping high-priority events during a downstream outage.
Sample Answer
Requirements & constraints:- Deliver events to flaky HTTP API with at-least-once semantics for high-priority items, bounded retries for low-priority, operational visibility, and safe replay.High-level design:- Ingest → Dispatcher → Retry Engine + Circuit Breaker → External HTTP API- Failed permanent/expired events → Dead-Letter Queue (DLQ) (persistent store like Kafka topic / S3 + metadata DB)- Priority tiering: High, Normal, Low (priority queueing)Error classification:- Transient (5xx, network timeouts, connection reset) → retryable- Throttling (429, 503 with Retry-After) → retryable with backoff honoring Retry-After- Client (4xx except 429) → non-retryable → DLQ immediately- Idempotency key present: safe to retry; otherwise add dedup metadataRetry strategy:- Exponential backoff with full jitter: - base = 500ms, max = 60s - attempt n wait = random_between(0, min(max, base * 2^(n-1)))- Max attempts: High=10, Normal=5, Low=3- Honor server Retry-After when provided (use max(Retry-After, backoff))Circuit breaker:- Per-endpoint CB with sliding window (60s, 100 requests)- Open if failure rate > 50% AND absolute failures > 20 in window- Open duration 30s, then half-open allowing 5 probe requests- Successful probes close CB; failed probes extend open with exponential backoff- When CB open: immediately queue retries into backlog and raise alertsDLQ schema (JSON):- id, priority, payload, timestamp, attempts, last_error_code, last_error_message, idempotency_key, original_topic, metadata, ttl, owner_team- Store in durable append-only store + index for queriesReplay tooling:- Web UI + CLI to filter (by id, time, error, priority), edit payload/headers (fix auth), re-enqueue to dispatcher with optional rate limit and start time- Supports bulk replay, dry-run mode, and automatic dedup checks using idempotency keys- Safe replay pipeline enforces priority and backpressureAvoid dropping high-priority events:- Separate high-priority queue/storage with reserved capacity/throttles- Token-bucket rate limiter ensures high-priority can burst through a capped fraction of throughput even during outage- Short-circuit lower priority retries when backlog exceeds threshold: either pause Low/Normal retries or escalate to manual review- Persist high-priority items across outages (replicated storage), and preferentially schedule their retries sooner and more attemptsObservability & ops:- Metrics: success/failure rates, queue depths by priority, retry counts, DLQ ingress rate, CB state- Alerts on rising DLQ rate, CB opens, backlog growth- Tracing logs include attempt number and backoff schedule- Run periodic DLQ audits and auto-replay rules for known transient causesTrade-offs:- More attempts & reserved capacity increase cost but reduce data loss for critical events- Immediate DLQ on 4xx prevents wasted retries; ensure good error parsing so real transient cases aren’t misclassified- Using per-endpoint CBs avoids global outage impact but adds complexity in state managementThis design balances reliability, operational safety, and cost while ensuring high-priority events are protected during downstream outages.
MediumSystem Design
35 practiced
Explain approaches to achieve a consistent snapshot when joining two streams in a distributed streaming engine. Discuss coordinating checkpoints, barrier propagation, handling skewed watermark progress, and how to resume joins after restoring from a snapshot.
Sample Answer
Requirements & constraints:- We need an exactly-consistent snapshot across two input streams so after a restore the join produces the same outputs (no duplicates/missing) and respects event-time semantics (watermarks).High-level approaches:1) Coordinated checkpoints (barrier-based/aligned):- Inject periodic checkpoints as barriers into both streams. Operators block processing of records after a barrier until barriers from all input channels arrive (barrier alignment), then take a local state snapshot and commit offsets. This yields globally-consistent snapshots and simplifies restore (restore operator state + committed source offsets).- Trade-off: alignment can stall processing when input rates/watermarks are skewed (backpressure).2) Unaligned or asynchronous checkpoints:- Snapshot in-flight buffers and positions without waiting for complete alignment (record sequence numbers / write-ahead log). Lower pause but more complex restore (must reprocess partial in-flight records deterministically) and higher metadata/IO.Coordination details:- Barrier propagation: barriers are streamed downstream with records; join operator must treat barriers per-input and only snapshot when barriers from all inputs for the same checkpoint id have arrived. On arrival, persist per-input offsets and join state atomically (use underlying checkpoint coordinator).Handling skewed watermark progress:- Idle-channel detection: mark slow/idle partitions as idle so watermarks propagate (avoids waiting for late but empty channels).- Watermark hold/buffering: hold events that might be needed until all relevant watermarks advance (store in keyed state + timers).- Backpressure mitigation: use bounded buffers + metrics; for heavy skewed partitions consider routing/partitioning changes or pre-aggregation.- For event-time joins, use watermark alignment strategies (hold until max(expected skew)) or use heuristics: if one stream’s watermark is far behind, you can delay emitting until it catches up or emit with a "late" policy.State snapshot & restore:- What to snapshot: keyed join state (per-key left/right buffers), timers (event-time/processing-time timers), and last persisted watermark/checkpoint id; source offsets (Kafka partition offsets) or source-specific positions.- On restore: restore operator keyed state and timers, reset source consumers to saved offsets, re-establish watermark to last known value (or compute from restored offsets). Resume processing from those offsets; because state contains buffered unmatched events and timers, the join semantics continue deterministically.- Exactly-once guarantees: combine coordinated checkpoints with transactional/atomic source sink commits (e.g., Kafka transactions or two-phase commit sink) so outputs aren't duplicated.Edge cases & trade-offs:- Late events after watermark: decide on side outputs or retractions.- Long-running skew: alignment stalls throughput — consider async checkpoints + durable in-flight logs or reduce checkpoint frequency.- Memory pressure: spill-to-disk or TTL for join buffers.Summary:Use barrier-coordinated checkpoints for simplicity and strong consistency; add idle detection and watermark-hold to handle skew; on restore, reset sources to checkpointed offsets and reload join buffers/timers so the join resumes deterministically. Consider unaligned checkpoints or source-level transactions if alignment cost is unacceptable.
Unlock Full Question Bank
Get access to hundreds of Data Reliability and Fault Tolerance interview questions and detailed answers.
