Batch and Stream Processing Questions

Covers design and implementation of data processing using batch, stream, or hybrid approaches. Candidates should be able to explain when to choose batch versus streaming based on latency, throughput, cost, data volume, and business requirements, and compare architectural patterns such as lambda and kappa. Core stream concepts include event time versus processing time, windowing strategies such as tumbling sliding and session windows, watermarks and late arrivals, event ordering and out of order data handling, stateful versus stateless processing, state management and checkpointing, and delivery semantics including exactly once and at least once. Also includes knowledge of streaming and batch engines and runtimes, connector patterns for sources and sinks, partitioning and scaling strategies, backpressure and flow control, idempotency and deduplication techniques, testing and replayability, monitoring and alerting, and integration with storage layers such as data lakes and data warehouses. Interview focus is on reasoning about correctness latency cost and operational complexity and on concrete architecture and tooling choices.

HardTechnical

79 practiced

Describe how to achieve 'effectively-once' semantics when writing from Flink to Kafka and to an RDBMS that does not support distributed transactions. Discuss use of write-ahead logs, outbox pattern, idempotent upserts, and coordination with Flink checkpoints to ensure consistency and recoverability.

Sample Answer

Goal: achieve "effectively-once" — each logical event is applied exactly once across Kafka and an RDBMS that lacks XA. Strategy combines Flink checkpoint coordination, transactional Kafka, an outbox / write-ahead log, and idempotent upserts in the DB.

1) Flink checkpoint-driven commits- Use Flink sinks that tie commits to checkpoints (TwoPhaseCommitSinkBase or transactional Kafka sink). Prepare work during pre-commit; only commit on checkpoint completion. On failure, Flink will abort in-flight transactions and re-run.

2) For Kafka- Use Kafka transactions (producer.initTransactions + begin/commitTransaction) integrated with Flink checkpointing so messages are visible only after a successful checkpoint — gives exactly-once to Kafka consumers.

3) For RDBMS without distributed transactions: Outbox / write-ahead log- Write a single atomic record to the DB outbox table inside the same DB transaction as your business update (this is local, so atomic).- Outbox row contains payload, dedup-key, and status.- A separate, idempotent dispatcher (or Flink job) reads committed outbox rows and publishes to Kafka (or downstream), marking rows as sent. The dispatcher must be idempotent and coordinate with Kafka transactions (produce and mark outbox-sent in the same DB transaction is not possible across systems). Instead: - Dispatcher reads outbox, writes to Kafka inside a Kafka transaction, and upon Kafka commit, updates outbox status. If dispatcher crashes, Kafka transaction aborts; records remain unsent and will be retried.

4) Idempotent upserts for sink-to-DB- When Flink itself needs to write to the RDBMS (not via outbox), use upserts keyed by a unique dedup-id (event id or sequence). Implement SQL like INSERT ... ON CONFLICT UPDATE or MERGE using the dedup key; ensure updates are deterministic and idempotent.- Combine with Flink two-phase commit pattern for JDBC: persist prepared transactions in a local WAL (or persistent state) tied to Flink checkpoint; commit only when checkpoint succeeds. If DB doesn’t support prepare/commit across restarts, use idempotent upserts + transactional local WAL to replay safely.

5) Write-Ahead Log (WAL) / Flink state- For sinks that can’t do 2PC, have Flink checkpoint store the pending operations (WAL) in durable state. On recovery, Flink replays pending operations; with idempotent upserts and dedup keys this yields effectively-once.

6) Coordination and recoverability- Ensure each action (DB change, Kafka publish) is correlated by a stable id.- Keep ordering guarantees: include monotonic sequence per key when necessary.- Monitor and alert on dispatcher stuck states (outbox rows not sent).

Trade-offs and practical notes:- Outbox shifts complexity to DB and dispatcher but avoids distributed transactions.- Kafka transactions + Flink checkpointing give strong guarantees to Kafka consumers.- Idempotent upserts are simplest but require schema support (unique constraints) and careful handling of non-idempotent side effects.- Test failure modes: dispatcher crashes between produce and outbox-mark, consumer duplicates, network partitions.

Example pattern summary:- Application writes business row + outbox row in one DB tx.- Dispatcher reads outbox, starts Kafka tx, writes messages, commits Kafka tx, then marks outbox row sent (retryable, idempotent).- Flink jobs that write directly use checkpoint-coordinated transactional sinks or WAL + idempotent upserts.

This combination yields effectively-once across Kafka and a non-XA RDBMS with recoverability and bounded complexity.

MediumTechnical

74 practiced

Sketch an architecture to provide end-to-end exactly-once processing for the pipeline: Kafka source -> Flink processing -> S3 Parquet sink. Account for Flink checkpointing, Kafka transactional consumption, and atomic file commits to S3. Explain assumptions, limitations, and how you would validate correctness.

Sample Answer

Requirements & assumptions:- Functional: end-to-end exactly-once from Kafka -> Flink -> Parquet files in S3.- Constraints: high throughput, failures/retries expected, S3 as object storage (eventual semantics on overwrite/deletes historically; assume modern S3 with read-after-write for new objects, but avoid relying on atomic renames).- Kafka supports transactional producers and committing offsets; Flink version supports checkpointing and TwoPhaseCommit sinks.

High-level architecture:1. Kafka topic with producers (non-transactional) and consumers.2. Flink job: - Use FlinkKafkaConsumer with enableCheckpointing and setCommitOffsetsOnCheckpoints(true) so offsets are included in Flink checkpoints. - Processing is deterministic/stateless or uses keyed state with RocksDB backend and durable checkpoints. - Sink: implement an exactly-once Two-Phase Commit sink (extend TwoPhaseCommitSinkFunction) that: - On pre-commit (onCheckpoint): flush partial Parquet files to temporary location (prefix /tmp/jobid/partition/txnid/), complete file writes locally or to S3 multipart upload but keep them under temp prefix. - On commit (when checkpoint is confirmed): atomically "make visible" by moving temp objects into final prefix (e.g., copy object to final path and CompleteMultipartUpload; or upload final object name during commit stage). The sink's transaction id is tied to Flink checkpoint id.3. Kafka offset management and transactions: - Flink writes offsets into checkpoint metadata; when a checkpoint completes, the job state includes offsets and the sink commits its transactions for that checkpoint. Optionally, leverage Kafka transactions if downstream writes back to Kafka; for S3 sink, Flink's TPC sink suffices.4. Atomic file commit strategy for S3: - Write files to temp prefix that are not read by consumers. - On commit, perform object copies to final prefix (or use CompleteMultipartUpload naming final key) and then delete temp objects. - Keep a commit manifest (JSON) per checkpoint listing final objects; publish manifest atomically (single object write). Downstream consumers/readers should only consider files referenced by manifests to avoid reading partial data.

Why this yields exactly-once:- Flink checkpoint is the coordination point: it snapshots Kafka offsets + sink transaction id + in-flight state.- Two-phase commit sink ensures files created for a checkpoint are only moved into final location when the checkpoint is successfully completed — preventing duplicates on retries.- When recovering from failure, Flink restores state and resumes from the saved Kafka offsets — avoiding reprocessing beyond the checkpoint.

Limitations and caveats:- S3 has no true atomic rename; copy+delete is used which can be eventually consistent in edge cases — using manifests minimizes risk.- Large object copies are expensive; prefer writing final object name via multipart upload to avoid copy.- Checkpoint frequency and Kafka transaction/producer timeouts must be tuned: long checkpoints could exceed Kafka.transaction.timeout.ms if Kafka transactions are used.- Backpressure and state size: ensure RocksDB and checkpoint storage (S3/HDFS) are tuned.

Validation and testing:- Deterministic integration tests: inject known messages, kill Flink at various points (during processing, during commit) and verify final S3 manifests contain exactly one representation of each input event (use event IDs and checksums).- Failure injection: simulate S3 copy failures, network partitions, Kafka broker restarts; verify recovery yields no duplicates and no data loss.- End-to-end property tests: tooling that produces N messages with unique IDs, then verify count of unique IDs in S3 equals N and checksums match; run under scale and while inducing faults.- Monitoring: expose metrics for checkpoints (latency, success/fail), sink commit failures, Kafka offsets, and manifest generation; alert on inconsistencies.

Operational notes:- Use durable checkpoint storage (S3/HDFS) for Flink checkpoints and savepoints.- Keep temp prefixes per-job & checkpoint and garbage-collect abandoned temp objects older than retention threshold.- Document assumptions about S3 semantics and enforce readers to consult manifests for final file lists.

HardSystem Design

82 practiced

Design an append-only, replayable event store for compliance audits that must retain immutable events for 7 years, support efficient replay by time range and partition, and allow redaction when legally required. Discuss storage formats, partitioning/indexing strategies, immutability guarantees, encryption-at-rest, and replay performance.

Sample Answer

Requirements (clarify): append-only, immutable retention 7y, efficient replay by time-range + partition, legal redaction, encryption-at-rest, auditability.

High-level design:- Ingest events into a write-ahead log (Kafka/Kinesis) for immediate durability and ordering.- Persist events long-term in immutable object storage (S3/GCS/Azure Blob) using a columnar/serialized file format (Avro/Parquet/ORC) wrapped by a table/manifest layer (Apache Iceberg/Delta Lake) to support time-range partitioning and fast manifests for replay.

Storage format & reasoning:- Avro/Parquet files per time/partition slice. Avro for schemaless/event streams (compact, row-oriented) or Parquet for analytical replay (columnar, faster scans). Use Avro for raw event files and periodically compact to Parquet for replay performance.- Maintain manifest metadata (Iceberg table metadata) referencing files, min/max timestamps, partition keys, record counts and file-level column statistics to avoid full scans.

Partitioning & indexing:- Primary partition = date/hour (year/month/day/hour) + logical partition key (tenant/customer, service). This yields efficient time-range + partition pruning.- Secondary index: lightweight key-value index (DynamoDB/Bigtable/Cassandra) mapping (partition, time-range) → file list or file pointers for sub-second lookups. Maintain file-level bloom filters/min-max to skip files.- Store a timeline metadata store (consistent, versioned) in a transactional DB (DynamoDB / RDS) or use Iceberg’s metadata tree for atomic snapshotting.

Immutability guarantees:- Use object-store features: S3 Object Lock in Governance/Compliance mode (WORM) for 7y retention; enable bucket-level immutability with lifecycle rules. Complement with write-once patterns: never mutate files, only add new manifests.- Cryptographic integrity: sign manifests/file manifests with HMAC or public-key signatures; maintain append-only audit ledger (append-only DB or blockchain-like ledger) recording writes, snapshots, actor, timestamp.- Access controls: restrict deletion rights; logs in CloudTrail/Audit.

Encryption-at-rest:- Envelope encryption: data encrypted with per-file data keys; keys encrypted with KMS CMK. Use customer-managed keys (CMKs) and strict audit of key rotation/use.- For tenant isolation, use per-tenant data keys (enables targeted key deletion if required).

Redaction (legal requirement):- Two modes: 1) Logical redaction: write a redaction tombstone event referencing event IDs/file ranges and mark them in manifest as redacted — replay layer filters them out. Maintain original data for audit, but mark as redacted in queries; keep an immutable audit trail of the redaction action (who, why, court order). 2) Cryptographic redaction ("shredding"): encrypt sensitive fields with a field-level key; on lawful redaction, perform key destruction/eviction for that tenant/field — rendered data unrecoverable. Combine with per-file field-level encryption or envelope per-tenant keys so only targeted records are destroyed.- Note trade-off: cryptographic shredding is irreversible and must be designed carefully to meet legal proofs; keep signed metadata proving redaction.

Replay performance:- Use manifest + partition pruning + file-level statistics to produce a minimal file list for the requested time-range and partition.- Parallel read/scan files using Spark or Presto; leverage Parquet column pruning/predicate pushdown.- Maintain periodic compacted snapshot files (larger Parquet files per partition) to reduce small-file overhead and speed sequential scans.- Cache hot partitions (recent days) in SSD-backed storage or a warmed Druid/ClickHouse for high-concurrency queries.- For low-latency replay of recent events, read from Kafka plus include compaction/upsert streams to reconstruct state.

Operational lifecycle & retention:- Enforce lifecycle: raw Avro → compact to Parquet → archive to Glacier/Cold Storage after N months; still under Object Lock during retention period.- Regular integrity checks: verify file signatures, manifest consistency, periodic checksum audits.

Trade-offs:- Object Lock + WORM gives strong immutability but increases cost and complexity in handling legal redaction (cryptographic shredding preferred for selective erasure).- Per-tenant keys enable targeted redaction but add key management overhead.- Secondary index increases write complexity and cost but greatly speeds targeted replay.

Monitoring & audit:- Audit trail of all operations (write, manifest changes, redactions) in an append-only ledger; alerting on manifest anomalies; regular compliance reports.

This design balances immutable storage, efficient time/partition replay, encryption, and legally-required redaction via tombstones plus cryptographic key eviction, using tried-and-tested building blocks (Kafka, S3 with Object Lock, Iceberg/Delta, KMS, DynamoDB).

MediumTechnical

84 practiced

Using a streaming SQL engine (Flink SQL or ksqlDB), write a query to compute user sessions given a streaming events table with schema: events(user_id STRING, event_time TIMESTAMP, event_type STRING). Define sessions by inactivity gaps of 30 minutes; output session_id, user_id, session_start, session_end, and event_count. Include watermark usage in your query.

Sample Answer

Approach: use event-time processing with watermarks and session windows (30-minute gap). Produce aggregated session rows per user with session start/end and event count. Below are Flink SQL and ksqlDB examples and brief notes.

Flink SQL (uses SESSION window):

sql

-- create source table with watermark on event_time
CREATE TABLE events (
  user_id STRING,
  event_time TIMESTAMP(3),
  event_type STRING,
  WATERMARK FOR event_time AS event_time - INTERVAL '1' MINUTE
) WITH (...);

SELECT
  CONCAT(user_id, '-', CAST(window_start AS STRING)) AS session_id,
  user_id,
  window_start AS session_start,
  window_end AS session_end,
  COUNT(*) AS event_count
FROM TABLE(
  SESSION(
    TABLE events,
    DESCRIPTOR(event_time),
    INTERVAL '30' MINUTE
  )
)
GROUP BY window_start, window_end, user_id;

ksqlDB (uses SESSION windows):

sql

CREATE STREAM events_stream (
  user_id VARCHAR,
  event_time VARCHAR,
  event_type VARCHAR
) WITH (KAFKA_TOPIC='events', VALUE_FORMAT='JSON', TIMESTAMP='event_time');

CREATE TABLE sessions AS
  SELECT
    user_id,
    WINDOWSTART AS session_start,
    WINDOWEND AS session_end,
    COUNT(*) AS event_count
  FROM events_stream
  WINDOW SESSION (30 MINUTES)
  GROUP BY user_id
  EMIT CHANGES;

Notes:- Watermarks handle late events; adjust allowed lateness and watermark offset to your ingestion skew.- session_id built from user+window_start for determinism.- Edge cases: out-of-order events near watermark, very long idle users, duplicate events — handle via deduplication and appropriate watermark/backpressure settings.

EasyTechnical

119 practiced

Describe tumbling, sliding, and session windows in stream processing. For each window type give one concrete analytics example (e.g., per-minute counts, moving averages, user sessions), and explain how window boundaries and late arrivals are handled differently.

Unlock Full Question Bank

Get access to hundreds of Batch and Stream Processing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.