Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.
EasyTechnical
36 practiced
Describe how to choose an effective partition key for a Kafka topic that will carry user activity events for online feature computation. Consider high-cardinality users, multi-tenant workloads, and hotspot avoidance. Provide concrete heuristics and an example key design for a system with 10M users where per-user ordering is required but throughput must be evenly distributed.
Sample Answer
Start from the requirement: per-user ordering means all events for a given user must land in the same partition. At the same time you must avoid hotspots and balance throughput across partitions for 10M users and multi-tenant traffic.Heuristics- Preserve ordering: the partitioning decision must be deterministic per user (user-level affinity).- Hash-based mapping: use a stable hash of (tenant_id, user_id) to map users to partitions — this gives even distribution for large cardinalities.- Partition count: choose many more partitions than brokers (e.g., 10–50× brokers) so Kafka can balance load. For 20 brokers, consider 200–1000 partitions.- Handle skew: detect heavy hitters and route them to dedicated partitions or apply rate-limiting/aggregation upstream.- Multi-tenant fairness: include tenant_id in the hash to avoid one tenant’s high-activity users dominating partitions reserved for another tenant, or implement per-tenant partition quotas.- Rebalancing: use consistent hashing or a mapping service to remap users without causing widespread reshuffling.Concrete key design (10M users, per-user ordering required)- Message key: "tenantId:userId" (string or binary)- Partitioner: partition = murmur3_hash("tenantId:userId") mod P, where P = 500 (example)- Rationale: murmur3 gives uniform distribution; including tenantId preserves multi-tenant isolation; P=500 spreads 10M users (~20k users/partition) to avoid hotspots while keeping per-user ordering.- Hotspot handling: during monitoring, if partition X is hot (due to few heavy users), move those users to dedicated partitions by updating a small routing table: if user in heavy-hitter list → partition = dedicated_partition_map[user]; otherwise use hash.Edge cases- Very heavy single-user workloads: route those users to separate topics/partitions or aggregate events before Kafka.- Changing P: increasing partitions requires re-keying mapping for new partitions; use consistent hashing or a mapping service to minimize movement.- Small tenants: monitor fairness and consider per-tenant caps if needed.This approach balances per-user ordering and even throughput while giving operational levers for hotspots and multi-tenant fairness.
MediumTechnical
41 practiced
Compare Apache Flink, Kafka Streams, Spark Structured Streaming, and Amazon Kinesis Data Analytics for implementing stateful stream processing for low-latency enrichments and windowed joins. Discuss programming model, state/backends, exactly-once support, scaling characteristics, operational complexity, and suitable use cases from an AI-engineer perspective.
Sample Answer
Situation: As an AI engineer choosing a stream-processing runtime for low-latency enrichments and windowed joins (e.g., augmenting inference results with user/profile embeddings, real-time feature joins), I compare four options across the requested dimensions.Programming model:- Apache Flink: Dataflow API (DataStream/Table). Rich event-time semantics, CEP, async IO. Java/Scala/Python.- Kafka Streams: Library embedded in apps. DSL and Processor API; lightweight, JVM-centric.- Spark Structured Streaming: Micro-batch (continuous mode optional). High-level SQL/DataFrame API; Python/Scala/Java.- Kinesis Data Analytics (KDA): Managed Flink or SQL runtime on AWS; similar to Flink if using Flink-based KDA.State/backends:- Flink: Pluggable state backends (RocksDB, in-memory) with efficient incremental checkpointing; scales to very large state.- Kafka Streams: RocksDB local state store with changelog topics for durability.- Spark: State store persisted to files (WAL + checkpointing); less optimized for very large keyed state.- KDA: Uses managed Flink state backend (RocksDB + S3 checkpoints) when Flink runtime selected.Exactly-once support:- Flink: Strong exactly-once with two-phase checkpoints and connectors supporting transactions.- Kafka Streams: Exactly-once semantics (EOS) via Kafka transactions (e.g., EOS v2); good for end-to-end with Kafka.- Spark: Exactly-once semantics for idempotent sinks or with sources/sinks that support atomic writes; micro-batch limits true low-latency EOS in some cases.- KDA: Matches Flink behavior when using Flink runtime; otherwise depends on SQL runtime features.Scaling characteristics:- Flink: Fine-grained task parallelism; scale-out with state migration; good for high throughput and low latency.- Kafka Streams: Scale by adding app instances; partition-driven scaling (must align tasks to topic partitions).- Spark: Scale via executors; micro-batch introduces latency and backpressure trade-offs; scaling stateful operators is heavier.- KDA: Autoscaling options on AWS but bound by managed limits; simpler scaling but less operational control.Operational complexity:- Flink: More operational overhead (deploying clusters, checkpoint storage, tuning RocksDB), but mature observability.- Kafka Streams: Easiest operationally if Kafka already exists—deploy apps as containers; simpler lifecycle.- Spark: Familiar if already using Spark; heavier resource footprint and complexity for low-latency tuning.- KDA: Lowest ops overhead (managed service) but less flexibility and potential vendor lock-in.Suitable AI-engineer use cases:- Flink: Real-time feature stores, streaming feature enrichment for online model serving, complex windowed joins, low-latency online learning pipelines with large keyed state.- Kafka Streams: Lightweight enrichment close to Kafka topics (e.g., feature cache lookups), lower operational burden when state size moderate and co-located with Kafka.- Spark Structured Streaming: Batch-like streaming for model training pipelines, aggregations over windows where ~1s+ latency is acceptable, or when reusing Spark ML pipelines.- Kinesis Data Analytics: Fastest path on AWS for prototypes or managed production if you want Flink features without managing infra.Recommendation summary:- For strict low-latency, large-state, complex event-time joins: Flink (or KDA Flink on AWS) is ideal.- For Kafka-first, moderate state and simpler ops: Kafka Streams.- For ML-centric ETL and near-real-time analytics with heavy reuse of Spark ecosystem: Spark Structured Streaming.Consider connector support (model stores, feature store, Redis, Kafka, S3), checkpointing/restore behavior, and GPU/serving integration when integrating model inference pipelines.
HardTechnical
42 practiced
Choose the most appropriate checkpoint storage (local disk, HDFS, S3/object storage) for a Flink cluster with 10TB of state. Discuss trade-offs in recovery time, durability guarantees, cost, and operational complexity, and recommend a configuration for a production environment with RPO of 15 minutes.
Sample Answer
Clarify requirements: 10 TB of state, production, RPO = 15 minutes (meaning checkpoints must be frequent enough and durable so we can restore to a point ≤15 minutes old). We assume RocksDB state backend (typical for large keyed state) and Flink incremental checkpoints are available.Comparison (recovery time | durability | cost | ops complexity)- Local disk (TaskManager local): - Recovery time: fastest if TM disks survive and accessible (local restores avoid re-downloading), but in practice failure of TM or node means data lost — full cluster restarts must fetch checkpoints from remote anyway. - Durability: weak — single-node/local disks are ephemeral; no replication across nodes. - Cost: low (no egress), uses node disks. - Complexity: low to operate but unsafe for production RPO.- HDFS (on-prem or HDFS cluster): - Recovery time: moderate — high throughput, HDFS can serve large checkpoint snapshots quickly; if co-located with Flink network, reads are fast. - Durability: strong — replication factor provides durability and consistency. - Cost: medium (storage hardware/ops). - Complexity: higher ops (manage HDFS HA, Namenode, scaling).- S3 / Object storage: - Recovery time: slower than local/HDFS for cold restore because many small objects need to be fetched; incremental checkpoints reduce data transfer. With parallel downloads and pre-warming, acceptable. - Durability: excellent (managed 11/13 9s), geo-redundant; strong for production. - Cost: low to medium (pay per GB/month + GET/PUT requests); egress costs if cross-region. - Complexity: low (managed service) but requires tuning for performance and consistent semantics (S3 eventual/strong consistency caveats).Recommendation for production (RPO 15 minutes):- Use object storage (S3-compatible) as the canonical checkpoint store and RocksDB incremental checkpoints as primary snapshot mechanism. Reason: durability and operational safety are paramount for production; S3 provides strong persistence and cheap long-term storage. Incremental checkpoints minimize data transferred/stored for 10 TB state.- Complement with HDFS if you run entirely on-prem and have an HDFS cluster with high throughput — then HDFS may give faster recovery and lower network egress costs. Avoid relying solely on local disk.Concrete configuration / best practices:- state.backend: RocksDBStateBackend (or unified StateBackend with RocksDB)- state.checkpoints.dir: s3a://my-bucket/flink-checkpoints/- state.savepoints.dir: s3a://my-bucket/flink-savepoints/- Enable incremental checkpoints: state.backend.rocksdb.incremental: true- Checkpoint interval: 5 minutes (300000 ms) — this gives two checkpoints within RPO window, balancing overhead and recovery granularity. Ensure checkpointing.timeout > interval * 2.- externalized.checkpoints: RETAIN_ON_CANCELLATION- Checkpoint concurrency/parallelism: tune checkpointing.max-concurrent-checkpoints (1–2) and min-pause-between-checkpoints to avoid overload.- S3 tuning: enable multipart/fast upload (fs.s3a.multipart.size ~ 64MB), fs.s3a.fast.upload=true, increase max connections and thread pool for parallel uploads/downloads.- Compression and incremental compaction: enable RocksDB compression and compact on checkpoint to reduce upload size.- Network/IO: ensure high network bandwidth between TMs and S3; use instance types with high I/O and burst network.- Test restore time: measure full restore time and ensure restore procedure meets RPO (consider warm standby cluster or continuous replication).- Backup strategy: periodic savepoints (e.g., daily) retained longer-term.- Monitoring & alerts: checkpoint duration, size, failure rate; automated rollback and health checks.- Disaster plan: region replication or cross-region bucket replication if needed for regional failures.Trade-offs summary:- S3 gives strongest durability with manageable cost and lower ops overhead; recovery time can be mitigated with incremental checkpoints and parallel downloads, so it is the recommended choice for production with 10 TB and 15-minute RPO unless strict very-fast recovery (minutes) is required and you control an HDFS cluster with better throughput—in that case HDFS is a viable alternative. Avoid local-only storage.
MediumTechnical
43 practiced
You need to build a materialized view that counts active users per minute using Kafka Streams KTable or Flink state. Describe how you would implement the stream->table pattern, keep the materialized view up-to-date, handle late events, and expose the view for low-latency queries.
Sample Answer
Requirements & approach:- Objective: maintain a materialized view that counts active users per minute (e.g., number of distinct users who performed activity in each 1-min bucket), kept up-to-date in streaming, tolerate late events, and serve low-latency queries.Stream→Table pattern (both options):- In Kafka Streams: read user-event topic, map events to minute-buckets (timestamp -> minute key), use groupByKey().aggregate() into a KTable backed by a state store (RocksDB). Use a tumbling event-time window of 1 minute with a unique user-set/count logic (see late-handling).- In Flink: assign event-time timestamps and watermarks, keyBy(minuteBucket), use a keyed state (MapState or HyperLogLog for dedupe/approx distinct) and a ProcessFunction or WindowFunction producing per-minute counts into managed state.Keeping view up-to-date:- Maintain per-minute structures: - Exact: maintain a SetState of userIds per minute (memory heavy) and count = set.size(). - Approx: use HyperLogLog in state for memory-efficient distinct counts.- On each event: compute minuteBucket, check/update user membership; if new, increment count and update state store. Emit the updated minute->count into the materialized KTable (Kafka Streams) or side-output / sink topic (Flink).Handling late events:- Use event-time semantics with watermarks. Configure allowed lateness (e.g., 2–5 minutes).- For Kafka Streams, use windowed aggregations with grace period to accept late records; for Flink, use window allowed lateness and onTimer to emit updates.- If event arrives after window is closed, optionally route to a re-processing/backfill job that updates state (and emits correction events). Ensure idempotency by storing per-user seen markers.Exposing for low-latency queries:- Use the local state store: Kafka Streams Interactive Queries expose RocksDB KTable for direct reads (low-latency, co-located). For Flink, use Queryable State (or expose via gRPC/REST from an FM-managed read service that periodically snapshots state or subscribes to the sink topic).- Also write the materialized view to a fast read-store: Redis, Aerospike or a caching tier keyed by minuteWindow for O(1) reads. Keep the sink as the single source of truth (update-stream of corrections).- Add TTL for old windows to bound state size; compact Kafka sink topic.Operational concerns:- Idempotency and deduplication: dedupe by event-id if available; persist per-user markers.- Scaling: partition by minuteBucket (and user hash) to balance load; ensure stateful operator parallelism aligns with Kafka partitions.- Monitoring: latency of updates, watermark lag, late-event rates, state sizes, and error correction metrics.This design provides exact or approximate active-user counts per minute, supports late data with bounded corrections, and offers low-latency reads via local state or a fast key-value store.
HardSystem Design
44 practiced
You're asked to design an end-to-end pipeline (Kafka -> Flink -> external DB) that must provide exactly-once updates to the external DB. Provide an architecture and pseudocode or high-level steps showing how you would use checkpoints, Kafka transactions, and a two-phase-commit sink (or alternative) to achieve end-to-end exactly-once. Discuss practical limitations and failure scenarios.
Sample Answer
Requirements & constraints:- Ingest high-throughput event stream (Kafka).- Stream processing with Flink, maintain state, and produce exactly-once updates to external DB (e.g., Postgres, Cassandra).- Tolerate node/process failures, provide end-to-end exactly-once semantics.High-level architecture:Kafka (producers) -> Kafka topics -> Flink Job (source with KafkaConsumer, stateful operators, checkpointing enabled) -> Two-phase-commit Sink (Flink TwoPhaseCommitSinkFunction) -> External DB (supports transactional commits) Use Kafka transactions for source/producer-side atomic writes; use Flink checkpoints coordinated by JobManager to drive sink 2PC commits.Key design elements / steps:1. Enable Flink checkpoints (periodic, durable, externalized) and set checkpointing mode to EXACTLY_ONCE.2. Use KafkaSource configured with Flink’s Kafka connector that participates in checkpoint barriers.3. Implement a TwoPhaseCommitSinkFunction in Flink: - beginTransaction(): start DB transaction (or create a local staging transaction id) - invoke(transaction, record): write record into transaction (use idempotent upserts or write to staging table) - preCommit(transaction): flush buffers, ensure writes are durable but not visible (e.g., write into a transactional table or local file) - commit(transaction): called only when checkpoint completes — commit DB transaction atomically - abort(transaction): rollback on failure4. Tie sink commits to checkpoint completion: Flink calls commit only after checkpoint success, ensuring atomic visibility.5. Optionally use an external coordinator / ledger table to track in-flight transactions for recovery.6. For external DBs lacking strong transactions, use an idempotent upsert pattern with deterministic keys and a monotonic sequence/provenance (e.g., event_id + version) and store last-applied per-partition in Flink state.Pseudocode (high-level):class TwoPhaseDBSink(TwoPhaseCommitSinkFunction): def beginTransaction(): return DB.begin_transaction() def invoke(tx, record): tx.upsert(record.key, record.value, record.event_id) def preCommit(tx): tx.flush() def commit(tx): tx.commit() def abort(tx): tx.rollback()Flink job:env.enableCheckpointing(30_000)source = FlinkKafkaSource(...)stream.keyBy(keyFn).process(statefulOps).sink(TwoPhaseDBSink())Practical limitations & failure scenarios:- External DB must support transactions with durability; otherwise emulate via idempotent upserts and deduplication. Without DB transactions, true end-to-end exactly-once is not achievable—only effectively-once.- Long-running transactions: if checkpoint interval << transaction duration, many open txns consume resources and risk locks/timeouts. Tune checkpointing and batch sizes.- Coordinator failures: if JobManager or sink process dies during commit phase, recovery must know which transactions were committed. Use external ledger or DB transactional metadata to reconcile.- Network partitions: commit may succeed but acknowledgement lost -> risk of duplicate commit attempts; make commit idempotent or record global transaction id visible in DB.- Kafka-side: producers should use Kafka transactions when writing to output topics; ensure transaction timeouts and transactional.id uniqueness across restarts.- Performance trade-offs: exactly-once adds latency (flush/commit on checkpoint). For high throughput, consider micro-batching and larger checkpoints.Takeaways:- Use Flink checkpoint-coordinated 2PC sink when external DB supports transactions; otherwise use idempotent upserts with durable deduplication state.- Carefully tune checkpoint interval, transaction lifetime, and state backend persistence; implement reconciliation/compaction for orphaned transactions.
Unlock Full Question Bank
Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.
