Batch and Stream Processing Questions

Covers design and implementation of data processing using batch, stream, or hybrid approaches. Candidates should be able to explain when to choose batch versus streaming based on latency, throughput, cost, data volume, and business requirements, and compare architectural patterns such as lambda and kappa. Core stream concepts include event time versus processing time, windowing strategies such as tumbling sliding and session windows, watermarks and late arrivals, event ordering and out of order data handling, stateful versus stateless processing, state management and checkpointing, and delivery semantics including exactly once and at least once. Also includes knowledge of streaming and batch engines and runtimes, connector patterns for sources and sinks, partitioning and scaling strategies, backpressure and flow control, idempotency and deduplication techniques, testing and replayability, monitoring and alerting, and integration with storage layers such as data lakes and data warehouses. Interview focus is on reasoning about correctness latency cost and operational complexity and on concrete architecture and tooling choices.

MediumSystem Design

74 practiced

You operate a streaming feature store that must meet a 500ms freshness SLA for online predictions. Describe an end-to-end architecture including ingestion, materialized feature computation (stateless and stateful), storage choices for low-latency reads, caching strategies, and how you'd ensure correctness and operational scalability under spikes.

Sample Answer

Requirements & constraints:- 500ms end-to-end freshness for online predictions- Support stateless (e.g., last_event) and stateful (e.g., rolling counts) feature computation- Correctness (no lost/duplicate updates, late-arriving events handled)- Operational scalability for traffic spikes

High-level architecture:1. Ingestion- Events -> Kafka (partitioned by entity_id). Producers include schema registry for compatibility and include event_ts.- Use small max batch sizes and linger.ms tuned to keep ingestion latency <50ms.

2. Stream processing / materialization- Use a stream processing engine with exactly-once semantics (Apache Flink or Kafka Streams).- Stateless features: transform + map operations computed per event and emitted immediately.- Stateful features: keyed state with RocksDB state backend inside Flink for large state per key (windowed counters, aggregations). Configure RocksDB to flush/snapshot frequently to meet recovery and correctness SLAs.- Periodically (or on change) emit materialized updates to: a) Local state store (RocksDB embedded) for ultra-low-latency co-located reads from the serving layer (see below). b) External low-latency read store (Redis Cluster or DynamoDB/Scylla) for multi-service access and persistence.

3. Online serving / low-latency reads- Primary: Co-located queryable state (Flink Queryable State or stateful microservices that read local RocksDB). This gives sub-ms local lookup when model serving colocated with processing node.- Secondary: Distributed cache / DB: - Redis Cluster (hot keys), with LRU eviction and TTL per feature; tuned client timeouts <50ms. - DynamoDB or Scylla for cold reads and larger capacity; use DAX/edge caching for sub-10ms reads.

4. Caching strategies- Client-side in-memory LRU cache in model-serving instances (ttl ~100–500ms) to absorb read spikes.- Read-through Redis cache: model server checks local cache -> Redis -> storage; writes/updates are pushed from stream processors to Redis.- Hot-key replication: maintain more replicas for heavy-hitter keys, or use consistent hashing with virtual nodes.

5. Correctness & data quality- Exactly-once processing and idempotent writes to external stores (use upserts with event_ts/version).- Watermarks and late-arrival logic: define acceptable lateness window; maintain separate “stale” flag if event older than threshold.- Schema registry and feature versioning so models reference stable feature IDs/versions.- End-to-end checks: consumer offsets, high watermark monitoring, per-key monotonic counters, periodic reconciliations (compare aggregates between batch and streaming).

6. SLA and scalability under spikes- Partitioning: increase Kafka partitions and Flink parallelism; key-by entity_id ensures state sharding.- Autoscaling: use metrics (lag, CPU, request latency) to scale stream processors and model-serving pods. Pre-warm extra capacity when traffic forecasted.- Backpressure controls: bound in-memory buffers, apply shedding policy for non-critical features; maintain graceful degradation (serve fewer features rather than fail).- Circuit breakers & timeouts in model-serving; fallback to last-known features when store not reachable.- RocksDB tuning: compaction settings, memory budgets per taskmanager; use SSDs and provisioned IOPS.- Load-test and chaos testing to validate 500ms SLA: synthetic spikes, failovers, network partitions.

Example flow (latency budget):- Producer -> Kafka publish: 20–50ms- Kafka -> Flink processing (stateless/stateful compute): 20–100ms- Flink write to Redis + embedded RocksDB: 5–20ms- Model server read from local cache/Redis: 1–20msTotal typical: ~50–200ms with headroom; under spikes ensure autoscale and caching keep under 500ms.

Operational monitoring & runbook:- Metrics: event-to-materialization latency, Kafka lag, processing time, read latency per store, cache hit ratio, error rates.- Alerts for lag >100ms, cache hit ratio drop, compaction stalls.- Runbooks: failover to cold store, flush caches, scale cluster, roll back processing jobs.

Trade-offs:- Co-located RocksDB offers lowest latency but ties compute and serving; external Redis provides shared access and persistence but adds network hop.- Exactly-once increases complexity but ensures correctness; at very high throughput you may accept idempotent-at-least-once with strong versioning.

This design balances low-latency reads (co-located state + cache), correctness (exactly-once, versioning, watermarks), and scalability (partitioning, autoscaling, backpressure) to reliably meet a 500ms freshness SLA.

HardTechnical

83 practiced

You need to deliver 1M inference requests/day at minimum cost with 99.9% uptime. Compare serverless (e.g., Lambda), container-based provisioned clusters, and batched GPU inference options. Propose a cost-optimized architecture that meets latency SLOs, including autoscaling rules, cold-start mitigation, and model compression/quantization options.

Sample Answer

Requirements & constraints:- Throughput: 1,000,000 inference requests/day ≈ 11.6 rps average, peak factor 10 → ~116 rps peak.- SLO: 99.9% uptime, latency SLO (assume p95 ≤ 200ms, p99 ≤ 500ms).- Cost minimized.

Option comparison (concise):- Serverless (Lambda/FaaS): Excellent for low management and bursty workloads; pays per-invocation. Downsides: cold starts, limited memory/GPU access, higher per-request cost at high sustained throughput; hard to do large-batch GPU work. Good if model small and stateless.- Container-based provisioned clusters (K8s with KNative/Autoscaler or ECS): Lower per-request cost for sustained traffic, can reserve warm capacity, supports GPUs, custom runtimes. More ops overhead.- Batched GPU inference (Triton/TF-Serving on GPU instances): Best latency/throughput per dollar for heavy neural nets; enables large batching to amortize GPU cost. Requires careful batching latency controls and queueing.

Recommended cost-optimized architecture:- Hybrid: provisioned container cluster for core baseline capacity + autoscaling to handle peaks; attach GPU-backed node pool for batched heavy models; use serverless for spiky low-CPU models or auth/edge routing. Components: - API gateway / edge LB with TLS + rate limiting. - Front-tier stateless containers (cheap CPU) for lightweight pre/post-processing, routing, can be serverless if tiny. - Inference tier: - CPU instances (pods) running optimized model for low-latency single requests (quantized CPU model, e.g., int8 via ONNX Runtime / Torch-TensorRT) sized to handle p50/p95 for small models. - GPU pool (Triton + batching) for high-throughput heavy models; set max batch size to meet p99 latency target via dynamic batching (max wait time). - Async queue (Kafka/Rabbit/SQS) for non-critical or batchable requests; worker pool consumes and uses large batches on GPU. - Metrics/observability + circuit breakers.

Autoscaling rules:- HPA on pods based on custom metric: requests/sec per replica + latency SLA. For CPU pods use target utilization (e.g., 60% CPU) and SLO-aware scaling: if p95 latency > threshold, scale up proactively.- Cluster Autoscaler for node scaling; GPU node pool scales based on pending GPU pod count and queue length.- For GPU batching: scale based on queue size and effective GPU utilization.

Cold-start mitigation:- Keep minimum replica counts (baseline capacity) equal to estimated steady-state traffic: min_replicas = ceil(peak_baseline_rps / per_pod_rps). For 116 rps peak and per-pod 20 rps → min 6 pods.- Warm-up lifecycle: periodic health probing to prevent idle termination; use lifecycle hooks to preload model into memory.- Use provisioned concurrency for serverless components where used.

Model compression & quantization options:- Distillation + pruning to reduce model size.- Quantize to int8 (static or dynamic) using ONNX Runtime or TensorRT for GPUs: preserves accuracy with large latency/cost wins.- Use model sparsity and compressed checkpoints; memory-mapped weights for faster load.- Mixed precision (FP16) on GPUs.

Latency SLO tactics:- Single-request path hits CPU-optimized quantized model for low-latency cases.- If model too heavy, route to GPU but enforce small dynamic batch (max wait 10ms) for online; otherwise use async queue for non-interactive.- Use adaptive batching: increase batch size when GPU utilization low, reduce when latency climbs.

Cost heuristics (example):- If sustained traffic, container+GPU batching often < serverless per-inference cost once >100k/day. Use serverless only for low-volume or unpredictable functions.

Trade-offs:- More ops complexity vs. lower cost.- Batching increases throughput but can add bounded latency; use hybrid routing to satisfy SLO.

Monitoring & ops:- Track per-model p50/p95/p99 latencies, queue depth, GPU utilization, cold-start rate.- Auto rollback triggers when accuracy/regression detected.

This hybrid design balances cost and latency: keep warm CPU-quantized replicas for immediate low-latency traffic and use GPU batched pools for throughput-efficient heavy inferences, with autoscaling and cold-start policies tuned to meet 99.9% uptime and SLOs.

MediumTechnical

84 practiced

Explain checkpoints and savepoints in streaming frameworks (e.g., Apache Flink). How do they enable fault tolerance and replayability for model retraining? Discuss trade-offs between checkpoint frequency, state size, storage backend, and recovery time.

Sample Answer

Checkpoints vs Savepoints (short definitions)- Checkpoint: automatic, periodic snapshot of operator state used by the streaming runtime (e.g., Flink) for exactly-once fault recovery. Managed by the framework, coordinated across parallel tasks.- Savepoint: user-triggered, durable snapshot intended for operational tasks (upgrades, migrations, manual rollback, or reproducible reprocessing). Not automatically garbage-collected and typically stored long-term.

How they enable fault tolerance and replayability for model retraining- Fault tolerance: Checkpoints let the job roll back to a consistent global state after a failure; operators resume processing from the checkpointed offsets so state and side-effects remain consistent (exactly-once semantics when integrated with sinks).- Replayability: Savepoints (or retained checkpoints) + deterministic source offsets allow you to restart a job from a specific point in time, replay input events and reproduce feature computation or training data pipelines for model retraining and debugging. Savepoints are ideal when you need stable, long-lived snapshots tied to model versions.

Trade-offs- Checkpoint frequency: Higher frequency reduces potential data reprocessing and recovery window but increases I/O and CPU overhead; more frequent small checkpoints suit low-latency pipelines, but may add backpressure.- State size: Large state (e.g., feature stores, model parameters, vocabularies) increases checkpoint duration and storage. Use state sharding, incremental checkpoints (Flink supports incremental RocksDB checkpoints) and TTL to reduce impact.- Storage backend: Local disk is fast but unsafe for node failures; durable object stores (S3, GCS) provide safety and shareability but add latency and egress cost. Choose backends that support multipart/streaming uploads and consider data locality for recovery speed.- Recovery time: Influenced by checkpoint size, backend throughput, and parallelism. Incremental checkpoints and parallel restore reduce recovery time. Trade-offs: faster recovery often means more storage and I/O during steady state.

Practical recommendations for ML engineers- Use frequent incremental checkpoints for production fault tolerance; take savepoints before model deploys or schema changes.- Keep savepoints associated with model artifacts and training metadata (input offsets, schema) so retraining is reproducible.- Monitor checkpoint duration and backpressure; tune checkpoint interval and RocksDB parameters.- Test restore and replay workflows regularly to validate reproducibility and estimate recovery windows.

MediumSystem Design

65 practiced

Design an Apache Flink job that computes per-user aggregates (e.g., clicks per 5-minute tumbling window) when events can be up to 2 minutes late. Describe partitioning strategy, watermark generation, allowed-lateness, state backend choice, checkpointing cadence, and how you'd scale the job to handle increasing traffic.

Sample Answer

Requirements:- Compute per-user counts (clicks) in 5-minute tumbling event-time windows.- Events may arrive up to 2 minutes late; correctness and fault-tolerance required.- Low latency for downstream ML features (~seconds) and ability to scale to high throughput.

High-level architecture:Event source (Kafka) → Flink stream (EventTime) → assignTimestamps & watermarks → keyBy(userId) → tumblingEventTimeWindow(5m) with allowedLateness(2m) → aggregate → sink (feature store/Redis / materialized view)

Partitioning:- keyBy(userId) for per-user aggregates so all events for a user go to same parallel instance.- Ensure Kafka topic has sufficient partitions and use consistent hashing of userId to partition to balance load. For very skewed users, use salt or hot-key mitigation (split heavy users across multiple keys + combine later).

Watermark generation:- Use periodic punctuated watermarks based on maxObservedEventTimestamp - maxOutOfOrderness.- Set maxOutOfOrderness = 2 minutes (or slightly higher e.g., 2m10s to be safe).- Example: Watermark = maxTsSeen - Duration.ofMinutes(2)

Allowed-lateness:- Set allowedLateness = 2 minutes so late events within 2m update window results.- Decide whether to emit updates (retractions) or to materialize final result after lateness; for ML features, emit updates and have consumers handle updates or use versioning.

State backend:- Use RocksDBStateBackend (local RocksDB + checkpoint to durable store like S3/GCS) for large state and memory efficiency.- Enable incremental checkpoints to reduce snapshot size.

Checkpointing and fault tolerance:- Enable checkpointing every 30s (tunable). For low-latency apps, 10–30s is typical.- Set checkpoint timeout (e.g., 5m) and at-least-once vs exactly-once: use exactly-once (Kafka two-phase commit or Flink Kafka sink) for feature correctness.- Configure minPauseBetweenCheckpoints (e.g., 10s) and externalized checkpoints for quick recovery.

Scaling:- Scale Flink parallelism: increase task slot count and parallelism for source, map, and window operators.- Ensure Kafka partitions >= Flink parallelism to allow rebalance.- Use rescalable state (savepoints) for manual scaling changes.- For sudden spikes, enable autoscaling (Kubernetes/FlinkKubernetesOperator) that increases jobManager/taskManager replicas and state backend resources.- Mitigate hot keys with dynamic splitting: detect heavy users and route to dedicated shards; pre-aggregate heavy users in upstream layer.

Operational considerations:- Monitor watermark lag, late event rates, checkpoint duration, and state size per operator.- Keep window state TTL or compaction to bound state growth.- Provide schema/versioning for downstream ML feature consumers; ensure idempotency or update semantics.

This design balances correctness for 2-minute lateness, scalable per-user aggregation, and production-grade fault tolerance for ML feature pipelines.

EasyTechnical

77 practiced

Define event time versus processing time in a stream processing system. Give an example where using processing time instead of event time leads to incorrect ML feature computation (e.g., rolling counts, sessionization). Explain the correction strategy (watermarks, event-time windows, and handling clock skew).

Unlock Full Question Bank

Get access to hundreds of Batch and Stream Processing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.