Batch and Stream Processing Questions

Covers design and implementation of data processing using batch, stream, or hybrid approaches. Candidates should be able to explain when to choose batch versus streaming based on latency, throughput, cost, data volume, and business requirements, and compare architectural patterns such as lambda and kappa. Core stream concepts include event time versus processing time, windowing strategies such as tumbling sliding and session windows, watermarks and late arrivals, event ordering and out of order data handling, stateful versus stateless processing, state management and checkpointing, and delivery semantics including exactly once and at least once. Also includes knowledge of streaming and batch engines and runtimes, connector patterns for sources and sinks, partitioning and scaling strategies, backpressure and flow control, idempotency and deduplication techniques, testing and replayability, monitoring and alerting, and integration with storage layers such as data lakes and data warehouses. Interview focus is on reasoning about correctness latency cost and operational complexity and on concrete architecture and tooling choices.

HardSystem Design

74 practiced

Design a multi-tenant streaming platform that enforces tenant isolation, per-tenant quotas (throughput and storage), and fair scheduling while minimizing cost. Discuss namespace design, resource quotas, logical vs physical isolation, and billing implications.

Sample Answer

Requirements & constraints- Functional: multi-tenant ingest, storage, processing, per-tenant throughput & storage quotas, strict tenant isolation, fair scheduling, usage-based billing.- Non-functional: cost-minimize, scalable to many small tenants and some large ones, low latency, operational simplicity, secure tenancy.

High-level architecture- Ingress layer (API/Gateway) → Tenant Router → Ingest Shard Pool (stateless collectors) → Messaging backbone (topics/partitions per tenant or shared) → Stream processors (worker pool) → Object/block storage + metadata DB → Billing & Quota Service → Admin/Monitoring.

Namespace design- Logical namespace per tenant: tenant_id scoped prefixes for topics, buckets, config (e.g., tenantX/{streams,meta,checkpoints}). Use hierarchical naming: tenant:environment:stream to support sub-tenants.- Map logical namespaces to physical resources via placement rules. Keep tenant identifiers immutable and part of metadata for audit.

Resource quotas- Throughput quotas: enforce at ingress and messaging layer. Token-bucket per-tenant enforced at API gateway and broker ingress; brokers expose backpressure if exceeded. Offer burst capacity with burst tokens billed separately.- Storage quotas: per-tenant retention policies + periodic enforcement via GC that rejects writes when quota exceeded. Store per-tenant metrics in metadata DB for real-time checks.- Soft vs hard quotas: soft quotas warn and throttle; hard quotas reject and trigger escalation.

Logical vs physical isolation- Default: logical isolation (shared compute/storage) with strong namespace & access controls for cost efficiency.- For high-security/compliance tenants: offer physical isolation (dedicated clusters, VM/node pools, or tenant-specific storage accounts). Use role-based access control and encryption keys per tenant (KMS) even in shared mode.- Trade-offs: logical = lower cost, higher utilization; physical = higher cost, stronger SLA and regulatory compliance.

Fair scheduling- Scheduler assigns processing jobs to worker pool using weighted fair-queueing based on tenant SLAs and recent usage. Implement multi-level scheduling: - Admission control at broker (per-tenant tokens). - Worker-level fairness: per-tenant queues with deficit round robin or weighted fair queuing. - Priority tiers: realtime vs batch; apply isolation via cgroups/namespaces for CPU/memory on worker nodes.- Preemption: long-running low-priority jobs can be preempted for higher-priority tenant traffic.

Monitoring, telemetry, and enforcement- Real-time metrics pipeline (per-tenant ingest rate, lag, storage usage). Alerts & dashboards. Enforcement loops: throttle -> escalate -> cut-off.- Audit logs for access and billing.

Billing implications- Metering: granular metrics (ingest bytes, egress bytes, retained storage GB-days, compute seconds, burst usage). Use a central billing pipeline that consumes usage stream.- Pricing model: base subscription + tiered usage (throughput & storage) + overage charges (burst/peak). For dedicated clusters, bill flat premium + usage.- Cost optimization: autoscale shared resources; use spot instances for batch workers; lifecycle policies to move cold data to cheaper storage (archive) with per-tenant opt-in.

Trade-offs and considerations- Complexity vs cost: prefer logical tenancy for most; offer physical tenancy as premium.- SLA enforcement requires careful instrumentation and fast control loops.- Security: encrypt at rest/in transit, tenant-scoped KMS keys, strict IAM policies.- Operational: automated tenant onboarding, quota templates, simulated load tests per tenant to avoid noisy-neighbor surprises.

This design balances isolation, fairness, and cost by combining namespace-based logical isolation with tokenized quota enforcement, fair scheduling algorithms, and optional physical isolation for sensitive tenants, while producing the telemetry needed for accurate billing.

HardSystem Design

80 practiced

A client requires GDPR 'right to be forgotten' across streaming ingestion and historical storage. Design pipeline changes to support deletion requests end-to-end: include tombstones, compaction, propagation to downstream systems, and auditing for compliance. Discuss trade-offs and limitations.

Sample Answer

Requirements & constraints:- Functional: accept user deletion requests (erase personal data across live stream ingestion and historical stores), confirm completion to user, support partial deletes (by user-id, email, device id).- Non-functional: GDPR-compliant audit trail, scalable (millions/day), low impact on ingest latency, eventual consistency acceptable, retention/backup legal holds.

High-level architecture:- Ingest: Kafka (topic-per-domain) with key = canonical subject-id (hashed).- Storage: Delta Lake / Iceberg on object storage for historical data; OLTP store for user profiles; downstream consumers (analytics, search, BI).- Control plane: Deletion service + audit store + orchestration (workflow engine).

Flow & components:1. Deletion request API: validates identity, creates Deletion Job with scope, reason, request-id, TTL, legalHold flag; writes request to deletion-commands topic.2. Tombstone generation: Deletion service emits tombstone events to Kafka (keyed by subject-id) and writes deletion marker metadata to central index (fast lookup).3. Stream handling: Consumers subscribe and on tombstone apply in-flight scrub logic: stop emitting PII, map to pseudonymized id, or drop records. Downstream connectors (CDC, sink) must interpret tombstone semantics.4. Historical compaction: Periodic compaction job reads Iceberg/Delta partitions, filters rows matching deletion index, rewrites files excluding PII or physically deletes rows (using file-level replace/delete APIs) and inserts compaction audit record. Use row-level deletes where supported; otherwise rewrite partitions.5. Downstream propagation: Sinks (Elasticsearch, warehouses) receive tombstones or compaction events; connectors support delete-by-key or reindex. For systems that can’t delete, mark records as deleted and exclude from queries.6. Auditing & compliance: Every action (request, tombstone emitted, consumer ack, compaction job, downstream deletion) logged immutably to append-only audit store (WORM-enabled object storage or blockchain-like ledger) with request-id, timestamps, actor, checksums, and signatures. Provide auditor/UI to show status and proofs.7. Backups & legal holds: Deletion service marks backups; disposable snapshots older than legalHold are scheduled for secure purge; for immutable backups, maintain legal/retention policy and provide justification to regulators.

Trade-offs & limitations:- Consistency: deletions are eventually consistent across systems; short windows where PII may persist.- Cost/performance: compaction and rewrite are I/O heavy—periodic batching reduces cost but increases latency to full deletion.- Downstream capabilities: third-party systems may not support deletes or fast reindexing; require adapters or contractual SLAs.- Backups & replicas: true removal from immutable backups may be impossible without re-encrypting or destroying keys—requires policy coordination.- Complexity: orchestration, id mapping, and audit increase system complexity and operational burden.

Operational recommendations:- Use canonical hashed subject-id and propagate it everywhere.- Implement strict access controls, monitoring, and test deletion workflows.- Provide SLA targets (e.g., tombstone propagation within minutes, historical purge within X days) and communicate limitations to legal/compliance.This design balances legal requirements, scale, and operational cost while making deletion end-to-end auditable and repeatable.

EasyTechnical

84 practiced

Explain Change Data Capture (CDC): what it is, common uses (e.g., streaming OLTP changes to analytic stores), and pros/cons of CDC vs batch extracts. What properties of the source DB affect CDC design?

Sample Answer

Change Data Capture (CDC) is a pattern/technique to capture and deliver only the data changes (inserts, updates, deletes) from a source system to downstream consumers in near real-time. Common implementations: log-based (read DB’s WAL/binlog/transaction log), trigger-based, or time-stamp/polling.

Common uses- Streaming OLTP changes into analytic stores or data lakes for near-real-time analytics and reporting- Feeding event-driven architectures, caches, search indexes, message buses- Synchronizing microservices and supporting near-real-time ETL/ELT pipelines- Auditing, replication, and cache invalidation

CDC vs batch extracts — pros and consPros of CDC:- Low latency: near real-time propagation of changes- Efficient: only deltas move, reducing data transfer and processing- Better for continuous pipelines and event-driven appsCons of CDC:- More complex: requires reliable ordering, exactly-once or idempotency handling- Operational overhead: schema evolution, schema drift, and monitoring- Integrations may require connector maturity and DB-specific handling

Batch extracts — pros:- Simpler to implement and reason about- Easier full-refresh or periodic consistency windowsCons:- Higher latency, more resource intensive, and often transfers redundant data

Source DB properties that affect CDC design- Durability of change logs (does DB expose a stable WAL/binlog?)- Transaction model and isolation level (multi-statement transactions, ordering guarantees)- Retention of logs and ability to replay (truncate/GC windows)- Throughput and write volume (affects connector throughput/backpressure)- Schema evolution support and DDL visibility- Support for native change streams/connectors (e.g., Debezium support, proprietary tools)- Replication/permission constraints (read-replicas, impact on production)

Design implications: prefer log-based CDC where available for minimal impact; plan for idempotency, ordering, exactly-once semantics or deduplication; add schema registry/versioning and monitoring; size retention windows and backfill strategies to handle outages.

MediumSystem Design

81 practiced

Design a pipeline for a client who needs hourly aggregated metrics stored in a data warehouse and also a near-real-time dashboard with sub-30s latency. Outline components, whether you'd use lambda or kappa, sources, stream engine, storage, and why. Include trade-offs in cost and operations.

Sample Answer

Requirements clarification:- Hourly aggregated metrics persisted to a data warehouse for analytics and billing.- Near-real-time dashboard with end-to-end latency <30s, accurate enough for operational monitoring.- Expect high-throughput event sources, some late-arriving events, and need for deduplication.

Recommended approach: Kappa architecture (stream-first, single processing path) using a durable log + stream processing to produce both realtime-served aggregates and hourly persistent aggregates. Kappa reduces code duplication and operational overhead versus Lambda.

High-level components:1. Sources: app events, mobile, web, DB CDC (Debezium) → produce immutable events to a durable log.2. Durable log / ingestion: Apache Kafka or AWS Kinesis (Kafka preferred for ecosystem & retention).3. Stream engine: Apache Flink or Apache Beam (Dataflow) for stateful, windowed aggregations, exactly-once semantics, watermarks for late data, deduplication. - Compute streaming rolling windows (e.g., tumbling 1h for hourly) and shorter windows for dashboard (1s–10s tumbling/sliding).4. Serving for dashboard: low-latency OLAP store like ClickHouse / Druid / Pinot or materialized state exposed via gRPC/REST (Flink state via Queryable State or materialize to Redis for ultra-low latency). - Choose Druid/Pinot if fast rollups and time-series queries needed; Redis for single-value counters.5. Data warehouse: Snowflake / BigQuery / Redshift. Stream engine writes hourly aggregate records (partitioned by hour) into DW via batch loads (S3 staging) or streaming ingestion (Snowpipe / BigQuery streaming).6. Monitoring / infra: Prometheus + alerting, Kafka Connect for sinks, schema registry (Avro/Protobuf), CI/CD for streaming jobs, SLAs, and data quality checks.

Why Kappa:- Single code path for realtime and hourly aggregates reduces bugs and maintenance.- Stateful stream engines support event-time semantics and late-arriving data, enabling correct hourly buckets.- Exactly-once or idempotent sinks minimize duplication.

Trade-offs:- Cost: Kappa with Flink + Kafka has higher runtime cost (always-on clusters) vs. Lambda can run batch jobs less frequently. Managed services (Dataflow, MSK, Snowflake) increase cost but reduce ops.- Operational complexity: Kappa requires expertise in stream processing, state management, monitoring. Lambda adds complexity in maintaining two codebases (stream + batch) and reconciling results.- Latency vs. cost: For sub-30s, streaming engine + serving store adds cost but meets SLA. Cheaper option: micro-batching (5–15s) if some slack allowed.- Durability: Kafka retention simplifies reprocessing; with Lambda, replaying batch pipelines is also possible but more error-prone.

Edge cases & reliability:- Implement watermarking and allowed-lateness policies; emit early partial aggregates and update them on late-arrival if needed.- Use schema registry and versioning, idempotent writes to DW, and transactional connectors (exactly-once Kafka sinks or S3 staging + atomic load).- Provide fallbacks: if stream processing backpressures, route to a temporary store and backfill hourly.

Recommendation summary:- Use Kappa with Kafka + Flink (or Dataflow) + Druid/ClickHouse for dashboard + Snowflake for hourly DW.- Accept higher infra cost for reliable sub-30s latency and simpler maintenance long-term. If client is cost-sensitive and can accept larger latency or eventual consistency, consider Lambda hybrid (streaming for dashboard, batch for DW).

EasyTechnical

73 practiced

Define event time and processing time in streaming systems. Provide an example where event time and processing time differ (e.g., mobile client offline then reconnects), explain the practical consequences for aggregation and joins, and how you would design to use event time correctly.

Unlock Full Question Bank

Get access to hundreds of Batch and Stream Processing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.