Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.
EasyTechnical
64 practiced
Explain event time versus processing time and why watermarks are important for windowing semantics. Provide an SRE-focused example: a streaming analytic that computes 5-minute event-time windows with late arrivals up to 2 minutes. How do watermarks, allowed lateness, and triggers affect correctness and resource usage?
Sample Answer
Event time vs processing time:- Event time is the timestamp when an event actually occurred (e.g., log timestamp). Processing time is when the stream processor sees the event. Using event time yields correct semantics when events are delayed or reordered; processing time yields low-latency but can produce incorrect results if data arrives late.Why watermarks matter:- A watermark is a low-bound estimate of event-time progress (e.g., “no events with event-time ≤ T are expected”). It lets the system decide when to close event-time windows despite out-of-order arrivals. Without watermarks you either wait forever (correct but unavailable) or close windows too early (fast but incorrect).SRE-focused example (5-minute event-time windows, 2-minute late arrivals):- Requirements: compute aggregates for each 5m event-time window; accept late events up to 2 minutes; emit final results after that.- Watermarks: advance watermark = max(event_time_seen) - allowed_skew. If watermark passes window_end (window_end ≤ watermark), the window can be fired. To allow 2m lateness, watermark must lag by at least 2m relative to max observed event time.- Allowed lateness vs triggers: - Allowed lateness: policy to accept late events and update previously emitted windows (e.g., keep state for +2m). This increases correctness (incorporates late arrivals) but increases state retention and re-computation. - Triggers: control when to emit partial/final results (e.g., early triggers on processing-time for low-latency previews, and a final trigger when watermark passes window_end + allowed_lateness).- Effects on correctness and resource usage: - Correctness: combining watermarks + allowed lateness ensures final results are stable after the lateness bound. If watermark lags too little, late events will be dropped or counted in a subsequent window (incorrect); if too conservative, results are delayed. - Resource usage: holding window state for extra allowed_lateness increases memory/disk and can increase CPU for merging updates. Early triggers increase output volume and downstream load. Watermark aggressiveness trades off latency vs risk of missing late events.- SRE recommendations: - Set watermark skew based on measured event delay distribution (P95/P99). - Monitor watermark lag, state size, and retractions/updates metrics. - Use early/accumulating triggers for operational visibility, but emit a final, idempotent result after watermark + allowed_lateness to ensure correctness. - Alert if watermark stalls or state growth exceeds thresholds (indicates upstream delays or misconfigured clocks).
EasyTechnical
43 practiced
Compare tumbling, sliding, and session windows in streaming systems. Give operational examples where each is appropriate (e.g., fixed-interval metrics vs sessionization). What SRE considerations affect window choices: state size, checkpoint frequency, and late-event handling?
Sample Answer
Tumbling, sliding, and session windows are common stream-windowing strategies; choose based on semantics, accuracy, and operational cost.Definitions & when to use- Tumbling windows: Non-overlapping, fixed-duration buckets (e.g., 1m). Good for fixed-interval metrics such as per-minute error rates, billing slices, or rolling dashboards where disjoint aggregates are acceptable.- Sliding windows: Fixed-duration windows that advance by a smaller hop (e.g., window=5m, slide=1m) producing overlapping results for finer-grained moving aggregates. Use for moving averages, anomaly detection, or smoothing latency trends.- Session windows: Dynamic windows that group events by inactivity gaps (e.g., user sessions with 30m gap). Use for user/sessionization, retention metrics, and engagement funnels where event-driven boundaries matter.Operational SRE considerations- State size: Sliding windows and long session windows keep more state (overlaps or per-key buffers). Plan memory and state-backend sizing; partitioning and TTLs reduce pressure.- Checkpoint frequency: More frequent checkpoints reduce recovery time but increase I/O throughput. For critical observability (alerts), choose shorter checkpoint intervals; for high-throughput metrics, balance checkpoint cost vs. acceptable recovery window.- Late-event handling: Decide watermarking and allowed lateness. Tight watermarks minimize state retention and latency but may drop late events. If correctness matters (billing, compliance), allow larger lateness and store side-outputs for late arrivals; otherwise prefer low-latency with best-effort processing.- Trade-offs & SLOs: Align window choice with SLOs (freshness vs. correctness). Monitor state growth, checkpoint durations, and restart times; alert on state-store saturation or prolonged recovery.
HardTechnical
36 practiced
Your Kafka cluster handles billions of events per day across thousands of partitions. You need to rebalance partitions and add brokers without downtime. Design an operational plan to perform reassignments safely, minimize data movement, and maintain cluster health metrics. Include monitoring thresholds and throttling strategies.
Sample Answer
Requirements & constraints:- Zero customer-visible downtime; durable ordering preserved.- Billions/day, thousands partitions → minimize cross-broker movement and network/io impact.- Maintain SLOs: 99.95% availability, P99 produce latency < X ms, consumer lag < Y.Plan (high level):1. Pre-check & capacity planning- Verify new brokers have same/deterministic config (logs dirs, disks, network, JVM).- Capacity target: <= 70% disk, <= 60% I/O utilization per broker post-move.- Run dry-run: gather current partition sizes, leader distribution, replica counts, ISR health.2. Create reassignment strategy (minimize movement)- Use rack-aware, weighted bin-packing: prefer placing replicas on new brokers only for partitions with largest imbalance or hot partitions.- Prioritize moving followers first; avoid moving leaders unless necessary.3. Automation & incremental execution- Generate reassignment plan using kafka-reassign-partitions (or custom tool) with small batches (e.g., 50–200 partitions per batch).- Use --throttle-bytes & dynamic broker config kafka.server.BrokerTopicStats to control replication throttling.Example throttling strategy- Start at conservative rate (e.g., 50 MB/s per broker), adjust based on health.- Throttling controls: - kafka-reassign-partitions --throttle 50MB/s (or set broker config: replica.replication.throttled.rate) - Use Cruise Control or custom controller to rate-limit per-broker.4. Monitoring & thresholds (must gate batches)- Key metrics: - Under-replicated partitions (URP): trigger immediate pause if > 0 (or > 1% for planned brief spikes). - ISR shrink events count: stop if > 5 in 15m. - Broker disk usage: stop batch if any broker > 80% disk. - Network IO: stop if replication throughput saturates > 70% NIC. - Produce/Fetch latency (P99): pause if increases > 30% over baseline for >5m. - Consumer lag: pause if lag growth rate > baseline*2 for >5m.- Alerting: automated PagerDuty on critical thresholds.5. Execution loop- For each batch: - Ensure cluster is healthy (URP=0, all brokers up). - Apply reassignment for followers only. - Set throttle to current rate. - Monitor 1-min and 5-min metrics. If any threshold breached -> reduce throttle by 50% or pause. - When finished, confirm replicas in-sync, update preferred leader for affected partitions (run preferred-leader election) only after replicas are caught up. - Rebalance leaders if needed in separate controlled wave.6. Post-migration tasks- Remove throttle, run cluster-wide preferred leader election if desired.- Recompute broker weights and capacity plan.- Run continuous monitoring for 24–72 hours.Rollback & safety- Plan idempotent reassignments (can resume).- Keep small batch size so reverts are quick: if severe issues, stop further batches and optionally reassign back by reversing plan.- Maintain snapshots of config and ZK/metadata backups.Tools & automation- Use Cruise Control for suggestions + automation with manual approvals.- Use Kubernetes/Ansible jobs to run kafka-reassign-partitions with JSON plans.- Build dashboards showing URP, ISR events, per-broker replication throughput, disk, NIC, produce/consume latency.Rationale: move followers first minimizes leader disruption; batching + throttling prevents resource saturation; strict gating based on URP/ISR/disk/network maintains availability.
HardTechnical
36 practiced
Propose a set of chaos engineering experiments to validate resilience of a streaming platform (Kafka + Flink). Include experiment goals, scenarios (broker failure, network partition, high GC, state backend corruption), safety controls, and metrics to measure the impact. How would you automate and roll out these experiments safely?
Sample Answer
Goal: validate end-to-end resilience of Kafka + Flink streaming pipeline by proving the system maintains SLOs, recovers without data loss/duplication, and exposes observable failure modes.Experiments (each includes hypothesis, steps, expected outcome):1) Broker failure (leader crash)- Hypothesis: Partition leaders failover within target latency and consumers resume with ≤1x duplicate messages.- Scenario: Kill Kafka broker process or node hosting multiple leaders.- Expected: ISR re-election < 30s, consumer lag spike bounded, no data loss.- Metrics: controller rebal time, leader election time, consumer lag, end-to-end processing latency, error rates, message offsets continuity.2) Network partition between Kafka and Flink- Hypothesis: Flink maintains processing progress using buffered state and resumes exactly-once (or at-least-once per config).- Scenario: Apply iptables drop on cluster subset for defined window (e.g., 60s).- Expected: Backpressure increases; Flink checkpoints fail/recover per policy.- Metrics: checkpoint success rate, backpressure metrics, sink throughput, checkpoint alignment time.3) High GC on brokers / task managers- Hypothesis: GC pauses cause bounded latency and don't corrupt state.- Scenario: Inject synthetic memory pressure or change JVM flags to trigger long GC.- Expected: Temporary latency spikes; system recovers without state loss.- Metrics: GC pause distribution, request latency percentiles, task manager heartbeats, failover counts.4) State backend corruption (simulated)- Hypothesis: Corrupted checkpoint manifests are detected and trigger safe rollback or restore.- Scenario: Tamper specific checkpoint files in durable storage or simulate read errors.- Expected: Job fails-fast or skips corrupted checkpoint per configured recovery; alerting triggers.- Metrics: checkpoint validation failures, job restart counts, data reprocessing volume.Safety controls- Run first in staging with production-like load and sampled topics.- Limit blast radius: single partition, single consumer group, small time window.- Schedule during low traffic, with manual abort and automated rollback hooks.- Canary flagging: feature flags to divert a small % of traffic.- Automated kill-switch: health checks that stop experiments on SLO breach.- Read-only or replay-safe sinks for experiments that risk duplication.Automation & rollout- Implement experiments as reproducible playbooks using chaos tools (Chaos Mesh/SteadyBit/Gremlin) + Kubernetes jobs and Ansible for VMs.- Parameterize: duration, target nodes, severity. Store configs in Git.- Pipeline: stage → runbook validation → dry run (staging) → canary (1% traffic), monitor automated guardrails → gradual ramp to prod subsets.- Integrate with CI/CD: require automated tests, monitoring dashboards, and on-call notification policies before increase.- Observability: dashboards, synthetic transactions, distributed tracing, and automated post-run reports (including metrics, logs, and root-cause signals).- Postmortem: every experiment produces an action list: fix gaps, adjust SLOs, and re-run until acceptance criteria met.This approach balances proving resilience, minimizing risk, and creating repeatable, automated experiments that feed continuous improvement.
MediumTechnical
41 practiced
Design a testing strategy for streaming pipelines that covers unit tests, integration tests, and production-like end-to-end tests. Include tools and frameworks you would use (e.g., Embedded Kafka, Flink test harness, Docker), methods for deterministic replay, and how to automate regression tests in CI/CD.
Sample Answer
Approach: split testing into three layers (unit, integration, production-like E2E), make tests deterministic, and automate them in CI/CD with gated promotion and scheduled regression runs.Unit tests (fast, developer feedback)- Tools: JUnit/PyTest, Mockito, local Flink Test Harness (OneInputStreamOperatorTestHarness, ProcessFunctionTestHarness) or Beam DirectRunner for transforms.- What to test: pure business logic, time-window calculations (use harness to advance event/process time), watermark handling, state updates.- Determinism: inject clocks, fixed RNG seeds, mock Kafka producers/consumers.Integration tests (component interactions)- Tools: Embedded Kafka (Embedded Kafka for JVM) or Testcontainers Kafka, Confluent Schema Registry test containers, Docker Compose, Flink MiniCluster or LocalCluster.- What to test: serialization/deserialization (Avro/Protobuf), connector configs, exactly-once semantics with checkpoints, consumer group behavior.- Methods: start ephemeral Kafka + schema registry, produce recorded test vectors, run pipeline against topics, read output topics and assert results. Use Flink savepoints to validate stateful upgrades.Production-like E2E tests (resilience, performance)- Tools: Kubernetes (kind/minikube for CI; real staging k8s for nightly), Docker, Testcontainers for local dev, k6 or Gatling for load, Prometheus + Grafana for metrics.- What to test: full ingest → processing → sink path, scaling (rebalance), failover (kill taskmanager/pod), rolling upgrades, checkpoint recovery, end-to-end latency, SLO violations.- Deterministic replay: store canonical input logs (Kafka topics exported to compressed files or offsets) in artifact storage; replay by producing them into topics at controlled rates; use fixed timestamps or event-time preserved; for replaying exactly stateful flows include starting from saved state (Flink savepoint) so behavior is reproducible.Regression & CI/CD automation- Pipeline stages: 1. Unit tests on every PR (fast). 2. Integration tests in CI using Testcontainers or small kind cluster (run with tags; parallel where possible). 3. E2E smoke tests in a staging namespace on merge (deploy via Helm/Kustomize). 4. Nightly full regression/perf tests in staging with production-like data volumes.- Tools: GitHub Actions / Jenkins / GitLab CI; use Docker layers for caching; Testcontainers for Kafka to keep CI hermetic.- Flakiness & observability: mark flaky tests, collect logs/artifacts (Kafka topics, savepoints, metrics) on failure; upload to CI artifacts. Gate merges on integration success and automated canary health checks.- Regression tracking: store baseline metrics, fail builds on latency/SLA regressions, add synthetic SLO tests that assert percentiles.- Rollback strategy: automated rollback if canary health degrades; keep savepoints for fast restore.Best practices summary:- Keep unit tests fast and deterministic with harnesses and injected clocks.- Use Testcontainers/embedded Kafka for reproducible integration runs.- Replay canonical input and start from savepoints for deterministic E2E.- Automate multi-stage CI with nightly regression and clear observability/artefact capture for debugging.
Unlock Full Question Bank
Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.
