Real Time and Batch Ingestion Questions
Focuses on choosing between batch ingestion and real time streaming for moving data from sources to storage and downstream systems. Topics include latency and throughput requirements, cost and operational complexity, consistency and delivery semantics such as at least once and exactly once, idempotent and deduplication strategies, schema evolution, connector and source considerations, backpressure and buffering, checkpointing and state management, and tooling choices for streaming and batch. Candidates should be able to design hybrid architectures that combine streaming for low latency needs with batch pipelines for large backfills or heavy aggregations and explain operational trade offs such as monitoring, scaling, failure recovery, and debugging.
HardTechnical
96 practiced
How would you test and validate end-to-end correctness of streaming pipelines that feed training data for ML models? Describe unit, integration, property-based tests, replaying archived events, canary replays, drift detection, and gating strategies to prevent bad data from reaching training/serving.
Sample Answer
Start by clarifying goals: ensure streaming pipeline correctness (schema, semantics, completeness, ordering), reproducibility of training data, and prevent bad data from reaching training/serving.Unit tests- Test transformations and feature extractors in isolation with deterministic small inputs (including nulls, duplicates, out-of-order timestamps). Use pytest/unittest and assert exact feature vectors, types, and error handling.- Mock external I/O (Kafka producers/consumers, cloud storage) so logic is validated fast.Integration tests- End-to-end tests in a sandbox cluster that wire up the real components (stream processors, Kafka, state stores, sinks). Inject synthetic event streams and validate outputs in sinks (Parquet/TFRecords). Verify event-time semantics, watermarking, windowing, and exactly-once guarantees where required.Property-based tests- Use Hypothesis or jqwik to generate broad families of inputs and assert invariants: idempotence of deduplication, monotonic timestamp ordering after watermarking, feature ranges (no NaNs unless expected), stable hashing of IDs. These catch edge-case combinatorics unit tests miss.Replaying archived events- Archive raw input event streams (compact storage with partition/time metadata). Periodically replay historical windows through pipeline in test environment to validate deterministic outputs and to catch divergence after code changes. Compare hash signatures of generated training datasets to baseline.Canary replays- Before committing live training data changes, run canary replays in production-like environment on a sampled slice (e.g., 0.1% traffic or a time-slice) and compare feature distributions, missing-rate, and label alignment. Surface diffs to alerting dashboards and require manual/automated approval.Drift detection- Continuously monitor feature distributions, cardinality, null rates, and schema changes with streaming metrics (KS/JS divergences, population stability index). For labels, monitor prediction vs. actual feedback. Track both population drift and concept drift; trigger periodic replay tests when thresholds exceeded.Gating strategies- Implement multi-layer gates: - Schema gate: reject events failing schema/validation at ingestion. - Sanity gate: block batches where key invariants (null rate, outlier counts, cardinality) exceed thresholds. - Canary gate: require successful canary replay and automated checks pass before promoting data to training. - Model gate: automate rollback or freeze training/serving if post-training evaluation degrades beyond SLA.Use feature flags / CI pipeline integration so promotion requires passing tests and stakeholder sign-off.Observability and tooling- Compute deterministic dataset fingerprints (stable hashing) and sample diffs. Store lineage metadata so any training dataset can be traced to input offsets. Automate alerts, runbooks, and a “retrain dry-run” that validates downstream model training and evaluation pipelines without promoting models.This combination—fine-grained unit/property tests, realistic integration & replay tests, automated canaries and drift detection, and strict gating—maintains end-to-end correctness while enabling safe iteration.
MediumBehavioral
73 practiced
Tell me about a time you led or contributed heavily to a migration of an ingestion pipeline (streaming or batch). Describe the situation, stakeholders, technical trade-offs you evaluated, how you handled schema and compatibility issues, and the outcome. Use STAR (Situation, Task, Action, Result).
Sample Answer
Situation: At my previous company we migrated a user-event ingestion pipeline from a batch-based Airflow + S3 job to a near-real-time Kafka Streams pipeline because ML features and online models needed fresher data for personalization. Stakeholders included ML engineers, data platform, backend services, product owners, and on-call SREs.Task: I was lead ML engineer responsible for designing the migration plan to deliver sub-minute features, preserve data quality, and avoid model regressions or downtime.Action:- Aligned requirements with stakeholders: freshness (<=60s), ordering guarantees, throughput (~10k events/s), and cost constraints.- Evaluated trade-offs: Kafka Streams for low latency and exactly-once semantics vs. Kinesis (managed service) vs. sticking with micro-batches. Chose Kafka with consumer-side deduplication because it balanced latency, control, and cost.- Designed schema strategy: adopted Avro with a strict compatibility policy. I introduced a versioned schema registry (Confluent) and enforced backward/forward compatibility checks in CI. For existing S3 batches I wrote a one-time adapter to convert legacy JSON to Avro with explicit nullable fields and default values.- Implemented a blue-green rollout: dual-write from backend services to both old batch path and new Kafka topic, and ran consumers that validated parity (counts, sampled events, key distributions). Built automated Canary tests comparing model predictions between batch-derived features and stream-derived features.- Coordinated on-call readiness: created runbooks, alerts for lag and schema errors, and a rollback plan to switch off producers to Kafka if issues arose.- Handled schema evolutions by using default values for added fields, deprecating fields with a compatibility window, and communication threads with product for planned changes.Result: Migration completed over 4 weeks with zero production downtime. Feature freshness improved from hours to <45s, model AUC improved by 2.3% on live traffic, and inference latency remained within SLA. Post-migration, incident count related to data freshness dropped 80%. Key learnings: invest early in schema governance, run parity checks before cutover, and involve SREs and product owners in acceptance criteria to avoid surprises.
EasyTechnical
94 practiced
Explain backpressure in streaming systems. Describe typical buffering strategies (in-memory queues, persistent queues, disk spill), reactive approaches vs push-based systems, and how to design ingestion to avoid OOM or message loss while keeping latency bounded.
Sample Answer
Backpressure is the mechanism by which a slower consumer signals a faster producer to reduce input rate so the system avoids resource exhaustion (OOM) or message loss. In streaming ML pipelines this prevents feature/serving layers from being overwhelmed when model inference or downstream writes lag.Buffering strategies:- In-memory queues: low latency, fast, but limited capacity — risk OOM under sustained bursts.- Persistent queues (e.g., Kafka, Pulsar): durable, can retain large volumes, provide replay and backpressure via consumer lag; moderate latency.- Disk spill (local spill-to-disk): hybrid—keep hot items in memory and spill overflow to disk to avoid OOM; increases tail latency.Reactive vs push-based:- Reactive (pull/credit-based): consumers request work at their capacity (Reactive Streams, Akka, gRPC flow control). Natural backpressure, predictable memory use.- Push-based: producers push messages; need external flow-control (throttling, acks) or risk buffer growth.Design to avoid OOM/message loss while bounding latency:- Use durable queues for ingest and keep producer retries/idempotency to avoid loss.- Apply backpressure: prefer pull-based consumption or implement TCP/HTTP flow-control and credits.- Set bounded in-memory buffers with clear overflow policy: drop-oldest/drop-newest with metrics, or redirect to persistent store.- Autoscale consumers (stateless inference workers) based on lag and latency SLOs; combine with concurrency limits per worker.- Prioritize messages (sampling, rate-limits) to protect latency-sensitive traffic.- Monitor consumer lag, queue lengths, memory; alert and automate scaling/flow-control.- Implement end-to-end timeouts and SLA-based dead-lettering for stuck messages.For ML: batch inference with adaptive batch sizes, backpressure-aware batching, and graceful degradation (serve simpler model) keep latency bounded while protecting system stability.
MediumTechnical
104 practiced
How do you ensure data lineage, governance, and reproducibility for pipelines that combine streaming ingestion and batch ETL? Describe metadata tracking, schema/versioning, access controls, and how replayability is supported for model retraining and audits.
Sample Answer
Approach: treat the pipeline as a product — capture metadata at every handoff, make artifacts immutable/versioned, enforce access via RBAC and audit logs, and design for deterministic replay from source events or materialized snapshots.Design (components & practices):- Metadata & lineage: instrument ingestion, ETL jobs, feature store, and model training to emit standardized lineage events to a metadata catalog (OpenLineage/Marquez or Apache Atlas). Capture dataset IDs, schema, source offsets/timestamps, job/run IDs, code commit SHAs, container image, and configuration.- Schema & versioning: use a schema registry for AVRO/Protobuf/JSON schemas and enforce compatibility rules. For tabular data, rely on table formats with versioning (Delta Lake / Iceberg / Hudi) so each write is a versioned snapshot; store schema history and evolution diffs.- Reproducibility & replayability: for streaming sources (Kafka/Kinesis), persist source topic, partition, and offset with ingestion watermark; enable replay by seeking offsets or reprocessing from event-time windows. For batch, persist input snapshot URIs (S3 object versions or Delta time-travel timestamps). Pin code artifact (git SHA), environment (container image digest), and hyperparameters via ML metadata (MLMD) so runs are deterministic.- Feature provenance & model retraining: use a feature store that records lineage from raw source to feature with transformation code/version. When retraining, reference the exact feature snapshot or replay raw events to recompute identical features; automated pipelines should accept a "replay token" (offsets/timestamp or snapshot id).- Access controls & governance: enforce IAM/RBAC for datasets, schema registry, model registry, and metadata catalog; use column-level encryption/masking and policy engines (e.g., Apache Ranger, Cloud DLP). Require approval workflows for schema changes and dataset sharing. Log all accesses to an append-only audit store with cryptographic hashes for tamper evidence.- Audits & validation: store hashes of dataset snapshots and model artifacts; enable automated validation checks (row counts, schema diffs, distribution drift) emitted into lineage events. Provide a UI/CLI to trace model prediction back to the raw event, transformation code, and training run.Example flow:1. Streaming ingestion writes raw events to Kafka and object store; ingestion job emits OpenLineage event with topic/offset range and S3 URIs.2. Batch ETL reads those S3 snapshots (or replays Kafka offsets), writes versioned Delta tables and registers schema change in registry.3. Feature store materializes features referencing Delta table version + transformation code SHA.4. Training job records MLMD run linking dataset snapshot ID, feature snapshot ID, code/image digest, hyperparams, and produces model artifact stored in model registry.5. For retrain/audit, operator provides snapshot ID or offsets; orchestrator rehydrates exact inputs and code to reproduce model. Lineage UI shows traceability and access logs.Why this works:- Versioned storage + schema registry makes data evolution explicit and reversible.- Lineage events tie data, code, and runs into a single traceable graph.- Storing offsets and immutable snapshots enables exact replay for retraining and regulatory audits.- RBAC and immutable audit logs ensure governance and compliance.Key metrics to track: percent of production models with end-to-end lineage, average time to reproduce a model, number of unauthorized access attempts, and % of automated retrains that exactly reproduce prior metrics.
HardTechnical
78 practiced
Your streaming system reports message loss after autoscaling events. Design experiments and diagnostic steps to identify the root cause spanning producer settings, broker behavior, consumer rebalancing, and network issues. Specify logs and metrics to capture and targeted tests to reproduce the issue.
Sample Answer
Approach: run controlled, observable experiments that isolate producer, broker, consumer, and network. Use deterministic synthetic traffic with unique, monotonic message IDs so you can detect loss vs duplication and tie events to autoscaling windows.1) Baseline instrumentation (what to capture)- Producer metrics/logs: record-send-rate, record-error-rate, retry-rate, record-queue-time, request-latency, inflight-requests, broker acks, producer logs at DEBUG. Capture producer config: acks, retries, max.in.flight.requests.per.connection, enable.idempotence, linger.ms, batch.size, buffer.memory, transactional.id.- Broker metrics/logs: MessagesInPerSec, BytesInPerSec, RequestHandlerPool metrics, UnderReplicatedPartitions, OfflinePartitionsCount, LogFlushRateAndTime, ISR changes, leader election/ preferred leader metrics, controller.log and server.log, JVM GC, disk I/O, socket-server request metrics, zookeeper/controller events.- Consumer metrics/logs: consumer_lag (per partition), commit-rate, failed-commits, poll-duration, rebalance-events, group-coordinator logs, max.poll.interval.ms, session.timeout.ms, auto.commit enabled, consumer log DEBUG.- Network/infra: tcp retransmits, device errors (ifconfig/ethtool), kernel drops, load balancer logs, cloud autoscaler events, CPU/memory on nodes, pod restarts, ephemeral storage, CNI metrics.- End-to-end: write/consume counters, message IDs, timestamps, and a dedup/seq check.2) Targeted experiments to reproduce- Controlled autoscale up/down: run synthetic producer/consumer at production QPS and trigger autoscaling (scale replicas + scale down). Observe loss window aligned to scaling events.- Producer-only stress: scale producers up/down without touching consumers/brokers to see if producers drop or timeout.- Broker perturbation: add/remove broker (or simulate rolling restart), change partition leaders, emulate slow disk to provoke leader moves.- Consumer rebalance tests: force rebalances (change group membership), sleep consumers to trigger max.poll.interval, vary session.timeout to see commit/offset loss.- Network fault injection: introduce packet loss, latency, or NAT timeouts during scale events.- Exactly-once checks: enable idempotence/transactions vs at-least-once and verify with sequence IDs.3) Repro tests & validation- Produce N messages with seq IDs; after events, consume and verify count, missing IDs, duplicates, and out-of-order.- Capture timeline: correlate autoscaler events, Kubernetes pod lifecycle events, Kafka controller logs (leader changes), producer/client timeouts, and consumer rebalance events.- If loss occurs only when consumer rebalances: check commits (are offsets committed before revoke?), use cooperative rebalancing (if supported), reduce max.poll.interval or increase commit frequency, or use manual commits during processing.- If loss aligns with broker leader moves/under-replicated partitions: inspect ISR shrink/leader election logs, check unclean.leader.election.enable (unsafe), check replication lags and disk pressure.- If loss aligns with producer timeouts/inflight drops: check retries, ack settings, idempotence; increase retries/backoff, set max.in.flight.requests.per.connection=1 or enable.idempotence for safe retries.- If loss aligns with network glitches: examine tcp retransmits, ephemeral port exhaustion, loadbalancer connection draining; add connection draining safeguards during scale, increase timeouts.4) Triaging matrix (quick mapping)- Missing messages + no consumer rebalances + broker leader changes → broker replication/ISR/leader election problem.- Missing messages + producer errors/timeouts/retries high → producer config/network causing dropped sends.- Missing messages + rebalance events during commit gaps → consumer commit/timing issue.- Intermittent loss with TCP errors → network/CNI/loadbalancer.5) Actionable mitigations to test- Enable producer idempotence and/or transactions; increase retries/backoff; set acks=all.- Harden consumers: use manual commits after processing, increase session/max.poll intervals, use cooperative rebalancing, ensure graceful shutdown on scale-down to finish commits.- Broker tuning: monitor/mitigate disk pressure, add broker capacity, avoid uncontrolled leader churn, disable unclean leader election.- Network: increase keepalives, tune LB draining, fix MTU/CNI issues.6) Deliverables for incident runbook- Exact list of logs/metrics to collect, synthetic test harness (producer/consumer) that emits sequence IDs and auto-runs autoscale scenarios, and a playbook: correlate timestamps, identify root cause, apply mitigations, repeat until no loss.This structured experiment plan lets you isolate which layer (producer, broker, consumer, network) is the root cause and validate fixes with reproducible tests.
Unlock Full Question Bank
Get access to hundreds of Real Time and Batch Ingestion interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.