Data Pipeline Scalability and Performance Questions

Design data pipelines that meet throughput and latency targets at large scale. Topics include capacity planning, partitioning and sharding strategies, parallelism and concurrency, batching and windowing trade offs, network and I O bottlenecks, replication and load balancing, resource isolation, autoscaling patterns, and techniques for maintaining performance as data volume grows by orders of magnitude. Include approaches for benchmarking, backpressure management, cost versus performance trade offs, and strategies to avoid hot spots.

MediumTechnical

30 practiced

Design a benchmarking and load-test strategy for a streaming pipeline (producer -> Kafka -> stream processor -> feature-store). Describe what to measure (throughput, end-to-end latency, p99, consumer lag, resource utilization), how to generate realistic traffic (key distribution, payload sizes, timing), how to isolate components, and which failure modes to simulate during tests.

Sample Answer

Start by clarifying goals: validate correctness, SLOs (throughput, E2E latency, p99), and resource cost under realistic ML-serving load.

What to measure- Throughput (msgs/sec and features/sec) per component and end-to-end.- End-to-end latency distribution (median, p95, p99) from producer timestamp to feature-store write.- Kafka consumer lag (records & time) and backlog growth rate.- Per-process resource utilization (CPU, memory, network, disk IOPS), GC/memory pressure for JVM services.- Error rates, retries, and dropped records.- Feature-store write latency and consistency (staleness).

Generate realistic traffic- Reuse production schemas and serialisation (Avro/Protobuf). Vary payload sizes following real percentiles; include occasional large records.- Key distribution: model partitioning/affinity—use realistic key skew (Zipf) and a uniform baseline to test hotspots.- Timing: replay real event traces if available. Otherwise synthesize bursts, steady-state, and diurnal patterns; include correlated events and TTLs.- Include feature computation complexity: light vs heavy transformations to emulate CPU-bound workloads.

Isolation strategy- Start with component-level tests: producer→Kafka, Kafka→processor, processor→feature-store.- Use canary clusters/namespaces and dedicated topics/partitions to avoid affecting prod.- Run end-to-end tests after component tuning.- Use resource limits and dedicated monitoring per component (Prometheus, JMX, p99 dashboards).

Failure modes to simulate- Broker failures and leader elections; slow ISR and partition unavailability.- Network partitions/high latency between components.- Consumer restarts, slow commits, and offset drift.- Backpressure: Kafka retention full, feature-store write throttling, DB connection pool exhaustion.- Resource exhaustion: CPU spikes, OOM, GC pauses.- Message corruption or schema evolution errors.

Validation & acceptance- Define SLOs (e.g., p99 < X ms, sustained throughput Y with <1% errors).- Run repeated tests (soak) for stability, and compare model-serving impact (latency/throughput) to ensure ML pipelines meet SLA.- Automate test harnesses and include observability dashboards & runbooks for each failure scenario.

HardSystem Design

37 practiced

Design a cross-region streaming replication solution that enables low-latency regional reads and supports global fault tolerance. Requirements: replication lag typically <5s, support regional read locality, and tolerate regional failures. Discuss active-passive vs active-active topologies, conflict resolution, metadata propagation, and tools such as MirrorMaker, Confluent Replicator, or custom replication.

Sample Answer

Requirements clarification:- Functional: global streams replicated across regions for low-latency reads (<5s typical lag), regional read locality, survive full-region outage.- Non-functional: high throughput (ML feature/event streams), strong availability, acceptable eventual consistency across regions.

High-level architecture:- Each region runs a local Kafka cluster (or Confluent Cloud regional cluster) that serves local producers/consumers (inference, feature store updates).- Cross-region replication layer replicates topics between clusters with at-most-once / exactly-once semantics depending on needs. Local reads hit local cluster for low latency.

Active-passive vs active-active:- Active-passive: one primary region accepts writes, others read-only replicas. Simpler conflict-free semantics and easier metadata consistency; replication tools (MirrorMaker/Replicator) suffice. Downside: primary regional failover increases write latency and single-write bottleneck.- Active-active (recommended for ML use-cases requiring low write latency globally): allow writes in any region and replicate bi-directionally. Requires conflict resolution and careful topic/key partitioning (see below). Better for global ingestion and low write latency.

Conflict resolution:- Prefer determinism: partition keys based on entity ID so same entity's events route to same partition across regions if possible. For truly concurrent updates, use last-write-wins with synchronized timestamps (NTP + logical clocks) OR vector clocks for stronger resolution. For critical model metadata, use a leader election (e.g., ZooKeeper/raft) per metadata namespace to avoid conflicts.- For feature stores: design idempotent update operations and include event version numbers; consumers apply higher-version wins.

Metadata propagation:- Replicate topic schemas (Schema Registry replication) and ACLs. Use schema registry multi-region replication (Confluent Schema Registry with active-active replication or replicate via REST). Propagate consumer offsets where needed using offset-sync utilities or by storing offsets in a replicated KV store (e.g., etcd/CockroachDB).

Tools and trade-offs:- MirrorMaker 2: open-source, good for simple uni/bi-directional replication, supports offset translation. Pros: lightweight; Cons: limited conflict resolution and enterprise features.- Confluent Replicator / Cluster Linking: enterprise-grade, handles GC/offsets, Schema Registry, Security; easier to operate for active-active with Confluent features.- Custom replication: build when you need application-aware merging (e.g., event transforms, deduplication, version-aware conflict resolution). Use Kafka Connect to transform events, and a distributed log-based reconciler (e.g., Kafka Streams) for merging.- Consider using a global log service (e.g., Kafka-on-Kubernetes with tiered storage or cloud-native global message buses) if operational overhead of custom replication is high.

Operational considerations:- Monitor replication lag, end-to-end latency, schema drift, and consumer liveness. Automate failover runbooks: advertise read endpoints via regional DNS/load balancer; on region failure, promote replicas for write if active-passive, or continue accepting writes in other regions if active-active.- Test partition rebalancing, network partitions, and clock skew scenarios.

Recommendation for ML role:- Use active-active with deterministic partitioning for entity-level writes, Confluent Cluster Linking (or MirrorMaker2 + custom conflict resolver) for replication, Schema Registry replication, and idempotent event design. This gives low-latency regional reads, tolerates regional failures, and keeps replication lag typically under 5s with properly provisioned network and async replication tuning.

MediumTechnical

31 practiced

Design a monitoring and tracing strategy to detect feature distribution drift upstream that could degrade model accuracy. Specify online and offline checks, metrics to compute (histograms, KL-divergence, KS-test), alert thresholds, and ideas for automated mitigation such as retrain triggers or feature freezes.

Sample Answer

Clarify goal & constraints: detect upstream feature distribution drift that may cause model accuracy degradation, fast enough to alert and (optionally) trigger automated mitigation without false alarms.

Online checks (real-time / streaming):- Per-feature running summaries over fixed sliding windows (e.g., 1h, 24h): count, mean, std, min/max, null-rate, quantile sketches (t-digest).- Compare current window to baseline (training or recent stable production) using: - KL-divergence on binned histograms for continuous features (use 50–200 bins, add smoothing). - Two-sample KS-test (p-value threshold) for continuous features. - Chi-squared / JS-divergence for categorical features; monitor top-k category mass and new/unseen categories. - Null-rate delta and cardinality changes for IDs.- Alert rules (tiered): - Warning: KL > 0.05 OR KS p < 0.05 OR top-category mass shift > 10% OR null-rate delta > 5%. - Critical: KL > 0.15 OR KS p < 0.01 OR top-category mass shift > 25% OR null-rate delta > 20%.- Rate-limit/aggregation: require persistent drift for N consecutive windows (e.g., 3) or sustained effect-size to reduce noise.

Offline checks (batch / daily):- Recompute histograms over larger windows (7/30 days); compute per-feature KL, JS, Wasserstein distance, KS, and population stability index (PSI).- Joint-distribution checks: monitor key feature pairs and model input embeddings using MMD or PCA-projection distance.- Label-shift proxy: track downstream model performance metrics (accuracy, AUC, calibration) and PSI between scored population and training labels when labels become available.

Practical considerations:- Minimum sample sizes: avoid tests with too few samples; flag low-sample windows separately.- Multiple testing: apply Bonferroni or FDR on many features; prioritize by feature importance (SHAP/weights).- Segmentation: compute metrics per critical cohort (geography, device, partner) to catch localized drift.- Baseline update: maintain rolling baseline (e.g., past 30 days) and long-term baseline (training) for comparisons.

Automated mitigation ideas:- Soft mitigation: - Feature freeze: temporarily stop using the drifting feature in scoring pipeline (zero/median impute or mask) for canary traffic; prefer if feature is known to cause volatility. - Canary routing: route X% of traffic to a model variant that ignores/drifts-handles the feature.- Retrain automation: - Trigger retrain when critical drift + observed downstream metric degradation OR when drift persists > T days; create automated pipeline to collect labeled data, retrain, validate on holdouts, and promote if pass metrics. - Shadow retrain: continuously train candidate models on recent data and evaluate in shadow to estimate improvement before promotion.- Rollback / human-in-loop: - If automated promotion fails tests, automatically rollback to last stable model and notify owners. - Require human sign-off for production changes if drift source is external (schema change, partner bug).- Root-cause and pattern detection: - Log feature distribution timeseries and top examples; attach example records to alerts. - Correlate drift events with upstream deployment, schema changes, partner IDs, or API failures.

Operational/alerting practices:- Attach feature importance weights and expected model sensitivity to every alert to prioritize.- Use three alert channels: dashboard (visualization + histograms), paged critical alerts, and email for warnings.- Capture full reproducible snapshot (baseline, current histograms, sample examples, model scores) in alert payload for fast triage.

Why this works:- Combining effect-size (KL/PSI/Wasserstein) with significance (KS p-values) and business-aware thresholds reduces false positives.- Tiered alerts + persistence requirement balance sensitivity and noise.- Automated soft mitigations and shadow retrains let you respond quickly while preserving safety through human oversight and rollback.

HardSystem Design

36 practiced

Design a global low-latency feature pipeline that supports both online inference and periodic training. Requirements: 100k TPS aggregate, end-to-end online feature freshness <50ms, multi-region availability (NA/EU/APAC), daily training dataset of ~1 PB, tolerable data loss <0.01%, feature versioning and rollback. Describe architecture, cross-region replication, caching strategies, consistency model, storage choices, and failure handling.

Sample Answer

Requirements clarification:- 100k TPS aggregate (read-dominant for online inference), end-to-end feature freshness <50ms, multi-region (NA/EU/APAC) active-active, daily training dataset ~1 PB, tolerable data loss <0.01%, feature versioning & rollback.

High-level architecture:- Client traffic -> Global load balancer (Anycast + DNS + traffic steering) -> Regional API Gateways -> Online Feature Store (low-latency read path) + Ingress stream for feature writes -> Event Buffer (regional Kafka/Rabbit with tiered storage) -> Offline Feature Lake for training -> Training cluster (distributed Spark or Flink + object storage).

Online feature path (freshness <50ms):- Write pipeline: producers -> regional ingress (gRPC) -> local write-ahead log (append-only) -> stream processor (Flink/Beam) that computes real-time aggregates and materializes features into an in-memory KV store (e.g., Redis/KeyDB or Aerospike) co-located in the region. End-to-end latency targeted by keeping materialization synchronous within region (<20ms) and replication async (<30ms budget).- Read path: model servers call regional feature cache (L1 in-memory), fallback to regional persistent store (L2 SSD-backed key-value like RocksDB via a service) if miss. L1 hit rate goal >99% to meet TPS.

Cross-region replication & consistency:- Active-active with per-entity primary region ownership for strong consistency on writes (hash/entity routing). For cross-region reads, use asynchronously replicated changelog from primary to replicas via compressing CDC (Kafka MirrorMaker / Confluent Replicator) with write sequence numbers and vector clocks.- Consistency model: hybrid — strong consistency for owner-region writes and monotonic-read/causal consistency for cross-region reads. For use cases needing strict global consistency, route to primary region or use synchronous quorum (rare, latency-penalty).

Storage choices:- Hot store: in-memory KV (Redis Cluster with persistence or Aerospike) for online serving.- Warm store: RocksDB-backed regional read-optimized store for cold misses.- Long-term/offline: object storage (S3 / GCS) + Parquet, and a tiered Delta Lake or Iceberg table for daily 1 PB dataset; metadata in Hive/Glue.- Changelog: Kafka with topic partitioning by entity key, replication factor across AZs and across regions (geo-replication).

Feature versioning & rollback:- Every feature materialization attaches a deterministic feature_id:version and schema. Materialize outputs into per-version namespaces (e.g., feature_name:vN:key -> value). Model metadata stores required versions; serving layer supports layered lookup and can atomically switch versions via metadata service. Rollback: flip model metadata to previous version; transient dual-writing to both versions for validation.

Caching strategies:- Multi-tier caching: L1 per-model in-memory cache (TTL very short, e.g., 50ms), L2 regional Redis cluster shard, L3 RocksDB. Use consistent hashing and hot-key handling (hot-key splitter, request coalescing) to avoid thundering herd. Adaptive TTLs and LFU eviction for feature popularity.

Training dataset generation:- Daily job consumes compacted Kafka changelogs + cold data from object store -> build feature tables in distributed compute (Spark) writing to Parquet/Delta with partition pruning. Use incremental joins via time-travel in Iceberg/Delta to produce 1 PB snapshot.

Failure handling & durability:- Write-ahead log persisted to local SSD and to Kafka (RF=3 across AZs) to meet <0.01% data loss. Cross-region replication provides disaster recovery; for full region loss, promote next region’s replica using changelog + last-offset reconciliation. Health checks and circuit breakers; backpressure applied to producers if stream lag exceeds thresholds. Automatic failover for regional caches via cluster manager + traffic steering.

Operational concerns:- Monitoring: SLOs (p99 latency <50ms, availability), observability on lag, cache hit rates, feature drift. Chaos testing (region failover drills).- Security: encryption in transit & at rest, IAM, audit logs for feature schema/version changes.- Trade-offs: prioritized low latency with per-entity primary-region ownership (sacrifices some global write flexibility) and async cross-region replication to meet latency & cost requirements.

This design provides sub-50ms freshness for regional reads, scales to 100k TPS by sharding and L1 cache, supports 1 PB daily training snapshots, offers versioned features and quick rollback, and keeps data loss well under 0.01% through durable logging and replication.

HardTechnical

35 practiced

Design an exactly-once end-to-end pipeline that consumes from Kafka, performs stateful transforms, and writes aggregated results to an external OLTP database. Explain how to use Kafka transactions, stream-processor checkpointing (e.g., Flink), two-phase commit, idempotent writes, and how to handle failure scenarios (processor restarts, partial commits). Discuss performance trade-offs.

Sample Answer

Requirements & guarantees:- End-to-end exactly-once from Kafka topic(s) → stateful stream transform → OLTP DB upserts/aggregates.- Tolerate processor restarts, partial failures, and at-least once delivery from Kafka clients; ensure no duplicate effects in DB.

High-level approach:- Use Kafka transactions for consuming + producing to Kafka (if intermediate topics used).- Use a stream processor with distributed checkpointing and transactional sink support (e.g., Flink with two-phase commit sink).- For OLTP DB writes, use either DB-native transactional two-phase commit (if supported) or idempotent upserts with monotonic write-ids to make writes replay-safe.

Components & flow:1. Source: Kafka consumer integrated with Flink’s Kafka connector that participates in checkpoint barriers.2. Stateful transform: Flink manages keyed state and snapshots state on checkpoint.3. Sink: Implement Flink TwoPhaseCommitSinkFunction: - In pre-commit phase (on checkpoint), the sink prepares a transaction (e.g., start DB tx and write to staging table or persist pending batch keyed by checkpoint-id). - On successful checkpoint completion, Flink’s job manager notifies tasks to commit the prepared transaction (finalize and make visible). - If DB supports XA/2PC, use it to coordinate commit; otherwise, implement idempotent finalization (see below).4. Idempotency pattern for DBs without XA: - Each write carries a unique transaction id composed of (job-id, operator-id, checkpoint-id). - Writes are upserts into a table keyed by business key + tx_id or use an idempotent compare-and-set based on last_applied_tx_id per key. - Commit = mark tx_id as committed; consumer of aggregated table only reads committed tx_ids.

Failure scenarios & handling:- Processor restart before checkpoint commit: Flink will restore from last successful checkpoint, reprocess messages from Kafka offset stored in checkpoint; because sink commit happens only on checkpoint completion, the previously uncommitted prepared writes are either rolled back by timeout or ignored (not visible). Reprocessing will replay the same writes with same tx_id; idempotency prevents duplicates.- Partial commit (prepare succeeded, commit failed due to DB crash): On restart, coordinator checks prepared-but-not-committed txs (via DB metadata). Either a recovery process completes commit or rolls back depending on consistency rules. If using idempotent finalization, re-applying commit is safe.- Kafka producer transactional failure: Use Kafka transactions only for Kafka sink paths; Flink’s KafkaSink integrates exactly-once by tying Kafka transaction lifecycle to checkpoints. If a Kafka transaction fails, it’s aborted and offsets not advanced; Flink will retry from last checkpoint.- Network partitions/timeouts: Ensure long-lived prepared transactions have TTL and have a recovery controller process to reconcile prepared transactions.

Why checkpoint + two-phase commit:- Checkpoint gives consistent view of stream offsets + operator state.- Two-phase commit ties external side-effects to that checkpoint: prepare during snapshot, commit on checkpoint complete. This achieves atomic visibility: either state + DB effects are both visible (commit) or neither (abort + restore).

Performance trade-offs:- Latency: Two-phase commit and transactional writes add latency—prepare/commit handshake and DB flushes. Using batching reduces overhead but increases end-to-end latency.- Throughput: Strong consistency (XA/2PC) limits parallelism and increases DB load. Idempotent upserts with application-level dedupe can be faster but requires more DB schema/logic and storage for tx metadata.- Complexity: Implementing idempotency and recovery logic increases system complexity but avoids XA shortcomings (poor performance, limited DB support).- Scalability: For high write scale, prefer sharded/partitioned DBs or an intermediate write-optimized store (e.g., transactional Kafka topic + CDC sink to OLTP) to offload coordination.- Operational: Need monitoring for long-prepared transactions, checkpoint backlog, and DB transaction GC.

Best practices:- Keep checkpoint interval tuned: too frequent increases commit overhead; too infrequent increases reprocessing window.- Use small idempotent writes and batched commits.- Add a reconciliation job that scans DB and Kafka committed tx_ids to repair stuck transactions.- Test failure modes (task kill, DB outage, network partition) in chaos tests.

This design gives end-to-end exactly-once by coupling Flink checkpoints with transactional/ idempotent external writes; choose XA when DB supports and scale is moderate, otherwise implement robust idempotency and reconciliation for better performance.

Unlock Full Question Bank

Get access to hundreds of Data Pipeline Scalability and Performance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.