Data Pipeline Scalability and Performance Questions

Design data pipelines that meet throughput and latency targets at large scale. Topics include capacity planning, partitioning and sharding strategies, parallelism and concurrency, batching and windowing trade offs, network and I O bottlenecks, replication and load balancing, resource isolation, autoscaling patterns, and techniques for maintaining performance as data volume grows by orders of magnitude. Include approaches for benchmarking, backpressure management, cost versus performance trade offs, and strategies to avoid hot spots.

MediumTechnical

32 practiced

Explain how to implement end-to-end exactly-once semantics for a pipeline Kafka → Flink → transactional PostgreSQL sink. Outline how Kafka transactions, Flink checkpointing/two-phase commit sink, and idempotent writes in the DB interact. Enumerate failure scenarios (task crash, network blips, coordinator failure) and how this architecture addresses each.

Sample Answer

Start by clarifying goal: end-to-end exactly-once from Kafka producer to Postgres rows — each input record applied exactly once in DB even with failures.

Architecture / components- Kafka: produce messages with transactional producers (optional). Topics with compacting partitions; offsets committed only when Flink confirms sinks.- Flink: consume with KafkaSource using "read_committed" semantics and enable checkpointing. Use Flink’s two-phase-commit (2PC) sink pattern (CheckpointedFunction + TwoPhaseCommitSinkFunction).- Postgres sink: implement transactional writes per Flink transaction (BEGIN … COMMIT). Prefer use of transactional tables or a changelog table plus idempotent upserts (INSERT ... ON CONFLICT) keyed by record-id. Optionally use Postgres two-phase commit (PREPARE TRANSACTION) if needed.

How they interact- On each Flink checkpoint, Flink’s 2PC sink starts a DB transaction (prepare in-memory state), writes rows but does not commit. Flink stores transaction metadata in checkpoint snapshot. Only after checkpoint completes (barrier reached across sources/operators) Flink’s JobManager triggers commit phase — sink commits the DB transaction. Kafka consumer offsets are committed as part of the checkpoint coordination (Flink writes offsets into checkpoint state), so offsets advance only when DB commit is durable.- If you use Kafka transactions end-to-end, Flink can be configured to participate (producer side) to atomically write to Kafka and other sinks; typically Flink manages Kafka offsets through checkpoints rather than Kafka transactions.

Failure scenarios- Task (sink) crash before checkpoint completion: checkpoint not completed => Flink aborts the in-flight DB transaction (no commit). On recovery, Flink restores last successful checkpoint, reopens a fresh DB transaction and replays events. Idempotent upserts or transactions avoid duplicates.- Task crash after checkpoint but before commit: Flink ensures checkpoint marked completed only when all barriers arrived. If commit hasn't executed and checkpoint claimed complete, JobManager will drive commit on recovered task using stored transaction metadata. If commit already in DB but crash prevented offset advancement elsewhere, offsets stored in checkpoint keep consistency.- Network blip during commit: if commit RPC times out, the sink must be able to probe DB for transaction state (committed/aborted) and retry safely. Using DB-side prepared transactions or idempotent upserts helps detect duplicate commits.- JobManager/coordinator failure: Flink’s checkpoint store (e.g., durable filesystem) and high-availability JobManager election ensure the new leader reads last completed checkpoint and replays from there, committing or aborting outstanding transactions per checkpoint metadata.- Kafka broker failure: Flink’s Kafka consumer with committed offsets in checkpoint will resume from stable offsets. If using Kafka transactional producers, ensure transaction timeouts and fencing are handled; Flink’s built-in Kafka sink integrates with checkpoints to avoid partial publishes.

Important patterns and trade-offs- Use DB transactions + Flink 2PC to tie commit to checkpoint; this gives exactly-once without external dedupe.- Idempotent writes (logical upserts with unique keys and versioning) are simpler and more robust if DB doesn’t support prepare/commit coordination.- Two-phase DB prepare (PREPARE TRANSACTION) offers stronger guarantees but increases operational complexity (prepared transactions left open on failures).- Keep checkpoints frequent enough to bound replay volume; tune transaction commit latency vs throughput.- Monitor for stuck prepared transactions and provide automated reconciliation.

Conclusion: combine Flink checkpoint-coordinated 2PC sink (or idempotent upserts) with Kafka consumption tied to checkpoints so offsets advance only after DB commit. This ensures each input is applied exactly once across crashes, network blips, and coordinator failures when implemented with proper transactional handling and recovery probes.

HardSystem Design

49 practiced

Design a scalable enforcement mechanism for per-tenant throughput quotas (for example, 1,000 events/sec) across a distributed global streaming platform that accepts ingress via HTTP endpoints, Kafka producers, and mobile SDKs. Discuss enforcement points (edge vs broker), token-bucket or leaky-bucket implementations, distributed counters and sharding of quota state, consistency windows, latency implications, and failure modes (stale counters, clock skew).

Sample Answer

Requirements (clarify):- Per-tenant global throughput quota (e.g., 1,000 events/s) across HTTP, Kafka, SDKs- Low ingress latency, high availability, global scale, fair enforcement, eventual consistency acceptable within bounded window

High-level architecture:Edge layer (regional POPs) → Global quota service (sharded) + local cache → Ingress brokers/routers → Ingest pipeline (Kafka/stream processors)

Enforcement points & rationale:- Primary enforcement at edge/regional POPs to minimize wasted traffic/latency and protect backend. Edges perform fast checks against local quota state.- Global quota service maintains authoritative counters and coordination; used for replenishment, reconciliation, and occasional hard-fail decisions.- Brokers also validate as a fallback (defense-in-depth) to catch misbehaving clients or edge failures.

Quota algorithm:- Token-bucket per tenant (tokens replenish at rate R = quota). Token-bucket supports bursts (useful for short spikes); leaky-bucket equivalent if strict smoothing desired.- Implement token-bucket both locally (fast, approximate) and globally (authoritative).

Sharding and distributed counters:- Partition tenants by consistent hash into quota shards (by tenant ID) hosted in a distributed, strongly-available store (e.g., Cassandra with lightweight transactions or Redis Cluster with CRDTs for counters).- Each shard maintains global token balance and replenishment state.- Edges hold a leased local token-bucket per tenant with lease TTL (e.g., 1s–5s) and a max local allowance proportional to regional traffic fraction to bound overshoot.

Consistency windows & trade-offs:- Local caches allow O(1) checks with eventual reconciliation — introduces bounded over-issue (tokens can be double-spent across edges during lease). Set lease TTL and local allowance so worst-case aggregate overshoot ≤ acceptable burst (e.g., 5–10%).- For strict quotas, reduce local allowance and increase sync frequency; higher coordination increases latency and load.

Sync protocol:- Edges request batch token grants from global quota service (atomic decrement per grant). Use optimistic batched requests to reduce round-trips.- Global service reconciles usage periodically (heartbeats) and applies corrective throttling if overshoot detected.

Latency implications:- Local checks: sub-ms–ms. Batched grant RPCs: tens–hundreds ms depending on WAN—use async replenishment to avoid blocking critical path.- For HTTP/SDK: check local bucket synchronously. For high-throughput Kafka producers: broker-level enforcement and token acquisition performed in pipeline with async replenish to avoid per-message RPC.

Failure modes & mitigations:- Stale counters (edge loses connectivity): local bucket continues serving until lease TTL; when TTL expires, edge stops granting tokens or falls back to restrictive mode. Use fail-open only if acceptable risk; prefer fail-closed for quota enforcement.- Clock skew: use logical time for token refill (use monotonic counters and server-driven replenishment); avoid TTL based on client clock.- Global service partition: shards can be replicated; on split-brain, prefer conservative behavior (reduce local allowance) and reconcile via write-ahead logs.- Burst abuse: limit per-connection or per-IP caps at edge in addition to tenant quota.- Consistency bugs: periodic global reconciliation job computes actual usage and issues credits/debits; alert and throttle if variance exceeds threshold.

Operational considerations:- Metrics: per-tenant accepted/rejected counts, overshoot, latency, grant success.- Config: per-tenant burst allowance, local lease TTL, shard placement strategy.- Testing: chaos tests for network partitions, clock drift, edge restarts.

Summary:Combine low-latency local token-buckets at edges with authoritative, sharded global counters and batched grants to balance latency and strictness. Tune lease TLLs and local allowances to control overshoot; prefer conservative defaults and strong observability to detect and remediate failure modes.

MediumTechnical

34 practiced

Given the schema below, propose a data lake partitioning and file layout optimized for common analyst queries that filter by date and region. Include recommended partition columns, file-size targets, and metadata/catalog strategy.

Schema:

events( event_id string, user_id string, event_time timestamp, event_type string, region string, properties map<string,string>)

Explain how your choices support both freshness and efficient historical scans.

Sample Answer

Recommended partitioning and file layout:

Partition columns- dt = DATE(event_time) (primary partition) — supports the most common time-based filters and efficient partition pruning.- region (secondary partition, low-to-medium cardinality) — allows queries that narrow by region without scanning unrelated regions.- Optional: event_type only if analysts frequently filter by it and cardinality is low; otherwise keep as column for predicate pushdown.

Example HDFS/S3 path layout- s3://bucket/events/dt=2025-12-05/region=EMEA/part-00001.parquet

File format & organization- Parquet with Snappy compression (columnar, good encoding/stats).- Sort rows within each file by event_time (ascending) and user_id to improve range scans and pushdown performance.- Cluster/compact periodically into larger files (see below).

File-size targets & write strategy- Target final Parquet file size: 128–512 MB (sweet spot for most query engines—enough I/O throughput without huge GC/memory pressure). Aim ~256 MB.- Ingest writes: allow small, frequent files (micro-batches) into a “hot” prefix (e.g., dt=2025-12-05/_hot/). Run scheduled compaction jobs (hourly/daily) to create target-size files into the final partition path. This preserves freshness while enabling efficient historical scans.

Metadata & catalog strategy- Use a table format with strong metadata: Apache Iceberg or Delta Lake (preferred) for ACID, schema evolution, partition metadata, and fast time-travel; fallback: Glue/Hive metastore with partition indexes.- Maintain partition-level statistics (min/max, null counts, distinct counts) in the catalog. Enable query engines to use stats for predicate pushdown.- For cloud: AWS Glue Data Catalog + Iceberg on S3, or Delta + Databricks metastore.- Use partition projection (where supported) to avoid listing millions of partitions and speed planning.

How this supports freshness vs historical scans- Freshness: Ingest to hot/small-file area to make data queryable quickly; use metadata (Iceberg/Delta commit) to atomically expose new files. Compaction runs asynchronously so writes are fast.- Historical scans: Compacted, sorted, and sized files allow scan-efficient sequential reads and better vectorized decompression; partition pruning by dt and region eliminates unrelated data early. Table metadata (manifests/statistics) prevents unnecessary file reads.- Operational: Schedule compaction frequency based on arrival rate (e.g., hourly for high-volume, nightly for low-volume). Retain raw immutable copy if reprocessing needed.

Additional optimizations- Keep properties map as MAP or JSON column but consider extracting commonly-filtered keys into their own columns for indexing/predicate pushdown.- Use dictionary encoding and zone maps (min/max) on low-cardinality columns (region, event_type).- Monitor small-file count and partition cardinality; consider bucketing or Z-ordering within partitions for heavy user_id-based lookups.

This design balances quick visibility of new events and efficient, large-scale historical analytics by combining partition pruning, compacted file sizes, and metadata-driven planning.

HardSystem Design

34 practiced

Architect a data ingestion and processing platform for a global ad-tech client that receives 50M events/sec peak, requires sub-second tail latency for bidding decisions, and must support both stateful stream processing for real-time features and offline storage for training. Provide a detailed component architecture (ingest tiers, transport, real-time engine, cold storage), partitioning and shard mitigation strategies, state management approach, replication model, autoscaling approach, and operational considerations like SLOs and runbooks. Explain major trade-offs and cost drivers.

Sample Answer

Requirements & constraints:- 50M events/sec peak globally; sub-second tail (<500ms p99/p999) for bidding decisions; stateful per-entity feature lookups & updates; offline durable storage for ML training; multi-region for geo-locality & DR; strong operational SLAs.

High-level architecture:1. Edge / Ingest Tier - Regional HTTP/gRPC collectors behind L4 load balancers, batching & lightweight validation. - Fast path: UDP/TCP multiplexers for low-latency clients with TLS termination.

2. Transport - Regional Kafka-like streaming (e.g., Confluent Kafka / Redpanda) with topic per logical feed and partitioning by buyer/auction key. - Use tiered brokers: hot partitions pinned to smaller set of brokers, cold partitions on cheaper storage.

3. Real-time Engine - Stateful stream processors (e.g., Flink/Materialize) deployed regionally for feature computation and enrichment. - API cache (in-memory KV: Aerospike / Redis AE) colocated with bidding systems for sub-ms lookups. Feature updates are written to stream and applied to state in memory.

4. Cold Storage / Offline - Append-only write path to object storage (S3/MinIO) in Parquet with time-partitioning and compaction. Daily/hourly materialized features exported for model training.

Partitioning & shard mitigation:- Hash by entity (advertiser/auction id) to guarantee state locality. Use consistent hashing with virtual nodes for even distribution.- Hot-key mitigation: detect skew and apply strategies: dynamic re-sharding, sample-and-broadcast for read-heavy keys, or split hot key by secondary dimensions with aggregation.- Backpressure path: if a partition lags, route non-critical telemetry to slower path and prioritize bidding events.

State management:- Large embedded state in Flink RocksDB with changelog streams for durability. Periodic snapshots to blob storage.- For ultra-low-latency lookups, materialize a subset of features into in-memory cache (LRU/size bounded), warmed by async CDC from stream state.

Replication model:- Cross-region replication of Kafka topics (active-active or active-passive depending on regulatory constraints) with at-least-once semantics.- State & cache: primary in region; async geo-replication of changelogs to DR region to rebuild state quickly. Use quorum writes for critical metadata.

Autoscaling:- Autoscale collectors and stateless components by CPU/requests.- Stateful processors scale via partition reassignment; maintain capacity headroom and use predictive scaling from traffic patterns. Use graceful rebalance windows and use standby task slots for quick failover.

Operational considerations:- SLOs: p99 read latency <5ms for cache, p99 end-to-end <500ms, availability 99.95%.- Monitoring: per-partition lag, GC, CPU, I/O, p99/p999 latencies, hot-key detection, tail metrics.- Runbooks: step-by-step for partition lag, broker failure, state corruption, cache eviction storms, auto-rebalance rollback.- Testing: chaos engineering, canary deployments, scale tests at 2x peak.

Trade-offs & cost drivers:- In-memory caches and high-throughput brokers are largest cost drivers (RAM, NVMe SSD). Choosing RocksDB + S3 reduces memory vs fully in-RAM but increases tail lookup latency.- Active-active multi-region increases complexity and cost but improves latency & availability.- Strict exactly-once semantics increase complexity and throughput overhead; often use idempotent/at-least-once with deduplication for cost/latency balance.

Why this design:- Combines durable, partitioned streaming for throughput with fast in-memory materialization for sub-second decisions, handles skew, supports ML training via compacted cold storage, and provides operational controls for safety and predictable autoscaling.

HardTechnical

31 practiced

A production Kafka cluster exhibits a hot partition caused by a tenant producing very high write amplification on a specific key range. Draft an operational runbook to detect, mitigate, and resolve the hot partition: include detection metrics (per-partition throughput, produce latency), immediate mitigations (rate-limit tenant, redirect writes, spin up topic replicas), medium-term fixes (salting, tenant-dedicated topics), and verification steps to confirm resolution without cluster-wide restarts.

Sample Answer

Runbook: Detect, mitigate, and resolve a Kafka hot-partition caused by a tenant’s high write amplification

1) Goals & constraints- Restore cluster health and consumer/producer SLAs without rolling restarts.- Minimize impact to other tenants; preserve data durability.

2) Detection (automated alerts)- Per-partition produce throughput (bytes/sec, msgs/sec) — threshold: >5x partition median for >1m.- Per-partition produce latency (p99, p50) and request queue length on leaders.- Partition leader CPU, network IO, and disk write stalls.- Broker-level replica lag and ISR changes.- Tenant-level client_id/key-pattern telemetry and topic key-distribution heatmap.

3) Immediate mitigation (within minutes)- Identify offending tenant keys and target partition(s).- Soft rate-limit producer traffic for that tenant via broker-side quotas or ingress gateway (throttle to safe rate).- Apply a targeted reject/429 policy at proxy if quota enforcement available.- Temporarily reassign partition leader to a less-loaded broker (use ControlledShutdown/PreferredReplicaElection carefully) — avoid full rebalance.- If supported, use per-tenant topic routing to redirect new writes to alternate topic (see redirect below).

4) Short-term actions (hours)- Create a mirrored topic with more partitions or dedicated topic for tenant; use producer-side switch to new topic (coordinate with tenant).- Increase replication factor for the hot partition to split leadership load? (No: replication increases IO — prefer partitioning/redirect.)- For immediate write-safety, enable log.segment.bytes smaller to allow faster flushes only if disk pressure low.

5) Medium-term fixes (days–weeks)- Key salting / hash-prefixing: change producer to add N-bit salt to keys to spread across partitions.- Tenant-dedicated topics or dedicated Kafka tenant cluster (for persistent high load).- Enforce per-tenant quotas and admission control at ingress/proxy layer.- Architect producer idempotence and batching changes to reduce amplification.- Automate detection + per-tenant dashboards and runbook playbooks.

6) Execution checklist & roles- On-call SRE: run detection queries, apply quotas, perform leader move with kafka-reassign-partitions tool.- Solutions Architect/Dev owner: coordinate tenant communication and producer changes.- Safety: snapshot controller logs, increase monitoring retention for postmortem.

7) Verification (no cluster restart)- Confirm per-partition throughput and p99 produce latency return to baseline for >30m.- ISR stable, replica lag normalized, no sustained controller rebalances.- Tenant-side success: downstream consumers see expected message flow (if redirected, validate consumer reconfiguration).- Run synthetic produce load matching tenant baseline to ensure even partition distribution after salting/dedicated topic.- Postmortem: root cause, quota thresholds adjustment, publish change plan.

Trade-offs- Rate-limiting minimizes service but impacts tenant SLAs.- Repartitioning requires producer changes — higher effort but sustainable.- Dedicated clusters cost more but provide isolation for noisy tenants.

Unlock Full Question Bank

Get access to hundreds of Data Pipeline Scalability and Performance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.