Distributed Systems Fundamentals Questions

Core principles and theory that underlie distributed computing systems. Includes understanding trade offs between consistency, availability, and partition tolerance, common consistency models such as eventual and strong consistency, replication and sharding strategies, load balancing and data partitioning, consensus algorithms and their guarantees, scalability and fault tolerance patterns, and how these concepts apply to infrastructure components such as databases, caches, service meshes, and load balancers. Candidates are expected to explain design choices, common failure modes, and how fundamental concepts influence architecture decisions for resilient and scalable systems.

HardTechnical

65 practiced

Design a streaming pipeline (using tools like Kafka Streams or Flink) to compute online features with exactly-once semantics and low latency. Explain state management, checkpointing, windowing for aggregations, and how to expose these features to low-latency inference endpoints.

Sample Answer

Requirements and constraints:- Exactly-once semantics for feature computations- Low-latency (<50–200ms) reads for inference- Support event-time aggregations (windows), late data, and incremental updates- Scalable, fault-tolerant, observable

High-level approach:- Use Flink (recommended) or Kafka Streams for stateful stream processing; Kafka as durable ingestion and changelog.- Compute features in streaming jobs and publish materialized feature values to an online store (Redis/KeyDB/Cassandra) for low-latency reads; keep feature change log in Kafka for recovery and audit.

State management & exactly-once:- Flink with RocksDB state backend to hold large keyed state (per-entity aggregates). RocksDB provides fast local disk-backed state.- Enable Flink checkpointing to durable storage (S3/GCS) with a reasonable interval (e.g., 5–30s) and configure timeout/retention.- Use Flink’s two-phase commit sink or Kafka transactions to write to both Kafka changelog topics and the online store with exactly-once semantics: - Preferred pattern: write feature updates to a Kafka “feature-changelog” topic within Flink’s checkpointed two-phase commit; have a separate idempotent consumer that materializes to Redis, or use Flink’s transactional sink to atomically write to Kafka and a staging buffer then commit on checkpoint.- Configure checkpointing and Kafka producer/consumer transactional IDs to avoid duplicates. Ensure operator state is checkpointed and restored deterministically.

Windowing and aggregations:- Use event-time processing with watermarks to compute windowed features (tumbling, sliding, session): - Example: 1h sliding window with 5min slide for rolling averages.- Implement user-defined process functions for complex features (e.g., percentile via digest sketches).- Handle late/out-of-order events via allowed lateness; decide whether to update materialized features on late arrivals (retract/update patterns).- For incremental aggregates use incremental algorithms: count/sum/avg via incremental state; quantiles via t-digest or KLL sketches; heavy hitters via CMSketch.

Exposing features for low-latency inference:- Materialize features to an online key-value store (Redis) keyed by entity id. Writes come from the feature-changelog topic (consumer that applies updates idempotently).- Ensure read-after-write consistency: pipeline writes a version or timestamp with each feature; inference can either wait for checkpoint-confirmed updates or read latest value and accept eventual consistency if tolerable.- Provide a service layer (gRPC/HTTP) that: - Fetches features from Redis with sub-ms lookups - Falls back to a “feature fetch” from a cold store (Cassandra) if missing - Supports bulk multi-entity fetches and caching- For ultra-low-latency model serving, colocate model server and Redis in same AZ; use connection pooling and pipelining.

Operational considerations:- Monitoring: track checkpoint latency/success, Kafka offsets, state size, RocksDB compaction metrics, sink latencies, feature drift.- Testing: end-to-end reproducible tests using recorded input topics and deterministic state restore; fuzz with out-of-order/late events.- Schema/versioning: evolve feature schemas via Avro/Protobuf with compatibility; include metadata (timestamp, version, TTL).- TTL & compaction: set TTL for online store and retention/compaction policy for changelog topics.

Example snippet (Flink pseudo):

java

// Key by entity, event-time with watermarks
stream
  .assignTimestampsAndWatermarks(...)
  .keyBy(event -> event.id)
  .window(SlidingEventTimeWindows.of(Time.hours(1), Time.minutes(5)))
  .aggregate(new IncrementalAgg(), new WindowProcess())
  .map(feature -> toKafkaProducerRecord(feature)) // transactional sink
  .sink(new FlinkKafkaProducer<>(... , flinkKafkaSemantic.EXACTLY_ONCE));

Trade-offs:- Exactly-once to external DB is hard; best practice is exactly-once to Kafka changelog and idempotent/transactional materializer to online store.- Shorter checkpoint intervals reduce data loss but increase overhead; tune with SLOs.

This design gives robust exactly-once feature computation, supports complex windowed aggregates, and serves features with low latency suitable for production inference.

MediumTechnical

77 practiced

Compare parameter server and AllReduce approaches for gradient synchronization in distributed training. For a sparse, high-cardinality recommendation model, which approach is preferable and why? Discuss network bandwidth and staleness implications.

Sample Answer

Short answer: For sparse, high-cardinality recommendation models a parameter-server (PS) architecture is generally preferable to AllReduce because PS supports sparse, partial parameter updates and selective network transfer, dramatically reducing bandwidth and memory pressure; AllReduce forces full dense synchronization and incurs higher bandwidth and wasted communication for sparse gradients.

Comparison:- Communication pattern: - AllReduce: collective, each worker exchanges and aggregates full gradient tensors (even if mostly zeros) via ring or tree algorithms. Efficient for dense tensors and GPUs with RDMA. - Parameter server: workers push only the non-zero indexed gradients (e.g., embedding rows) to servers; servers aggregate and apply updates, workers pull required parameters.- Bandwidth: - Sparse recommendation models have extremely high embedding-cardinality but each batch touches a tiny subset. PS sends only touched rows → much lower bandwidth and network I/O. - AllReduce would transmit full embedding shards or dense gradients, causing orders-of-magnitude higher traffic and potential network bottlenecks.- Staleness: - PS can be configured synchronous (lockstep) or asynchronous (ASGD). ASGD reduces latency and improves throughput at cost of parameter staleness; this can be acceptable for recommendation embeddings where SGD noise is tolerable. Synchronous PS avoids staleness but increases wait time. - AllReduce is typically synchronous, giving consistent, non-stale gradients — beneficial for convergence but expensive here.Recommendation and trade-offs:- Prefer a sharded parameter-server for embeddings with sparse push/pull and possibly bounded staleness (e.g., stale-synchronous parallel, SSP) to balance throughput and convergence. Use techniques: compression (quantization, sparsification), caching hot embeddings on workers, colocating hot-partition servers, and rate-limiting updates.- If your cluster has extremely fast RDMA interconnect and embeddings are small/dense or workload becomes dense, AllReduce may be viable. Otherwise PS yields better network efficiency and practical scalability for high-cardinality sparse recommendation models.

HardSystem Design

83 practiced

Design a global cache invalidation strategy for feature caches used by inference, where feature updates must propagate within a bounded staleness window across regions. Explain invalidation messages, TTLs, and trade-offs between push-based invalidation and conservative TTLs.

Sample Answer

Requirements and constraints:- Functional: feature cache entries (per-user/per-entity) used by inference must reflect upstream updates within bounded staleness S (e.g., 5s–2min).- Non‑functional: global multi-region deployment, low latency for inference, high availability, tolerable network partitions.- Scale: millions of keys, high write and read QPS.

High-level architecture:- Regional inference services with local feature cache (in-memory + optional local Redis).- Global change-log (append-only) of feature updates (e.g., Kafka/CDC stream) and a lightweight Invalidation Service per region subscribed to the stream.- Optional global coordination (consistent timestamp authority or logical clocks).

Invalidation messages:- Payload: {key, feature_version, update_ts, tombstone_flag, optional payload_hash}- Sent via message bus to regional Invalidation Services. Use batching for throughput; include sequence offsets for resumability.- Regions apply message if local cache version < feature_version. Support idempotent apply.

TTL strategy:- Conservative TTLs: set TTL ~= S to bound staleness without push. Simple, no infra cost, but increases load on origin and read latency after expiration.- Push-based invalidation: messages proactively evict/update entries to near-zero staleness. Requires reliable delivery and backpressure handling.

Hybrid approach (recommended):- Use push invalidations for hot keys and critical features; use conservative TTL for cold keys.- Maintain per-key metadata: last_update_ts, hot_flag. Hot promotion via read/write frequency or explicit marking.- On push failure or long partition, TTL provides safety net: TTL <= S ensures bounded staleness.

Delivery guarantees & trade-offs:- At-least-once messaging: simplest; idempotency required. Duplicate invalidations are fine.- Exactly-once is expensive globally; avoid unless strictly necessary.- Ordering: per-key ordering is important—use key-partitioned streams so updates per key are ordered.

Handling partitions and latency:- If region disconnected, allow reads from local cache with note of increased staleness; fall back to origin on cache-miss or if feature critical.- Use monotonic versioning/timestamps to avoid regressions.

Operational considerations:- Metrics: invalidation lag distribution, stale-read rate, message delivery success, per-key hotness.- Backpressure: Invalidation Service should throttle/shed non-critical invalidations; rely on TTL instead.- Testing: chaos tests (packet loss, consumer restart) and spike tests.

Summary of trade-offs:- Push minimizes staleness and origin load but adds delivery complexity and operational cost.- Conservative TTL is simple and robust but increases origin load and worst-case staleness equals TTL.- Hybrid (push for hot/critical, TTL for others) delivers best balance for global ML inference where most predictions depend on low-latency, fresh features.

HardTechnical

68 practiced

Explain how Raft implements leader election and log replication. In an unreliable network, what conditions can cause split-brain or stale leaders, and how would you detect and remediate them in a metadata service used by ML pipelines?

Sample Answer

Raft implements leader election and log replication to provide a single-source-of-truth consensus. Leader election: each node starts as follower; if it doesn't hear a heartbeat from a leader within a randomized election timeout, it becomes a candidate, increments its term, and requests votes from peers. A candidate becomes leader when it collects a majority of votes for the same term. Votes are granted only once per term and only to candidates with up-to-date logs (lastLogTerm/lastLogIndex check), preventing split decisions when logs diverge.

Log replication: the leader accepts client writes, appends entries to its log, and sends AppendEntries RPCs (heartbeats) to followers. Followers append entries and reply. An entry is committed once it’s stored on a majority; the leader then applies committed entries to state machine and informs followers of commitIndex during subsequent AppendEntries.

Unreliable networks can cause split-brain or stale leaders when:- Network partitions isolate a majority from the current leader, allowing the minority containing a candidate to form a new majority and elect a new leader (partition heal can reveal conflicting leaders).- Message loss or long delays cause followers to time out and elect another leader while the original still believes it’s leader (stale leader).- Clock drift coupled with poorly chosen timeouts increases simultaneous candidacies.

Detection in a metadata service for ML pipelines:- Monitor term/leader id changes, election frequency, and commitIndex gaps. Sudden rapid elections, divergent commitIndexes across nodes, or clients receiving conflicting leader addresses indicate issues.- Track RPC latencies, dropped heartbeats, and partitioned peer sets via health checks and network-level telemetry.

Remediation strategies:- Tune randomized election timeouts relative to observed network latencies (longer timeouts reduce split elections).- Enforce majority-write/read policies for metadata updates; ensure clients use the cluster’s current leader or use leader-aware client libraries with retries and redirects.- Add automated fencing: when a new leader is elected, include term-based fencing tokens in metadata write operations so a stale leader’s writes get rejected by followers (followers check term before accepting).- Implement alerting and automated failover: if a node shows divergent logs or abnormal elections, isolate and perform rolling recoveries, or snapshot-and-restart followers to resync from leader.- For ML pipelines specifically, make metadata mutations idempotent, store operation logs to reconcile inconsistent client views, and add consistency checks before critical actions (e.g., model promotion) so stale reads don’t trigger irreversible steps.

These combined detection and remediation steps keep metadata correctness high while minimizing ML pipeline disruption.

EasyTechnical

76 practiced

List common load-balancing algorithms (round-robin, least-connections, consistent hashing, weighted) and briefly describe when each is a good fit for ML model inference services. Include considerations for sticky sessions and GPU-backed pods.

Sample Answer

Round-robin- What: Rotate requests evenly across healthy instances.- When good for ML inference: Simple, low-overhead; works well when all replicas are identical in capacity and inference latency is stable (e.g., CPU-based micro models).- Trade-offs: Ignores per-instance load/latency; can overload slower pods.

Least-connections- What: Send to the pod with the fewest active connections/requests.- When good: Useful when inference time varies (variable batch sizes or models with unpredictable latency). Helps avoid queuing on busy pods.- Trade-offs: Requires accurate connection tracking; may not reflect GPU memory or throughput constraints.

Weighted (round-robin or least-connections with weights)- What: Assign weights proportional to capacity (vCPU, GPU count, throughput).- When good: Heterogeneous clusters (mixed GPU/CPU pods, different model sizes). Route more traffic to higher-capacity nodes.- Trade-offs: Need to compute/adjust weights as load or pod resources change.

Consistent hashing- What: Hash request key to a stable backend set.- When good: Useful if you want request affinity to cached model artifacts or per-user state (e.g., session-level caches, warmed model contexts). Minimizes cache misses when pods hold large warmed caches.- Trade-offs: Uneven distribution if key skewed; less effective for pure stateless inference.

Sticky sessions / session affinity- Use when per-session state ( warmed model, cached features, GPU-resident context) improves latency/throughput.- Prefer short affinity TTLs and combine with consistent hashing or cookie-based affinity. Beware of reduced overall balancing and reduced fault tolerance if many sessions stick to a few pods.

GPU-backed pods considerations- Schedule-aware routing: Prefer weight-based routing that accounts for GPU count, memory, and current GPU utilization (use metrics like nvidia-smi).- Avoid sending many concurrent small requests to a single GPU pod—batching on GPU is efficient; use a gateway that supports batching decisions or route by capacity.- Fast failover: If a GPU pod crashes, ensure affinity TTLs expire quickly and health checks detect GPU OOMs.- Consider autoscaling by GPU metrics and use node selectors/taints to keep GPU workloads on appropriate nodes.

Practical recipe- Default: weighted round-robin for heterogeneous ML clusters.- If latency varies: least-connections + weights.- If caching/warmth matters: consistent hashing or short sticky sessions.- Always combine with health checks, autoscaling based on GPU utilization/queue length, and observability (latency, GPU metrics) to adapt balancing strategy.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.