Distributed Systems Fundamentals Questions

Core principles and theory that underlie distributed computing systems. Includes understanding trade offs between consistency, availability, and partition tolerance, common consistency models such as eventual and strong consistency, replication and sharding strategies, load balancing and data partitioning, consensus algorithms and their guarantees, scalability and fault tolerance patterns, and how these concepts apply to infrastructure components such as databases, caches, service meshes, and load balancers. Candidates are expected to explain design choices, common failure modes, and how fundamental concepts influence architecture decisions for resilient and scalable systems.

HardTechnical

65 practiced

Outline a membership and failure-detection protocol suitable for very large clusters (thousands of nodes) that tolerates unreliable failure detectors and partitions. Explain the SWIM approach including gossip-based membership dissemination, indirect probes, suspicion, and anti-entropy, and describe practical implementation details such as partial views, probe amplification, and handling churn.

Sample Answer

**Overview / goal**Design a membership + failure-detection protocol for thousands of nodes that tolerates unreliable detectors and partitions by combining scalable gossip, lightweight probes, suspicion, and background anti-entropy — i.e., the SWIM family.

**SWIM core concepts**- Membership dissemination: each node keeps a partial view (small random subset, e.g., O(log N) or constant like 30–50 peers) and gossips membership updates to them.- Periodic direct probes: node A selects a target B, sends a ping; expecting ack within timeout.- Indirect probes (probe amplification): if B doesn't reply, A asks k random helpers from its partial view to perform ping-req to B and forward any ack back — reduces false positives under loss.- Suspicion mechanism: after failed direct+indirect probes mark B as SUSPECT rather than DEAD; suspicion is gossiped with incarnation numbers. Suspect state gives time for B to refute.- Anti-entropy (gossip dissemination): piggyback membership changes on probes; periodic full gossip/rumor-mongering ensures eventual consistency and heals partitions.

**Practical implementation details**- Partial views: size trades off reachability vs traffic. Use adaptive partial views (hybrid: mostly random + a few well-known seeds/routers).- Probe amplification k: choose 3–5 for high-loss environments; increase timeouts multiplicatively with RTT variance.- Suspect timers & incarnation numbers: allow node to refute by issuing higher incarnation; split-brain avoided by preferring higher incarnation and local monotonic counters.- Handling churn: use rapid anti-entropy rounds during sustained churn, increase gossip fanout temporarily, and use push–pull gossip to converge faster.- Load control: stagger probe schedule (randomized intervals), rate-limit forwarded probe-reqs, and compress membership vectors (delta encoding / bloom filters) for large update sets.- Partition tolerance: on partition, both sides can progress; when healing, anti-entropy + incarnation resolution reconciles states.- Observability & ops: expose metrics (suspect counts, gossip lag), adjustable timeouts, and a safe graceful leave protocol to reduce false suspicions during scale-down.

This yields a scalable, robust protocol suitable for cloud clusters of thousands, balancing low per-node overhead with strong eventual membership correctness.

HardTechnical

82 practiced

A replicated messaging queue is used for order ingestion. Explain how CAP theorem considerations affect design choices for the queue. If network partitions are possible, describe a design that prioritizes availability and its implications on ordering, duplication, and delivery guarantees, and then describe a design that prioritizes consistency and how it would behave under partition.

Sample Answer

**CAP theorem baseline**

CAP says in presence of a network partition you can choose either Consistency (every client sees same view) or Availability (every request succeeds) but not both. For a replicated order-ingestion queue this choice drives replication, leader election, and client guarantees.

**Design that prioritizes Availability (AP)**

- Approach: use multi-master / leaderless replication or fast local writes with asynchronous anti-entropy (e.g., Dynamo-style or Cassandra with W + R ≤ N).- Behavior under partition: - Writes accepted on any reachable replica → system stays available. - Ordering: global order cannot be guaranteed; per-producer sequence numbers help local ordering but cross-producer order may be inconsistent. - Duplication: high chance of duplicates after reconciliation; deliver at-least-once unless dedup applied. - Delivery guarantees: typically eventual consistency; implement idempotent consumers, dedup tables (with TTL), or client-side sequence numbers to achieve effectively-once semantics.- Trade-offs: low latency and high uptime, complexity in reconciliation, increased storage for tombstones/dedup metadata.

**Design that prioritizes Consistency (CP)**

- Approach: leader-based replication with quorum writes or single-writer partitions (e.g., Kafka partition leader, Raft).- Behavior under partition: - If leader is unreachable, either reject writes or stall until leader election completes → availability is reduced. - Ordering: strong ordering preserved per partition; global ordering can be enforced via single partition or centralized sequencer. - Duplication: lower; with leader and durable commit you can provide exactly-once semantics with transactional commits and careful producer/consumer protocols. - Delivery guarantees: can offer linearizability or strong consistency; clients may see errors/timeouts during partitions.- Trade-offs: stronger correctness, simpler consumer logic, but potential downtime and higher tail latency during leader elections.

**Practical considerations for Systems Engineer**

- Choose AP when uptime is critical and clients can handle retries/idempotency.- Choose CP when strict ordering and correctness are required (financial orders).- Hybrid: partition by order key and use CP within partitions, AP across replicas with async cross-region replication; add dedup, sequence numbers, observability and SLA-aligned timeouts.

EasyTechnical

69 practiced

Explain the difference between sharding (horizontal partitioning) and replication. When is each technique appropriate? Describe a simple architecture where you combine sharding and replication to achieve both scale and fault tolerance, and explain how you would route requests to the correct shard.

HardSystem Design

83 practiced

Design an observability pipeline for a platform with 10k services that produces petabytes of logs and traces per day. Describe ingestion architecture, sampling strategies (head-based vs tail-based), indexing and search trade-offs, retention and cost controls, tenant isolation, and strategies to enable debugging of high-cardinality issues without overwhelming storage or query latency budgets.

Sample Answer

**Clarify goals & constraints**- 10k services, PBs/day telemetry, multi-tenant, sub-second trace queries, cost-limited, secure tenant isolation.

**High-level architecture**- Lightweight agents (Fluent Bit, OpenTelemetry) -> Kafka (sharded by tenant/service) -> Stream processors (Flink/KS) for enrichment/early-aggregation -> Cold/Hot storage: - Hot index (Elasticsearch/ClickHouse/txn-index) for recent window (hours–days) - Cost-optimized object store (S3/ADLS) with indexed pointers + Parquet for long-term- Control plane for sampling and policies; query API and UI layer.

**Ingestion & reliability**- Backpressure via Kafka quotas, per-tenant ingress tokens, SLO-based throttling.- Schema registry + protobuf to validate events early.

**Sampling: head vs tail**- Head-based: apply deterministic lightweight rules at agent for rate limits, tag-only traces, and preserve representative set; low CPU.- Tail-based: route raw traces to short-term buffer in Kafka/streaming state; run adaptive policies (error/anomaly detection, latency spikes, rare-span detection) to retain full traces selectively.- Hybrid: head-filtered baseline + tail-selectors for anomalies. Keep sample-decisions versioned and recorded in metadata to reconstruct.

**Indexing & search trade-offs**- Index high-value fields (trace_id, service, env, error_flag, user_id hashed) in hot index; store full spans/blobs in object store with pointers.- Use inverted index for logs; use columnar store (ClickHouse) for high-cardinality metrics/aggregates.- Trade-off: broader indexing = faster queries but high cost; mitigate by time-windowed hot tiers and pre-aggregations.

**Retention & cost controls**- Tiered retention: minutes->hours (hot SSD), days (warm HDD), months->years (cold object store).- Per-tenant budgets and quota enforcement, autosampling when budget exceeded, compression (snappy/zstd), deduplication, delta-encoding.- Lifecycle policies automated with policy-as-code.

**Tenant isolation & security**- Logical isolation via Kafka partitions and storage prefixes; RBAC and mTLS for ingestion; IAM policies per tenant for S3; per-tenant resource quotas and billing meter.- Multi-tenant index patterns with encryption-at-rest and KMS per org for sensitive tenants.

**High-cardinality debugging strategies**- Sparse indexing: index hashed keys plus tag cardinality metadata to avoid exploding index size.- On-demand full-fidelity retrieval: when a query hits a high-cardinality key, fallback to tail-buffered raw data or S3 retrieval with precomputed pointers.- Use fingerprinting and cardinality sketches (HyperLogLog) to detect hot keys and surface them in UI.- Adaptive sampling that preserves all error/slow-request traces and keeps head-sampled representative spans per high-cardinality dimension.- Provide query-time server-side aggregation, sampling-aware query rewrites, and explain-plan to show sampling bias.

**Operational considerations**- Observability for the observability stack itself (meta-metrics).- Chaos-tests for backpressure, tenant blast radius testing.- Cost + SLA telemetry fed to a control loop to auto-tune sampling thresholds.

HardSystem Design

61 practiced

Design a globally-distributed shopping cart service used by millions worldwide. Requirements: low read latency from nearest region for browsing, allow offline adds (mobile), preserve cart state across devices, support eventual or strong consistency at checkout, and resolve concurrent modifications during cart merge. Describe architecture, storage/replication choices, conflict-resolution strategies, and common failure modes.

Sample Answer

**High-level summary**Design a multi-region cart service with local reads, offline adds, cross-device sync, and configurable consistency at checkout. Use region-local caches for low latency, durable global storage for sync, and CRDT/operation-log merging for conflicts.

**Architecture**- Edge/API tier in each region (TLS + auth). Mobile/web clients talk to nearest edge.- Local write-ahead log (WAL) + local Redis cache for fast reads and offline buffering.- Global Cart Service backend: per-user cart metadata and append-only operation store (Kafka/LogDB) replicated across regions.- Background sync workers reconcile local WAL -> global log and propagate deltas.

**Storage & replication**- Hot path: regional Redis (replicated within region) for reads/low-latency updates.- Durable: globally-replicated append-only store (Kafka with MirrorMaker 2 or a geo-replicated DB like Spanner/Cockroach) for strong serialization when needed.- Snapshot store: S3/GCS for periodic cart snapshots and long-term recovery.

**Consistency & checkout**- Eventual by default: merges use CRDTs (observed-remove set with quantities) or operation transforms so concurrent adds/removes converge.- Strong at checkout: acquire per-cart global lease (via distributed lock in Spanner/etcd) then materialize final cart from global log and apply business rules atomically.

**Conflict resolution**- Use operation-based CRDTs: each client op = (user, op_id, timestamp, op). Merge by causal order; removals include tombstones to avoid reappearance.- On cart-merge (device sync), prefer last-writer-for-each-item quantity with monotonic counters or apply max(quantity) depending on UX; expose merge summary to user on significant conflicts.

**Failure modes & mitigations**- Network partition: local adds buffered in WAL; sync retries with backoff; on prolonged partition merge via CRDTs.- Lost messages: use durable append-only logs and idempotent ops (op_id).- Clock skew: use logical clocks/vector clocks or Lamport timestamps for ordering.- Hot partitions (popular carts): sharding by user-id with rebalancing and rate limiting.- Lease failure at checkout: fallback to optimistic concurrency with compare-and-swap on global snapshot and user-visible retry.

This design balances low-latency UX, offline capability, and safe strong consistency at checkout while keeping merge logic auditable and recoverable.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.