Distributed Systems Fundamentals Questions

Core principles and theory that underlie distributed computing systems. Includes understanding trade offs between consistency, availability, and partition tolerance, common consistency models such as eventual and strong consistency, replication and sharding strategies, load balancing and data partitioning, consensus algorithms and their guarantees, scalability and fault tolerance patterns, and how these concepts apply to infrastructure components such as databases, caches, service meshes, and load balancers. Candidates are expected to explain design choices, common failure modes, and how fundamental concepts influence architecture decisions for resilient and scalable systems.

EasyTechnical

102 practiced

Define eventual consistency formally and describe the guarantees it provides. Give two real-world examples (for instance DNS and distributed cache invalidation) that use eventual consistency and explain expected convergence behavior and practical staleness bounds you might engineer.

HardTechnical

123 practiced

Design an algorithmic approach to detect and recover from split-brain in an active-active cluster. Include detection mechanisms (quorum checks, external arbiters), safe reconciliation strategies for diverged writes, fencing mechanisms to prevent stale primaries from writing, and discuss trade-offs between availability and correctness.

Sample Answer

Approach summary: treat split‑brain as a membership + state divergence problem. Detect with quorum/witness checks, stop new writes on suspected partitions, fence stale primaries, then reconcile divergences using safe merge strategies (application-aware or convergent CRDTs) or manual conflict resolution when necessary. Balance: stricter correctness reduces availability; favor depending on SLA.

Detection mechanisms- Majority quorum (N/2+1): any partition without quorum must reject writes.- Witness/arbitrator node: lightweight external process in different failure domain to break ties when even-sized clusters split.- Heartbeats with monotonic epochs: nodes exchange leader epoch and last-committed index; missing or inconsistent epochs trigger suspicion.- Cross-check: compare commit index and checksums for recent log segments to detect divergence.

Fencing mechanisms- Epoch/term-based fencing: leader increments epoch on election; all writes tagged with epoch. Nodes reject writes with older epochs.- External lease/token (e.g., Zookeeper, etcd): grant lease to active leader; lease expiry/fencing prevents stale leaders from accepting writes.- Hardware/software STONITH for aggressive safety in stateful systems.

Safe reconciliation strategies- Prevent divergence first: prefer quorum-write (majority) for commits.- If divergence occurs: - For commutative, convergent data (counters, sets): use CRDTs to auto-merge. - For append-only logs: replicate and use deterministic merge by timestamp + origin id, then de-duplicate. - For general mutable state: compute operation logs / intents; attempt automatic merge (three-way merge using base + branch A + branch B); if conflicts remain, mark records as conflicted and route to human or business-rule resolver. - Use tombstones/causal metadata to avoid resurrecting deletes.- Reconciliation workflow: snapshot pre-merge, run dry-run merge with validation tests, apply merged state under new epoch with fencing, verify checksums.

Implementation considerations- Keep arbiter/stateless witness small and highly available.- Record durable operation log with causal metadata (vector clocks or hybrid logical clocks) to help deterministic merges.- Monitor RPO (how much data may be lost) and RTO (time to reconcile); provide admin tools for inspecting conflicts.- Test failure scenarios with chaos engineering.

Trade-offs- Availability vs correctness: quorum/lease/fencing favors correctness (potentially reduced availability during partitions). Allowing minority writes (optimistic active-active) improves availability but requires complex reconciliation and may violate invariants.- Complexity vs automation: CRDTs automate merges but constrain data model. Application-level reconciliation preserves semantics but needs operational overhead.

Metrics and safety- Enforce CI for reconciliation logic, audit logs, and ability to rollback.- Track conflict rate, reconciliation duration, number of manual interventions.This design prevents silent data corruption, provides deterministic recovery paths, and surfaces unavoidable human decisions when semantics require it.

EasyTechnical

82 practiced

Compare synchronous (HTTP/gRPC) and asynchronous (message-based) communication between microservices. Discuss trade-offs in latency, fault isolation, complexity, back-pressure handling, and when you would pick one over the other for specific types of interactions (e.g., checkout vs. analytics ingestion).

HardSystem Design

79 practiced

Design a distributed transaction solution for an online marketplace that must update inventory and billing atomically across services. Compare Saga (choreography versus orchestration) with 2PC; outline compensating actions, idempotency, the role of a coordinator, failure modes, and how you would monitor and debug in production.

Sample Answer

Requirements & constraints:- Atomicity across Inventory and Billing for a marketplace order: either both succeed (reserve inventory, charge buyer) or effects are undone.- High availability, low latency, cross-service autonomy, eventual consistency acceptable (no global lock).- Services deployed independently, possible partial failures, retries, network partitions.

High-level options:1) Two-Phase Commit (2PC): coordinator asks participants to prepare/commit.- Pros: strong atomicity across participants.- Cons: blocking (locks resources), single point coordinator, poor availability under failures, bad for long-running business ops (billing), not cloud-friendly across microservices/databases.

2) Saga Pattern (recommended): sequence of local transactions with compensating transactions.- Choreography: each service emits events and next service reacts. - Pros: simple, no central coordinator. - Cons: implicit flow hard to observe, brittle for complex flows.- Orchestration: central Saga Orchestrator controls steps and triggers compensations on failure. - Pros: explicit state machine, easier observability and error handling. - Cons: single logical coordinator (can be made highly available).

Design (Orchestrated Saga):- Components: Order Service (orchestrator), Inventory Service, Billing Service, Event Log / Durable Queue, Idempotency store, Compensation handlers.- Flow: 1. Order Service creates Saga instance (state: PENDING), publishes ReserveInventory command. 2. Inventory reserves items (local TX), persists reservation, replies SUCCESS → Order Service sends ChargeBilling command. 3. Billing charges card (local TX), persists charge id, replies SUCCESS → Order Service completes Saga and publishes OrderConfirmed. 4. On any failure, orchestrator issues compensating actions in reverse order (e.g., RefundCharge, ReleaseInventory).

Compensating actions:- Inventory: release reservation or restock.- Billing: refund or void authorization (use separate idempotent Refund/void API).- Each compensating action is a well-defined business transaction implemented by the service.

Idempotency:- All commands/statements carry Saga ID + step ID; services store processed IDs and results. Replayed or retried commands check store and return previous outcome.- Use at-least-once messaging with dedupe via idempotency keys; ensure compensating actions are idempotent.

Coordinator role:- Maintains Saga state machine (persisted); retries steps with backoff; decides timeouts, escalation to manual intervention for long-failed sagas.- Make coordinator HA: leader election (e.g., Kubernetes with leases) and persistent Saga store (RDB/NoSQL). Use optimistic concurrency to avoid duplicates.

Failure modes & mitigations:- Participant failure mid-saga: orchestrator retries; if timeout, trigger compensations already completed.- Network partition: idempotent retries and persistent state; if partition isolates coordinator, standby takes over using persisted state.- Partial side-effects (e.g., charge succeeded but inventory reserve failed): compensating refund; ensure billing supports void vs refund semantics to minimize cost.- Permanent failures (payment disputes): surface to operations/UX with clear user messaging and manual remediation workflow.

Monitoring & debugging:- Central Saga dashboard listing saga instances, state transitions, timestamps, retries, and duration.- Correlated distributed traces (OpenTelemetry) across services using Saga ID as trace/span attribute.- Structured logs including Saga ID, step, outcome, retry count.- Metrics: sagas started/succeeded/failed, average duration, compensation rate, retry counts, queue lengths.- Alerts: high compensation rate, long-running sagas, repeated transient errors.- Tools: Queryable event store for replay, ability to replay or manually trigger compensations; runbook for common failure scenarios.

Trade-offs summary:- 2PC gives strict ACID but impractical for microservices (availability & scaling). Orchestrated Saga provides business-level atomicity with better availability and observability at the cost of eventual consistency and needing compensations. Use orchestration for complex flows and choreography for simpler, linear flows where coupling is low.

EasyTechnical

77 practiced

What does idempotency mean in the context of distributed APIs? Provide an example design for an idempotent payment or order endpoint, describing how clients generate keys, how the server stores outcomes, and how expiration or garbage collection of idempotency records is handled.

Sample Answer

Idempotency in distributed APIs means that making the same request multiple times (due to retries, network errors, or client bugs) has the same effect as making it once — no duplicate side effects (e.g., double charge). The server must detect repeated attempts and return the original outcome rather than performing the operation again.

Example design for an idempotent payment/order endpoint:

- Client responsibility (how to generate keys) - Client generates a unique Idempotency-Key per logical user intent (UUIDv4 or deterministic hash of user+cart+timestamp). Key must be stable across retries for the same intent and unique for different intents. - Client sends header: Idempotency-Key: <key> with request body and relevant metadata (user-id, amount).

- Server responsibility (storing outcomes) - On receiving request: 1. Lookup idempotency record by (Idempotency-Key, user-id) in a fast key-value store (Redis, DynamoDB). 2. If record not found, create a record with state=IN_PROGRESS, request hash, timestamp, and attempt to process the payment (use transactional or externally-consistent calls to payment provider). 3. On success/failure, atomically update record to state=COMPLETED or FAILED with response payload (status code, body, transaction id, error code). 4. Return stored response to any client that retries with same key. - Concurrency: Use compare-and-set or DB conditional write to ensure only one worker moves IN_PROGRESS → processing; others read IN_PROGRESS and either wait/poll or return 409/202 with location to poll.

- Data model example - Key: idempotency:{user_id}:{idempotency_key} - Value: {state, request_hash, response_body, response_status, tx_id, created_at, last_updated}

- Expiration / garbage collection - Keep records long enough to cover typical retry windows (e.g., 24–72 hours). Set TTL on key-value store to auto-expire. - For longer retention needs (audit), persist completed records to audit store and remove from fast store after TTL. - Use background GC to remove stale IN_PROGRESS records older than a threshold (mark as FAILED or allow safe retry). - Provide API to clients to query status if IN_PROGRESS persists.

Notes and caveats:- Include request hashing to detect changed payloads with same key: if request differs, return 400 to avoid silent mismatches.- Protect against replay across users by scoping keys to user/account.- Idempotency keys are for deduplication of non-idempotent operations (POST); GET/PUT semantics differ.- Choose TTL balancing cost vs risk of duplicate processing after expiry.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.