Distributed Systems Design and Trade-offs Questions

Evaluate the candidate's ability to solve complex, multi-layered distributed-systems design problems by making reasonable assumptions, articulating trade-offs, and handling edge cases. Candidates should show how to decompose problems that span networking, caching, persistence, and performance optimization; select architectures and algorithms with explicit trade-off analysis (e.g. speed versus simplicity, consistency versus availability, synchronous versus asynchronous communication); and consider failure modes including network failures, device or resource limitations, and concurrent access patterns. Strong responses include clear assumption statements, alternative approaches, complexity and cost considerations, testing and validation strategies, and plans to monitor and mitigate operational risk (circuit breakers, rate limiting, backpressure, observability).

EasyTechnical

84 practiced

Explain the circuit breaker pattern and how it prevents cascading failures in distributed systems. Describe the typical circuit states, transition triggers, and how you would implement a circuit breaker for an HTTP client including threshold selection and reset behavior.

Sample Answer

The circuit breaker pattern prevents cascading failures by stopping calls to a failing downstream component after errors exceed a threshold, giving the failing service time to recover and avoiding resource exhaustion across the system.

Typical circuit states and transitions:- CLOSED: Normal operation; requests flow through and errors are counted. - Transition to OPEN when error rate or consecutive failures exceed configured threshold within a rolling window.- OPEN: Short-circuit mode; requests immediately fail-fast (or return a fallback) to avoid stressing the downstream service. - Transition to HALF-OPEN after a configured timeout (sleep window).- HALF-OPEN: Probe state; allow a limited number of trial requests to test recovery. - If trial requests succeed (below failure threshold), transition back to CLOSED and reset counters. - If a trial request fails, transition immediately back to OPEN and restart the timeout.

Implementing a circuit breaker for an HTTP client:- Metrics: track rolling counts (consecutive failures or failure ratio) and request latencies. Use a fixed-size sliding window or exponentially-weighted moving average to avoid unbounded memory.- Threshold selection: - Consecutive-failure threshold: suitable for low-latency critical calls (e.g., 3–5 consecutive failures). - Error-rate threshold: for high-volume calls, use percentage (e.g., 50% errors over last N requests or 10s window with min requests guard like 20 requests). - Configure conservative defaults, allow per-endpoint tuning based on SLA and traffic patterns.- Reset behavior: - Open timeout (sleep window): start small (e.g., 5–30s) and consider exponential backoff for repeated open cycles. - Half-open probes: limit concurrent probes (e.g., 1 or a small number) to avoid thundering herd. - On success: gradually restore (reset counters) and optionally use success streak requirement (e.g., 2–3 successful probes) before fully closing.- Practical considerations: - Provide fallback responses or default behavior to maintain user experience. - Instrumentation and alerts for state changes and metrics. - Thread-safety and distributed coordination: single-process breaker per client is common; for multi-instance systems, consider shared health signals (service discovery, health checks) rather than trying to synchronize breaker state.- Example behavior: HTTP client returns cached/fallback response immediately when OPEN, attempts one probe after 10s in HALF-OPEN; if probe succeeds twice, close; if it fails, open for 30s and double timeout next time.

This approach reduces latency spikes, prevents resource exhaustion, and isolates faults so failures don’t cascade through the system.

MediumTechnical

70 practiced

Explain the bulkhead pattern and how you would apply it to isolate failures in a critical payments subsystem so that user-facing search and discovery remain responsive. Include examples at process/thread level, queue level, and deployment-level, and describe metrics to measure effectiveness.

HardSystem Design

96 practiced

Architect a streaming analytics pipeline that reads from an ordered distributed log, transforms events, and updates downstream aggregates with exactly-once semantics. Describe how to manage source offsets, checkpointing, idempotent or transactional sinks, and recovery behavior to guarantee correctness during failures and reprocessing.

Sample Answer

Requirements & constraints:- Read from an ordered distributed log (e.g., Kafka, Kinesis) preserving partition ordering.- Transform events (enrichment, aggregations) and update downstream aggregates with exactly-once semantics.- Support failures, reprocessing, and scalable parallelism.

High-level architecture:Producer → Distributed Log (partitions) → Stream Processor (stateful operators) → Sink (aggregates DB / materialized view)

Core components and responsibilities:1. Source reader: consumes committed offsets per partition, emits records with their source offset metadata.2. Processing engine: performs transformations and stateful aggregations using local state stores (embedded RocksDB) and a changelog stream for state durability.3. Checkpoint coordinator: periodically snapshots in-flight offsets + local state progress and persists checkpoint metadata.4. Sink connectors: either transactional sinks (two-phase commit) or idempotent sinks with deduplication keys.

Exactly-once design and mechanisms:- Use atomic checkpoint including: - Per-partition consumed offset - Local operator state snapshot (or applied changelog offset) - In-flight output transaction boundaries- For transactional sinks: implement a two-phase commit (prepare/commit) per checkpoint. On checkpoint N: - Begin transaction T_N at sink - Emit outputs tied to T_N - On successful local snapshot, commit T_N - Persist checkpoint metadata mapping offsets → T_N - On recovery, only commit or abort transactions based on persisted checkpoint state.- For idempotent sinks (preferred when transactions not supported): include deterministic deduplication keys (e.g., partition+offset, event-id). Sink must dedupe by key (idempotent upserts) or maintain last-applied offset per upstream partition.

Offset & state management:- Maintain checkpoint-aligned offsets: do not advance committed source offsets until the corresponding checkpoint is durable and outputs are committed.- Option A (changelog-based): persist operator state changes to a changelog topic; checkpoints only store last applied changelog offset. Recovery replays changelog to rebuild state deterministically, then continue from source offsets > checkpoint.- Option B (snapshot-based): snapshot state to durable storage and store snapshot version + source offsets in checkpoint.

Checkpointing cadence & trade-offs:- Frequent checkpoints = lower replay window and faster recovery, higher overhead.- Larger checkpoints = less frequent IO but longer potential reprocessing and recovery time.

Recovery behavior:- On restart, read latest successful checkpoint: - Restore operator state (from snapshot or replay changelog up to saved changelog offset). - Restore sink transaction state: if a transaction for checkpoint was prepared but not committed, consult checkpoint metadata and either commit (if checkpoint marked successful) or abort. - Resume consuming from saved source offsets (next offset = last_committed_offset + 1).- If reprocessing older data (retrospective job): assign deterministic keys and use idempotent sinks or run a versioned write (write-then-swap) so aggregates remain correct.

Correctness considerations:- Ensure deterministic processing: stateless transforms deterministic; stateful ops must be order-preserving per key and use consistent partitioning.- Make side effects transactional or idempotent (external HTTP calls are problematic — buffer and make them idempotent).- For aggregated counters, use upserts with last-offset metadata per key to avoid double-counting.

Scaling and availability:- Partition stream by key to enable parallel, ordered processing.- Use a distributed checkpoint coordinator (leader election) to avoid split-brain.- Retain changelog/topic retention long enough to allow recovery.

Common trade-offs:- Two-phase commit gives stronger guarantees but complicates sink implementation and increases latency.- Idempotent upserts scale easier but require sinks that can dedupe and store last-seen metadata.

Summary: Guarantee exactly-once by atomically coupling source offsets, operator state, and sink commits via durable checkpoints. Prefer changelog-backed state + checkpoint-aligned transactions or idempotent sinks with deterministic dedupe keys to ensure correctness under failures and reprocessing.

EasyTechnical

138 practiced

Compare synchronous and asynchronous communication between microservices. For each mode list three advantages and three disadvantages. Give one concrete example scenario (e.g., login, order placement) where you'd choose synchronous and one where asynchronous is preferred, stating latency and consistency assumptions.

HardSystem Design

68 practiced

Design a reconciliation and repair system to restore eventual consistency between two services (orders and shipments) after messages were duplicated or lost due to a broker outage. Describe detection methods, idempotent replay, checkpoints, monotonic sequence numbers, throttling of repairs to avoid overload, and operator tools to inspect and apply fixes.

Sample Answer

Requirements & constraints:- Goal: restore eventual consistency between Orders and Shipments after broker outage causing message loss/duplicates.- Non-functional: minimal downtime, avoid overloading services, auditability, safe for retries.

High-level architecture:- Reconciliation service reads canonical state from each service (primary DB snapshots or change-log), computes deltas, and issues repair actions (replay/create/cancel events) via a controlled repair queue to target service APIs or event topic.

Detection methods:- Dual-source comparison: compare authoritative order state (Orders DB or CDC log) vs. inferred shipments (Shipments DB or SIP logs).- Invariants & checksums: per-order hashes, totals, expected shipment counts, statuses.- Sequence-gap detection: use monotonic sequence numbers or event offsets from broker/CDC to detect missing ranges.- Duplicate detection: identify multiple events affecting same business key with same sequence id/timestamp/hash.

Idempotent replay & repair actions:- Require every event/repair to carry a globally-unique id (repair_id) and original monotonic event_id.- Target endpoints expose idempotent handlers: store processed_event_ids with TTL/retention; ignore replays with seen ids; apply compare-and-set semantics for state transitions.- Repairs should include a before/after expected version (optimistic concurrency) to avoid stomping concurrent legitimate changes.

Checkpoints & monotonic sequence numbers:- Maintain persistent checkpoint per reconciliation worker: last_processed_event_id per partition/key and last_repair_id applied.- Use monotonic event sequence numbers (or increasing CDC LSNs) embedded in events so workers can resume and detect gaps deterministically.- Periodic checkpoint flush to durable store (preferably strongly consistent KV like etcd or RDBMS) to survive crashes.

Throttling and safety:- Repair queue implements rate-limiting, concurrency limit per target service, and backoff on 429/5xx responses (exponential with jitter).- Prioritize repairs: high-severity (e.g., unpaid shipped orders) first; batch low-risk fixes.- Circuit breaker per service to pause repairs if error rates spike.- Staged rollout: dry-run mode that flags intended changes without applying; then apply in waves.

Operational tooling:- Dashboard showing divergence metrics, checkpoints, pending repair queue, per-key history and event timeline.- Inspect tool: view order+shipment snapshots, full audit trail of events, computed diff, suggested repair.- One-click actions: apply suggested repair, schedule repair window, or simulate.- Safety controls: require approval or multi-signer for destructive repairs; enable transactional rollbacks where supported.- Logs & audit: immutable audit log with repair_id, operator, timestamp, outcome.

Implementation notes & trade-offs:- Relying on CDC (Debezium) gives authoritative LSNs; if not available, use service-level versioning.- Storage of processed_event_ids grows—use TTL and compaction by checkpointing.- Idempotency table can be sharded by key to scale.- Stronger consistency (synchronous repair) increases load; prefer eventual, prioritized repairs with throttling.- Test extensively with fault-injection: duplicate messages, delayed delivery, partial failures.

This design detects gaps/duplicates deterministically, replays idempotently, persists progress via checkpoints and sequence numbers, protects services with throttling/circuit breakers, and provides operators with tools to inspect, approve, and audit fixes.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Design and Trade-offs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.