Distributed Systems Principles and Tradeoffs Questions
Fundamental concepts and engineering trade offs for systems that run on multiple machines or across data centers. Topics include consistency models such as strong eventual and causal consistency; the trade off between consistency availability and partition tolerance; conceptual understanding of consensus and leader election algorithms such as Paxos and Raft; replication and partitioning strategies including leader follower and multi leader approaches; failure modes including network partitions partial failures clock skew and split brain; mitigation patterns such as retries with idempotency exponential backoff circuit breaker and bulkhead; conflict detection and state reconciliation strategies; considerations for distributed transactions and eventual reconciliation; monitoring and observability including logs metrics and distributed tracing; testing strategies including fault injection and chaos engineering; and reasoning about how these choices affect correctness latency complexity and operational cost. Interviewers will probe the candidate on choosing appropriate consistency and replication schemes explaining failure modes and designing systems that remain correct and available under realistic failure scenarios.
MediumTechnical
31 practiced
Create a six-week fault-injection and chaos-engineering plan to increase resilience of an online retail checkout flow. Specify a sequence of experiments (from low to higher blast radius) such as database read latency injection, region failover, downstream service timeouts, and pod kills. For each experiment provide success criteria, monitoring checks, rollback plan, stakeholder communication, and how to capture learnings and remediation tasks.
Sample Answer
**Six-week plan — progressively increasing blast radius**Week 1: Read-only DB latency injection (canary)- Success: no checkout failures; p95 latency < threshold- Monitor: errors, latency, cart abandonment- Rollback: remove chaos pod; notify opsWeek 2: Cache layer eviction & degraded cache- Success: cache miss handling OK, origin load acceptable- Rollback: restore cache nodesWeek 3: Downstream payment timeout simulation (canary region)- Success: fallback to payment retry/queued payments- Rollback: disable injection, replay queuedWeek 4: Pod kills for non-critical services (notification) cross-region- Success: no data loss; retries/compensations work- Rollback: scale up podsWeek 5: Regional failover of checkout region (traffic shift)- Success: failover completed within RTO; orders processed consistently- Rollback: revert trafficWeek 6: High-blast run: simultaneous DB latency + payment degradations (controlled window)- Success: SLOs within agreed error budget; manual escalation thresholds unchanged- Rollback: emergency kill experiments, failover to standbyFor each experiment include:- Monitoring checks: SLOs, error budgets, business metrics (checkout completions), and infra signals- Rollback plan: immediate stop of chaos, autoscaling, traffic reroute- Stakeholder comms: pre-announce, runbook, postmortem- Capture learnings: automated runbook updates, remediation tickets, and prioritized fixes.Start small, validate observability and rollbacks, then increase scope and stakeholders involvement.
HardTechnical
37 practiced
Design a split-brain detection and prevention strategy for a multi-master database replicated across data centers. Include fencing mechanisms (epoch numbers, leases, token-based fencing), automated detection heuristics (heartbeat anomalies, checksum divergence), operator runbooks for reconciliation, and a safe data-merge strategy for divergent but commutative operations versus non-commutative conflicts.
Sample Answer
**Situation / Goal**: Prevent and recover from split-brain in a multi-master DB across datacenters while preserving safety and minimizing downtime.**Fencing mechanisms**:- Epoch numbers: each master write carries a monotonically-increasing epoch from a centralized arbiter; nodes reject writes with stale epoch.- Leases: short, renewable leader leases (e.g., 2–5s) stored in a highly-available coordinator; expiration prevents stale leaders.- Token-based fencing: use hardware/software tokens (e.g., cloud metadata tokens or KMS-signed capability) that the active master must present; revoking token denies further writes.**Automated detection heuristics**:- Heartbeat anomalies: missed heartbeats beyond threshold triggers suspicion; require correlated network telemetry before action.- Checksum divergence: periodic state checks (Merkle trees) detect data drift; large deltas mark potential split-brain.- Operation logs: conflict frequency spikes and replication lag > threshold trigger automated investigation.**Operator runbook (reconciliation)**:- Detection → Isolate suspected partitioned node (revoke token, remove from quorum) → Capture consistent snapshots and operation logs → If checksums match, rejoin via incremental replication; if divergent, escalate.- Human-in-loop steps for non-deterministic conflicts: notify SRE + DB owner, present divergence report, and require explicit confirmation to merge.**Safe data-merge strategy**:- Commutative ops: merges via CRDTs or idempotent last-writer-wins with vector clocks; automated reconciliation using operation-idempotency and causal order.- Non-commutative conflicts: preserve history and surface conflict resolution to application owners. Prefer application-level merge handlers or present both versions and run compensating transactions. Use compensating audits and OLAP-style reconciliation for data that cannot be auto-merged.**Preventive policies & testing**:- Enforce leases + token fencing in all masters, periodic chaos tests (network partitions, lease expiration), and automated rollback windows.**Outcome**: This combination prevents unsafe concurrent writers, provides fast automated remediation for common cases, and escalates complex conflicts to humans with clear data artifacts for safe reconciliation.
HardTechnical
25 practiced
Deeply analyze Raft's log replication: explain how leader election, AppendEntries, commit rules, and log compaction (snapshotting) interact. Describe concrete scenarios that cause log divergence between leader and follower (for example, follower with stale entries from prior leaders) and provide a safe, step-by-step recovery plan to reconcile follower logs without violating Raft's safety properties.
Sample Answer
**Situation / Scope**Explain Raft pieces and how they interact: leader election, AppendEntries, commit rules, snapshotting. Then enumerate divergence scenarios and a safe recovery plan.**Leader election & AppendEntries (brief)**- Leader election picks a leader with the most up-to-date log (term, lastIndex). Leaders send AppendEntries RPCs carrying prevLogIndex/prevLogTerm and entries; followers accept only if their log matches prev fields.**Commit rules**- Leader considers an entry committed when a majority of servers store it in the same term (including leader)—this prevents committing entries from old terms without a later entry in current term.**Snapshotting/log compaction**- When a follower lags far behind, leader may send InstallSnapshot RPCs containing compacted state and last included index/term; follower replaces prefix with snapshot state and adjusts nextIndex.**Divergence scenarios**1) Follower has entries from a prior leader that the current leader doesn't have (different term or missing suffix).2) Follower appended speculative entries but never committed due to election; current leader's log has conflicting entries at same indexes.3) Follower behind beyond leader's retention window (leader has snapshot-only for prefix).**Safe step-by-step recovery plan**1) On AppendEntries rejection, leader decrements nextIndex for that follower and retries—this finds the last matching index via binary/backoff probing to limit RPCs.2) If nextIndex <= leader.firstLogIndex (snapshot threshold), send InstallSnapshot: include lastIncludedIndex/Term. Follower replaces its prefix and sets lastApplied <= lastIncludedIndex.3) After snapshot install, leader sets nextIndex = lastIncludedIndex + 1 and sends AppendEntries for remaining entries.4) Ensure commit follows Raft rules: leader only advances commitIndex when majority has replicated entries from leader's term; do not force-commit entries from older terms.5) Maintain persistent term and votedFor to avoid skipping elections.**Why safe**- Matching prevLogIndex/Term prevents overwriting divergent committed entries on any server.- Snapshot install reconciles followers that cannot reach match via incremental backoff without violating safety because snapshots include last included terms and indices and apply state deterministically.**Operational notes**- Use exponential/backoff probing or binary search to reduce AppendEntries churn.- Monitor follower replication lag, snapshot frequency, and retention windows so followers rarely need full snapshots.- Use metrics/alerts for repeated InstallSnapshot events (indicates slow follower or GC misconfiguration).This plan preserves Raft invariants: never apply a log entry unless it is committed and guarantee leader learned majority before advancing commitIndex.
EasyTechnical
30 practiced
Explain the CAP theorem and provide concrete examples of how each of the three properties (consistency, availability, partition tolerance) manifests in a cloud-based microservices application. For each pairwise trade-off (CA, CP, AP), name a real-world service (for example: global user profile store, chat service, payment ledger), justify which two properties you'd prioritize, and describe one operational consequence of that choice (monitoring, scaling, or failover behavior).
Sample Answer
**CAP theorem recap**In presence of network partitions you must choose between consistency (C) and availability (A). Partition tolerance (P) is non-negotiable in distributed cloud systems.**How properties manifest**- Consistency: reads reflect latest writes (e.g., single-region SQL master). Example: payment ledger.- Availability: system answers requests even under some failures (e.g., cached profile reads).- Partition tolerance: system continues operating despite region/network split (e.g., degraded read-only global storefront).**Pairwise trade-offs & examples**- CA (Consistency + Availability): example — internal config store in single datacenter. Prioritize C and A; consequence: no partition tolerance — global failover requires manual DR and monitoring of datacenter health.- CP (Consistency + Partition tolerance): example — payment ledger. Prioritize C and P; consequence: during partitions some regions reject writes (reduced availability) and require monitoring for rejected requests and automated failover gating.- AP (Availability + Partition tolerance): example — global user profile cache or social feed. Prioritize A and P; consequence: potential inconsistent reads; monitoring must track divergence metrics and background reconciliation systems to heal conflicts.
EasyTechnical
34 practiced
Explain the problems clock skew and inconsistent timestamps create in a distributed system (examples: event ordering, log correlation, TTL expiry). List and describe three concrete mitigation strategies you would adopt when designing cross-region logging and distributed event ordering.
Sample Answer
**Problems caused by clock issues**Clock skew and inconsistent timestamps break event ordering, make cross-service log correlation unreliable, and cause wrong TTL expiry (early or late deletions).**Three mitigation strategies**1) Synchronized clocks + NTP/Chrony with monitoring- Deploy disciplined NTP/Chrony with discipline to hardware RTC and monitor offset metrics (alert if >50ms). Useful for log timestamps.2) Use logical clocks for ordering- Lamport or hybrid logical clocks (HLC) to attach monotonic logical timestamps to events — ensures causal ordering even when physical clocks drift.3) Include provenance and store both physical+logical timestamps- Logs/events include server timestamp, HLC, and region id. For TTLs rely on leader-based expiry or centralized coordinator to avoid premature expirations across regions.These combined give robust cross-region logging and event ordering while retaining human-readable physical times.
Unlock Full Question Bank
Get access to hundreds of Distributed Systems Principles and Tradeoffs interview questions and detailed answers.
