Distributed Systems Principles and Tradeoffs Questions

Fundamental concepts and engineering trade offs for systems that run on multiple machines or across data centers. Topics include consistency models such as strong eventual and causal consistency; the trade off between consistency availability and partition tolerance; conceptual understanding of consensus and leader election algorithms such as Paxos and Raft; replication and partitioning strategies including leader follower and multi leader approaches; failure modes including network partitions partial failures clock skew and split brain; mitigation patterns such as retries with idempotency exponential backoff circuit breaker and bulkhead; conflict detection and state reconciliation strategies; considerations for distributed transactions and eventual reconciliation; monitoring and observability including logs metrics and distributed tracing; testing strategies including fault injection and chaos engineering; and reasoning about how these choices affect correctness latency complexity and operational cost. Interviewers will probe the candidate on choosing appropriate consistency and replication schemes explaining failure modes and designing systems that remain correct and available under realistic failure scenarios.

MediumTechnical

34 practiced

You observe a sudden spike in 95th percentile latency for a checkout request path from 200ms to 2s across regions. Outline an observability-driven runbook using logs, metrics, and traces to identify root cause and verify a remediation. Include which signals to check first and what quick experiments to run.

Sample Answer

Runbook (observability-driven) — goal: identify root cause of 95th‑pct checkout latency jump (200ms → 2s) and verify remediation.

1) Triage / quick signals (first 5–10 min)- Alert/dashboard: confirm spike time, affected regions, % of users impacted.- Traffic & error metrics: request rate (RPS), success rate, HTTP 5xx/4xx, retry rates.- Latency heatmap / p95 by region, by endpoint, by user segment, and by host/pod.- Recent deploys/config changes/feature flags in CI/CD and infra (within last 60m).

2) Traces (next 10–20 min)- Query distributed traces for slow traces (p95/p99). Identify common slow spans (DB, external APIs, auth, payment gateway).- Use trace waterfall: find span causing tail latency, count of spans with high duration, and whether slow spans are upstream or downstream.- Check span attributes: database queries, SQL text, cache miss rates, network timeouts.

3) Metrics & infra (parallel)- Per-host/pod CPU, memory, GC pauses, thread pool saturation, event-loop latency.- DB metrics: slow queries, locks, connections, replication lag, CPU.- Cache metrics: hit/miss rate, eviction, latency.- Network: increased RTT, packet loss, LB health checks, connection errors.

4) Logs (investigate correlated traces)- Filter logs for slow request IDs / trace IDs; search for exceptions, timeouts, retries.- Look for patterns: authentication failures, rate-limiter rejects, backpressure logs, circuit-breaker tripping.

5) Quick experiments (safe, low-cost)- Canary rollback of last deploy or toggle related feature flag for affected regions.- Traffic shift: divert a small percentage to a healthy region/version to validate whether problem is code/deploy vs infra/third‑party.- Scale up: increase replicas or DB connection pool temporarily to see if latency drops (tests if resource exhaustion).- Throttle retries or disable noncritical downstream calls to assess impact.- Run synthetic requests with tracing to reproduce and isolate span causing delay.

6) Root-cause confirmation- Correlate remediation change with reduction in p95 across same cohorts and traces showing shorter problematic span.- Validate no increase in errors, and check SLOs, business metrics (conversion rate).- Post-mortem: capture timeline, root cause, detection gaps, and actions (monitoring, alert tuning, runbook updates).

Signals to prioritize: p95 latency by region/host, trace span durations, error rates, recent deploys/flags, DB/queue/cpu saturation.

EasyTechnical

26 practiced

Describe the circuit breaker pattern at a high level. Explain the closed, open, and half-open states and give an example of metrics or conditions that should trigger each state transition in a service-to-service call scenario.

EasyTechnical

33 practiced

When decomposing a monolith into microservices, what signals do you use to decide service boundaries? List at least four indicators that suggest a module should become a separate service and explain how distributed-system concerns influence your decision.

MediumTechnical

35 practiced

Compare Paxos and Raft as consensus protocols for implementing replicated state machines. Focus on safety and liveness guarantees, understandability for implementers, and when you would pick one over the other for a new service.

Sample Answer

Situation: You're choosing between Paxos and Raft to implement a replicated state machine for a new service. Here's a comparison focused on safety, liveness, implementer understandability, and selection guidance.

Safety:- Both Paxos (classic/Multi-Paxos) and Raft provide strong safety: they never decide conflicting values; logs stay consistent across non-faulty replicas. Both guarantee linearizability when used correctly (leader-based variants ensure a single authoritative sequence).

Liveness:- Both require a majority of nodes and some partial synchrony to make progress. Raft’s leader election and heartbeat timers make liveness behavior more predictable in practice. Paxos can make progress similarly but often relies on careful leader selection; naive Paxos implementations risk livelock under contention.

Understandability / Implementability:- Raft was explicitly designed for understandability: clear leader role, explicit election, log replication, and snapshot mechanics. Its state machine and invariants are easier to reason about and implement correctly.- Paxos (and its Multi-Paxos optimization) is more minimal and mathematically elegant but harder to get right in an engineering implementation — many subtle corner cases (reconfiguration, log compaction) are tricky.

When to pick which:- Pick Raft when you want faster, safer engineering delivery: easier to implement, debug, and maintain; many production-quality implementations/libraries exist (etcd/consensus modules). Good for most new services needing consensus for configuration, metadata, or small-scale replicated state.- Pick Paxos if you need compatibility with an existing Paxos-based ecosystem, or you require a protocol variant with specific theoretical properties (e.g., certain optimizations or formal proofs), and you have expertise to implement and validate it. For highly customized consensus research or when integrating with systems already using Paxos families, use Paxos.

Trade-offs summary:- Raft = engineering-friendly, predictable liveness tuning, widespread ecosystem.- Paxos = compact theory, potentially more flexible, higher implementation risk.

Recommendation: For a typical new service, choose Raft for faster, less error-prone delivery unless you have a strong reason to prefer Paxos.

HardSystem Design

31 practiced

Design an observability plan for microservices running across three regions to reliably trace a user transaction end-to-end. Include sampling strategy, how to propagate trace context and baggage, tag/attribute design for low-cardinality metrics, storage/retention trade-offs and how to balance signal quality with cost.

Sample Answer

Requirements & constraints:- End-to-end trace of a user transaction across 3 regions with <500ms added latency for instrumentation, support root-cause and SLO debugging, cost bounded, high cardinality user IDs must not explode metrics.

High-level architecture:- Local sidecars in each region collecting traces/metrics (e.g., OpenTelemetry Collector) → regional ingest/agent → central tracing backend (managed or self-hosted, e.g., Jaeger/Tempo or vendor) with long-term metrics in a TSDB (Prometheus remote write -> Cortex/Thanos) and traces in object store cheaper tier.

Trace context & baggage propagation:- Use W3C Trace Context (traceparent) for trace identifiers and tracestate for vendor hints. For minimal baggage, allow only a few well-defined keys (e.g., "session_type", "feature_flag") and enforce max size (<= 512 bytes) at service boundary. Strip/ignore unknown baggage at ingress to prevent blowups. Libraries instrument HTTP/gRPC to automatically propagate headers.

Sampling strategy:- Multi-tier sampling: 1. 100% lightweight recording of trace IDs and low-cardinality metadata (service, region, route, status, latency bucket) for metrics and correlation. 2. Head-based adaptive sampling at edge: probabilistic base rate (e.g., 1%) + rules to keep all traces that are errors, high latency (>SLO threshold), or match important transactions (payment, checkout). 3. Tail-based sampling at regional collectors: keep additional traces if anomalies detected (spike in error rate), using buffered window (e.g., 30s) to retroactively retain interesting traces.- Ensure sampling decision stored in trace flags so downstream keeps consistent decision.

Tag/attribute design for low-cardinality metrics:- Only attach low-cardinality attributes to metrics: service, region, environment, endpoint name (normalized), HTTP status class (2xx/3xx/4xx/5xx), SLO bucket.- High-cardinality data (user_id, email, device_id, full URL) go to trace spans only (not to aggregated metrics). For occasional user-level debugging, record hashed user_id or a sampled user_id tag with controlled cardinality (e.g., keep at most 1k unique user samples per hour).

Storage & retention trade-offs:- Hot path: store full traces for last 7–14 days in fast-access store for debugging.- Cold path: move older traces (30–90 days) to object store (S3) with compressed chunks and index only sampled keys.- Metrics: keep high-resolution (10s) for 14 days, 1m for 90 days, aggregated daily for 13 months.- Use tiered storage and retention policies driven by business importance and compliance.

Balancing signal quality with cost:- Prioritize trace retention for error/slow traces and critical flows; apply lower sampling for background jobs.- Use envelope of deterministic rules + ML/anomaly detector to increase sampled traces only when signal suggests value.- Monitor instrumentation overhead (CPU, network) and evaluate ROI: if certain spans rarely aid debugging, reduce instrumentation level.- Periodic audits: review top N high-cardinality tags and remove or convert them to span-only annotations; enforce tag whitelist.

Operational considerations:- Provide developers SDKs with default sampling config and easy overrides per-service.- Dashboards: link traces from metrics (trace_id in logs), use trace sampling heatmaps.- Security/privacy: redact PII before adding to spans; enforce encryption in transit and at rest.- Test plan: simulate cross-region failure, verify trace continuity, and validate tail-sampling retention.

This plan ensures high signal for debugging critical transactions while keeping cardinality low and costs predictable.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Principles and Tradeoffs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.