Distributed Systems Fundamentals Questions

Core principles and theory that underlie distributed computing systems. Includes understanding trade offs between consistency, availability, and partition tolerance, common consistency models such as eventual and strong consistency, replication and sharding strategies, load balancing and data partitioning, consensus algorithms and their guarantees, scalability and fault tolerance patterns, and how these concepts apply to infrastructure components such as databases, caches, service meshes, and load balancers. Candidates are expected to explain design choices, common failure modes, and how fundamental concepts influence architecture decisions for resilient and scalable systems.

HardSystem Design

68 practiced

Design monitoring and alerting to detect silent data corruption in a distributed object store. Include integrity verification methods (checksums, Merkle trees), sampling frequency, repair workflows, how to avoid false positives, and how an SRE team would prioritize repairs to minimize user impact while preserving durability guarantees.

Sample Answer

Requirements & constraints:- Detect silent data corruption (bit flips, disk/controller bugs) with high confidence.- Minimize user-visible errors and repair-induced load.- Preserve durability (e.g., N-replication or erasure coding).- Reasonable operational cost.

High-level design:- Hybrid integrity checks: per-object strong checksums (e.g., CRC64 or BLAKE2b) at write time + Merkle trees per shard for scalable range verification + optional content-addressed hashes (SHA-256) for critical objects.- Metadata stores checksum version and validation status; consistent hashing maps objects to replicas.

Integrity verification strategy:- Writes: compute and store checksum in object metadata and update Merkle tree leaf.- Reads: validate checksum on read-path; if mismatch, serve from healthy replica + trigger repair.- Background scrubbing: - Continuous light-weight sampling: random 0.1–0.5% of objects per hour per storage node to detect widespread issues quickly. - Targeted full scrubs: run offline full Merkle-verify per shard weekly/monthly depending on data criticality and error rates. - Priority scrubs for newly attached disks, after firmware updates, or if SMART signals degrade.

Repair workflow:1. Detection (read-time or scrub) → mark object as suspect.2. Cross-check replicas: fetch checksum/hash from other replicas and Merkle proofs to avoid full downloads.3. If majority agrees on a correct copy, overwrite corrupt replica atomically and update Merkle tree and metadata.4. If all replicas disagree, escalate to data-loss workflow: try reconstruct from erasure-coded fragments or use latest object version from backups; alert SRE/owners.

Avoiding false positives:- Require multi-step confirmation before treating as corruption: - Re-read same block; re-check checksum. - Consult Merkle proof and per-replica metadata. - Use quorum-based validation: only declare corruption when local replica disagrees with >=1 healthy replica (or >N/2 in quorum systems).- Timestamps and version checks ensure transient write-in-flight states aren’t misclassified.

Monitoring & alerting:- Metrics: - Corruption detected/sec, confirmed repairs/sec, scrub coverage %, per-disk and per-node corruption rate, repair latency, repair throughput, SMART/failure signals.- Alerts: - P0: multiple confirmed corruptions in short window on same node/disk → immediate on-call page (possible hardware fault). - P1: rising corruption rate above baseline → paged to SRE with automated targeted scrubs. - P2: single confirmed corruption repaired → paged to on-call only if >SLO threshold.- Dashboards: heatmap by shard, trend lines, scrub backlog.

Prioritization to minimize user impact & preserve durability:- Score objects by: criticality (service SLO), access frequency, number of healthy replicas, time-since-last-verified, and repair cost (network/CPU).- Schedule repairs: 1. Hot objects (high read rate + any corruption) — immediate repair from healthy replica, prioritized to restore user reads. 2. Objects with insufficient healthy replicas (e.g., only one good replica left) — high priority to avoid durability loss. 3. Cold objects with full redundancy — background repair when cluster load low.- Rate-limit repairs per node and use bandwidth shaping to avoid impacting tail latency; favor local repairs when possible to reduce cross-dc traffic.- For erasure-coded objects, reconstruct minimal fragments needed; prefer repair to same availability zone to keep rebuild cost low.

Operational playbooks & automation:- Automated confirm-and-repair runbooks (idempotent), with manual escalation only for ambiguous or multi-replica mismatch cases.- Auto-trigger targeted deep-scrub after firmware updates or SMART anomalies.- Periodic drills and post-incident blameless reviews; collect metrics to tune sampling frequency and thresholds.

Trade-offs:- Stronger hashes and more frequent scrubs increase CPU/I/O cost; mitigate with sampling, Merkle summary checks, and quorum confirmation to reduce false positives.- Faster repair reduces window of vulnerability but increases load—use prioritization and rate-limiting.

This approach balances early detection, low false-positive rate, automated safe repairs, and SRE-prioritized actions to preserve durability and minimize user impact.

HardTechnical

69 practiced

Your JVM-based services are experiencing rare but severe tail latencies (99.99th percentile) due to long garbage collection pauses. As an SRE, propose mitigation strategies at the JVM configuration level (GC selection/tuning), application level (object allocation reduction), and system level (container limits, CPU pinning). Describe how you would measure improvements and safely roll out changes.

Sample Answer

Situation: We have rare but severe 99.99th‑percentile pauses traced to GC. As an SRE I’d attack this in three layers (JVM, app, system), measure with targeted observability, and roll changes out safely.

JVM-level mitigations (selection & tuning)- Choose a low‑pause collector: prefer ZGC or Shenandoah for very large heaps and sub‑millisecond pauses; G1 with tuned pause targets if Z/Shen unavailable.- Example flags (G1): -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ParallelRefProcEnabled- Example flags (ZGC): -XX:+UseZGC -Xmx8g- Heap sizing: right‑size Xms=Xmx to avoid expansion pauses; avoid excessive heap > memory pressure on hosts.- GC ergonomics: tune region size (G1), survivor ratios, and concurrent thread counts (-XX:ConcGCThreads, -XX:ParallelGCThreads).- Enable detailed GC logging for analysis: -Xlog:gc*:file=/var/log/gc.log:time,uptime,level,tags

Application-level mitigations (reduce allocation & latency amplification)- Reduce allocations: reuse buffers (ByteBuffer pools), avoid short‑lived temporary objects in hot paths, use primitive arrays, and use StringBuilder vs concatenation in loops.- Leverage escape analysis/stack allocation by the JIT (keep methods simple).- Use off‑heap or memory-mapped IO for very large caches.- Batch and coalesce I/O and logging to avoid amplification of GC pressure.- Add allocation telemetry (allocation profiler, async-profiler) and hotspot flamegraphs to find alloc hotspots.

System-level mitigations (containers/CPU/memory)- Container limits: give JVM exclusive memory, set Xmx less than container memory; use cgroup v2 with memory.high instead of abrupt OOM.- CPU pinning / isolcpus: isolate GC threads to dedicated cores or use cpuset to avoid CPU contention with noisy neighbors.- Set cpu.shares and cpuset for predictable CPU; ensure NUMA locality and set -XX:+UseNUMAAffinity if appropriate.- Use hugepages for large heaps to reduce TLB pressure.- Avoid oversubscription on nodes; reserve cores for GC if needed.

Measurement & instrumentation- Baseline: capture p50/p95/p99/p99.9/p99.99 latency histograms, error rates, throughput.- GC observability: parse GC logs, JFR recordings, jstat, and export GC pause durations to Prometheus (histograms with explicit high buckets to capture 99.99).- Profiling: async-profiler/jcmd/jfr to identify allocation callsites and stop-the-world durations, flamegraphs for CPU and allocations.- Correlate pauses with CPU steal, cgroup throttling, host OOM/killing, and network/backpressure events.

Safe rollout plan1. Local & staging validation: reproduce with stress tests (heap sizes, traffic patterns) and measure GC pause distributions.2. Canary: deploy to a small percentage of traffic or isolated hosts. Compare histograms and GC logs against baseline.3. Gradual ramp: increase canary scope while monitoring SLOs and alerts for regressions; keep automated rollback triggers on increased p99.99 or error rate.4. Communication & runbook: document expected behavior, thresholds, and rollback steps; schedule windows for risky changes like collector switches.5. Post‑rollout: keep elevated monitoring for 24–72 hours, capture JFR on anomalies, and run postmortem if regressions occur.

Tradeoffs & safeguards- ZGC/Shenandoah reduce pause times but may use more CPU; validate throughput impact.- Lowering Xmx reduces GC work but increases OOM risk—monitor OOMs closely.- CPU isolations improve latency but reduce packability and utilization.

This approach combines concrete JVM flags, application refactors, and system-level isolation, backed by profiling, metrics, and a conservative canary rollout to eliminate rare, severe GC tail latencies while safeguarding availability.

HardTechnical

81 practiced

Design an experiment and a practical detection algorithm to identify 'retry storms'—situations where many clients repeatedly retry failing requests causing cascading failures. Describe instrumentation required (client-side and server-side), detection thresholds (e.g., error-rate x retry-rate correlation), automated mitigation controls (exponential backoff enforcement, bulkheads, circuit-breakers), and a test plan to verify fixes.

Sample Answer

Situation: We want an experiment and a practical detection algorithm to catch "retry storms"—high-volume client retries causing cascading failures.

Experiment design (overview)- Inject controlled failures (HTTP 5xx or TCP resets) to a subset of service instances for a short window (30–120s) and observe client and server telemetry to validate detection and mitigations.- Run at low traffic hours against canary traffic (1–5% of users) with kill-switch and rollbacks.

Instrumentation required- Client-side: - Emit events: request_id, timestamp, endpoint, outcome (success/fail), retry_count, backoff_policy, client_id/region. - Sampled logs and metrics: retry_rate = retries / total_requests per minute; time-between-retries histogram.- Server-side: - Per-endpoint request_rate, error_rate (5xx), queue_length, latency_p95/p99, active_connections. - Per-client and per-IP counters: requests/min, retries/min, rejected/min. - Tracing: propagate parent IDs to link initial request -> retries.

Detection algorithm (practical)- Compute rolling windows (1m, 5m) per (client_id, endpoint, region): - error_rate = failed_requests / total_requests - retry_ratio = retry_requests / total_requests- Alert when: - error_rate > E_thresh (e.g., 5%) AND retry_ratio > R_thresh (e.g., 20%) AND retry_rate_growth > G_thresh (e.g., 2x over 1m) - OR server KPIs degrade: queue_length ↑ by 3x and p99 latency ↑ by 2x concurrent with retries.- Use correlation score: score = w1*norm(error_rate)+w2*norm(retry_ratio)+w3*norm(latency_increase); trigger if score > 0.8.

Automated mitigation controls- Enforce server-side rate limits and bulkheads: per-client concurrent requests cap, token bucket with small burst.- Global/canary circuit-breaker: when score high, return 429 with Retry-After or 503 with Accept-Retry header.- Backoff enforcement gateway: if client does not respect exponential backoff, gateway performs backoff on behalf (rejects or delays requests) and marks client for progressive throttling.- Graceful shedding: route excess traffic to degraded read-only or cached responses.- Feedback loop: emit metrics when mitigation applied; auto-scale only if sustained legitimate load (distinguish retries vs new users).

Test plan to verify fixes1. Unit tests: simulate client behavior (honoring/ignoring backoff) and verify detection score and mitigation decisions.2. Integration canary: run injected-failure experiment on 1% traffic; confirm detection triggers and automated mitigations reduce server load by >50% within 60s.3. Chaos test: random partition + mass-retry simulation, verify no cascading failures and SLOs remain within error budget.4. Postmortem: validate logs/traces for root cause, false-positive rate < 1% across tests, and rollback path works.

Key trade-offs- False positives vs availability: conservative thresholds reduce accidental shedding but increase risk of cascade.- Client cooperation: prefer server-side protective controls because not all clients implement backoff.

This plan gives measurable detection, automated protection, and a testable verification path.

EasyTechnical

62 practiced

List common failure modes in distributed systems (e.g., network partition, node crash, disk full, resource exhaustion, time skew). For each failure mode give a one-line operational mitigation SREs can deploy quickly to reduce customer impact and name a monitoring metric or alert that would indicate the problem.

Sample Answer

1) Network partition — Mitigation: Enable traffic routing to healthy regions (failover/DNS weight shift) and increase request retries with exponential backoff. Metric/alert: inter-region packet loss or TCP connection failure rate > X% and increased cross-region RPC latency.

2) Node crash — Mitigation: Auto-restart via orchestration (k8s restartPolicy / auto-scaling group replacement) and route traffic away during recovery. Metric/alert: node/process crash count or pod restart rate > threshold.

3) Disk full — Mitigation: Trigger automated log rotation/retention and free space script or remove noncritical data, then throttle writes. Metric/alert: disk utilization (%) > 85% or inode exhaustion.

4) Resource exhaustion (CPU/memory) — Mitigation: Auto-scale replicas, OOM killer tuning, and limit/cgroup enforcement; restart runaway processes. Metric/alert: sustained CPU or memory usage > set baseline or OOM events.

5) Time skew / NTP drift — Mitigation: Re-sync NTP/chrony and quarantine divergent hosts; rely on monotonic clocks for IDs. Metric/alert: host clock offset > 100ms or frequent TLS/kerberos errors.

6) Service overload / high latency — Mitigation: Enable rate limiting / circuit breakers and shed nonessential traffic; scale horizontally. Metric/alert: p95/p99 latency spike or error rate increase.

7) Dependency failure (DB/cache down) — Mitigation: Failover to replicas, switch read-only mode, or return degraded responses. Metric/alert: dependency connection errors, increased DB connection latency, or cache miss storm.

8) Configuration drift / bad deploy — Mitigation: Roll back deploy, apply known-good config, and enable feature flags. Metric/alert: deployment failure rate, surge in error budgets, or config checksum mismatch.

9) Security compromise — Mitigation: Isolate affected hosts, revoke credentials, and apply emergency patch. Metric/alert: unusual auth failures, elevated outbound traffic, or IDS alerts.

10) Hardware degradation (disk I/O, NIC errors) — Mitigation: Evacuate workload from host and replace hardware; use degraded-IO mode. Metric/alert: disk I/O error rate, SMART warnings, or interface error counters.

Each mitigation should be codified as runbooks and automated where possible to reduce MTTR.

EasyTechnical

66 practiced

Explain the difference between replication and sharding. As an SRE, describe a scenario where you would use replication primarily for availability and durability, and another where you'd shard primarily for scale. Discuss operational trade-offs such as failover complexity, metadata overhead, and rebalancing pain during node failures.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.