Distributed Systems Troubleshooting Questions

Focused on diagnosing incidents specific to distributed architectures and multi service systems. Candidates should be able to detect and analyze network latency packet loss service to service communication failures cascading failures load balancing misconfiguration and data consistency anomalies. The topic covers observability practices such as distributed tracing aggregated metrics and logs correlation identifiers health checks and alerting; instrumentation strategies for cross service request flow mapping; and remediation patterns such as timeouts retries circuit breakers backpressure and resynchronization. Interviewers assess the ability to reason about partitioning and consistency models reproduce issues safely across services and propose mitigation and longer term fixes for distributed failure modes.

HardTechnical

26 practiced

Root-cause analysis case: provided signals below, produce a short RCA describing the causal chain, immediate remediation steps, and long-term fixes.

Metrics: service-a p95=800ms, error-rate=10% for last 15mTraces (sample): gateway -> service-a (span 600ms) -> service-b (span 580ms) -> redis (span 560ms)Logs from service-b: repeated 'ERR connect timeout to redis host:6379'

Write an RCA summary: causal chain, short-term mitigations, and long-term engineering/operational fixes.

Sample Answer

Summary (root cause):- Primary cause: Redis became unreachable/latent (connection timeouts) causing service-b's Redis calls to block ~560ms per request. Service-b threads/requests queued and timed out, producing repeated "ERR connect timeout to redis host:6379".- Causal chain: Redis connectivity/timeouts → service-b RPCs to Redis slow/fail → service-b spans ≈580ms and returns errors/slow responses → service-a waits on service-b (span 600ms) → service-a p95 spikes to 800ms and error-rate increases to 10% → gateway surfaces user-visible errors.- Probable contributing factors: insufficient Redis capacity/connection limits, exhausted connection pool or network issue between service-b and Redis, lack of fail-fast/circuit-breaker and inadequate observability on Redis resource saturation.

Immediate remediation (short-term, action within minutes–hours):1. Failover/restore Redis: promote replica or restart Redis instance if healthy; confirm cluster/network routes.2. Reduce incoming load: temporary traffic throttle or enable rate-limiting at gateway to protect downstream.3. Clear stuck resources: restart service-b (or recycle worker processes) to free blocked connections; if using connection pooling, restart to reset pools.4. Apply emergency configuration: lower service-b Redis client timeout to fail faster (short term) and enable fail-open behavior or return cached/partial responses.5. Re-route traffic to healthy regions/replicas if available; notify on-call and stakeholders.

Long-term fixes (engineering + operational):- Resilience patterns: - Implement circuit breakers and bulkheads in service-b to fail fast and prevent thread/connection exhaustion. - Add client-side timeouts, retry with exponential backoff and jitter, and fallback responses (caching, degraded mode).- Capacity & configuration: - Tune Redis connection pool, increase max connections, and ensure proper keepalive/timeouts. - Right-size Redis (memory, CPU) and add replicas/sharding/cluster for capacity and HA; enable automatic failover.- Observability & alerting: - Add Redis-specific metrics (connection usage, blocked clients, latency percentiles, slowlog) and alert thresholds. - Link trace/span alerts to downstream resource alerts so correlated symptoms trigger automated runbooks.- Automation & runbooks: - Create runbook for Redis connectivity incidents (failover steps, sanity checks, rollback). - Automate failover and graceful degradation paths (circuit-breaker triggers → serve cached data).- Testing & SLOs: - Define SLOs/error budgets for service-b and Redis; run chaos/chaos-net tests (network partitions, Redis failure) to verify degradations and recovery. - Load-test Redis and service-b to validate connection limits and latency under production-like traffic.

Expected outcome:- Short-term actions reduce p95 and error rate by removing the Redis bottleneck and preventing service-b from exhausting resources.- Long-term fixes prevent recurrence by making service-b tolerant of Redis issues and ensuring Redis capacity/HA and faster detection/automation.

MediumTechnical

22 practiced

A Kafka consumer group's lag is steadily increasing for a topic. Describe a structured troubleshooting approach: which metrics (consumer throughput, processing time, rebalance frequency) and logs to inspect, how to determine whether producer, broker, or consumer is the bottleneck, and short- and long-term remediations (parallelism, partitioning, backpressure).

Sample Answer

Start with a clear hypothesis loop: measure → isolate → validate → remediate.

1) Immediate metrics to check- Consumer: records-consumed/sec, fetch-latency, max-poll-interval, commit-latency, processing time per-record, consumer-lag (per-partition).- Broker: bytes-in/bytes-out, request-latency, under-replicated-partitions, ISR changes, disk I/O, network throughput.- Producer: records-produced/sec, produce-latency, batch-size, retries/acks.- Cluster-wide: partition distribution, consumer group offsets, rebalance frequency, JVM GC, CPU/memory on clients and brokers.

2) Logs to inspect- Consumer client logs: deserialization errors, poll timeout, commit failures, frequent rebalances, exceptions.- Broker logs: leader election, disk errors, throttle logs, network errors.- Producer logs: failed sends, timeouts, throttling messages.

3) How to determine the bottleneck- If producer rate << broker bytes-in or topic log end offset not advancing → producer-side.- If log end offset advancing but consumer offsets lag growing and consumer CPU/GC or processing-time/sec high → consumer-side processing bottleneck.- If end offset not moving and brokers show high request-latency, I/O saturation, or throttling → broker-side.- Use A/B tests: stop consumers—if log end offset keeps growing quickly, producers are healthy; stop producers—if consumers catch up, producer was source.

4) Short-term remediations (low friction)- Restart stuck consumers if they’re stuck in long GC or hung.- Add more consumer instances (if partition count allows) or temporarily increase consumer parallelism (thread pools) to consume backlog.- Apply client-side backpressure: slow producers or increase retention to avoid data loss.- Increase fetch.max.bytes / max.poll.records carefully to improve throughput.

5) Long-term fixes- Repartition topic to increase parallelism (only with careful coordination).- Optimize consumer processing: batch processing, async I/O, backpressure-aware design, increase instance resources.- Tune broker hardware/IO/network or scale brokers, adjust retention/segment size, and enable partition reassignment for balance.- Implement better monitoring/alerting on lag thresholds and capacity planning; enforce SLOs and runbooks.

6) Trade-offs and caution- More partitions increase throughput but complicate ordering and consumer rebalances.- Increasing parallelism inside a consumer can complicate offset commits—use idempotency or transactional patterns.- Repartitioning requires producer/consumer compatibility and may need rolling deployments.

Conclude by recommending a runbook: monitor per-partition lag, isolate with stop/start tests, apply short-term fixes to restore service, then plan long-term capacity/architecture changes validated by load tests.

MediumTechnical

21 practiced

You suspect packet loss between two microservices in the same cluster causing TCP retransmits and increased response times. Which tcpdump or Wireshark filters and commands would you run to detect retransmissions and packet drops? What TCP fields and packet patterns indicate loss or duplicate ACKs? Provide example tcpdump commands you would use to capture relevant traffic.

Sample Answer

Approach: capture the TCP conversation between the two microservices, then inspect for retransmissions, out-of-order packets, duplicate ACKs, zero-window, and SACK info. Use tcpdump for capture and Wireshark/tshark display filters to detect patterns.

Useful TCP fields / packet patterns that indicate loss:- Retransmission: same TCP sequence number sent again. Wireshark tags: tcp.analysis.retransmission or tcp.analysis.fast_retransmission.- Fast retransmit: sender retransmits after triple duplicate ACKs; look for tcp.analysis.fast_retransmission and preceding 3+ duplicate ACKs (same ACK number, increasing repetition).- Duplicate ACKs: ACKs with identical ack field repeated (tcp.analysis.duplicate_ack).- Out-of-order: tcp.analysis.out_of_order — receiver sees seq not matching expected.- Zero window: window size = 0 (TCP header field) indicates receiver can't accept data.- SACK blocks: presence or absence of SACK options helps infer recovery behavior.- Retransmit due to timeout: retransmit with longer RTT and no preceding duplicate ACKs.- TCP reset (RST) or excessive retransmits → connection issues.

tcpdump capture examples (replace IP_A, IP_B, PORT):

bash

# Capture all TCP between two hosts on interface eth0, full packet, write pcap
sudo tcpdump -i eth0 -s 0 -w /tmp/micro.pcap host 10.0.1.5 and host 10.0.1.8 and tcp

# Capture specific service port (e.g., port 8080)
sudo tcpdump -i eth0 -s 0 -w /tmp/micro.pcap host 10.0.1.5 and host 10.0.1.8 and tcp port 8080

# Capture and print verbose packets with sequence/ack info (useful short live check)
sudo tcpdump -i eth0 -n -vvv 'host 10.0.1.5 and host 10.0.1.8 and tcp' -c 200

Inspect with Wireshark/tshark:

bash

# Open in Wireshark: analyze tcp.analysis.* flags, follow TCP stream, enable expert info
wireshark /tmp/micro.pcap

# Use tshark to list retransmissions / duplicate ACKs
tshark -r /tmp/micro.pcap -Y "tcp.analysis.retransmission || tcp.analysis.fast_retransmission || tcp.analysis.duplicate_ack" -T fields -e frame.number -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e tcp.seq -e tcp.ack -e tcp.window_size

# Show duplicate ACK count trends
tshark -r /tmp/micro.pcap -Y "tcp.analysis.duplicate_ack" -T fields -e frame.number -e tcp.ack

Quick live Wireshark display filters:- tcp.analysis.retransmission- tcp.analysis.fast_retransmission- tcp.analysis.duplicate_ack- tcp.analysis.out_of_order- tcp.window_size == 0- tcp contains "SACK" or tcp.options.sack

Troubleshooting steps:- Capture both sides (client & server IPs) to confirm where retransmit originates.- Correlate timestamps and RTTs (tcp.analysis.ack_rtt) to see if retransmits follow timeouts.- Check NIC error counters, kube CNI logs, and iptables/iptables-save for drops if retransmits are frequent.- If you see many duplicate ACKs followed by fast retransmit → likely intermediate packet loss. If retransmits without duplicate ACKs and long delay → possible path loss or severe congestion/timeouts.

Edge notes:- In high-throughput environments use -s 0 and write to disk to avoid truncation.- Add 'and not port 22' to tcpdump filter to avoid SSH noise when capturing from remote hosts.

EasyTechnical

27 practiced

Describe the circuit breaker pattern for service-to-service communication. Explain the main states (closed, open, half-open), which metrics should drive state transitions (error rate, consecutive failures, latency), and how you'd tune timeout windows and probe behavior in half-open. Also discuss risks of aggressive breakers and interactions with retries.

EasyTechnical

26 practiced

You're on-call and receive a pager: Service-A error rate jumped from 0.2% to 5% starting 3 minutes ago; CPU and memory look normal. Outline a 10-minute triage plan: what dashboards/metrics/logs/traces/commands you check first, how you determine blast radius, what temporary mitigations you might apply, and what you communicate to stakeholders in the first 10 minutes.

Sample Answer

Minute 0–1: quick situational snapshot- Acknowledge the pager, set a 10‑minute timer, and open the primary incident channel (Slack/PagerDuty incident room). Post initial blameless note: “Investigating Service‑A error rate spike (0.2%→5%) starting ~3min ago. CPU/mem normal per alert. Will update every 5 mins.”- Assign yourself as lead and request on-call backup if available.

Minute 1–4: triage dashboards and high-level signals- Check service SLOs/error budget dashboard and global health panel (requests/sec, error rate, latency P50/P95/P99).- Review dependency heatmap (downstream/upstream errors) and deployment/CI dashboard for recent deploys in last 30 min.- Inspect load balancer / API gateway metrics (5xx by backend), and ingress rates to rule out traffic surge or malformed requests.- Quick commands: kubectl get pods -n svc-a -o wide; kubectl logs -l app=service-a --since=5m | tail -n 200; check autoscaler and node health (kubectl top nodes).

Minute 4–7: dig into logs and traces to find error pattern- Search logs for common error signatures and timing (rate-limited, auth failures, DB timeouts, 5xx code distribution) and group by endpoint, client IP, user agent, or region.- Check distributed traces for increased error span counts / root cause service. Use sampling to find a recent failing trace.- Determine blast radius: is it all instances, one AZ, a region, specific endpoints, or a client subset? Use filters: instance ID, pod name, region, API key, client IP range.

Minute 7–9: temporary mitigations- If a bad deploy detected: roll back to last known good revision (or divert traffic to previous version).- If error concentrated on particular endpoint or heavy client: add a temporary rate limit or feature flag to disable noncritical functionality.- If downstream DB/service is failing: enable circuit breaker / degrade gracefully (return cached results), increase timeouts cautiously.- If healthy instances exist: scale up replicas or shift LB weights away from failing AZ/instances.

Minute 9–10: communications and next steps- Post an update in incident channel: what you’ve observed (error rate, affected endpoints/regions, suspected cause or “investigating”), immediate mitigation applied (rollback/rate‑limit), and ETA for next update (5–15 min).- Notify stakeholders: alert oncall owner, page dev lead if deploy suspected, inform product on potential customer impact and whether to pause releases.- Create incident ticket with timeline, commands run, and links to dashboards/log queries.

Key reasoning: prioritize signals that narrow scope quickly (is it traffic, deploy, or dependency). Apply low-risk mitigations first (traffic steering, rate limits, rollback) and communicate often with concise facts and next ETA.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.