Advanced Debugging and Root Cause Analysis Questions

Systematic approaches to complex debugging scenarios: intermittent failures, race conditions, environment-dependent issues, infrastructure problems. Using logs, metrics, and instrumentation effectively. Differentiating between automation issues, environment issues, and application defects. Experience with advanced debugging tools and techniques.

HardSystem Design

28 practiced

Design an instrumentation plan to capture root-cause data for intermittent errors in a high-throughput system without introducing significant overhead. Include options like eBPF sampling, conditional tracing triggered by anomaly thresholds, on-demand debug snapshots, and ephemeral detailed captures around detected anomalies.

Sample Answer

Requirements & constraints:- Capture root-cause data for intermittent, low-rate errors in a high-throughput service without adding measurable latency or CPU overhead.- Data must be selective (avoid continuous full-tracing), fast to enable diagnosis, privacy-aware, and storage-cost-controlled.- Support production, on-demand, and post-incident investigation modes.

High-level approach:- Multi-tiered, low-overhead baseline telemetry + dynamic escalation to richer captures only around detected anomalies.

Architecture / components:1. Lightweight baseline telemetry - Metrics (prometheus): error rates, latency histograms, tail latency, percentiles. - Structured logs with sampling (1-5% by default) and minimal context (request id, route, status). - Health guards (SLO-based alerts + anomaly detector).

2. eBPF continuous sampling - Kernel-space eBPF probes to sample stack traces, syscall/IO patterns, and per-pid resource counters at low rate (e.g., 1–10Hz aggregated per service instance). - Aggregate and transmit summaries (heatmaps of stacks, syscall frequency) to reduce volume. - Extremely low overhead; safe sandboxed observers.

3. Conditional tracing (dynamic) - Tracing agent (OpenTelemetry/Jaeger) in “standby” with tracing headers but not recording payloads. - Trigger policy: when metric anomaly or error spike crosses threshold (absolute or statistical), flip targeted instances or trace ids to “record full trace” for a short window (e.g., 30–120s). - Policy can include sampling keys (user id, request path) to bias capture to suspicious requests.

4. On-demand debug snapshots - Operator API to request ephemeral capture on specific instance(s): full flamegraph, heap/profile, thread stacks, and request context. Snapshots are time-bounded and stored in secure ephemeral storage. - Use sidecar to collect app-level context (goroutine stacks, request headers) with safe-rate limits.

5. Ephemeral detailed captures around anomalies - When an anomaly detector marks a request as likely problematic (e.g., repeated retries, long tail > threshold), automatically attach detailed capture for that request only: full trace, request/response bodies (masked), environment, and eBPF stack sample before/after. - Store these artifacts with TTL (e.g., 7 days) and size quota.

Data flow:- Metrics/anomaly detector ➜ on anomaly → orchestrator instructs agents (via control-plane: Kafka/consul/agent API) to enable conditional tracing/eBPF uplift on selected hosts/trace-ids → agents stream rich artifacts to secure object store + indexed metadata to observability backend.

Scalability & overhead controls:- Multi-level sampling: baseline metrics (100%), logs sampled low, eBPF low-frequency sampling, conditional tracing only on triggers.- Rate limits, quotas per service, instance-level CPU cap for eBPF/tracing.- Aggregate on-host summaries to reduce network egress.- Progressive backoff: if capture rate too high, widen thresholds or reduce capture window.

Privacy & security:- Redact PII at capture point, use encryption in transit/rest, RBAC for access to debug snapshots.- Audit logs for who triggered captures.

Trade-offs:- Pros: minimal steady-state overhead, high fidelity when needed, fast targeted root-cause.- Cons: may miss some rare events between samples; requires robust anomaly detection to trigger captures and operator discipline to manage artifacts/quota.

Operational considerations:- Test policies in staging with synthetic anomalies.- Build dashboards showing capture rate, storage consumption, and false-positive rate.- Post-incident: use captured artifacts for RCA and refine trigger thresholds and sampling biases.

This plan balances continuously low overhead telemetry with powerful, ephemeral deep-capture mechanisms (eBPF, conditional tracing, on-demand snapshots) to reliably surface root causes for intermittent errors while controlling cost and risk.

HardTechnical

31 practiced

Intermittent TLS handshake failures occur between services across datacenters. Describe a troubleshooting plan that includes collecting endpoint TLS logs, using openssl s_client, taking packet captures (tcpdump/wireshark), checking cipher-suite compatibility, MTU/fragmentation issues, certificate validation errors, and proxies/load-balancer timeouts. List commands and what to look for.

Sample Answer

Start with scope & reproduce: identify affected services, frequency, client/server IPs, datacenters, times. Capture an incident window for correlated data.

1) Collect endpoint TLS logs- App/server TLS (nginx, envoy, haproxy, JVM): increase log level to DEBUG for TLS handshakes, enable TLS library logs (OpenSSL/boringssl).- What to look for: handshake errors (SSL routines), timeouts, alert descriptions (e.g., decrypt_error, bad record mac, certificate verify failed), client hello/cipher selection logs, session resumption attempts.

Examples: enable more verbose Envoy logs or JVM:

bash

# For nginx: increase error_log level, reload
sed -i 's/error_log .*;/error_log \/var\/log\/nginx\/error.log debug;/' /etc/nginx/nginx.conf
nginx -s reload

2) Use openssl s_client to exercise and inspect TLS- Basic connect, show cert chain:

bash

openssl s_client -connect host:443 -servername example.com -showcerts -state -debug

- Force specific TLS version / ciphers to reproduce:

bash

openssl s_client -connect host:443 -tls1_2 -cipher 'ECDHE-RSA-AES128-GCM-SHA256'

- What to look for: successful handshake, certificate chain, verify return code, ServerHello cipher, session resumption, SNI behavior, any errors printed (handshake failure, no ciphers offered).

3) Packet capture (tcpdump/Wireshark)- Capture during failure window on both sides if possible:

bash

sudo tcpdump -i eth0 host A.B.C.D and port 443 -w /tmp/tls_hs.pcap

- If high traffic, use capture filters for SYN/ACK/TLS handshake bytes:

bash

tcpdump -w /tmp/hs.pcap 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 or port 443'

- In Wireshark, filter by: - tls.handshake.type == 1 (ClientHello) or tls.handshake.type == 2 (ServerHello) - tcp.analysis.retransmission, tcp.analysis.rtt, ip.frag- What to look for: - Is ClientHello sent and ServerHello returned? Partial handshakes? TLS Alerts (e.g., fatal) - Retransmissions, zero window, or resets (RST) mid-handshake - IP fragmentation or overlapping fragments - Differences between successful and failed traces (cipher negotiation, SNI present/missing)

4) Cipher-suite compatibility- Compare ClientHello cipher list vs ServerHello chosen cipher from traces/openssl.- Check server config for allowed ciphers and TLS versions; check client libraries (old OpenSSL, Java 8u versions) may lack modern ciphers.- Use test matrix:

bash

# list ciphers client can propose
openssl ciphers -v 'ALL:eNULL' | column -t

- What to look for: missing common ciphers, server preferring unsupported cipher, TLS version downgrade/renegotiation problems.

5) MTU / fragmentation issues- Test large TLS handshake packets (ClientHello with many extensions/certs can exceed MTU).- Determine path MTU and fragmentation:

bash

# from client: find PMTU (Linux)
tracepath -n host
# ping with DF to test fragmentation
ping -M do -s 1472 host

- What to look for: ICMP "fragmentation needed" messages, ClientHello fragmented across IP packets or blocked by middleboxes; Wireshark shows IP fragmentation or TLS records split abnormally. If fragmentation causes failure, reduce MSS/MTU on endpoints or disable TCP Segmentation Offload (TSO/GRO) for debugging:

bash

sudo ethtool -K eth0 tso off gso off gro off

6) Certificate validation errors- Check certificate chain, expirations, CA trust, hostname (SNI), CRL/OCSP timeouts.- Use openssl verify and s_client verify:

bash

openssl verify -CAfile /etc/ssl/certs/ca-bundle.crt server.pem

- Check OCSP responses in s_client output (-status) and server OCSP stapling.- What to look for: missing intermediates, wrong SAN, OCSP timeout/failure causing delays or aborts.

7) Proxies / Load-balancer / Timeout interactions- Inspect LB/proxy logs and idle/timeouts. Handshake may be interrupted if upstream timeouts too low or health checks misconfigured.- What to look for: Nginx/haproxy timeouts, connection resets, half-closed sockets, TCP close during handshake.- Commands:

bash

# haproxy stats (example)
echo "show stat" | socat stdio /var/run/haproxy.sock
# check connection states on host
ss -tna | grep ESTAB

- Validate persistence and TCP keepalive/timeout settings align across chain. Increase timeouts temporarily to see effect.

8) Correlate telemetry & mitigate- Correlate traces: logs + openssl + pcap timelines. Look for consistent failure signature (same cipher, source IP, datacenter).- Temporary mitigations: enable less strict cipher set, increase LB timeouts, reduce MTU, disable TLS resumption, roll back recent changes, route traffic to healthy datacenter.

9) Long-term fixes & monitoring- Harden monitoring: expose TLS handshake success rates, TLS alert metrics, cipher downgrade counts, certificate expiry alerts.- Automate packet capture on failure patterns, maintain documented mitigation playbooks.

Key pointers:- Always capture both client and server sides when possible.- Compare successful vs failed flows; small differences reveal root cause.- Pay special attention to middlebox behavior (firewalls, IDS) that may drop fragmented packets or strip TLS extensions.

MediumSystem Design

28 practiced

Design a reproducibility test harness that captures production requests (anonymized), replays them against a staging environment to reproduce flaky bugs, and avoids side effects on downstream systems. Describe components, storage, rate-limiting, anonymization, and safety mechanisms for replaying production traffic.

Sample Answer

Requirements:- Capture representative production requests with headers/body, metadata (timing, auth-less IDs).- Anonymize PII and secrets before storing.- Replay to staging or isolated sandbox to reproduce flaky bugs without affecting downstream systems.- Rate-limit and throttle replays; preserve order/temporal patterns optionally.- Auditability, config-driven selection, and ability to rollback/disable.

High-level architecture:- Capture Agent (edge/service mesh or reverse-proxy plugin) → Sanitizer/Anonymizer → Event Store (immutable) → Replay Orchestrator → Staging Environment(s) + Sandbox Hooks → Observer & Comparator → Dashboard/Alerting.

Components and responsibilities:- Capture Agent: non-blocking, streams a copy of requests (and responses optionally) to async pipeline (e.g., Kafka). Minimal latency and sampling controls.- Sanitizer/Anonymizer: pipeline consumer that applies deterministic hashing/masking rules, strips auth tokens, removes cookies, redacts bodies per schema, and replaces IDs with stable pseudonyms for correlation.- Event Store: append-only store (object store + metadata DB). Stores original schema version, sampling rate, provenance, and retention policy. Use encryption at rest and access controls.- Replay Orchestrator: schedules replays, applies rate-limiting, configurable traffic shaping (burst, ramp, preserve order), injects test headers, and maps endpoints to staging versions.- Sandbox Hooks & Downstream Failsafes: staging uses feature flags to stub or record downstream calls, redirect external integrations to mock endpoints, and enforce circuit breakers and resource quotas.- Observer & Comparator: captures staging responses, compares diffs to production behavior (where available), collects logs/traces, and surfaces potential flakiness.

Rate-limiting and safety:- Global and per-service throttle: default low throughput (e.g., 1 req/sec) with token-bucket for bursts. Configurable guardrails per test.- Dependency isolation: DNS routing or service mesh rules to route external calls to mocks; enforce side-effect-free modes (read-only DB replica or snapshot).- Non-destructive mode: replay header forces idempotency, e.g., use test tenant or sandbox accounts; automatically rewrite destructive verbs (POST→POST with test parameters) or block them.- Quotas & time windows: only replay during maintenance windows or on-demand with approvals for higher throughput.- Kill-switch & circuit-breaker: central abort API and health checks that stop replays if error rates or latency exceed thresholds.

Anonymization & privacy:- Schema-driven redaction: JSON schemas define sensitive fields; use format-preserving hashing for keys to maintain referential integrity.- Differential access controls: separate teams’ access to raw vs. anonymized data; audit logs for all accesses.- Compliance: configurable retention, encryption, and support for data subject deletion workflows.

Observability, auditing, and workflow:- Dashboard to schedule runs, show diffs, correlate traces (using pseudonymous IDs), and attach incidents.- Automated triage: mark reproducible runs, attach logs/traces, and open tickets.- CI integration: allow replay of specific recorded sessions as part of staging test suites.

Trade-offs:- Stronger anonymization reduces forensic fidelity; keep reversible but auditable escapes accessible under strict controls for incident work.- Full fidelity replays risk side effects; prefer mocking critical integrations and using read-only replicas.

This harness balances reproduction fidelity with privacy and safety, enabling SREs to reproduce flaky bugs reliably while preventing downstream impact.

EasyTechnical

21 practiced

What are the differences between logs, metrics, and distributed traces for debugging and RCA? For an incident that affects only 0.1% of requests, which telemetry source would you consult first and why, and what signals do you expect to see in each?

Sample Answer

Logs, metrics, and distributed traces serve complementary roles in debugging/RCA:

- Metrics: Aggregated numeric time-series (counts, latencies, error rates, percentiles). Great for detecting that something is wrong, alerting, and measuring blast radius and trends.- Traces: Per-request timeline across services (spans). Best for understanding causal flow, locating the service/span where latency or errors originate, and seeing high-tail behavior for individual requests.- Logs: High-detail textual records (events, stack traces, contextual fields). Use for deep inspection once you’ve isolated the offending request(s) — exact exception, input data, config values.

If an incident affects only 0.1% of requests, start with traces (assuming you have request-level tracing and enough sampling). Reason: the problem is at the request granularity and unlikely to move metrics significantly; traces let you find the few slow/erroring request flows and identify the problematic service/span quickly. Recommended workflow:1. Check metrics to confirm whether overall error/latency rate shows a tiny bump and to get the time window and affected endpoints.2. Query traces for those endpoints/time window, filter for errors or high-latency tails, and inspect spans to find the failing service or RPC.3. Once you have a candidate trace/request-id, consult logs for that request-id to get stack traces, payloads, config, and exact error messages.

Signals to expect:- Metrics: tiny increase in error count or latency p99/p999; overall traffic and SLOs likely still within thresholds; alerts may not have fired if thresholds are coarse.- Traces: a subset of traces showing errors/exceptions, long spans, retry loops, or an abnormal span timing in a particular downstream service; repeated pattern across the 0.1%.- Logs: per-request error messages, stack traces, validation failures, or unusual input that caused the codepath to fail; correlated request-id, headers, or user identifiers.

Caveats: tracing sampling can miss rare failures—temporarily reduce sampling or use error-focused capture. If tracing isn’t available, use metrics to narrow time/endpoint and scan logs by request attributes or small time windows.

EasyTechnical

30 practiced

What is a flaky test? Describe common root causes of flakiness in CI pipelines (timing, resource contention, test order dependency, external service reliance) and list concrete strategies you would use to detect and triage flaky tests in a large test suite.

Unlock Full Question Bank

Get access to hundreds of Advanced Debugging and Root Cause Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.