Systematic Troubleshooting and Debugging Questions

Covers structured methods for diagnosing and resolving software defects and technical problems at the code and system level. Candidates should demonstrate methodical debugging practices such as reading and reasoning about code, tracing execution paths, reproducing issues, collecting and interpreting logs metrics and error messages, forming and testing hypotheses, and iterating toward root cause. Topic includes use of diagnostic tools and commands, isolation strategies, instrumentation and logging best practices, regression testing and validation, trade offs between quick fixes and long term robust solutions, rollback and safe testing approaches, and clear documentation of investigative steps and outcomes.

EasyTechnical

48 practiced

A stateful service reports unhealthy status and fails its health check. The service stores data on local disk. Enumerate at least five possible root causes and for each give a quick command or check to distinguish disk I/O issues from application logic failures.

Sample Answer

**Overview**Below are five+ plausible root causes for a stateful service failing health checks, with a concise diagnostic check or command to help distinguish disk I/O problems from application-logic failures. I frame each as: cause → quick check (what indicates disk I/O vs app logic).

1) Disk full / no free inodes - Check: free space and inodes

bash

df -h /var/lib/myservice && df -i /var/lib/myservice

- Disk I/O signal: 0B available or 100% inodes. - App logic signal: space OK but app logs show write errors or panics.

2) Underlying device errors (hardware / SMART) - Check: kernel/device errors and SMART health

bash

dmesg | egrep -i 'ext4|sd|error|I/O' -n && smartctl -H /dev/sda

- Disk I/O signal: repeated I/O errors, SMART FAIL. - App logic signal: no kernel I/O errors; failures appear only in app stack traces.

3) High disk latency / saturated IOPS - Check: realtime I/O stats and top consumers

bash

iostat -xz 1 3 && iotop -b -n 5

- Disk I/O signal: high await/svctm, %util near 100, io-consuming processes. - App logic signal: normal I/O metrics but service process CPU/thread stuck or exceptions.

4) Filesystem corruption / mount read-only - Check: mount status and attempts to write a temp file

bash

mount | grep '/var/lib/myservice' && touch /var/lib/myservice/.healthcheck_write 2>&1 || echo fail

- Disk I/O signal: filesystem mounted ro or write fails with I/O errors. - App logic signal: writes succeed but app still reports unhealthy.

5) Permission / ownership issues on storage path - Check: ownership/SELinux and ability to open files as service user

bash

ls -ld /var/lib/myservice && sudo -u myservice_user test -w /var/lib/myservice && echo write-ok

- Disk I/O signal: permissions prevent write but disk health metrics normal. - App logic signal: permissions correct; failures in app logs (exceptions, config errors).

6) Network-mounted volume (NFS/EBS) problems - Check: show mount type, latency, and rpc stats

bash

mount | grep myvolume && cat /proc/mounts && showmount -e nfs-server 2>/dev/null

- Disk I/O signal: NFS hangs, retransmits, or remote unreachability. - App logic signal: mount healthy locally; application stack traces point to logic.

7) Application-level bug (corrupt DB, crash loops) - Check: service logs, open files and sockets

bash

journalctl -u myservice -n 200 && lsof -p $(pidof myservice)

- Disk I/O signal: logs show I/O exceptions; system metrics align with disk faults. - App logic signal: clear stack trace, assertions, config errors without underlying I/O errors.

Tips: correlate timestamps (dmesg/iostat) with service log timestamps to decide root cause quickly.

HardTechnical

30 practiced

A large service fails TLS handshakes intermittently after a certificate rotation, with 'bad certificate' errors despite clients and servers running in the same VPC. Propose a step-by-step troubleshooting and remediation plan: how to verify certificate chains, SNI, OCSP stapling, intermediate cert presence, clock skew, client/server trust stores, and rolling update strategies that avoid breaking connections.

Sample Answer

**Situation & goal**Provide a safe, step-by-step troubleshooting and remediation plan to restore reliable TLS after a certificate rotation causing intermittent "bad certificate" errors.

**Verify certificate chain**- From client and server hosts, fetch and inspect chain:

bash

# show cert chain and SNI
openssl s_client -connect service.example.com:443 -servername service.example.com -showcerts

- Confirm leaf -> intermediates -> root order and that intermediates are served by the server.

**Check SNI**- Ensure clients send correct SNI and server virtual-host config uses it. Reproduce both with and without -servername to see behavior.

**OCSP / Stapling**- Query stapled OCSP response from server:

bash

openssl s_client -connect host:443 -status -servername host

- If stapling is missing/expired, check server TLS config and OCSP responder reachability; enable stapling and renewal.

**Intermediate cert presence**- If chain incomplete, add missing intermediate(s) to server cert bundle and reload. Validate with SSL Labs or openssl.

**Clock skew**- On a few affected hosts, check system time (ntp/chrony). Certificate validity failures often from >5 min drift:

bash

date; sudo chronyc tracking

**Client & server trust stores**- Verify clients trust the CA chain (root/intermediate) and haven't cached old intermediates. For JVM/containers, confirm cacerts and restart processes.

**Rolling update strategy (avoid breaking connections)**- Use connection draining, blue/green or canary rollout: - Mark instance as draining in load balancer; wait for existing connections to finish. - Replace certs and reload TLS daemon non-disruptively (graceful restart). - Deploy small canary group, monitor handshake errors and metrics (TLS errors, 4xx/5xx). - If errors persist, rollback to previous cert bundle quickly.

**Observability & remediation**- Collect server and client TLS logs, correlate timestamps. Add metrics/alerts for TLS handshake failures and OCSP stapling errors.- If cached old certs on clients cause intermittent failures, advise clients to restart or flush TLS caches and provide timeline for forced refresh.

Result: systematic verification of chain, SNI, OCSP, time, and trust stores plus a safe rolling update pattern to avoid breaking connections.

MediumTechnical

29 practiced

Your CI pipeline exhibits flaky tests that pass locally but fail intermittently in CI containers. Describe a systematic triage plan to root cause flakiness: isolating environment differences, resource constraints, test interdependence, container runtime, timing issues, and use of deterministic seeds or sandboxing.

Sample Answer

**High-level goal**Create a repeatable triage playbook to turn intermittent CI failures into deterministic, actionable root causes.

**Step 1 — Reproduce consistently**- Re-run the failing job with full logs and increased verbosity; preserve failing container image and job ID.- Reproduce locally using the exact container image and entrypoint:

bash

docker pull my-ci-image:job-123 && docker run -it --env-file=.env my-ci-image:job-123 /bin/bash
# run the same test command from CI
pytest tests/failing_test.py -k failing_case -q --maxfail=1

**Step 2 — Isolate environment differences**- Compare OS packages, Python/Java/Node versions, environment variables, locale, time zone, ulimits.- Use `diff` on `pip freeze`/`npm ls`/`dpkg -l` outputs and `env | sort`.

**Step 3 — Check resource constraints**- Re-run tests with constrained CPU/memory similar to CI using cgroups or Docker flags and monitor:

bash

docker run --cpus=0.5 --memory=512m ...

- Observe OOM, CPU throttling, timeouts, and kernel logs.

**Step 4 — Detect test interdependence/order issues**- Run tests in random order and single-threaded:

bash

pytest -k failing_case --dist=loadfile --maxfail=1 -x
pytest --maxfail=1 --random-order

- Run the whole suite, bisect failing order with `pytest --maxfail` or `git bisect` on test order.

**Step 5 — Investigate container/runtime differences**- Check seccomp, AppArmor, cgroups, storage drivers, mount namespaces and ephemeral filesystem behaviors. Reproduce on same runtime (containerd vs docker).- Compare kernel versions and sysctl settings.

**Step 6 — Timing and nondeterminism**- Add logging, timestamps, and increased timeouts. Use tools like `strace` or `perf` to inspect blocking syscalls.- Run tests under a deterministic scheduler (single CPU) or add sleeps to expose races.

**Step 7 — Deterministic seeds and sandboxing**- Ensure tests that rely on randomness accept a seed; enforce CI to pass/save the seed:

bash

pytest --randomly-seed=12345

- Sandbox flaky tests (isolated containers, temp dirs, network namespaces). Use ephemeral databases or test doubles.

**Step 8 — Metrics, automation, and remediation**- Automate re-run and capture artifacts (core dumps, network traces). Add flaky-test annotation and quarantine until fixed.- Implement CI gate: require reproducible failure locally or attach deterministic seed before marking pass.

**Outcome & trade-offs**- This plan prioritizes reproducible debugging and isolates environmental vs code causes. Trade-offs: increased CI time and temporary quarantining of tests; but yields long-term reliability.

MediumTechnical

35 practiced

You inherit a large legacy monolith with almost no observability. Propose a practical plan to add observability incrementally with minimal risk: what instrumentation to add first, how to enrich logs and propagate context, smoke tests to validate changes, and canary strategies to deploy observability improvements.

Sample Answer

**Clarifying assumptions**I’d target low-risk, high-value observability first: error rate, latency, request flow. Use OpenTelemetry-compatible tooling so future work is portable.

**Instrumentation to add first**- Add structured JSON logs (log adapter) with minimal change: timestamp, level, service, host, request_id, user_id (if available).- Emit basic metrics (request count, latency histogram, error count) via a pull exporter (Prometheus client) or push gateway if necessary.- Add distributed tracing headers (W3C Trace-Context) at ingress/egress and start creating spans for request entry/exit.

**Enrich logs & propagate context**- Generate a correlation/request_id at the edge (load balancer or API layer) if none exists; inject into HTTP headers and message metadata.- Ensure all log writers include request_id and span_id. Use middleware/sidecar to inject context for legacy modules (nginx/LB, service proxies).- Store structured logs in ELK/Opensearch or Loki; traces to Jaeger/Zipkin; metrics to Prometheus/Grafana.

**Smoke tests to validate changes**- Automated synthetic requests that assert: - JSON log lines contain request_id + status + latency. - Prometheus metrics update (counter increments, latency histogram buckets). - Trace is created and contains expected spans (entry → handler → db).- Run smoke tests in CI and against canary instances.

**Canary strategy**- Non-invasive rollout: deploy log/metric exporters first with sampling and rate limits.- Canary a small percentage of instances (1–5%) behind feature flags or routing rules. Monitor: - CPU/memory overhead, error rates, latency change, log volume.- Gradually increase traffic to canary while validating smoke tests and SLOs; rollback if any regressions.

**Risk mitigation & ops**- Start with read-only sidecars or agent-based exporters to minimize code changes.- Use sampling and throttling to avoid storage/CPU spikes.- Document schema and tag conventions; add dashboards and alerts tied to SLOs before broad rollout.

MediumTechnical

37 practiced

You deploy a code change that causes intermittent request timeouts only under high load. Describe an experimental plan to reproduce the issue reliably: name tools you'd use (e.g., locust, wrk, Gatling), which synthetic load patterns to try (ramp, spikes, concurrency), metrics and sampling frequencies to collect, and how to control variables to prove causality.

Sample Answer

**Goal & hypothesis**Reproduce intermittent timeouts under high load to prove whether the new change causes increased tail latency/connection exhaustion.

**Tools**- Load generators: locust or k6 for realistic user flows; wrk or Gatling for high-concurrency HTTP microbenchmarks.- Observability: Prometheus + Grafana, OpenTelemetry traces, pprof/flamegraphs, tcpdump or ss/netstat.- System: sar/iostat, vmstat, netstat, dstat for host metrics.

**Experiment plan**1. Baseline: run synthetic traffic against pre-change version (canary or rollback) to record baseline metrics.2. Stress patterns: - Ramp: linear increase over 10–30 min to find load threshold. - Spike: sudden jump to target QPS for 1–5 min to trigger transient resource exhaustion. - Sustained high concurrency: constant high connections for 30+ min to exercise leaks. - Realistic mixes via locust scenarios (think time, session flows).3. Reproduce: run identical scenarios against post-change; alternate runs (A/B) to control time-varying noise.

**Metrics & sampling**- Request latency percentiles (p50, p95, p99, p999) — sample histogram every 5s.- Error rate/timeouts per second — 1s resolution.- Throughput (req/s) — 1s.- CPU, memory, iowait — 5s.- Open sockets/FDs, connection queues (syn backlog), threadpool utilization — 1–5s.- GC pauses and heap usage (if JVM) — 1s.- Network retransmits, drops — 5s.- Collect flamegraphs/pprof when issue occurs.

**Controlling variables / proving causality**- Repeatability: run each scenario multiple times at same wall-clock/time-of-day to reduce noise.- Binary toggle: compare with service version A (pre-change) and B (post-change) under identical load; keep traffic generator seed/requests identical.- Disable autoscaling & background jobs to avoid external scaling events.- Pin instance types, network, and placement; use isolated test env or dedicated nodes.- Controlled rollback/feature-flag flip during run to observe immediate metric changes (causal signal).- Correlate traces with system metrics and tcpdump to rule out network issues.- Statistical check: compute difference in p99/p999 and error rates with confidence intervals across repeated runs.

**Outcome**If post-change consistently raises tail latency/timeouts under identical controlled loads and rolling back/feature-flag eliminates them immediately, the change is causal. Use collected traces and profiles to pinpoint the root cause (e.g., blocking syscall, connection leak, thread pool starvation).

Unlock Full Question Bank

Get access to hundreds of Systematic Troubleshooting and Debugging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.