Systematic Troubleshooting and Debugging Questions

Covers structured methods for diagnosing and resolving software defects and technical problems at the code and system level. Candidates should demonstrate methodical debugging practices such as reading and reasoning about code, tracing execution paths, reproducing issues, collecting and interpreting logs metrics and error messages, forming and testing hypotheses, and iterating toward root cause. Topic includes use of diagnostic tools and commands, isolation strategies, instrumentation and logging best practices, regression testing and validation, trade offs between quick fixes and long term robust solutions, rollback and safe testing approaches, and clear documentation of investigative steps and outcomes.

EasyTechnical

34 practiced

Describe how structured logging (e.g., JSON logs with fields) differs from unstructured logging and why structured logs are helpful for systematic debugging at scale. Provide example fields you would include for a public-facing HTTP service and explain how those fields make root-cause analysis easier.

HardTechnical

30 practiced

Design a stepwise approach to instrument and debug an intermittent TLS handshake failure between clients and a regional load balancer. Include what artifacts to collect (pcap captures, openssl s_client output, certificate chains, load balancer logs), how to isolate whether the failure is due to client certificate mismatch, incorrect SNI routing, cipher/version incompatibility, or expired/revoked intermediate certificates, and safe testing steps to reproduce the problem without impacting all clients.

Sample Answer

**Approach — overview**1) Reproduce reliably on a small scope, collect artifacts, then triage by hypothesis (client cert mismatch, SNI routing, cipher/version, cert chain issues). Work iteratively and non-disruptively.

**Artifacts to collect**- PCAPs at client, LB front-end, and back-end (tcpdump filter for TLS):

bash

sudo tcpdump -i eth0 -w /tmp/tls.pcap port 443 and host <client_ip or lb_ip>

- openssl s_client outputs (with SNI, cert chain, and chosen cipher):

bash

openssl s_client -connect lb.example.com:443 -servername lb.example.com -showcerts

- Full certificate chains (PEM), OCSP/CRL responses, and cert serials- Load balancer logs (access, error, TLS debug) with timestamps and connection IDs- Backend server TLS logs and application logs- Metrics/traces (latency, TLS handshake errors) from monitoring

**Stepwise isolation**1) Correlate timestamps/connection IDs across artifacts to tie a failed handshake to a pcap and LB log entry.2) Client cert mismatch: - Check openssl s_client with client cert/key:

bash

openssl s_client -connect lb:443 -cert client.pem -key client.key -CAfile ca.pem

- Inspect LB requiring client auth (mutual TLS) and verify CA bundle and subject DN allowed list.3) Incorrect SNI routing: - Test with different SNI values; confirm backend selection in LB logs.

bash

openssl s_client -connect lb:443 -servername correct.example.com
openssl s_client -connect lb:443 -servername wrong.example.com

- Use pcap to verify SNI in ClientHello (if TLS1.3 or ESNI considerations).4) Cipher/version incompatibility: - Force specific TLS versions/ciphers:

bash

openssl s_client -connect lb:443 -tls1_2 -cipher 'ECDHE-RSA-AES128-GCM-SHA256'

- Compare ClientHello cipher list to server's accepted suite in LB/backend logs and wireshark.5) Expired/revoked/intermediate certs: - Inspect full chain from s_client -showcerts; check validity dates and issuer chain. - Query OCSP or CRL for revocation.

bash

openssl ocsp -issuer issuer.pem -cert cert.pem -url http://ocsp.responder/

**Safe testing / repro without impacting all clients**- Use canary testing: route a small percentage to a debug LB pool or use traffic mirroring.- Test from isolated client VM that reproduces the client environment; use curl/openssl with identical SNI/certs.- Rate-limit and schedule tests during low-traffic windows; disable automated retries to avoid flood.- If possible, enable LB debug on a single listener or use feature flags to capture extra logging for only affected IPs.- When changing configs (certs/cipher lists), apply to a single backend or staging first, validate, then roll out with gradual traffic shift.

**Outcome & remediation**- Once root cause identified, fix (update CA bundle, adjust SNI routing table, add supported ciphers, renew intermediates), validate via the same s_client/pcap tests, then monitor metrics for disappearance of handshake failures.

HardTechnical

30 practiced

Describe how you would debug a Heisenbug: an intermittent race condition that disappears when you add logging or run under a debugger. Provide reproducibility strategies, non-invasive instrumentation techniques (kernel tracing, hardware watchpoints, record-and-replay), and specific Linux tools and approaches (rr, ftrace, perf, GDB) that minimize perturbation of timing.

Sample Answer

**Approach overview (why this matters)** I’d treat a Heisenbug as a timing-sensitive concurrency bug and focus first on reproducibility with minimal perturbation, then use non-invasive tracing and record-and-replay to inspect the failing execution.

**Reproducibility strategies**- Run targeted stress tests in a controlled environment: fixed CPU frequency, isolated cores (cpuset), IRQ affinity, disable turbo/CPU power management to reduce nondeterminism.- Use workload amplification: increase concurrency, reduce sleeps/timeouts, inject synthetic load (cpu/mem/io) to raise failure probability.- Capture environment: container snapshot, kernel version, coredumps, and perf events.

**Non-invasive instrumentation techniques**- Kernel tracing with ftrace/tracepoints: low-overhead, can log scheduler, lock events, syscalls with minimal timing change.- perf record -e sched:sched_switch, cpu-clock to collect samples and context switches non-invasively.- Hardware watchpoints & ptrace-based breakpoints sparingly; watchpoints are lower overhead than software breakpoints.- Record-and-replay with rr (userland): reproduce and step through without altering timing; ideal for deterministic userspace bugs.- Kernel-level record-replay (if needed) or use kcov/kasan in staging with care.

**Specific Linux tools & approaches**- rr: run the failing test under rr to get deterministic replay and use gdb to step; good for single-process/userland races.- ftrace / trace-cmd: enable specific events (sched, sys_enter/sys_exit, lockdep) and correlate timestamps.- perf: perf record/perf script for stack traces; perf sched for scheduling anomalies.- GDB: use hardware watchpoints (watch *(int*)addr) to stop on memory writes; limit scope to reduce perturbation.- kprobe/uprobes: attach to functions without recompilation when need kernel/user function entry logging.

**Workflow**1. Harden environment and amplify failure.2. Try rr for reproducible userland replay; if successful, debug with gdb/rr replay.3. If rr cannot capture (multi-process, kernel interactions), use ftrace/perf with precise tracepoints to narrow window.4. Use perf/ftrace logs to identify suspect region; add targeted hardware watchpoints under gdb or use controlled probes.5. Validate fix and rerun stress tests in CI with high concurrency and trace sampling enabled.

**Trade-offs**- RR is powerful but only userland and single-node; ftrace/perf add less perturbation but produce large data. Balance sampling granularity with overhead.

This sequence minimizes timing perturbation while providing deterministic paths to root cause and fix.

MediumTechnical

33 practiced

You're investigating frequent CI test flakiness: tests pass locally and on most agents but fail consistently on one Ubuntu 20.04 runner with cryptic filesystem errors. Outline how you would isolate environmental differences between runners, gather evidence (logs, process lists, kernel versions, mounts, ulimits), and propose fixes to make tests deterministic across runners.

Sample Answer

**Clarify scope & goal**I’d make tests deterministic on all runners while preserving performance — isolate why Ubuntu 20.04 runner shows filesystem errors.

**Gather evidence (commands to run)**Run on both a good runner and the failing runner and collect artifacts:

bash

uname -a
lsb_release -a
cat /proc/version
mount | sed -n '1,200p'
df -hT
cat /proc/mounts
ulimit -a
ps aux --sort=-rss | head -n 50
dmesg --ctime | tail -n 200
journalctl -u docker.service --no-pager -n 200
stat /path/to/failing/file || ls -l /path/to/failing
strace -o /tmp/test.strace -f ./run-failing-test

Capture CI job logs, runner agent logs, and timestamps; save copies for diffs.

**Isolate differences**- Automated diff of outputs (git-like):

bash

diff -u runner_good.txt runner_bad.txt

- Compare kernel, mount options (e.g., noexec, nodev), filesystem types (ext4 vs overlayfs), cgroup versions, container runtimes, storage drivers (overlay2 vs aufs), SELinux/AppArmor, and available disk/inodes.- Check ulimits, open files, tmpfs sizes, and throttling/IOPS limits.

**Hypotheses & targeted tests**- If mount options differ: reproduce by remounting read-write or matching options.- If overlayfs/driver issue: run the job in a container with the same storage driver or switch to bind mounts.- If ulimit/file-descriptor exhaustion: increase limits in runner config and retest.- If kernel bug/dmesg shows errors: upgrade kernel or use runner image pinned to known-good kernel.- If intermittent race on temp dirs: ensure tests use unique per-job temp directories, fsync where required, or run under tmpfs to reduce latency.

**Fixes to deploy**- Short-term: add environment check step at CI start that fails fast with actionable message if runner mismatches; add retries and stronger assertions in tests.- Medium-term: standardize runner images (immutable images or Docker container images) and enforce IaC-managed configs; pin kernel and storage drivers or use hosted runners.- Long-term: containerize test environment fully (same image for all runners), add e2e smoke job that validates runner before executing tests.

**Automation & monitoring**- Add a CI job that periodically compares runner diagnostics and alerts on divergence.- Store artifacts centrally; correlate failures with runner metadata and node health (disk, memory, IO).

This approach narrows root cause quickly, produces reproducible test environments, and prevents future divergences.

EasyTechnical

28 practiced

Explain the role of Service-Level Indicators (SLIs) and Service-Level Objectives (SLOs) in prioritizing debugging and remediation work. Provide a concrete example of an SLI/SLO pair for a public HTTP API (include metric and objective) and explain how an SLO breach should change incident response priorities.

Unlock Full Question Bank

Get access to hundreds of Systematic Troubleshooting and Debugging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.