Systematic Troubleshooting and Debugging Questions

Covers structured methods for diagnosing and resolving software defects and technical problems at the code and system level. Candidates should demonstrate methodical debugging practices such as reading and reasoning about code, tracing execution paths, reproducing issues, collecting and interpreting logs metrics and error messages, forming and testing hypotheses, and iterating toward root cause. Topic includes use of diagnostic tools and commands, isolation strategies, instrumentation and logging best practices, regression testing and validation, trade offs between quick fixes and long term robust solutions, rollback and safe testing approaches, and clear documentation of investigative steps and outcomes.

HardTechnical

34 practiced

You have three high-priority user-facing bugs reported simultaneously and only two engineers available to work on them. How do you prioritize which to fix first? Describe the factors you consider, how you communicate decisions, and how you allocate limited resources to minimize customer impact and risk.

Sample Answer

Situation: Three high-priority, user-facing bugs arrived simultaneously while only two SREs were available. Each bug could affect availability, data integrity, or performance for customers.

Task: Decide order of fixes, allocate two engineers, minimize customer impact and risk, and communicate clearly to stakeholders.

Action:- Rapid triage (I led a 15–20 minute Triage): quickly classify each bug by impact (SLO/SLA breach, number of users affected, data loss vs. annoyance), scope (single service vs. cascade across systems), reproducibility, and rollback/mitigation options.- Quantify risk: used monitoring/alerts to estimate affected traffic and error rates (e.g., 10% of requests failing vs. 0.1%), and checked whether the incident consumes error budget.- Prioritize by objective criteria: 1. Safety/data-loss > availability impacting many users > degraded performance for some users. 2. If two bugs have similar severity, prioritize the one with an easy mitigation or higher blast radius.- Allocate engineers: - Engineer A: immediate hotfix for highest-impact bug (data loss/major SLO breach) — pair if required for risky deploys. - Engineer B: start mitigation for second-highest (deploy circuit-breaker, rate limit, routing rule) and begin investigation for the third if mitigation is cheap. - If a fix is long-running, rotate: one engineer remains on containment while the other cycles to next highest-priority remediation.- Use safe deployment practices: feature flags, canary, automated rollback, and short maintenance windows if required.- Communication: - Sent an initial incident update within 15 minutes to stakeholders (PMs, Support, Engineering): impact summary, priority decision, temporary mitigations, expected next update time. - Provided clear runbook actions and ownership: "A — fix bug #1; B — mitigate bug #2 and begin triage for #3." - Regular updates every 30–60 minutes and a post-incident summary with root cause, timeline, and follow-ups.- Minimize customer impact and risk: - Apply short-term mitigations to reduce user exposure (traffic routing, feature toggles). - Avoid risky, large-scope changes during emergency; prefer incremental fixes and automated rollbacks. - If necessary, coordinate a quick maintenance page or degraded-mode notice to set expectations.

Result / Learning:- This approach minimizes user-facing impact by fixing highest-risk issues first while containing others, keeps stakeholders informed, and preserves safe deployment practices. After the incident I codified the triage decision matrix and added playbook steps for parallel resource allocation so future simultaneous incidents resolve faster and more predictably.

HardTechnical

39 practiced

A Heisenbug disappears when you add logging or run under a debugger. Propose at least four techniques to investigate and capture the bug without perturbing behavior significantly. Discuss trade-offs for each technique.

Sample Answer

Situation: You're debugging a Heisenbug — it vanishes when you add logging or attach a debugger. Below are four+ low-perturbation techniques an SRE can use to observe and capture the bug, with concrete notes and trade-offs.

1) eBPF-based tracing (bcc/bpftrace)- What: Attach probes to kernel/syscall/uprobes in userspace to collect stack traces, latencies, events with microsecond overhead.- Why: Runs in-kernel, very low overhead and safe to deploy on production for short windows.- Trade-offs: Limited history buffer; complex BPF programs risk verifier rejection; debugging logic in restricted environment; sampling granularity may miss very short-lived races.

2) Probabilistic / sampled logging- What: Add extremely low-rate (1/10k–1/1M) log sampling or conditional traces on rare invariants; tag correlated request IDs.- Why: Minimal perturbation; can capture rare events while keeping volume low.- Trade-offs: May miss the event; requires good correlation keys to connect partial traces.

3) Hardware-assisted tracing (Intel PT / LBR)- What: Use CPU last-branch recording or Intel Processor Trace to capture execution paths with negligible timing impact.- Why: Very low perturbation and high fidelity instruction-level trace.- Trade-offs: Large data volume; requires kernel support/tools to collect/interpret; not always available in cloud VMs.

4) Record-and-replay / deterministic replay (rr, checkpoint+replay, CRIU)- What: Record one execution (syscalls, signals) and replay deterministically to reproduce under debugger.- Why: Lets you debug non-deterministic bugs offline without perturbing original run.- Trade-offs: Recording overhead can be high; some syscalls or network interactions complicate replay; not trivial for multi-process/distributed systems.

5) Fine-grained snapshotting + core dumps- What: Trigger memory+thread snapshots or minidumps on invariant violation (using ulimits, coredump_filter, or custom watchdog).- Why: Capture full state at failure moment for offline analysis without live instrumentation.- Trade-offs: Dumps are heavy; must ensure privacy/security of dumps; may miss transient scheduling order.

6) Low-overhead distributed tracing and causal logging- What: Use lightweight span propagation (e.g., W3C tracecontext) and collect timing/metadata instead of full logs.- Why: Correlates multi-service events with tiny payloads.- Trade-offs: Still may alter timing slightly; needs instrumentation across services.

Guiding strategy: start with non-invasive sampling (eBPF + sampled traces), add targeted hardware tracing or record-replay when you have a reproduction, and use snapshot dumps when you need full state. Always consider privacy, data volume, and the risk of adding new failure modes when choosing a tool.

MediumTechnical

37 practiced

A pod in Kubernetes is in CrashLoopBackOff. Detail the kubectl commands and investigative steps you would run to determine whether the crash is due to application error, image problems, misconfigured liveness/readiness probes, or node resource exhaustion. Include how to collect logs and events and an approach to repro locally.

Sample Answer

Start by gathering basic status and recent events:

- Check pod status and node:

bash

kubectl get pods -o wide --namespace myns
kubectl describe pod my-pod-<id> --namespace myns
kubectl get events --namespace myns --sort-by=.lastTimestamp

- Inspect container logs (current and previous restarts) to see application errors or stack traces:

bash

kubectl logs my-pod-<id> -c my-container --namespace myns
kubectl logs my-pod-<id> -c my-container --previous --namespace myns

- If logs are empty, verify image start issues (entrypoint/command, missing binaries) by checking pod spec and image:

bash

kubectl get pod my-pod-<id> -o yaml --namespace myns
kubectl describe pod my-pod-<id> | grep -A5 "Containers:"
kubectl rollout history deploy/my-deployment --namespace myns

- Probe misconfiguration: examine liveness/readiness settings and container lifecycle hooks in the pod YAML. Temporarily disable probes to confirm:

bash

kubectl patch deploy my-deployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"my-container","livenessProbe":null,"readinessProbe":null}]}}}}' --namespace myns

- Check resource exhaustion on node (CPU/memory, OOMKills):

bash

kubectl top pod my-pod-<id> --namespace myns
kubectl top node <node-name>
kubectl describe node <node-name> | grep -A5 "Allocatable\|Non-terminated"
kubectl get pod my-pod-<id> -o jsonpath='{.status.containerStatuses[0].lastState}'

- If you need a shell, run a debug pod on same node or use ephemeral container:

bash

kubectl debug -it my-pod-<id> --image=busybox --target=my-container --namespace myns
kubectl alpha debug node/<node-name> -it --image=busybox

- Collect cluster-wide logs/events and pod-level diagnostics for postmortem:

bash

kubectl logs deploy/my-deployment --namespace myns --since=1h
kubectl get events --all-namespaces --sort-by=.lastTimestamp > /tmp/events.log
kubectl get pods -o yaml --namespace myns > /tmp/pods.yaml

Reproduce locally:- Run same image with env/args on your laptop to validate startup:

bash

docker run --rm -e VAR=val --name local-test my-registry/my-image:tag

- Or use kind/minikube to recreate the deployment with same resource limits and probes:

bash

kubectl apply -f pod-debug.yaml --namespace debug

Diagnosis checklist:- If logs show stack trace or config error → application bug/config.- If logs empty + image exits quickly → image/entrypoint issue.- If liveness probe fails repeatedly but app healthy on manual curl → probe misconfiguration (tweak thresholds, initialDelaySeconds).- If OOMKilled or node pressure → resource limits/requests mis-sized or node exhaustion.

Always capture logs, pod YAML, events and timestamps, then roll back or patch probes and redeploy with increased verbosity for verification.

MediumTechnical

37 practiced

You suspect a memory leak in a Java service running in production. Describe the sequence of diagnostic steps you would perform to identify the leak: what JVM flags, heap dump tools, GC logs, and analysis tools you would use and how you would interpret object retention graphs to find the leaking allocation site.

Sample Answer

Situation: Production Java service shows steadily increasing memory usage and rising OOMs. I would follow a safe, evidence-driven sequence to identify the leak.

1) Confirm and gather signals- Check monitoring: heap used, GC pause frequency/duration, thread count, file descriptors. Correlate with deploys/traffic.- Pull recent GC metrics and heap usage graphs to confirm a leak (heap used trending up between GCs).

2) Enable/collect non-invasive JVM diagnostics (if not already)- GC logging (Java 8/11+): - Java 8: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/var/log/gc.log -XX:+PrintHeapAtGC - Java 11+: -Xlog:gc*:file=/var/log/gc.log:time,uptime,level- jstat for live GC stats: jstat -gc <pid> 1000- Avoid enabling heavy profilers in prod unless controlled.

3) Capture heap dumps (safe practice)- Use jcmd to request a heap dump to disk (non-invasive): jcmd <pid> GC.heap_dump /tmp/heap.hprof- Alternatively use jmap -dump:live,format=b,file=/tmp/heap.hprof <pid> (be cautious on large heaps).- Take multiple dumps over time to confirm growth (t1, t2).

4) Collect allocation/profiler data (optional)- For sampling allocation hotspots use async-profiler (low-overhead): ./profiler.sh -d 30 -e alloc -f alloc.svg <pid>- For more detail, use YourKit/Flight Recorder in staging.

5) Analyze GC logs- Look for: - Increasing frequency of full GCs or long GC pauses. - “Promotion failure” or “Last resort GC” messages. - Heap occupancy before and after GC: if used after full GC continues to grow, live set increases — leak likely.- Tools: GCViewer, GCEasy, or jClarity Censum to visualize.

6) Heap dump analysis- Open .hprof in Eclipse MAT, VisualVM, or IntelliJ Memory Analyzer.- First view “Summary”: dominant retained sizes and suspect classes.- Use the Dominator Tree to find objects holding most retained memory. Retained size = memory freed if that object released.- Drill down on top dominators (e.g., large Collections, caches, byte[] arrays).- Inspect incoming references and “Paths to GC Roots” for a suspicious object to see why it’s reachable. - Look for long-lived static maps, caches, ThreadLocals, classloader leaks, or unclosed resources (InputStreams, JDBC connections).- Use MAT queries: - Leak Suspects report - OQL queries to find patterns, e.g. find all byte[] > threshold.

7) Map retained objects to allocation site- If allocation sites are not visible in heap dump, use allocation profiling (async-profiler or JVM Flight Recorder with allocation stack traces) in a controlled environment to capture where large objects are allocated.- In MAT, check if objects have “stack trace” info (requires -XX:+HeapDumpOnOutOfMemoryError with -XX:+HeapDumpAfterFullGC sometimes preserves more info) — otherwise use allocation profiler to correlate high-retained types with code paths.

8) Validate and remediate- Reproduce in staging with same workload and attach detailed profiler to confirm allocation site.- Fix code: clear caches, use weak/soft references where appropriate, close resources, limit cache sizes, or fix logic causing retention.- Deploy fix, monitor GC/heap metrics and regression tests.

Interpretation tips when reading retention graphs:- High shallow size with low retained size usually not a leak; high retained size indicates root retention.- Dominator tree node high retained size + single incoming reference from a static or long-lived object = likely leak root.- Repeated object graph patterns (many same key in map) point to application-level structures (maps/lists) as the leak source.- Allocation flamegraphs or allocation sites with persistent allocations over time map to the code that must be changed.

Commands summary:jcmd <pid> GC.heap_infojcmd <pid> GC.heap_dump /tmp/heap.hprofjstat -gc <pid> 1000async-profiler: ./profiler.sh -d 60 -e alloc -f alloc.svg <pid>

This workflow prioritizes low-risk collection, GC-log analysis, targeted heap dumps, dominator-tree inspection in MAT, and allocation profiling to pinpoint the allocation site before applying a code fix and monitoring the outcome.

MediumTechnical

39 practiced

You see an increase in 95th percentile latency for an API endpoint in Prometheus. Describe the step-by-step approach you would take to triage: what PromQL queries you would use, how to map to spans/traces, and which system and application metrics you would inspect next.

Sample Answer

1) Quick-sanity & scope- Confirm the alert window and when p95 rose. Query p95 over time:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="api", handler="/v1/foo"}[5m])) by (le))

- Compare baseline (e.g., last 24h) vs spike:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="api", handler="/v1/foo"}[5m])) by (le)) 
  or on() vector(0) # plot trend

2) Narrow by dimension (instance/replica/region/route)- Per-instance p95:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="api", handler="/v1/foo"}[5m])) by (le, instance))

- Per-pod / per-node rates:

sum(rate(http_requests_total{job="api", handler="/v1/foo"}[1m])) by (instance)

If only some instances show high p95, suspect local saturation.

3) Correlate traffic & errors- Request rate, 5xx rate:

sum(rate(http_requests_total{job="api", handler="/v1/foo"}[1m])) by (instance)
sum(rate(http_responses_total{job="api", code=~"5..", handler="/v1/foo"}[1m])) by (instance)

- Latency vs concurrency:

sum(rate(process_cpu_seconds_total{job="api"}[1m])) by (instance)

4) Map to traces/spans- If instrumentation includes trace IDs in logs/metrics: find requests during spike (timestamp ± window) and extract trace ID. If not, use APM UI (Jaeger/Zipkin/NewRelic): - Filter traces by service name + endpoint + time range + duration > p95 threshold. - Look for common slow spans: DB, external HTTP, cache miss, GC pause.- If you expose a metric like trace_span_duration_seconds_sum or custom tags, query:

sum(rate(span_duration_seconds_sum{service="api", span="db_query"}[5m])) by (instance)

Trace examples: look for increased db call count per request or repeated retries.

5) Inspect system & app metrics- CPU / load, memory, GC pauses:

node_load1, node_cpu_seconds_total, process_resident_memory_bytes, jvm_gc_pause_seconds_count (or equivalent)

- IO / network:

node_disk_io_time_seconds_total, node_network_receive_bytes_total

- Thread pools / queue lengths / connection pools: - instrumented metrics: http_server_threads_active, worker_queue_length, db_pool_active, db_pool_wait_count- Downstream services: - DB p95: histogram_quantile(0.95, sum(rate(db_query_duration_seconds_bucket[5m])) by (le)) - External API latencies and error rates

6) Hypothesis & test- Example hypotheses: CPU-saturated pod, increased DB latency, cache eviction leading to more DB calls, GC spikes, unhealthy nodes.- Test by isolating: scale replicas, drain suspect pod, replay traffic, or run synthetic requests while inspecting traces.

7) Remediation & follow-up- Short-term: route traffic away from bad pods, increase replicas, roll back deploy.- Medium-term: add finer instrumentation (per-span metrics, request_id in logs), alerts for downstream p95, alert on divergence between p95 and median.- Postmortem: root cause, mitigation, SLO impact, additional dashboards.

Key reasoning: first quantify the scope with PromQL, localize by dimension, correlate with traffic/errors and system saturation, map to spans to find slow component, then act (isolate/scale/rollback) and prevent with better instrumentation and targeted alerts.

Unlock Full Question Bank

Get access to hundreds of Systematic Troubleshooting and Debugging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.