Debugging and Systematic Troubleshooting Questions
Diagnosing defects methodically: reproducing failures, forming and testing hypotheses, reading stack traces and logs, bisecting changes, and reasoning about error handling and edge cases. Covers a disciplined root-cause approach that applies from local bugs to production issues, distinct from embedded hardware-level debugging. A universally probed engineering-craft skill.
Explain the practical differences between debugging at the application level versus the infrastructure level (network, storage, compute). Give two examples of failures that look similar at first glance but have different root causes at each level, and describe how you would distinguish between them.
Sample Answer
Direct answer
Application-level debugging asks "why did this code produce the wrong result," and its evidence lives inside your process: variables, call stacks, exceptions, business logic. Infrastructure-level debugging asks "why couldn't the correct code even run correctly," and its evidence lives outside your process: network reachability, disk and memory pressure, DNS, container scheduling, resource limits. The two failure modes can look identical from the outside (a request fails, a page times out) while requiring completely different evidence to distinguish.
Structured elaboration
The practical test: if you could run the exact same code on a machine with unlimited, perfectly-behaving resources and a clean network, would the bug still happen? If yes, it's application-level (a logic error, a bad state transition, an unhandled edge case). If no, it's infrastructure-level (the code was never wrong, but its environment failed to give it what it needed).
Two concrete examples that look alike but diverge in root cause:
- "Requests are timing out." Application-level cause: a newly introduced N+1 query pattern (fetching a list, then running one extra database query per item in that list instead of one combined query, so 100 items means 101 round trips instead of 1) makes one endpoint genuinely slow under load, so it exceeds a client timeout. Infrastructure-level cause: the container's memory limit is too tight, so the process is spending most of its time in garbage collection (or worse, getting OOM-killed (the operating system forcibly terminating the process because it used more memory than it was allowed to) and restarted), and the timeout is really a symptom of resource starvation that has nothing to do with the query logic. Both present as "P99 latency spiked" in a dashboard.
- "Intermittent 500 errors on one specific endpoint." Application-level cause: a race condition in that endpoint's handler corrupts shared state under concurrent access. Infrastructure-level cause: that endpoint happens to be the one that calls out to a dependency whose DNS resolution is flaky, so the failures are really network-layer and would happen on ANY endpoint that made the same external call.
How to distinguish them in practice: check infrastructure-level signals FIRST because they're cheap and rule out a whole class of causes at once: container restarts/OOM kills, CPU throttling, disk pressure, DNS/network error rates, host-level resource graphs. If those are clean, the evidence points inward to application logic, and now stack traces, code paths, and business-logic invariants become the productive place to look. Going the other direction (deep-diving application logs before checking whether the host was even healthy) wastes time chasing red herrings in code that was never the problem.
Worked example
Given "checkout intermittently fails for large carts": check host-level memory/CPU graphs for the checkout service during the failure window (infra-level, five minutes). If those are flat and unremarkable, the next cheapest infra check is whether the failures correlate with a specific downstream dependency's error rate (still infra-adjacent, still cheap). Only once both come back clean do you profile the actual cart-processing code path, because now the evidence has ruled out "the environment failed the code" and pointed at "the code itself has a large-cart-specific bug," likely a data-size-dependent logic error, not a data-size-dependent resource limit.
Trade-offs and pitfalls
The trap is assuming the layer based on which tools you personally know best rather than what the symptom actually implies. A backend engineer with no infra background will often burn an hour reading application logs for a problem that a five-second look at a container-restart graph would have explained. The fix is cheap and disciplined: always take the five-minute infra-layer look first, even if you expect it to come back clean, because ruling it out is what makes the deeper application-level dig trustworthy.
Given a function that sometimes throws an exception deep inside a library you cannot modify, explain how you would instrument the codebase to capture a stack trace and the relevant contextual variables without changing the library's own code. Describe approaches for at least two of Python, Java, or Node.js.
Sample Answer
Direct answer
You can capture a stack trace and the relevant local state without touching the library's source by instrumenting at the BOUNDARY where you call into it: wrap the call site in a try/catch (or its language equivalent) that re-raises after logging, install a global/uncaught-exception hook that fires regardless of where the exception originates, or attach a debugger/tracer that breaks on the exception type without modifying any code at all.
Structured elaboration
Three complementary approaches, from least to most invasive:
- Wrap the call site, not the library. Since you control the code that CALLS into the library, even if you can't modify the library itself, catching the exception at your own call site and logging the full stack trace, the exception type/message, and any locally-relevant variables (the arguments you passed in, the current request context) before re-raising gives you a permanent, low-overhead capture point. In Python:
try: library_call(...) except Exception: logger.exception("context: %s", relevant_vars); raise. In Java: a try/catch around the call that logs viae.printStackTrace()or a structured logger before rethrowing. In Node.js: wrapping in a try/catch for synchronous calls, or.catch()on the returned Promise for async ones. - Install a global uncaught-exception/unhandled-rejection hook. This catches cases where the exception surfaces somewhere you didn't anticipate wrapping: Python's
sys.excepthook, Java'sThread.setDefaultUncaughtExceptionHandler, Node'sprocess.on('uncaughtException', ...)andprocess.on('unhandledRejection', ...). These fire regardless of exactly where inside the library the exception originated, at the cost of being a last-resort catch-all rather than a precisely-scoped one. - Attach a debugger or tracer without changing code, when you need MORE than a stack trace, such as full local variable state at the moment of the throw. Most debuggers support "break on exception" (Python's
pdbwithpython -m pdb, orimport pdb; pdb.set_trace()combined with a signal, or a conditional breakpoint set on the exception type in a full IDE debugger; Java's debugger supports exception breakpoints directly in most IDEs) which pauses execution at the exact throw site, inside the library's own code, letting you inspect the full call stack and locals without ever editing the library.
Worked example
A third-party HTTP client library occasionally raises an unlabeled ConnectionError with no useful message, and you need to know what request triggered it. Wrapping the call site: try: response = third_party_client.get(url, timeout=5) except ConnectionError: logger.exception("third_party_client failed for url=%s, timeout=%s", url, 5); raise. This doesn't change a single line of the library, but every future occurrence now logs the exact URL and timeout that triggered it, alongside the library's own stack trace, immediately giving you the context needed to distinguish "this URL is consistently unreachable" from "this only fails under a specific timeout value."
Trade-offs and pitfalls
A global uncaught-exception hook is a safety net, not a substitute for a targeted wrap at the call site: it tells you SOMETHING failed somewhere, but without the call-site context (what arguments were in play, what business operation was in progress) it's often not enough to actually diagnose the issue, only to know it happened. The debugger approach is the most informative but requires either a reproducible local trigger or an interactive session, so it's best reserved for cases where the logged stack trace alone isn't enough to form a hypothesis.
Explain Raft and Paxos at a level useful for debugging production clusters: how leader election, log replication, and commit rules work, and what causes leader churn and split-brain. Given a cluster experiencing repeated leader elections and elevated latency, outline a triage plan to identify and remediate the root cause.
Sample Answer
Direct answer
Raft and Paxos both solve the same problem, getting a cluster of nodes to agree on an ordered log of operations despite failures, via leader election and log replication; understanding the mechanism is what lets you read a "repeated leader elections, elevated latency" symptom as a specific, diagnosable pattern (leader churn) rather than a vague "the cluster is unhappy," and triage it by checking the SPECIFIC conditions known to cause churn.
Structured elaboration
How leader election and log replication work (at the level needed for debugging): nodes elect a single leader via a voting process requiring a majority (quorum); the leader accepts writes, appends them to its own log, and replicates them to followers; an entry is considered "committed" (safe, durable) once a majority of nodes have it in their log, at which point it's applied and acknowledged to the client. If the leader stops being heard from (a network partition, a crash, or simply being too slow to send heartbeats within the follower's timeout), followers time out and trigger a NEW election.
What causes leader churn and split-brain:
- Leader churn (repeated re-elections) happens when something makes the CURRENT leader repeatedly fail to maintain its leadership within the cluster's timeout window: network instability between the leader and enough followers to lose quorum-heartbeat delivery, the leader itself being overloaded (GC pauses, CPU starvation) and failing to send heartbeats in time even though it's technically still alive, or election-timeout values tuned too aggressively relative to the actual network's latency variance, causing followers to time out and call new elections even during normal, brief latency blips.
- Split-brain (two nodes both believing they're the leader simultaneously) is specifically what BOTH Raft and Paxos are designed to prevent via the quorum/majority requirement: a true split-brain (as opposed to a brief, correctly-resolved dual-candidacy during an election) should be structurally impossible if the algorithm and quorum configuration are correct, so observing genuine split-brain behavior is a strong signal of either a configuration bug (a quorum size that doesn't actually require a true majority, often from a MISCOUNTED total node count after a scaling change) or a client incorrectly caching a stale leader address and continuing to write to it without validating leadership.
Triage plan for a cluster with repeated leader elections and elevated latency:
- Check the ELECTION frequency and correlate with network metrics between the current leader and followers specifically; frequent elections correlating with network latency spikes or packet loss point at a network-stability issue, not an application bug.
- Check the LEADER's own resource health (CPU, GC pause times, disk I/O for its log-write path) during the windows just before each re-election; a leader failing to send heartbeats because it's itself resource-starved is a very different fix (resource/config tuning on that node) than a network issue.
- Check the configured election-timeout relative to observed network latency percentiles. A timeout set too close to the network's normal P99 latency will trigger spurious elections on ordinary latency variance; the safe margin should be several multiples of observed latency variance, not just comfortably above the median.
- Check quorum configuration explicitly if there's ANY suspicion of split-brain specifically, confirming the configured cluster size and required-majority calculation match the ACTUAL number of voting members, especially after any recent scaling event (adding or removing a node) that could have left a stale configuration on some nodes.
Worked example
Correlating election timestamps against network metrics shows every re-election lines up with brief packet-loss spikes on the network path to one specific follower, not a general cluster-wide issue; the current election-timeout is set close to the network's typical P95 latency (the response time slower than 95% of measurements. P50, P95, and P99 all describe the same idea at different strictness levels: P50 is the typical case, P95 and P99 capture progressively rarer, worse outliers), meaning even ordinary jitter occasionally exceeds it. Widening the election timeout to a safer multiple of observed P99 latency (rather than P50/P95) stops the spurious re-elections without addressing the underlying, minor network jitter directly, since the actual issue was a mistuned timeout, not a fundamentally broken network path.
Trade-offs and pitfalls
Widening the election timeout too far trades away FAILOVER SPEED for stability: a genuinely dead leader will now take longer to be detected and replaced, so the timeout should be tuned to comfortably exceed normal latency variance without being needlessly conservative; the right value depends on the actual, measured network characteristics of the specific deployment, not a generic default copied from documentation.
Describe an incident where implementing automated retries with exponential backoff caused an unexpected production problem, such as cascading retries or duplicate writes. How did you detect the issue, debug the root cause, and redesign the retry strategy afterward?
Sample Answer
Direct answer
A well-known failure mode: adding exponential-backoff retries without also bounding total concurrent retries or coordinating across clients can turn a brief, minor downstream slowdown into a self-inflicted overload, since every client independently retries at roughly the same moments, producing synchronized bursts (a "thundering herd") or, worse, an ever-growing number of in-flight retried requests that outpaces the downstream system's ability to drain them, causing duplicate writes if the underlying operation wasn't idempotent.
Structured elaboration
The mechanism, concretely: a downstream dependency slows down briefly. Every client calling it starts timing out around the same time, and every client's retry logic backs off and retries on a similar schedule (especially if backoff has no random jitter, all clients retry at almost exactly the same intervals). This creates synchronized retry storms that hit the ALREADY-STRUGGLING downstream system with a burst of additional load right as it's trying to recover, potentially preventing recovery entirely or extending the outage well beyond what the original blip would have caused. If the retried operation was a write that wasn't idempotent (no dedup key, no "already applied" check), some of those retries succeed on requests that had ALSO already succeeded on an earlier attempt whose success acknowledgment was merely delayed or lost, producing duplicate writes.
How the issue was detected: typically via a mismatch between the "small blip" the monitoring initially showed and a much larger, longer-lasting incident that followed; checking request-rate graphs for the affected downstream dependency reveals a sharp, synchronized spike in incoming request rate that has no correlate in genuine user traffic, which is the signature of a retry storm rather than organic load growth. A duplicate-writes symptom (if applicable) is typically caught downstream, in data-quality checks or a customer report of a double-charged transaction, and root-caused back to the retry logic by correlating timestamps of the duplicate writes against the retry schedule.
Redesigning the retry strategy:
- Add jitter to the backoff, so retries from different clients spread out over a window instead of clustering at the same instants, directly reducing the synchronized-burst effect.
- Cap total concurrent retries or add a circuit breaker, so that once a downstream dependency is confirmed unhealthy, clients stop hammering it with fresh attempts and instead fail fast (or queue) until health is confirmed to be restored, rather than every client independently continuing to retry indefinitely.
- Make the underlying operation idempotent wherever the retried operation has a side effect (a write, a charge, a message send), using a deterministic idempotency key so a duplicate retry is safely recognized and no-op'd by the receiver, removing the duplicate-write risk regardless of how the retry logic behaves.
- Bound total retry attempts and total retry TIME, not just per-attempt backoff, so a request doesn't retry indefinitely against a dependency that's genuinely down for an extended period, which just prolongs and worsens the retry-storm effect.
Trade-offs and pitfalls
Retries with backoff are the right default response to TRANSIENT failures, and removing them entirely in reaction to this incident would be an overcorrection that makes the system more fragile against genuinely brief blips; the fix is making the retry strategy safe under the failure modes it can itself cause (synchronized bursts, duplicate side effects), not abandoning retries. The general lesson for any team adding retry logic for the first time: always pair it with jitter and idempotency from the start, rather than adding them reactively after the first retry-storm incident.
Your payment provider intermittently returns 502 errors, causing checkout failures for customers. Describe the debugging steps you would take to determine whether the root cause is your own integration, the provider itself, the network, or a configuration issue, and describe short-term mitigations to reduce customer impact while you investigate.
Sample Answer
Direct answer
First establish WHO owns the fault, before trying to fix anything: check whether the 502s originate from your own service (a bug in how you call the provider), the network path between you and them, your configuration (wrong endpoint, expired credentials, timeout misconfigured), or the provider itself. A 502 specifically means an upstream server returned an invalid response to a gateway, which already narrows the search: it usually means something between you and the provider's actual application server broke, not that your request was malformed (that would more often be a 4xx).
Structured elaboration
- Check your own logs for the exact request/response pair. Capture the full request you sent (headers, body, timing) and the full 502 response, including any body the provider returned. A 502 with a body from the provider's own error page suggests an issue near their edge, not deep in their systems.
- Check timing and correlation. Is the failure rate correlated with your request VOLUME (suggests rate limiting or a capacity issue on their side), with TIME OF DAY (suggests their scheduled maintenance or your own traffic pattern), or with a SPECIFIC request shape (suggests your own payload triggers an edge case in their processing)?
- Rule out your own network and configuration. Confirm DNS resolves to the expected endpoint, TLS handshakes succeed, and you're not accidentally hitting a sandbox/staging URL in production. Check whether a recent config or credential change on your side coincides with when the 502s started.
- Check the provider's status page and support channels. Many payment providers publish real-time incident status; if others report the same symptom at the same time, that's strong evidence the fault is upstream, not yours.
- Determine if it's truly intermittent or has a pattern. A steady low background rate of 502s can be normal for any third-party dependency at scale; a sudden step-change is what actually indicates an incident, on either side.
Worked example
A checkout service seeing 502s from a payment provider: logs show the failures are NOT correlated with request volume (rules out simple rate limiting) but ARE correlated with a specific payment method (Apple Pay tokens specifically), while card payments succeed at the normal rate. That pattern points at the provider's Apple-Pay-specific processing path, not a general outage or a problem in the checkout service's own code, since the same service, same network path, and same general request shape succeed for card payments. Confirmed by checking the provider's status page, which shows a partial incident affecting exactly that payment method.
Trade-offs and pitfalls
Short-term mitigation while you investigate: implement a retry with backoff for 502s specifically (a 502 is often transient), and if you can identify a stable pattern like the Apple-Pay-specific example, temporarily route that payment method to a fallback or clearly surface the failure to the user rather than silently retrying a request that will keep failing the same way. The trap is assuming "third-party" automatically means "not my problem to investigate further": even a genuine provider-side incident is worth root-causing on your end, both to build an accurate mitigation and because sometimes what looks like a provider outage is actually your own malformed request that only fails for a specific payload shape.
Unlock Full Question Bank
Get access to all 42 Debugging and Systematic Troubleshooting interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.