Debugging and Systematic Troubleshooting Questions
Diagnosing defects methodically: reproducing failures, forming and testing hypotheses, reading stack traces and logs, bisecting changes, and reasoning about error handling and edge cases. Covers a disciplined root-cause approach that applies from local bugs to production issues, distinct from embedded hardware-level debugging. A universally probed engineering-craft skill.
Explain, with examples, how cognitive biases such as confirmation bias, anchoring, and sunk-cost fallacy can hinder a debugging investigation. Describe concrete practices, such as pair debugging, rotating investigators, hypothesis logs, and clear acceptance criteria, that you have introduced on a team to mitigate these biases.
Sample Answer
Direct answer
Confirmation bias, anchoring, and sunk-cost fallacy each distort a root-cause investigation in a specific, predictable way: confirmation bias makes you notice evidence supporting your first guess and discount evidence against it; anchoring makes an early, possibly wrong hypothesis dominate the rest of the investigation even after better evidence emerges; and sunk-cost fallacy keeps you investigating a disproven lead because of time already invested in it, rather than switching based on current evidence. Concrete team practices (pair debugging, rotating investigators, hypothesis logs, explicit acceptance criteria) counter each of these by introducing structure that doesn't rely on individual willpower to overcome the bias.
Structured elaboration
Confirmation bias: once you suspect a cause, you unconsciously interpret ambiguous evidence as supporting it and are quicker to dismiss evidence that doesn't fit. In debugging, this looks like reading a log line as "consistent with my theory" when a more neutral read would call it inconclusive, or stopping the investigation the moment ANY supporting evidence appears rather than continuing to look for disconfirming evidence too.
- Mitigation: hypothesis logs. Writing down each hypothesis and what specific evidence would DISCONFIRM it, before looking for evidence, forces a falsifiable framing up front; when you later find evidence, you check it against the pre-written disconfirmation criteria rather than retroactively deciding it "counts" as support.
Anchoring: the first plausible explanation offered (by you, or by someone else on the call) tends to dominate the rest of the investigation's framing, even as better evidence appears, because everyone's mental model has already organized around it.
- Mitigation: rotating investigators / a fresh pair of eyes. Someone joining the investigation LATE, without the accumulated anchor, will naturally form hypotheses from the current evidence rather than the initial framing, and is often the person who notices the anchor was wrong; deliberately bringing in a fresh perspective partway through a long investigation is a structural way to interrupt this.
Sunk-cost fallacy: having spent two hours pursuing one lead makes it psychologically harder to abandon, even once evidence stops supporting it, because abandoning it "wastes" the time already spent (which is, of course, already spent either way, and not actually recoverable by continuing).
- Mitigation: explicit acceptance criteria and timeboxing set in advance. Deciding, BEFORE starting to investigate a specific hypothesis, what evidence would confirm it and how much time is reasonable to spend testing it, makes the "abandon or continue" decision a pre-committed rule rather than an in-the-moment judgment call that sunk cost can distort.
Pair debugging as a mitigation for all three simultaneously: a second person, thinking independently, is less likely to share the exact same anchor or the exact same sunk-cost attachment to a specific lead, and naturally provides a real-time check on confirmation bias by asking "does that evidence actually support that, or are we reading it generously?"
Worked example
A team investigating an intermittent failure anchors early on "it's probably the recent deploy" (a reasonable first guess, given the timing). Two hours in, with the deploy's code reviewed thoroughly and nothing found, sunk cost starts to argue for continuing to scrutinize that same deploy rather than considering it possibly unrelated. A hypothesis log, written at the start, had specified "if the failure recurs on a service instance that predates this deploy, that disconfirms the deploy hypothesis"; checking that specific, pre-committed criterion shows the failure DID recur on an older instance, cleanly disconfirming the deploy theory despite two hours of sunk investigation into it. A rotating fresh investigator, brought in specifically because the original two were stuck, asks a question neither anchored investigator had considered ("what else changed around that time besides the deploy?") and identifies an unrelated infrastructure change that actually explains the failure.
Trade-offs and pitfalls
These practices have a real cost (pairing takes two people's time instead of one; hypothesis logs take a few minutes to write that could otherwise go straight into investigating), and the trade-off is worth it specifically for investigations that are ALREADY taking a long time or where the cost of a wrong conclusion is high; for a five-minute, low-stakes bug, the overhead of formal hypothesis logging isn't proportionate to the risk these biases actually pose in that context.
Explain Raft and Paxos at a level useful for debugging production clusters: how leader election, log replication, and commit rules work, and what causes leader churn and split-brain. Given a cluster experiencing repeated leader elections and elevated latency, outline a triage plan to identify and remediate the root cause.
Sample Answer
Direct answer
Raft and Paxos both solve the same problem, getting a cluster of nodes to agree on an ordered log of operations despite failures, via leader election and log replication; understanding the mechanism is what lets you read a "repeated leader elections, elevated latency" symptom as a specific, diagnosable pattern (leader churn) rather than a vague "the cluster is unhappy," and triage it by checking the SPECIFIC conditions known to cause churn.
Structured elaboration
How leader election and log replication work (at the level needed for debugging): nodes elect a single leader via a voting process requiring a majority (quorum); the leader accepts writes, appends them to its own log, and replicates them to followers; an entry is considered "committed" (safe, durable) once a majority of nodes have it in their log, at which point it's applied and acknowledged to the client. If the leader stops being heard from (a network partition, a crash, or simply being too slow to send heartbeats within the follower's timeout), followers time out and trigger a NEW election.
What causes leader churn and split-brain:
- Leader churn (repeated re-elections) happens when something makes the CURRENT leader repeatedly fail to maintain its leadership within the cluster's timeout window: network instability between the leader and enough followers to lose quorum-heartbeat delivery, the leader itself being overloaded (GC pauses, CPU starvation) and failing to send heartbeats in time even though it's technically still alive, or election-timeout values tuned too aggressively relative to the actual network's latency variance, causing followers to time out and call new elections even during normal, brief latency blips.
- Split-brain (two nodes both believing they're the leader simultaneously) is specifically what BOTH Raft and Paxos are designed to prevent via the quorum/majority requirement: a true split-brain (as opposed to a brief, correctly-resolved dual-candidacy during an election) should be structurally impossible if the algorithm and quorum configuration are correct, so observing genuine split-brain behavior is a strong signal of either a configuration bug (a quorum size that doesn't actually require a true majority, often from a MISCOUNTED total node count after a scaling change) or a client incorrectly caching a stale leader address and continuing to write to it without validating leadership.
Triage plan for a cluster with repeated leader elections and elevated latency:
- Check the ELECTION frequency and correlate with network metrics between the current leader and followers specifically; frequent elections correlating with network latency spikes or packet loss point at a network-stability issue, not an application bug.
- Check the LEADER's own resource health (CPU, GC pause times, disk I/O for its log-write path) during the windows just before each re-election; a leader failing to send heartbeats because it's itself resource-starved is a very different fix (resource/config tuning on that node) than a network issue.
- Check the configured election-timeout relative to observed network latency percentiles. A timeout set too close to the network's normal P99 latency will trigger spurious elections on ordinary latency variance; the safe margin should be several multiples of observed latency variance, not just comfortably above the median.
- Check quorum configuration explicitly if there's ANY suspicion of split-brain specifically, confirming the configured cluster size and required-majority calculation match the ACTUAL number of voting members, especially after any recent scaling event (adding or removing a node) that could have left a stale configuration on some nodes.
Worked example
Correlating election timestamps against network metrics shows every re-election lines up with brief packet-loss spikes on the network path to one specific follower, not a general cluster-wide issue; the current election-timeout is set close to the network's typical P95 latency (the response time slower than 95% of measurements. P50, P95, and P99 all describe the same idea at different strictness levels: P50 is the typical case, P95 and P99 capture progressively rarer, worse outliers), meaning even ordinary jitter occasionally exceeds it. Widening the election timeout to a safer multiple of observed P99 latency (rather than P50/P95) stops the spurious re-elections without addressing the underlying, minor network jitter directly, since the actual issue was a mistuned timeout, not a fundamentally broken network path.
Trade-offs and pitfalls
Widening the election timeout too far trades away FAILOVER SPEED for stability: a genuinely dead leader will now take longer to be detected and replaced, so the timeout should be tuned to comfortably exceed normal latency variance without being needlessly conservative; the right value depends on the actual, measured network characteristics of the specific deployment, not a generic default copied from documentation.
Find and fix the bug in this JavaScript async function, where a missing await causes unpredictable ordering and errors:
async function processItems(items, processor) {
items.forEach(async item => {
await processor(item);
});
console.log('done');
}
Explain the root cause and provide a corrected version that guarantees 'done' prints only after every item has been processed.
Sample Answer
Direct answer
Array.prototype.forEach does not await its callback: it fires each async callback and moves on immediately, so console.log('done') runs before any of the await processor(item) calls have resolved. The fix is to replace forEach with a construct that actually waits, either a for...of loop with await inside it (sequential) or Promise.all over items.map(...) (concurrent).
Structured elaboration
forEach was designed before async/await existed and its callback's return value, including a Promise, is simply discarded. Passing an async function as the callback doesn't change this: each invocation still returns a Promise that forEach never looks at, so forEach itself completes synchronously (having merely started every item, not finished any of them) the instant it has called the callback once per array element. The console.log('done') line after the forEach call then runs immediately, while the async work is still in flight in the background.
Worked example
async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }
// Fixed, distinct per-item delays (not Math.random()) so the interleaving below
// is deterministic and reproduces identically on every run, not just illustrative.
const delays = { 1: 20, 2: 10, 3: 30 };
// BUGGY
async function processItemBuggy(x) { await sleep(delays[x]); console.log(' processed', x, '(buggy)'); }
async function processBuggy(items) {
items.forEach(async item => { await processItemBuggy(item); });
console.log('done (buggy)');
}
// FIXED: sequential
async function processItemFixed(x) { await sleep(delays[x]); console.log(' processed', x, '(fixed)'); }
async function processFixed(items) {
for (const item of items) { await processItemFixed(item); }
console.log('done (fixed)');
}
(async () => {
console.log('--- buggy run ---');
await processBuggy([1, 2, 3]);
await sleep(100); // let the buggy run's background work finish before the next section starts
console.log('--- fixed run (sequential for-of) ---');
await processFixed([1, 2, 3]);
})();
Executed output (buggy items finish asynchronously in the background; with the fixed per-item delays above, item 2 always resolves first, then item 1, then item 3, after done has already printed):
--- buggy run ---
done (buggy)
processed 2 (buggy)
processed 1 (buggy)
processed 3 (buggy)
--- fixed run (sequential for-of) ---
processed 1 (fixed)
processed 2 (fixed)
processed 3 (fixed)
done (fixed)
done (buggy) prints immediately, before any processed line: forEach fired all three async callbacks and moved on without waiting for any of them. The processed (buggy) lines do eventually print, once their fixed delays resolve, in delay order (2, then 1, then 3, not the input order), rather than before done. The for...of version correctly prints all three processed lines, strictly in input order, before done.
Trade-offs and pitfalls
There are two valid fixes with different semantics, and picking the wrong one is itself a common mistake: a for...of loop processes items strictly one at a time (useful when order matters or you must not overwhelm a downstream dependency with concurrent calls), while await Promise.all(items.map(item => processor(item))) processes all items concurrently and finishes as soon as the slowest one does (faster, but only safe if the operations are independent and the downstream system can handle concurrent load). Silently reaching for forEach out of habit, rather than deliberately choosing sequential versus concurrent semantics, is exactly how this bug class gets reintroduced even by developers who already know the rule.
Explain the practical differences between debugging at the application level versus the infrastructure level (network, storage, compute). Give two examples of failures that look similar at first glance but have different root causes at each level, and describe how you would distinguish between them.
Sample Answer
Direct answer
Application-level debugging asks "why did this code produce the wrong result," and its evidence lives inside your process: variables, call stacks, exceptions, business logic. Infrastructure-level debugging asks "why couldn't the correct code even run correctly," and its evidence lives outside your process: network reachability, disk and memory pressure, DNS, container scheduling, resource limits. The two failure modes can look identical from the outside (a request fails, a page times out) while requiring completely different evidence to distinguish.
Structured elaboration
The practical test: if you could run the exact same code on a machine with unlimited, perfectly-behaving resources and a clean network, would the bug still happen? If yes, it's application-level (a logic error, a bad state transition, an unhandled edge case). If no, it's infrastructure-level (the code was never wrong, but its environment failed to give it what it needed).
Two concrete examples that look alike but diverge in root cause:
- "Requests are timing out." Application-level cause: a newly introduced N+1 query pattern (fetching a list, then running one extra database query per item in that list instead of one combined query, so 100 items means 101 round trips instead of 1) makes one endpoint genuinely slow under load, so it exceeds a client timeout. Infrastructure-level cause: the container's memory limit is too tight, so the process is spending most of its time in garbage collection (or worse, getting OOM-killed (the operating system forcibly terminating the process because it used more memory than it was allowed to) and restarted), and the timeout is really a symptom of resource starvation that has nothing to do with the query logic. Both present as "P99 latency spiked" in a dashboard.
- "Intermittent 500 errors on one specific endpoint." Application-level cause: a race condition in that endpoint's handler corrupts shared state under concurrent access. Infrastructure-level cause: that endpoint happens to be the one that calls out to a dependency whose DNS resolution is flaky, so the failures are really network-layer and would happen on ANY endpoint that made the same external call.
How to distinguish them in practice: check infrastructure-level signals FIRST because they're cheap and rule out a whole class of causes at once: container restarts/OOM kills, CPU throttling, disk pressure, DNS/network error rates, host-level resource graphs. If those are clean, the evidence points inward to application logic, and now stack traces, code paths, and business-logic invariants become the productive place to look. Going the other direction (deep-diving application logs before checking whether the host was even healthy) wastes time chasing red herrings in code that was never the problem.
Worked example
Given "checkout intermittently fails for large carts": check host-level memory/CPU graphs for the checkout service during the failure window (infra-level, five minutes). If those are flat and unremarkable, the next cheapest infra check is whether the failures correlate with a specific downstream dependency's error rate (still infra-adjacent, still cheap). Only once both come back clean do you profile the actual cart-processing code path, because now the evidence has ruled out "the environment failed the code" and pointed at "the code itself has a large-cart-specific bug," likely a data-size-dependent logic error, not a data-size-dependent resource limit.
Trade-offs and pitfalls
The trap is assuming the layer based on which tools you personally know best rather than what the symptom actually implies. A backend engineer with no infra background will often burn an hour reading application logs for a problem that a five-second look at a container-restart graph would have explained. The fix is cheap and disciplined: always take the five-minute infra-layer look first, even if you expect it to come back clean, because ruling it out is what makes the deeper application-level dig trustworthy.
You're handed a stack trace and a short log snippet showing a NullPointerException-like error in a microservice. What are the first five concrete steps you take to triage the issue, and why? Be specific about the commands, artifacts, or data points you would request or inspect.
Sample Answer
Direct answer
With a stack trace and log snippet in hand, the first five steps are: (1) read the stack trace's top frame to identify the exact line and null value involved, (2) check the log immediately before the exception for the request/input context, (3) determine whether this is reproducible or a one-off, (4) check recent deploys or config changes around the failure's first occurrence, and (5) assess blast radius (how many requests/users affected) to decide urgency before digging further.
Structured elaboration
- Read the stack trace's top frame precisely. A NullPointerException-like error names the exact line where a null was dereferenced; note the specific variable and the surrounding code, since that tells you WHAT was null, which immediately narrows the space of "why" hypotheses.
- Check the log lines immediately before the exception for request context: what input, user, or upstream call was in flight when the null occurred. This is often the fastest way to form a concrete hypothesis rather than staring at the code in the abstract.
- Determine reproducibility. Try to trigger the same code path with the same or similar input locally or in staging; a reliably reproducible case turns the rest of the investigation into ordinary debugging, while a genuinely intermittent one requires the log/trace evidence to carry more of the weight.
- Check what changed recently. A deploy or config change around the time this error first appeared is a strong lead; correlate the error's first-seen timestamp against the deploy history before assuming it's an old, dormant bug that just got triggered by unusual input.
- Assess blast radius and urgency, specifically before spending more time investigating: is this a single edge-case request, or is it happening at a rate that indicates a systemic problem affecting many users? This determines whether the next step is "keep investigating calmly" or "mitigate immediately, diagnose in parallel."
Concrete artifacts/data points to request or inspect at each step: the exact request ID or correlation ID tied to the failing request (to pull its FULL log trail, not just the exception itself); the deploy/change history for the service in the relevant time window; a count or rate of this specific exception over time (one occurrence versus a sustained rate materially changes urgency); and, if available, a sample of two or three OTHER occurrences of the same exception, to check whether they share a common input pattern or are scattered and unrelated.
Worked example
A NullPointerException at OrderService.java:212, in a block computing a discount based on customer.getLoyaltyTier(). The log line just before shows the request was for a specific customer ID; pulling that customer's full record shows their loyaltyTier field is null, unlike the typical customer record which always has a default tier set. Checking recent changes shows a new customer-import path (added in a deploy two days prior) that bypasses the usual account-creation flow and its default-value logic. That's the concrete root cause: a specific new code path creates customer records without the default tier, and any downstream code that assumes the tier is always set will fail exactly like this, for exactly the customers created through that path.
Trade-offs and pitfalls
The temptation under time pressure is to jump straight to "add a null check" without doing steps 2 through 4, which fixes THIS symptom but leaves the actual root cause (customer records being created in an inconsistent state) free to cause a different failure somewhere else that assumes the same invariant. A null check is a reasonable immediate mitigation once you understand the mechanism, but the investigation isn't complete until you know WHY the null got there in the first place.
Unlock Full Question Bank
Get access to all 29 Debugging and Systematic Troubleshooting interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.