Debugging and Systematic Troubleshooting Questions

Direct answer

Diagnosing heap corruption in a long-running C++ process means catching the corruption AT THE POINT IT HAPPENS, not at the point it eventually crashes, because by the time a corrupted heap causes a visible crash the actual buggy write can be far away in both time and code from the crash site. The standard toolchain: run under AddressSanitizer (ASAN) or Valgrind to catch buffer overflows and use-after-free at the exact instruction that causes them, enable core dumps for any crash that does occur, and analyze allocator/heap-metadata corruption patterns to distinguish overflow from use-after-free from double-free.

Structured elaboration

1. Reproduce under a memory-safety sanitizer first, since manual code review rarely finds heap corruption directly. ASAN instruments every memory access and reports the exact read/write, its size, and a full stack trace the moment an out-of-bounds or use-after-free access occurs, rather than only when the corrupted memory later causes a visible crash. Valgrind's memcheck provides similar coverage without recompilation, at a significant runtime-speed cost, useful when you cannot rebuild with sanitizer flags (e.g., against a shipped binary).
2. Enable core dumps (ulimit -c unlimited and a configured core pattern) so that if the process does crash outside a sanitizer run, you have a snapshot to analyze post-mortem rather than only a stack trace at the crash site, which for heap corruption is frequently NOT where the actual bug is.
3. Distinguish the failure category from the corruption pattern. A buffer overflow typically corrupts adjacent heap metadata or neighboring allocations (found by ASAN's "heap-buffer-overflow" report, pinpointing both the overflowing write and the allocation it overflowed); a use-after-free corrupts memory that has already been returned to the allocator and possibly reused by something else (ASAN's "heap-use-after-free," which also reports where the memory was freed); a double-free corrupts the allocator's own internal free-list structures and often crashes deep inside malloc/free rather than in application code.
4. Analyze allocator patterns for intermittent, hard-to-reproduce cases. If the corruption is rare enough that a full sanitizer run in production isn't practical, techniques like periodic heap-consistency checks (lighter-weight, built-in memory-allocator self-checks you can turn on without a full sanitizer run: malloc_zone_check on macOS, mallopt(M_CHECK_ACTION, ...) on glibc-based Linux) or a hardened allocator (e.g., Electric Fence, or glibc's tunable malloc debugging) can narrow down. These are fallback options for when a full ASAN/Valgrind run genuinely isn't practical; in most cases the sanitizer approach described above is what you'd actually reach for first when corruption first occurs without full ASAN overhead.
5. Fix and validate under the sanitizer, not just by re-running normally. A fix that makes the crash go away without re-confirming under ASAN can just mean the corruption still happens but no longer crashes visibly, which is worse: the bug is still there, now hidden again.
6. Validate in CI, not just once by hand. A one-off local ASAN run proves the fix today; it does not stop the same bug class from coming back next quarter. Add a dedicated CI job that builds the affected binary (or its unit/integration test suite) with -fsanitize=address and runs it on every pull request, and treat any sanitizer finding as a build-blocking failure rather than a warning. Because ASAN's overhead (roughly 2-3x) is usually acceptable for a test suite even when it would be too costly for production, this is normally affordable as a required CI check; if the full suite is too slow to run on every commit, run it on a schedule (e.g., nightly) against main, with new-code paths covered on every PR at minimum.

Worked example

A service crashes roughly once a week with no consistent stack trace, a classic heap-corruption signature. Running the exact same workload under ASAN in a staging environment reproduces a heap-buffer-overflow within minutes: a fixed-size buffer used to serialize a variable-length message is written one byte past its allocation when the message hits a specific length boundary. ASAN's report gives the overflowing write's stack trace directly, which is not the same location as any of the crash sites seen in production, confirming why the crash location alone was misleading: production was crashing wherever the corrupted metadata happened to be read next, sometimes far from the actual bug.

Trade-offs and pitfalls

ASAN and Valgrind both add significant CPU and memory overhead (roughly 2-3x for ASAN, often 10-20x for Valgrind), so neither is normally run in production continuously; the practical approach is reproducing the workload in staging under the sanitizer, or running a canary instance with ASAN enabled if the bug is too rare to reproduce off of production traffic. Treating the crash-site stack trace as the bug location, without sanitizer confirmation, is the single most common way heap-corruption investigations go in circles.

Direct answer

Rubber-duck debugging is the practice of explaining your code or problem, line by line, out loud, to an inanimate object (or any passive listener), on the theory that the act of ARTICULATING your assumptions forces you to notice the gap between what you believe the code does and what it actually does; used collaboratively, you become the "duck," a real listener whose job is to stay quiet and let the other person talk through it, only interjecting with clarifying questions, not answers.

Structured elaboration

Why it works: most debugging time is lost not to hard problems but to an unexamined assumption ("of course X is true here") that the explainer has never actually had to state out loud; the discipline of narrating each step forces exactly that statement, and the mismatch often becomes obvious the moment it's spoken, even before the listener says anything at all.

How to use it collaboratively to help a stuck teammate:

1. Sit with them and ask them to explain the problem from the beginning, including what they expect to happen and what's actually happening, as if you know nothing about the code (even if you do); resist the urge to jump in with your own theory immediately, since the goal is for THEM to find the gap through their own narration, not for you to hand them the answer.
2. Ask clarifying, not leading, questions when something seems glossed over: "what does this variable contain at this point?" rather than "isn't this variable actually null here?"; a clarifying question preserves the self-discovery effect, while a leading question short-circuits it, in effect just answering the question yourself, in a slightly slower way.
3. Notice when they pause or hesitate on a specific line, since that hesitation is itself a signal, often the exact spot where their explanation doesn't fully hold together, even if THEY haven't consciously noticed it yet; gently returning to that point ("can you say more about what happens right there?") is more effective than moving on.
4. Let silence do some of the work. Resisting the urge to fill every pause with your own guess gives them room to keep narrating and often arrive at the insight themselves, which is both faster (no back-and-forth debugging your OWN, potentially wrong, theory) and more valuable for their own learning than being handed the answer.

When to escalate to active pair programming instead of continuing to guide:

• When the explanation reveals a gap in UNDERSTANDING (not just an overlooked line) that narration alone won't resolve, e.g., a genuine misunderstanding of how a language feature or library behaves, where providing the missing knowledge is more useful than more questions.
• When you notice, through their narration, something concrete they clearly haven't seen (a specific line, a specific value) and pointing it out directly is faster and kinder than continuing an extended Socratic process once the value of self-discovery has been exhausted for this specific bug.
• When time pressure genuinely doesn't allow for the (often somewhat slower) self-discovery process, in which case switching to active pairing, working the problem together directly, is the pragmatic choice, with an explicit acknowledgment that you're switching modes for a good reason, not simply losing patience.

Worked example

A teammate stuck on why a function returns stale data: walking them through explaining the code line by line, they narrate "and then this checks the cache... and returns it if it's still valid... I wanted a sixty-second expiry, so I set the value to sixty thousand..." and pause, mid-sentence, on "sixty thousand," suddenly noticing they'd been assuming the codebase's convention was MILLISECONDS (so they multiplied their intended sixty seconds by 1000 before storing it), when the surrounding code that actually reads this field treats it as SECONDS directly, with no conversion; storing 60000 where the code expects 60 means the real expiry is 60000 seconds (about 16.7 hours), roughly 1000x longer than the sixty seconds they intended, which is exactly why the data was going stale. They found this entirely through their own narration; the only input needed was staying quiet and letting them keep talking through it.

Trade-offs and pitfalls

Rubber-duck debugging (with a real, silent listener) works best for a SPECIFIC kind of stuck-ness, an unexamined assumption, not for a genuine knowledge gap or a problem requiring information the person simply doesn't have; recognizing which kind of "stuck" you're looking at, and switching to direct guidance or active pairing when it's the latter, is what keeps the technique from wasting time on a problem it was never suited to solve.

Direct answer

The out-of-bounds write is almost certainly a signed/unsigned index mismatch or an off-by-one in the removal logic itself, both classic in-place-array-manipulation pitfalls that "works in tests" precisely because typical test arrays are small and don't happen to exercise the exact boundary condition that triggers the bug, while production data occasionally does.

Structured elaboration

1. Reproduce with a boundary-focused test case first, rather than trying to guess from code review alone: specifically test with an array where duplicates occur near the END of the array (the last one or two elements), since in-place removal algorithms that shift elements left as duplicates are removed are especially prone to write-index errors exactly at the tail, where there's less "room" for an off-by-one to go unnoticed.
2. Check for a signed/unsigned index mismatch specifically, since the question calls it out directly: if the write index is computed via a subtraction that can go negative in an edge case (e.g., removing duplicates from a very short array, or one that's ENTIRELY duplicates), and that index is stored in an unsigned integer type, a negative result wraps around to a huge positive value, producing a write far outside the array's actual bounds, which is a classic C/C++/embedded-systems bug (also possible in any language with explicit unsigned integer types) and matches the "out-of-bounds write" symptom precisely, as opposed to a simple off-by-one which would typically write just one slot too far, not wildly out of bounds.
3. Check for integer overflow in any index arithmetic if the array or index values could be large enough to approach the integer type's limit, though this is less likely to be the specific cause here unless the array sizes involved are unusually large.
4. Check for aliasing/in-place-mutation hazards: if the removal logic reads from and writes to the SAME underlying array simultaneously (common in an in-place algorithm), a read that should happen BEFORE a corresponding write, but doesn't due to a loop-ordering bug, can produce corrupted results distinct from, but sometimes confused with, an out-of-bounds write; confirming which specific failure mode is occurring (via the debugger or a sanitizer) avoids fixing the wrong mechanism.
5. What tests to add once the specific mechanism is found: boundary cases specifically (duplicates at the very start, very end, an array that's entirely duplicates, an array with no duplicates at all, a single-element array, an empty array), since these are exactly the cases most likely to expose an off-by-one or signed/unsigned bug that a "typical" test case with duplicates scattered comfortably in the middle would never exercise.

Worked example

Reproducing with an array that's ENTIRELY duplicate values (an aggressive boundary case) triggers the crash reliably, where a more typical mixed test case doesn't. Stepping through with a debugger shows the write index computed as write_pos = write_pos - 1 at one point in the loop, intended to "back up" one slot when a duplicate is detected, but under this specific input pattern, write_pos reaches zero and then goes negative on the next iteration; since it's declared as an unsigned type, that negative value wraps to a very large positive number, and the subsequent array write at that "index" is wildly out of bounds, matching the crash symptom exactly. Fix: change the index variable to a signed type (allowing the intermediate negative value to be handled correctly, with an explicit check before it's ever used as an actual array index) or restructure the loop logic to avoid ever needing to decrement below zero in the first place, whichever better fits the surrounding code's conventions.

Trade-offs and pitfalls

A naive fix like clamping the index to zero whenever it "looks" negative, without understanding WHY it went negative, risks silently producing a different, subtler correctness bug (skipping or duplicating an element) instead of a crash, which is arguably worse, since it fails silently rather than loudly; understanding the exact arithmetic that produces the negative value, and fixing the LOGIC rather than just guarding the symptom, is what actually resolves it correctly.

Direct answer

Confirmation bias, anchoring, and sunk-cost fallacy each distort a root-cause investigation in a specific, predictable way: confirmation bias makes you notice evidence supporting your first guess and discount evidence against it; anchoring makes an early, possibly wrong hypothesis dominate the rest of the investigation even after better evidence emerges; and sunk-cost fallacy keeps you investigating a disproven lead because of time already invested in it, rather than switching based on current evidence. Concrete team practices (pair debugging, rotating investigators, hypothesis logs, explicit acceptance criteria) counter each of these by introducing structure that doesn't rely on individual willpower to overcome the bias.

Structured elaboration

Confirmation bias: once you suspect a cause, you unconsciously interpret ambiguous evidence as supporting it and are quicker to dismiss evidence that doesn't fit. In debugging, this looks like reading a log line as "consistent with my theory" when a more neutral read would call it inconclusive, or stopping the investigation the moment ANY supporting evidence appears rather than continuing to look for disconfirming evidence too.

• Mitigation: hypothesis logs. Writing down each hypothesis and what specific evidence would DISCONFIRM it, before looking for evidence, forces a falsifiable framing up front; when you later find evidence, you check it against the pre-written disconfirmation criteria rather than retroactively deciding it "counts" as support.

Anchoring: the first plausible explanation offered (by you, or by someone else on the call) tends to dominate the rest of the investigation's framing, even as better evidence appears, because everyone's mental model has already organized around it.

• Mitigation: rotating investigators / a fresh pair of eyes. Someone joining the investigation LATE, without the accumulated anchor, will naturally form hypotheses from the current evidence rather than the initial framing, and is often the person who notices the anchor was wrong; deliberately bringing in a fresh perspective partway through a long investigation is a structural way to interrupt this.

Sunk-cost fallacy: having spent two hours pursuing one lead makes it psychologically harder to abandon, even once evidence stops supporting it, because abandoning it "wastes" the time already spent (which is, of course, already spent either way, and not actually recoverable by continuing).

• Mitigation: explicit acceptance criteria and timeboxing set in advance. Deciding, BEFORE starting to investigate a specific hypothesis, what evidence would confirm it and how much time is reasonable to spend testing it, makes the "abandon or continue" decision a pre-committed rule rather than an in-the-moment judgment call that sunk cost can distort.

Pair debugging as a mitigation for all three simultaneously: a second person, thinking independently, is less likely to share the exact same anchor or the exact same sunk-cost attachment to a specific lead, and naturally provides a real-time check on confirmation bias by asking "does that evidence actually support that, or are we reading it generously?"

Worked example

A team investigating an intermittent failure anchors early on "it's probably the recent deploy" (a reasonable first guess, given the timing). Two hours in, with the deploy's code reviewed thoroughly and nothing found, sunk cost starts to argue for continuing to scrutinize that same deploy rather than considering it possibly unrelated. A hypothesis log, written at the start, had specified "if the failure recurs on a service instance that predates this deploy, that disconfirms the deploy hypothesis"; checking that specific, pre-committed criterion shows the failure DID recur on an older instance, cleanly disconfirming the deploy theory despite two hours of sunk investigation into it. A rotating fresh investigator, brought in specifically because the original two were stuck, asks a question neither anchored investigator had considered ("what else changed around that time besides the deploy?") and identifies an unrelated infrastructure change that actually explains the failure.

Trade-offs and pitfalls

These practices have a real cost (pairing takes two people's time instead of one; hypothesis logs take a few minutes to write that could otherwise go straight into investigating), and the trade-off is worth it specifically for investigations that are ALREADY taking a long time or where the cost of a wrong conclusion is high; for a five-minute, low-stakes bug, the overhead of formal hypothesis logging isn't proportionate to the risk these biases actually pose in that context.

Direct answer

Investigate by looking at the DISTRIBUTION of latencies the patch changed, not just the two summary numbers: a change that helps the median while hurting the tail usually means the patch introduced a new, occasional slow path (a lock, a retry, a cold-cache miss, a GC pause) that wasn't present before, even though it made the common case faster. The fix should target that specific slow path, ideally without giving back the average-case win.

Structured elaboration

1. Confirm the shape of the regression with a full latency histogram, not just average and p99 in isolation. Did the whole distribution shift, or did a new secondary "hump" appear at the tail while the bulk of requests got faster? These imply very different causes.
2. Look for what the patch changed that could introduce occasional cost. Common culprits: added caching (fast on hit, slow on the now-rarer miss, especially a cold cache after deploy), added batching (fast on average, but a request unlucky enough to wait for a batch to fill sees added latency), a lock or synchronization primitive introduced to make the common path more efficient, or a retry/backoff path that wasn't there before.
3. Correlate slow outliers with a specific condition. Pull the individual traces for requests in the new p99 tail and look for what they have in common: a specific input size, a cache miss, contention with a background job, a specific shard or partition.
4. Reproduce the tail behavior in isolation once you have a hypothesis, ideally with a small, targeted load test that forces the suspected condition (a cold cache, high concurrency, a large payload) rather than waiting for it to occur naturally in production traffic.
5. Fix the specific slow path, not the whole change. The goal is almost never to revert the improvement; it's to bound the cost of the new slow path (a timeout, a smaller batch window, a non-blocking fallback) while keeping the average-case gain.

Worked example

A patch that adds request coalescing (batching several concurrent identical lookups into one backend call) drops average latency by 30% but pushes p99 up by 4x. The latency histogram shows a new cluster of requests waiting close to the full batch window (say, 50ms) even when they were the only request in flight, because the coalescing logic always waits for the window to close before dispatching, even with nothing to batch against. The fix: dispatch immediately if no other request has joined the batch within the first few milliseconds, keeping the win for genuinely concurrent traffic while removing the unnecessary wait for solo requests, which is exactly the group hurting the p99.

Trade-offs and pitfalls

A tempting shortcut is to just revert the patch to restore the old p99, which throws away a real average-case improvement to fix a bug that's usually addressable with a smaller, targeted change. The other trap is chasing the p99 number without ever pulling individual slow traces: two very different mechanisms (a cold-cache path and a lock-contention path) can both show up as "p99 got worse," and the fix for one does nothing for the other.