Debugging and Systematic Troubleshooting Questions

Direct answer

Confirmation bias, anchoring, and sunk-cost fallacy each distort a root-cause investigation in a specific, predictable way: confirmation bias makes you notice evidence supporting your first guess and discount evidence against it; anchoring makes an early, possibly wrong hypothesis dominate the rest of the investigation even after better evidence emerges; and sunk-cost fallacy keeps you investigating a disproven lead because of time already invested in it, rather than switching based on current evidence. Concrete team practices (pair debugging, rotating investigators, hypothesis logs, explicit acceptance criteria) counter each of these by introducing structure that doesn't rely on individual willpower to overcome the bias.

Structured elaboration

Confirmation bias: once you suspect a cause, you unconsciously interpret ambiguous evidence as supporting it and are quicker to dismiss evidence that doesn't fit. In debugging, this looks like reading a log line as "consistent with my theory" when a more neutral read would call it inconclusive, or stopping the investigation the moment ANY supporting evidence appears rather than continuing to look for disconfirming evidence too.

• Mitigation: hypothesis logs. Writing down each hypothesis and what specific evidence would DISCONFIRM it, before looking for evidence, forces a falsifiable framing up front; when you later find evidence, you check it against the pre-written disconfirmation criteria rather than retroactively deciding it "counts" as support.

Anchoring: the first plausible explanation offered (by you, or by someone else on the call) tends to dominate the rest of the investigation's framing, even as better evidence appears, because everyone's mental model has already organized around it.

• Mitigation: rotating investigators / a fresh pair of eyes. Someone joining the investigation LATE, without the accumulated anchor, will naturally form hypotheses from the current evidence rather than the initial framing, and is often the person who notices the anchor was wrong; deliberately bringing in a fresh perspective partway through a long investigation is a structural way to interrupt this.

Sunk-cost fallacy: having spent two hours pursuing one lead makes it psychologically harder to abandon, even once evidence stops supporting it, because abandoning it "wastes" the time already spent (which is, of course, already spent either way, and not actually recoverable by continuing).

• Mitigation: explicit acceptance criteria and timeboxing set in advance. Deciding, BEFORE starting to investigate a specific hypothesis, what evidence would confirm it and how much time is reasonable to spend testing it, makes the "abandon or continue" decision a pre-committed rule rather than an in-the-moment judgment call that sunk cost can distort.

Pair debugging as a mitigation for all three simultaneously: a second person, thinking independently, is less likely to share the exact same anchor or the exact same sunk-cost attachment to a specific lead, and naturally provides a real-time check on confirmation bias by asking "does that evidence actually support that, or are we reading it generously?"

Worked example

A team investigating an intermittent failure anchors early on "it's probably the recent deploy" (a reasonable first guess, given the timing). Two hours in, with the deploy's code reviewed thoroughly and nothing found, sunk cost starts to argue for continuing to scrutinize that same deploy rather than considering it possibly unrelated. A hypothesis log, written at the start, had specified "if the failure recurs on a service instance that predates this deploy, that disconfirms the deploy hypothesis"; checking that specific, pre-committed criterion shows the failure DID recur on an older instance, cleanly disconfirming the deploy theory despite two hours of sunk investigation into it. A rotating fresh investigator, brought in specifically because the original two were stuck, asks a question neither anchored investigator had considered ("what else changed around that time besides the deploy?") and identifies an unrelated infrastructure change that actually explains the failure.

Trade-offs and pitfalls

These practices have a real cost (pairing takes two people's time instead of one; hypothesis logs take a few minutes to write that could otherwise go straight into investigating), and the trade-off is worth it specifically for investigations that are ALREADY taking a long time or where the cost of a wrong conclusion is high; for a five-minute, low-stakes bug, the overhead of formal hypothesis logging isn't proportionate to the risk these biases actually pose in that context.

Direct answer

The loop's exclusive upper bound, end = start + page_size, is computed without checking against items.size(), so on the final page (whenever the total item count is not an exact multiple of page_size) the loop reads past the end of the vector. That's undefined behavior, not a clean "missing item": it can silently return garbage values appended after the real data, or crash outright, depending on what memory happens to sit past the buffer. The fix clamps end to min(start + page_size, items.size()).

Structured elaboration

vector::operator[] performs no bounds checking in the standard library implementations most compilers ship, so reading items[i] for an i at or beyond items.size() is undefined behavior: the compiler is not required to produce any particular result, including a crash. This is precisely what makes the bug dangerous in practice: it can appear to "work" (returning plausible-looking but wrong data) far more often than it visibly crashes, which is worse for debugging because nothing flags the failure.

Worked example

#include <vector>
#include <cstdio>
using namespace std;

vector<int> get_page(const vector<int>& items, int page, int page_size) {
  int start = page * page_size;
  int end = start + page_size;         // BUG: not clamped to items.size()
  vector<int> out;
  for (int i = start; i < end; ++i) {
    out.push_back(items[i]);            // reads past the end on the last page
  }
  return out;
}

int main() {
  vector<int> items = {10, 20, 30, 40, 50, 60, 70};
  auto result = get_page(items, 2, 3);   // start=6, end=9
  printf("result.size() = %zu\n", result.size());
  printf("result = [");
  for (size_t i = 0; i < result.size(); ++i) {
    printf("%d%s", result[i], i + 1 < result.size() ? ", " : "");
  }
  printf("]\n");
  return 0;
}

Compiled and run without a sanitizer:

$ clang++ -std=c++17 -o s13 s13.cpp && ./s13
result.size() = 3
result = [70, 0, 0]

Index 6 (70) is valid; indices 7 and 8 are out of bounds, and in this un-sanitized build happened to read zeroed heap memory, appending two garbage 0s instead of stopping after the one real item. Compiling the identical source with AddressSanitizer instead of catching it silently makes the undefined behavior fail loudly and precisely:

$ clang++ -std=c++17 -fsanitize=address -o s13_asan s13.cpp && ./s13_asan
==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603000001c1c
READ of size 4 at 0x603000001c1c thread T0
    #5 0x... in get_page(std::vector<int, std::allocator<int>> const&, int, int) s13.cpp
    #6 0x... in main s13.cpp
0x603000001c1c is located 0 bytes after a 28-byte region

The fixed version:

#include <vector>
#include <algorithm>
#include <cstdio>
using namespace std;

vector<int> get_page(const vector<int>& items, int page, int page_size) {
  int start = page * page_size;
  int end = min(start + page_size, (int)items.size());
  vector<int> out;
  for (int i = start; i < end; ++i) out.push_back(items[i]);
  return out;
}

int main() {
  vector<int> items = {10, 20, 30, 40, 50, 60, 70};
  auto result = get_page(items, 2, 3);
  printf("result = [");
  for (size_t i = 0; i < result.size(); ++i) {
    printf("%d%s", result[i], i + 1 < result.size() ? ", " : "");
  }
  printf("]\n");
  return 0;
}

$ clang++ -std=c++17 -o s13_fixed s13_fixed.cpp && ./s13_fixed
result = [70]

No out-of-bounds access, and the correct single item for the last page.

Trade-offs and pitfalls

The original bug report described the symptom as "missing last item," which is one PLAUSIBLE manifestation of this same defect (if the garbage values happened to look like empty/default entries rather than visibly wrong numbers, a caller might interpret the page as short one item), but as this run shows, the actual manifestation of unclamped out-of-bounds reads is undefined and build-dependent: it can be silent garbage, a crash, or, less commonly, something that happens to look correct by luck. The lesson generalizes: never trust that undefined behavior manifests the same way twice, and prefer .at() (which throws std::out_of_range on a bad index) over operator[] during development specifically because it converts silent UB into a loud, debuggable failure.

Direct answer

Investigate by looking at the DISTRIBUTION of latencies the patch changed, not just the two summary numbers: a change that helps the median while hurting the tail usually means the patch introduced a new, occasional slow path (a lock, a retry, a cold-cache miss, a GC pause) that wasn't present before, even though it made the common case faster. The fix should target that specific slow path, ideally without giving back the average-case win.

Structured elaboration

1. Confirm the shape of the regression with a full latency histogram, not just average and p99 in isolation. Did the whole distribution shift, or did a new secondary "hump" appear at the tail while the bulk of requests got faster? These imply very different causes.
2. Look for what the patch changed that could introduce occasional cost. Common culprits: added caching (fast on hit, slow on the now-rarer miss, especially a cold cache after deploy), added batching (fast on average, but a request unlucky enough to wait for a batch to fill sees added latency), a lock or synchronization primitive introduced to make the common path more efficient, or a retry/backoff path that wasn't there before.
3. Correlate slow outliers with a specific condition. Pull the individual traces for requests in the new p99 tail and look for what they have in common: a specific input size, a cache miss, contention with a background job, a specific shard or partition.
4. Reproduce the tail behavior in isolation once you have a hypothesis, ideally with a small, targeted load test that forces the suspected condition (a cold cache, high concurrency, a large payload) rather than waiting for it to occur naturally in production traffic.
5. Fix the specific slow path, not the whole change. The goal is almost never to revert the improvement; it's to bound the cost of the new slow path (a timeout, a smaller batch window, a non-blocking fallback) while keeping the average-case gain.

Worked example

A patch that adds request coalescing (batching several concurrent identical lookups into one backend call) drops average latency by 30% but pushes p99 up by 4x. The latency histogram shows a new cluster of requests waiting close to the full batch window (say, 50ms) even when they were the only request in flight, because the coalescing logic always waits for the window to close before dispatching, even with nothing to batch against. The fix: dispatch immediately if no other request has joined the batch within the first few milliseconds, keeping the win for genuinely concurrent traffic while removing the unnecessary wait for solo requests, which is exactly the group hurting the p99.

Trade-offs and pitfalls

A tempting shortcut is to just revert the patch to restore the old p99, which throws away a real average-case improvement to fix a bug that's usually addressable with a smaller, targeted change. The other trap is chasing the p99 number without ever pulling individual slow traces: two very different mechanisms (a cold-cache path and a lock-contention path) can both show up as "p99 got worse," and the fix for one does nothing for the other.

Direct answer

With a stack trace and log snippet in hand, the first five steps are: (1) read the stack trace's top frame to identify the exact line and null value involved, (2) check the log immediately before the exception for the request/input context, (3) determine whether this is reproducible or a one-off, (4) check recent deploys or config changes around the failure's first occurrence, and (5) assess blast radius (how many requests/users affected) to decide urgency before digging further.

Structured elaboration

1. Read the stack trace's top frame precisely. A NullPointerException-like error names the exact line where a null was dereferenced; note the specific variable and the surrounding code, since that tells you WHAT was null, which immediately narrows the space of "why" hypotheses.
2. Check the log lines immediately before the exception for request context: what input, user, or upstream call was in flight when the null occurred. This is often the fastest way to form a concrete hypothesis rather than staring at the code in the abstract.
3. Determine reproducibility. Try to trigger the same code path with the same or similar input locally or in staging; a reliably reproducible case turns the rest of the investigation into ordinary debugging, while a genuinely intermittent one requires the log/trace evidence to carry more of the weight.
4. Check what changed recently. A deploy or config change around the time this error first appeared is a strong lead; correlate the error's first-seen timestamp against the deploy history before assuming it's an old, dormant bug that just got triggered by unusual input.
5. Assess blast radius and urgency, specifically before spending more time investigating: is this a single edge-case request, or is it happening at a rate that indicates a systemic problem affecting many users? This determines whether the next step is "keep investigating calmly" or "mitigate immediately, diagnose in parallel."

Concrete artifacts/data points to request or inspect at each step: the exact request ID or correlation ID tied to the failing request (to pull its FULL log trail, not just the exception itself); the deploy/change history for the service in the relevant time window; a count or rate of this specific exception over time (one occurrence versus a sustained rate materially changes urgency); and, if available, a sample of two or three OTHER occurrences of the same exception, to check whether they share a common input pattern or are scattered and unrelated.

Worked example

A NullPointerException at OrderService.java:212, in a block computing a discount based on customer.getLoyaltyTier(). The log line just before shows the request was for a specific customer ID; pulling that customer's full record shows their loyaltyTier field is null, unlike the typical customer record which always has a default tier set. Checking recent changes shows a new customer-import path (added in a deploy two days prior) that bypasses the usual account-creation flow and its default-value logic. That's the concrete root cause: a specific new code path creates customer records without the default tier, and any downstream code that assumes the tier is always set will fail exactly like this, for exactly the customers created through that path.

Trade-offs and pitfalls

The temptation under time pressure is to jump straight to "add a null check" without doing steps 2 through 4, which fixes THIS symptom but leaves the actual root cause (customer records being created in an inconsistent state) free to cause a different failure somewhere else that assumes the same invariant. A null check is a reasonable immediate mitigation once you understand the mechanism, but the investigation isn't complete until you know WHY the null got there in the first place.

Direct answer

A concrete story: a service occasionally returned stale pricing data to a subset of users, detected via a customer complaint rather than any internal alert (since the values were plausible-looking, just wrong, not obviously broken); the root cause traced to a caching layer that keyed its cache entries incorrectly, causing two logically-distinct pricing contexts to collide and overwrite each other's cached value, and the permanent fix corrected the cache key's uniqueness rather than just adjusting the cache's expiry time.

Structured elaboration

How it was detected: a customer support ticket reported seeing a price that didn't match what should have applied to their account tier, with no corresponding error or alert on the engineering side, since the returned value was a real, validly-formatted price, just the WRONG one; this is a useful detail because it illustrates a class of bug (returning plausible-but-wrong data) that's structurally invisible to error-rate-based monitoring, and only surfaces via a downstream consumer noticing a substantive discrepancy.

How it was reproduced: confirming the report wasn't a one-off required identifying the PATTERN, not just the single instance; checking whether other users on the same account tier around the same time window also received an unexpected price showed a small but real cluster, ruling out "one weird one-off" and confirming a systemic, reproducible mechanism worth a full investigation.

Debugging tools and techniques used: traced the pricing-lookup code path for the affected requests, and found it flows through an in-memory cache keyed, it turned out, on account tier ALONE rather than on the combination of account tier AND region (pricing legitimately varies by both); when two users on the same tier but different regions made requests close together in time, the second request's result could overwrite the first's cache entry under the shared, insufficiently-specific key, and a THIRD user (same tier, either region) arriving shortly after could then receive whichever region's price happened to be cached most recently, regardless of their own actual region.

The root cause: a cache key that didn't include every dimension the underlying value actually varied by, a classic caching-correctness bug: the cache was implicitly promising "this value is valid for anyone with this tier," when the real invariant needed was "this value is valid for anyone with this tier AND this region."

The permanent fix implemented: updated the cache key to include region alongside tier, restoring the correct invariant; also added a specific integration test that exercises exactly this scenario (two regions, same tier, interleaved requests) to catch a regression of this specific mechanism in the future, since the original bug had shipped without any test covering this particular combination of dimensions.

What you learned that helps you avoid similar bugs: whenever introducing a cache, explicitly enumerate every dimension the cached value can legitimately vary by, and verify the cache key includes ALL of them, not just the ones that happen to be obvious or top-of-mind at implementation time; a caching bug of this shape is especially dangerous specifically because it fails SILENTLY (no error, no crash, just occasionally-wrong data) and is invisible to typical error-rate monitoring, which argues for treating "does this cache key capture every dimension of variation" as a deliberate design-review question on any future caching work, not something to verify only after a bug report arrives.

Trade-offs and pitfalls

The tempting quick fix, once the symptom (stale/wrong cached price) was understood, would have been to simply reduce the cache's TTL (expiry time), which would have reduced the WINDOW during which a collision could produce visibly wrong data without fixing the actual collision mechanism at all; the permanent fix specifically addressed the cache KEY's correctness, not the expiry duration, since a shorter TTL would have masked the bug's visible frequency without removing its root cause.