Systematic Debugging and Root Cause Analysis Questions

Debugging a multithreaded C++ crash starts with reproducing under a sanitizer, since a plain crash trace from a race often just shows whichever thread happened to fault, not the real conflicting access.

Verified example: build and run under ThreadSanitizer

#include <pthread.h>
#include <stdio.h>
int count = 0;
void* t(void*_) { for(int i=0;i<1000000;i++) count++; return NULL; }
int main(){ pthread_t a,b; pthread_create(&a,NULL,t,NULL); pthread_create(&b,NULL,t,NULL);
  pthread_join(a,NULL); pthread_join(b,NULL); printf("%d\n", count); }

Build with gcc -fsanitize=thread -g -o race race.c && ./race: TSan reports a data race on count with both thread backtraces, and the program's printed total is reliably less than 2000000 because count++ is a non-atomic read-modify-write. Fix with an atomic or a mutex:

#include <stdatomic.h>
atomic_int count = 0;
void* t(void*_) { for(int i=0;i<1000000;i++) atomic_fetch_add(&count, 1); return NULL; }

Core dumps, breakpoints, thread backtraces

For a crash rather than a silent wrong-answer race: enable core dumps, attach gdb/lldb to a live hung/crashing process (or load the core), and use thread apply all bt to get every thread's backtrace at once, since the crashing thread is often a victim of another thread's earlier bad write, not the culprit itself. Watchpoints (watch <var>) catch the exact instruction that last modified a corrupted value. Helgrind (valgrind's race detector) is a slower but useful second opinion when TSan isn't available for the toolchain in use.

Trade-offs and pitfalls

Sanitizers add real overhead (roughly 2-20x depending on tool), so they run in CI/staging on representative inputs, not as the default production build; a race that only appears under specific production concurrency may need a staging load test built specifically to reproduce that concurrency level before the sanitizer can catch it.

An OOM kill and a memory leak look similar (the process dies for lack of memory) but have different signatures and different fixes.

The distinction

An OOM kill is the kernel (or a container's cgroup limit) terminating a process that exceeded its allowed memory abruptly, often under a sudden allocation spike; a memory leak is memory usage climbing steadily over time because something keeps holding references that should have been released.

Telemetry to distinguish them

• RSS over time: a leak shows a slow, steady upward slope over hours/days; a transient spike-then-kill shows a sharp jump right before death with no earlier trend. For example, a leak might read 400 MB at hour 0, 450 MB at hour 1, 510 MB at hour 2, and 590 MB at hour 3 before finally hitting a 600 MB limit, a fairly steady climb; a transient spike instead sits flat around 400 MB for hours and then jumps straight from 400 MB to 2 GB in under a minute right before the kill, with no earlier upward trend at all.
• Heap vs. resident set: if heap usage (managed allocations) grows in lockstep with RSS, the leak is likely in application-level allocations; if RSS grows without heap growing correspondingly, look at off-heap/native memory (buffers, mmap'd regions, native library allocations).
• GC metrics: a leak typically shows GC running more frequently and reclaiming less each time as the leak accumulates dead-but-still-referenced objects; a transient spike shows a sudden allocation burst GC can't keep up with, without necessarily reflecting a long-term trend.
• OOM killer logs (dmesg, journalctl -k): report the exact process and the memory state at kill time, useful for confirming it was the kernel/cgroup limit rather than an application-level crash.

Trade-offs and pitfalls

Treating every OOM kill as a leak leads to chasing a nonexistent slow leak when the real fix is raising a memory limit or smoothing a legitimate spike (e.g., batch size); conversely, dismissing a real leak as "just a spike" because a single graph looks noisy delays the fix until it recurs. The time-series shape (gradual slope vs. sudden step) is the deciding evidence either way.

"Time-travel" debugging for production means being able to replay a specific past request in isolation, later, with full instrumentation, rather than only being able to observe it live once.

Approach and trade-offs

Capture the inputs, any non-deterministic events (timestamps, random values, external call responses), and network interactions for a request at the time it happens (or a sampled subset of requests), store them, and provide a replay mechanism that re-executes the exact same logical request in an isolated environment with a debugger attached or heavier logging enabled, well after the fact.

Key design trade-offs: privacy and storage (capturing full request/response bodies indefinitely for every request is expensive and risky; capture is usually scoped to a sampled subset, or triggered specifically for requests that error or exceed a latency threshold, with sensitive fields redacted before storage), and performance overhead (recording enough detail to make a request truly replayable adds latency/CPU cost on the hot path, so this capability is usually opt-in per-request or triggered by an anomaly rather than always-on for 100% of traffic).

A concrete instance

Take a POST /checkout request: the capture records the request body (cart contents, a discount code lookup), the exact timestamp used to compute a time-limited discount, the random idempotency key generated for the payment call, and the actual response the payment gateway returned, say {"status": "approved", "authCode": "9F3A21"}. All of that gets written to the capture store as a small JSON record keyed by request ID. Later, replaying it means re-running the same /checkout code path but substituting the recorded timestamp, idempotency key, and payment-gateway response for the real ones, with a debugger attached, so the exact sequence of decisions the code made (which discount branch it took, what it sent the payment gateway) plays out identically to the original failing run, without re-calling the real payment API.

Why this helps under pressure

Instead of trying to catch a rare bug live (which may never recur in the same shape), an on-call engineer can pull the exact failing request from the capture store and replay it as many times as needed, with as much instrumentation as needed, without any further risk to production, turning a one-shot, high-pressure live debugging session into a repeatable offline one.

Trade-offs and pitfalls

This only helps for bugs where the captured inputs are actually sufficient to reproduce the behavior; if the bug depends on shared state at the exact moment (another concurrent request, a race with a background job) rather than purely this request's own inputs, replay in isolation won't reproduce it, and the design needs to be honest about that limitation rather than oversold as a universal fix.

The goal is to capture enough detail to diagnose an intermittent failure without paying the overhead of full verbose tracing on every request, all the time.

Approach

1. Baseline cheap signal always on: lightweight metrics and low-cardinality logs run continuously with near-zero overhead, giving you the "something is wrong, roughly when" signal.
2. Conditional/triggered detailed capture: when an anomaly threshold fires (an error, a latency outlier, a specific status code), automatically escalate to a detailed, ephemeral capture for a short window around that event: full request/response payloads (redacted), a stack trace, or a short eBPF trace of the exact syscalls/queries involved, stored only for that flagged window rather than continuously.
3. eBPF/kernel-level sampling for very low-overhead, always-on visibility into syscall latency or scheduling behavior without modifying the running application, useful when you cannot add application-level instrumentation cheaply.
4. On-demand debug snapshots: an operator-triggered dump of in-process state (thread stacks, queue depths, cache sizes) for a live investigation, taken sparingly since it can itself add latency.

Trade-offs and pitfalls

The riskiest failure mode is instrumentation changing the behavior you're trying to observe (turning a heisenbug into something that no longer reproduces once instrumented) - the anomaly-triggered design specifically avoids this by keeping steady-state overhead near zero and only paying the instrumentation cost in the rare window right around a real anomaly. The second risk is threshold tuning: too sensitive and you capture (and pay for) noise constantly; too loose and you miss the actual event. Start conservative and tighten based on how often the trigger fires versus how often it actually captures something useful in review.

git bisect performs binary search over commit history: you mark one commit known-good and one known-bad, it checks out the midpoint, you (or a script) report good/bad, and it repeats, isolating the first bad commit in O(log n) tests. A Python helper can drive this directly through the git CLI via subprocess, one commit at a time, instead of leaving the whole loop to a shell one-liner.

Function interface

import subprocess


def bisect_find_regression(
    good_commit: str,
    bad_commit: str,
    test_command: list,
    repo_path: str = ".",
    retries: int = 5,
    failure_threshold: int = 1,
) -> str:
    """
    Uses `git bisect` (via the git CLI) to find the first commit between
    good_commit and bad_commit that introduced a failure of `test_command`.

    Returns the SHA of the first bad commit.

    Assumptions:
      - repo_path is a clean git checkout with no uncommitted changes: bisect
        checks out different commits in place, and this function does not
        stash or restore working-tree edits.
      - test_command is a list suitable for subprocess.run, e.g.
        ["pytest", "-k", "regression_case"].
      - a commit whose build/setup itself fails (not the test under
        investigation) must be treated as untestable so bisect skips it
        instead of misattributing the regression to an unrelated break;
        this function does that by checking for a distinguished exit code.
      - failure_threshold / retries handle test flakiness: the test is run
        up to `retries` times on a commit and only counted "bad" if it
        fails at least `failure_threshold` times, so one flaky failure on
        an actually-good commit does not derail the search.
    """
    subprocess.run(["git", "bisect", "start"], cwd=repo_path, check=True)
    subprocess.run(["git", "bisect", "bad", bad_commit], cwd=repo_path, check=True)
    subprocess.run(["git", "bisect", "good", good_commit], cwd=repo_path, check=True)

    def _verdict_for_current_checkout() -> str:
        failures = 0
        for _ in range(retries):
            proc = subprocess.run(test_command, cwd=repo_path)
            if proc.returncode == 125:
                return "skip"
            if proc.returncode != 0:
                failures += 1
        return "bad" if failures >= failure_threshold else "good"

    try:
        while True:
            verdict = _verdict_for_current_checkout()
            out = subprocess.run(
                ["git", "bisect", verdict],
                cwd=repo_path,
                capture_output=True,
                text=True,
                check=True,
            )
            if "is the first bad commit" in out.stdout:
                return out.stdout.splitlines()[0].split()[0]
    finally:
        subprocess.run(["git", "bisect", "reset"], cwd=repo_path, check=True)

How it runs tests

Each iteration runs test_command against whatever commit git bisect has currently checked out, using its process exit code as the verdict: 0 means good, a nonzero code (other than 125) means bad, and 125 means "cannot be tested, skip this commit" (the convention git bisect run itself uses, so a wrapper script that already follows it plugs straight into this function's _verdict_for_current_checkout). This was verified end to end against a small real git repository: five commits, a passing test on the first two, a bug introduced on commit four (test starts failing) with an unrelated no-op commit after it, and bisect_find_regression correctly returned the exact commit that introduced the failure.

Handling flaky tests

A test that is flaky will make bisect converge on the wrong commit, because a single failing run at a good commit looks identical to a real regression. _verdict_for_current_checkout runs the test up to retries times per commit and only reports "bad" once at least failure_threshold of those runs failed, so an occasional flake on a good commit does not get reported as bad, while a commit that fails consistently still gets reported bad quickly. Tune retries/failure_threshold to the test's known flake rate: a test that fails 1 in 20 runs when truly "good" needs enough retries that a false-bad verdict is unlikely (e.g. requiring 2+ failures out of 5 runs), while a very stable test can use retries=1.

Assumptions about the environment

• The repo has a linear, bisectable history between good_commit and bad_commit (a single monotonic transition, not multiple unrelated interleaved regressions in the range).
• test_command is runnable from a fresh checkout with no manual setup steps the script doesn't perform (dependencies are installed as part of test_command itself, or already present).
• The caller has already established that good_commit truly passes and bad_commit truly fails before starting; the function does not re-verify the endpoints.

Trade-offs and pitfalls

Bisect assumes a single monotonic transition from good to bad; if the bug is intermittent even in its "bad" state, or if multiple unrelated changes landed in the range, the repeat-N-times guard above is what keeps the search from being derailed, and batched-commit CI setups should bisect at the smallest unit of change actually available, not the batch. Because git bisect reset runs in a finally block, the repo is always left back on its original branch even if the loop is interrupted or a commit is genuinely untestable end to end.