Systematic Debugging and Root Cause Analysis Questions

A segmentation fault means the process touched memory it isn't allowed to access; the investigation goal is to turn an intermittent, hard-to-catch native crash into a reproducible, analyzable one.

Tools and steps

1. Reproduce with maximum diagnostic power on: run under gdb directly if it reproduces quickly enough, or with AddressSanitizer (ASan) for a build that instruments memory accesses and reports the exact faulting access (use-after-free, buffer overrun) with a much more precise message than a bare SIGSEGV. For example, ASan might report something like heap-use-after-free ... WRITE of size 4 at 0x60200000eff0 thread T0, immediately naming the access type and the freed allocation's address, rather than the bare Segmentation fault (core dumped) you would get without it.
2. If it's rare in production (roughly once a day) and can't be reproduced on demand: ensure core dumps are enabled (ulimit -c, container-aware core_pattern), add crash-time backtraces via a signal handler if the runtime supports it, and run a staging replica under ASan/valgrind with realistic traffic to try to surface the same class of bug faster than waiting for the rare production occurrence.
3. Analyze the core dump: match the exact binary and libraries, load into gdb, get the backtrace (bt), and inspect the faulting frame's variables; valgrind (slower, but very precise) or a saved ASan report narrows it to a specific allocation/free pair when the fault is memory-safety related rather than a plain null/wild pointer.
4. Distinguish your code vs. a library vs. OS/kernel: check whether the faulting frame's symbol belongs to your binary, a linked library (possible ABI (Application Binary Interface: the low-level contract, calling conventions, struct layouts, symbol names, that a compiled library and the code linking against it must agree on) mismatch or a known bug in that library version), or kernel/OS code (rare, but check dmesg/kernel logs for corroborating hardware or driver errors).

Trade-offs and pitfalls

ASan and valgrind both add real runtime overhead (ASan roughly 2x, valgrind much more), so they're a staging/canary tool, not something you leave on in full production traffic; the practical pattern is running a small percentage of production-like traffic through an instrumented canary continuously, rather than choosing between "always on everywhere" and "never on until it crashes again."

Debugging a multithreaded C++ crash starts with reproducing under a sanitizer, since a plain crash trace from a race often just shows whichever thread happened to fault, not the real conflicting access.

Verified example: build and run under ThreadSanitizer

#include <pthread.h>
#include <stdio.h>
int count = 0;
void* t(void*_) { for(int i=0;i<1000000;i++) count++; return NULL; }
int main(){ pthread_t a,b; pthread_create(&a,NULL,t,NULL); pthread_create(&b,NULL,t,NULL);
  pthread_join(a,NULL); pthread_join(b,NULL); printf("%d\n", count); }

Build with gcc -fsanitize=thread -g -o race race.c && ./race: TSan reports a data race on count with both thread backtraces, and the program's printed total is reliably less than 2000000 because count++ is a non-atomic read-modify-write. Fix with an atomic or a mutex:

#include <stdatomic.h>
atomic_int count = 0;
void* t(void*_) { for(int i=0;i<1000000;i++) atomic_fetch_add(&count, 1); return NULL; }

Core dumps, breakpoints, thread backtraces

For a crash rather than a silent wrong-answer race: enable core dumps, attach gdb/lldb to a live hung/crashing process (or load the core), and use thread apply all bt to get every thread's backtrace at once, since the crashing thread is often a victim of another thread's earlier bad write, not the culprit itself. Watchpoints (watch <var>) catch the exact instruction that last modified a corrupted value. Helgrind (valgrind's race detector) is a slower but useful second opinion when TSan isn't available for the toolchain in use.

Trade-offs and pitfalls

Sanitizers add real overhead (roughly 2-20x depending on tool), so they run in CI/staging on representative inputs, not as the default production build; a race that only appears under specific production concurrency may need a staging load test built specifically to reproduce that concurrency level before the sanitizer can catch it.

An error isolated to a single availability zone (or, similarly, a single canary region behind a feature flag) is a strong, specific clue: whatever differs between that AZ/region and the others is the prime suspect, not the application code in general.

Investigating an AZ-specific failure

1. Compare metrics and logs across AZs for the same time window and the same traffic mix, not in isolation, to confirm the failure really is AZ-specific and not just coincidentally more traffic hit that AZ.
2. Check configuration and image differences between AZs: are all AZs running the exact same deployed version, the same instance type/generation, the same underlying AMI (Amazon Machine Image, the base disk snapshot each instance boots from)/base image? Infrastructure that "should be identical" across AZs sometimes isn't, due to a partial or stuck rollout. For example, diffing AMI IDs across AZs might show AZ-b is still running an older ami-0821 while the other two AZs already rolled forward to ami-0925, which alone would explain failures isolated to that one AZ.
3. Validate whether the root cause is infra (a networking issue specific to that AZ, a degraded underlying host), network (cross-AZ latency/connectivity to a dependency), or application code (a code path only exercised by that AZ's specific traffic characteristics, unlikely if config/image are truly identical).

The feature-flag/canary-region variant

The same reasoning applies when a config change deployed via feature flag only shows failures in the canary region: check whether the failure is genuinely the flag's new code path (isolate by disabling just the flag in that region and observing) versus an environment difference the canary region happens to have that has nothing to do with the flag (e.g., the canary region was already running a slightly different base image before the flag rollout even started).

Trade-offs and pitfalls

It's tempting to assume "it's regional, so it must be infra"; but a code path that's genuinely only exercised in one region (due to routing, data locality, or a region-specific config value) can produce an application bug that looks identical to an infra issue until you specifically rule out configuration/image parity first.

A heisenbug is a bug whose behavior changes or disappears when you try to observe it, most commonly because the act of observing (adding logging, attaching a debugger, running under instrumentation) changes timing enough to avoid the exact interleaving or memory layout that triggers it; the name is a nod to the observer effect.

A useful related taxonomy

• Bohrbug: a solid, deterministic, always-reproduces-the-same-way bug (named for a fixed atomic model), the "normal" kind.
• Heisenbug: changes or vanishes under observation (timing-sensitive races, uninitialized-memory-dependent behavior).
• Mandelbug: has such complex, seemingly chaotic causes (long causal chains, feedback effects) that it appears random even though it's technically deterministic given full state.

Practical techniques for a heisenbug

1. Non-invasive instrumentation that adds minimal timing perturbation: kernel-level tracing (eBPF, ftrace) or hardware watchpoints rather than application-level logging, since the goal is to observe without changing the very timing you're trying to catch.
2. Record-and-replay (rr and similar tools) captures a real execution once, including all nondeterministic inputs, so it can be replayed deterministically afterward for as many debugger sessions as needed without needing to reproduce live again.
3. Statistical/stress approaches: run under heavy load or with deliberately altered scheduling (thread priority tweaks, artificial delays at suspected contention points) to increase the trigger rate rather than trying to catch a rare natural occurrence.
4. Treat a changed symptom under observation itself as diagnostic evidence that timing (not logic) is the mechanism, which should redirect the investigation toward concurrency/ordering rather than deeper into the business logic.

Trade-offs and pitfalls

Chasing a heisenbug with heavier and heavier logging is the classic trap: each added log line changes timing further and can make the bug harder to catch, not easier; the more effective path is minimal, targeted probes plus record-and-replay rather than an escalating amount of instrumentation.

Classifying an incoming failure into "automation/test failure," "environment/infra failure," or "application defect" automatically speeds up triage by routing each case to the right owner without a human reading every failure first.

Design

1. Data sources: test/CI logs, historical flakiness scores per test, infra health signals (was the CI runner/host healthy at the time), and application error signatures (does this match a known defect's error pattern).
2. Heuristics or ML features: correlate the failure with recent deploys/changes (points at application defect), with a specific CI runner or environment (points at infra), or with a test's own historical flakiness rate (points at automation/test issue); a simple, explainable heuristic model (a decision tree over these signals) is usually preferable to an opaque ML model here, since triage decisions need to be auditable.
3. Low-confidence handling: route ambiguous cases to a human reviewer rather than forcing a guess, and track how often the low-confidence bucket is used as a signal that the heuristics need improvement.
4. Integration with bug trackers and accuracy measurement: auto-file or auto-tag issues with the classified bucket, and track precision/recall against human-confirmed ground truth over time to catch the classifier drifting as the codebase and test suite evolve.

A worked instance

Suppose checkout_flow_test fails: the recent deploy touched an unrelated billing module (no file-path correlation with this test), the CI runner reported disk pressure at the exact failure timestamp (an infra signal), and this specific test has a 0% historical flake rate over its last 200 runs. Those three signals together classify the failure as environment/infra with medium confidence, routing it to the infra on-call rather than the billing team.

Confirmed related pattern: security-specific instances

The identical three-way triage applies to CI-only security-test failures and intermittent vulnerability-scan failures: is it the test/scanner itself being flaky, an environment difference (scan scope, plugin version, timing), or a genuine regression in the application. The same signal categories (recent-change correlation, environment/infra health, historical flakiness) resolve it without needing security-specific heuristics.

Trade-offs and pitfalls

An automated classifier that's wrong with high confidence is worse than one that correctly flags low confidence, since a confidently-wrong "it's just flaky" classification can suppress attention on a genuine regression; bias the system toward flagging uncertainty rather than optimizing purely for classification accuracy on the easy cases.