Systematic Debugging and Root Cause Analysis Questions

Direct answer

A single-frame glitch that worsens under load smells like a race between the sim and render threads. Most engines double-buffer so the renderer always reads a stable snapshot, and a missing or misplaced sync point lets it read a buffer mid-write.

Structured elaboration

• Check the sim-to-render handoff: sim writes buffer A while render reads buffer B, then they swap; an unfenced swap lets render start reading A before sim finishes writing it.
• A telltale symptom is a "torn" read: position updates correctly but rotation, written moments later in the same update, still shows the old value for one frame.
• Widen the race by adding artificial CPU load, since "worse under load" points at timing, not deterministic logic.
• Fix with a genuine synchronization point, an atomic buffer-pointer swap, so render never starts before sim fully publishes its writes.

Worked example

$\frac{1000}{60} \approx 16.67 \text{ ms per frame}$

An unfenced swap can let render read buffer A microseconds before sim finishes writing it, producing exactly one frame of torn state before the next swap self-corrects it.

Trade-offs and pitfalls

Locking every field eliminates the race but can stall the render thread, reintroducing frame-time problems; prefer a lock-free atomic swap. Verify the fix with the same artificial-load test, not just its apparent disappearance.

What the interviewer probes next

How you'd design the swap to avoid stalling either thread, and how you'd detect a torn read automatically.

Debugging a multithreaded C++ crash starts with reproducing under a sanitizer, since a plain crash trace from a race often just shows whichever thread happened to fault, not the real conflicting access.

Verified example: build and run under ThreadSanitizer

#include <pthread.h>
#include <stdio.h>
int count = 0;
void* t(void*_) { for(int i=0;i<1000000;i++) count++; return NULL; }
int main(){ pthread_t a,b; pthread_create(&a,NULL,t,NULL); pthread_create(&b,NULL,t,NULL);
  pthread_join(a,NULL); pthread_join(b,NULL); printf("%d\n", count); }

Build with gcc -fsanitize=thread -g -o race race.c && ./race: TSan reports a data race on count with both thread backtraces, and the program's printed total is reliably less than 2000000 because count++ is a non-atomic read-modify-write. Fix with an atomic or a mutex:

#include <stdatomic.h>
atomic_int count = 0;
void* t(void*_) { for(int i=0;i<1000000;i++) atomic_fetch_add(&count, 1); return NULL; }

Core dumps, breakpoints, thread backtraces

For a crash rather than a silent wrong-answer race: enable core dumps, attach gdb/lldb to a live hung/crashing process (or load the core), and use thread apply all bt to get every thread's backtrace at once, since the crashing thread is often a victim of another thread's earlier bad write, not the culprit itself. Watchpoints (watch <var>) catch the exact instruction that last modified a corrupted value. Helgrind (valgrind's race detector) is a slower but useful second opinion when TSan isn't available for the toolchain in use.

Trade-offs and pitfalls

Sanitizers add real overhead (roughly 2-20x depending on tool), so they run in CI/staging on representative inputs, not as the default production build; a race that only appears under specific production concurrency may need a staging load test built specifically to reproduce that concurrency level before the sanitizer can catch it.

Direct answer

Identical inputs with different outcomes means the simulation itself isn't bit-for-bit deterministic, usually floating-point non-associativity, hash-container iteration order, or uninitialized state. Hash the full simulation state every tick on both clients and compare to find the exact first divergent tick, rather than eyeballing the visual result minutes later.

Structured elaboration

• Compute an identical checksum over every entity's relevant state at the end of each tick; send just the hash.
• The first tick where hashes differ is the divergence point; work backward from there.
• Check for non-associative float math across CPU paths (SSE (SSE = a CPU's vectorized floating-point instructions, which can accumulate rounding slightly differently than the plain, non-vectorized "scalar" path for the same math) vs scalar), hash-map enumeration order, or an RNG seeded differently per client.
• Fix by forcing one deterministic code path, fixed-point math or ordered containers, for anything feeding simulation state.

Worked example

$10 \times 60 \times 30 = 18{,}000 \text{ ticks in a 10-minute match at 30 ticks/sec}$

A per-tick rounding error as small as $10^{-7}$ seems negligible once, but compounding across 18,000 ticks turns it into a visible, multi-unit position divergence after a few minutes.

Trade-offs and pitfalls

Fixed-point math guarantees determinism but is more work, and mixing it with leftover float code reintroduces the exact bug. A per-tick hash adds bandwidth and CPU but catches desyncs immediately instead of relying on a vague player report.

What the interviewer probes next

How you'd keep the state hash cheap enough not to affect frame time, and which parts of a simulation are safe to leave non-deterministic.

A core dump is a snapshot of a process's memory and register state at the moment of a crash; analyzing it lets you reconstruct the call stack and inspect variable state without having been attached live when it happened.

Step by step

1. Ensure core dumps are enabled and captured (ulimit -c unlimited, and on Linux confirm /proc/sys/kernel/core_pattern writes somewhere retrievable, since containers often need this configured explicitly or the dump is silently discarded on exit).
2. Match symbols to the exact binary. The core dump alone is just raw memory; you need the exact binary and shared libraries (same build, same versions) plus debug symbols to make sense of it. A version mismatch is the single most common reason analysis fails or produces nonsense frames.
3. Load into a debugger: gdb -c core.1234 /path/to/exact/binary, then bt (backtrace) for the call stack, thread apply all bt for all threads if it's a multithreaded crash, and inspect specific frames/variables with frame N and print. A sample bt output might look like:

#0  0x00007f3a91a2b1e0 in process_record (rec=0x0) at parser.c:142
#1  0x00007f3a91a2a9f4 in handle_batch (batch=...) at worker.c:88
#2  0x00007f3a91a29c10 in main (argc=3, argv=...) at main.c:24

Frame #0 naming a file and line inside your own codebase (parser.c:142, not a library path) is what points the fault at your code rather than a dependency; a null rec argument here would be the concrete lead to chase.
4. In a container, retrieve the core file from the container's filesystem or a shared volume before the container is removed (crashes often trigger automatic container restart/cleanup that deletes the evidence), and be aware ASLR (address space layout randomization) shifts addresses between runs, so symbol resolution must account for the load offset recorded in the core file, not assume a fixed base address.
5. For a proprietary binary with no source, work with whatever symbol table is available (even partial), and if none exists, escalate to the vendor with the exact reproduction steps and binary version, since blind byte-level analysis without any symbols rarely yields an actionable fix.

Trade-offs and pitfalls

Core dumps can contain sensitive memory (credentials, user data); production processes handling sensitive data need a policy for who can access a dump and how long it's retained. The other common failure is capturing the dump but not the exact matching build artifacts alongside it, which makes the dump unanalyzable months later when the build has moved on.

A stack trace is the ordered list of function calls active at the moment of a crash or exception, from the outermost caller down to the exact frame where the failure occurred; reading it top-down (or bottom-up depending on convention) tells you the call path that led to the failure, and the top frame usually names the immediate failing operation.

Worked example in a compiled language

A native C service crashes with SIGSEGV, and gdb's backtrace after loading the core shows:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000555555555149 in process_record (rec=0x0) at record.c:88
#1  0x000055555555518a in handle_batch (batch=0x7fffffffe010) at batch.c:42
#2  0x00007ffff7dab083 in main (argc=2, argv=0x7fffffffe128) at main.c:15

This says: at line 88 of record.c, process_record was called with rec equal to a null pointer, called from handle_batch at batch.c:42, called from main. The fix path is the same as in any language: go to record.c:88, find what dereferences rec, and trace backward through handle_batch/main to find why a null record reached that call in the first place (a failed earlier lookup that should have short-circuited? an unchecked allocation failure upstream?).

A Go service shows a similar shape on panic, and because the Go toolchain embeds function names and line numbers by default, the trace already reads like the C example above (main.processRecord(...) at record.go:88, called from main.handleBatch, etc.) without any extra symbolication step, unless the binary was explicitly stripped at build time (-ldflags="-s -w").

When the trace shows only addresses, no names

The C example above assumed symbols were present. In a stripped binary (built without -g, or with symbols stripped after linking), the same crash instead shows raw memory addresses with no names:

#0  0x0000555555555149 in ?? ()
#1  0x000055555555518a in ?? ()

The fix is to symbolicate: obtain the exact binary and matching debug-symbols file used in the deployed build (same version, same compiler flags), then map each address back to a source line, using addr2line -e ./process -f -C 0x0000555555555149 (for C/C++), or go tool addr2line <binary> for a stripped Go binary, or gdb loaded with the correctly matched binary/symbol file instead of the stripped one. A version mismatch between the crashing binary and the symbols you're resolving against is the most common reason this step fails silently, producing plausible-looking but wrong function names.

Trade-offs and pitfalls

Treating the stack trace as the full answer is the common mistake: it tells you where the program noticed the problem, not necessarily why the bad value got there in the first place, which can be several calls earlier. The trace narrows the search; it rarely IS the root cause on its own. Symbolication has its own pitfall: resolving addresses against a binary that merely looks like the right one (same filename, wrong build) produces confident-looking wrong answers, so pin the exact build artifact (checksum or build ID) alongside the core dump when capturing it.