Systematic Debugging and Root Cause Analysis Questions

When the two live-incident options are specifically a rollback of the latest deploy or toggling a feature flag that should revert the new behavior, the choice comes down to which one restores the previous behavior more completely and more verifiably, not simply which is faster to execute.

Comparing the two options

Feature flag toggle: fast (a config change, often seconds to propagate) and narrowly scoped if the flag genuinely gates all of the new behavior; the risk is that the flag may not cover every code path touched by the change (a partial flag, or a change that also altered a shared library or schema outside the flag's reach), in which case toggling it off looks like it worked but leaves some of the regression in place.
Rollback of the latest deploy: restores a fully known-good binary, so it does not depend on the flag actually covering all of the change; the cost is it also reverts any other, unrelated changes bundled in the same deploy, and a rollback typically takes longer to execute and verify than flipping a flag.

Assessing risk

The deciding question is confidence that the flag fully covers the regression: if the change was built and tested specifically to be flag-gated (and QA verified the flag-off path matches the prior release), toggle the flag first since it is faster and reversible with less collateral. If there is any doubt the flag is complete (schema changes, shared state, a partial rollout of the flag itself), prefer the rollback, since a flag toggle that only partially fixes the incident quietly wastes time while customers keep seeing errors.

Preserving data and logs

Before acting, capture the current error logs, traces, and the exact deploy/flag state (which version is live, which flag values are set, for which cohorts), since a rollback in particular can make the failing state harder to inspect afterward; snapshot dashboards and pull a sample of failing requests first if it costs only seconds.

Performing the action and verifying restoration

Toggle the flag (or execute the rollback) for a narrow slice first if that is safely possible (one region, one host group) to confirm the fix actually restores correct behavior before flipping it globally; then verify against the original symptom directly (the specific error rate, endpoint, or user cohort that was affected) rather than a generic drop in overall errors, since a coincidental dip can be mistaken for a fix.

Communication and avoiding further disruption

Announce which action was taken and its known side effects before taking it where possible (for example: "we are toggling flag X off, which also reverts behavior Y that shipped with it") so stakeholders are not surprised, and avoid stacking a second untested change on top while the first action's effect is still being verified, since two simultaneous changes make it impossible to attribute the outcome to either one.

Trade-offs and pitfalls

Defaulting to the flag toggle purely because it is faster, without first confirming it actually covers the full blast radius of the change, is the most common mistake in this scenario; when in doubt about coverage, the rollback's completeness is worth its slower execution time.

Investigating a heisenbug requires accepting the standard toolkit (logging, attaching a debugger) is off the table, since both perturb the exact timing you need to observe.

Non-invasive strategies, roughly in order of overhead

1. Kernel/low-level tracing (ftrace, eBPF, perf): observe scheduling and syscall timing with far less perturbation than application-level logging, since they don't run inside the process's own critical path the same way. (ftrace is Linux's built-in kernel function tracer; eBPF lets you run small sandboxed programs inside the kernel to observe events cheaply; perf is Linux's low-overhead sampling profiler; all three watch the system from outside the process rather than adding code inside it.)
2. Hardware watchpoints: a CPU-level trap that fires when a specific memory address is written, letting you catch the exact write that corrupts shared state without instrumenting the code path at all.
3. Record-and-replay (rr) (Mozilla's open-source record-and-replay debugger): capture one real occurrence (including all syscalls and nondeterministic inputs) once, then replay it deterministically as many times as needed in a full debugger session, which sidesteps the observer-effect problem entirely after the initial capture.
4. Increase trigger probability instead of adding instrumentation: run under heavier concurrent load, or add artificial scheduling pressure (deliberately delay specific operations) to make the natural race window occur more often, without touching the code path being investigated.

Confirmed generalization

The same four techniques apply whether the observing party is a human with a debugger, or an application's own logging framework: anything that runs synchronously in the critical section changes its timing. This is why four independent worked cases across roles (SRE, general engineering, systems engineering, QA) all converge on the same toolset: rr/record-and-replay, kernel-level tracing (ftrace/eBPF), perf sampling, and GDB used only against a replayed/captured session rather than the live race.

Trade-offs and pitfalls

These tools have a real learning curve and setup cost (eBPF and rr both require specific kernel/OS support and practice); the payoff is a heisenbug that would otherwise burn days of guess-and-check becomes a captured, replayable artifact you can step through as many times as needed, which is usually worth the setup cost for anything that recurs.

A stack trace is the ordered list of function calls active at the moment of a crash or exception, from the outermost caller down to the exact frame where the failure occurred; reading it top-down (or bottom-up depending on convention) tells you the call path that led to the failure, and the top frame usually names the immediate failing operation.

Worked example in a compiled language

A native C service crashes with SIGSEGV, and gdb's backtrace after loading the core shows:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000555555555149 in process_record (rec=0x0) at record.c:88
#1  0x000055555555518a in handle_batch (batch=0x7fffffffe010) at batch.c:42
#2  0x00007ffff7dab083 in main (argc=2, argv=0x7fffffffe128) at main.c:15

This says: at line 88 of record.c, process_record was called with rec equal to a null pointer, called from handle_batch at batch.c:42, called from main. The fix path is the same as in any language: go to record.c:88, find what dereferences rec, and trace backward through handle_batch/main to find why a null record reached that call in the first place (a failed earlier lookup that should have short-circuited? an unchecked allocation failure upstream?).

A Go service shows a similar shape on panic, and because the Go toolchain embeds function names and line numbers by default, the trace already reads like the C example above (main.processRecord(...) at record.go:88, called from main.handleBatch, etc.) without any extra symbolication step, unless the binary was explicitly stripped at build time (-ldflags="-s -w").

When the trace shows only addresses, no names

The C example above assumed symbols were present. In a stripped binary (built without -g, or with symbols stripped after linking), the same crash instead shows raw memory addresses with no names:

#0  0x0000555555555149 in ?? ()
#1  0x000055555555518a in ?? ()

The fix is to symbolicate: obtain the exact binary and matching debug-symbols file used in the deployed build (same version, same compiler flags), then map each address back to a source line, using addr2line -e ./process -f -C 0x0000555555555149 (for C/C++), or go tool addr2line <binary> for a stripped Go binary, or gdb loaded with the correctly matched binary/symbol file instead of the stripped one. A version mismatch between the crashing binary and the symbols you're resolving against is the most common reason this step fails silently, producing plausible-looking but wrong function names.

Trade-offs and pitfalls

Treating the stack trace as the full answer is the common mistake: it tells you where the program noticed the problem, not necessarily why the bad value got there in the first place, which can be several calls earlier. The trace narrows the search; it rarely IS the root cause on its own. Symbolication has its own pitfall: resolving addresses against a binary that merely looks like the right one (same filename, wrong build) produces confident-looking wrong answers, so pin the exact build artifact (checksum or build ID) alongside the core dump when capturing it.

Direct answer

Pull Flow Log records for the source and destination ENIs (Elastic Network Interfaces, the virtual network cards attached to each instance) and read the action field. A REJECT for that exact tuple means a security group or NACL (Network Access Control List, a stateless, subnet-level firewall, separate from the per-instance security group) is blocking it, while ACCEPT records with the app still timing out mean the problem is above the network layer entirely.

Structured elaboration

• Query Flow Logs (Athena or CloudWatch Insights) filtered to the incident window and the ENIs/ports involved.
• On REJECT, check both the security group and the NACL, since NACLs are stateless and can block the return leg even when the security group allows the request.
• On ACCEPT with no timely response, look at DNS resolution, the TLS handshake, or the destination process itself, none of which Flow Logs show.
• No records at all suggests routing, a missing route table entry or peering/Transit Gateway (a managed hub that routes traffic between multiple VPCs and on-premises networks over VPN or dedicated connections) misconfiguration, rather than a security rule.

Worked example

action=REJECT for 10.0.1.15:443 -> 10.0.2.20:5432 conclusively points at SG/NACL rules; action=ACCEPT for the same tuple with a client-side timeout redirects the investigation entirely toward the destination service instead.

Trade-offs and pitfalls

Flow Logs sample and aggregate rather than log every packet, so very brief issues can be underrepresented. They carry no payload detail, so ACCEPT doesn't mean the request was handled correctly.

What the interviewer probes next

Why NACLs being stateless matters for return traffic, and how you'd alert on a REJECT spike for a given path.

Tripled logging cost usually means volume grew faster than the value extracted from it; the fix is to cut volume selectively, not uniformly, so the signal that actually gets used survives.

A practical plan

1. Set log-level policy by environment and default: debug logging is fine to leave on in staging but must default to info/warn in production, with a way to raise it temporarily and narrowly (one instance, one request ID, a short TTL) rather than fleet-wide and indefinitely.
2. Sample high-volume, low-value lines (e.g. successful health checks, routine polling) instead of dropping them entirely, so you can still detect a rate change without paying to store every instance.
3. Aggregate/rollup where the individual line rarely matters: turn "1000 identical retry log lines" into one line with a count, and rely on metrics (which are cheap) for anything that's fundamentally a counter, saving log storage for things that need the specific detail (a stack trace, a specific failing payload).
4. Redact and shorten before storage, not after: strip large payloads or PII at the point of logging rather than logging everything and cleaning it up downstream, since the ingestion cost is already paid by the time cleanup happens.
5. Set retention tiers: keep full-fidelity logs for a short, cheap window (days) and only aggregated/rolled-up summaries for the longer compliance window, instead of one flat retention policy for everything.

Monitoring the change itself

Track "log volume per request" and "percentage of debug-triage sessions where the needed line was missing" as the two competing metrics, so cost cuts can be validated against not silently destroying the ability to debug, rather than declared successful purely because the bill went down.

Trade-offs and pitfalls

The main risk is over-trimming: cutting a log line that turns out to be the one thing needed during the next incident. The mitigation is a staged rollout of each cut (reduce, watch for a sprint, then commit) plus keeping an emergency dial to re-enable full verbosity narrowly and fast when an active incident needs it.