Systematic Debugging and Root Cause Analysis Questions

A short list of error classes, checked reflexively, catches a large share of bugs faster than reasoning from scratch each time, especially under time pressure.

The checklist and where each shows up

• Off-by-one: a paginated API returning one too many/few items, or a loop boundary; assertion/test: check the exact boundary index (first and last page/element) explicitly, not just a "normal" middle case.
• Null/bounds: an unchecked null or an index beyond array/collection length; assertion: a boundary test with an empty collection and a single-element collection.
• Integer overflow: a size or count computation exceeding the integer type's range (a buffer-size calculation, a large-array binary search midpoint); assertion: a test with values near the type's max, not just small representative numbers.
• Index errors: signed/unsigned confusion, or mixing 0-indexed and 1-indexed conventions across two components; assertion: explicit tests at index 0 and index length-1.
• Race conditions: state mutated by more than one thread/request without synchronization; assertion: a concurrency stress test, not just a serial unit test.

Using assertions and invariant checks under time pressure

Adding a small, well-placed assertion (e.g., asserting an index is within expected bounds, or a computed size is non-negative) right before the suspect operation turns a silent wrong answer into an immediate, precise failure at the exact point of violation, which is much faster to act on than a downstream symptom several calls later. In production, convert a temporary debugging assertion into either a proper validated error path (fail gracefully with a clear error) or a monitored invariant check (log and alert, don't crash), rather than leaving raw assertions that could themselves crash production on an edge case.

Trade-offs and pitfalls

This checklist is a starting triage list, not a replacement for understanding the actual code; its value is in ruling several common causes in or out in the first few minutes, which is exactly when time pressure makes a systematic pass most valuable.

Direct answer

Errors specifically under light traffic point away from capacity and toward a timeout mismatch. If the backend's keep-alive timeout is shorter than the load balancer's idle timeout, the backend can silently close a pooled connection the LB still thinks is usable.

Structured elaboration

• AWS ALB defaults to a 60-second idle timeout; many app servers default their keep-alive much lower (older Node.js: 5 seconds).
• A gap longer than the backend's timeout between requests on a pooled connection means the backend already closed it when the LB tries to reuse it, producing a reset (502) or hang (504).
• Under heavy traffic, connections get reused often enough to avoid the gap; under light traffic, gaps past the shorter timeout are common, explaining the clustering.
• Confirm by checking backend logs for connection resets at the same timestamps as the LB errors.

Worked example

$5 \text{ s (backend keep-alive)} < 60 \text{ s (ALB idle timeout)}$

Any idle gap over 5 seconds on a pooled connection creates the failure window.

Trade-offs and pitfalls

Fix by setting the backend's timeout larger than the LB's, for example 65s versus 60s, never the reverse. Lowering the LB's timeout to match also "fixes" it but trades errors for far more frequent connection setup overhead.

What the interviewer probes next

Why the fix direction must be backend-longer-than-LB, and how a synthetic low-traffic canary would catch this.

Auth and session races reliably show up "only under load or in parallel test runs" because they depend on two operations touching shared identity state (a session record, a signing key, a token cache) landing close enough in time to interleave, something a single-threaded manual test never triggers.

Confirming the token-refresh race

Reproduce under concurrent load (many simultaneous refresh calls for the same user/session), instrument the refresh path with fine-grained timestamps around the read-check-write of the session/token state, and look for two refresh calls whose logged operations interleave (both read "no active session," both then create one, producing a duplicate). Concretely, this might look like: request A reads "no active session" at t=0ms, request B reads the same "no active session" state at t=2ms (before A's write has landed), so both requests independently create a session row for the same user, and the second write is the duplicate that later produces an intermittent 401 when the first session is looked up inconsistently. Verify the fix under the same parallel load, not just a single retest, since the fix needs to survive the interleaving that caused the bug, not merely pass a serial test.

Root causes in this class of bug

A signing-key rotation racing with in-flight validation is the same shape: a request validated against the old key arrives just as the key store swaps to the new key, and if the validation path doesn't accept a short overlap window for the previous key, legitimate requests fail for the few minutes until caches fully converge. The fix is usually a deliberate overlap window (accept both old and new key for a bounded grace period) rather than an instantaneous cutover, plus idempotency (making an operation idempotent means running it twice has the same effect as running it once, so a duplicate retry can't create a second session) on session creation (a unique constraint or a check-and-set on a stable session key) so a race that does occur produces one session, not two.

Trade-offs and pitfalls

Making the refresh path fully serialized (a lock per user) removes the race but adds latency and a new single point of contention; a narrower fix (idempotent creation, an accept-both-keys window) usually gets the correctness without that cost, and is the preferred approach unless contention is proven to be a non-issue.

Print/log statements and an interactive debugger solve different problems, and the time-pressure framing usually favors prints.

When each wins

Prints/logs win when: you can't attach a debugger (production, a phone-screen with no IDE, a containerized script), the bug involves timing (a debugger's pauses can hide or change a race), or you need to see a value across many iterations rather than one paused moment.
A debugger wins when: you need to inspect complex object state interactively without predicting in advance which fields matter, or step precisely through a tricky branch a single log line can't capture.

Best practices for temporary debug logs

Use a consistent format (timestamp, a short tag, the value), log at decision points (before/after a branch, before/after I/O), and remove or demote them before merging so they don't become permanent noise. For a containerized script under a time-box, a single well-placed line printing the exact input and the exact point of divergence is usually faster than configuring remote debugging.

Trade-offs and pitfalls

Prints scattered without a plan just relocate the guessing; the discipline that makes them fast is deciding which value, at which point would distinguish your competing hypotheses before adding the line, the same discipline a debugger session needs anyway.

"Crashes on certain files, has a stack trace, but we can't reproduce" is a minimal-reproduction problem: the goal is turning a vague report into a small, deterministic failing case a developer can act on directly.

Step-by-step

1. Gather everything available from the report: the exact stack trace, environment details (OS, browser/runtime version), and, critically, the actual file that was being uploaded if at all retrievable (with the customer's permission/anonymized if sensitive), since "certain files" strongly suggests a specific file characteristic (size, encoding, an edge-case format variant) is the trigger.
2. Build a small harness/script that attempts the same operation against a range of candidate file variations (different sizes, encodings, malformed headers) rather than manually retrying one file at a time.
3. Reduce to the smallest repro: once any variation reproduces the crash, apply a binary-search/delta-debugging approach (repeatedly cut the failing file roughly in half, or remove one chunk at a time, retesting after each cut to check whether the crash still reproduces) until you have the smallest input that still crashes, which is far easier for a developer to reason about than the original large file. Concretely: the original 40 KB file crashes; trimmed to the first 20 KB it still crashes; trimmed further to 10 KB it stops crashing, narrowing the trigger to somewhere in bytes 10 KB-20 KB; re-verifying the crash reproduces at each trim step, not just at the end, is what keeps this reduction trustworthy.
4. Verify the candidate fix against both the minimal repro and the original reported file (if available) before handing back to QA, since a fix that only addresses the minimized case might miss a related edge case present in the original file but stripped out during minimization.

A related, nondeterministic-production variant

For a production job that fails nondeterministically under load with an obscure exception: capture the exact environment and inputs at failure time, collect logs/traces immediately (before they age out), and apply the same reduce-and-bisect idea to the input space specifically, isolating external dependencies (mock/stub them) so the investigation narrows to your own code's behavior rather than variance introduced by a dependency.

Trade-offs and pitfalls

Minimization can occasionally strip away the actual triggering condition if done too aggressively or without retesting at each step; always confirm the crash still reproduces after each reduction, not just at the end, or you risk shipping a "fix" for a simplified case that no longer matches the real bug.