Systematic Debugging and Root Cause Analysis Questions

Reproducing a production race deterministically means removing the randomness from timing, not just running the code more times and hoping.

Approach

1. Force the interleaving instead of hoping for it: use a deterministic delay or explicit synchronization points (barriers, controlled thread scheduling) to make two operations race on purpose, the same way you'd construct an adversarial test case for a known suspect interaction.
2. Detection tooling: a thread sanitizer (TSan) flags a real data race even if it didn't cause an observable bad outcome on that specific run; stress tests (high concurrency, tight loops) increase the odds of hitting the natural window without forcing it.
3. Instrumentation: fine-grained, per-operation logs with high-resolution timestamps and a vector clock or logical sequence number per request let you reconstruct, after the fact, the actual interleaving order that occurred in a failing production case, rather than guessing.
4. Durable fixes, in order of typical preference: a lock around the critical section (simplest, correct); optimistic concurrency control (a version/CAS (compare-and-swap, meaning update a value only if it still matches the version you last read, otherwise reject and retry) check before commit, better under high read/low write contention); idempotency at the API/write layer so a duplicate or reordered operation doesn't corrupt state even if the race still occurs.

Worked link to a concrete case

A distributed cache with multiple writer threads showing intermittent state inconsistency is the same shape: two writers race on a read-modify-write of a shared entry. A lock or compare-and-swap on that entry, or moving to an operation that's inherently commutative (e.g. an increment instead of a read-then-set), removes the race rather than papering over its symptom with retries.

A concrete trace of the race and the fix

Suppose two concurrent requests both update the same account balance: request A reads version=3, balance=100 at t=0ms; request B reads the same version=3, balance=100 at t=2ms, before A's write has landed. Both compute a new balance from that stale read and try to commit. With a version/CAS check, whichever commit arrives second is rejected because the stored version is no longer 3 (A's write already advanced it to 4), forcing that request to re-read the fresh value and retry instead of silently overwriting A's update.

Trade-offs and pitfalls

Adding a lock reduces throughput; optimistic concurrency control avoids that cost but adds retry logic and complexity, and is the wrong choice if conflicts are frequent (high retry rate can be worse than a lock). The choice should be driven by measured contention, not assumed in advance.

Silent data corruption is the dangerous case precisely because nothing crashes or errors; the system keeps running and reporting success while the data itself is wrong, so detection has to be built in deliberately rather than relying on something failing loudly.

Detection techniques

• Checksums/CRCs: compute and store a checksum alongside data at write time, and verify it on read; a mismatch proves corruption occurred somewhere between write and read without needing to know the mechanism.
• Write-then-read verification: immediately read back what was just written and compare, catching corruption introduced by the write path itself (a common check for critical writes).
• Periodic scrubbing: proactively read and verify all stored data on a schedule (not waiting for an application to happen to read a corrupted record), surfacing corruption before something downstream actually consumes the bad data.
• Replication consistency checks: compare checksums/digests across replicas that should be identical; a divergence flags corruption on at least one replica, even before any single reader notices.
• Alerting: any of the above should page/alert immediately on a detected mismatch, since silent corruption left undetected for a long window is much harder to recover from cleanly.

What a detected mismatch actually looks like

For example, object orders/2026-07-14/part-0091 is written with checksum a1b2c3; on the next scheduled scrub, recomputing the checksum over the stored bytes returns d4e5f6 instead, a mismatch. That single mismatch is what flags this specific object, not the whole dataset, for recovery; checking replica 2 for the same object shows its checksum still matches the original a1b2c3, so remediation restores from replica 2 rather than attempting to patch the corrupted bytes in place.

Recovery and remediation

Automated remediation should default to safe rollback/recovery from a known-good replica or backup rather than attempting to "fix" the corrupted bytes in place, and should verify the recovered data against its checksum before considering the incident closed. When corruption is only discovered later (e.g., in nightly backups already taken), the recovery plan needs to first identify precisely which backups are still clean (checksum each one against its recorded value, working backward from the most recent), then restore to the last confirmed-clean point, and only then investigate how corruption entered the pipeline in the first place so it doesn't happen again.

Verified related cases

The same detection discipline (checksums, cross-region replication checks, versioned objects, reprocessing from a known-good source) applies directly to bit-flip corruption discovered in cloud object storage, and a distributed storage cluster (e.g., Ceph) returning occasionally corrupted objects is diagnosed the same way: inspect cluster health/scrub state, run an explicit scrub-and-verify pass, and repair from a known-good replica rather than attempting an in-place patch of a corrupted object.

Trade-offs and pitfalls

Scrubbing and cross-replica checks cost real I/O and compute; the practical trade-off is running them on a schedule frequent enough to bound the "how much could have silently corrupted before we noticed" window to an acceptable size, not running them continuously at full intensity, which is rarely necessary and can itself compete with production traffic for I/O bandwidth.

A correlation ID is a single identifier generated at the edge of a request and propagated through every hop, so logs, metrics, and traces from otherwise-independent services can be joined into one end-to-end picture.

Design

1. Generate once, at the edge (the first service to see the request), and reject or ignore any incoming ID from an untrusted external source unless you specifically want to correlate with an external partner's identifier.
2. Propagate via headers (e.g. X-Request-Id or the W3C traceparent format) on every downstream call, and via message baggage for async work (queue messages, background jobs) so the ID survives a hop that isn't a synchronous HTTP call.
3. Attach it to every log line and trace span emitted while handling that request, not just at the top-level handler.
4. For third-party/legacy components you cannot instrument, enrich logs at your own ingress/egress boundary around that component (the request going in, the response coming out) so you can at least bound how much time was spent inside the black box, even without visibility inside it.
5. Query time: given an ID, a single query across the log/trace store reconstructs the full path, ordered by timestamp, across every service that touched the request.

Trade-offs and pitfalls

High-cardinality IDs are cheap to log but expensive to index if you try to make every field high-cardinality; keep the correlation ID itself indexed and unique, but avoid also indexing every other high-cardinality field (like raw user input) at the same tier, or storage cost balloons. Sampling (only tracing 1% of requests in detail) is fine for aggregate analysis but means a specific customer's specific failing request may not have full trace data; the mitigation is to force full capture whenever an error is detected on that request, regardless of the sampling decision made at the start.

Direct answer

Pull Flow Log records for the source and destination ENIs (Elastic Network Interfaces, the virtual network cards attached to each instance) and read the action field. A REJECT for that exact tuple means a security group or NACL (Network Access Control List, a stateless, subnet-level firewall, separate from the per-instance security group) is blocking it, while ACCEPT records with the app still timing out mean the problem is above the network layer entirely.

Structured elaboration

• Query Flow Logs (Athena or CloudWatch Insights) filtered to the incident window and the ENIs/ports involved.
• On REJECT, check both the security group and the NACL, since NACLs are stateless and can block the return leg even when the security group allows the request.
• On ACCEPT with no timely response, look at DNS resolution, the TLS handshake, or the destination process itself, none of which Flow Logs show.
• No records at all suggests routing, a missing route table entry or peering/Transit Gateway (a managed hub that routes traffic between multiple VPCs and on-premises networks over VPN or dedicated connections) misconfiguration, rather than a security rule.

Worked example

action=REJECT for 10.0.1.15:443 -> 10.0.2.20:5432 conclusively points at SG/NACL rules; action=ACCEPT for the same tuple with a client-side timeout redirects the investigation entirely toward the destination service instead.

Trade-offs and pitfalls

Flow Logs sample and aggregate rather than log every packet, so very brief issues can be underrepresented. They carry no payload detail, so ACCEPT doesn't mean the request was handled correctly.

What the interviewer probes next

Why NACLs being stateless matters for return traffic, and how you'd alert on a REJECT spike for a given path.

A hotfix is a fast, narrowly-scoped patch to restore correct behavior immediately; a long-term fix addresses the underlying design or process issue, typically taking longer and carrying lower risk of introducing a new problem.

When to choose which

Choose a hotfix when user impact is active and ongoing, the fix is small and well-understood (low blast radius), and a proper fix would take meaningfully longer than the acceptable time to restore service. Choose to invest in the long-term fix directly when there's no active user impact yet (a bug caught before it ships, or a near-miss), or when the "quick" fix would itself be risky/complex enough that it's not actually faster or safer than doing it right.

Risks of hotfixes

They frequently trade correctness for speed in a way that creates hidden technical debt (a special-cased branch nobody remembers the reasoning for), can mask the actual root cause (the symptom goes away, but the underlying condition that caused it is still there and can resurface differently), and sometimes introduce a new, narrower bug because they were reviewed and tested less thoroughly than a normal change under time pressure.

Communication and documentation after a hotfix

Document, at minimum: what the hotfix specifically does and does not address, why it was chosen over a full fix, and a tracked follow-up item for the durable fix with an owner and rough timeline, communicated to the team (not just left in a commit message), since an undocumented hotfix is the most common way "temporary" becomes permanent by default.

Trade-offs and pitfalls

The single biggest failure mode across roles (SRE, general engineering, systems engineering) is the same: a hotfix applied under pressure with no tracked follow-up quietly becomes the permanent state of the system, carrying its narrower risk profile forward indefinitely instead of the brief window it was meant for.