Systematic Debugging and Root Cause Analysis Questions

Direct answer

Pull Flow Log records for the source and destination ENIs (Elastic Network Interfaces, the virtual network cards attached to each instance) and read the action field. A REJECT for that exact tuple means a security group or NACL (Network Access Control List, a stateless, subnet-level firewall, separate from the per-instance security group) is blocking it, while ACCEPT records with the app still timing out mean the problem is above the network layer entirely.

Structured elaboration

• Query Flow Logs (Athena or CloudWatch Insights) filtered to the incident window and the ENIs/ports involved.
• On REJECT, check both the security group and the NACL, since NACLs are stateless and can block the return leg even when the security group allows the request.
• On ACCEPT with no timely response, look at DNS resolution, the TLS handshake, or the destination process itself, none of which Flow Logs show.
• No records at all suggests routing, a missing route table entry or peering/Transit Gateway (a managed hub that routes traffic between multiple VPCs and on-premises networks over VPN or dedicated connections) misconfiguration, rather than a security rule.

Worked example

action=REJECT for 10.0.1.15:443 -> 10.0.2.20:5432 conclusively points at SG/NACL rules; action=ACCEPT for the same tuple with a client-side timeout redirects the investigation entirely toward the destination service instead.

Trade-offs and pitfalls

Flow Logs sample and aggregate rather than log every packet, so very brief issues can be underrepresented. They carry no payload detail, so ACCEPT doesn't mean the request was handled correctly.

What the interviewer probes next

Why NACLs being stateless matters for return traffic, and how you'd alert on a REJECT spike for a given path.

Tripled logging cost usually means volume grew faster than the value extracted from it; the fix is to cut volume selectively, not uniformly, so the signal that actually gets used survives.

A practical plan

1. Set log-level policy by environment and default: debug logging is fine to leave on in staging but must default to info/warn in production, with a way to raise it temporarily and narrowly (one instance, one request ID, a short TTL) rather than fleet-wide and indefinitely.
2. Sample high-volume, low-value lines (e.g. successful health checks, routine polling) instead of dropping them entirely, so you can still detect a rate change without paying to store every instance.
3. Aggregate/rollup where the individual line rarely matters: turn "1000 identical retry log lines" into one line with a count, and rely on metrics (which are cheap) for anything that's fundamentally a counter, saving log storage for things that need the specific detail (a stack trace, a specific failing payload).
4. Redact and shorten before storage, not after: strip large payloads or PII at the point of logging rather than logging everything and cleaning it up downstream, since the ingestion cost is already paid by the time cleanup happens.
5. Set retention tiers: keep full-fidelity logs for a short, cheap window (days) and only aggregated/rolled-up summaries for the longer compliance window, instead of one flat retention policy for everything.

Monitoring the change itself

Track "log volume per request" and "percentage of debug-triage sessions where the needed line was missing" as the two competing metrics, so cost cuts can be validated against not silently destroying the ability to debug, rather than declared successful purely because the bill went down.

Trade-offs and pitfalls

The main risk is over-trimming: cutting a log line that turns out to be the one thing needed during the next incident. The mitigation is a staged rollout of each cut (reduce, watch for a sprint, then commit) plus keeping an emergency dial to re-enable full verbosity narrowly and fast when an active incident needs it.

for f in $(ls testdir) performs unquoted word-splitting on the output of ls: the shell splits on whitespace, so any filename containing a space is broken into multiple separate "words," and filenames containing shell glob characters or newlines can misbehave further.

for f in $(ls /var/data/input); do
  process "$f"
done

Verified demonstration

Against a directory containing "file one.txt", "file two.txt", and "normal.txt", the buggy version processes five malformed tokens: [file], [one.txt], [file], [two.txt], [normal.txt], exactly as predicted, silently corrupting both multi-word filenames into two garbage entries each.

Corrected, minimal implementation

for f in /var/data/input/*; do
  process "$f"
done

Running the fixed version against the identical directory correctly processes all three files with their full, intact names: [testdir/file one.txt], [testdir/file two.txt], [testdir/normal.txt]. For filenames that could contain newlines specifically (which even the glob form can't fully guard against in an early loop over find's plain output), the fully robust pattern is: find /var/data/input -maxdepth 1 -type f -print0 | while IFS= read -r -d '' f; do process "$f"; done. (IFS is bash's word-separator variable; setting it empty for the duration of read stops it from trimming leading or trailing whitespace out of each filename it reads.)

Complexity

No algorithmic complexity change; both are O(n) in the number of files.

Edge cases and tests

Filenames with spaces (verified above), a filename starting with a dash (which some commands could misinterpret as a flag: quoting alone doesn't protect against this, a -- separator or explicit path prefix does), an empty directory (loop body should simply not execute), and filenames containing newlines (only the find -print0/read -d '' form handles this correctly; the plain glob form is safe for spaces but not for embedded newlines).

Isolating the root cause of an environment-specific security incident starts from having a trustworthy definition of "normal" to compare against, since without one, every observed difference looks equally suspicious.

What a known-good baseline looks like

A snapshot of expected configuration, dependency versions, network policy, and behavioral metrics (error rates, latency, auth success rates) for an environment when it is known to be working correctly, captured and version-controlled the same way infrastructure-as-code is, not reconstructed from memory after the fact.

Using it to isolate root cause faster

Diff the current, incident-affected environment against the baseline systematically: configuration drift, dependency/library version differences, network/firewall rule differences, and any recent unlogged manual change. A difference found this way is a concrete, falsifiable hypothesis ("this environment has a different TLS cipher suite enabled") rather than a vague "something's different," and each diff item can be tested independently (temporarily aligning that one setting to baseline and observing whether the symptom clears) rather than changing many things at once.

A concrete worked case

A secret-rotation job succeeding in one environment and failing in another with identical code and container image is a textbook baseline-diff case: since the code is provably identical, the cause must be in the environment, and diffing permissions (IAM/service-account differences), cloud metadata service behavior, network egress rules, and any config drift between the two environments will surface the actual difference far faster than re-reading the job's code for a bug that isn't there.

Trade-offs and pitfalls

A baseline that isn't kept current (infrastructure changes without updating the baseline snapshot) becomes actively misleading, flagging legitimate intentional changes as suspicious drift; treating the baseline as a living, versioned artifact rather than a one-time snapshot is what keeps this technique useful over time.

Silent data corruption is the dangerous case precisely because nothing crashes or errors; the system keeps running and reporting success while the data itself is wrong, so detection has to be built in deliberately rather than relying on something failing loudly.

Detection techniques

• Checksums/CRCs: compute and store a checksum alongside data at write time, and verify it on read; a mismatch proves corruption occurred somewhere between write and read without needing to know the mechanism.
• Write-then-read verification: immediately read back what was just written and compare, catching corruption introduced by the write path itself (a common check for critical writes).
• Periodic scrubbing: proactively read and verify all stored data on a schedule (not waiting for an application to happen to read a corrupted record), surfacing corruption before something downstream actually consumes the bad data.
• Replication consistency checks: compare checksums/digests across replicas that should be identical; a divergence flags corruption on at least one replica, even before any single reader notices.
• Alerting: any of the above should page/alert immediately on a detected mismatch, since silent corruption left undetected for a long window is much harder to recover from cleanly.

What a detected mismatch actually looks like

For example, object orders/2026-07-14/part-0091 is written with checksum a1b2c3; on the next scheduled scrub, recomputing the checksum over the stored bytes returns d4e5f6 instead, a mismatch. That single mismatch is what flags this specific object, not the whole dataset, for recovery; checking replica 2 for the same object shows its checksum still matches the original a1b2c3, so remediation restores from replica 2 rather than attempting to patch the corrupted bytes in place.

Recovery and remediation

Automated remediation should default to safe rollback/recovery from a known-good replica or backup rather than attempting to "fix" the corrupted bytes in place, and should verify the recovered data against its checksum before considering the incident closed. When corruption is only discovered later (e.g., in nightly backups already taken), the recovery plan needs to first identify precisely which backups are still clean (checksum each one against its recorded value, working backward from the most recent), then restore to the last confirmed-clean point, and only then investigate how corruption entered the pipeline in the first place so it doesn't happen again.

Verified related cases

The same detection discipline (checksums, cross-region replication checks, versioned objects, reprocessing from a known-good source) applies directly to bit-flip corruption discovered in cloud object storage, and a distributed storage cluster (e.g., Ceph) returning occasionally corrupted objects is diagnosed the same way: inspect cluster health/scrub state, run an explicit scrub-and-verify pass, and repair from a known-good replica rather than attempting an in-place patch of a corrupted object.

Trade-offs and pitfalls

Scrubbing and cross-replica checks cost real I/O and compute; the practical trade-off is running them on a schedule frequent enough to bound the "how much could have silently corrupted before we noticed" window to an acceptable size, not running them continuously at full intensity, which is rarely necessary and can itself compete with production traffic for I/O bandwidth.