Systematic Debugging and Root Cause Analysis Questions

Given process sometimes returns null, the debugger session should target both places that can produce that outcome: the explicit return null; branch, and the return finalize(intermediate); branch, since finalize returning null on some input would also make process return null without ever reaching the explicit return null; line.

public String process(Request r) {
    String intermediate = transform(r.getPayload());
    if (intermediate.length() > 0) {
        return finalize(intermediate);
    }
    return null;
}

Breakpoints and checks

1. Set a breakpoint at the start of process, and a conditional breakpoint at the if (or just after transform returns) so it only stops on calls where the eventual return value is null, not every call.
2. Inspect r.getPayload() at the top: is the raw payload itself empty or unexpected?
3. Step into transform and inspect its return value directly: is intermediate genuinely empty, or is transform throwing/silently returning empty on some input path? Also check whether transform can return null: if it ever does, intermediate.length() would throw a NullPointerException rather than return null from process, which is a different symptom worth distinguishing from a plain empty-string result.
4. Watch intermediate.length(): confirm whether the branch is taking the return null path because the input was legitimately empty, or because transform has a bug that produces an empty result for input that should not be empty.
5. Set a second breakpoint at the return finalize(intermediate); line and step into finalize, inspecting its return value before it propagates back. This path is easy to overlook because it isn't the visually obvious return null; line, but finalize returning null for some non-empty intermediate reproduces the exact same symptom from the outside.

Runtime conditions to inspect

There are three live hypotheses, not two: "upstream input was actually empty" (correct-by-design null, via the explicit branch), "transform mishandled a non-empty input and produced an empty result" (a bug in transform, via the explicit branch), and "finalize itself returned null for a non-empty intermediate" (a bug in finalize, via the other branch entirely). The debugger session's job is to look at r.getPayload(), intermediate, and finalize's return value side by side on a failing call to tell which of the three is true, rather than assuming the explicit return null; line is the only source.

Trade-offs and pitfalls

A conditional breakpoint that only triggers on a null-producing call avoids stepping through hundreds of successful calls to find the rare failing one; without it, this kind of intermittent-looking bug becomes tedious to debug interactively. Fixating only on the explicit return null; statement because it's the one line that literally says "null" is the trap here: it's easy to miss that finalize's return value flows straight back out unchecked.

A short list of error classes, checked reflexively, catches a large share of bugs faster than reasoning from scratch each time, especially under time pressure.

The checklist and where each shows up

• Off-by-one: a paginated API returning one too many/few items, or a loop boundary; assertion/test: check the exact boundary index (first and last page/element) explicitly, not just a "normal" middle case.
• Null/bounds: an unchecked null or an index beyond array/collection length; assertion: a boundary test with an empty collection and a single-element collection.
• Integer overflow: a size or count computation exceeding the integer type's range (a buffer-size calculation, a large-array binary search midpoint); assertion: a test with values near the type's max, not just small representative numbers.
• Index errors: signed/unsigned confusion, or mixing 0-indexed and 1-indexed conventions across two components; assertion: explicit tests at index 0 and index length-1.
• Race conditions: state mutated by more than one thread/request without synchronization; assertion: a concurrency stress test, not just a serial unit test.

Using assertions and invariant checks under time pressure

Adding a small, well-placed assertion (e.g., asserting an index is within expected bounds, or a computed size is non-negative) right before the suspect operation turns a silent wrong answer into an immediate, precise failure at the exact point of violation, which is much faster to act on than a downstream symptom several calls later. In production, convert a temporary debugging assertion into either a proper validated error path (fail gracefully with a clear error) or a monitored invariant check (log and alert, don't crash), rather than leaving raw assertions that could themselves crash production on an edge case.

Trade-offs and pitfalls

This checklist is a starting triage list, not a replacement for understanding the actual code; its value is in ruling several common causes in or out in the first few minutes, which is exactly when time pressure makes a systematic pass most valuable.

When the two live-incident options are specifically a rollback of the latest deploy or toggling a feature flag that should revert the new behavior, the choice comes down to which one restores the previous behavior more completely and more verifiably, not simply which is faster to execute.

Comparing the two options

Feature flag toggle: fast (a config change, often seconds to propagate) and narrowly scoped if the flag genuinely gates all of the new behavior; the risk is that the flag may not cover every code path touched by the change (a partial flag, or a change that also altered a shared library or schema outside the flag's reach), in which case toggling it off looks like it worked but leaves some of the regression in place.
Rollback of the latest deploy: restores a fully known-good binary, so it does not depend on the flag actually covering all of the change; the cost is it also reverts any other, unrelated changes bundled in the same deploy, and a rollback typically takes longer to execute and verify than flipping a flag.

Assessing risk

The deciding question is confidence that the flag fully covers the regression: if the change was built and tested specifically to be flag-gated (and QA verified the flag-off path matches the prior release), toggle the flag first since it is faster and reversible with less collateral. If there is any doubt the flag is complete (schema changes, shared state, a partial rollout of the flag itself), prefer the rollback, since a flag toggle that only partially fixes the incident quietly wastes time while customers keep seeing errors.

Preserving data and logs

Before acting, capture the current error logs, traces, and the exact deploy/flag state (which version is live, which flag values are set, for which cohorts), since a rollback in particular can make the failing state harder to inspect afterward; snapshot dashboards and pull a sample of failing requests first if it costs only seconds.

Performing the action and verifying restoration

Toggle the flag (or execute the rollback) for a narrow slice first if that is safely possible (one region, one host group) to confirm the fix actually restores correct behavior before flipping it globally; then verify against the original symptom directly (the specific error rate, endpoint, or user cohort that was affected) rather than a generic drop in overall errors, since a coincidental dip can be mistaken for a fix.

Communication and avoiding further disruption

Announce which action was taken and its known side effects before taking it where possible (for example: "we are toggling flag X off, which also reverts behavior Y that shipped with it") so stakeholders are not surprised, and avoid stacking a second untested change on top while the first action's effect is still being verified, since two simultaneous changes make it impossible to attribute the outcome to either one.

Trade-offs and pitfalls

Defaulting to the flag toggle purely because it is faster, without first confirming it actually covers the full blast radius of the change, is the most common mistake in this scenario; when in doubt about coverage, the rollback's completeness is worth its slower execution time.

A nightly metrics pipeline drifting lower while the live production model stays stable is strong evidence the pipeline itself changed, not the model: the investigation should first rule out the pipeline before doubting the model.

RCA plan

1. Checksum the inputs: compare the checksum/hash of the dataset the nightly job consumed against the expected/previous dataset; a silent upstream data change (a new day's data with a schema drift, a partial load, a different sample) is the most common cause of "the eval script says worse, but production says fine."
2. Dataset versioning: pin and log the exact dataset version/snapshot used for each nightly run so a regression can be attributed to a specific version bump rather than an ambiguous "sometime between runs."
3. Golden artifacts: maintain a small, fixed, hand-verified reference dataset with known expected metric values; rerun the nightly evaluation code against that golden set specifically, and if it now produces different numbers than its own historical golden-set baseline, the bug is in the evaluation code, not the data; if the golden-set numbers are unchanged, the regression is genuinely in the day's input data. For example, the golden set's accuracy might hold steady at 0.842 across both last night's run and a rerun today, while the day's actual input checksum changed from abc123 to def456; that combination, golden set stable, real input checksum different, is the concrete evidence that the input data changed and the evaluation code did not.
4. Compare the nightly pipeline's own historical metric on a fixed replay dataset run-over-run, isolating whether the pipeline's code changed behavior independent of data.

Trade-offs and pitfalls

Skipping straight to "the model regressed" without checking the pipeline first risks an expensive, unnecessary retraining or rollback cycle for a bug that was actually in the measurement, not the thing being measured; the golden-artifact check is cheap and should run before any model-level conclusion is drawn.

A login/SSO flow that mostly succeeds but fails for a small percentage needs a first-10-minutes triage that narrows scope before touching any code.

First 5-10 minutes

Check logs for the specific failing requests (error type, exact step in the flow where they fail), compare metrics for the failing slice against the overall success rate (is the failure rate flat over time or did it start at a specific moment, which would point at a recent change), and check for recent changes (a deploy, a config change, an identity-provider-side change) around the same window before assuming the bug is new code versus an external dependency shift. Concretely, this might turn up something like: the failing requests all log error=token_exchange_timeout region=eu-west-1, account for about 0.4% of logins over the last 20 minutes, and started right at 09:14, four minutes after a 09:10 deploy, specific enough evidence to go straight to that deploy's diff rather than guessing.

The MFA-enrollment-only variant

When only a subset of users in specific regions can't complete MFA enrollment while login works fine for everyone: apply the same narrowing discipline specifically to the dimensions that define the affected subset (which regions, which identity provider, which user attribute), since a failure isolated to specific regions strongly suggests a regional configuration or identity-provider-latency difference rather than a universal code bug (which would affect all regions equally).

Trade-offs and pitfalls

The common mistake is changing code based on a guess before confirming scope; spending the first several minutes purely on scoping (which users, which region, since when) is what turns "some login attempts fail" from a vague, hard-to-act-on report into a specific, testable hypothesis, and skipping that step tends to produce a fix that doesn't actually address the real, narrower cause.