Systematic Debugging and Root Cause Analysis Questions

Inference results differing for the same user on the same model version usually trace to a source of nondeterminism the system didn't intend: sampling, non-deterministic GPU ops, model warm-up state, or input preprocessing that isn't as stable as assumed.

Investigation plan

1. Rule out sampling first: if the model uses stochastic decoding (temperature/top-k sampling) rather than greedy/deterministic decoding, differing outputs are expected, not a bug; confirm the decoding config actually used at inference time.
2. Check for non-deterministic ops: certain GPU kernels (some cuDNN (NVIDIA's GPU library that implements the low-level convolution math deep-learning frameworks call into) convolution algorithms, atomic-based reductions) are not bit-exact deterministic by default; verify whether determinism flags were set (e.g. forcing deterministic algorithms) and whether that setting differs across the serving fleet.
3. Model warm-up state: confirm whether the model or its runtime has any stateful warm-up (cached graph optimizations, JIT compilation choices) that could differ between a cold and warm instance serving the same request.
4. Input preprocessing: verify the exact preprocessing pipeline is byte-for-byte identical for the same logical input across requests (floating-point order-of-operation differences, locale-dependent formatting, or non-deterministic tokenization can all silently vary the input tensor).

Verified: seeding gaps produce exactly this class of bug

The same root cause class shows up, and can be directly reproduced, in the RNG layer around a model rather than the model itself. Two concrete, executed examples:

1) Cross-library seeding gap (seeding random but forgetting numpy, or vice versa, so only part of the pipeline is actually reproducible):

import random
import numpy as np

def naive_seed_everything(seed):
    random.seed(seed)          # BUG: forgets numpy (or any other RNG source)

def full_seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)       # FIX: seed every RNG source the pipeline touches

# naive: random.random() is reproducible run-to-run, np.random.rand() is NOT,
# because something else in the process advances numpy's global RNG state
# between runs. Executed result:
#   naive : random() reproducible=True   numpy reproducible=False
#   fixed : random() reproducible=True   numpy reproducible=True

2) Worker-seeding bug in a multi-worker data loader (seeding every worker with the same fixed value at init makes every worker's "shuffle" produce the identical, not independently random, sequence):

import random

def buggy_worker_shuffle(worker_id, base_seed, dataset):
    rng = random.Random(base_seed)                       # BUG: ignores worker_id
    order = dataset[:]
    rng.shuffle(order)
    return order

def fixed_worker_shuffle(worker_id, base_seed, dataset, epoch=0):
    rng = random.Random(base_seed + worker_id * 1000 + epoch)  # FIX: per-worker, per-epoch
    order = dataset[:]
    rng.shuffle(order)
    return order

# Executed with 4 workers, dataset of 20 items, base_seed=7:
#   buggy_worker_shuffle -> all 4 workers produce the IDENTICAL order
#   fixed_worker_shuffle -> all 4 workers produce DISTINCT orders, and the
#                           same base_seed across two separate runs reproduces
#                           the identical set of 4 orders (still deterministic,
#                           just no longer duplicated across workers)

Both snippets were executed directly (Python 3, stdlib random + numpy); the "buggy" variants reproduce exactly the failure mode described (unintended duplication / partial irreproducibility), and the "fixed" variants confirm the corrected behavior (independent-but-reproducible per-worker sequences; fully reproducible cross-library output).

Trade-offs and pitfalls

Forcing full determinism (deterministic GPU algorithms, single-threaded data loading) is a useful diagnostic mode but often costs real throughput; it's typically enabled to investigate, then a narrower fix (proper independent-but-reproducible seeding, confirming decoding config) is shipped rather than leaving full determinism mode on in production permanently.

Presenting an incomplete solution honestly and calmly is itself a skill being evaluated: the goal is to demonstrate structured judgment under a real constraint, not to hide the gap.

What to present

Show the working parts clearly and explicitly (what does pass, and why you're confident it's correct), be transparent and specific about which scenarios still fail (not vague: name the exact failing case), give a prioritized next-step plan (what you would do first, second, and why, if given more time), and note which test cases you'd add next to guard against regressions once the fix lands.

The timeout/complexity-mismatch variant

When the issue is a solution that's correct but too slow (O(n^2), about to time out on the largest inputs) with 10 minutes left: state the trade-off plainly (this works and is correct, but won't pass the largest test cases in time), propose the concrete algorithmic direction you'd take with more time (what data structure or approach would get it to the needed complexity), and if there's time, implement the simpler, safe fallback rather than risking an unfinished, more complex optimization that might not compile or might introduce a new bug under the remaining time pressure.

Demonstrating calm, structured ownership

Frame the status update the same way you would during a real production incident: what's known, what's not yet known, what you're doing about the gap, rather than apologizing repeatedly or trying to hide the incomplete state, since the interviewer (or, in production, a stakeholder) is evaluating the judgment and communication as much as the code itself.

Trade-offs and pitfalls

Attempting a risky, more complex fix in the final minutes rather than falling back to a known-safe simpler version is a common mistake: running out of time mid-rewrite with a previously-working solution now broken is worse than stopping at a known, clearly-communicated partial state.

The same binary-search reasoning that isolates a bad commit also isolates a bad element inside a large input: split the input in half, test each half (or its complement) against the failure, and recurse into whichever piece still reproduces the failure.

Applying it manually and automating it

Given a dataset or a large DAG of transforms that fails somewhere inside it: run the process on the first half; if it still fails, recurse into that half, otherwise recurse into the second half. Concretely, on a 16-row dataset where row 11 is the one that crashes the model: round 1 tests rows 1-8 (passes) and 9-16 (fails), so the search keeps 9-16; round 2 tests 9-12 (fails, since it still contains row 11) and 13-16 (passes), keeping 9-12; round 3 tests 9-10 (passes) and 11-12 (fails), keeping 11-12; round 4 tests row 11 alone (fails) and row 12 alone (passes), isolating row 11 as the minimal failing input in exactly 4 rounds instead of checking all 16 rows one at a time. Automating this needs a test function that returns pass/fail for an arbitrary subset, and the loop halves the search space each round, so a million-row dataset localizes to the failing row(s) in about 20 rounds instead of one at a time.

Nondeterministic failures

If the failure only reproduces some fraction of the time, a single test on a candidate subset is not trustworthy: a subset that doesn't fail on this run might still contain the true trigger. The standard adjustment is to run each candidate subset multiple times and only accept "does not reproduce" after several clean runs, and to keep the last few known-failing subsets so a wrongly-discarded half can be revisited if the search dead-ends.

Trade-offs and pitfalls

Binary search over data assumes the failure is caused by the presence of specific elements, not by the size of the input (e.g. a memory limit hit only at full scale); if halving the input consistently stops reproducing regardless of which half is kept, that is itself a signal the bug is scale-dependent rather than content-dependent, and the investigation should pivot to resource limits instead of continuing to bisect.

This function is supposed to sum the first k elements but starts iterating from index 1 instead of 0, silently skipping the first element and off by one on both ends.

def sum_first_k(nums, k):
    total = 0
    for i in range(1, k):   # BUG: should start at 0
        total += nums[i]
    return total

Verified execution against nums = [10, 20, 30, 40, 50]

• k=0: returns 0 (happens to be correct by coincidence, since range(1,0) is empty).
• k=1: returns 0, but the correct answer is nums[0] = 10. Wrong.
• k=5: returns 140 (20+30+40+50, only 4 terms), but the correct answer is the sum of all 5 = 150. Wrong.
• k=6: raises IndexError, since range(1,6) reaches index 5, one past the end of a 5-element list.

Corrected implementation

def sum_first_k(nums, k):
    if k <= 0:
        return 0
    if k > len(nums):
        raise ValueError(f"k={k} exceeds list length {len(nums)}")
    return sum(nums[:k])   # or: sum(nums[i] for i in range(k))

Complexity

O(k) time (or O(n) worst case if k approaches n), O(1) extra space beyond the slice.

Edge cases

k=0 (return 0, no elements summed), k=1 (return exactly nums[0]), k equal to len(nums) (sum the whole list), k greater than len(nums) (raise explicitly rather than silently truncating or index-erroring), and an empty nums list with k=0 (return 0).

Unit tests to verify

assert sum_first_k([10,20,30,40,50], 0) == 0, assert sum_first_k([10,20,30,40,50], 1) == 10, assert sum_first_k([10,20,30,40,50], 5) == 150, and a test confirming k > len(nums) raises rather than silently misbehaving.

A nightly metrics pipeline drifting lower while the live production model stays stable is strong evidence the pipeline itself changed, not the model: the investigation should first rule out the pipeline before doubting the model.

RCA plan

1. Checksum the inputs: compare the checksum/hash of the dataset the nightly job consumed against the expected/previous dataset; a silent upstream data change (a new day's data with a schema drift, a partial load, a different sample) is the most common cause of "the eval script says worse, but production says fine."
2. Dataset versioning: pin and log the exact dataset version/snapshot used for each nightly run so a regression can be attributed to a specific version bump rather than an ambiguous "sometime between runs."
3. Golden artifacts: maintain a small, fixed, hand-verified reference dataset with known expected metric values; rerun the nightly evaluation code against that golden set specifically, and if it now produces different numbers than its own historical golden-set baseline, the bug is in the evaluation code, not the data; if the golden-set numbers are unchanged, the regression is genuinely in the day's input data. For example, the golden set's accuracy might hold steady at 0.842 across both last night's run and a rerun today, while the day's actual input checksum changed from abc123 to def456; that combination, golden set stable, real input checksum different, is the concrete evidence that the input data changed and the evaluation code did not.
4. Compare the nightly pipeline's own historical metric on a fixed replay dataset run-over-run, isolating whether the pipeline's code changed behavior independent of data.

Trade-offs and pitfalls

Skipping straight to "the model regressed" without checking the pipeline first risks an expensive, unnecessary retraining or rollback cycle for a bug that was actually in the measurement, not the thing being measured; the golden-artifact check is cheap and should run before any model-level conclusion is drawn.