Systematic Debugging and Root Cause Analysis Questions

Locating a Java memory leak means finding what's still referenced that shouldn't be, and the standard toolchain (jmap/jcmd, a heap analyzer) exists to answer exactly that question.

Step-by-step

1. Confirm the trend: RSS (Resident Set Size, the process's actual physical memory usage as reported by the OS)/heap usage climbing steadily over the observation window (here, 48 hours to OOM) rather than a one-off spike.
2. Capture heap dumps at multiple points in time, not just one: a single dump only shows current state; two or three dumps spaced hours apart let you diff which object types grew between them, which is far more diagnostic than one static snapshot.
   bash

   jcmd <pid> GC.heap_dump /tmp/heap-$(date +%s).hprof
   # or: jmap -dump:live,format=b,file=heap.hprof <pid>
   

3. Analyze with a heap analyzer (Eclipse MAT, jvisualvm): use the "dominator tree" / leak-suspect report to find which object graph retains the most memory, then trace its retained path to GC roots (what is holding onto it: a static field, a listener registered and never removed, an unbounded cache/collection). For example, the leak-suspect report might show a SessionCache object retaining 2.1 million Session instances, about 400 MB, all reachable through one static Map field that's never cleared on logout, which is the retained path pointing directly at the fix.
4. Confirm the fix: after the code change, repeat the same two-snapshot diff over a soak test and confirm the previously-growing type has stopped growing.

Common root causes worth checking first

Static/long-lived caches without eviction, event listeners registered but never unregistered, ThreadLocal values not cleared in a pooled-thread environment, and unbounded queues/collections accumulating faster than they drain.

Trade-offs and pitfalls

Taking a heap dump on a live production instance pauses that JVM briefly (duration scales with heap size), so schedule it for a low-traffic moment or a canary instance rather than the busiest host; and a single heap dump without a second comparison point tempts a wrong conclusion, since the biggest object type in one snapshot isn't necessarily the one that's growing.

This function is supposed to sum the first k elements but starts iterating from index 1 instead of 0, silently skipping the first element and off by one on both ends.

def sum_first_k(nums, k):
    total = 0
    for i in range(1, k):   # BUG: should start at 0
        total += nums[i]
    return total

Verified execution against nums = [10, 20, 30, 40, 50]

• k=0: returns 0 (happens to be correct by coincidence, since range(1,0) is empty).
• k=1: returns 0, but the correct answer is nums[0] = 10. Wrong.
• k=5: returns 140 (20+30+40+50, only 4 terms), but the correct answer is the sum of all 5 = 150. Wrong.
• k=6: raises IndexError, since range(1,6) reaches index 5, one past the end of a 5-element list.

Corrected implementation

def sum_first_k(nums, k):
    if k <= 0:
        return 0
    if k > len(nums):
        raise ValueError(f"k={k} exceeds list length {len(nums)}")
    return sum(nums[:k])   # or: sum(nums[i] for i in range(k))

Complexity

O(k) time (or O(n) worst case if k approaches n), O(1) extra space beyond the slice.

Edge cases

k=0 (return 0, no elements summed), k=1 (return exactly nums[0]), k equal to len(nums) (sum the whole list), k greater than len(nums) (raise explicitly rather than silently truncating or index-erroring), and an empty nums list with k=0 (return 0).

Unit tests to verify

assert sum_first_k([10,20,30,40,50], 0) == 0, assert sum_first_k([10,20,30,40,50], 1) == 10, assert sum_first_k([10,20,30,40,50], 5) == 150, and a test confirming k > len(nums) raises rather than silently misbehaving.

Auth and session races reliably show up "only under load or in parallel test runs" because they depend on two operations touching shared identity state (a session record, a signing key, a token cache) landing close enough in time to interleave, something a single-threaded manual test never triggers.

Confirming the token-refresh race

Reproduce under concurrent load (many simultaneous refresh calls for the same user/session), instrument the refresh path with fine-grained timestamps around the read-check-write of the session/token state, and look for two refresh calls whose logged operations interleave (both read "no active session," both then create one, producing a duplicate). Concretely, this might look like: request A reads "no active session" at t=0ms, request B reads the same "no active session" state at t=2ms (before A's write has landed), so both requests independently create a session row for the same user, and the second write is the duplicate that later produces an intermittent 401 when the first session is looked up inconsistently. Verify the fix under the same parallel load, not just a single retest, since the fix needs to survive the interleaving that caused the bug, not merely pass a serial test.

Root causes in this class of bug

A signing-key rotation racing with in-flight validation is the same shape: a request validated against the old key arrives just as the key store swaps to the new key, and if the validation path doesn't accept a short overlap window for the previous key, legitimate requests fail for the few minutes until caches fully converge. The fix is usually a deliberate overlap window (accept both old and new key for a bounded grace period) rather than an instantaneous cutover, plus idempotency (making an operation idempotent means running it twice has the same effect as running it once, so a duplicate retry can't create a second session) on session creation (a unique constraint or a check-and-set on a stable session key) so a race that does occur produces one session, not two.

Trade-offs and pitfalls

Making the refresh path fully serialized (a lock per user) removes the race but adds latency and a new single point of contention; a narrower fix (idempotent creation, an accept-both-keys window) usually gets the correctness without that cost, and is the preferred approach unless contention is proven to be a non-issue.

This retry loop's root risk is that it retries on any failure, including the ambiguous case where the payment gateway actually succeeded but the response was lost (a timeout, a dropped connection), which can charge the card multiple times since there is no idempotency key tying repeated attempts to the same logical charge.

function charge(card, amount):
  for i in 1..5:
    res = call_payment_gateway(card, amount)
    if res.success: return true
  return false

Specific failure modes

1. No idempotency key: each retry is indistinguishable to the gateway from a brand-new charge request; if attempt 2 succeeds after attempt 1 actually also succeeded (but its response was lost to a network timeout), the customer is charged twice.
2. Retries on ambiguous outcomes: the code retries on any non-success response, including a timeout where the charge may have gone through server-side; a truly failed charge (declined card) and an ambiguous timeout should not be treated identically.
3. No backoff: five immediate retries can also amplify load on the gateway during a real outage, worsening a cascading failure rather than helping.

Concrete fixes

function charge(card, amount, idempotency_key):
  for attempt in 1..5:
    res = call_payment_gateway(card, amount, idempotency_key=idempotency_key)
    if res.success:
      return true
    if res.error_type == "declined":       # a real, final failure
      return false
    sleep(backoff_with_jitter(attempt))     # only retry on genuinely transient errors
  return false                              # exhausted retries: surface for manual reconciliation

The idempotency key (generated once per logical charge attempt, sent on every retry) is the core fix: a well-designed payment gateway will recognize a repeated key and return the original result instead of creating a second charge, making retries safe regardless of which attempt's response was actually lost.

Complexity and edge cases

No algorithmic complexity change; the design fix is about correctness under ambiguity, not performance. Edge cases: the gateway itself doesn't support idempotency keys (then a client-side dedup ledger, checking "was this logical charge already attempted" before issuing a new request, is the fallback), a declined card (must not retry), and all 5 attempts exhausted with no clear success/failure (must surface as "unknown, needs reconciliation," never silently treated as a clean failure that's safe to retry again later from scratch).

A CI-only, non-locally-reproducing GPU OOM under a hard 30-minute clock needs a plan built around commands that surface the actual memory numbers fast, not guesswork.

First 5 minutes: diff the environment with real commands

# On the CI runner (or in its logs), find the GPU type and total memory
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv
# Diff CI's job config against local: batch size, precision, GPU count
diff <(cat ci/config.yaml) <(cat local/config.yaml)

Check whether CI shares one GPU across multiple concurrent jobs (a common cost-saving CI setup): if two jobs are scheduled onto the same device, each effectively sees half the memory nvidia-smi reports as free.

Next 10 minutes: reproduce CI's exact conditions, not approximate ones

import torch
# Cap memory to match CI's actual per-job budget instead of using the full local GPU
torch.cuda.set_per_process_memory_fraction(0.5, device=0)
torch.cuda.empty_cache()

try:
    run_training_step(batch_size=CI_BATCH_SIZE)  # use CI's config value, not local's
except torch.cuda.OutOfMemoryError:
    print(torch.cuda.memory_summary(device=0, abbreviated=True))

torch.cuda.memory_summary() shows allocated vs. reserved memory and the largest free block, which distinguishes a hard ceiling breach (allocated is genuinely near the total) from fragmentation (reserved is high but no single free block is large enough for the next allocation).

Next 10 minutes: bisect whether it is a ceiling breach or fragmentation

If memory_summary() shows many small free blocks but no contiguous block large enough for the failing allocation, it is fragmentation building up across test cases in the same process; if allocated memory alone already approaches the device total, it is a real ceiling breach (too large a batch/model for the available memory, or tensors never released between test cases).

Last 5 minutes: cheapest safe mitigation to unblock CI

# ci/config.yaml - minimal change to restore a green CI while investigation continues
gpu_tests:
  batch_size: 16       # was 64; matches what CI's smaller/shared GPU can hold
  parallel_jobs: 1      # was 2; stop co-scheduling two GPU-heavy jobs on one device

# add between GPU-heavy test cases as a stopgap
import gc, torch
gc.collect()
torch.cuda.empty_cache()

File the deeper investigation (genuine leak, fragmentation, or under-provisioned CI resources) as a follow-up; the batch-size/serialization change is a mitigation, not the root-cause fix.

Complexity and edge cases

No algorithmic complexity change; the mitigation trades CI throughput (smaller batches, serialized GPU jobs) for a passing pipeline. Edge case: reducing batch size can quietly change what a test is actually validating (a model behavior test tuned for batch_size=64 may behave differently at 16); note this explicitly in the follow-up ticket so the mitigation does not silently become the permanent, untracked test configuration.