Advanced Debugging and Root Cause Analysis Questions

Systematic approaches to complex debugging scenarios: intermittent failures, race conditions, environment-dependent issues, infrastructure problems. Using logs, metrics, and instrumentation effectively. Differentiating between automation issues, environment issues, and application defects. Experience with advanced debugging tools and techniques.

HardTechnical

29 practiced

A service is suffering from occasional long-tail latency spikes caused by garbage collector (GC) pauses. Explain how you would detect that GC is the root cause using metrics and traces, what JVM (or other runtime) flags you would examine or tune, and how you would instrument and monitor to prevent future regressions.

HardTechnical

25 practiced

A large test suite exhibits order-dependent failures: individual tests pass when run alone but fail when run in suite. Describe how you would diagnose and fix order-dependent tests: include techniques like test isolation, fixture design, resetting global state, randomizing test order, using deterministic seeds, and writing assertions to catch hidden dependencies.

Sample Answer

**Problem & goal**Explain and eliminate order-dependent (flaky) tests so suite is deterministic and each test can run in any order.

**Diagnosis steps**- Reproduce: run full suite repeatedly and log failures; use continuous integration to capture flakes.- Bisect ordering: run subsets or binary search (e.g., run halves) to find which test(s) introduce dependency.- Localize: run suspect tests in different positions to determine which one changes environment.

**Techniques to fix**

- Test isolation - Ensure each test sets up and tears down its own environment; avoid relying on prior tests. - Use framework fixtures that scope to test case (not module/global). Example: in pytest, use function-scoped fixtures.

- Fixture design & resetting global state - Reset singletons, global caches, environment variables, DB state, file system artifacts between tests. - Use transaction rollback or test-specific databases. For shared resources, create disposable resources (temp dirs, unique DB schemas).

- Randomize test order + deterministic seeds - Add randomized ordering during CI to surface order dependencies. When a failure appears, rerun with the recorded seed to reproduce. - Example (pseudocode): set RNG seed at test start and log it so failing seed can be reused.

- Assertions to catch hidden dependencies - Add explicit preconditions and postconditions: assert initial state (no lingering listeners, zero-rows in table) and cleanup assertions. - Use invariants checks in teardown that fail fast when global state is polluted.

**Practical example**- If Tests A then B fail: make B assert DB empty at start; change B to create its own DB schema and clean up; or change A to rollback transactions or clear global caches in teardown.

**Prevention & best practices**- Prefer pure unit tests with mocks for external systems.- Enforce CI job that randomizes order and fails on nondeterminism; require recorded seed for debug.- Add linters/coverage that detect shared mutable state usage.

Outcome: tests become deterministic, easier to debug, and CI stability improves.

HardTechnical

31 practiced

You are given the following Java concurrency snippet. Identify the potential deadlock and explain why it can happen. Propose at least three different fixes or mitigations (code-level and architectural). Include how you would detect such a deadlock in production.

java

public class Account {
  private final Object lockA = new Object();
  private final Object lockB = new Object();

  public void transferTo(Account other, int amount) {
    synchronized (lockA) {
      // prepare
      synchronized (other.lockB) {
        // perform transfer
      }
    }
  }

  public void receiveFrom(Account other, int amount) {
    synchronized (other.lockA) {
      synchronized (lockB) {
        // apply receive
      }
    }
  }
}

Sample Answer

**Brief identification of the deadlock**- Deadlock arises because transferTo locks this.lockA then other.lockB, while receiveFrom locks other.lockA then this.lockB. Two threads can each hold one lock and wait for the other — classic circular wait.

**Why it can happen**- Lock ordering is inconsistent: one path acquires A then B, the other acquires B then A (on different instances), so two threads can mutually block.

**Three fixes / mitigations**

1) Consistent global lock ordering (code-level)

java

public void transferTo(Account other, int amount) {
  Account first = (System.identityHashCode(this) < System.identityHashCode(other)) ? this : other;
  Account second = (first == this) ? other : this;
  synchronized (first.lockA) {
    synchronized (second.lockB) {
      // transfer
    }
  }
}

- Use stable tie-breaker (ids) to ensure all threads lock accounts in same order.

2) Use ReentrantLock with tryLock + timeout and retry (code-level)

java

private final ReentrantLock la = new ReentrantLock();
private final ReentrantLock lb = new ReentrantLock();

public boolean transferTo(Account other, int amount) throws InterruptedException {
  if (la.tryLock(100, MILLISECONDS)) {
    try {
      if (other.lb.tryLock(100, MILLISECONDS)) {
        try { /* transfer */ return true; } finally { other.lb.unlock(); }
      }
    } finally { la.unlock(); }
  }
  return false; // retry or report failure
}

- Avoids permanent blocking; implement exponential backoff and retries.

3) Architectural: serialize via a single coordinator / message queue- Push transfer requests to a dedicated thread or service that processes transfers sequentially (actor model). Eliminates multi-lock contention at the app level.

**How to detect deadlock in production**- Capture thread dumps (jstack/hs-stack) on high CPU or no-progress events and look for BLOCKED/WAITING cycles.- Use JVM tools: Java Flight Recorder, async-profiler, or VisualVM to detect locked monitors.- Monitor metrics: threads blocked count, lock contention, sudden drop in throughput. Alert when threads remain blocked > threshold.- Automated tests: add concurrency stress tests and use tools like jcstress, FindBugs/SpotBugs (deadlock detectors), and CI-level thread-deadlock detectors.

**QA perspective / testing**- Create unit+stress tests that spawn many concurrent transfers/receives across accounts, assert liveness, and fail on timeouts.- Add chaos tests that randomly pause threads to increase likelihood of deadlock and validate the recovery strategy.

EasyTechnical

25 practiced

A flaky automated test sometimes fails in your CI pipeline but passes locally most of the time. Outline the initial triage steps you would take to determine whether this is a flaky test (test issue), an environment issue (CI infra/config), or an application defect. Include specific commands/tools to collect evidence, how you would reproduce locally or in an isolated environment, and what CI artifacts you would capture (logs, screenshots, core dumps, container snapshots).

Sample Answer

**Situation & goal** I need to determine if a flaky CI failure is the test, CI environment/config, or an application bug by collecting reproducible evidence and isolating variables.

**Initial triage steps**1. Reproduce locally and isolate: - Run the single test repeatedly with verbosity and seed (example pytest):

bash

pytest tests/test_foo.py::test_bar -q -k test_bar --maxfail=1 -vv --count=50

- If flaky interacts with order, run entire test module or with random order plugin:

bash

pytest --random-order --random-order-bucket=module

2. Reproduce CI environment: - Pull same commit, use same Docker image or container used by CI:

bash

docker pull myorg/ci-image:sha
git checkout <commit>
docker run --rm -it -v $(pwd):/app myorg/ci-image:sha /bin/bash
pytest tests/test_foo.py::test_bar

- If CI runner is available, SSH/exec into failing job container (or re-run job with debug): capture exact env:

bash

env | sort > ci-env.txt
uname -a; lsb_release -a

3. Collect artifacts & evidence: - Test logs and stdout/stderr (JUnit XML, pytest -q --junitxml=report.xml). - Full CI job log, timestamps, node labels, Docker/container IDs. - Screenshots for UI tests (Selenium/WebDriver), saved on failure. - Core dumps / traces: enable core dump, capture with gcore or coredumpctl:

bash

ulimit -c unlimited
gcore <pid>
coredumpctl dump > core.dump

- Network traces if flaky is timing/network: tcpdump or curl traces. - Container snapshot for postmortem:

bash

docker commit <container-id> flaky-investigation:latest
docker save flaky-investigation:latest -o image.tar

4. Narrow cause: - If reproducible only in CI image → environment/config (missing service, timeouts, resource limits). - If reproducible locally after running same container → app defect or test relying on timing/state. - If non-reproducible but fails under parallel/seed/order changes → flaky test (race, timing, shared state). - Use git bisect for recent regressions:

bash

git bisect start
git bisect bad <bad-commit>
git bisect good <good-commit>

5. Next actions & reporting - Attach collected artifacts to bug: logs, report.xml, screenshots, core dumps, container image, env snapshot, exact command and CI job link. - Propose fixes: make test deterministic (mock external services, reset state), increase timeouts, pin CI image, or fix app race condition. Include reproduction steps in ticket.

This approach produces concrete evidence to classify and resolve flaky failures efficiently.

EasyTechnical

25 practiced

You have a new feature behind a feature flag and customers report intermittent errors. Explain how you would use the feature flag to isolate the faulty behavior, gather targeted telemetry from affected users, and roll out a safe rollback or gradual exposure while minimizing customer impact.

Unlock Full Question Bank

Get access to hundreds of Advanced Debugging and Root Cause Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.