Systematic Troubleshooting and Debugging Questions

Covers structured methods for diagnosing and resolving software defects and technical problems at the code and system level. Candidates should demonstrate methodical debugging practices such as reading and reasoning about code, tracing execution paths, reproducing issues, collecting and interpreting logs metrics and error messages, forming and testing hypotheses, and iterating toward root cause. Topic includes use of diagnostic tools and commands, isolation strategies, instrumentation and logging best practices, regression testing and validation, trade offs between quick fixes and long term robust solutions, rollback and safe testing approaches, and clear documentation of investigative steps and outcomes.

HardTechnical

54 practiced

Hard: Design a strategy to detect and debug database deadlocks that only occur under production scale. Include diagnostic SQL (e.g., examining lock tables), instrumentation for lock wait times, and application-level changes (e.g., lock ordering, retries with backoff). Explain how to validate the mitigation in production safely.

Sample Answer

Start by clarifying scope and goals: detect root cause of deadlocks that appear only at production scale, gather reproducible diagnostics, add monitoring to catch long lock waits, and deploy application mitigations safely with validation.

Diagnostics (runtime SQL to capture locks / waits)- Periodically sample lock state and active transactions:

sql

-- PostgreSQL: show blocking/waiting
SELECT pid, now() - pg_stat_activity.query_start AS age, query, state,
       wait_event_type, wait_event
FROM pg_stat_activity
WHERE state <> 'idle' AND (wait_event IS NOT NULL OR now() - query_start > interval '1s');

-- current locks
SELECT t.relkind, l.locktype, l.mode, l.GRANTED, a.query, a.pid
FROM pg_locks l
JOIN pg_stat_activity a ON l.pid = a.pid
LEFT JOIN pg_class t ON l.relation = t.oid;

For MySQL/InnoDB inspect INFORMATION_SCHEMA.INNODB_LOCKS and INNODB_LOCK_WAITS and SHOW ENGINE INNODB STATUS for deadlock traces.

Instrumentation- Emit metrics: lock_wait_duration (histogram), lock_acquired_count, deadlock_count, blocked_transactions.- Inject tracing: annotate spans where transactions acquire locks and include table/key identifiers.- Create alert: rate(deadlock_count[5m]) > threshold or p95(lock_wait_duration) > X ms.

Application-level mitigations- Enforce consistent lock ordering across code paths (document ordering by resource id or table priority).- Reduce transaction scope: move non-essential work outside transactions; use SELECT ... FOR UPDATE only when necessary.- Use optimistic concurrency where possible (version/timestamp).- Implement retry with exponential backoff and jitter for transactions that fail with deadlock:

python

def txn_with_retry(func, retries=5):
    backoff = 0.05
    for i in range(retries):
        try:
            return func()
        except DeadlockError:
            time.sleep(backoff + random.uniform(0, backoff))
            backoff *= 2
    raise

Validation strategy (safe production rollout)1. Canary: enable instrumentation and mitigation for a small % of traffic (feature flag).2. Observe metrics (deadlock_count, latency, error rate) and tracing to confirm reduction of deadlocks and no regression.3. Run synthetic load tests against a production-like dataset in a staging cluster using captured query patterns to reproduce scale.4. Gradual ramp: increase canary percentage while monitoring.5. Post-mortem: retain full deadlock traces/logs for 48–72 hours; if mitigation introduces other issues (increased latency or contention), roll back.

Why this works- Combining diagnostics (DB lock tables + engine deadlock traces) with telemetry gives visibility at scale.- App-level changes (ordering, smaller txns, retries) reduce the chance of circular waits and make failures transient and recoverable.- Canary + metrics ensures safe validation without global risk.

HardTechnical

59 practiced

Hard: Explain how you'd diagnose a memory leak in a large Java service running on Kubernetes that shows increasing RSS and eventually OOMs. Discuss how you'd use jmap, jstack, GC logs, heap dumps, FlameGraphs, and kube tools. Provide a step-by-step plan from detection to validation of a fix and discuss the impact of taking a heap dump in production.

Sample Answer

Situation: A large Java service on Kubernetes shows steadily increasing RSS and eventually OOMs. I would follow a disciplined detection → diagnosis → fix → validation plan.

1) Detect & triage- Use kube tools: kubectl top pod, kubectl describe pod, check events, HPA scaling, node pressure. Confirm RSS growth is per-pod and not transient (e.g., memory spike on GC).- Collect timestamps, pod names, container IDs, image tags, and recent deploys.

2) Capture runtime evidence (non‑invasive first)- Enable/collect GC logs (if not already): -Xlog:gc*,gc+heap=debug,file=/logs/gc.log:time,level. GC logs show promotion, full GCs, metaspace or fragmentation.- Get jstack (thread dumps) from the container to see thread states and possible blocked threads: kubectl exec -> jstack <pid> or use jcmd.

3) Heap inspection- If GC logs indicate increasing used heap and promotion failures, take a heap dump. Warning: heap dump can pause the JVM and spike disk I/O; on large heaps it may freeze the process. Prefer doing this on a replica or during low traffic. Use jmap -dump:live,format=b,file=/tmp/heap.hprof <pid> or jcmd GC.heap_dump.- If production risk is high, replicate the problem in a load/staging environment and capture dumps there first.

4) Analyze heap dumps & profiles- Load HPROF into Eclipse MAT to find biggest dominator trees, leak suspects (suspected leak suspects report).- Use async-profiler/async-profiler FlameGraphs (CPU and allocation profiler) to identify high-allocation code paths; produce allocation flame graphs to find allocation hotspots.- Use FlameGraphs for memory allocation (async-profiler -e alloc) and for CPU to rule out GC pressure from hot loops.

5) Cross-check threads and GC: correlate jstack snapshots with GC logs and allocation flamegraphs to see if particular threads are allocating or retaining objects.

6) Narrow root cause & fix- Common causes: unbounded caches, thread-local collections, listeners not removed, native memory (DirectByteBuffer/leaks), classloader leaks from redeploys.- Implement code fix (e.g., bounded cache, weak references, close ByteBuffer, remove listeners). Add metrics and unit/integration tests reproducing leak patterns if possible.

7) Validate- Deploy fix to staging and run load tests; monitor RSS, heap used, GC pauses, allocation rates.- Deploy canary to production with increased observability: heap/GC logs, Prometheus JVM metrics (jvm_memory_used_bytes), p99 latencies, and restart strategy.

8) Postmortem & prevention- Add continuous monitoring, OOM alerts, heap-at-risk alerts, GC logging, periodic heap-dump retention policy, and automated sampling profilers during elevated memory usage.

Impact of taking a heap dump in production- Can pause the JVM (stop-the-world) and consume disk I/O and space; creating a dump of a large heap can take seconds to minutes and may temporarily increase memory footprint. Always prefer taking a dump on a replica or during low traffic and ensure sufficient disk space. If unavoidable, prepare for impact: schedule maintenance window, use pod disruption budget, or spin up a debug replica.

This process combines kube primitives, GC logs, jstack/jmap, heap analysis, and FlameGraphs to move from symptom to validated fix while minimizing production risk.

EasyTechnical

31 practiced

Describe how you would use `git bisect` to find the commit that introduced a regression which causes a failing unit test. Include the high-level commands you would run and how you'd automate bisect if the test can be run non-interactively.

Sample Answer

Approach: use binary search between a known-good commit and a known-bad commit. If the unit test can run non-interactively, you can fully automate git bisect.

Manual bisect commands:1. Start bisect:

bash

git bisect start
git bisect bad            # current commit with failing test
git bisect good <sha-good>  # last known good commit

2. Git will checkout a midpoint; run your test:- If test fails: git bisect bad- If test passes: git bisect good3. Repeat until git reports the first bad commit:

bash

git bisect reset

Automated bisect (non-interactive test):Create a script that returns exit code 0 on success (good) and non-zero on failure (bad), e.g.:

bash

#!/bin/bash
# run_tests.sh — exit 0 if tests pass, non-zero if fail
set -e
pytest tests/test_regression.py -q

Make executable:

bash

chmod +x run_tests.sh
git bisect start
git bisect bad
git bisect good <sha-good>
git bisect run ./run_tests.sh
# git bisect will stop and print the first bad commit
git bisect reset

Key notes:- Ensure working tree is clean before starting.- If tests require build or environment setup, include those steps in the script.- For flaky tests, run them multiple times or add deterministic checks to avoid false positives.- After finding the commit, inspect diff and write a unit test/patch to fix the regression.

MediumTechnical

37 practiced

Medium: A complex bug appears only in production under high load and cannot be reproduced locally. Describe isolation and hypothesis-testing strategies you would use (e.g., sampling, feature flags, traffic mirroring, staged rollout) and explain how each helps validate the root cause without disrupting users.

Sample Answer

I’d follow an evidence-driven, low-risk isolation and hypothesis-testing plan so production users aren’t disrupted.

1) Fast telemetry & targeted sampling- Turn up verbose tracing for a tiny fraction (0.1–1%) of requests using sampling keys. Capture spans, request payloads, latency, error stacks.- Why: gives high-fidelity data from real traffic to reproduce failure patterns without overwhelming logs or impacting performance.

2) Feature flags and toggle-driven experiments- Gate suspect code paths behind a flag; flip the flag for a small user cohort or internal accounts.- Why: isolates the component quickly and lets us compare error rates/metrics between on/off groups to confirm causality.

3) Traffic mirroring (shadowing)- Mirror a percentage of live traffic to a staging cluster that runs diagnostic builds (read-only or isolated DB).- Why: exercises production traffic at scale with no user impact; if the bug appears in shadow, we can iterate safely.

4) Canary / staged rollout- Deploy fixes to a canary subset (single region / 1–5% of pods) with strict health checks and auto-rollbacks.- Why: validates fixes under real load with automated rollback to protect users.

5) Controlled load-testing with recorded traces- Recreate high-load patterns in a performance lab using captured traffic to reproduce race conditions or resource exhaustion.- Why: deterministic reproduction aids debugging (profiling, thread dumps) without touching prod.

6) Hypothesis-driven experiments- Form 1–2 falsifiable hypotheses (e.g., “DB connection leak when concurrency > N”).- Design the smallest test (toggle, sampled tracing, mirrored traffic) that would change measurable outputs (error rate, conn count). Run, observe, update hypothesis.

7) Safety nets and observability- Add circuit breakers, rate limits, and alerting before experiments. Monitor key SLOs and have an on-call rollback plan.

Example flow: enable sampled tracing for 0.5% → observe connection metrics suggesting leak → mirror traffic to staging with a patched client to confirm behavior → roll out patch to a 2% canary with auto-rollback → full rollout.

This approach balances gathering actionable data with minimal user impact and fast recovery.

EasyTechnical

27 practiced

Write a short Python helper that wraps a function and logs execution time and exceptions for diagnostic purposes. Signature: `def instrument(func):` should return a wrapper. The wrapper should log start/end timestamps and exception stack traces using the standard `logging` module. Provide the implementation and a brief explanation of why such an instrumentation wrapper is useful in debugging.

Sample Answer

Approach: implement instrument as a decorator that wraps func, logs start and end timestamps (human-readable and high-resolution elapsed time) and logs exceptions with full stack trace using logging.exc_info. Use functools.wraps to preserve metadata.

python

import logging
import functools
import time
from datetime import datetime

logger = logging.getLogger(__name__)

def instrument(func):
    """Decorator that logs start/end timestamps and exceptions for diagnostics."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        start_ts = datetime.utcnow().isoformat() + "Z"
        logger.info("START %s at %s args=%s kwargs=%s", func.__qualname__, start_ts, args, kwargs)
        try:
            result = func(*args, **kwargs)
            return result
        except Exception:
            # Logs exception with stack trace
            logger.exception("EXCEPTION in %s", func.__qualname__)
            raise
        finally:
            end_time = time.perf_counter()
            elapsed = end_time - start_time
            end_ts = datetime.utcnow().isoformat() + "Z"
            logger.info("END   %s at %s elapsed=%.6fs", func.__qualname__, end_ts, elapsed)
    return wrapper

Why useful: This instrumentation gives immediate visibility into when a function runs, how long it takes, and the full stack trace when it fails — essential for diagnosing performance regressions and intermittent errors in production. It’s low-overhead, reusable across functions, preserves original signature and exceptions, and integrates with existing logging/monitoring pipelines. Edge cases: avoid logging extremely large args (sanitize or truncate), and consider async support (use an async wrapper) if decorating coroutines.

Unlock Full Question Bank

Get access to hundreds of Systematic Troubleshooting and Debugging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.