Debugging and Recovery Under Pressure Questions

Covers systematic approaches to finding and fixing bugs during time pressured situations such as interviews, plus techniques for verifying correctness and recovering gracefully when an initial approach fails. Topics include reproducing the failure, isolating the minimal failing case, stepping through logic mentally or with print statements, and using binary search or divide and conquer to narrow the fault. Emphasize careful assumption checking, invariant validation, and common error classes such as off by one, null or boundary conditions, integer overflow, and index errors. Verification practices include creating and running representative test cases: normal inputs, edge cases, empty and single element inputs, duplicates, boundary values, large inputs, and randomized or stress tests when feasible. Time management and recovery strategies are covered: prioritize the smallest fix that restores correctness, preserve working state, revert to a simpler correct solution if necessary, communicate reasoning aloud, avoid blind or random edits, and demonstrate calm, structured troubleshooting rather than panic. The goal is to show rigorous debugging methodology, build trust in the final solution through targeted verification, and display resilience and recovery strategy under interview pressure.

HardTechnical

124 practiced

You are seeing an intermittent segmentation fault in a C++ service that occurs roughly once per day. Core dumps are limited. Under production pressure, outline a detailed plan to capture more information and reproduce the crash: enabling core dumps, adjusting ulimit, adding diagnostics (backtraces on crash), running ASAN/valgrind in staging, and ways to minimize performance impact while collecting more data.

Sample Answer

Situation: An intermittent segfault (~once/day) in production with limited core dumps. Goal: capture reliable diagnostics and reproduce the crash with minimal production impact.

Plan (ordered, with commands/examples and risk controls):

1) Immediate low-risk prod changes to capture more info- Enable core dumps system-wide (or per-service) and route to safe location: - echo '/var/coredumps/%e.%p.%t.core' > /proc/sys/kernel/core_pattern - mkdir -p /var/coredumps && chown appuser:appgroup /var/coredumps- Raise core-size for service user or systemd unit: - ulimit -c unlimited (in startup script) OR in systemd: - [Service] LimitCORE=infinity- Ensure permissions and disk quota; rotate cores (logrotate) and rate-limit writes (see next).- Use coredumpctl on systemd systems to retrieve cores: coredumpctl info <PID> / coredumpctl dump

2) Add safe, low-overhead crash diagnostics in-app- Install an async-signal-safe SIGSEGV handler that: - Captures a lightweight backtrace (backtrace_symbols_fd()) and writes to a file or journal. Example: use backtrace()/backtrace_symbols_fd() (works in signal handler).- Also call abort() to trigger core dump after logging.- Optionally integrate Google Breakpad/minidump or Sentry for minidumps (smaller than full cores).- Example flow: on SIGSEGV -> write timestamp + backtrace to /var/log/app/crash-<pid>.log -> raise(SIGABRT).

3) Capture full symbolized backtraces from cores (post-mortem)- Build and keep matching debug-symbol package (strip binaries but keep separate .debug).- Use gdb: gdb -c core /path/to/binary --batch -ex "thread apply all bt full"- Automate: script to pull core & run gdb to produce human-readable trace and upload to artifact store.

4) Reproduce in staging with heavier tools- Build debug/ASAN/UBSAN builds: - Compile with -g -O1 -fsanitize=address,undefined -fno-omit-frame-pointer - Run under ASAN in staging (ASAN will abort with stack + heap info). Ensure ASAN_OPTIONS=halt_on_error=1:prefer_static=false:detect_leaks=0- Use Valgrind for memory errors if ASAN doesn't catch (much slower). Run targeted scenarios, replay production-like load.- Use deterministic stress tests: replay traffic, increase concurrency, extend run-time, fuzz inputs to surface race or heap corruption.

5) Minimize production performance impact- Don't enable ASAN/Valgrind in prod. Instead: - Canary approach: deploy instrumented builds to 1-2 canary hosts. - Sampling: attach lightweight continuous stack-sampling profiler (perf, eBPF) across fleet to detect hotspots/races without full tracing. - Rate-limit core/minidump generation: only collect full core for first N crashes per hour (script checks existing dumps) or capture minidump then skip. - Use user-space minidump (Breakpad) which is much smaller and cheaper than full cores. - Limit backtrace depth and use async-signal-safe logging to minimize risk.- Use correlating logs/metrics: add unique request IDs, log context before crash windows to help reproduce.

6) Repro strategy- Correlate timestamps + host + logs to find common inputs/state.- Replay exact traffic (recorded requests) in staging and run under ASAN/valgrind.- Add additional assertions and sanitizer builds on branches touching suspicious code.- If suspecting race, reproduce under stress testers (helgrind/tsan): compile with -fsanitize=thread for race detection (staging only).

7) Follow-up and automation- Automate core collection, symbolization, and alerting (upload bt to ticketing).- Add regression tests for discovered causes and add CI checks (UBSAN/TSAN on PRs where relevant).

Why this order: enable safe collection first (low risk) so future rare crash yields data; add in-process lightweight logging to maximize chance of immediate diagnosis; then reproduce in staging under heavier, slower tools to root-cause; control production risk via canaries, sampling, and rate-limits.

Edge considerations:- Cores may contain secrets—secure storage and access controls required.- Signal handlers must be async-signal-safe; avoid heavy allocations in handler.- ASAN/TSAN are not for prod due to overhead; use limited canaries.- Keep debug symbols matched to deployed binaries (versioned).

This plan balances immediate data collection, controlled risk in prod, and rigorous reproduction in staging.

HardTechnical

75 practiced

A numerical algorithm occasionally misclassifies inputs near a decision boundary due to floating-point rounding. Explain how you would debug and verify whether floating-point precision is the root cause (unit tests with extreme magnitudes, high-precision arithmetic check), propose concrete fixes (epsilon comparisons, Kahan summation, stable algorithms), and describe regression tests you would add to prevent recurrence.

Sample Answer

First I’d confirm whether floating‑point rounding is the root cause by reproducing and isolating failure cases and comparing results with higher precision.

Debug & verification steps- Reproduce deterministically: capture inputs that misclassify and write failing unit tests.- High‑precision check: compute the same decision using a higher‑precision library (Python’s decimal with increased context, or mpmath) and compare outcomes; if high‑precision flips the decision, precision is likely the culprit.- Magnitude sweep & pathological inputs: unit tests over extreme magnitudes (very small/large values, subnormal numbers) and near-boundary perturbations (±ulp) to see sensitivity.- Statistical tests: run randomized/Monte‑Carlo tests and log distance to decision boundary distribution.- Instrumentation: log intermediate values, condition numbers, and relative errors.

Concrete fixes (with examples)- Use robust comparisons: avoid direct equality; use relative epsilon tailored to scale.

python

import math
def almost_equal(a, b, rel=1e-12, abs_tol=1e-15):
    return abs(a-b) <= max(rel*max(abs(a),abs(b)), abs_tol)

- Reformulate math to avoid catastrophic cancellation (e.g., use stable quadratic formula form or log-sum-exp for probabilities).- Kahan summation for long sums to reduce error:

python

def kahan_sum(nums):
    s = 0.0
    c = 0.0
    for x in nums:
        y = x - c
        t = s + y
        c = (t - s) - y
        s = t
    return s

- Use compensated algorithms / pairwise summation, or libraries with higher precision (long double, 128-bit, MPFR) where necessary.- Guardrails: clamp values, normalize input ranges, scale variables to bring values into numerically stable ranges.

Regression tests to prevent recurrence- Add unit tests for known failing inputs and their high‑precision reference outputs.- Property-based tests (Hypothesis / QuickCheck): generate wide ranges and assert stability properties (monotonicity, bounds).- Ulp-based tests: check that results don’t change for ±1–2 ulp perturbations unless mathematically expected.- CI: run an extended numeric test suite on merge (including high‑precision reference checks periodically or for PRs touching numeric code).- Performance guard: measure and assert acceptable numeric error (max relative/absolute error thresholds).

This approach isolates the cause, applies minimal invasive fixes (epsilon, Kahan, reformulation), and prevents regressions with deterministic and property tests integrated into CI.

MediumTechnical

124 practiced

You're fixing a production bug that may take hours. Draft the communication strategy and three templated messages: (1) initial public status (within 10 minutes), (2) 30-minute internal update with technical info, and (3) final resolution summary. For each, explain what technical detail to include or omit for external vs internal audiences.

Sample Answer

Communication strategy (summary):- Triage fast, announce within 10 minutes, set expectations (impact, scope, ETA estimate), form incident team and owner, provide regular cadence updates (every 30 mins) internally and less-frequently externally, escalate as needed, and post a final RCA/resolution within 24–48 hours.- Audience rule: external = customers/stakeholders — clear impact, mitigation steps, ETA, and reassurance; omit low-level root cause or sensitive logs. Internal = engineers/ops — include logs, error traces, rollback/runbook steps, ownership, and next actions.

(1) Initial public status (within 10 minutes)Subject: Service outage affecting [feature/service] — investigatingBody: We are aware of an outage affecting [who/how many, e.g., “payments for some users”]. Engineers are investigating. Impact: [users affected / degraded / partial]. No action required by customers. We’ll provide another update within 60 minutes or sooner. Incident owner: [name/contact]. Apology and short reassurance.

What to include/omit: Include impact, scope, ETA, contact. Omit technical root cause, stack traces, and internal remediation steps.

(2) 30-minute internal update (technical)Subject: INC#[id] — Update (30m): cause hypothesis & next stepsBody: Current status: degradation persists; affected component: service-A (Kubernetes pod crashloop); error seen: “OutOfMemoryError” in logs (snippet below). Actions taken: scaled replicas, rolled back commit abc123, disabled feature flag xyz. Next steps: collect heapdumps, run memory profiler, engage backend team. Owner: [name]. Estimated time to resolution: ~2–4 hours.Attach: relevant log snippets, metrics dashboards, runbook link.

What to include/omit: Include stack traces, metrics, config changes, commands run, and mitigation steps. Omit customer-identifying data and long raw logs not needed for diagnosis.

(3) Final resolution summarySubject: Incident resolved — [service] restored (INC#[id])Body: Summary: Incident started at [time], caused by memory regression in service-A introduced by commit abc123 leading to pod crashes; mitigation: rolled back commit and scaled replicas; full service restored at [time]. Impact: [users affected, downtime duration]. Root cause: faulty allocation pattern in module X. Remediation: reverted commit, added automated memory tests, deployment gate, monitoring alert for OOM. Action items & owners: [list] (RCA to follow by [date]). Thanks for your patience; contact [owner] for questions.

What to include/omit: External customers: include impact, resolution, and customer-facing prevention steps (no internal code diffs). Internal: include full RCA, code link, test coverage additions, and retrospective items.

HardTechnical

78 practiced

Design an automated pipeline that, given a failing test and a range of commits, finds the minimal code change introducing the bug by combining git-bisect with input/test minimization and build caching. Explain how you would handle flaky tests, expensive builds, and how results are presented to developers for fast fixing.

Sample Answer

Requirements & goals:- Given: failing test + commit range- Output: minimal committing change that introduced bug, a minimized reproducible test input, build artifacts to reproduce locally, and a clear report for devs- Constraints: flaky tests, expensive builds, CI resources

High-level pipeline:1. Prepare: checkout tip of range, reproduce failure locally to capture test command and environment. Record baseline build cache keys.2. Bisect orchestration: automated controller uses git-bisect to pick commits. For each candidate commit: - Restore build cache (artifact keys derived from commit, target, and dependencies). - Perform incremental build; if cache hit, skip full compile. - Run test under controlled sandbox (isolated container, fixed seeds, determinism flags) with instrumentation to capture failure trace.3. Test/result minimization: when the test fails at a commit, run delta-debugging on inputs/arguments and on test steps (binary search over test harness) to produce a minimal reproducer. Also perform a minimal code-diff isolation: apply git diff hunks to a clean workspace to confirm smallest hunk that reproduces.4. Flaky-test handling: each candidate is evaluated with N repeated runs (N adaptive: e.g., 3→7) and statistical rules (e.g., require >=95% failure certainty) before marking commit bad. Maintain historical flakiness scores; skip bisect branches with high flakiness unless reproducible.5. Cost & parallelism: parallelize bisect jobs across independent commit subranges using a prioritized queue (earliest likely culprits first by heuristics like recent edits to test or touched files). Use aggressive build caching + content-addressable artifact store (CI cache, remote build cache such as Bazel or sccache). For expensive native builds, run unit-level test-only builds or use “reproducer builds” limited to impacted modules by dependency graph analysis.6. Tooling & feedback: - Produce a concise report: culprit commit hash, author, minimal diff hunk, minimized test inputs, stack traces, deterministic repro command, and confidence score. - Attach artifacts: cached build tarball, container image, failing logs, flakiness history. - Create an automated PR (or issue) against culprit commit with labels, suggested fix locations, and a one-click “reproduce in CI” button. - Provide a web UI timeline showing bisect path, run durations, cache hit/miss, and retry counts.Trade-offs & safeguards:- Conservative vs aggressive minimization: keep original failing run alongside minimized case to avoid false positives.- Resource limits: abort if budget exceeded and provide best-effort candidate list.This pipeline yields fast, high-confidence bug origin identification while minimizing wasted build time and surfacing actionable repros to developers.

HardTechnical

73 practiced

You're leading recovery from a bad release that broke user-facing workflows. Create a prioritized, time-bound action plan: immediate mitigations (rollback or feature flag), customer communication, monitoring and verification, data repair/backfill plan, and a post-incident retrospective. Assign roles and criteria for declaring the incident resolved.

Sample Answer

1) Immediate triage (0–15 min)- Declare incident and appoint Incident Commander (IC) — Release Owner or senior SRE.- Stand-up short war-room (Zoom/Slack) with IC, SRE, Release Engineer, Product Manager (PM), Customer Success (CS), Data Engineer, QA.- Goal: Stop user-impacting change. IC decides mitigation path using criteria below.

2) Immediate mitigations (15–60 min)- Decision criteria: - If full rollback is safe and quick (<30–60 min, minimal DB/schema drift) → execute rollback. - If rollback risky or long (migrations, incompatible schemas) → flip feature flag(s) to disable faulty behaviour.- Roles: - Release Engineer/SRE: execute rollback or flip flags and validate deployment pipeline. - QA: smoke-test critical workflows after mitigation.- Timebox: mitigation completed and basic verification within 60 minutes.

3) Customer communication (30–90 min)- CS + PM draft an initial status message: brief summary, impact, mitigation in progress, ETA for next update.- IC approves; CS posts to status page, notifies affected customers (support triage) and internal stakeholders.- Schedule frequent updates (every 30–60 minutes) until stabilized.

4) Monitoring & verification (continuous; intensify first 4 hours)- SRE implements or verifies: - Business metrics: success rate of key workflows, error rates, latency, user-facing exceptions, affected user count. - Canary/health checks and synthetic transactions across critical paths.- Acceptance criteria to declare “mitigated”: - Key workflow success rate restored to ≥ 99% of baseline. - Error rate for related components down to baseline. - No new regressions for 2 consecutive monitoring intervals (e.g., 30–60 min).- QA runs end-to-end tests and sanity checks on production traffic if safe (shadow/canary).

5) Data repair / backfill plan (1–72+ hours)- Triage data integrity: Data Engineer + Backend owner determine damage scope (lost writes, partial writes, inconsistent state).- Short-term: stop further writes to affected paths if they cause corruption (via flag/traffic routing).- Repair strategy: - Reconstruct from logs/event store/queue replay where possible. - Idempotent backfill scripts with dry-run mode; run on subset and validate. - If manual reconciliation needed, CS prepares list for targeted customer fixes.- Timeline: - Analysis and plan within 4 hours. - Dry-runs and validation 6–24 hours. - Full backfill: scheduled with change window and monitoring; expected completion depends on data size (document ETA).- Roles: Data Engineer (lead), Backend owner (approver), QA (validate), IC (coordinate customer impact and timing).

6) Post-incident retrospective (72 hours after containment)- IC schedules blameless postmortem within 3–7 days with stakeholders: engineers, SRE, PM, CS, QA.- Outputs: - Timeline and root cause analysis (RCA). - Concrete action items with owners and deadlines (tests, automation, improved rollback, runbooks). - Preventive measures: stricter canary policy, deployment guardrails, pre-deploy schema checks, feature-flag coverage.- Follow-up: track action items to completion in engineering board.

7) Incident resolved criteria (must all be met)- Technical: production workflows restored to baseline and verified for at least 2 consecutive monitoring intervals.- Data: either no data loss or repair/backfill completed and validated for affected datasets.- Communication: affected customers notified of resolution and remediation plan; status page updated.- Documentation: RCA drafted and action items assigned.

8) Example assignment summary- IC: Release Owner (or senior SRE)- SRE/Release Engineer: mitigation, monitoring, deployment tasks- Backend owner / Data Engineer: data triage & backfill- QA: verification & test automation- PM: customer impact prioritization, approves comms- CS: customer communications and support triage

This plan balances speed (containment), safety (avoid risky rollbacks), clear ownership, measurable verification, and a structured path to repair and learning.

Unlock Full Question Bank

Get access to hundreds of Debugging and Recovery Under Pressure interview questions and detailed answers.

Join thousands of developers preparing for their dream job.