Systematic Troubleshooting and Debugging Questions

Covers structured methods for diagnosing and resolving software defects and technical problems at the code and system level. Candidates should demonstrate methodical debugging practices such as reading and reasoning about code, tracing execution paths, reproducing issues, collecting and interpreting logs metrics and error messages, forming and testing hypotheses, and iterating toward root cause. Topic includes use of diagnostic tools and commands, isolation strategies, instrumentation and logging best practices, regression testing and validation, trade offs between quick fixes and long term robust solutions, rollback and safe testing approaches, and clear documentation of investigative steps and outcomes.

EasyTechnical

28 practiced

You must update a cluster of backend servers. Compare the trade-offs between doing a rolling restart (one node at a time) and restarting the entire cluster all-at-once. Describe scenarios where each approach is preferable, and list the operational checks and preconditions you would enforce before starting a rolling restart.

HardTechnical

39 practiced

You are the sysadmin lead and must decide between applying an immediate hotfix that temporarily reduces outages and launching a 3–6 month project to redesign a fragile part of the infrastructure. Describe the criteria and decision process you would use to choose an option: include risk assessment, SLO impact, resource allocation, stakeholder communication plan, a tentative timeline for both options, and how you'd measure success post-decision.

Sample Answer

**Decision framework — overview**I frame the choice by scoring short-term mitigation vs long-term remediation across: risk, SLO impact, cost/resources, time-to-value, and stakeholder impact. I aim for a defensible, measurable decision.

**Risk assessment**- Hotfix: low dev effort, immediate reduction in outages; risk = regression, incomplete root cause fix, technical debt accrual. Mitigations: canary deployment, quick rollback playbook, extra monitoring.- Redesign: reduces recurrence risk, improves operability, but risks scope creep, schedule slippage, and temporary instability during migration. Mitigations: phased migration, feature flags, automated tests.

**SLO impact**- Quantify current error budget burn and how much each option reduces outages (estimate). If error budget exhausted or near, prefer hotfix to regain breathing room. If recurring incidents consume >50% of cycles over quarters, prioritize redesign.

**Resource allocation & timeline**- Hotfix path (1–2 weeks): triage, implement fix, code review, canary, rollback plan, full rollout, postmortem. Team: 1–2 engineers + on-call rotation.- Redesign path (3–6 months): requirements, design, prototyping, tests, incremental migration, documentation, runbooks. Team: 2–4 engineers + occasional SRE/QA support; include buffer for ops tasks.

**Stakeholder communication**- Immediate: notify stakeholders of chosen short-term action, expected impact, rollback criteria, and monitoring plan. - For redesign: present roadmap, milestones, resource needs, risk register, and monthly status with demos. Obtain sign-off from product, security, and finance.

**Success metrics**- For hotfix: reduction in incident rate within 30 days, error budget recovery, mean time to detect/resolve (MTTR) improvement.- For redesign: sustained reduction in incidents over 90 days, improved MTTR, improved capacity/throughput, reduced operational toil, successful runbook drills.

I’d choose hotfix if SLOs are imminently at risk and the hotfix buys safe time; choose redesign when incidents are chronic, cost of outages exceeds project cost, and resources are available.

HardTechnical

35 practiced

A distributed storage cluster (e.g., Ceph) is returning occasionally corrupted objects to clients. Describe step-by-step how you would investigate: what cluster health and metadata to inspect, how to run scrubbing and verification, how to repair corrupt objects safely, methods to verify client-visible integrity, and how to communicate risk and mitigation to stakeholders.

Sample Answer

**Situation overview & first actions**- Treat as high-severity: corruption can be silent. Triage to contain scope before mass repair.- Notify stakeholders (ops, app owners) with immediate risk, impacted pools/users, and a short maintenance window if intrusive actions needed.

**Inspect cluster health & metadata**- Check global health and recent events:

bash

ceph status
ceph health detail
ceph -s

- Identify affected pools/PGs/OSDs:

bash

ceph pg dump pgs_brief | grep -E "corrupt|inconsistent|stale"
ceph osd tree
ceph pg <pgid> query

- Audit object metadata and checksums (RADOS/Xattrs, RBD/rgw metadata).

**Run scrubbing and verification**- Start targeted scrubs (avoid cluster-wide load spike):

bash

# schedule deep-scrub for specific PGs/pool
ceph osd pool scrub <pool> --deep-scrub
ceph pg deep-scrub <pgid>

- Monitor scrub logs and events:

bash

ceph health detail
journalctl -u ceph-osd -f

- For RBD/CEPHFS/RGW: use rbd info / rbd diff / fsck tools or rgw-admin to verify object checksums/etags.

**Safe repair workflow**- Prefer read-only verification, then repair only verified corrupt replicas.- Back up affected objects/snapshots before write actions:

bash

# export object as backup
rados -p <pool> get <object> /tmp/<object>.bak

- Use Ceph repair primitives:

bash

ceph pg repair <pgid>        # repairs PG replica set using healthy copies
rados repair -p <pool> <object>  # for specific object (if available)

- If OSD hardware suspected: mark out, reweight, replace OSD, let rebalance complete; avoid simultaneous removals.

**Verify client-visible integrity**- Re-fetch objects from clients and compute checksums:

bash

sha256sum /path/to/object
# compare with known-good checksum or backup

- For S3/rgw, compare ETag/MD5; run application-level integrity tests and sampling reads.- Run systemic tests (consistency checks, fsck for CephFS, rbd bench/verify).

**Prevent recurrence & root-cause analysis**- Check OSD SMART, kernel/syslog, network errors, scrubbing schedule, Ceph version bugs.- Enable/verify checksums at layers (filestore/bluestore checksum config), update to patched Ceph if bug identified.

**Communication to stakeholders**- Immediate: what happened, systems/pools affected, user impact, mitigation steps in progress, expected ETA.- Follow-up: root cause, objects affected count, actions taken (backups, repairs, OSD replacements), risk of data loss, prevention steps and timeline for fixes/patches.- Provide rollback/restore plan and recommended actions for app owners (revalidate checksums, reingest data if needed).

**Outcome & lessons**- Prioritize targeted scrubbing, backups before repair, and clear stakeholder updates. Implement monitoring for silent corruption (periodic deep-scrubs, S.M.A.R.T. alerts, checksum audits).

MediumTechnical

31 practiced

A recent configuration change caused a spike in 502 errors. Design a safe rollback plan that minimizes customer impact: include immediate steps to rollback, verification checks to confirm success, strategies for partial rollbacks/canaries, and communication points with stakeholders. Also list safeguards to avoid repeating the same mistake on the next deployment.

Sample Answer

**Immediate rollback steps**- Stop new traffic to affected release: set load balancer/service to drain new connections for that config group.- Revert to last-known-good config from versioned store (Git/Consul/Parameter Store) and deploy via CI pipeline with the “rollback” tag.- If hotfix required, apply safe revert via feature flag toggle or config flag rather than full deploy.

**Verification checks**- Monitor 502 rate, overall error rate, and 95th/99th latency; expect 502s to return to baseline within 1–3 minutes.- Run synthetic health checks and representative user transactions (login, API calls) against canary and prod.- Inspect application and proxy logs for absence of the previous error signature and confirm no new critical errors.- Check downstream dependencies and circuit-breaker states.

**Partial rollbacks / canary strategies**- Use canary rollback: revert config on a small subset of instances (10–25%) and observe health for 5–15 minutes before full rollback.- Phased rollback by AZ/region to limit blast radius.- Use feature flags to disable only the offending functionality while leaving other changes in place.

**Communication points**- Create an incident channel (Slack/MS Teams) with on-call, SRE, product owner, and support leads.- Post timeline: detection, rollback start, verification milestones, and estimated time to resolution (ETR).- Notify customer-facing teams (support/CS) with guidance for responses and status updates every 15–30 minutes until resolved.- Produce a post-incident report summarizing root cause, impact, timeline, and action items.

**Safeguards to prevent recurrence**- Enforce config review and automated validation tests in CI (schema, type checks, dry-run apply).- Require change approvals for config that touches gateways/proxies and cascade settings.- Add automated canary gating in deployment pipeline: block full rollout if canary health checks fail.- Implement automatic rollback on threshold breaches (error rate, latency).- Improve observability: structured logs, tracing, and synthetic monitors covering critical flows.- Update runbooks and run a tabletop review; add postmortem remediation owners and deadlines.

EasyBehavioral

32 practiced

Describe the key fields and structure you would include in a troubleshooting runbook or investigation document after resolving a production issue. Provide a short template (bulleted list) that an on-call engineer could follow and update during an incident.

Unlock Full Question Bank

Get access to hundreds of Systematic Troubleshooting and Debugging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.