Incident Management and Response Questions

Covers operational handling of production outages and service incidents across the full lifecycle from preparation through detection, triage, containment, mitigation, recovery, and post incident review. Interviewers assess monitoring and observability signals, alerting thresholds and on call rotation, severity classification and escalation paths, incident command and coordination, runbooks and playbooks, immediate containment and mitigation techniques to minimize customer impact, restoration and recovery procedures, and evidence capture when relevant. Candidates should be able to describe root cause analysis practices, blameless post incident reviews, tracking remediation and follow up actions, driving cross functional ownership of fixes, and how incident learnings feed into long term reliability improvements and tooling or automation. Senior level expectations include organizing incident response teams for production reliability, defining severity levels and escalation policies, balancing rapid decisions with risk management, and continuously improving processes, runbooks, and instrumentation.

HardTechnical

62 practiced

Explain how to preserve and present a tamper-evident chain of custody for digital evidence gathered during an operational incident (e.g., logs, memory dumps, config snapshots) to support legal or compliance investigations. Describe practical steps and tooling you would use.

Sample Answer

**Situation & goal**Preserve tamper-evident chain of custody for logs, memory dumps, and config snapshots so evidence is admissible and auditable for legal/compliance reviews.

**High-level approach**1. Collect with forensically-sound tools2. Generate immutable hashes and signatures3. Record detailed metadata and transfers4. Store in write-once/append-only storage5. Maintain access logs and audit trail

**Practical steps**- Preparation: maintain documented incident playbook, pre-authorized tools, and a secure collector host (isolated jumpbox).- Collection: - Logs: use secure log-forwarding (rsyslog/Fluentd over TLS) or export original files with timestamps. - Memory: use volatility-compatible dumps via LiME (Linux) or DumpIt/FTK Imager (Windows) from the jumpbox. - Configs: use git snapshot (read-only clone) or export with CLI > file.- Hashing & signing: - Immediately compute SHA-256 and SHA-512 for each artifact. - Sign hashes with an HSM-backed key or at minimum GPG key controlled by evidence custodian.- Metadata & custody record: - Record who collected, tool/version, host, UTC timestamp, hash, signature, reason, chain-of-custody form (CSV/JSON). - Use an append-only ledger (WORM storage or blockchain-backed/logging) to store entries.- Secure storage: - Store artifacts and signed hashes in immutable storage (AWS S3 Object Lock with legal-hold/Governance mode or WORM on-prem NAS). - Keep offline air-gapped backups on encrypted removable media; label and log physical transfers.- Access & audit: - Enforce least privilege, MFA, and logged access to storage; enable object-level logging (CloudTrail, SIEM). - Preserve original copies; work on duplicates for analysis.- Tooling examples: - Collection: LiME, DumpIt, FTK Imager, tcpdump, rsync (read-only), Fluentd - Hash/sign: sha256sum/sha512sum, OpenSSL, GPG, PKCS#11/HSM - Storage/audit: AWS S3 Object Lock, Azure Immutable Blob, WORM NAS, Elastic SIEM, Splunk, CloudTrail - Case management: TheHive, RTIR, or simple signed JSON/CSV with secure git (read-only repo)- Validation & reporting: - Show verifies: hash verification steps, signature verification, audit logs. - Produce formal report with timeline, signed by custodian, and preserve chain-of-custody artifacts.

**Why this works**Immediate hashing+signing prevents undetected modification; immutable storage and detailed metadata show who handled evidence and when; auditable logs and controlled access satisfy legal/compliance requirements.

HardTechnical

55 practiced

Define how Service Level Objectives (SLOs) and error budgets should influence incident response priority and release blocking policies. Provide a concrete example mapping SLO breach scenarios to on-call actions and deployment freezes.

EasyTechnical

52 practiced

How should a systems administrator verify backups and test restores for critical systems? Describe a practical verification plan (frequency, environments, test cases) and what metrics you would track to ensure backups are reliable.

MediumTechnical

64 practiced

Explain how you would run a blameless post-incident review after a major outage. Describe the structure of the postmortem document, how to facilitate the review meeting, how to track action items, and how to ensure follow-through and accountability without blaming individuals.

Sample Answer

**Situation & goal**After a major outage I run a blameless post-incident review to restore service confidence, capture root causes and preventive actions, and improve runbooks and alerts — not to punish individuals.

**Postmortem document structure**- Summary: incident timeline, impact (services/users), duration, business impact (SLA/Mttr).- Timeline: minute-by-minute annotated events from alerts to remediation.- Contributing factors: technical and human/system-level issues (e.g., misconfigured firewall rule, cascade from auto-scaling).- Root cause analysis: causal chain (why-why), not finger-pointing.- Mitigations taken during incident.- Action items: owner, priority, due date, verification criteria.- Learnings and follow-ups: runbook changes, monitoring, testing, training.

**Facilitating the review meeting**- Invite cross-functional attendees (on-call, SRE, network, app owners, product).- Set ground rules: blameless, evidence-based, focus on systems/processes.- Walk timeline aloud, ask clarifying questions, surface contributing factors.- Use neutral language and encourage psychological safety; call out systemic issues (e.g., unclear runbook, insufficient alerts).

**Tracking action items & ensuring follow-through**- Log actions in a tracked system (ticket tracker or board) with clear owners, deadlines, and verification steps.- Triage actions by severity: P0 (must), P1 (recommended), P2 (nice-to-have).- Assign a reviewer to verify completion and effectiveness (e.g., runbook tested in staging).- Add recurring check-ins: 1-week and 1-month follow-ups on status; report metrics (MTTR, number of similar incidents).- Reward improvements and publicize wins to reinforce culture.

**Accountability without blaming**- Frame failures as opportunities to harden systems and processes.- Use data and concrete examples; avoid naming unless permissioned for learning.- If human error is root cause, focus on system design changes (guardrails, automation, better alerts, training) rather than punishment.

MediumTechnical

55 practiced

Describe how you would build and run an incident simulation program (tabletop and live) across product, engineering, platform, and support teams. Include scenario selection, scheduling, success metrics, incident commander rotation, and how to ensure learnings translate to concrete reliability improvements.

Unlock Full Question Bank

Get access to hundreds of Incident Management and Response interview questions and detailed answers.

Join thousands of developers preparing for their dream job.