Incident Response and Troubleshooting Questions

Approach to diagnosing and resolving production incidents, outages, and critical failures under time pressure. Covers systematic triage, identifying root causes, maintaining service availability, coordinating with stakeholders, prioritizing safety and mitigation steps, postmortem practices, and learning from incidents to prevent recurrence. Interviewers expect examples showing technical troubleshooting, communication during crises, decision making under pressure, and follow through in remediation and documentation.

HardTechnical

90 practiced

A critical production database shows logical corruption limited to a subset of user records while underlying storage appears healthy. As the Systems Administrator responsible for recovery, outline in detail how you would contain the issue, perform forensic analysis, perform selective restoration to minimize data loss, verify data integrity post-restore, and communicate with legal/compliance and impacted stakeholders.

Sample Answer

**Containment (first 30–60 mins)** - Isolate DB instance (set to read-only or stop writes via app feature/DB-level) to prevent further corruption propagation. - Snapshot storage and take filesystem-level and DB logical backups (cold copy) for forensic baseline. - Preserve logs: transaction logs, binary logs, audit logs, OS syslogs — mark chain and copy to WORM storage. - Raise incident, notify on-call/management and align RTO/RPO expectations.

**Forensic analysis** - Reproduce corruption against a copy: restore snapshot to isolated sandbox. - Correlate timeline: map corrupted user IDs to DB transactions, application deploys, cron jobs, user actions. - Use DB tools (checksum, DBCC, pg_verifychecksum, etc.) to identify patterns/affected tables, rows, indices. - Search audit logs, recent schema changes, failed jobs, or third-party integrations. Document evidence and hash preserved artifacts.

**Selective restoration / recovery plan** - Identify last-known-good transactions per affected user using transaction logs/wal. - If row-level recovery possible: use point-in-time recovery (PITR) to a time just before corruption and extract only affected rows into staging. - If PITR not possible, restore most recent full backup to staging and apply incremental logs to recover up to safe point; export corrected user records. - Apply fixes via transactional batch updates with idempotent scripts, test on staging, and apply during maintenance window.

**Verification post-restore** - Run integrity checks: referential checks, checksums, counts, application-level acceptance tests for affected users. - Compare hashes and audit trails between staging and production. - Monitor metrics and logs intensively for 24–72 hours.

**Compliance & stakeholder communication** - Notify Legal/Compliance immediately with preserved evidence, scope (which users/data types), potential impact, and steps taken. - Provide regular status updates: initial containment, forensic findings, recovery ETA, verification results, and final post-mortem. - If breach or regulatory notification required, coordinate timelines and messaging with Legal. - Produce incident report: timeline, root cause, remediation, preventive actions (additional monitoring, RBAC changes, automated backups verification).

**Preventive actions** - Implement improved alerts, automated backup verification, immutable backups, stricter change controls, and periodic restore drills.

MediumTechnical

53 practiced

During a high-severity payments outage, describe how you would coordinate between engineering, SRE, customer support, and product teams. Define the roles you would assign (incident commander, communications lead, SMEs), the cadence of updates, the content and audience of status messages, and how you would resolve conflicts between competing priorities like security holds vs restoring transactions.

HardTechnical

49 practiced

You are managing a major outage that may be the result of a security incident (possible data exfiltration). Explain how you would coordinate responsibilities between operations (service restoration) and security (containment/forensics). Describe which actions might hinder forensic evidence, how you would document steps, and how you would brief executives and legal/compliance stakeholders.

Sample Answer

**Situation & high-level approach**As a systems administrator I’d split responsibilities immediately: Operations focuses on rapid, reversible service restoration; Security leads containment and forensic preservation. I’d act as the operational lead in coordination with the security incident responder.

**Coordination & responsibilities**- Operations (my team): - Isolate affected systems at network edge or via ACLs/segmentation to stop spread while preserving integrity. - Failover to healthy instances or scale up unaffected services to restore customer-facing functionality. - Implement temporary mitigations (rate-limits, disable exposed endpoints) that are reversible and documented.- Security / Forensics: - Acquire volatile and persistent evidence (memory dumps, disk images, network captures) before changes. - Triage logs, IOC matching, timeline building, chain-of-custody. - Decide on legal holds and escalation to law/regulators.

**Actions that hinder forensics**- Rebooting or patching compromised hosts without prior evidence capture (destroys memory).- Overwriting logs (rotating or truncating) or running broad cleanup scripts.- Performing destructive restores or reimaging before imaging disks.- Allowing automated log pruning/retention jobs to run.

**Documentation & chain-of-custody**- Use an incident ticket with timestamps, actor names, commands run, and justification for each action.- Capture MD5/SHA256 hashes of images, record tool versions, and store forensic images in write-once storage.- Maintain an evidence log (who, what, when, why) and preserve original media.

**Executive & Legal briefing**- For execs: concise risk/status slide — scope, customer impact, ETA for restoration, high-level containment steps, business risks and mitigations.- For legal/compliance: provide full timeline, preserved artifacts, chain-of-custody, indicators of compromise, and recommended regulatory notifications. Offer to coordinate with counsel before releasing external statements.

I’d enforce a “preserve-first, remediate-second” policy, escalate quickly if exfiltration indicators appear, and keep clear two-way communication between ops and security throughout.

HardSystem Design

60 practiced

Propose a chaos engineering program for a large enterprise aimed at proactively discovering weaknesses that lead to incidents. Define program scope, safe blast radius policies, types of experiments (network partitions, instance terminations, resource throttling), CI/CD integration, metrics for program success, and governance and approval processes.

Sample Answer

**Program overview (goal)** I would establish a staged chaos engineering program to proactively surface infrastructure weaknesses before they become incidents, focusing first on non-production, then canary production, then broader production.

**Scope & phasing** - Phase 0: runbooks, incident-response readiness, stakeholder signoff. - Phase 1: dev/staging clusters, single-region services, low-risk infra (logging, metrics). - Phase 2: canary production (subset of users, feature flags). - Phase 3: full production with strong controls.

**Safe blast radius policies** - Blast radius defined by service, region, and user-impact classes. - Default guardrails: run only during business off-hours for risky experiments, automated circuit breaker to abort on SLA/ERR spike, quota limits per team (e.g., max 5% traffic). - Required rollback/test window and automated health checks before escalation.

**Experiment types & examples** - Network partitions: isolate app tier from DB in staging to validate retry/backoff. - Instance termination: kill a node to verify auto-scaling and stateful failover. - Resource throttling: throttle disk IOPS/CPU on a database replica to test read scaling and fallback. - Clock skew, DNS failures, IAM policy revocation tests for auth flows.

**CI/CD integration** - Experiments as code in repo (playbooks + scripts). - Pre-merge checks run lightweight chaos (container restarts) in CI pipelines for microservices. - CD pipelines gate deployments on successful chaos checks in canary environment. - Automated experiment runner with scheduling, results stored in artifacts.

**Metrics for success** - Mean Time To Detect (MTTD) and Mean Time To Recover (MTTR) for injected failures. - Reduction in production incidents attributable to covered failure modes. - Coverage: percentage of services with at least monthly experiments. - Experiment pass/fail rate, false-positive aborts, and business-impact incidents avoided.

**Governance & approval** - Chaos board (SRE, SysAdmin, Product, Security) approves blast radius and escalation policies. - Pre-approval matrix: low-risk experiments auto-approved; medium/high require runbook, owner, and on-call signoff. - Post-mortem mandatory for any failed experiment causing customer-visible impact; learnings feed backlog.

I would run monthly reviews with stakeholders, update playbooks, and gradually expand scope as confidence and automation grow.

MediumTechnical

72 practiced

An incident may require forensic collection on hosts. Describe best practices for collecting evidence on Linux and Windows servers in a way that preserves integrity and chain of custody while minimizing service interruption. Include how you would document the process and which artifacts you prioritize capturing first.

Sample Answer

**Approach (brief)** Preserve integrity and chain-of-custody by performing non-invasive collection first, documenting every action, using forensically-sound tools, and minimizing live system changes. When possible, capture volatile data before powering down; if shutdown is required, create disk images from a trusted forensic workstation.

**Immediate prioritized artifacts (order)** 1. Volatile memory (RAM) — use LiME on Linux or DumpIt on Windows 2. Network connections & sockets — ss / netstat / Get-NetTCPConnection 3. Running processes and loaded modules — ps aux / tasklist + listdlls 4. Authentication logs, sudo/PowerShell history — /var/log/auth*, Windows Security event log 5. Disk metadata & important files — /etc, registry hives (SYSTEM, SAM, SECURITY), key config files 6. Timestamps and filesystem metadata — use read-only imaging

**Commands & tools (examples)** - Linux: sudo ss -tunap; sudo ps auxwww; sudo ausearch; create memory dump with LiME - Windows: Get-Process; netstat -ano; wevtutil epl Security.evtx; use vssadmin only to list

**Preservation & minimal disruption** - Mount evidence storage read-only; if imaging, use ddrescue or FTK Imager via network from a clean host - Avoid installing new agents; prefer portable, signed tools - If service-critical, snapshot VMs or use hypervisor-level snapshots for later analysis

**Documentation & chain-of-custody** - Record timestamped log: who, when, tool + version, command outputs collected, hashes (SHA256) of images, storage location and transfer method - Photograph console/network connections and server state before changes - Sign and store custody forms; calculate and verify hashes whenever moving copies

**Rationale & trade-offs** Prioritize volatile state because it’s lost on reboot. Balance service continuity by using snapshots or live collection from a remote forensic host. Maintain forensic integrity via read-only access, hashing, and detailed logs so evidence remains admissible and reproducible.

Unlock Full Question Bank

Get access to hundreds of Incident Response and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.