Incident Command and Crisis Leadership Questions

Situation & goal
I’m the incident coordinator for a high-severity cloud outage where Team A wants immediate containment (isolate affected instances) and Team B wants a longer diagnostic-first approach. My goals: contain impact fast, preserve forensic evidence, and keep stakeholders informed.

Immediate steps (decision-making)

• Triage fast: ask both teams to state immediate risks/benefits (30–60s each). Use runbook criteria (safety, data loss, blast radius) to score options.
• Choose containment-first if user impact or data exfiltration risk is high; choose investigation-first if containment would destroy volatile evidence and impact is moderate. If unclear, prefer a short-lived containment-with-reversible-rollback plan.
• Call the tie-breaker (on-call engineering lead or SRE manager) only if runbook doesn’t resolve it.

Containment actions (cloud-specific)

• Isolate resources via security groups/NACLs or move instances to a quarantine subnet rather than terminate.
• Snapshot disks (EBS snapshots, GCP persistent disk snapshot) and export memory dump if possible.
• Turn on/collect CloudTrail, VPC Flow Logs, ASG/activity logs, and copy logs to an immutable S3 bucket with object lock.
• Freeze deployments (pause CI/CD) and revoke affected credentials.

Stakeholder alignment & communication

• Immediately post a concise incident update: action taken, rationale, owner, ETA for next update (every 15 min).
• Assign clear RACI: who contains, who investigates, who communicates externally.
• Schedule brief syncs for decisions that require longer discussion; escalate blockers to incident commander.

Post-containment

• Verify service stability, validate rollback plan, continue evidence collection.
• Transition to post-incident analysis with preserved artifacts, timeline, and agreed next steps (hotfix vs. long-term fix) and ensure both teams contribute.

Why this works

• Uses playbook + measurable criteria for quick, defensible decisions.
• Preserves forensic evidence by snapshotting and quarantining rather than destroying resources.
• Keeps stakeholders informed and ownership clear so containment and long-term fixes proceed in parallel.