Incident Command and Crisis Leadership Questions

Short definition
An Incident Commander (IC) is the single point of technical and tactical authority during a high-severity production outage. As a systems administrator I lead coordination, set priorities, and keep stakeholders informed so engineers can focus on fixes.

Core responsibilities

• Triage and set goals: quickly assess impact (services/users affected, SLA risk) and define a clear incident objective.
• Prioritize work: rank mitigation tasks (restore service, reduce blast radius, collect diagnostics).
• Coordinate resources: assign roles (owner for networking, storage, app logs) and avoid duplicated effort.
• Communicate cadence: establish regular updates (e.g., every 10–15 min early, lengthen as situation stabilizes).
• Escalate appropriately: involve senior engineers, SRE lead, or management when scope exceeds team ability or SLA is at risk.
• Timebox actions: set short timeboxes (5–15 min) for hypotheses; if no progress, pivot or escalate.
• Delegate & hand off: delegate tactical tasks but retain IC role until stability; hand off after formal briefing, documented state, and overlap for at least one update cycle.

Example
Situation: DB cluster failover impacting web tier. I declared IC, set objective “restore read/write within 30 min,” assigned DB engineer to failover, networking to validate routes, and logging to gather WALs. I gave 10-min timeboxes, provided 10-min cadence updates to stakeholders, escalated to DBA manager at 20 min when metrics showed replication lag, and handed off to on-call lead after service returned and a postmortem owner was assigned.

This approach keeps decisions centralized, communications predictable, and responders focused on technical work.

Situation & immediate goal
Within first 0–5 minutes I declare command, assess scope (affected services, customer impact, SLAs) and set objective: restore service or mitigate customer-facing impact.

Forming the response team (5–15m)

• Triage lead (on-call sysadmin) — investigates logs/metrics.
• Network lead — checks routers, firewall, load balancers.
• App/deploy lead — evaluates recent deployments.
• SRE/DB lead — database health and rollbacks.
• Communications coordinator — external/internal updates.
  I assign roles, contact via phone + Slack incident channel, and start a shared conference bridge.

Priorities & decisions (15–45m)

• Priority 1: stop customer-facing harm (apply mitigations like traffic routing, feature flags, rate limits).
• Priority 2: identify root cause with logs, alerts, recent deploys.
• Decide rollback vs mitigation: if recent deploy clearly correlated and rollback time < mitigation risk, authorize rollback; if rollback risks data loss or longer downtime, implement mitigations and hotfix.

Coordination & cross-team activities (15–75m)

• Use incident channel for async work; bridge for synchronous coordination.
• Assign specific investigation tasks with owners and ETA (e.g., “collect logs 15m — Alice”).
• Run 15-minute rapid syncs to update actions and re-prioritize.

Stakeholder communication

• Status cadence: initial statement at 10m (acknowledge, scope, ETA for next update), then updates every 15 minutes or on major events.
• Channels: PagerDuty for urgent on-call alerts; Slack incident channel and bridge for responders; email/Ops status page and designated AM/PM exec DM for customers/leadership.

Artifacts & post-incident

• Maintain timeline in shared doc (timestamps, actions, owner, outcome).
• Capture logs/metrics snapshots and export alert history immediately.
• Ensure post-mortem ownership assigned before incident closes and evidence archived for RCA.

I emphasize clear ownership, short feedback loops, and decisions driven by minimizing customer impact and risk.

Incident Commander's Decision Checklist — Systems Administrator

Purpose: Rapid, objective guidance to (a) declare Major Incident, (b) escalate to Execs, (c) authorize partial/full shutdowns, (d) approve customer communications.

---

A. Declare Major Incident

• Objective criteria (pick any 2): outage affecting >20% users or >1 critical service; RTO >4 hrs; security breach with confirmed data exfiltration; cascading failures across regions.
• Evidence to gather: monitoring alerts, incident timeline, affected services, estimated impact.
• Stakeholders to notify: On-call SRE/engineer, Lead Sysadmin, Incident Manager, Product Owner.
• Approval gate: Incident Commander (IC) + Lead Sysadmin concur.
• Action: Open major-incident channel, initiate war room, start timeline log.

---

B. Escalate to Executives

• Objective criteria: service outage >8 hrs; revenue-impacting >$X/hr; legal/regulatory exposure; reputational risk (major customer SLA breach).
• Stakeholders: CTO, Head of Ops, Legal, PR lead, Customer Success lead.
• Approval gate: IC + Ops Manager + Head of Product sign-off.
• Action: Send exec brief with impact, mitigation plan, ETA, and next steps.

---

C. Authorize Partial/Full Service Shutdown

• Objective criteria: containment reduces risk (security), prevents data corruption, or hardware failure causing wider damage; estimated benefit > risk.
• Stakeholders: IC, Lead Sysadmin, DB Admin (if DB involved), Network Lead, Product Owner.
• Approval gate: IC + Lead Sysadmin + one technical peer (DB/Network) AND Ops Manager for full shutdown.
• Action: Execute shutdown runbook, record changes, inform stakeholders, enable rollback procedures.

---

D. Approve Customer-Facing Communications

• Objective criteria: communication necessary when outage >30 minutes or impacts SLAs/customers or security incident confirmed.
• Stakeholders: IC, Customer Success, PR, Legal.
• Approval gate: PR + Legal + IC sign-off (exec sign-off for major reputational incidents).
• Action: Publish status page update and templated message; schedule follow-ups and post-incident report.

---

Notes: Timestamp every decision, log rationale, and list next checkpoints (e.g., 30/60/120 min). Maintain single source of truth (incident doc).

Decision framework (fast risk assessment)

• Triage impact: severity (S1/S2), customer-visible scope, data loss vs degraded performance.
• Safety-first: if customer data corruption or security breach suspected → immediate rollback.
• If degradation non-destructive and partial mitigations reduce impact quickly → mitigate and investigate.
• Use RACI: Owning oncall makes call with Approval from engineering lead and Product for customer impact.

Data required (quick checks)

• Deployment timeline and changelog, recent commits/releases.
• Monitoring: error rates, latency, traffic by service/region, logs (sampling), DB anomaly metrics.
• Rollback playbook readiness, backups, migration scripts, DB schema changes.
• Customer-facing ticket/LSAT counts.

Stakeholders to involve (fast)

• Engineering lead, release owner, SRE/DevOps, Product, Support, Security if applicable, and Incident Commander.

Rollback risk considerations

• DB schema or irreversible migrations—rollback may be impossible or risky.
• Cross-service dependency state mismatches, cache/invalidation needs, data divergence.
• Rollback duration and blast radius vs mitigation speed.

Communication & timelines

• Announce decision within 5–10 minutes: state action (rollback or mitigation), reason, expected ETA, and next update cadence (e.g., every 15 minutes).
• Use incident bridge, status page, and support templates. After stabilization, run postmortem including decision rationale, metrics, and remediation plan.

Situation & Task
During a production incident a junior suggested repeatedly restarting a clustered node to “clear state.” That risks split-brain or filesystem corruption. As incident commander I had to stop the risky action, protect data integrity, coach them under time pressure, and restore service safely.

Actions

• Immediately pause further restarts and take control of the console.
• Explain briefly why repeated restarts risk data loss (unclean mounts, split-brain in corosync/pacemaker, in-progress disk writes).
• Implement a safe mitigation: create a storage snapshot or volume-level backup (LVM snapshot, cloud volume snapshot) and export logs (journalctl, dmesg, cluster logs).
• If node must be removed, fence it cleanly (STONITH) or isolate it at the network level rather than rebooting repeatedly.
• Promote a healthy peer to take over services (crm resource migrate / pacemaker failover) or use the load balancer to shift traffic.
• Pair with the junior engineer while I run commands; verbalize intent and ask them to read logs and prepare rollback steps — short coaching moments: cause, risk, safer alternative.
• If uncertain, escalate to senior engineer or follow runbook; record every action in the incident timeline.

Result & Learning
We preserved data, restored service via failover, and the junior learned safer troubleshooting patterns (use snapshots, fencing, consult runbooks). Post-incident I led a blameless review to update runbooks and add a checklist: snapshot -> fence -> failover -> verify.