Incident Command and Crisis Leadership Questions

Situation: During a high-severity outage (database failover flapping) at 2am, the on-call lead and I were both paged; I was coordinating cross-team comms and needed someone to execute a rollback and runbooks because the lead engineer was tied up with upstream vendor calls. A junior SRE with 6 months on call experience was available.

Task: I needed them to perform a controlled rollback of a recent schema migration, run health checks, and gather logs—without making the production state worse.

Action:

• Clarified the objective and success criteria: "Rollback migration X to version v2, verify DB connections <1% error rate, app latency <200ms, and collect app/db logs for 15 minutes."
• Broke the work into explicit steps and handed them a checklist: pre-checks, rollback command, post-checks, and escalation triggers.
• Paired over screen share for the pre-checks (10 minutes): I watched them run the safety queries, confirm backups and replica sync, and validate maintenance window flags.
• Added safety guards: prevented destructive commands unless two approvals were given (me + on-call lead), used a read-only snapshot to rehearse, and set a temporary traffic shift to a healthy region via the load balancer before the rollback.
• Gave clear escalation thresholds (e.g., error rate >5% or replication lag >30s) and stayed available on a dedicated Zoom while they executed.
• Had them narrate each command and expected result out loud to ensure understanding.

Result: The rollback completed in 22 minutes, error rate dropped from 8% to 0.7%, and latency returned to normal. No additional data loss occurred. The junior SRE executed precisely and gained confidence.

Follow-up coaching & process changes:

• Conducted a 1:1 debrief the next day using the incident timeline; praised decisions and corrected minor gaps (improving query timeouts).
• Ran a tabletop session with the team to practice the checklist and refine escalation thresholds.
• Updated the runbook to include the safety rehearsal step, added automated pre-check scripts to enforce backup and replica health, and introduced a mandatory two-person approval for production rollbacks.
• Scheduled mentorship: weekly pairing sessions and a mandatory on-call runbook certification for new SREs.

What I learned: Breaking high-pressure tasks into clear steps, pairing during execution, and baking safety checks into runbooks lets junior engineers contribute safely while accelerating their growth.

Situation: As an on-call SRE team, prolonged high-severity incidents and frequent paging cycles can push the team into fatigue and cognitive overload, degrading decisions and increasing risk.

Signs (signals):

• Slower acknowledgement and longer mean time to respond/repair (MTTA/MTTR)
• Repeated human errors, missed runbook steps, or incorrect rollbacks
• Multiple overlapping incidents without clear ownership
• Team members reporting exhaustion, reduced situational awareness, or making simple mistakes
• Increase in “noise” alerts being ignored or suppressed ad-hoc
• Decline in post-incident follow-through (missing postmortems or actions)

Immediate actions (during incident) as Incident Commander:

• Yellow card: enforce role clarity—assign IC, communications lead, bridge owner, and secondary responders; rotate cognitive load
• Triage: reduce scope—declare mitigation goals (protect customers first, then restore full service)
• Reduce noise: silence non-actionable alerts and throttle noncritical pages
• Enforce short breaks: schedule 15–30min rotations for responders; bring fresh eyes for complex decisions
• Use checklists/runbooks and read-back confirmations to avoid mistakes
• Communicate transparently to stakeholders to reduce ad-hoc interruptions

Longer-term actions:

• Rebalance on-call load and increase staffing or shadow rotations to rebuild capacity
• Reduce alert fatigue: tune alert thresholds, add deduplication and SLO-based alerting
• Automate repetitive remediation (playbooks → runbooks → automated playbooks)
• Improve runbooks and run regular war-room drills to practice escalation
• Implement wellness policies: maximum consecutive on-call days, guaranteed recovery time, and on-call pay/time-off
• Conduct blameless postmortems focused on process and capacity, and track remediation to completion

Result: These steps restore operational capacity, reduce error rates, improve MTTR, and prevent future overload by addressing both technical and human factors.

Situation: At a previous company we experienced a global, high-severity outage affecting our main API and web users across regions during peak hours.

Task: As the on-call SRE lead, I had to set up and run a clear incident communication plan so technical teams, product, support, execs, and external customers stayed informed and aligned.

Action:

• Stakeholder mapping & responsibilities
  • Engineering (on-call, incident commander, SWAT): triage, mitigation, RCA artifacts.
  • Product: prioritize user-facing work, customer impact assessment.
  • Support/CS: customer-facing answers, FAQs, backlog routing.
  • Executives: business impact, decision authority, stakeholder escalation.
  • Legal/PR: review external messaging if data exposure or major market impact.
• Cadence of updates
  • Initial: immediate acknowledgement within 5–10 min.
  • Rapid cadence: every 15 min for first hour (or until a working mitigation).
  • Stabilize: every 30–60 min while degraded.
  • Resolution: final update within 30 min of full recovery, followed by postmortem timeline within 72 hours.
• Channels & formats
  • Bridge (video/voice + chat): primary coordination for responders.
  • Status page: public incident banner + timeline updates.
  • Social/Email: customer-facing updates (sent when confirmed facts change).
  • Internal Slack/channel: running log, timelines, decisions.
• Templated messages (short, consistent)
  • Initial: “We are aware of a service outage impacting [scope]. Investigating. ETA: TBC. Next update: +15m.”
  • Progress: “Root cause: [high-level]. Mitigation in progress: [actions]. Impact: [scope]. Next update: +30m.”
  • Resolution: “Service restored at [time]. Cause: [brief]. Next: postmortem by [date].”
• Escalation criteria
  • Execs: sustained outage >30–60 min affecting >X% revenue/traffic or key customers, or when action requires business trade-offs.
  • Legal/PR: confirmed or suspected data breach, regulated-data exposure, or negative press/social amplification above threshold.
  • Immediate paging if incident affects payments, compliance, or critical SLAs.

Result: This structured plan kept teams aligned, reduced redundant work, enabled clear customer communication, and allowed leadership to make timely trade-offs. Postmortem showed faster time-to-mitigation and positive stakeholder feedback. Learned to codify thresholds and automate status-page pushes to remove human delay.

Situation: Eight-hour SEV1 with major customer impact; executives demand timelines and are publicly assigning blame.

Task: As Incident Commander (IC), my role is to lead remediation, coordinate communications, protect responders, and deliver a blameless postmortem.

Action:

• Establish command structure immediately:

  • IC (me): single point of decision/communication and prioritization.
  • Roles: Communication Lead (exec & external), Tech Lead, Scribe, Runners (fetch logs/permissions), and Subject-Matter Experts (DB, infra, app).
  • Create an incident channel and a short-lived war room (virtual/physical).

• Structured updates and cadence:

  • Publish a concise incident status every 15–30 minutes to execs and stakeholders using the template: Impact | Current Hypothesis | Actions in Progress | Blockers | Next ETA.
  • For example: “Impact: 60% of auth traffic failing. Hypothesis: DB connection pool exhaustion. In-progress: scaling DB read replicas, increasing pool limits; investigating trace IDs. Blocker: config change approval. Next ETA: 30m.”

• Protect the response team:

  • Enforce a “no-external-interruptions” rule: all exec questions go to Communication Lead.
  • Keep the core remediation team focused in the war room; use Runners to surface non-technical needs.
  • Rotate responders every 2–3 hours to avoid burnout; onboard replacements during handoff.

• Manage executive communications:

  • Be transparent, data-driven, and set expectations. Provide the three-line summary for execs and a public-ready statement vetted by Legal/PR.
  • When pressured for timelines, give ranges tied to current mitigations (e.g., “If config rollback succeeds, restoration in 20–30m; if not, fallback X in 90m”), and commit to the next ETA.
  • Escalate decisions through IC; avoid engineering distraction by routing exec queries to Communication Lead.

• Keep responders focused on remediation:

  • Prioritize actions by impact and effort (quick mitigations first).
  • Maintain a visible task board (short tasks with owners).
  • Stop non-essential experiments; require approval for risky changes.

• Ensure a blameless review:

  • After stabilization, preserve data: logs, timelines, config diffs, runbooks.
  • Schedule a post-incident review within 72 hours with stakeholders; invite only contributors and decision-makers.
  • Use a blameless template: Timeline, Contributing Factors, Root Causes, What Went Well, Action Items (owner + due date).
  • Track action items to completion and publish a customer-facing summary if needed.

Result/Learning:

• This approach centralizes communication, shields engineers, gives execs timely, honest information, and produces concrete follow-ups. It preserves team morale and yields systemic fixes rather than finger-pointing.

Overview: A three-tiered certification (IC1, IC2, IC3) that combines competency-based learning, hands-on simulation, peer evaluation, and measurable promotion gates. Each level builds incident scope, decision-making, and leadership.

IC1 — Incident Commander (Individual contributor)
Learning objectives:

• Triage, communication, runbook execution, basic RCA participation
• Use monitoring/alerting, runbook edits, and safe rollback procedures
  Assessments:
• Written quiz on runbooks, SLOs, and escalation paths
• 1-hour tabletop simulation of a single-service outage with recorded comms
• Peer review of post-incident write-up
  Promotion criteria:
• 3 successful simulations (≥80%), 2 real incidents as acting IC with mentor sign-off, written test ≥85%
  Maintenance:
• Monthly 30‑min drills, quarterly tabletop, mandatory runbook refresher annually

IC2 — Senior IC (Cross-service, coordination)
Learning objectives:

• Coordinate multi-team incidents, prioritize error budget trade-offs, resource allocation, incident strategy
• Mentor IC1s, lead postmortems with blameless facilitation
  Assessments:
• Multi-hour simulated multi-service incident with injected chaos; assess decision logs and comms
• Written scenario-based exam (prioritization, SLO trade-offs)
• 360° peer review (engineers, PMs, on-call)
  Promotion criteria:
• 6 months as IC1 with documented incidents, 5 simulations (including 2 multi-team), peer score ≥4/5, mentor endorsement
  Maintenance:
• Biweekly drills (rotating scenarios), semiannual cross-team war games, ongoing mentorship duties

IC3 — Principal IC (Strategy, policy, escalation authority)
Learning objectives:

• Drive incident strategy for high-severity events, liaise with execs, design incident policy and capacity planning
• Run organization-wide incident retros, SLO governance
  Assessments:
• Full-scale crisis simulation (24-hour exercise) with exec communication evaluated
• Policy design assignment and oral defense
• Leadership peer review and external evaluator
  Promotion criteria:
• 12+ months at IC2, documented leadership in major incidents, successful crisis exercise, executive sponsor approval
  Maintenance:
• Annual full-scale crisis drill, quarterly SLO governance reviews, lead one postmortem per quarter

Support elements:

• Digital learning portal with micro-modules, recorded simulations for asynchronous review
• Certification badge, expiry after 12 months without maintenance activity
• Appeals and remediation path: targeted coaching, supervised incidents, re-assessment

Metrics:

• Time-to-ack, time-to-resolution, postmortem quality score, drill pass rates, and confidence surveys drive continuous improvement.