Incident Command and Leadership Questions

Covers the skills and responsibilities required to lead and coordinate high severity incident responses as an incident commander or incident lead. Candidates should be able to explain how they direct and prioritize response activities, maintain and communicate an incident timeline and decision log, delegate roles, and make timely decisions with incomplete information. Includes practices for coordinating multi team responses across functions such as network security, threat intelligence, operations, legal, privacy, and executive stakeholders, as well as managing evidence handling, handoffs, and escalation paths. Evaluators will assess communication strategies for technical teams and nontechnical stakeholders, running war rooms or command centers, maintaining composure under pressure, and managing stakeholder expectations during unfolding incidents. At senior levels, candidates are expected to demonstrate experience commanding complex incidents, balancing operational urgency with investigative and compliance needs, documenting decisions for post incident review, and establishing or improving incident command processes and communication protocols.

HardTechnical

44 practiced

An outage may trigger regulatory notification in multiple jurisdictions with different timelines and content requirements. As the incident commander, outline a compliant and auditable process to determine which regulators must be notified, coordinate messaging and timing with legal/privacy teams, and maintain an evidence trail that meets audit and discovery requirements.

Sample Answer

Requirements & constraints:- Multiple jurisdictions may require notification based on impacted users, data types (PII, financial, health), and thresholds/time-to-notify (e.g., 72 hrs GDPR, state breach laws).- Notifications must be auditable and defensible in discovery.

Process (stepwise, auditable):

1) Triage & Data Inventory (0–30 min)- As IC, run predefined checklist: services impacted, affected data types, estimated user count by region (use real-time telemetry + geo-IP), and systems involved.- Automate data pulls (logs, DB queries) with immutable, time-stamped exports stored in a write-once audit bucket.

2) Jurisdictional Determination (30–60 min)- Run an automated rule engine that maps affected regions + data types + thresholds to required regulators (maintained by Legal/Compliance). Produce a decision log recording inputs, rule versions, timestamp, and IC sign-off.

3) Legal/Privacy Coordination (60–120 min)- Convene pager/war room with Legal/Privacy. Share the decision log and evidence extracts.- Use templated notification drafts that include required fields per jurisdiction; Legal approves edits in the shared system (versioned collaboration tool).

4) Messaging & Timing Plan- Legal sets timing per jurisdiction; IC schedules outbound notifications and internal communications. Maintain a timeline artifact (sequence of planned sends with time zones and authority).

5) Execution & Proof- Send notifications via approved channels (secure portal, registered email). Each send produces cryptographic evidence: signed email receipts, delivery logs, or system-generated receipts. Store these in the audit bucket.

6) Evidence Trail & Preservation- All artifacts (raw logs, exports, decision engine input/output, chat transcripts, meeting minutes, draft versions, final sent messages, receipts) are stored with WORM/immutable settings, access logs, and retention policy. Hash and timestamp each artifact; store hashes in separate tamper-evident ledger.

7) Post-Incident Review & Compliance Reporting- Produce a compliance packet including timeline, decision log, legal approvals, and artifacts. Run retention and e-discovery export procedures.

Tools & controls:- Rule engine (config-driven), immutable storage (S3 WORM), versioned docs (confluence/git), secure comms (recorded war room), cryptographic hashing, RBAC and audit logging.

Why this works:- Automates repeatable determinations, ensures Legal is engaged early, creates immutable, timestamped evidence for audits and discovery, and minimizes risk of missed jurisdictional obligations.

EasyTechnical

44 practiced

What is an escalation path in incident management? Describe how you would design escalation tiers, who to include at each tier (technical and non-technical), and the measurable thresholds or triggers you would use to move an incident to the next tier in a typical SRE organization.

HardTechnical

43 practiced

Senior leadership requests an audited, tamper-evident record of all critical incident decisions and time-stamped communications for potential legal review. Design a decision-logging process (tools, cryptographic or process controls) that preserves confidentiality, ensures integrity, supports authorized audits, and defines retention and access policies.

Sample Answer

Requirements (clarify): immutable, tamper-evident, time‑stamped decision + comms, confidentiality, authorized auditability, retention + legal‑hold.

High-level design:- Ingest: Incident responders capture decisions/communications via a controlled UI (chat/decision form) integrated with ticket (e.g., Jira/ServiceNow) and incident chat (e.g., Slack with dedicated #incidents).- Canonical append-only log: a dedicated Decision Ledger service that accepts signed entries and writes to immutable storage.

Tools & components:- Frontend: Incident CLI/web form + integrations to collect: actor, role, timestamp (UTC), decision text, rationale, affected services, related artifacts (screenshots, logs).- Authentication/authorization: SSO (OIDC/SAML) + RBAC via IAM (Okta/Azure AD), MFA required for sign-off actions.- Cryptographic integrity: - Each entry is signed client‑side using per-user keypairs stored in an HSM-backed keystore (AWS KMS/HSM). Signatures include canonicalized payload + previous-entry hash (hash chain), producing an append-only tamper-evident chain. - Ledger records also store SHA-256 digest of any attached binary; attachments stored in WORM storage (S3 Object Lock/GCP Object Retention) encrypted at rest (KMS). - Periodic public anchor: weekly Merkle root anchoring to an external notary (e.g., publish Merkle root to a permissioned blockchain or to a public timestamping service) for extra non-repudiation.- Storage and immutability: - Primary ledger: write-once DB (e.g., PostgreSQL with append-only table + WAL retention) + daily snapshots to immutable object store with Object Lock enabled. - Backups: encrypted, versioned snapshots stored in separate region.- Confidentiality: - Encrypt entries at rest with envelope encryption; per-entry access control lists. Sensitive fields (PII) are redacted/encrypted with separate key policy; access requires justified request and approval.- Access & audit: - Audits granted via short-lived, logged decryption keys (KMS grants) and timebound RBAC roles. All access (read/write/list) logged to immutable audit logs (CloudTrail/Stackdriver) and correlated with ledger entries. - Provide an Audit API that returns verifiable entry + signature + inclusion proof (Merkle path) so auditors can validate chain and signature offline.- Retention & legal-hold: - Policy: default retention = 7 years (example) in WORM. Legal hold mechanism locks retention per-case and prevents deletion; only Legal + Security can create/remove holds, actions logged and signed.- Operational controls: - Change management for ledger service is strictly controlled: infra as code, peer-reviewed changes, canary deploys. Monitoring/alerts on tamper indicators (hash mismatch), unexpected deletion attempts, KMS access anomalies.- Example entry flow: 1. Responder fills decision form -> client signs payload with user private key (held in HSM or client‑side hardware token). 2. Ledger verifies signature, appends entry with previous hash, returns entry id. 3. System persists entry and attachment digests to S3 Object Lock, publishes updated Merkle root to notary. 4. Read/access requires IAM authorization; any access generates an audit event and (optionally) requires an access justification.

Trade-offs and considerations:- HSM-backed keys improve non-repudiation but add operational complexity.- Public anchoring increases cost but strengthens legal defensibility.- Balance redaction vs. forensic needs; maintain separate encrypted forensic store accessible only with multi-party approval.

This design preserves confidentiality (encryption + RBAC), ensures integrity and tamper-evidence (hash chain + signatures + external anchoring + immutable storage), supports authorized audits (verifiable signatures, inclusion proofs, logged access), and enforces retention/legal-hold policies.

MediumTechnical

43 practiced

Provide a template (fields and sample entries) for documenting chain of custody for digital artifacts during an incident involving suspected data exfiltration. Explain how you'd maintain access control to the artifacts and how the document is shared with security and legal teams without compromising evidence integrity.

Sample Answer

Chain-of-Custody Template (fields + sample entry)- Case ID: INC-2025-045- Artifact ID: A-001- Artifact Type: Compressed DB dump (gzip)- Source Host: db-prod-3.example.com (10.0.3.12)- Collection Time (UTC): 2025-11-20T14:22:10Z- Collected By (Name/Role): J. Chen / SRE On-call- Collection Method & Tool: ssh -> sudo pg_dump | gzip, SHA256 computed via sha256sum- Hash (SHA256): b1f2...9a3c- Storage Location (path/ID): /forensics/readonly/INC-2025-045/A-001.gz (WORM-backed)- Access Restriction Level: Confidential — Legal & IR team only- Transfer History: - 2025-11-20T15:00Z — J. Chen -> Forensics S3 (write), signed transfer log - 2025-11-21T09:10Z — M. Singh (Legal) requested read-only snapshot- Notes / Observations: Contains suspicious large SELECTs logged 2025-11-19- Signatures: J. Chen (collector) — digital signature fingerprint: 04ab...

Maintaining Access Control and Integrity- Capture: Collect on isolated network segment; use read-only operations where possible; compute and record cryptographic hashes immediately.- Storage: Store artifacts in write-once-read-many (WORM) or object store with immutable buckets; enable versioning and strict lifecycle policies.- Access Control: Enforce least privilege via IAM roles and short-lived credentials (OIDC or AWS STS). Gate access with multi-party approval (IR + Legal) recorded in an approvals ledger.- Audit & Logging: All access via jump-host with session recording and append-only audit logs (syslog/ELK), record operator identity, timestamp, and purpose.- Sharing with Security & Legal: Provide hashed, read-only copies or pre-signed time-limited URLs; never send via email. When Legal needs copies, require documented approval, record transfer entries in the document, and have recipient verify hash before and after transfer.- Chain Integrity: Require digital signatures (GPG) on custody entries and periodic re-hashing on long-term storage. Preserve original media; if analysis requires changes, work on forensic copies and log every derivative.- Practical SRE considerations: Automate hash computation and COC entry creation as part of collection playbooks, integrate with incident ticket (linked by Case ID), and ensure runbooks list exact commands/tools used so evidence is reproducible.

HardSystem Design

66 practiced

You're rolling out an automated rollback tool that SREs can trigger during incidents. Define the policies and safeguards to prevent harmful or cascading rollbacks, including scope-limiting, explicit approval gates, dependency checks, pre-rollback verification, and post-rollback validation steps.

Sample Answer

Requirements & constraints:- Allow SRE-triggered automated rollback during incidents while preventing harmful/cascading impact.- Must limit scope, require approvals for risky actions, verify dependencies, perform pre- and post-checks, and produce auditability and quick manual override.- Target low-latency for critical rollbacks but prioritize safety.

High-level design:- Rollback Controller (service) + Policy Engine + Dependency Graph DB + Verification Runner + Orchestrators/Agents + Audit & Monitoring.- UI/CLI + Incident Integration (PagerDuty/Slack) for triggers; RBAC + MFA for approvals.

Key policies & safeguards

1) Scope-limiting- Multiple scoping dimensions: service, deployment unit (pod, instance, region), tag/commit hash, traffic % (canary rollback), and time window.- Default: conservative scope (single instance or 5% traffic) with template presets for emergency (broader) vs safe (narrow).- Explicit scope required in trigger; wildcard scopes forbidden unless senior approver & justification logged.

2) Explicit approval gates- Role-based gates: small-scope rollbacks (<=5% or single instance) can be auto-approved by on-call SRE; larger scopes require two-person approval (SRE + owner) or timer-based escalation.- Approval methods: signed API call, UI confirmation, or cryptographic attestations; require MFA for high-impact rollbacks.- Approval rules encoded in Policy Engine (JSON/YAML), versioned and auditable.

3) Dependency checks- Maintain a dependency graph (service → downstream services, DB migrations, feature flags).- Before rollback, verify: - No incompatible API/version constraints in downstreams - No pending DB schema changes tied to target deploy - Feature-flag state compatibility - Active long-lived jobs or consumers (e.g., Kafka consumers) that would mis-handle version skew- If dependency checks fail, block automatic rollback and present safe alternatives (quarantine nodes, throttle traffic).

4) Pre-rollback verification (dry-run)- Dry-run simulation that checks orchestration plan, ensures idempotency, computes blast radius.- Capture pre-rollback system snapshot: current traffic routing, config, metrics baselines, error budgets, and topology.- Run smoke tests or synthetic transactions against a rollback candidate in an isolated canary (if possible) or staging mirror.- Apply rate limiting and throttling parameters (e.g., rollback at X instances per minute).

5) Execution controls & safety mechanisms- Phased execution: canary rollback → monitor → progressive rollout with automatic stop on error thresholds.- Circuit-breakers: if error rate, latency, or SLO breach crosses configured thresholds, automation halts and optionally reverts.- Abort hooks: manual abort immediacy with guaranteed stop and mitigation steps.- Cooldown windows and rollback quotas per service per timeframe to avoid oscillation.

6) Post-rollback validation- Automated validation suite: synthetic user journeys, health probes, telemetry checks (latency, error rate, traffic distribution), DB consistency checks.- Canary metrics compared to pre-rollback baselines; require pass criteria before continuing progressive rollback.- If validation fails, automation either auto-rollback to previous stable artifact or pause and flag for human intervention.- Full audit trail with timestamps, approvers, diffs, and telemetry snapshots; notifications to stakeholders and incident system.

Operational & governance considerations- Policy Engine stores policies per service with safe defaults; changes to policy require change control.- Chaos-safety tests in staging: periodically simulate rollbacks to validate the automation.- Logging, immutable audit, and post-incident playbooks generated automatically for RCA.- Trade-offs: heavier checks increase time-to-rollback; mitigate by precomputing dependency graphs, running readiness probes continuously, and keeping rollback artifacts ready.

Why this prevents cascading harm- Scope-limiting and phased canaries minimize blast radius.- Dependency checks prevent incompatible state flips.- Approval gates and RBAC ensure human judgment for risky actions.- Pre/post verification plus circuit-breakers stop harmful progress early and enable safe recovery.- Auditing and governance prevent abuse and enable continuous improvement.

Unlock Full Question Bank

Get access to hundreds of Incident Command and Leadership interview questions and detailed answers.

Join thousands of developers preparing for their dream job.