Covers the design and operation of incident response programs and the creation and maintenance of actionable runbooks and playbooks for production systems. Candidates should be able to explain the incident lifecycle from detection and classification through investigation, escalation, remediation, and post incident analysis. Topics include severity definitions and assessment, escalation procedures, team roles and responsibilities, communication protocols during incidents, on call rotations, alert triage, and coordination across teams during outages. Also includes designing automated remediation steps where appropriate, integrating runbooks with monitoring and alerting systems, maintaining playbooks for common failure modes such as malware, data exfiltration, denial of service, and account compromise, and conducting blameless post incident reviews and continuous improvement. Candidates should be able to discuss metrics for measuring response effectiveness such as mean time to detect, mean time to repair, and response success rate, and describe approaches to improve those metrics over time.
MediumTechnical
71 practiced
You detect a rise in failed login attempts indicative of account compromise attempts. As a Solutions Architect, propose a playbook that covers detection and triage, immediate containment actions (e.g., rate-limit, lock, require MFA), user notification messaging, decision criteria for forced password reset versus monitored lockout, and coordination with security and customer support teams.
Sample Answer
Summary: A pragmatic, repeatable playbook to detect, triage, contain, notify, and coordinate when we observe increased failed-login attempts suggesting credential stuffing or brute force.Detection & Triage- Alert triggers: spike in failed auth rate > X baseline (e.g., 5x 15-min rolling avg), IP reputation hits, multiple usernames from same IP/subnet, login attempts across many geographic regions.- Initial enrichment: collect IPs, user IDs, user agents, device fingerprints, timestamps; run IPs through threat intel (TOR, VPN, known botnets); correlate with recent password reset or MFA changes.- Triage severity (Low/Medium/High) based on volume, breadth (many accounts) and evidence of successful compromises.Immediate Containment (automated + manual)- Automated controls (within 60s): - Apply progressive rate-limiting per IP and per account (exponential backoff). - Block known bad IPs/subnets and fingerprinted bots at WAF/CDN. - Require step-up: challenge with CAPTCHA or device fingerprinting.- Manual / high-severity actions: - Temporarily lock accounts with confirmed successful logins from suspicious vectors. - Force MFA requirement for affected accounts (if not enabled). - For widespread attack, enable global stricter throttling windows.User Notification Messaging- Tiered templates: - Informational (failed attempts, no evidence of breach): “We detected repeated sign-in attempts to your account. No unusual activity detected. Consider enabling MFA.” - Action required (suspicious activity or locked): “We temporarily locked your account after suspicious sign-in attempts. To regain access, complete account verification and a password reset. We recommend enabling MFA.” - Security incident (confirmed compromise): “We detected unauthorized access to your account. We have locked the account and require a password reset + MFA. Contact support if you see unfamiliar activity.”- Include clear, short steps, support contact, and reassurance about data protection.Decision Criteria: Force Password Reset vs Monitored Lockout- Force password reset when: - Evidence of successful unauthorized login (session created, changes to profile, transactions). - Attack used credential stuffing across many accounts with confirmed compromises. - Account is high-value (financial/admin) or contains sensitive PII.- Monitored lockout when: - High volume of failed attempts but no successful login and no suspicious post-login activity. - Low-risk account; use escalation to require password change if attempts persist.- Define TTLs: temporary lockouts auto-expire after X hours with soft verification; forced resets require immediate user action.Coordination & Escalation- Security Ops: handoff ticket with enriched data, recommended containment, and IOCs to block. Run forensic sessions for confirmed compromises.- Customer Support: provide playbook scripts, notification templates, verification flows, and escalation path for irate or compromised users.- Legal/PR: escalate if widescale leakage or regulatory impact.- Post-incident: update IAM rules, blocklists, device fingerprint rules, and run targeted MFA push for affected cohorts.Metrics & Post-mortem- Track mean time to detect/contain, % accounts forced-reset, false positives, customer impact.- Conduct blameless postmortem and update thresholds, notification wording, and automation playbooks based on findings.
MediumTechnical
75 practiced
You are tasked with reducing MTTR by 30% in the next quarter for a set of critical services. As a Solutions Architect, propose a prioritized plan with five concrete initiatives spanning people, process, and technology (e.g., instrumentation changes, runbook improvements, automation). For each initiative, describe expected impact and how you would measure success.
Sample Answer
Goal: reduce MTTR by 30% in one quarter for critical services. Prioritized plan (highest to lower priority) with expected impact and success metrics:1) Prioritized alert triage + runbook rewrites (Process / People)- Action: audit top 10 alert types, simplify to actionable alerts and attach concise runbooks (symptoms → diagnosis → remediation steps).- Impact: faster decision-making; fewer escalations.- Measure: MTTR per alert type, % of incidents resolved by first responder, time-to-first-action. Success: 30% reduction in MTTR for incidents triggered by those alerts; >60% resolved by first responder.2) Instrumentation improvements & SLO-based alerting (Technology)- Action: add high-cardinality traces and key business metrics; move from threshold to SLO/burn-rate alerts; tag traces with deployment/service metadata.- Impact: earlier and more precise detection, fewer noisy alerts.- Measure: MTTD, false-alert rate, alert-to-incident ratio. Success: MTTD drops by 20%, false-alerts down >40%.3) Runbook automation & one-click remediation (Automation / Tech)- Action: convert frequent manual remediations to safe automated playbooks (feature flags, autoscale, circuit breaker resets) triggered from pager or runbook UI.- Impact: eliminate manual toil and reduce human error/time.- Measure: % incidents using automation, median remediation time for automated steps. Success: automation used in ≥25% incidents and reduces their MTTR by 50%.4) War-room drills + on-call enablement (People / Process)- Action: weekly 60–90 min blameless tabletop drills for top incident types; update on-call handbook and escalation matrix.- Impact: improves muscle memory, faster coordination.- Measure: time-to-first-action in drills vs baseline, post-drill survey confidence. Success: 15–25% faster coordination metrics and improved confidence scores.5) Post-incident analytics & continuous improvement loop (Process / Tech)- Action: enforce SLA for postmortems with RCA, define corrective tasks, track through backlog; instrument dashboards to show MTTR trends.- Impact: systemic root causes fixed, prevents repeat incidents.- Measure: % action items completed within sprint, recurrence rate of same incident, dashboarded MTTR trend. Success: 80% of corrective actions completed within 4 weeks and recurrence down by 50%.Rollout plan: 0–2 weeks audit/priority, 2–6 weeks implement instrumentation and runbook rewrites, 4–10 weeks deploy automation and run drill cadence, continuous postmortem enforcement. Monitor weekly; adjust based on metrics.
MediumTechnical
68 practiced
Scenario: The payments service is intermittently returning 500 errors for ~7% of transactions across multiple regions and customer impact is trending up. As a Solutions Architect, outline the incident response plan: initial triage steps to determine scope and root cause, teams and SMEs to involve, which runbook sections to follow, immediate mitigations to reduce customer impact, and longer-term fixes to prioritize.
Sample Answer
Situation: Production payments service is returning intermittent 500s for ~7% of transactions across regions and customer impact is increasing.Initial triage (first 15–30 min)- Verify alerts and scope: confirm error rate, affected regions, payment types, time windows via monitoring (Datadog/Prometheus), logs, and APM traces.- Identify blast radius: % of users, merchant tiers, failover paths, payment provider involvement.- Check recent changes: deployments, config changes, infra autoscaling, certificates, third-party API changes, feature flags.- Reproduce: run synthetic transactions against representative endpoints and capture full request/response and correlation IDs.Teams & SMEs to involve immediately- On-call SRE/Platform engineer (lead)- Payments backend owner / service owner- API gateway/load-balancer engineer- Database specialist (if persistence involved)- Third-party payments provider/processor SME- Security (if auth/crypto issues suspected)- Communications: Incident commander + Product/Support for customer messagingRunbook sections to follow- Incident initiation and severity classification- Escalation matrix and contact list- Monitoring and diagnostics checklist (logs, traces, metrics)- Mitigation playbooks (circuit breakers, rate-limiting, failover)- Rollback and deployment rollback procedures- Post-incident RCA and retrospective templateImmediate mitigations to reduce customer impact (minutes–hours)- Enable error-rate circuit breaker to stop retry storms and protect downstream systems.- Route new traffic to healthy regions or fallback payment processor if available.- Disable recent risky feature flags or rollback recent deployment to last known-good.- Apply temporary rate-limits or priority routing for high-value merchants.- Return graceful failures with useful error codes/messages and guidance to customers/support.Short-term containment (hours)- Increase logging/trace sampling for affected flows, add targeted debug logs.- Throttle/backpressure non-critical background jobs.- Coordinate with third-party provider to confirm their health and apply agreed mitigations.Longer-term fixes to prioritize (days–weeks)- Root-cause corrective changes (code fixes, config fixes) with tests.- Improve observability: add end-to-end tracing, business-metric SLOs on payment success rate, and better dashboards/alerts for early detection.- Harden resilience: idempotency, retries with exponential backoff, bulkhead and circuit-breaker patterns, multi-processor failover.- Release pipeline safety: add canary deployments, feature-flag staging, automated rollback.- Capacity & chaos testing: load tests and scheduled chaos experiments for payment paths.- Postmortem & action items with owners, deadlines, and customer remediation plan if needed.Communication- Incident commander: provide status updates cadence (e.g., every 30 min) to stakeholders and public status page.- Support/CS: share temporary workarounds, impacted scope, and ETA.- After resolution: publish RCA, impact metrics, and timeline of fixes.This plan balances rapid containment to protect customers with structured diagnostics to find and fix root cause while ensuring clear ownership and communication.
HardTechnical
62 practiced
You deployed a large-scale schema change affecting billions of rows that now causes production outages. Design a safe rollback or mitigation strategy: include options for logical rollbacks, partial rollbacks, feature flags, dual-read strategies, compensating migrations, rehearse steps, required tools, and communication with downstream systems and customers.
Sample Answer
Situation: A large schema migration on billions of rows is causing production outages. Below is a practical, prioritized rollback and mitigation strategy a Solutions Architect would lead.Immediate containment (first 30–60 mins)- Pause deployments and stop any jobs that write to affected tables.- Put affected services into degraded mode via global feature flag (read-only where possible).- Throttle traffic using load balancer or circuit-breaker rules.- Notify SRE, DBAs, product, and on-call owners.Rollback options (choose based on situation & SLA)1. Logical rollback (fastest when available)- Apply the inverse DDL/DML that reverses schema change (e.g., rename column back, drop new index) using online DDL tools.- Use transactional scripts and smaller batched statements to avoid long locks.2. Partial rollback / mitigation- Revert only problem pieces (e.g., drop problematic index, remove constraint) while keeping safe parts of migration.- Add compatibility views or columns to map old API to new schema.3. Dual-read / dual-write strategy (short-term)- Re-route reads to a stable replica that still serves the previous schema via a compatibility layer or a read adapter service.- For writes, write to both old and new formats (idempotent), with background reconciliation.4. Compensating migrations (data-level fixes)- Run batched compensating jobs that correct corrupted rows, using idempotent transactions and rate limiters.Rehearse, safety and tooling- Maintain pre-approved rollback playbooks stored in runbooks with exact SQL and time estimates.- Use online-DDL tools (pt-online-schema-change, gh-ost, native DB online alter), change-data-capture (Debezium/Kafka) for sync, and feature-flag platforms (LaunchDarkly, Flagsmith).- Use migration orchestration (Flyway/liquibase + staged scripts) and automated canary tests.Execution steps for a full logical rollback1. Validate backup/RPO — ensure recent consistent backup and binlogs available.2. Quiesce writes and redirect to maintenance mode.3. Execute rollback DDL in small batches on replicas; promote once validated.4. Replay binlogs if needed or run reconciliation jobs for missed writes.5. Run comprehensive smoke tests and canaries on replicas before restoring traffic.6. Gradually reinstate traffic (circuit-breaker increase), monitor metrics and errors.Communication plan- Immediate incident bulletin: impact, scope, mitigation actions, expected ETA.- Regular updates every 15–30 minutes to stakeholders and downstream integrators.- Post-incident: detailed RCA, data integrity verification, client-facing summary and compensation if needed.Key safeguards & lessons- Always stage schema changes behind feature flags and backwards-compatible design (expand-contract pattern).- Test migrations on production-scale clones and rehearse rollback playbooks.- Automate observability: schema-change dashboards, slow-query and constraint-failure alerts, and data drift monitors.This approach balances speed (containment) with safety (batched, reversible steps), preserves data integrity, and maintains clear communications with engineering and customers.
EasyTechnical
59 practiced
Explain the difference between Mean Time To Detect (MTTD), Mean Time To Acknowledge (MTTA), and Mean Time To Repair (MTTR). Given this timeline: alert fired at 10:05, first human acknowledgment at 10:10, remediation started at 10:20, service restored at 11:05 — calculate MTTD, MTTA, and MTTR and explain any assumptions.
Sample Answer
MTTD, MTTA and MTTR measure different parts of the incident lifecycle:- MTTD (Mean Time To Detect): average time from incident start to when it is detected (alerted). It measures observability/detection capability.- MTTA (Mean Time To Acknowledge): average time from detection/alert to the first human acknowledgement. It measures alerting and on-call responsiveness.- MTTR (Mean Time To Repair): average time from the start of remediation (or sometimes from acknowledgement) to service restoration. It measures how long it takes to fix and recover.From the timeline:- Alert fired: 10:05- First human ack: 10:10- Remediation started: 10:20- Service restored: 11:05Assumptions: treat alert time as detection time (incident start = alert); treat remediation start as the start of repair.Calculations:- MTTD = detection time − incident start = 0 minutes (alert = detection)- MTTA = ack − alert = 10:10 − 10:05 = 5 minutes- MTTR = restore − remediation start = 11:05 − 10:20 = 45 minutesNote: If your org defines MTTR from acknowledgement, MTTR would be 11:05 − 10:10 = 55 minutes. Be explicit about definitions when reporting metrics. As a Solutions Architect, I recommend documenting definitions, instrumenting timestamps at key lifecycle events, and tracking these metrics per service to drive improvements in monitoring, on-call routing, and runbooks.
Unlock Full Question Bank
Get access to hundreds of Incident Response and Runbook Design interview questions and detailed answers.
