Incident Severity Classification and Escalation Questions
Assigning severity to incidents, deciding when and whom to page, and moving an issue up defined escalation paths. Covers severity matrices, escalation criteria, triage decisions, and scope-of-authority calls about when to pull in more responders or leadership. The routing and prioritization discipline that sits at the front of the incident lifecycle.
Design an incident escalation policy for an organization that currently has none. Cover detection, severity assignment, primary/secondary responders, who receives executive notifications and when, customer communication templates, concrete SLAs for the first 0-72 hours, and how to test and iterate the policy regularly.
Sample Answer
Requirements & constraints:
- Protect customer data & uptime; meet contractual SLAs; minimize MTTR; clear accountability; auditable logs.
High-level policy:
- Detection & Triage
- Sources: monitoring alerts (SRE), customer-reported tickets, automated health checks, sales/CS reports.
- On alert, on-call engineer must acknowledge within 5 minutes and perform triage within 15 minutes (confirm, gather scope, initial workaround).
- Severity Assignment (examples)
- Sev 1 (Critical): full service outage or data loss impacting >50% customers — immediate page 24/7.
- Sev 2 (High): major feature degraded impacting >10% — page business-hours on-call, escalate to 24/7 if unresolved in 30m.
- Sev 3 (Medium): partial degradation, workaround exists — ticket and notification to product owner within 4h.
- Sev 4 (Low): cosmetic or non-urgent — backlog.
- Responders & Escalation Matrix
- Primary responder: on-call engineer (SRE/Platform) — 0–30m actions.
- Secondary responder: engineering lead — escalate if unresolved at 30m (Sev1) or 2h (Sev2).
- Tertiary: Architect/Solutions Architect + Product + Legal — escalate at 2h (Sev1) or 8h (Sev2).
- Incident Commander (IC): assigned for Sev1/2 within 30m; rotates from senior SRE or Solutions Architect.
- Executive Notifications
- Sev1: Execs (CTO, VP Eng, Head of Sales, Head of Customer Success) notified at 30m and updated hourly until containment.
- Sev2: Execs notified at 4h if customer impact persists; summary at 24h.
- Notifications include impact, customer lists, mitigation, owner, ETA.
- Customer Communication Templates (short examples)
- Initial (within 60m for Sev1): “We’re investigating an outage affecting [product/region]. Impact: [short]. Next update by [time].”
- Follow-up: status, workaround, affected customers, ETA.
- Resolution: root cause summary, actions taken, mitigation, expected delivery for permanent fix.
- SLAs: First 0–72 hours (concrete)
- Acknowledge alert: <=5m
- Initial triage & customer notification (Sev1): <=60m
- Incident Commander assigned (Sev1/2): <=30m
- Containment (Sev1): <=4h (goal); if not contained, provide hourly updates
- Recovery (service restored) target: Sev1 <=12h, Sev2 <=48h
- Postmortem published: Sev1 within 72h draft, final within 14 days; Sev2 draft within 7 days.
- Testing, Runbooks & Iteration
- Maintain runbooks per service; automated playbooks for common failure modes.
- Quarterly tabletop exercises (simulate Sev1/Sev2) with execs, SRE, Sales, CS.
- Monthly on-call fire drills (inject alerts) to verify paging, acknowledgement SLAs.
- After-action: every real incident triggers blameless postmortem, assigned action items with owners and due dates; track metrics: MTTA, MTTR, customer impact, communication timeliness.
- Biannual policy review with stakeholders; update escalation thresholds, contact lists, templates.
Trade-offs & rationale:
- Fast acknowledgement prevents escalation delays; IC model centralizes decisions and communication. Hourly exec cadence balances information needs without noise. Testing cadence mixes frequency for readiness and lower overhead.
This policy can be adapted per customer contracts and scaled across regions/services.
Design an escalation path and SLA framework for customer issues that require SA involvement after deployment. Define severity levels, response and resolution targets per level, escalation owners, handoffs to engineering or customer success, and sample templates for customer communication at each stage.
Sample Answer
Requirements & goals:
- Rapid, predictable response for post-deployment issues requiring SA involvement
- Clear severity definitions, measurable SLA targets, named escalation owners, smooth handoffs to Eng/CS, and reusable customer communications.
Severity levels & SLAs:
- Sev 1 (Critical): production-down or major data loss; widespread customer impact.
- Response: 30 minutes
- Resolution target: 4 hours (workaround in 1 hour if full fix >4h)
- Owner: On-call SA + Incident Lead (Customer Success Eng + Eng on-call)
- Escalation: immediate paging to Eng on-call; daily exec updates
- Sev 2 (High): major degraded functionality for many users, no data loss.
- Response: 1 hour
- Resolution target: 8 business hours (workaround in 4 hours)
- Owner: Assigned SA; escalate to Eng within 2 hours if no triage
- Handoff: handover doc + recording to Eng; CS notified for customer coordination
- Sev 3 (Medium): single-user impact or non-urgent functional bug.
- Response: 4 business hours
- Resolution target: 3 business days
- Owner: SA; queued for engineering backlog with prioritization input from CS
- Sev 4 (Low): feature requests, documentation issues.
- Response: 1 business day
- Resolution target: next release/roadmap
- Owner: SA product/CS for roadmap discussions
Escalation path & handoffs:
- Triage by SA within Response SLA; create incident record (severity, impact, logs, steps to reproduce, workaround).
- If needs code change or deep debug, SA escalates to Eng on-call with a clear One-Pager: business impact, repro, logs, data sample, priority.
- CS handles customer comms and expectation setting; SA provides technical brief and status.
- For cross-team or executive escalation, Incident Lead coordinates weekly/daily updates and post-incident review.
Roles:
- SA: technical triage, workaround, handoff package
- Eng on-call: root cause, patch/fix
- Customer Success (CS): customer updates, SLA tracking, business priority
- Incident Lead (rotating): coordinates communications, exec briefings, RCA ownership
Sample customer templates (concise):
Initial Acknowledgement (within Response SLA)
"Hi [Customer], we received your report of [issue]. Severity: [Sev X]. Our SA is triaging and will respond within [response SLA]. We’ll provide next update by [time]. — [SA name]"
After Triage / Workaround
"Hi [Customer], triage complete. Cause: [summary]. Temporary workaround: [steps]. We’ve assigned this to [Eng/Team] for a permanent fix. Expected ETA for resolution: [X]. We’ll update by [time]. — [SA name], CC: [CS lead]"
Escalation to Engineering
"Hi [Customer], we’ve escalated to Engineering for a code fix due to [reason]. Impact: [customers affected]. Current mitigation: [workaround]. Target resolution: [ETA]. We will provide status every [frequency]. — [Incident Lead]"
Resolution & RCA
"Hi [Customer], incident resolved at [time]. Fix applied: [what]. Root cause: [summary]. Preventive actions: [steps]. We will share full RCA and timeline for any follow-up by [date]. Thank you for patience. — [SA], [CS]"
Post-incident follow-up
"Attached: RCA, timeline, and planned preventive measures. If you want a technical review session, we can schedule one."
Implementation notes:
- Use an incident tracker (Jira/ServiceNow) with severity SLA automation and escalation policies.
- Maintain runbooks per common failures, and a post-incident retrospective within 5 business days.
- Measure KPIs: SLA adherence, MTTR, number of Sev1s, CS satisfaction after incidents.
This framework balances fast customer-facing action by CS, deep technical resolution by SA/Eng, and measurable SLAs to set expectations.
Explain the typical responsibilities of a junior Solutions Architect for incidents across severity levels P1 (critical) to P4 (low). For each severity include expected response time, primary actions, who to notify, and required documentation or post-incident tasks.
Sample Answer
P1 — Critical (service down / data loss / major customer impact)
- Expected response time: Immediate (15 minutes) and join incident bridge.
- Primary actions: Triage to identify scope and rollback/mitigation steps; apply hotfix or failover; gather logs/metrics; execute documented runbook steps; stabilize system.
- Who to notify: Incident manager, engineering on-call, product owner, account executive, Sales/Client success for affected customers, senior architect/manager.
- Documentation / post-incident: Create incident record with timeline, root cause hypothesis, actions taken, customer impact; run blameless postmortem within 48–72 hours with RCA, corrective actions, and SLA credit determinations; update runbooks and architecture if needed.
P2 — High (partial outage / degraded functionality for many)
- Expected response time: 30–60 minutes.
- Primary actions: Assess degradation, implement mitigations (rate limits, graceful degradation, targeted restarts), collect diagnostics, monitor impact.
- Who to notify: Engineering lead, product owner, account rep, on-call support.
- Documentation / post-incident: Incident summary in ticketing system, RCA within 5 business days if recurring or impactful, update monitoring/alerts and playbooks.
P3 — Medium (single customer impact or non-critical feature broken)
- Expected response time: 4 hours.
- Primary actions: Reproduce issue, provide workaround to customer, schedule fix with engineering, gather logs for troubleshooting.
- Who to notify: Assigned engineer, account owner, support engineer.
- Documentation / post-incident: Ticket with troubleshooting steps, timeline, planned remediation ETA; include in weekly review if trend observed.
P4 — Low (cosmetic, minor config issue, feature request)
- Expected response time: 24–72 hours (triage during business day).
- Primary actions: Log issue, prioritize with product backlog, provide guidance or documentation to customer.
- Who to notify: Product manager and support owner as appropriate.
- Documentation / post-incident: Update backlog/user story, include notes in architecture docs if change required.
General notes:
- Always follow SLAs and escalation paths; communicate status regularly to stakeholders; ensure evidence (logs, screenshots, timestamps) is attached to every incident record.
Propose a prioritization framework you would present to executive stakeholders that combines SLO breaches, revenue impact, user count affected, regulatory risk, and time-to-fix for triaging incidents. Provide a short formula or scoring approach and explain trade-offs.
Sample Answer
Situation: Executive stakeholders need a clear, repeatable way to triage incidents that balances technical reliability (SLO breaches) with business risk (revenue, users, regulatory) and operational cost (time-to-fix). I propose a weighted scoring framework that produces a single "Priority Score" and maps to action SLAs.
Scoring approach (normalize each factor 0–100, higher = worse):
- SLO breach severity (S) = percent deviation from target mapped to 0–100
- Revenue impact (R) = estimated hourly revenue loss mapped to 0–100 (log-scale)
- Users affected (U) = percent of active users affected mapped to 0–100
- Regulatory risk (G) = categorical: None=0, Low=50, High=100
- Time-to-fix (T) = estimated MTTR in hours mapped inversely: shorter T → higher urgency if impact present (map to 0–100 where small T keeps score high to prioritize quick interventions)
Priority Score = 0.35S + 0.25R + 0.20U + 0.15G + 0.05*T
Rationale:
- Weight S highest because recurring SLO breaches degrade customer trust and long-term business value.
- Revenue and user impact follow; regulatory risk is critical but rarer, so a strong categorical boost covers compliance emergencies.
- Time-to-fix is low weight to avoid penalizing hard-to-fix legacy issues; it nudges towards fast wins.
Thresholds → Actions:
- 80–100: Immediate executive alert, full incident response, customer communications
- 60–79: High priority engineering focus, mitigation plan within 1 hour
- 30–59: Investigate and schedule remediation within sprint
- <30: Monitor and backlog
Trade-offs and considerations:
- Simplicity vs precision: fewer weights keeps it explainable to execs; customize weights per product line if revenue-critical.
- Estimation noise: revenue and user estimates may be rough; use ranges and conservative assumptions.
- Gaming risk: guardrails required so teams don’t under-report S to de-prioritize work.
- Periodic calibration: review weights and mappings monthly using incident postmortems and business outcomes.
This framework gives executives a transparent, business-aligned triage score while preserving operational nuance for engineering.
How would you define incident severity levels (P1–P4) for a multi-tenant SaaS product? Specify concrete criteria (impact, users affected, SLA exposure) and who should be alerted at each level.
Sample Answer
I define P1–P4 by concrete, operational criteria (impact, scope, SLA risk) and clear alerting/playbook ownership so responses scale predictably across tenants.
P1 — SEVERE (Business-critical outage)
- Criteria: Complete service down or data corruption affecting all tenants or a single major tenant; core functionality unavailable; SLA/commitment breached or imminent; revenue/contract risk; security breach with exfiltration.
- Users affected: ~100% or a single paying tenant with mission-critical SLAs.
- SLA exposure: Immediate/ongoing breach.
- Alerting/owners: All-hands page + on-call SRE, incident commander (senior SRE/engineering manager), Solutions Architect on-call, Product PM, Sales/Account Executive for impacted customers, Legal/Compliance and Exec on-call if security/contract at risk.
P2 — HIGH (Major degradation)
- Criteria: Significant feature degraded for many tenants or complete outage for a small subset; workaround exists but causes operational pain; potential SLA violation within hours.
- Users affected: Multiple tenants or a single paid tenant non-critical.
- SLA exposure: Likely if not resolved in hours.
- Alerting/owners: On-call SRE, engineering lead, Solutions Architect, Product PM, designated Sales rep for affected customers.
P3 — MEDIUM (Partial impairment)
- Criteria: Non-critical feature failures, performance degradation affecting limited users, intermittent errors; no immediate SLA breach.
- Users affected: Small subset of users/tenants.
- SLA exposure: Unlikely.
- Alerting/owners: Team on-call, engineering owner, Solutions Architect for technical guidance; Customer Success informed if reported by customers.
P4 — LOW (Minor / Planned)
- Criteria: Cosmetic bugs, minor performance variance, documentation/UX issues, planned maintenance.
- Users affected: Minor or single-user impact; no SLA risk.
- Alerting/owners: Triage by product/engineering during normal hours; Solutions Architect for consult as needed; tracked in backlog.
Notes / best practices:
- Use measurable thresholds (error-rate > X%, latency > Y ms, % tenants affected) mapped to each level.
- Automate detection/alerting and include runbooks per severity with escalation timelines.
- For multi-tenant, always state tenant scope in incident title and prioritize compensations/communication per SLA tiers.
That is every published Incident Severity Classification and Escalation question for Solutions Architect so far. Browse the other topics in this category, or practice this one interactively.