Incident Classification and Severity Questions

Focuses on structured approaches to classifying incidents and assigning severity levels to drive appropriate response, escalation, and communication. Covers defining severity criteria based on customer impact, affected services, scope of impact, and regulatory concerns, mapping severity to response playbooks and on call rotations, establishing escalation paths and communication cadences, defining service level objectives and response time targets, coordinating cross functional responders, and creating runbooks and automated tooling to enforce the framework. Also includes governance topics such as reviewing and refining severity definitions from post incident analyses, training responders on the framework, and adjusting thresholds to reduce false positives and ensure consistent prioritization.

MediumTechnical

37 practiced

Describe how you would map incident severities to SLO error budget policy actions. Provide concrete policy examples such as: if error budget burn > 50% in 24 hours then pause non-essential deploys and trigger an on-call incident review (Sev-1); if error budget burn between 20-50% then increase monitoring and require pre-merge checks (Sev-2). Explain how these mappings help balance velocity and safety.

Sample Answer

Start by defining measurable signals and thresholds: error-budget burn rate (EBBR) over a rolling window (e.g., 24h), current remaining error budget, and incident count/severity. Map ranges of EBBR to discrete policy actions tied to deploy controls, monitoring, and human escalation.

Concrete policy examples:- Sev-1 (EBBR > 50% in 24h OR remaining budget < 10%): immediate safety posture - Pause all non-essential/deferred deployments (automated CI/CD gate) - Trigger on-call incident review and war room within 15 minutes - Enable expanded tracing and debug-level logs for affected services - Block automatic roll-forward of feature flags; require rollback plan- Sev-2 (EBBR 20–50% in 24h): elevated caution - Require pre-merge CI checks to include new load and integration tests - Increase dashboarding cadence and anomaly alerts; add temporary synthetic tests - Restrict canary sizes and require manual approval for production rollout - Schedule a focused reliability review with engineering owners within 24 hours- Sev-3 (EBBR 5–20% in 24h): watchful - Tighten alert thresholds, add short-term throttling or rate-limits - Notify product and release managers; require deployment postmortem for any failed deploys- Normal (EBBR < 5%): normal operations, standard SLO guardrails

Automation and tooling:- Implement policy engine that reads EBBR + SLO state and enforces CI/CD gates (e.g., pause pipelines, adjust canary config)- Integrate with alerting/on-call tools to auto-create incidents and route to appropriate responders- Record every enforced action for audit and postmortem correlation

Why this balances velocity and safety:- Graduated responses preserve developer velocity when risk is low (only lightweight checks). As risk rises, controls tighten incrementally—blocking risky operations only when necessary—so teams aren’t overly constrained but systemic instability triggers immediate safety actions.- Automating enforcement removes friction and ambiguity, enabling fast, consistent decisions and faster recovery while making the cost of reliability visible (error budget) to product and engineering stakeholders.

MediumSystem Design

52 practiced

Design a severity classification rubric for a microservices-based e-commerce platform handling approximately 100M requests/day. Define four severity levels and, for each level, supply measurable criteria: 5xx error rate threshold, p99 latency threshold, percent of total revenue affected, number of orders failing per minute, and an example business impact. Explain how you would validate and iterate on these thresholds after launch.

HardSystem Design

31 practiced

Design an SLA-driven automatic mitigation system where the declaration of a Sev-0 triggers pre-approved mitigations (traffic-shift, circuit-breakers, emergency throttles) automatically unless a human veto is issued within 2 minutes. Address safety and authorization (who can pre-approve mitigations), audit and rollback, testing and canarying, and failure modes (partial mitigation, mitigation causing new failures).

Sample Answer

Requirements:- Automatic execution of pre-approved mitigations when a Sev-0 is declared, unless vetoed within 2 minutes.- Mitigations: traffic-shift, circuit-breaker, emergency throttles.- Strong safety, authorization, auditable actions, testability, rollback, and robust handling of failure modes.

High-level design:- Incident Orchestrator (IO): listens to Sev-0 declarations (from pager, monitoring, or human), starts a 2-minute veto timer, and triggers mitigation playbooks if no veto.- Mitigation Engine (ME): executes atomic mitigation steps via adapters (LB API, service mesh, rate-limiters).- Policy Store: stores pre-approved mitigations per service with required approvers, scoped limits, canary configs, and TTLs.- Authz & Approval Service: enforces who may pre-approve and who may veto; integrates with RBAC/SCIM and requires 2FA for high-impact approvals.- Audit & Rollback Service: logs actions, stores pre/post state snapshots, and supports automated rollback runbooks.- Canary Runner & Safety Guard: executes mitigations incrementally (canary percentage, health probes); aborts and rolls back if health degrades beyond threshold.

Key flows:1. Sev-0 declared → IO reads Policy → starts 2-minute veto window and notifies approvers.2. If no veto → IO instructs ME to run Canary Runner with pre-defined steps and probes.3. On success → scale mitigation to target; on failure → automatic rollback + escalate.

Safety & Authorization:- Pre-approval requires combination of role + signed policy artifact; critical mitigations need multi-approver (two-person) sign-off.- Veto allowed to on-call SREs and service owner; veto action is authenticated and audited.

Audit & Rollback:- All API calls are immutably logged (who, what, when, diff).- ME snapshots config/state before changes; rollback playbook can be automated or manual depending on policy.- Post-incident report generated automatically.

Testing & Canarying:- Every mitigation has a canary plan: initial small percent, health probes (latency, error-rate, user journeys), time window and step increments.- Staging rehearsals (chaos/DR drills) run weekly; synthetic traffic validates rollback.

Failure modes & mitigations:- Partial mitigation (only some replicas updated): use idempotent, versioned config changes and state verification; retry with exponential backoff.- Mitigation causes new failures: Safety Guard monitors probes and reverses changes automatically; escalate to humans if rollback fails.- Orchestrator failure: ME has independent fallback (local abort/rollback) and uses leader-election for IO HA.- Authorization compromise: emergency killswitch requires multi-party offline confirmation.

Trade-offs:- Faster automatic mitigation improves MTTR but increases risk of false-positive actions—mitigated with short veto window, canarying, and multi-approver for high-impact actions.- Complexity vs safety: invest in automation, testing, and audit to keep trust.

Metrics:- MTTR, false-mitigation rate, rollback rate, canary pass/fail rate, time-to-veto, and audit completeness.

MediumTechnical

51 practiced

You notice a recurring pattern where brief traffic spikes from noisy clients cause Sev-1 alerts. Outline a plan to reduce these false positives while ensuring real incidents are still reliably surfaced. Include immediate mitigations (cooldowns, client throttling), metric smoothing or aggregation, topology-aware alerting (per-customer vs global), and longer-term architectural changes to reduce blast radius.

Sample Answer

Situation: Recurrent brief traffic spikes from noisy clients are triggering Sev-1 alerts, overwhelming on-call and masking real incidents.

Plan (immediate → medium → long-term):

Immediate mitigations (hours):- Cooldowns: add short alert for the symptom with for: 5m in Prometheus so momentary blips don't surface as Sev‑1.- Client throttling / circuit-breakers: deploy rate-limits at edge (API gateway / LB) to smooth noisy clients and return 429s rather than causing downstream saturation.- Auto‑mute noisy sources temporarily while we investigate (alert suppression based on client ID).

Example alert tweak (Prometheus):

yaml

- alert: HighErrorRate
  expr: sum by(customer)(rate(request_errors_total[1m])) / sum by(customer)(rate(request_total[1m])) > 0.05
  for: 5m
  labels:
    severity: page

Metric smoothing / aggregation (days):- Use rate() or increase() with longer windows (1m→5m) and add p50/p95 baselines to avoid reacting to spikes.- Aggregate per-customer and global; compute anomaly score vs recent baseline (rolling 1h) before escalating.

Topology-aware alerting:- Split alerts: per-customer (exhaustion of single tenant) → severity page only for high-value tenants or when >N customers affected; global aggregated alerts trigger Sev‑1.- Add cardinality checks: only escalate global alert if the number of impacted customers > threshold.

Long-term architectural changes (weeks→months):- Improve isolation: per-tenant queues, worker pools, or rate-limiters to reduce blast radius.- Backpressure and graceful degradation: prioritized traffic paths, feature flags to shed non-critical load.- Autoscaling policies tuned for transient bursts (scale based on sustained load, not instantaneous spikes).- Invest in anomaly detection (ML) to reduce false positives and enrich alerts with customer/context data.

Validation and process:- Run canary of alert changes, monitor missed incident risk for 2–4 weeks.- Update runbooks to reflect per-customer logic.- Postmortem after any missed incident; iterate thresholds and isolation mechanisms.

This approach balances immediate noise reduction with topology-aware alerting and longer-term isolation to keep real incidents visible while minimizing false positives.

MediumSystem Design

30 practiced

Design an automated runbook execution flow for a common Sev-2 incident: worker queue backlog leading to processing delays. Describe: trigger conditions, pre-execution safety checks (idempotence, approvals), automated remediation steps (scale workers, retry policies), verification steps, rollback hooks, and human-in-the-loop decision points. Show how to abort automation safely if post-checks fail.

Sample Answer

Requirements & goal:- Detect and remediate worker-queue backlog (Sev-2) to restore processing latency within SLOs while avoiding cascading failures or double-processing.

Trigger conditions:- Alert when: queue length > threshold T (e.g., 95th percentile backlog > X messages for >5 min) OR processing rate drops by >Y% and queue depth growth rate >Z/min.- Verify correlated signals: worker CPU/memory, error rates, downstream service latency.

Pre-execution safety checks:- Idempotence: ensure remediation actions (scale, requeue) are idempotent or guarded by runbook state tokens (operation-id).- Check for in-flight dedup keys / visibility timeout semantics to avoid double-processing.- Approvals: automated runbook runs up to a safe tier (e.g., auto-scale up to N workers). For higher-impact actions (drain queues, purge messages, change visibility timeout > threshold), require 1-click approval from on-call via Slack/PagerDuty.

Automated remediation steps (ordered, with backoff):1. Throttle producers (soft) if supported.2. Increase consumer concurrency: trigger auto-scale of worker pool + update autoscaler min replicas. Use incremental steps (e.g., +20% up to cap), wait 60s between steps.3. Adjust transient retry and visibility timeouts to reduce lock contention (bounded change).4. Re-enqueue only messages in error-state after dedup verification; do NOT mass-purge.5. If worker errors spike, roll back scale and switch to "debug mode" (enable verbose logs) and notify humans.

Verification:- Post-step checks after each action: queue depth trending down for 3 consecutive intervals, processing rate restored, error rate within baseline.- Run synthetic consumer/test message through whole path to ensure end-to-end processing.

Rollback hooks and abort:- Define preconditions for rollback: if after an action the error rate > threshold OR duplicate-processing detected OR downstream SLA violated.- Rollback actions: revert scaling to previous replica count, restore previous timeout/retry settings, re-enable producers.- Abort automation safely: before destructive actions set a “safety window” (e.g., 5 minutes) during which automation pauses and requires manual abort/confirm. If post-checks fail, automation runs rollback hooks automatically and posts an incident update with root-cause indicators and recommended manual steps.

Human-in-the-loop decision points:- Approve escalations beyond automated caps.- Decide to divert traffic or perform message-level remediation (inspect & requeue).- Confirm mass purge or schema-affecting fixes.

Auditability & observability:- Log every step with operation-id in central store; emit metrics for action success/failure and duration; provide a single runbook UI showing state, diffs, and one-click approve/abort.

Safety principles:- Prefer incremental, reversible actions.- Never mass-delete/reprocess without human approval.- Keep idempotence and strong telemetry to detect duplicates and side effects quickly.

Unlock Full Question Bank

Get access to hundreds of Incident Classification and Severity interview questions and detailed answers.

Join thousands of developers preparing for their dream job.