Incident Severity Classification and Escalation Questions

Severity: Sev-1 (Critical)

Justification (measurable criteria):

• Scope: 7% HTTP 500 rate in eu-west-1 over 8 minutes with 120 distinct user-facing checkout failures → large fraction of active transactions likely impacted (if baseline traffic is moderate/high, this represents many blocked purchases).
• Business impact: Payments failing = direct revenue loss and poor user experience; 120 reports in 8 minutes indicates escalation risk and potential revenue/severity breach of SLOs for checkout success.
• Data loss/risk: NullPointer in payment processor suggests logic-level failure — low risk of data corruption but high risk of double-charges or incomplete orders if retries are mishandled; must verify idempotency and transactionatomicity.

Immediate three actions (first 15 minutes):

1. Mitigate: Roll back or disable the recent deploy(s)/feature flags in eu-west-1 affecting payments (or divert traffic) to restore success rate — aim to reduce 500s immediately.
2. Triage & gather evidence: Attach logs and span traces for failed requests, capture sample request IDs/user IDs, stacktrace of NullPointer, recent code/config changes and deploy timestamps; mark incident and notify on-call backend/payment engineer and product owner.
3. Protect customers & prevent data issues: Pause automated retries that could cause duplicate charges, put a temporary error page explaining issue for checkout if needed, and open a safe manual queue for failed payments to reconcile once fixed.

Requirements & constraints:

• Scale: 50+ independently owned services
• Goal: enforce error budgets & severity escalation with minimal friction
• Non-functional: low-latency decisions, tamper-resistant metrics, auditable exceptions

High-level design:

1. SLO/Error-Budget Platform (centralized control plane)

   • Stores SLO definitions, SLIs, rolling window, and policy rules per service
   • Integrates with metrics backend (Prometheus/Thanos, BigQuery) and CI/CD systems (GitHub Actions, Jenkins)
   • Exposes REST/gRPC API and UI for owners

2. Automated enforcement engine

   • Continuously evaluates SLIs against SLOs (e.g., 30d window, 5m data freshness)
   • On threshold crossings:
     • Soft action at warning (notify owners, create incident, mute noisy alerts)
     • Hard action at depletion (automated gates):
       • Pause CD pipelines via CI/CD API (set branch protection, block merge PRs)
       • Post a status check failure referencing error-budget ticket
       • Trigger canary rollback automation if recent deploy correlates with spike
   • Actions are idempotent and timeboxed; automatic re-evaluation to resume when budget recovers

3. Human approval & exceptions flow

   • Self-serve exception requests via platform: owner provides justification, rollback/mitigation plan, risk window
   • Two-tiered approvers:
     • Service-level approver (on-call/product owner) can grant short-lived exceptions
     • Reliability Review Board (cross-functional SRE + Product) for multi-hour/critical exceptions
   • All exceptions recorded with TTL, audit trail, and required postmortem

4. Data freshness & accuracy guarantees

   • Ingest pipeline enforces max-staleness (e.g., 5m) with watermarking
   • Use two independent signal paths: primary (Prometheus) and validation path (sampling into long-term store)
   • SLA on metric freshness (SLO for observability): alert when telemetry lag > threshold
   • Recompute error budget deterministically from raw samples; store immutable evaluation events for audit

5. Detecting & preventing metric gaming

   • Require canonical SLIs with deterministic aggregation (avoid owner-defined ad-hoc counters)
   • Cross-validate SLI with secondary signals (e.g., user-facing latency vs internal success-rate)
   • Anomaly-detection on SLI patterns to detect injects/spikes inconsistent with traffic patterns
   • Rate-limit synthetic test traffic; mark synthetic signals and exclude or annotate them
   • Require code-reviewed SLI changes and gated rollout (feature flag + canary) — SLO platform rejects SLI schema changes without review

6. Organizational model

   • Federated ownership: teams define SLOs for their services within enterprise guardrails (minimum SLO, severity classifications)
   • Central SRE reliability guild:
     • Maintains platform, enforcement policies, audit, incident escalation playbooks
     • Reviews exceptions and runs quarterly reliability reviews
   • Reliability KPIs in engineering performance reviews (shared accountability)
   • Monthly cross-team reliability forum to align priorities and evolve guardrails

Trade-offs & rationale:

• Centralized enforcement preserves enterprise reliability while giving teams autonomy to set realistic SLOs.
• Automated brakes reduce blast radius but are reversible with clear human approval paths to avoid blocking delivery indefinitely.
• Multi-path telemetry and audit trails reduce risk of gaming and ensure decisions are defensible.

Implementation notes:

• Use policy-as-code (Open Policy Agent) to encode gating rules; integrate webhook-based CI/CD enforcement.
• Store audit & evaluation events in immutable log (e.g., Bigtable/BigQuery + signed entries).
• Start with conservative automations (notify → soft block → hard block) and expand as trust grows.

Approach: maintain an in-memory, time-indexed map that groups alerts by (service, root_cause) inside a 5-minute sliding window. For each group keep recent alerts (timestamped), aggregated severity (e.g., max or weighted average), count, and last-seen timestamp. Periodically (or on write) evict >5m old alerts and emit/update consolidated incidents.

import time
from collections import deque, defaultdict, namedtuple
from threading import Lock, Thread, Event

Alert = namedtuple("Alert", ["id", "service", "root_cause", "severity", "timestamp"])
GROUP_KEY = lambda a: (a.service, a.root_cause)

class GroupState:
    def __init__(self):
        self.alerts = deque()   # items: (timestamp, severity, id)
        self.count = 0
        self.agg_severity = 0.0
        self.last_emit = 0

    def add(self, alert):
        self.alerts.append((alert.timestamp, alert.severity, alert.id))
        self.count += 1
        # Example aggregation: weighted average by recency simple approach -> keep max for correctness
        self.agg_severity = max(self.agg_severity, alert.severity)
        self.last_seen = alert.timestamp

    def evict_older_than(self, cutoff):
        changed = False
        while self.alerts and self.alerts[0][0] < cutoff:
            _, sev, _ = self.alerts.popleft()
            self.count -= 1
            changed = True
        # recompute agg if needed (O(n) only when changed and agg could have been evicted)
        if changed:
            self.agg_severity = max((s for (t,s,i) in self.alerts), default=0.0)

class Deduper:
    def __init__(self, window_seconds=300, emit_callback=None, eviction_interval=5):
        self.window = window_seconds
        self.groups = defaultdict(GroupState)
        self.lock = Lock()
        self.emit_cb = emit_callback or (lambda k, s: None)
        self.stop_ev = Event()
        self.eviction_interval = eviction_interval
        Thread(target=self._evictor, daemon=True).start()

    def ingest(self, alert: Alert):
        now = time.time()
        key = GROUP_KEY(alert)
        with self.lock:
            group = self.groups[key]
            group.add(alert)
            # immediate emit/update can be based on policy; emit updated incident
            incident = self._build_incident(key, group)
        self.emit_cb(key, incident)

    def _build_incident(self, key, group: GroupState):
        return {
            "service": key[0],
            "root_cause": key[1],
            "count": group.count,
            "severity": group.agg_severity,
            "last_seen": getattr(group, "last_seen", None)
        }

    def _evictor(self):
        while not self.stop_ev.is_set():
            cutoff = time.time() - self.window
            with self.lock:
                to_delete = []
                for key, group in list(self.groups.items()):
                    group.evict_older_than(cutoff)
                    if group.count == 0:
                        # optionally emit resolved incident
                        self.emit_cb(key, {"resolved": True})
                        to_delete.append(key)
                    else:
                        # emit updated incident if needed
                        self.emit_cb(key, self._build_incident(key, group))
                for k in to_delete:
                    del self.groups[k]
            self.stop_ev.wait(self.eviction_interval)

    def shutdown(self):
        self.stop_ev.set()

Key points:

• Sliding window implemented by timestamped deque per group; eviction is O(n_group) when pruning.
• Aggregation uses max severity for deterministic correctness; can be changed to weighted average but requires recomputation on eviction.
• Thread-safe with coarse lock; can shard by service to reduce contention.

Complexity:

• Ingest: amortized O(1) append; occasional O(n_group) when eviction causes recompute of agg severity.
• Memory: O(total alerts in window).
• Evictor: scans groups every eviction_interval -> O(num_groups + total_alerts_in_window) per run.

Eventual consistency and distributed concerns:

• In distributed ingestion, alerts may arrive out-of-order or duplicated. Use idempotency keys, compare timestamps, and use watermarking to decide lateness cutoff. For multi-node scaling, shard by service/root_cause (consistent hashing) and use a single writer per shard or use a distributed state store (e.g., Redis Streams with consumer groups) to ensure a single source of truth per group.
• Emissions are eventually consistent: incidents may be updated multiple times as late alerts arrive; design downstream consumers to handle updates and resolved messages idempotently.

Testing under high throughput:

• Unit tests for grouping, eviction, edge cases (out-of-order, duplicates, same timestamp).
• Load test: produce alerts at target QPS with realistic skews; measure latency, memory, GC.
• Chaos tests: delay/reorder alerts, inject duplicates, node restarts.
• Benchmark sharding strategies and lock contention; profile hot groups and add backpressure or sampling if necessary.
• Validate correctness via golden-model aggregator and property-based tests asserting counts and window membership.

Recommended on-call rotation & escalation for a 24/7 SRE team supporting multiple services:

Rotation

• Primary on-call: 1-week shift, daily overlap handoff window (30 min) at shift start/end. Rotate weekly to avoid burnout.
• Secondary (backup): assigned weekly, reachable within escalation timeline.
• Pager duty for on-call manager/lead for major incident coordination (monthly rotation).
• Roster includes service owners for each critical service.

Severity mapping, paging, and SLA

• Sev‑1 (service down / data loss / customer-impacting ≥ critical SLO breach)
  • Who paged: Primary on-call immediately, Secondary after timeout, Service owner(s), On-call manager, Product lead (if customer-visible).
  • Ack SLA: 1 minute. Escalation timeline: escalate to Secondary and manager at 5 minutes; if unacknowledged at 10 minutes, call and notify exec on-call.
• Sev‑2 (degraded service, partial impact, elevated error rates)
  • Who paged: Primary on-call and Service owner.
  • Ack SLA: 5 minutes. Escalation: escalate to Secondary at 15 minutes; include manager at 30 minutes if not resolved.
• Sev‑3 (non-urgent, single-user or minor degradation)
  • Who paged: Primary on-call via low-urgency channel (ticketing/async).
  • Ack SLA: 1 hour. Escalation: escalate to Secondary or on-call manager after 4 hours or per SLAs.

Escalation mechanics

• Use automated runbooks in alert payloads; include playbook link, impact, recent logs, runbook steps.
• Use multi-channel escalation (pager → SMS → phone call → conference bridge) for Sev‑1.

Handoffs & overlapping incidents

• At shift handoff: 30-minute synchronous overlap to review active incidents, runbook steps taken, next actions, and owner assignments. Document in incident timeline.
• If incident spans shifts: outgoing remains primary until formally handed off; handoff only after briefing and shared post-it checklist on incident board. Handoff recorded in incident ticket with time, reasons, and acceptance by incoming primary.
• During major incidents, assign a dedicated Incident Commander (IC) who persists across rotations unless formally relieved; relieve only with overlap briefing to maintain continuity.

Additional practices

• Enforce max one high-severity on-call week per quarter per engineer where possible.
• Run regular drills and post-incident reviews; update runbooks within 48 hours of incident closure.

Situation: I'm incident commander for a Sev-1 during a major marketing campaign causing ~3x traffic.

15-minute triage checklist (minutes 0–15)

1. Declare incident and owner
   • I announce I am IC, set incident channel (Slack/Zoom), and assign note-taker and scribe.
2. Gather high-priority telemetry (first 5 minutes)
   • Service health: error rates, 5xx/4xx by service/endpoint (APM/ELK/NewRelic).
   • Latency: P50/P95/P99 for front-end, API, DB calls.
   • Capacity: CPU, memory, thread/pool saturation, queue lengths, connection counts.
   • Traffic patterns: ingress rate, geo distribution, new vs returning, rate of growth.
   • Downstream/backends: DB replica lag, third-party API error rates, cache hit/miss.
   • Deployments: recent deploys, config changes, feature flags toggled.
3. Triage decision: mitigation vs diagnosis
   • If system integrity at risk (data loss, cascading failures) -> immediate mitigation (rate-limit, fail-open/closed, circuit-breakers, toggle campaign feature flag).
   • If degraded but stable -> prioritize quick diag steps (identify overloaded tier) while monitoring mitigation.
4. Immediate mitigations to apply (parallel)
   • Enable autoscaling policies or increase capacity (pre-approved runbooks).
   • Enable rate-limiting / throttling for non-essential traffic (e.g., bulk API, analytics).
   • Turn off non-critical background jobs, async processors, and batch jobs.
   • Flip campaign feature flag to reduced mode (e.g., lower personalization).
   • Route traffic away from unhealthy instances via LB drain.
5. Stakeholders to notify (first 10 minutes)
   • Engineering on-call squads for affected services, platform team, product/marketing lead, Customer Support lead, and CTO/ops leader.
   • Provide status: impact summary, actions underway, estimated next update.
6. Decisions to defer
   • Full root-cause analysis, long-term capacity changes, schema migrations, rolling restarts across all regions (unless critical).

Communication cadence (first hour)

• T+0:30 — Initial incident declared with impact, scope, and mitigations in incident channel and to stakeholders.
• T+15 — Short update: telemetry summary, mitigations applied, next steps, and ETA for next update.
• T+30 — Progress update: results of mitigations, whether traffic stabilized, any new actions.
• T+45 — If unresolved, escalate mitigation intensity or rollbacks; update stakeholders.
• T+60 — Decision point: restore normal traffic or continue mitigations; assign owners for RCA and recovery plan.

Throughout: concise, timestamped updates, one voice from IC, and ensure Customer Support has a tailorable message.