Comprehensive Interview Preparation Guide for Staff-Level Site Reliability Engineer (FAANG Standards)

Requirements & goals:

• Ensure every incident yields a consistent postmortem, tracked action items with owners/due dates, automated ticket creation, and visibility on dashboards to reduce regression risk.

Postmortem template (standard fields):

• Incident ID / Severity / Services affected
• Timeline (detection → mitigation → recovery) with timestamps
• Root cause analysis (contributing factors)
• Impact (users, SLOs, cost)
• Mitigations applied / Rollback steps
• Action items (title | owner | priority | due date | ticket ID | status)
• Postmortem author + reviewers
• Lessons learned + preventive measures
• Link to logs/trace/grafana panels

Automation pipeline (high-level flow):

1. Trigger: Incident closed/triaged in PagerDuty → webhook to “postmortem pipeline” service.
2. Create doc: Pipeline instantiates a postmortem from template in Confluence/Notion/Git repo, pre-filling incident metadata and timeline snippets (from alert payload, Logging/Tracing links).
3. Extract AI-assisted RCA: Optionally run a lightweight parser/LLM to suggest timeline and potential causes for reviewer validation.
4. Action extraction: UI or parser turns identified actions into structured items.
5. Ticket creation: For each action, pipeline creates tickets in Jira/ServiceNow via API, assigns owner (from on-call/rotation mapping or manual), sets priority and due date, and links ticket → postmortem → incident.
6. Enforcement: Add ticket to a tracked board with required SLA (e.g., 2-week auto-escalation). Create recurring reminders in Slack and calendar.
7. Dashboard & metrics: Emit events to metrics store and Grafana/Looker:
   • Postmortems created per incident
   • Action-item completion rate and age
   • Mean time to remediate preventive actions
   • Regression incidence per service
8. Follow-through: Weekly pipeline job queries open postmortem tickets, nudges owners, and if overdue triggers manager escalation; quarterly audits ensure closure and verifies implemented changes (code reviews, runbook updates).

Integration specifics:

• Use webhook-based triggers (PagerDuty/PagerTree/Custom) → central service (Python/Go) that calls Confluence API + Jira REST API.
• Store canonical links in incident metadata (logs, traces, dashboards).
• Use service ownership mapping (GitHub CODEOWNERS or internal registry) to suggest owners.
• Expose Grafana dashboard panels linking to postmortems and open action items; add alerts for regression rate or rising open-action backlog.

Data flow (simplified):
PagerDuty webhook → postmortem-service → create doc + parse timeline → generate tickets (Jira) → emit metrics to Prometheus → Grafana dashboard → reminders/Slack notifications.

Scalability & reliability:

• Stateless postmortem-service behind autoscaling; queue (Kafka/SQS) for bursts.
• Idempotent handlers to avoid duplicated docs/tickets.
• RBAC so only authorized edits update action statuses.

Trade-offs:

• Full automation of RCA risks false causes — keep human validation step.
• LLM suggestions speed drafting but must be auditable.
• Tight enforcement vs. team burden: prefer SLAs + escalation rather than automatic ticket ownership reassignment.

Success metrics:

• % incidents with completed postmortems within 48h
• Action-item completion rate within due date (>90%)
• Regression rate per service (trend down)
• Reduction in repeat incidents from same root cause

This pipeline balances automation (reduce toil) with human review (accuracy), integrates with common ticketing and dashboard tools, and creates measurable accountability to prevent regressions.

NetworkPolicy in Kubernetes is purely additive: there is no "deny" rule you author directly. Once any policy selects a pod for a given direction (ingress or egress), that pod becomes default-deny in that direction, and only traffic matching some policy's explicit allow rule gets through. So finding "which policy is responsible" for a block usually means finding a policy that selects the destination pods and either has no matching rule at all, or a rule that is narrower than the traffic actually needs, not finding an explicit deny statement anywhere.

Step 1: find every policy that could affect the destination

kubectl get networkpolicy -n prod -o json | jq '.items[] | {name: .metadata.name, podSelector: .spec.podSelector, policyTypes: .spec.policyTypes}'

Example output:

{"name": "default-deny-ingress", "podSelector": {}, "policyTypes": ["Ingress"]}
{"name": "allow-internal-prod", "podSelector": {"matchLabels": {"tier": "internal"}}, "policyTypes": ["Ingress"]}

An empty podSelector: {} matches every pod in the namespace. Here, default-deny-ingress selects all pods in prod for Ingress and defines no ingress rules at all, meaning every pod in prod denies all inbound traffic unless a second, more specific policy also selects it and allows something. allow-internal-prod only helps pods labeled tier: internal; if the backend pods are not labeled that way, they get no exception and stay fully closed.

Step 2: reproduce and confirm which rule (or absence of one) is the block

kubectl -n web exec -it frontend-pod -- curl -sv --connect-timeout 5 http://backend.prod.svc.cluster.local:80

A connection that hangs until timeout (as opposed to an immediate connection refused) is the characteristic signature of a NetworkPolicy drop: the packet is silently discarded rather than actively rejected, since the destination pod's application never even sees the SYN.

Then inspect the specific policy selecting app=backend in prod:

kubectl describe networkpolicy default-deny-ingress -n prod

PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
Allowing ingress traffic:
  <none> (Selected pods are isolated for ingress connectivity)

"Selected pods are isolated for ingress connectivity" with no rules listed is the exact confirmation: this policy is the one making backend pods deny-by-default, and no other policy in prod currently grants an exception for traffic from web.

Step 3: minimal fix

Add a narrowly scoped allow rule rather than widening or removing the existing deny-all; NetworkPolicy design should stay additive and specific.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-frontend-to-backend
  namespace: prod
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: web
      podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 80

This uses kubernetes.io/metadata.name, the label Kubernetes automatically applies to every namespace (set to the namespace's own name) since it became a default-on feature around Kubernetes 1.21, rather than assuming a custom name=web label already exists on the web namespace. Relying on the built-in label avoids a second failure mode: a namespaceSelector that silently matches nothing because the expected custom label was never applied.

kubectl apply -f allow-web-frontend-to-backend.yaml
kubectl -n web exec -it frontend-pod -- curl -sS http://backend.prod.svc.cluster.local:80 -o /dev/null && echo OK

Trade-offs and pitfalls

• A podSelector: {} combined with policyTypes: [Ingress] and zero rules is easy to misread as "no policy" when it is in fact the strictest possible one (deny everything); always check for this shape first, since it is the single most common cause of "NetworkPolicy is blocking traffic unexpectedly" reports.
• namespaceSelector and podSelector inside one from entry are ANDed together (both must match the same source pod); a common mistake is listing them as two separate entries in the from list, which ORs them instead and unintentionally allows any pod in the web namespace, or any app=frontend pod in any namespace, rather than the intersection actually intended.
• Widening an existing broad policy (adding an exception to default-deny-ingress itself) instead of adding a new, narrowly scoped policy makes the security intent harder to audit later; prefer additive, purpose-named policies over editing a shared deny-all policy in place.
• This fix only affects Ingress on the backend side; if prod or web also enforces Egress policies, the frontend pods' egress must independently allow traffic to backend, or the same symptom recurs from the other direction.

Direct answer

Say what the success metric was, where the baseline number came from, and what target you set, before you describe the outcome. The interviewer is probing whether "success" was defined up front and grounded in a real measurement, versus decided retroactively once you knew how things turned out.

How to define and defend success criteria

Pick a metric tied to an outcome, not an output. "We shipped the feature" is an output; "time-to-first-task dropped" or "escalation volume fell" is an outcome. Output-only "success" criteria are a common tell that the metric was picked after the fact.

Establish the baseline honestly. State exactly where the baseline number came from: an existing dashboard, a manual count over a defined window, or a proxy metric if the exact one didn't exist yet. If you had to estimate, say so and say how.

Set the target with reasoning, not a round number pulled from nowhere. A defensible target is anchored to something: a competitor benchmark, a prior period's rate of improvement, or the minimum change needed to matter to the business.

Leading vs. lagging metrics:

Pairing one of each lets you show early progress without over-claiming the final outcome too soon.

Worked example (skeleton)

Baseline: support tickets tagged "onboarding confusion" averaged 20 per week over the prior month, pulled directly from the ticketing system's tag filter. Target: cut that to under 10 per week within two months of shipping a redesigned onboarding flow, a 50% reduction chosen because it was the minimum drop the support lead said would let them reallocate a headcount from triage to other work. Guardrail: overall support volume tracked in parallel, to catch a bug where confused users simply stopped filing tickets instead of getting unconfused.

Trade-offs and pitfalls

• Don't present an output metric as if it were the success criterion; "we launched on time" is a milestone, not a definition of success.
• Be ready to say exactly where the baseline number came from; "around 20 a week" with no source is the kind of claim that collapses under a follow-up question.
• Watch for target numbers that look precise but have no derivation behind them; a round target with stated reasoning is more credible than a falsely precise one with none.
• A guardrail metric (something that would catch you gaming the primary metric) is what separates a senior answer from a junior one here.

Direct answer

Stay neutral by asking, not telling, both sides. Get underneath the stated positions by finding what each person is actually protecting, revenue predictability, technical risk, reputation, past experience, and negotiate from that underlying interest instead of from the positions they walked in with.

Structured elaboration

The technique is separating positions from interests: a position is what someone is demanding, an interest is why they are demanding it.

1. Meet each person separately first if the joint room is too charged for either to speak candidly. Ask direct, non-leading questions: what happens if this goes the other way, what are you actually worried about? People often state a harder position in front of the other person than they hold privately.
2. Listen for the interest underneath the stated position. Someone insisting on a hard deadline may actually be worried about looking unreliable to their own stakeholders, not about the date itself. Someone insisting on a particular technical approach may be protecting against repeating a past incident, not defending the approach for its own sake.
3. Stay neutral by not adopting either position as your own recommendation early. Reflect both interests back to each person to confirm you understood correctly, so neither feels unheard before you move forward.
4. Once the interests are visible, look for an option that satisfies both, rather than a compromise that splits the difference on the original positions, which usually satisfies neither interest fully.
5. Bring both people into the same room only once you understand both sides well enough to keep the conversation anchored on interests instead of letting it slide back into restating positions.
6. Get an explicit, verbal agreement to the resolution from both people in the room, then document it along with the reasoning, not just the outcome. Decisions that skip the why tend to get quietly re-litigated later.

Worked example

A sales leader wants a fully featured launch by end of quarter. Engineering leadership wants a two-quarter delay to avoid instability. The positions look flatly opposed, but talking to each separately surfaces that sales actually needs one specific, demonstrable capability for a committed customer conversation, not literally everything, and engineering is actually worried about one particular unstable subsystem, not the whole feature set. A phased plan that ships the specific capability sales needs now, built on the stable parts of the system, while the risky subsystem gets its rework on its own timeline, satisfies both underlying interests even though neither person's original position was granted in full.

Trade-offs and pitfalls

Interest-based framing is slower than just picking a side. If a decision genuinely has to happen today, spend the time finding interests quickly rather than skipping the step entirely, a decision made purely from positions rarely survives contact with implementation.
Staying neutral does not mean staying silent when one side's underlying interest is simply wrong on the facts, a risk that is not real, a deadline that is not actually fixed. Neutrality is about the process, not about withholding a correction you are confident in.
If you only interview people separately and never get them into the same room to hear each other directly, you risk becoming a relay each side later blames for misrepresenting them. A well-framed joint session is usually still necessary, not optional.

Direct answer

Resharding a live cluster safely means moving keys gradually with dual-read or dual-write support during the transition, never flipping all traffic to the new layout at once, and keeping a clear rollback path if verification fails partway through.

Structured elaboration

• Partitioning approach: consistent hashing (with virtual nodes) over naive modulo hashing, since modulo remaps nearly the entire keyspace on any node-count change, while consistent hashing only remaps the fraction of keys that fall in the affected ring segment.
• Handling node add/remove: when adding a node, only the keys now owned by that node (per the ring) need to migrate; a migration worker copies those specific keys from their old owner to the new owner, then updates routing once the copy is verified complete for that range.
• Mitigating hot partitions during migration: migrating a hot key's range can itself create load on both the old and new owning nodes simultaneously; throttle migration bandwidth for ranges known to be hot, and consider migrating them last, once the migration process itself is validated on cooler ranges.
• Routing metadata: store the current ring/shard-to-node mapping in a strongly consistent, low-latency store (e.g., a small dedicated configuration service, or the cache cluster's own management plane) that all clients or proxies consult; a stale routing table during migration is what causes reads/writes to hit the wrong node.
• Safe resharding sequence: (1) begin dual-write for the affected key ranges (write to both old and new owner), (2) backfill/copy existing data for those ranges to the new owner, (3) verify the new owner has complete, correct data, (4) switch reads to the new owner for those ranges, (5) stop writing to the old owner and reclaim its capacity.

Worked example

For a Redis-based cache holding session objects for 200 million active users, resharding from 20 to 25 nodes moves roughly 20 percent of the keyspace under consistent hashing ($1 - 20/25 = 20\%$). Migrating that fraction gradually, at a throttled rate (e.g., a few thousand keys per second) rather than all at once, keeps the source and destination nodes' load within normal operating bounds throughout the multi-hour migration window, instead of a spike that itself risks an incident.

Trade-offs and pitfalls

Skipping the dual-write/verification step to move faster risks a window where a client reads from the new owner before it has the complete, correct data, silently serving wrong results (not just a miss); always verify before cutting reads over. A stale routing table cached by a client during migration is a common source of "works on most nodes, fails intermittently" bugs; keep the routing metadata's own propagation latency well inside your migration pace.

Start by defining the two models briefly: Centralized Incident Command (CIC) uses a small, trained incident command center (IC) that coordinates detection, triage, decisions and communications. Decentralized Empowered-Responder (DER) gives service teams or on-call engineers authority to act independently, with lightweight coordination.

Centralized Incident Command

• Advantages: consistent decision-making, single source of truth for status/communications, easier cross-service prioritization, preserves exec-level escalation path. Good for high-severity outages affecting many services.
• Risks: bottlenecked decisions, slower local remediation, single-point-of-failure if IC is overloaded or understaffed, reduced domain-context in decisions.
• Scaling: scale by staffing rotating IC shifts, runbooks, playbooks, and tooling (war-rooms, status pages). Beyond certain incident volume/variety, IC becomes overwhelmed.
• Governance: strict roles (commander, scribe, communications), playbooks, simulated drills, SLA for IC response times, auditing and runbook maintenance.

Decentralized Empowered-Responder

• Advantages: fast local remediation, leverages domain expertise, encourages ownership and rapid iteration, reduces coordination overhead for localized issues.
• Risks: inconsistent communications, conflicting mitigations across teams, difficulty prioritizing cross-cutting incidents, harder to present unified postmortems.
• Scaling: requires reliable observability, automated guardrails (circuit breakers, feature flags), standardized alerting taxonomy, and runbook templates to keep coherence as teams grow.
• Governance: clear boundaries of authority, escalation criteria, mandatory post-incident reporting, shared incident metadata standards, and periodic cross-team drills.

When to adopt which (criteria)

• Prefer CIC when incidents are high blast-radius, regulatory-sensitive, cross-domain, or require executive coordination; or when org maturity for decentralized ops is low.
• Prefer DER when services are well-isolated, teams are experienced, ownership culture exists, and fast local fixes reduce customer impact.
• Hybrid: most large enterprises should adopt hybrid: empower responders for low-to-medium severity and local issues while IC activates for multi-service or high-impact incidents. Implement trigger criteria (severity thresholds, affected-service-count, legal/PR risk) that automatically flip into centralized mode.

Operational recommendations

• Define clear severity-to-governance mapping and automated escalation rules.
• Invest in standardized tooling: incident templates, shared dashboards, chatops, and a central incident metadata store.
• Regularly exercise both modes (game days) and refine decision thresholds.
• Measure: mean time to acknowledge, time to mitigate, communication lag, and postmortem quality to tune the balance.

Direct answer
For a global, latency-sensitive user-profile subsystem, a reasonable decomposition splits Profile Storage, Authentication, Preferences, and Avatar/Media Processing into separate services, each with a clearly different access pattern and failure-isolation need, communicating synchronously only where a request genuinely can't proceed without an immediate answer.

Structured elaboration
Authentication is kept separate because it's a security-sensitive, extremely high-frequency, low-latency dependency that nearly every other request needs to check, and isolating it means an issue in less-critical functionality (like avatar processing) can never take down the ability to authenticate a request. Profile Storage owns the core profile fields (name, settings, account state) and needs to be read on nearly every request, so it's optimized for fast, simple reads, kept narrowly scoped to just that core data rather than accumulating every profile-adjacent feature over time. Preferences (notification settings, UI preferences, and similar lower-stakes, less frequently-read data) is split out specifically because it doesn't need the same latency or availability guarantees as core profile and authentication data; if the Preferences service is slow or briefly unavailable, the rest of the profile experience should still work with sensible defaults, whereas Authentication being unavailable is a much more serious failure. Avatar/media processing is naturally asynchronous (resizing, transcoding, or moderating an uploaded image takes real time and doesn't need to block the rest of the profile experience) and has a very different resource profile (CPU/IO-heavy batch-style work) from the fast, simple reads the other services handle, making it a clear candidate for its own service with its own scaling behavior.

On communication choices: reading a user's core profile (needed on most page loads) should be synchronous, since the caller can't meaningfully proceed without it; updating a display avatar can be asynchronous, since the caller doesn't need to wait for image processing to finish, just for the upload to be accepted. Failure isolation follows from the split: if Avatar Processing is degraded, users can still authenticate, read their profile, and use the rest of the product, with a graceful fallback (a default avatar, or the previous one) rather than the whole profile experience failing.

Worked example
For team coordination across these services: the team owning Authentication needs the strictest change-management discipline given its security sensitivity and blast radius if it breaks, and its API contract with the other three services should be the most stable and carefully versioned of the four; Profile Storage's API needs to stay backward-compatible for the many other services and clients that read from it directly; Preferences and Avatar Processing, being lower-stakes and more independently used, can iterate faster with a lighter review process, as long as their optional, non-blocking nature is preserved (nothing else should come to depend on them synchronously in a way that would undo the isolation benefit).

Trade-offs and pitfalls
The most common mistake in a decomposition like this is letting Profile Storage's scope creep to absorb Preferences or other adjacent data over time ("it's just one more field"), which quietly recreates the coupling the split was meant to avoid; keeping the boundary honest requires treating a new field's home as a deliberate decision (does this need Profile Storage's stricter availability and latency guarantees, or can it live in the more relaxed Preferences service) rather than defaulting to wherever's most convenient to add it.

Direct answer

subprocess.run blocks the calling thread until the child process exits (or times out); asyncio.create_subprocess_exec returns control to the event loop immediately and lets you await the child's completion alongside other concurrent work. The choice is really about what else your script needs to be doing while the external command runs.

Blocking behavior and concurrency model

subprocess.run is simplest for a script that runs external commands sequentially, one after another -- there's no event loop to reason about, no await syntax, and stdout/stderr capture is a single synchronous call. If you need to run several external commands concurrently, the sync option is threads (a ThreadPoolExecutor calling subprocess.run per worker) -- which works, but each thread is a real OS thread, so it doesn't scale cleanly past a few hundred concurrent subprocess launches.

asyncio.create_subprocess_exec fits when the automation is ALREADY asyncio-based (talking to async HTTP clients, other async I/O) and you want subprocess launches to be just another awaitable alongside that work, sharing the same single-threaded event loop rather than spinning up a thread pool. It also composes naturally with asyncio.gather and semaphores for bounding how many subprocesses run at once, without the OS-thread overhead of the threaded approach.

Ease of implementation

subprocess.run wins here decisively for anything simple: subprocess.run([...], capture_output=True, timeout=30, check=True) is a single, easy-to-read line. The asyncio version requires creating the subprocess, then separately await-ing communicate() or manually pumping the stdout/stderr streams, and wrapping the whole thing in an async function -- meaningfully more ceremony for the same basic 'run a command and get its output' task.

When to prefer asyncio for SRE automation

Prefer asyncio when the automation's dominant cost is genuinely concurrent I/O-bound work at meaningful scale -- for example, running the same health-check command over SSH against 500 hosts in parallel, where you want hundreds of subprocesses in flight without hundreds of OS threads. Prefer sync subprocess.run (possibly with a modest thread pool) for anything with a handful of sequential or lightly-parallel external calls, a one-off maintenance script, or anywhere the extra async ceremony would cost more in review/maintenance burden than it saves in throughput. A useful rule of thumb: if you're not already in an async codebase and you're not launching dozens+ of concurrent subprocesses, subprocess.run (with threads if you need modest parallelism) is very likely simpler, more debuggable, and just as correct.

Trade-offs and pitfalls

Mixing the two carelessly is the most common real bug: calling a blocking subprocess.run from inside an async function (instead of create_subprocess_exec or running it in an executor) blocks the entire event loop, silently stalling every OTHER concurrent task the automation was supposed to be running -- turning what looked like a concurrent asyncio program into an accidentally-sequential one, with no error raised to tell you.

A steady 12-month decline coinciding with a 3x increase in release frequency is a strong, though not proof-positive, correlational signal that increased deploy velocity is outpacing the reliability discipline (testing, canarying, rollback speed) needed to support it safely.

Structured elaboration

Probable causes to investigate: whether the SAME proportion of releases are now causing incidents (in which case the decline is simply a volume effect of shipping more, and the underlying release quality hasn't actually gotten worse per-release) versus whether a GROWING proportion of releases are now causing incidents (a genuine quality regression in the release process itself, perhaps because testing or review rigor hasn't scaled alongside the increased velocity). Immediate remediation: a temporary, deliberate SLOWING of release velocity while root-causing continues, paired with tightening the canary/rollback process (shorter time-to-detect, faster automatic rollback) so that whatever incidents do occur have a smaller blast radius and shorter duration even before the root cause is fully addressed.

Worked example

Analysis reveals whether the incident-per-release RATE has grown (not just the raw release count): if releases per month tripled (say from 10 to 30) and incidents per month also roughly tripled (from 1 to 3), the per-release incident rate stayed constant, suggesting the decline is a volume effect the team can address by improving detection/rollback speed rather than a fundamental quality regression; if incidents per month grew MORE than proportionally (say from 1 to 8), that's a real per-release quality regression, pointing toward insufficient testing or review scaling with the new velocity, warranting a harder look at whether CI coverage, canary duration, or review rigor were quietly cut corners to sustain the faster pace.

Trade-offs and pitfalls

Whether SLOs should be adjusted in the short term: the honest answer is usually NO, not yet; the declining trend from 99.99% to 99.85% likely reflects a genuine, fixable process gap rather than the target having become inherently unachievable, and relaxing the SLO in response to a trend that root-causing might well reverse risks permanently normalizing a preventable regression instead of fixing it. Process changes to prevent recurrence: as release velocity scales, testing and canary infrastructure need proportional investment (not a fixed amount set once and never revisited), and a periodic check specifically correlating release frequency against incident rate (the exact analysis this question describes) should become a routine, recurring health check, not a one-off investigation triggered only after the decline became this severe.

Situation: Postmortem shows several Sev-2 incidents should have been Sev-1 — indicating inconsistent severity classification that risks delayed escalations and customer impact.

Proposal (process + tooling):

1. Mandatory severity review cadence

• For every incident created, require an initial severity assessment by the on-call engineer and a 15‑minute confirmation by the triage lead/IC. If disagreement, escalate to the escalation policy owner within 30 minutes.
• Add a mandated severity check at postmortem kickoff (24–72 hours) to confirm final classification and rationale.

2. Severity-thermostat metrics

• Track: % incidents reclassified upward within 7 days, median time-to-escalate, false-negative Sev-1 (missed Sev-1), customer-impact minutes.
• Target: <2% upward reclassifications, median time-to-escalate <15 mins for Sev-1.

3. Automated guardrails that flag misclassifications

• Integrate rules in incident creation (PagerDuty/Jira/SRE tooling): if incident meets predicates (service down > X, >Y% error rate, SLO breach, multiple customers affected), auto-flag as potential Sev-1 and require a confirmation step before closing.
• Build a classifier using historical incidents (features: SLO breach, error rate, number of affected regions/customers, rollback required) to surface likely-severity mismatches to triage leads.
• Add CI-style pre-close checks that prevent closing a Sev-2 if certain signals (ongoing SLO breaches) exist.

4. Improved training

• Quarterly severity calibration sessions with simulated incidents; publish a one-page decision tree and canonical examples for Sev-0/1/2/3.
• New-hire bootcamp includes a severity-playbook module and hands-on tabletop exercises.

5. Lightweight appeals/audit process

• Create a 2-step appeals flow: submit appeal within 7 days with evidence → 48-hour review by a rotating Severity Audit Panel (senior SRE + product + support). Panel can reclassify and annotate lessons.
• Maintain an audit log of appeals and panel decisions.

Measurement and improvement

• Weekly dashboard: upward reclassification rate, time-to-first-escalation, classifier precision/recall, customer-impact minutes. Run monthly retros to tune thresholds and training.
• Aim for progressive improvement: monthly reduction targets and report to reliability steering.

Accountability

• Incident owner/on-call: initial severity assignment and documentation.
• Triage lead/IC: 15-min confirmation and escalation decision.
• Reliability engineering manager: owns severity metrics and classifier maintenance.
• Severity Audit Panel: governance and appeals.
• SLIs/SLOs/product owners: ensure SLO definitions map to severity rules.

Example: After enabling guardrails, we expect auto-flagging to catch ~80% of missed Sev-1s; appeals provide human oversight and feed training data to retrain the classifier.

This combination of fast human checks, measurable telemetry, automated guardrails, continuous training, and a short appeals loop reduces misclassification risk while keeping processes lightweight and accountable.