Root Cause Analysis and Corrective Actions Questions

Covers methods and practices for identifying and eliminating the underlying causes of incidents and problems, and for ensuring effective remediation. Topics include structured analysis techniques such as five whys and fishbone diagrams, causal factor mapping, and evidence gathering to move beyond surface symptoms to systemic root causes like control gaps, training deficiencies, process defects, unclear policies, cultural issues, or supervisory failures. Includes postmortem practices such as blameless facilitation, creating psychological safety so people speak openly, designing postmortem templates, documenting findings, and avoiding postmortem fatigue by applying proportional review. Covers designing, prioritizing, tracking, and verifying corrective actions and remediation plans, including metrics and acceptance criteria for when an action is considered effective. Senior level skills include facilitating cross functional postmortems, establishing governance and feedback loops, converting incident learnings into continuous improvement, balancing quick fixes with long term prevention, and building systems to ensure remediation ownership and ongoing measurement.

MediumTechnical

71 practiced

A critical runbook failed during a live outage. Propose a validation and maintenance process for runbooks so they stay accurate and effective. Include validation cadence, automated checks, ownership, and how to capture lessons from real incidents.

Sample Answer

Validation & maintenance process (runbook-as-code)

1) Ownership & lifecycle- Assign an explicit runbook owner (primary) and a secondary on-call reviewer. Owners are responsible for accuracy, tests, and updates.- Store runbooks in git (md or structured YAML) with PRs, templates, and required approvers.

2) Validation cadence- Continuous: every PR must pass automated checks (see below).- Quarterly: owners perform a manual review and a tabletop walkthrough with the on-call team.- Annually: executive review to ensure alignment with SLOs and architecture changes.- After every incident: mandatory update within 48 hours (see lessons capture).

3) Automated checks (CI)- Linting/format checks (structure, required fields: symptoms, pre-reqs, impact, rollback).- Link & credential checks (no hard-coded secrets; verify referenced playbooks exist).- Smoke execution: automated dry-run or mock-run of steps that can be safely executed (API calls simulated, idempotent CLI steps executed in a sandbox).- Validation tests: runbook test harness that asserts expected outputs, timing, and alert suppression logic. Fail CI on regressions.- Telemetry harness: synthetic exercises (daily/weekly) to validate alert -> runbook mapping and that runbook triggers the right playbook.

4) Capturing lessons- Post-incident: run a blameless postmortem that includes a "runbook fidelity" section. Record: - Which steps failed or were missing - Time-to-action vs expected - Suggested edits (explicit diff in runbook repo)- Link postmortem to runbook PR; require owner sign-off to merge fixes.- Track metrics: runbook success rate, time-to-first-action, number of manual ad-hoc steps added. Surface in dashboard.

5) Practice & readiness- Regular game-days / chaos experiments that exercise runbooks end-to-end.- New hires and on-call rotations perform at least one supervised real run of critical runbooks.

6) Small safety rules- Mark destructive steps clearly; require an explicit "confirm" and privileged gating.- Include rollback and verification steps; provide expected log patterns and monitoring queries.

Why this works: runbook-as-code + CI enforces correctness early; cadence + drills keep human processes fresh; postmortems close the loop so incidents improve operational playbooks rather than erode trust.

EasyTechnical

74 practiced

Describe the 'Five Whys' technique and provide a step-by-step application for this incident: 'A key API returned 502 errors for 45 minutes after a deploy, impacting 20% of users.' Show five levels of why and conclude with an actionable root cause and one remediation.

Sample Answer

The Five Whys is a root-cause technique that repeatedly asks "Why?" (typically five times) to move from symptomatic failures to an actionable root cause. It's useful in incidents to avoid stopping at surface fixes and to identify process or system changes that prevent recurrence.

Applied to: "A key API returned 502 errors for 45 minutes after a deploy, impacting 20% of users."

1) Why did the API return 502 errors?- Because the upstream service returned invalid responses or timed out, leading the API gateway to respond with 502.

2) Why did the upstream service return invalid responses/timings?- Because the service instances failed health checks and began returning 500-level errors or dropped connections under load.

3) Why did instances fail health checks / error under load after the deploy?- Because the new deployment introduced a blocking initialization routine that delayed readiness, causing the load balancer to route traffic to instances not fully ready.

4) Why was traffic routed to not-fully-ready instances?- Because readiness probes were too permissive (immediate 200) and the deployment strategy (rolling update with minimal overlap) allowed traffic during instance warm-up.

5) Why were readiness probes permissive and deployment strategy risky?- Because there was no runbook enforcing stricter probe criteria or a staged deployment policy; engineering defaults allowed fast rollouts without explicit warm-up verification.

Actionable root cause:- Deployment released change with inadequate readiness checks and an aggressive rollout strategy, allowing traffic to hit unready instances that performed blocking initialization and produced 502s.

One remediation (high-impact, single change):- Enforce stricter readiness probe that verifies full initialization (e.g., internal health endpoint that checks dependencies), and change deployment policy to staged canary rollouts (start 1% traffic, validate metrics for X minutes before ramp). Also add a pre-deploy checklist requiring a warm-up plan for changes that affect startup paths.

HardTechnical

92 practiced

Design a governance framework that integrates SLOs and error budgets across multiple teams so RCA and corrective actions are prioritized automatically. Explain enforcement, exception handling, and how to measure compliance without stifling feature velocity.

Sample Answer

Requirements:- Per-service SLOs (availability/latency/error-rate) with clear owners, measurement windows (e.g., 7d/30d), and defined error budget burn rates.- Cross-team visibility and automated prioritization of RCA/corrective work when error budgets approach thresholds.- Minimal friction for feature delivery unless reliability risk is high.

High-level framework:1. SLO registry & telemetry: central SLO catalog (Git-backed) + metrics pipeline (Prometheus/SignalFX) publishing real-time SLO status and burn rate.2. Automated policy engine: encode governance rules (thresholds: GREEN <50% burn, YELLOW 50–80%, RED 80–100%, BLOCK >100%) in an engine (e.g., Open Policy Agent + Orchestrator).3. Prioritization automation: when a service reaches YELLOW, create a high-priority "Reliability Task" (ticket in Jira) assigned to the SLO owner; at RED generate mandatory RCA workflow with due dates and lightweight mitigation playbook; at BLOCK trigger temporary feature gate (see enforcement).4. Enforcement & feature gating: integrate with CI/CD (feature flag service and deployment pipeline). BLOCK state auto-attaches a deployment guard that prevents non-critical feature merges unless a compensating action is taken (e.g., temporary rollout freeze, extra canary guard). Approvals require a documented mitigation and Product/SRE sign-off.5. Exception handling: support time-boxed exception requests via the SLO registry (owner proposes compensating controls and alternate SLOs), reviewed by Reliability Council (rotating cross-team panel). All exceptions are logged, expire automatically, and must include metrics proving risk mitigation.6. RCA & corrective actions workflow: standardized RCA template, automated evidence collection, and action item tracking with SLAs. Error-budget burn causes automatic scheduling of blameless postmortems and assignment of corrective backlog items with priority and estimated effort.7. Measuring compliance without killing velocity:- KPIs: % services with up-to-date SLOs, % of incidents with completed RCA within SLA, mean time to remediate reliability actions, fraction of deploys blocked vs. approved with mitigations.- Signal-based leniency: only enforce deploy block at BLOCK; YELLOW/RED produce elevated prioritization but not hard stops.- Define "low-risk" change tag that bypasses gating after automated risk checks (short scope, tested, limited blast radius).8. Feedback loops & continuous improvement: weekly SLO review dashboard, monthly Reliability Council review of metrics and exceptional cases, and quarterly adjustments to SLOs/policies.

Why this works:- Automation enforces consistency and reduces human bottlenecks.- Policy tiers preserve velocity early (alerts and prioritization) while only applying hard blocks when risk is high.- Exception process keeps flexibility with governance and auditability.- Measured KPIs align incentives (teams fix reliability proactively) without blanket freezes.

MediumTechnical

81 practiced

You have five candidate corrective actions for a recurring database outage. Design a prioritization matrix using at least four criteria (e.g., impact reduction, recurrence probability, engineer-hours, business cost). Show a sample weighted scoring formula and apply it to rank these hypothetical actions: schema rollback, connection pool tuning, query rewrite, read-replica addition, and automated failover.

Sample Answer

Approach: build a simple weighted scoring matrix with 4 criteria relevant to SRE trade-offs, score each action 1–5 (5 = best), and compute weighted total = sum(weight * score). For cost/effort criteria higher is better (so low engineer-hours or low business cost get higher scores).

Criteria and weights:- Impact reduction (how much outage impact is reduced) — weight 0.40- Recurrence probability reduction (likelihood incident recurs) — weight 0.25- Engineer-hours to implement (lower = better) — weight 0.20 (score inverted)- Business cost / infra cost (lower = better) — weight 0.15 (score inverted)

Scoring rules: 5 = very good (large impact reduction / low hours / low cost), 1 = poor.

Apply to actions (scores as [Impact, Recurrence, EngHours, Cost]):

1) Schema rollback — [3, 3, 4, 5]- quick to revert schema changes; moderate impact and recurrence reduction; low effort/cost

2) Connection pool tuning — [4, 4, 5, 5]- often reduces connection storms and recurrence; low effort and cost

3) Query rewrite — [5, 4, 3, 4]- can eliminate hotspots (high impact), moderate recurrence reduction, moderate effort

4) Read-replica addition — [3, 3, 2, 2]- improves read scalability but doesn't fix root cause; higher infra cost and effort

5) Automated failover — [4, 5, 2, 3]- reduces downtime recurrence and impact, but moderate engineering effort and operational cost

Weighted totals (calculate = 0.4*I + 0.25*R + 0.2*E + 0.15*C):

1) Schema rollback = 0.4*3 + 0.25*3 + 0.2*4 + 0.15*5 = 1.2 + 0.75 + 0.8 + 0.75 = 3.52) Connection pool tuning = 0.4*4 + 0.25*4 + 0.2*5 + 0.15*5 = 1.6 + 1.0 + 1.0 + 0.75 = 4.353) Query rewrite = 0.4*5 + 0.25*4 + 0.2*3 + 0.15*4 = 2.0 + 1.0 + 0.6 + 0.6 = 4.24) Read-replica addition = 0.4*3 + 0.25*3 + 0.2*2 + 0.15*2 = 1.2 + 0.75 + 0.4 + 0.3 = 2.655) Automated failover = 0.4*4 + 0.25*5 + 0.2*2 + 0.15*3 = 1.6 + 1.25 + 0.4 + 0.45 = 3.7

Ranking (highest to lowest):1. Connection pool tuning — 4.352. Query rewrite — 4.203. Automated failover — 3.704. Schema rollback — 3.505. Read-replica addition — 2.65

Recommendation and reasoning:- Immediate: implement connection pool tuning (low effort, high impact, fast ROI) and schedule query rewrites for longer-term removal of hotspots.- Medium-term: automated failover to reduce MTTR and recurrence.- Schema rollback remains a tactical mitigation during incidents.- Read-replicas are lowest priority here: useful for scaling reads but won’t address root cause and has higher cost.

Notes: adjust weights if business priorities shift (e.g., if minimizing recurrence is paramount, raise recurrence weight). Validate estimates with metrics (query latency, connection counts, change history) before execution.

HardTechnical

74 practiced

Propose a program to turn incident learnings into a continuous improvement backlog. Describe intake, prioritization rules, owner assignment, checkpoints for verification, and incentives that encourage teams to close improvements rather than mark them done prematurely.

Sample Answer

Situation: Repeated incidents produced ad-hoc fixes but no sustained reliability improvements. I designed a formal “Incident Improvement Backlog” program to ensure learnings become measurable fixes.

Program overview (intake)- Every P0–P2 post-incident review (PIR) creates 1–3 backlog candidates using a template: symptom, root cause, proposed change, estimated effort, risk, SLO impact, acceptance criteria.- Non-PIR findings (pager noise, oncall suggestions) can file low-effort candidates via a lightweight form in the same backlog.

Prioritization rules- Auto-score each candidate by: SLO impact (40%), incident frequency (20%), customer impact (20%), effort (20%). Normalize to a 0–100 priority score.- Required bands: Critical (≥75) -> must be scheduled in next sprint; High (50–74) -> roadmap within 4 sprints; Medium/Low scheduled by capacity or batched into improvement sprints.- Tie-breakers: hit to error budget, regulatory/security exposure, and cross-team dependencies.

Owner assignment- Assign to the oncall service owner or the team most aligned with system ownership. If cross-functional, assign a primary owner and a secondary stakeholder from each affected team.- Owners must add technical tasks, estimate effort, and propose rollout/verification plans within one week of intake.

Checkpoints & verification- Definition of Done requires: code change or runbook update, automated test(s), deployment to staging, and a Verification Checklist logged in the ticket.- Verification steps: QA in staging, simulated fault test (if applicable), monitoring/alert rule changes, and a 14–30 day monitoring window post-deploy to ensure no regression.- A small “Improvement Review Board” (rotating SRE + product rep) performs quarterly audits of closed items—random sample of 10%—and can reopen items not meeting acceptance criteria.

Incentives to prevent premature closure- Policy: tickets closed without verification enter a “reopen tax” — they must be reassigned and reprioritized with a 20% added effort penalty.- Scorecard & recognition: teams earn reliability points for verified improvements; accumulate toward quarterly rewards (team budget for tooling/training).- Performance metrics: track fraction of improvements fully verified, mean time from intake→verification, and reduction in recurrence rate. Tie part of oncall rotation bonus and team OKRs to verified improvements, not simply closed tickets.- Psychological safety: blameless postmortems, public dashboards showing verified impact to celebrate wins.

Why it works- Objective scoring reduces bias; ownership clarity avoids orphaned items; verification + audits prevent gaming; incentives align team goals to real reliability impact rather than checkbox closure.

Unlock Full Question Bank

Get access to hundreds of Root Cause Analysis and Corrective Actions interview questions and detailed answers.

Join thousands of developers preparing for their dream job.