Cultural Fit and Working Style Questions

Five-point executive pitch (key messages)

1. Outcome: Cut outage frequency by 50% within 12 months — directly improving availability, customer trust, and revenue predictability.
2. ROI: Fewer outages reduce incident cost (engineering hours, SLA credits, lost sales) and speed up feature delivery long-term by reducing context-switch cost. We expect payback within 9–12 months.
3. Scope & approach: Focused program—SLO tuning, error budget governance, chaos testing, automated runbooks, and backlog of top-10 reliability fixes—delivered in quarterly milestones.
4. Trade-off transparency: Some feature schedules shift short-term; we’ll use error budgets to prioritize urgent features and preserve roadmap visibility.
5. Measurement & governance: Monthly executive dashboard + quarterly review; we’ll course-correct with data, not opinions.

Anticipated objections and responses

• “We can’t delay revenue features.” Response: Quantify outage cost vs. feature value; use canary releases and error-budget gating to release high-value features safely.
• “This is just engineering overhead.” Response: Frame as risk mitigation with measurable ROI and legal/brand protection.
• “How do we know 50% is achievable?” Response: Pilot data from similar efforts and targeted fixes yield conservative 30–60% in industry benchmarks.
• “Will this slow engineering morale?” Response: Reduced on-call fatigue increases productivity/retention; include developer enablement as a program deliverable.
• “What if it fails?” Response: Stage gates, pilot, and kill-switch metrics ensure we stop or pivot early.

Data / KPIs to include

• Outage frequency (incidents/month) and MTTR — baseline and target (−50% frequency, MTTR −30%)
• Customer-facing uptime (%) and SLA breaches / credits avoided
• Incident cost per month (engineering hours × fully-loaded cost + revenue impact)
• Number of high-severity incidents and time to resolve runbook coverage %
• Developer context-switch time and sprint velocity impact pre/post pilot
• Error budget consumption rate and percentage of releases blocked/allowed by policy

Closing ask: Approve a 12-month pilot with $X budget for tooling, 0.5–1.0 FTE SRE lead, and quarterly checkpoints to validate ROI and rebaseline priorities.

Workshop goal: teach blameless postmortems and drive adoption across engineering teams so incidents become learning opportunities and reliability improves.

Agenda (3 hours):

• 0:00–0:15 — Intro: objectives, psychological safety, what “blameless” means
• 0:15–0:35 — Principles: incident taxonomy, RCA vs. contributing factors, SRE guardrails (SLOs/error budgets)
• 0:35–1:05 — Walkthrough of a strong postmortem (real anonymized example)
• 1:05–1:20 — Break
• 1:20–2:00 — Exercise 1: tabletop postmortem on a supplied incident (roles: incident commander, reporter, engineer)
• 2:00–2:30 — Exercise 2: write a concise postmortem using template; peer review for tone/actionability
• 2:30–2:50 — How to turn postmortems into work: prioritization, playbook updates, small experiments
• 2:50–3:00 — Next steps, distribution channels, office hours, Q&A

Practical exercises:

• Tabletop simulation with timeline reconstruction, “5 whys” → identify contributing factors (not individual failures)
• Rapid write: 10–15 minute postmortem draft focusing on timeline, impact, root contributing factors, remediation, and prevention
• Peer review to flag blaming language and convert into systemic wording

Templates to distribute:

• One-page postmortem template: Summary, Timeline, Impact (users/metrics), Contributing factors, Action items (owner + ETA), Detection lead time, Follow-ups
• Action-item tracker with risk, priority, verification criteria
• Rubric for blameless language + checklist for “good” postmortem
• Playbook/update PR template to link postmortem to code/docs

Metrics to track (adoption & effectiveness):

• Adoption: % incidents with postmortems within X days; % postmortems using template; number of teams regularly submitting
• Quality: median time-to-postmortem; rubric score distribution (completeness, blamelessness, actionable items)
• Impact: % action items completed on time; changes linked to reduced recurrence; SLO/error budget trend pre/post
• Cultural: survey scores on psychological safety and perceived learning from incidents (quarterly)

Rollout tips:

• Pilot with one team, collect feedback, iterate templates
• Make postmortems visible (read-only dashboard) and celebrate learning (not people)
• Offer monthly office hours and a lightweight async review to sustain adoption.

Clarify constraints and goals:

• Requirements: protect legal exposure, follow PR guidance on limited disclosure, preserve customer trust and public confidence, restore system reliability.
• Constraints: legal/PR must approve external messaging; account team needs to retain customer; engineering must fix root cause quickly.

Plan (coordination + timeline):

1. Triage (0–1 hour): SRE leads a technical incident summary—impact scope, affected services, mitigation steps taken, current status, next milestones. Share in a private channel with PR, Legal, and Account Manager (AM).
2. Joint decision call (within 1–2 hours): PR, Legal, AM, SRE, and Engineering manager align on what can be said publicly vs. privately to the customer. SRE presents factual, non-speculative technical points and ETA for remediation.
3. Dual-track messaging:
   • Private customer message (AM + PR, reviewed by Legal): transparent impact, immediate mitigation, remediation ETA, compensation/next steps, single point of contact. Tone: empathetic, accountable, technical specifics appropriate to customer level.
   • Public statement (PR + Legal, technical sign-off by SRE): brief acknowledgement, scope, status, commitment to investigation, invite affected customers to a private channel. Avoid admission of negligence or legal language that increases liability.
4. Remediation & Evidence (ongoing): SRE documents logs, timelines, root cause hypothesis, and remediation steps in an incident timeline to support both PR messaging and Legal review.
5. Escalation & Compensation (24–72 hours): AM negotiates with customer using documented remediation plan; SRE provides technical assurance (post-fix tests, SLOs). Legal approves any contractual remedies.
6. Post-incident review (within 7 days): blameless RCA with PR and Legal invited to understand communications impact; produce a public postmortem sanitized for legal review and approved by PR/Legal and the customer if required.

Example content snippets:

• Private to customer: “We deeply regret the repeated outages. Our SRE team has identified [high-level cause], implemented [mitigation], and expect full restoration by [ETA]. We will provide hourly updates and a dedicated technical contact. We propose [compensation].”
• Public tweet: “We’re aware of service disruptions affecting some customers. Our teams are actively investigating and mitigating. We’ll share a detailed update soon and are available directly to affected customers.”

Why this works:

• Aligns stakeholders early to avoid contradictory messages.
• SRE supplies factual, time-stamped evidence to protect company legally.
• AM/PR preserve customer trust by offering direct, empathetic communication and tangible remediation.
• Legal reduces exposure by controlling admissions and approvals.

Metrics/controls:

• Establish a fast-path approval workflow (e.g., 1-click legal sign-off in incident channel).
• Track time-to-first-response public/private, time-to-remediation, customer satisfaction post-incident, and follow-up SLA credits.

Situation: A postmortem showed repeated delays during incidents because multiple microservices had ambiguous owners — alerts bounced between teams and runbooks conflicted.

Approach / Task: As the SRE lead on the postmortem action items, I needed to (1) clarify ownership, (2) update a canonical service-ownership matrix (RACI), and (3) communicate and enforce the changes so future handoffs wouldn’t fail.

Actions I took:

• Rapid discovery: I ran an inventory of impacted services (logs, alert routes, git service dirs) and listed current “implicit” owners from recent deploys and pager rotations.
• Stakeholder workshop: Convened product, platform, and team leads for a 60–90 minute session to review each service’s critical flows, dependencies, and operational responsibilities.
• Drafted RACI per service: For each service I proposed Responsible (primary on-call), Accountable (team lead that makes changes), Consulted (upstream/downstream teams), and Informed (product/ops stakeholders). I used explicit criteria for Responsibility: deploy authority, on-call commitment, and codebase ownership.
• Governance & sign-off: Sent the draft RACI to stakeholders for 48-hour review, captured objections, and iterated until teams formally signed off in PR comments.
• Source of truth & automation: Committed the RACI matrix and updated runbooks to a repo under “/operations/service-ownership.md”. Updated alert routing and on-call schedules in PagerDuty/OpsGenie to match Responsible entries. Added badge in service README linking to owner and runbook.
• Communication & training: Announced changes in engineering-all and in dedicated Slack channels, ran a 30-minute walkthrough for on-call teams, and added a checklist to on-call playbooks (how to escalate if ownership is incorrect).
• Follow-up cadence: Added this as a recurring quarterly review item and included ownership validation as part of any service onboarding checklist.

Result / Metrics:

• Within one month, incident triage time improved: mean time to acknowledge (MTTA) dropped ~35% and incidents reopened due to handoff confusion fell to zero for those services.
• Teams reported clearer expectations; future postmortems referenced the RACI matrix as authoritative.
• Key learning: Ownership must be explicit, version-controlled, and tied to tooling (alert routing + on-call) to be effective — social agreement alone doesn’t scale.

Situation: During a Sev1 outage I ran the incident bridge for our customer-facing API to restore service quickly and avoid duplicated effort.

Roles I assign (clear, minimal overlap):

• Bridge Lead (Incident Commander): owns scope, priorities, and final decisions. Keeps the bridge focused and prevents scope creep.
• Scribe (Recorder): documents timeline, actions taken, commands run, ownership, and timestamps in a shared incident document (Google Doc/Confluence) visible to all.
• Communications Lead (Comms): writes customer/internal updates, syncs with PM/ops, posts to status page and Slack channel.
• Tech Leads / Investigators: domain experts for networking, DB, app, infra; each reports findings and actions to Bridge Lead.
• Runbook Owner / Operator: executes safe remediation steps (rollbacks, restarts) only after Bridge Lead authorizes.
• Observer / Escalation Liaison: monitors external dependencies and escalates to execs if needed.

Information flows I enforce:

• One source of truth: the scribe document. All diagnostic output, commands, and decisions get copied there.
• Verbal updates only for critical changes; every action must be logged immediately by scribe.
• Investigators report findings to Bridge Lead; Lead decides and instructs Runbook Owner to execute.
• Comms drafts updates from scribe + Bridge Lead approval, then publishes.

Status updates strategy to minimize duplication/misinformation:

• Fixed cadence: brief status every 10 minutes (or agreed cadence) and on-state-change.
• Use a single incident Slack channel and pin the scribe doc and current status line.
• Status template: impact, user-visible symptoms, recent actions, next actions, owner, ETA.
• Enforce “announce, don’t act”: No one executes a change unless Bridge Lead has announced it on the channel and the scribe logged it.
• Post-incident: scribe final timeline + action items, then schedule blameless postmortem.

This structure keeps decision paths clear, prevents simultaneous conflicting actions, and ensures stakeholders get consistent, timely information.