Emotional Intelligence and Relationship Building Questions

1. What exactly is the alert/pager text and its severity (S1/S2/etc.)?

• Why: confirms scope and priority, ensures correct escalation & runbook.
• Degraded vs down: "down" increases urgency and may trigger full incident bridge; "degraded" may allow limited mitigation while investigating.

2. Which service/component and region/cluster is affected?

• Why: narrows blast radius and points to specific telemetry (hosts, k8s namespace, AZ).
• Degraded vs down: for "down" assume global impact and check multi-region failover; for "degraded" focus on regions with elevated errors/latency.

3. When did the issue start and is it ongoing or intermittent?

• Why: establishes timeline, correlates with deployments/config changes and traffic spikes.
• Degraded vs down: "down" often has a sharp start (deployment/rollback candidate); "degraded" may show gradual trend — look at rolling changes or capacity saturation.

4. What customer-facing symptoms / user impact are observed (errors, slow responses, data loss)?

• Why: helps prioritize remediation vs diagnostics and informs stakeholders.
• Degraded vs down: "down" => major outage communications and potential rollback; "degraded" => customer guidance and mitigation (rate-limiting, scaling).

5. Any recent changes (deploy, config, infra, traffic, certs) around the time window?

• Why: changes are highest-probability root causes; quickly identify revert candidates.
• Degraded vs down: "down" makes recent deploys prime suspects; "degraded" also consider load changes or resource exhaustion.

6. Which logs/metrics/alerts are abnormal and pointers to dashboards or runbooks?

• Why: directs where to look first (error codes, 5xx spikes, CPU, latency).
• Degraded vs down: "down" look for node/instance termination and healthcheck failures; "degraded" inspect latency, queue depth, GC, and saturation metrics.

7. Is there a known workaround or mitigation in place (traffic reroutes, feature flags, scaling)?

• Why: prevents duplicate work and may buy time for root cause.
• Degraded vs down: "down" may already have failover engaged — verify health of failover; "degraded" look for temporary throttles or partial rollbacks to stabilize service.

8. Who else is on-call for related services and do we need to assemble an incident bridge?

• Why: ensures cross-team coordination and faster resolution when dependencies are involved.
• Degraded vs down: "down" typically requires immediate multi-team bridge and stakeholder notifications; "degraded" may escalate more selectively based on impact.

Use answers to pick quick actions: check runbooks, gather evidence (screenshots, query outputs), communicate status every 5–10 minutes, and decide whether to mitigate (scale/route/rollback) or continue diagnostics.

Situation: In a war room I see developers pushing a risky quick fix while SRE recommends rolling back to the last known-good release. As incident commander I need a clear decision in 20 minutes.

Immediate goal: Minimize customer impact and overall time-to-recovery while avoiding actions that could worsen the incident.

Step 1 — Rapid information gathering (first 5 minutes)
Questions I ask aloud and record:

• What exactly is the user-visible symptom and scope (regions, customers, endpoints)?
• When did the regression start and what changed in this deployment?
• What telemetry shows (errors, latency, traffic patterns, saturation)?
• What does the quick fix do, and how confident are you in it? (tests, code review, canary)
• How long to fully implement and validate the quick fix end-to-end?
• How long to perform a rollback and any database/migration implications?
• Any non-obvious stateful side effects (schema changes, data loss, cache invalidation)?
• Are there workarounds to mitigate user impact immediately?

Step 2 — Stakeholder consultation (minutes 5–10)

• Dev lead / engineer who authored the fix (technical details, confidence)
• SRE lead (rollback plan, runbook, estimated RTO)
• Product/PM (customer impact / business priorities)
• QA or on-call tester (canary validation)
• Security if the issue touches auth/data
  I synthesize inputs on a single line: estimated time-to-recover (TTR) for each path, risk of escalation, and blast radius.

Decision criteria I use (prioritized, applied in minutes 10–15)

• Time-to-recovery (shorter preferred) and certainty of success
• Customer impact: active errors vs degraded performance; SLA/SLO risk
• Rollback complexity and side effects (migrations, cascading failures)
• Risk of making the situation worse (unknown code paths, stateful changes)
• Ability to validate rapidly (canary or smoke tests)
• Long-term technical debt vs immediate stability

Make the decision (minute 15)

• If rollback has low complexity and high certainty with shorter TTR, choose rollback.
• If quick fix has tested, small blast radius, and can be validated in a canary within acceptable TTR, allow quick fix with strict controls.
  I state the decision succinctly, assign owners, and set concrete success criteria and timers (e.g., “Rollback initiated in 2 min; smoke test in 5; if errors not reduced by X% in 10 min we revert to plan B”).

Communication plan (minutes 15–20)

• To Both Teams (live war room): Announce decision, rationale (criteria above), owner assignments, and exact timeline and rollback triggers. Example: “We’re rolling back because rollback TTR=6 min vs quick-fix TTR≥30 min and rollback has no migrations. Devs: stand by to triage post-rollback. SREs: execute runbook now.”
• To Wider Stakeholders (email/Slack incident channel): Short update with the chosen action, expected customer impact, ETA for next update.
• To Customers (if needed): Prepared templated message via support/ops if incident affects SLAs.
• After action: schedule a post-incident review and capture missed safeguards.

Tone and facilitation

• Stay neutral, data-driven, time-box decisions, enforce binary triggers (if X not met in Y minutes, escalate), and keep delegations explicit. My role is to remove ambiguity, enforce the decision, and ensure rapid validation and clear communication.

I would run a parallel technical and communication plan so customers feel heard while we gather evidence.

Who to involve (internal):

• On-call SRE + primary service owner (developer)
• Observability/telemetry lead (logs/metrics/tracing)
• API/gateway and networking owner if applicable
• Customer success/technical account manager (for customer liaison)
• Escalate to on-call manager if multiple customers or SLAs impacted

What to ask customers (specific, actionable):

• Exact timestamps (including timezone) of observed errors
• Request IDs, correlation IDs, or trace IDs from client logs
• Full request/response snippets (headers, status codes, payloads) redacted for PII
• Client environment: SDK/version, region, IP, retry behavior
• Frequency pattern: single request vs repeated, user impact, reproducible steps

Managing expectations:

• Acknowledge receipt quickly (within 1 business hour)
• Tell them what you will do and by when (e.g., “We’ll investigate for 4 hours and update every 2 hours”)
• Share interim findings even if inconclusive; commit to a final summary
• Offer mitigations/workarounds if available (retries, alternate endpoints)
• Close loop with postmortem and timeline when resolved

Sample phrasing (apology without admitting unknown cause):

• “I’m sorry you experienced this — thank you for reporting it. We’re investigating now and would appreciate the request-ids and timestamps so we can correlate with our traces. We’ll update you in two hours with progress.”
• If confirming action: “We apologize for the disruption. We don’t yet have a root cause, but we’re prioritizing correlation of your request-ids with our traces and will share findings and mitigations as soon as possible.”

Metrics I would track (and why they change behavior)

1. Mean Time to Acknowledge (MTTA)

• What: Average time from alert firing to someone beginning remediation.
• Why it drives behavior: Short MTTA incentivizes on-call readiness, clear alert routing, and owning alerts. If MTTA is long, teams will invest in better alert routing, runbooks, and paging policies.

2. Mean Time to Resolve (MTTR)

• What: Time from alert to incident resolution (or mitigation).
• Why it drives behavior: Highlights the need for reliable runbooks, playbooks, automated remediation, and fewer manual toil steps. Teams see direct impact of investments that speed recovery.

3. Percentage of Incidents Caused by Known Failure Modes (Repeat Rate)

• What: Fraction of incidents matching previously documented root causes.
• Why it drives behavior: High repeat rate forces permanent fixes over firefighting; motivates root-cause remediation, postmortem action completion, and issue tracking.

4. Error Budget Burn Rate (per week)

• What: Rate at which an SLO’s error budget is consumed relative to expected.
• Why it drives behavior: Fast burn signals prioritization of reliability work and throttles feature push; aligns product and engineering trade-offs with measurable business risk.

Short email to engineering managers

Subject: Quick wins to reduce outages — 4 reliability metrics to track this quarter

Hi all —

To make reliability work visible and prioritized, I propose we start reporting four behavioral metrics weekly: MTTA, MTTR, Repeat Rate (incidents from known failure modes), and Error Budget Burn Rate. Each metric ties directly to actions teams can take (better alerting/runbooks, automation, permanent fixes, and release pacing). Example: reducing MTTA by 50% last quarter cut our outage impact window by 30% on service X.

If you’re open, I’ll share a dashboard and run a 30‑minute sync to walk through targets and low-effort improvements each team can implement in 2–4 weeks. Small changes here prevent big outages later.

Thanks — happy to set the meeting and tailor targets per service.

Goal: change interrupting/dismissive behavior into collaborative, respectful on-call participation so incidents resolve faster and team psychological safety improves.

1. Coaching goals (SMART)

• Short term (2 weeks): reduce interruptions during calls by 50%; ask at least one clarifying question before offering a solution in 80% of incidents.
• Medium term (8 weeks): consistently use active listening behaviors and contribute constructively; peer feedback score ≥4/5.
• Long term (3 months): lead incident calls without dismissing suggestions; mentor others.

2. Coaching techniques & schedule

• 1:1 weekly 30–45min sessions for 8 weeks: set expectations, review incidents, role-play, share feedback.
• Shadowing: coach observes 2 on-call rotations; immediate private debrief after incidents.
• Peer feedback loop: rotating buddy gives short written feedback after each shift.

3. Role-play & feedback scripts

• Role-play scenario: noisy incident; coach plays engineer offering idea. Pause when the junior interrupts; rewind and ask them to respond using an agreed script.
• Feedback script (in-the-moment): “When you spoke over Sam earlier, it made it hard to evaluate their idea. Can you tell me what you heard from them?” Follow with: “In future, try holding your thought and asking one clarifying question first.”
• Reinforcement script: “I noticed you waited and asked a clarifying question—that helped us reach a fix faster. Thank you.”

4. Active-listening exercises

• Mirroring drill: after a colleague speaks for 60s, paraphrase their point in one sentence before responding.
• Question-first rule: practice asking exactly one clarifying question before proposing a solution for 5 consecutive incidents.
• Silent-take: count to 3 before replying; coach times and logs compliance.
• Pair retell: during post-incident, each participant summarizes the other’s contribution.

5. Measurement & success metrics

• Behavioral metrics: % of incidents with interruption events (tracked via meeting transcript or observer notes), % of responses that begin with paraphrase/clarifying question.
• Perception metrics: weekly peer-feedback scores, post-incident psychological-safety pulse (1–5).
• Operational metrics: mean time to acknowledge/respond, mean time to resolve (monitor for no negative impact).
• Review cadence: weekly review of metrics in 1:1; 8-week formal evaluation.

6. Escalation plan if no improvement

• After 8 weeks with <30% improvement: formal performance improvement plan (PIP) with clear expectations and timelines, required training (communication workshop), and limited solo on-call duties.
• If still insufficient after PIP: remove on-call privileges and reassign role responsibilities until remedial training completed; involve manager/HR for next steps.

This plan combines clear goals, practice, real-time feedback, measurable outcomes, and fair escalation so the engineer can learn constructive behaviors while protecting team reliability and safety.