Understanding the Role and First 90-Day Plans Questions

Framework: treat SRE initiatives as investment projects—estimate costs, map benefits to business metrics, monetize benefits where possible, and compute simple ROI and payback while accounting for uncertainty and time value.

1. Metrics to track

• Direct cost savings: reduced on-call overtime, contractor hours, cloud spend ($)
• Availability & reliability: MTTR, MTTD, incidents/sec, downtime minutes → convert to lost-revenue or SLA penalties avoided ($/min)
• Developer productivity: number of deploys/week, mean cycle time, time saved per engineer → translate to salary-equivalent savings
• Customer impact: churn reduction, NPS, feature adoption lift → revenue impact estimates
• Risk reduction: probability of major outage reduction → expected loss avoided

2. Attribution approach

• Baseline period and trend-adjusted counterfactual (pre/post with seasonality)
• Use A/B or canary rollouts where possible (partial rollout to measure lift)
• Tag work items and incidents to initiatives; track time spent and incident types before/after
• Use conservative attribution (e.g., credit 50–80% of observed improvement to SRE when other changes overlap)

3. Handling time horizons

• Short-term wins: automation for repetitive ops, runbook scripts, alert tuning — fast payback, low uncertainty
• Long-term bets: major architecture refactors, observability platform migrations — high upfront cost, model multi-year NPV, use staged milestones and milestones-based go/no-go
• Discount future benefits (use company WACC) and show payback period and NPV alongside simple ROI

4. Example scoring/prioritization

• Compute expected annual benefit ($) − annualized cost ($) → ROI% and payback months; rank by ROI, but include: risk, strategic value, SLO impact.
• Sensitivity analysis for optimistic/pessimistic outcomes.

5. Concrete examples

• High ROI: Automating database failover and runbooks that cut average incident duration from 60→10 min. Benefit: 50 incidents/year × 50 min saved × $100/min business impact = $250k/yr vs $40k dev time → quick payback.
• Medium ROI: Alert deduplication and tuning reducing paging noise—improves on-call satisfaction and reduces churn; benefits partly qualitative but measurable via reduced escalations and overtime.
• Low ROI: Large-scale migration to a marginally cheaper cloud region with minimal performance gains; high migration risk and long payback—defer or re-scope.

This approach ties SRE work to business dollars, balances short vs long term, and uses conservative attribution and sensitivity checks so leadership can prioritize investments objectively.

SLI definition

• SLI: Fraction of authorization requests that are “successful” within an allowed latency. For each API request in the measurement window:
  • Success = HTTP 2xx with authorization decision returned within 500 ms.
  • Degraded = HTTP 2xx but latency > 500 ms OR HTTP 5xx with automatic retry success within 2 s (counts as partial).
  • Failed = all other outcomes (5xx without retry success, 4xx client errors that are server-caused).
• Numeric SLI:
  SLI = (Success_count + 0.5 * Degraded_count) / (Total_count − Excluded_count)

Measurement window

• Calendar month (e.g., 1st 00:00 to last day 23:59:59 UTC). Use a rolling monthly report for operational visibility but the SLO target applies per calendar month.

SLO and monthly error budget

• SLO: 99.99% availability per month.
• Allowed monthly error = (1 − 0.9999) * measurement_seconds.
  Example (30-day month): seconds = 30243600 = 2,592,000
  Allowed failure seconds = 0.0001 * 2,592,000 = 259.2 s ≈ 4 minutes 19 seconds.
• Using the SLI formula above, any weighted failed-request-time that exceeds 259.2s (or equivalent request-count proportion) consumes the error budget.

Planned maintenance and exclusions

• Pre-scheduled maintenance windows are excluded if:
  • They are announced ≥72 hours in advance,
  • The window is recorded in the maintenance calendar, and
  • The service is taken down via coordinated change control.
• Exclusion method: subtract requests and seconds during the maintenance window from denominator (Excluded_count and excluded measurement_seconds). Limit total excluded time per month (e.g., ≤ 1% of month) to avoid abuse.

Accounting for partial failures (rationale & practice)

• Use weighted counting (0.5 for degraded) to reflect reduced user impact vs total failure.
• Instrumentation: API gateway and auth service emit metrics: request_count, successful_count, degraded_count, failed_count, latency_histogram with labels. Compute SLI from these metrics in Prometheus or metrics pipeline.
• For latency-based degraded definition, store per-request timing; use aggregate counts to compute weighted SLI.

Operational policies

• Alert when error budget burn rate exceeds thresholds (e.g., 50% budget used in first half of month, or burn rate indicating full consumption within 24 hours).
• On budget exhaustion: restrict risky launches, require SRE approval for deploys, increase on-call staffing and mitigation playbooks.
• Post-incident: attribute consumed budget to incidents in postmortem and refine thresholds (latency boundary, retry behavior).

This definition gives a precise numerator/denominator, an exact monthly error budget (example above), rules for excluding planned maintenance, and a reproducible way to handle partial/degraded responses.

Incident Runbook Template (SRE)

1. Header: Incident ID, Service, Severity, Start time, Runbook version
   Purpose: Provide immutable incident metadata for tracking and auditing.

2. Detection criteria
   Purpose: Define the exact alerts/metrics/log patterns that indicate this incident so responders can confidently identify it.

3. Immediate triage checklist
   Purpose: Step-by-step questions and quick commands to confirm scope, blast radius, customer impact, and whether to page additional teams.

4. Mitigation / workaround steps
   Purpose: Fast, minimally invasive actions to reduce customer impact while preserving evidence for root cause analysis.

5. Rollback / safe-recovery steps
   Purpose: Clear commands and criteria to revert recent changes or safely restore service if mitigations fail or cause regressions.

6. Owner and escalation path
   Purpose: Identify the primary on-call owner, secondary contacts, and escalation timeline so responsibility is explicit.

7. Communication templates

• Stakeholder (engineers/ops): short technical status update with current impact, actions in progress, ETA.
• Executives: one-paragraph summary: impact, customer/business effect, immediate action, and expected recovery time.
  Purpose: Ensure consistent, audience-appropriate messaging and reduce cognitive load while communicating.

8. Run commands & links (playbooks, dashboards, logs, runbook scripts)
   Purpose: Centralize the exact tools and URLs needed to act quickly.

9. Post-incident follow-up tasks (RCA owner, data to collect, timeline, action items, SLO impact, next meeting)
   Purpose: Ensure learning, accountability, and remediation are scheduled to prevent recurrence.

10. Notes / timeline (append as incident progresses)
    Purpose: Record decisions and timestamps for transparency and postmortem analysis.

I evaluate fixes using four axes: blast radius, permanence (frequency & root cause), maintenance cost, and testing complexity. For each axis I score whether the change favors a code/system fix or an operational (runbook/temporary automation) solution, then weigh against SLO/error budget and team capacity.

Criteria:

• Blast radius: Large blast radius → prefer code changes with safety controls (feature flags, canary). Small, well-contained impact → runbook or short automation ok.
• Permanence/frequency: Recurring or root-cause fixes → implement in code. One-off or extremely rare events → runbook.
• Maintenance cost: Low ongoing maintenance and economical to own → code. High lifecycle cost (requires frequent updates across teams) → prefer runbook until you can centralize.
• Testing complexity & risk: If fix is easy to test in CI/staging and low risk → code. If it requires complex live-state manipulation or risks side effects → start with automated runbook and iterate to code once procedures stabilize.

Examples:

• Implement in code: A memory-leak mitigation that triggers daily OOMs across many nodes (high blast radius, frequent). We implement automatic circuit breaker and backpressure in service code, guarded by rollout and monitored by metrics.
• Runbook/temporary automation: A vendor API rate-limit that rarely spikes during a specific monthly batch (small blast radius, infrequent). Create a documented runbook plus a small script to backfill/disable offending batches; convert to permanent automation if frequency increases.

I document the decision, track technical debt, and set a review deadline to reassess runbook fixes for codification.

Situation: A major incident started immediately after a deployment and I must choose between rollback vs fix-forward.

Decision framework — evaluate these factors quickly and in order of impact:

• Customer impact: measure scope (percent of requests/users affected), severity (data loss, transaction failure), and SLO breach risk. High-impact or data-loss → favor rollback.
• Stateful migrations: are DB schema or data migrations involved? Destructive/irreversible migrations strongly favor rollback or freeze traffic to prevent corruption.
• Rollback risks: can we safely revert (idempotent deploys, backward-compatible schema, config changes)? If rollback might leave partial state or cause more outage, lean fix-forward.
• Detection latency & blast radius: how long ago did issue start and how fast it’s spreading? Fast-spreading issues push to rollback to stop damage.
• Deployment size & complexity: single-service vs cross-cluster. Small, isolated deploys are easier to rollback quickly.
• Observability & revert testing: do we have reliable health checks and an automated rollback path? If yes, rollback is low-risk.
• Team readiness & coordination: who’s available to execute rollback or patch? Time to mitigation matters.

How I’d make the call

• Rapidly gather metrics (errors, latency, traffic), check migrations, and consult change logs.
• If customer-impacting, reversible deployment, or ongoing data corruption → initiate immediate rollback (automated where possible), notify stakeholders, and open incident channel.
• If rollback risks are high (complex migration) but a contained, fast fix exists (feature flag off, route cut, hotfix) with testable rollback mitigation → implement fix-forward while isolating impacted traffic.

Communication

• Immediately post decision in incident channel with: decision, rationale (key factors), owner(s), ETA for mitigation, and next steps.
• Notify product and support with customer-facing workaround/expected timeline.
• After mitigation, run postmortem focusing on decision rationale, what worked, and improvements (feature flags, safer migrations, automated rollbacks).
  This framework balances safety, speed, and long-term system integrity.