Practices for building and maintaining relationships with stakeholders, achieving alignment on goals scope timelines and success criteria, and managing expectations across functions and levels. Topics include tailoring communication and metrics to different audiences, negotiating trade offs and realistic timelines, coaching partners on prioritization, documenting decisions and governance, handling scope creep and midstream changes, maintaining transparency with roadmaps status reports and decision logs, and establishing escalation protocols. Candidates should show tactics for earning buy in without formal authority, coordinating operational handoffs, protecting teams from unnecessary friction, and measuring the health and effectiveness of stakeholder relationships and long term alignment.
HardTechnical
61 practiced
You must present a decision to stakeholders: re-architect a brittle subsystem (6–9 months, high cost) versus implement short-term remediation (1 month, increased ongoing maintenance). Create a one-page decision memo summarizing technical trade-offs, risk profile, cost-benefit analysis, timelines, KPIs to monitor during a short-term fix, and recommended oversight if you choose the remediation route.
Sample Answer
Decision Memo: Re-architect vs Short-term RemediationContext: A brittle subsystem causing frequent incidents, degraded performance, and operational overhead. Two options: Full re-architecture (6–9 months, high upfront cost) or short-term remediation (≈1 month, lower upfront cost, higher ongoing maintenance).Technical trade-offs- Re-architect: - Pros: Eliminates root-cause design limitations (scalability, single points of failure), reduces long-term MTTR and operational toil, easier to automate, supports future feature velocity. - Cons: Large scope, integration risk, resource diversion, deployment complexity, potential regression/new bugs.- Remediation: - Pros: Fast risk reduction, immediate incident reduction, preserves roadmap momentum, lower initial cost. - Cons: Band-aid fixes increase technical debt, higher cumulative maintenance cost, fragile workarounds may mask latent bugs.Risk profile- Re-architect: Higher project risk (schedule slippage, integration), lower operational risk post-delivery. Business risk during migration window.- Remediation: Lower near-term project risk, higher medium-term operational and reliability risk (recurrent incidents, capacity constraints).Cost-benefit (30-month horizon)- Re-architect: Upfront dev cost C_high; operational cost O_low. Total ≈ C_high + 30*O_low.- Remediation: Upfront cost C_low; operational cost O_high. Total ≈ C_low + 30*O_high.Estimate breakeven if 30*(O_high - O_low) > (C_high - C_low). Qualitatively, if remediation keeps causing weekly incidents (≥ X hours/person effort), re-architecture likely more cost-effective within 18–30 months.Timelines- Re-architect: 6–9 months (design 4–6 weeks, implementation iterations 4–7 months, testing/migration 4–8 weeks).- Remediation: 2–6 weeks (identify hotspots 1 week, implement fixes 1–3 weeks, testing/rollback 1 week).KPIs to monitor during remediation (must be instrumented before deployment)- Availability: 99.9% service uptime (SLO target baseline)- MTTR: median and 95th percentile incident resolution time (goal: reduce by 30% within 4 weeks)- Incident frequency: number of P1/P2 incidents per week (target: ≤ baseline/2)- Error rate: request error rate and downstream failure rate- Latency: p95 and p99 end-to-end latency for critical flows- Operational toil: engineering hours/week spent on incidents and manual fixes- Capacity/utilization: thread/queue/CPU/backlog metrics for hotspotsInstrument dashboards and alerts with thresholds and alert fatigue controls.Recommended oversight for remediation route- Appoint a remediation owner (SRE lead) with weekly 30–60m stakeholder sync.- Define a 60–90 day remediation runway with two gates: - Gate 1 (2 weeks): observable KPIs instrumented and initial fixes applied. - Gate 2 (8 weeks): KPI targets met OR decision to escalate to re-architect.- Enforce timeboxed fixes only (no feature creep); track technical debt explicitly (JIRA epics).- Run blameless reviews after each incident; weekly trend review with Product, Engineering, and Finance.- Reserve budget for emergency rollback and for a scoped proof-of-concept re-architecture spike if remediation fails.RecommendationIf leadership requires immediate risk reduction and cannot fund a long project now, proceed with timeboxed remediation with strict KPI gates and the oversight above. If incident toil, cost, or business growth indicates sustained strain (projected breakeven < 24 months), prioritize re-architecture to reduce long-term risk and costs.
EasyTechnical
80 practiced
You lead an incident postmortem for a Sev2 outage affecting a subset of customers. Describe how you'd tailor the postmortem content and format for three audiences: on-call engineers, product managers, and executives. For each audience include which metrics, level of technical detail, remediation actions, and follow-up cadence are appropriate.
Sample Answer
For a Sev2 outage affecting a subset of customers I’d produce three tailored postmortems:1) On-call engineers- Metrics: error rate by endpoint, latency percentiles (p50/p95/p99), request volume, CPU/memory, instance counts, rollback/deploy timestamps, alert firing timeline.- Technical detail: full timeline, traces, logs, stack traces, config diffs, exact code/infra change, reproducer steps, runbook gaps.- Remediation actions: immediate hotfix/rollback steps, automation to prevent recurrence (tests, schema checks), improved alerts with runbook links, throttling or circuit-breaker changes.- Follow-up cadence: short daily sync for 3–5 days until fixes validated; post-change verification runbook and owner assigned.2) Product managers- Metrics: customer impact (users affected, sessions lost), feature usage affected, duration, SLA/SLO breach risk, business KPIs (revenue/transactions) if relevant.- Technical detail: high-level root cause and why this feature subset was impacted; no low-level logs.- Remediation actions: timeline for customer-facing fixes, compensations if needed, prioritization of reliability vs feature work.- Follow-up cadence: status updates at 24h and weekly until deployment complete; include verification metrics and customer communications.3) Executives- Metrics: scope (percent of customers), duration, business impact estimate, SLO breach and error budget consumption, recurrence risk (low/medium/high).- Technical detail: one-paragraph cause, key systemic weakness (process or architecture), not code-level.- Remediation actions: summary of mitigations (short-term containment + long-term investments), required resources or trade-offs, proposed timeline and expected risk reduction.- Follow-up cadence: single executive summary within 48h, then weekly until key milestones met, and a 30/90-day review to validate ROI.Across all audiences: assign clear owners, deadlines, and acceptance criteria for each remediation; publish links to detailed artifacts for those who want deeper dives.
HardTechnical
82 practiced
Multiple teams define 'request success' differently, causing inconsistent SLIs across the platform. Design a harmonization process: discovery phase to catalog definitions, a canonical taxonomy of SLI types, mapping of legacy SLIs to canonical ones, governance for future SLI changes, and tooling for enforcement and reporting.
Sample Answer
Situation / overview: Multiple teams measure "request success" differently (HTTP 2xx, business validation, partial successes), causing inconsistent SLIs, noisy alerts, and unreliable SLOs. I’d implement a five-part harmonization program: discovery, canonical taxonomy, mapping, governance, and tooling/enforcement.Discovery (catalog)- Run automated inventory: scan Prometheus/OpenMetrics exporters, Grafana dashboards, alert rules, service repos, and SDK usage to extract existing SLI definitions and labels.- Interview 1:1 with owners for ambiguous cases and capture business intent (what user outcome they care about).- Output: a machine-readable catalog (YAML/JSON) listing service, endpoint, metric name, labels, current definition, owner, rationale.Canonical taxonomy of SLI types- Define a small orthogonal set: Availability (success/failure), Latency (pXX, p95, p99), Correctness (business validation), Freshness (staleness/TTL), Throughput (QPS/backpressure).- For each type specify canonical metric shape (gauge, counter, histogram), semantic labels, and recommended aggregation (per-route, per-customer tier).- Provide concrete examples: "Availability = count(successful_user_visible_responses)/count(total_user_requests) where success = HTTP 2xx AND business_flag=true".Mapping legacy SLIs to canonicals- Create transformation rules: pattern-based rewrites and adapter functions to compute canonical metrics from existing metrics (e.g., combine error counters + validation counters).- Implement a migration plan: dual-writing period where services emit legacy + canonical metrics; compare and reconcile for 2-4 prod cycles.- Example rule: legacy metric http_requests_total{code=~"2.."} -> availability_success = sum_over(label) of matching counts; for services with business-level validation, multiply by validation_passed counter to get correctness SLI.Governance for future SLI changes- SLI change process: changes via PR against a central "SLI Catalog" repo. PR must include owner, reason, impact analysis, and test plan.- Approval flow: automated checks + SLI Review Board (rep from SRE, Product, Security) approves high-impact changes; low-impact (non-customer-facing) auto-approve after CI checks pass.- Versioning + audit: catalog entries are versioned; changelog and approval metadata stored; grandfathering rules for historical SLOs.Tooling & enforcement- Central SLI library (language SDKs) that exposes canonical metrics/labels and helper functions; teams import to emit correct metrics.- Exporter adapters: Prometheus recording rules to derive canonical metrics where direct emission impossible.- CI gates: repo linter that validates metric names, labels, and that any SLI change has an approved PR; policy-as-code (Open Policy Agent) enforces standards before merge.- Dashboards & reporting: central Grafana with templated SLO panels, SLI health views, and SLA reports; weekly automated reports with drift detection between legacy and canonical metrics.- Alerts: SLO-based alerting (error budget burn rate); map legacy alerts to canonical ones during migration to avoid duplication.Rollout and success metrics- Pilot with 3 services (frontend, payments, search). Dual-write for 2 weeks, reconcile, adjust rules, then onboard more teams.- KPIs: percentage of services emitting canonical SLIs, reduction in false-positive alert rate, alignment score (manual audit), improved SLO accuracy. Target: 90% coverage in 3 months, 50% fewer noisy alerts.Trade-offs and risks- Upfront engineering cost and temporary duplicate metrics/alerts. Mitigation: prioritize high-impact services first, provide SDK and migration playbooks, and automate recording rules.This approach combines people/process (governance and owners), clear technical standards (taxonomy + SDK), and automation (adapters, CI gates, dashboards) so SREs and product teams share a single, auditable definition of "request success."
HardSystem Design
63 practiced
Design a resilient, low-latency escalation protocol that spans three support tiers (on-call engineers, platform SRE, executive operations) and multiple regions. Include notification channels, thresholds that trigger each tier, SLA targets for escalation times, failover if primary contacts are unreachable, and audit logging to verify escalations occurred.
Sample Answer
Requirements & constraints:- Global, multi-region coverage with <5 min detection-to-first-notify for Sev1; tolerate region loss; three escalation tiers: On-call Engineers (Tier 1), Platform SRE (Tier 2), Exec Ops (Tier 3).- Reliable delivery, auditability, low-latency, automated failover, and correlation across regions.High-level design:- Monitoring -> Alerting Engine -> Escalation Orchestrator -> Notification Gateway -> Recipients- Centralized Escalation Orchestrator deployed multi-region (active-active) with strongly consistent datastore (e.g., CockroachDB) for canonical state and audit logs; regional cache for latency.Trigger thresholds & SLA targets:- Sev1 (P0): service-down / latency > 5x SLO or error rate spike > 5σ. Escalation timeline: 0–2 min notify Tier1; if no ACK by 5 min escalate Tier2; if unresolved/ no ACK by 15 min escalate Tier3. Target: 95% first-ack within 2 min, full Tier3 notification within 15 min.- Sev2 (P1): partial degradation. Notify Tier1 within 5 min; escalate to Tier2 after 30 min; Tier3 only if 2+ hours or business impact grows.- Sev3: informational.Notification channels & delivery:- Primary: mobile push + SMS + phone call (voice), secondary: email, tertiary: Slack/Teams + incident portal. Use provider multiplexing (Twilio + PagerDuty + internal SMS gateway) to avoid single vendor failure.- Messages include incident ID, impact summary, playbook link, and one-click ACK/CLAIM endpoint.Failover & unreachable handling:- Orchestrator retries exponential backoff + alternate channel sequentially. If primary contact unreachable, escalate to on-call buddy list for same region, then cross-region on-call pool, then Tier2.- Health checks on notification providers; if provider outage detected, automatically switch to backup provider via feature-flagged routing.Audit logging & verification:- Immutable audit stream (append-only) written to distributed ledger (WAL -> S3 + Kafka + long-term DB). Each escalation event logs: timestamp, alert id, recipients attempted, channel, delivery status, ACK/CLAIM timestamps, escalation actions, operator notes.- Provide an incident verification API and dashboard showing timeline, latencies, and SLA compliance metrics. Retain logs 1 year; exportable for postmortem.Operational controls & safety:- Rate limits to avoid notification storms; deduplication by fingerprinting; automated suppression windows during maintenance.- Regular runbooks, monthly simulated drills (dark-launch and game days) to validate timings and on-call rotations.Trade-offs:- Strong consistency increases cross-region latency slightly but ensures single source of truth for escalation state; mitigated by local caching for fast reads. Using multiple providers increases cost but materially improves resilience.
EasyTechnical
61 practiced
What information should be included in an SRE decision log (architectural decision record) to ensure traceability and stakeholder alignment? Provide a template with fields (for example: decision summary, alternatives considered, owner, date, impact, rollback criteria, stakeholder approvals) and explain why each field matters.
Sample Answer
Include the following fields in every SRE decision log (Architectural Decision Record) to ensure traceability, clear rationale, and stakeholder alignment:- Decision ID: unique, short identifier (e.g., ADR-2025-012). Why: enables trace links from incidents, runbooks, PRs, and changelogs.- Title / Summary: one-line description of the decision. Why: quick scan for relevance.- Date & Status: decision date and current status (proposed/approved/deprecated). Why: shows lifecycle and when to re-evaluate.- Owner / Decision Maker: person or team responsible. Why: clarifies who enforces and answers follow-ups.- Context & Problem Statement: background, constraints, metrics/SLOs affected. Why: links decision to operational needs and accepts trade-offs.- Decision: concise statement of chosen approach. Why: definitive record of what was decided.- Alternatives Considered: short list of alternatives with pros/cons and estimated effort/risk. Why: documents trade-offs and rationale.- Impact & Scope: systems, teams, SLOs, cost, security, compliance, and rollout windows. Why: helps stakeholders assess consequences and plan work.- Implementation Plan & Owners: steps, timeline, required changes (runbooks, CI, infra). Why: enables execution and coordination.- Rollback / Mitigation Criteria: explicit triggers and steps to revert or mitigate if issues arise. Why: reduces blast radius and speeds recovery.- Monitoring & Validation: metrics, dashboards, alerting, success criteria. Why: measurable verification that decision meets goals.- Dependencies & Related ADRs/PRs/Runbooks: links to code, tickets, docs, and previous decisions. Why: traceability across systems.- Stakeholder Approvals / Review Log: names, dates, and any objections or conditions. Why: evidences alignment and accountability.- Review / Re-evaluation Date: when to revisit (e.g., 6–12 months). Why: prevents stale decisions from persisting.Using this template ensures decisions are actionable, auditable, and connected to reliability goals—reducing ambiguity during incidents and simplifying future changes.
Unlock Full Question Bank
Get access to hundreds of Stakeholder Management and Alignment interview questions and detailed answers.
