Stories demonstrating your ability to drive completion, overcome obstacles, and deliver outcomes despite constraints. This includes managing ambiguity, making progress with incomplete information, and maintaining momentum. At entry level, focus on times you saw something that needed to be done and took initiative, or when you stuck with a challenge until it was resolved.
HardTechnical
55 practiced
You are the SRE lead for a platform used by thousands of downstream services. Uptime SLOs were missed repeatedly last quarter. You must deliver a 3-month plan to materially improve reliability (e.g., from 99.5% to 99.99%) across critical services despite constrained budget and hiring freeze. Outline prioritization, technical investments, process changes, stakeholder buy-in approach, quick wins, and metrics to measure progress.
Sample Answer
Situation: Last quarter our platform repeatedly missed uptime SLOs (avg 99.5%) affecting thousands of downstream services and burning through our error budgets.Goal / Task: In 90 days raise effective reliability to 99.99% for critical services (P0/P1), while constrained by limited budget and hiring freeze.90-day plan (prioritization & timeline)- Week 0–1: Triage & prioritization — run a 2‑day reliability workshop with product owners to identify top 10 critical services by customer impact, frequency of incidents, and error-budget consumption. Freeze non-essential changes on these services.- Weeks 1–3: Quick wins — implement immediate low-cost fixes: - Enforce circuit breakers and client-side retries with backoff in gateways. - Tighten health checks and fail-open/close semantics for load balancers. - Increase alert thresholds to reduce noise; add runbook links to alerts.- Weeks 3–8: Technical investments (low-cost, high-impact) - Introduce lightweight request throttling and graceful degradation patterns. - Automate rollback and canary promotion for deployments (CI/CD config changes). - Centralize and standardize SLO/SLI definitions and dashboards for top services.- Weeks 8–12: Process & hardening - Mandatory post-incident reviews within 48h with concrete remediation owners. - Weekly error-budget reviews with product and platform leads; gate risky changes. - Launch “reliability guild” rotating on-call SMEs to coach teams on resiliency patterns.Stakeholder buy-in- Present impact: translate uptime delta into revenue/customer-impact and developer toil.- Commit to transparent cadence: weekly reliability reports, concise remediation tracker, and a single source-of-truth dashboard.- Offer low-effort developer incentives (office hours, templates, PR reviews) rather than hiring.Metrics to measure progress (and targets)- Primary: SLO compliance for top 10 services: move from 99.5% → 99.99% within 90 days for P0/P1.- Secondary: Mean time to detect (MTTD) ≤ 5 min, mean time to restore (MTTR) reduced by 50%, incident frequency down 40%, error-budget burn rate ≤ planned.- Operational: % of services with automated rollbacks/canaries, % of alerts with runbooks.Trade-offs & risks- Short-term focus on critical services may defer noncritical work; communicate scope clearly.- Some technical debt remains; plan follow-up quarter for deeper platform refactors.Expected result: Within 90 days, achieve measurable SLO improvement for critical services through low-cost automation, process discipline, clearer SLIs, and stakeholder-aligned governance — hitting the error-budget and restoring trust without new hires.
EasyBehavioral
59 practiced
Describe an early-career situation where you were assigned an unfamiliar operational task (e.g., a deployment, database migration, incident lead). Explain how you learned what you needed quickly, who you involved, how you mitigated risk, and the outcomes that demonstrated you delivered results despite being new to the task.
Sample Answer
Situation: In my first SRE role I was asked to lead a production database migration from a single primary to a primary‑replica setup for a customer‑facing service. I’d never run a live DB migration or owned the runbook before.Task: My goal was to perform the migration with zero customer-visible downtime and a safe rollback path.Action:- I quickly mapped the scope: schema size, replication options, maintenance windows, and backup/restore time. I read the vendor docs and previous migrations, then distilled a checklist into a one‑page runbook.- I involved senior SREs and the DB engineer for design review, and scheduled a pre‑migration dry run in a staging environment with the dev team.- To mitigate risk I automated steps (backup, schema validation, replica catch‑up checks) with scripts, created a verified snapshot backup, and defined clear abort criteria and rollback steps. I also opened a Slack incident channel and assigned on‑call coverage for the window.- During the migration I followed the runbook, monitored replication lag and error rates, and had the rollback tested and ready.Result: Migration completed within the maintenance window with <1% latency blip and no data loss. Post-migration monitoring showed stable error rates and a 30% improvement in read throughput. The runbook I created became the team’s standard template and I received mentorship praise for clear documentation and risk management.This taught me to combine rapid learning, targeted mentorship, automation, and clear rollback plans when owning unfamiliar operational tasks.
EasyTechnical
71 practiced
Give an example where you automated a repetitive operational task (e.g., log rotation, cleanup job, deployment step, alert suppression) that improved team efficiency. Describe the problem, the automation you built (language/tooling), how you validated it, and the measurable impact (time saved, reduced incidents, fewer human errors).
Sample Answer
Situation: Our production Kubernetes cluster accumulated orphaned PVCs and old nightly job logs. Every two weeks on-call engineers manually identified and deleted stale PVCs and rotated logs—about 3 hours of combined work and occasional accidental deletion that caused a minor outage once.Task: Reduce manual effort, eliminate human-error risk, and enforce retention policy.Action:- I built a safe automated cleanup service in Python using the Kubernetes client library and cronjob scheduling via Kubernetes CronJob.- Logic: list PVCs and jobs older than a configurable threshold (30 days), cross-check owner references and “protected” label, simulate deletions in dry-run mode, then delete in batches with exponential backoff and rate limits.- Added unit tests for parsing/age logic, integration tests against a kind cluster, and a GitHub Actions workflow to run dry-runs daily and alert via Slack for removals.- Implemented feature flags and an “approve” channel for initial two-week rollout.Result:- Reduced manual cleanup time from ~3 hours biweekly to ~0.5 hours/month (mainly review) — ~90% time savings.- Eliminated further accidental deletions; zero cleanup-related incidents in 9 months.- Team feedback: fewer pager distractions and clearer retention compliance.
EasyTechnical
60 practiced
Tell me about a time you reduced alert fatigue by cleaning up noisy alerts or improving alert routing. Explain how you identified which alerts to change, the criteria you used for suppression or severity change, the steps you took to implement changes, and how you measured improvement (e.g., reduced pages, improved MTA/MTTR).
Sample Answer
Situation: Our team was getting ~150 on-call pages/week from PagerDuty, many for low-impact transient alerts (flapping host checks, noisy 5xx spikes from a 3rd-party) which caused alert fatigue and longer MTA.Task: I was asked to reduce noise and improve signal-to-noise so on-call could focus on real incidents and reduce mean time to acknowledge (MTA) and mean time to recovery (MTTR).Action:- Audit & identify: I exported 90 days of PagerDuty incidents and our monitoring alerts (Prometheus + Alertmanager) and ranked alert rules by page count, duplicate triggers, and average time-to-resolve. Top offenders were 10 rules generating 60% of pages.- Define criteria: For each rule I used: severity impact (SLO violation/immediate user impact), recurrence (how often it pages per week), mean resolution time, and flapping rate. If an alert triggered >10 times/week with <5% leading to actual incident, I flagged for suppression or severity downgrade.- Implement changes: - Suppressed low-impact flapping host checks with a 5-minute for: duration and alertmanager grouping to reduce duplicates. - Downgraded non-actionable alerts (single 5xx spike) from P1->P3 and routed them to a "platform-observe" channel instead of on-call. - Added runbook links and automated remediation (restart job) for a class of known transient failures so human acknowledgement wasn’t required. - Rolled changes as a staged release, monitored for false negatives, and coordinated with dev teams via a changelog.- Measure & iterate: Tracked weekly pages, MTA, MTTR, and SLO burn.Result: Within four weeks pages dropped from 150→55/week (63% reduction). Median MTA improved from 8m→2.5m and median MTTR for high-severity incidents improved from 42m→28m. SLO burn remained stable. We kept a dashboard and periodic review to avoid over-suppression.Learning: Use data-driven criteria, stage changes, automate remediation where safe, and maintain visibility so suppression doesn’t hide real failures.
EasyBehavioral
54 practiced
Tell me about an instance where you had to make a quick operational decision during an incident without all the desired data. What choice did you make (e.g., scale up, rollback, throttle), what factors guided your decision, how did you execute it, and what was the outcome?
Sample Answer
Situation: Late one Friday, our primary API started showing elevated error rates and latency after a deploy to production. Alerts were firing and customer impact was visible in synthetic tests, but detailed logs and tracing were delayed because the logging pipeline was backlogged. We had ~30% of requests failing and load was steady.Task: As the on-call SRE, I needed to restore availability quickly without full observability.Action:- I used available signals: error rate, latency dashboards, recent deploy metadata, and Canary health (canary was healthy), which suggested the issue was systemic but likely related to a traffic pattern change.- Risk assessment: rollback could disrupt ongoing fixes; scaling up would buy time but might hide a bad code path; throttling or rate-limiting user traffic would reduce customer impact but risk SLA breaches.- I chose a conservative two-step approach: first apply request throttling to non-essential endpoints and reduce concurrency for a noisy backend service (quick config change via our load balancer). Simultaneously, I triggered a controlled scale-up of the failing service by increasing pod replica count by 50% to relieve resource pressure while we collected richer diagnostics.- I communicated the actions to stakeholders and set a 15-minute checkpoint.Result: Error rate dropped from 30% to under 5% within 10 minutes, latency returned to near-normal, and the extra capacity allowed logs/traces to catch up. With full traces we identified a memory leak in a third-party library used by a non-critical endpoint; we rolled back that component in the next maintenance window and deployed a fix. Post-incident, I added a synthetic test for that endpoint and an alert to detect logging backlog earlier.Learning: In incidents with incomplete data, combine conservative mitigations (throttling) with temporary capacity increases to buy time for diagnosis, prioritize customer impact, communicate clearly, and convert the root cause into automation or monitoring to prevent recurrence.
Unlock Full Question Bank
Get access to hundreds of Deliver Results / Bias for Action interview questions and detailed answers.
