Assess a candidate's structured approach to solving problems and their ability to communicate their thinking clearly, regardless of whether the problem is technical, analytical, or business in nature. Look for: clarifying requirements and open questions before diving in, explicitly stating assumptions, breaking a complex or ambiguous problem into smaller components, proposing and comparing multiple approaches, explaining trade offs in plain language, narrating reasoning step by step as the work progresses, verifying a proposed solution (including edge cases, failure modes, or counterexamples), and adapting the approach when new information or constraints appear. Emphasis is on logical rigor, the ability to adjust the level of detail for different audiences (technical peers vs non-technical stakeholders), and continual communication so the interviewer can follow the candidate's reasoning and decisions throughout.
EasyTechnical
66 practiced
An internal analytics pipeline reports stale data because database replication lag has grown unpredictably. As the SRE on-call, break this problem into investigation components (e.g., monitoring, workload, schema changes, network, resource contention). For each component, list the exact data you would collect, owners to contact, and the first three diagnostic steps you would perform to narrow down the root cause.
Sample Answer
I’d break the investigation into five components. For each I list exact data to collect, owners to contact, and first three diagnostic steps.1) Monitoring/alerts- Data: replication lag time series (seconds), alert history, host timestamps, Prometheus metrics (replica_io_wait, replica_sql_delay), Grafana dashboards, recent deploy/change logs.- Owners: Monitoring/observability team, on-call DBA.- Steps: 1. Verify alert validity and correlate lag spikes with timestamps across systems. 2. Check metric cardinality: which replica(s) affected, host vs region scope. 3. Pull recent deploys/config changes around spike times.2) Workload- Data: query frequency, slow-query log, long-running transactions, batch job schedules, ETL job logs, transaction rates (TPS).- Owners: Analytics team, ETL owners, application service owners.- Steps: 1. Identify heavy queries or jobs starting at lag onset via slow-query and processlist. 2. Correlate ETL/job start times with lag pattern. 3. Temporarily pause or throttle suspected jobs to see if lag improves.3) Schema/DDL changes- Data: recent DDLs, schema migration history, index rebuild events, binlog/replication format.- Owners: DB schema owners, release manager.- Steps: 1. Check for recent ALTER/INDEX operations around the time lag grew. 2. Inspect binlog size and events per second for DDL spikes. 3. If DDL found, review whether it caused table locks or large binlog events.4) Network- Data: packet loss, RTT, interface errors, route changes, MTU, cloud network events, VPN/peering logs.- Owners: Network/Cloud infra team, on-call network engineer.- Steps: 1. Run traceroute/ping between primary and replicas; check cloud network incident dashboard. 2. Inspect host network interfaces for errors and NIC saturation. 3. Correlate lag with network packet drops or maintenance windows.5) Resource contention (CPU/IO/memory)- Data: iostat, vmstat, top/ps metrics, disk queue length, disk latency, CPU steal, IOPS, FS utilization, cgroup limits.- Owners: Host/infra SRE, storage team.- Steps: 1. Check replica and primary host CPU, disk latency, and I/O queues at lag times. 2. Look for noisy neighbors: other processes consuming I/O/CPU or backups/snapshots. 3. If I/O-bound, move replica to different disk, limit backups, or scale IOPS and observe change.Across all components:- Correlate timelines to find causation, not just concurrency.- If immediate mitigation needed: promote healthier replica for analytics, throttle ETL, or temporarily re-route traffic.- After resolution, run a blameless postmortem with owners and add targeted alerts (e.g., sudden binlog growth, long DDLs, sustained disk latency).
HardTechnical
72 practiced
A noisy alert triggers 60 times per day. The team suggests simply raising the threshold to silence it, but you worry that will hide regressions. Design an experiment and rollout plan to reduce noise while minimizing the risk of missing real incidents. Include hypothesis, metrics to measure (false positive/negative rates, pages/hr), canary rollout steps, rollback criteria, and how you'd communicate results to stakeholders.
Sample Answer
Hypothesis:Tuning the alert (smarter thresholds + short-term suppression + contextual filters) will reduce noisy pages by ≥80% without increasing missed real incidents (false negatives ≤ baseline + 5%).Metrics to measure (baseline then during experiment):- Pages/hour and pages/day (primary noisy-signal metric)- True positive rate, false positive rate, false negative rate (defined by incidents correlated to on-call pages and postmortem-confirmed outages)- Time-to-detect and time-to-acknowledge for real incidents- Alert fatigue: % of pages acknowledged within 5 minutes vs ignored- SLO/error-budget impact (errors/sec or latency percentiles related to this alert)Experiment design:1. Capture baseline for 14 days: record every alert, associated signal values, whether it correlated to an incident, and operator action.2. Build deterministic replayable filter rules and a secondary ML/anomaly detector on historical data to propose improved thresholds and contextual suppression (e.g., suppress when upstream known-degraded flag set).3. Create two parallel alert channels: A (current) and B (tuned). For canary, route 5% of service instances to emit only tuned alerts; keep current alerts on all.Canary rollout steps:- Canary-0 (lab): Run tuned logic on historical data and simulated traffic — verify expected reduction and no missed incidents.- Canary-1 (5% traffic, 48 hours): Enable tuned alerts to a small subset; duplicate alerts still sent to a shadow log. On-call receives both original and tuned alerts but tuned is marked “canary”.- Canary-2 (25% traffic, 3 days): If metrics satisfied, expand. Continue shadowing and compare.- Canary-3 (100% traffic, 7 days): Full rollout if safe.Statistical criteria:- Require at least 90% confidence (A/B test) that pages/day decreased and false negative rate not increased beyond +5% absolute.- Minimum observation windows: 48 hours for short signals, 7 days to capture weekly patterns.Rollback criteria:- Immediate rollback if: - Pages/hr caused by tuned alerts for confirmed incidents increases by >5% absolute vs baseline OR - Any missed production incident attributable to the tuned alert (false negative confirmed) - Time-to-detect for real incidents increases >25% - Error budget burn rate increases above predefined threshold- Automated: if canary cohort shows X continuous missed detections (e.g., 2 confirmed misses in 24h), revert.Validation & monitoring:- Shadow mode: always keep original alert logic in shadow for 14 days post-rollout to compare.- Dashboard: real-time comparison of TP/FP/FN, pages/hr, SLO metrics, with alert annotations.- Post-canary review: runbook for on-call to annotate each canary alert as TP/FP/FN to improve classifier.Communication plan:- Pre-experiment RFC to stakeholders (prod eng, SRE, on-call, PM): scope, hypothesis, metrics, rollback plan, timeline.- Daily brief during canary to on-call and execs via summary channel; immediate paging for rollback triggers.- Weekly stakeholder report with dashboards, statistical test results, examples of alerts suppressed and any near-misses.- Post-rollout postmortem and decision memo: publish results, lessons learned, and updated runbooks.Outcome targets:- Reduce noisy pages from 60/day → ≤12/day within rollout window.- Maintain detection fidelity (FN increase ≤5% absolute) and preserve SLOs.This approach balances conservative canary rollout, statistical validation, operator feedback, and clear rollback rules to reduce noise without hiding regressions.
MediumSystem Design
77 practiced
Design a monitoring dashboard for the notifications system that will be used by on-call engineers and executives. Specify which metrics (e.g., p50/p95/p99 latency, success rate, queue depth, SLO burn rate, alert counts) you would show, the visualization type for each, and how you would create two views (detailed for engineers, summary for execs). Also describe alert thresholds and who they should notify.
Sample Answer
Requirements & scope:- Support on-call incident response (detailed, high-cardinality, drillable)- Executive summary (health, trends, SLO status)- Real-time + historical (last 1h/24h/30d) and alerting integrationEngineer (detailed) view — panels and visualizations:1. Latency histograms & percentile time series: p50/p95/p99 line chart (timeseries with percentile overlays), plus heatmap of request latency by endpoint. Purpose: triage slow endpoints.2. Success rate (200s vs errors) timeseries and per-team stacked bar breakdown.3. Queue depth / backlog gauge + time series per queue (gauge + area chart).4. Throughput (req/sec) heatmap and timeseries.5. Error breakdown: pie/chart and top-N error types (table) with recent traces links.6. SLO burn rate chart (line) with shaded error-budget window.7. Alert activity feed and recent incidents list (table).8. Correlated infra: CPU/memory of workers, consumer lag (timeseries).Executive (summary) view — panels and visualizations:1. Overall health score (big numeric + green/yellow/red) derived from SLO adherence.2. SLO compliance cards: % met (last 7d) with sparkline.3. High-level p95 latency and success-rate KPI small multiples (single-number + trend).4. Active critical alerts count (badge) and last incident SLA impact (card).5. Capacity & queue risk: current queue depth vs capacity threshold (gauge).6. 30d error budget consumption (donut).Alert thresholds and routing:- Critical (notify PagerDuty + #oncall-sre Slack + SMS): SLO burn rate > 14x sustained for 5m OR success rate < 99.5% for 2m OR p99 latency > X ms (service-specific) + sudden queue depth growth > 50% in 2m.- High (notify Slack #sre + email): SLO burn rate 4–14x for 15m OR success rate 99.5–99.9% for 10m OR queue depth > 75% capacity.- Warning (ticketing + email): degraded trends (p95 increase > 2x baseline for 1h) or sustained error spike < thresholds.- Routing by owner tag: alerts include service, owning team, runbook link, runbook steps; if no acknowledgement in 5m escalate to on-call manager.Design notes:- Use sample rate aggregation for p50/p95/p99 (histogram buckets) to avoid tail loss.- Make all charts drillable to traces/logs (link to tracing/ELK).- Keep exec view low-noise: show SLO & business impact; engineers get high-cardinality telemetry.
MediumTechnical
76 practiced
A sudden spike in database connections caused tail latency to spike across your service. Propose at least three mitigations spanning software-level, architectural, and operational changes. For each approach explain how it works, trade-offs (latency, complexity, cost), estimated implementation effort, and expected short-term vs long-term impact on reliability.
Sample Answer
1) Software-level: enforce connection pooling + timeouts and resilient client behavior- How it works: Use a client-side pool (or a shared pool like pgbouncer) to cap concurrent DB connections, set per-query timeouts and short retry-with-backoff. Prevents thundering herd and keeps DB from being overwhelmed.- Trade-offs: Slight added latency from queueing when pool is full; risk of connection contention if pool sizes misconfigured. Low hardware cost; moderate complexity to tune.- Effort: Small–medium (days to 2 weeks) to instrument and roll out changes across services.- Impact: Short-term: immediately reduces tail by limiting concurrent load. Long-term: improves stability and predictable latency; requires periodic tuning.2) Architectural: add caching / read replicas + DB proxy layer- How it works: Offload reads to caches (Redis/Memcached) and read replicas; put a lightweight proxy to route and limit connections and implement retry/circuit-breaker logic.- Trade-offs: Cache staleness, increased system complexity, additional infrastructure cost (replicas, cache nodes). Potential consistency trade-offs for reads.- Effort: Medium–high (2–8 weeks) depending on app changes; longer for cache keying and replica-aware queries.- Impact: Short-term: moderate relief if easily cacheable data present. Long-term: large reliability gain and reduced DB load; better scalability.3) Operational: rate limiting, autoscaling, and runbook + monitoring improvements- How it works: Implement service-side rate limits and priority queuing; enable autoscaling for DB tier (read replicas, CPU/memory or RDS instance resizing) and add alerts for connection count, queue length, tail latency; create incident playbooks.- Trade-offs: Rate limiting can degrade UX; autoscaling costs more; runbook creation is labor.- Effort: Medium (1–4 weeks) to configure policies, dashboards, and runbooks.- Impact: Short-term: immediate containment by shedding noncritical traffic; rapid scaling reduces pressure. Long-term: improved operational maturity, faster incident response, and predictable capacity planning.Combined approach: apply pooling + timeouts immediately, add rate-limits/runbooks for containment, and plan caching/replicas and DB autoscaling for sustained resilience.
HardTechnical
62 practiced
Write a persuasive one-page memo to senior leadership arguing for a six-week reliability sprint to address technical debt. Include the types of evidence you would collect to support the case (incident trends, customer impact, ops cost), proposed success metrics, risk mitigation for ongoing product timelines, and how you would coordinate with product managers to minimize business disruption.
Sample Answer
To: Senior LeadershipFrom: SRE TeamSubject: Request for a 6‑Week Reliability Sprint to Reduce Technical Debt and Lower Operational RiskSummary: Over the past 12 months our production incidents, latency spikes, and rollback frequency have steadily increased. I propose a focused six‑week reliability sprint to pay down targeted technical debt that is driving customer impact and ops cost. This is an investment that will immediately reduce outages, lower MTTR, and free engineering capacity long‑term.Evidence we will present:- Incident trends (weekly incident count, MTTR, burn rate against SLOs) with timelines showing deterioration- Customer impact (support tickets, NPS dips, percent of revenue affected by outages)- Ops cost (on‑call hours, emergency deploys, post‑incident engineering hours, contractor spend)- Root cause mapping linking top 20 incidents to specific debt (e.g., brittle deploy scripts, untested DB migrations, missing observability)Proposed scope & success metrics:- Fix top 6 debt items that cause ~70% of incidents- Reduce weekly incident count by 50% and MTTR by 40% within 90 days- Increase SLO compliance from X% → Y% and reduce on‑call burn by 30%- Track deploy success rate and restore time per incidentRisk mitigation & timeline alignment:- Freeze new feature merges only for services in scope; cherry‑pick critical fixes allowed- Reserve 20% of sprint capacity for critical business work- Phased rollout with feature flags and canary deployments to avoid broad user impact- Maintain hotfix path for revenue‑critical launchesCoordination with Product Managers:- Joint prioritization session to select debt items whose remediation yields highest product value- Clear acceptance criteria and release gates co‑owned by SRE and PMs- Roadmap visibility: publish sprint plan, daily syncs, and a public risk register to keep stakeholders informed- Offer follow‑up demos showing reliability ROI to re‑invest savings into product featuresConclusion: A six‑week, targeted reliability sprint is a high‑ROI intervention that reduces customer pain and operational cost while protecting product timelines through careful coordination and phased rollouts. I welcome approval to finalize scope and present the data package.
Unlock Full Question Bank
Get access to hundreds of Problem Solving and Communication interview questions and detailed answers.
