Production and Operational Awareness Questions

Understanding of production systems, monitoring, incident response, and operational responsibilities of backend engineers. Comfortable discussing debugging, deployments, and troubleshooting.

MediumTechnical

84 practiced

A Kubernetes cluster shows intermittent node-level CPU saturation causing pod evictions and increased request latency. Provide a step-by-step triage plan including kubectl commands, specific metrics to inspect (node/pod CPU, throttle metrics, cgroup limits), immediate mitigations to reduce impact, and long-term remediation strategies.

Sample Answer

Situation: Intermittent node CPU saturation is evicting pods and increasing latency. Triage must be fast (reduce impact) and thorough (find root cause).

Triage plan — quick checklist and kubectl commands1) Confirm symptoms & scope- Check node and pod CPU at cluster level:

bash

kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu | head

- Inspect events and recent evictions:

bash

kubectl get events --all-namespaces --sort-by=.lastTimestamp
kubectl get nodes
kubectl describe node <node-name>   # look for "Eviction", "OutOfcpu" messages

2) Identify noisy pods / patterns- Top consumers on the problematic node:

bash

kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name> -o wide
kubectl top pod -n <ns> --sort-by=cpu

- Check QoS class / requests/limits:

bash

kubectl get pod <pod> -n <ns> -o yaml | yq '.spec.containers[] | {name: .name, requests: .resources.requests, limits: .resources.limits}'

3) Inspect throttling and cgroups (node-level)- If you can SSH into node:

bash

# container runtime: find container id, then inspect cgroup throttle stats
cat /sys/fs/cgroup/cpu,cpuacct/docker/<container-id>/cpu.stat
# or for containerd:
cat /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/.../<container-id>/cpu.stat

- From monitoring (Prometheus) check: - container_cpu_usage_seconds_total (pod/container) - container_cpu_cfs_throttled_seconds_total and throttled_periods_total - node_cpu_seconds_total (util by mode) - kube_node_status_allocatable_cpu_cores and kube_node_status_capacity_cpu_cores - kube_pod_container_resource_requests/limits

4) Check kubelet and cluster config- Kubelet reserved and eviction thresholds:

bash

kubectl get node <node> -o json | jq '.status.allocatable, .metadata.annotations'
# Inspect kubelet config on node / kubelet-config ConfigMap
kubectl -n kube-system get configmap kubelet-config-<version> -o yaml

Immediate mitigations (reduce customer impact)- Evacuate/cordon problematic node(s) to stop new pods:

bash

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-local-data --force

- Scale up capacity quickly: increase node pool or add instances (cloud provider / cluster autoscaler)- Throttle/route traffic to healthier pods: temporarily scale down rollout or use traffic-split to healthier regions- Reduce noncritical workloads: add node taints and move batch/low-priority pods off- Patch runaway pods: restart or roll back offending deployment

Root-cause analysis steps- Correlate spikes with deployments, cronjobs, backups, batch jobs, or cron traffic patterns- Check for CPU throttling metrics — high throttled_periods_total indicates limits too low- Inspect requests vs. limits: pods with high limits but low requests can cause scheduler packing then burst -> throttling for guaranteed vs burst QoS- Verify system/kube-reserved not starving kube-system processes- Confirm kernel or cgroup issues, noisy neighbor behavior, or kernel boot/config changes

Long-term remediation & prevention- Right-size resource requests and limits: enforce request > 0 for meaningful scheduling; use LimitRange and resource quotas- Use proper QoS: set requests ~= expected usage for Burstable; Guaranteed for critical services- Implement HPA (and VPA where appropriate) with sensible targets and PDBs- Enable cluster autoscaler and tune scale-up/scale-down policies- Improve observability: ensure Prometheus + node exporter + cAdvisor metrics, alert on node CPU saturation, throttling ratio, and eviction events- Tune kubelet eviction thresholds and cpu-manager policy for latency-sensitive workloads; consider dedicated CPUs (CPU manager static policy) for latency-critical pods- Introduce scheduling isolation: taints/tolerations, pod anti-affinity, node pools for noisy batch jobs- Capacity planning and load testing to validate headroom and SLOs- Build runbook: clear steps to cordon/drain, scale, and identify noisy pods; post-incident blameless review and remediation plan

Key metrics to alert on- Node CPU utilization % (sustained > 80–90%)- container_cpu_cfs_throttled_seconds_total / throttled_periods_total ratio- kube_node_status_allocatable_cpu_cores vs sum(pod requests)- Eviction events and kubelet eviction thresholds triggered- Pod restart rate and request latency SLO breaches

This plan gives immediate commands to stabilize the cluster, diagnostic metrics to pinpoint root cause (throttling, requests/limits, kubelet settings), and remediation steps (short-term mitigation and long-term fixes) tailored for SRE priorities: reliability, observability, and capacity.

EasyTechnical

90 practiced

Explain the difference between SLI, SLO, SLA, and error budget for a REST JSON API that serves external customers. For example, the service currently reports 99.95% successful request rate over a 30-day window. Define a concrete SLI you would measure, express the SLO, and compute the monthly error budget in minutes; mention measurement windows, edge cases and how partial failures (e.g., degraded latency) factor in.

Sample Answer

SLI (concrete): successful-request-rate = (number of HTTP requests in a window that return HTTP 2xx with valid JSON body within latency threshold) / (total requests from external customers in that window). I’d record request outcome, status code, JSON parse success, and latency.

SLO (example): 99.95% successful-request-rate over a 30-day rolling window.

Error budget (monthly 30-day window): 30 days = 30 * 24 * 60 = 43,200 minutes. Allowed failure fraction = 1 - 0.9995 = 0.0005. Error budget = 0.0005 * 43,200 = 21.6 minutes of equivalent failure per 30 days.

Measurement windows & practice:- Use rolling 30-day windows for operational decisions and 7-day/1-day short windows for alerting.- Compute by counting failed requests’ duration-equivalent: either minutes of total unavailability (if whole service is down) or request-count fraction as defined above.

Edge cases & counting rules:- Client errors (4xx) should be excluded from success metric if caused by client (invalid auth, bad input), or tracked separately; server-side 5xx and timeouts count as failures.- Partial failures (degraded latency): include latency threshold in SLI (e.g., success requires <500 ms). Requests exceeding threshold count as failures (degraded service consumes error budget).- Valid JSON but semantic error: treat as failure if it breaks client contract.- Retries: count the original client-visible request outcome (if retry is transparent to client and succeeds, count success); ensure de-duplication to avoid double-counting.- Sampling: if high QPS, sample deterministically but ensure statistical significance and store numerator/denominator consistently.

Instrumentation & alerts:- Alert when 7-day SLO burn rate > 4x or remaining error budget < 50% mid-window.- Log reasons for failures for post-incident analysis and to refine SLI (e.g., separate latency vs correctness).

This approach ties availability and user experience into a measurable SLI, a clear SLO, and a precise error budget (21.6 minutes/month) that includes partial degradations.

MediumTechnical

83 practiced

You need to perform capacity planning for a stateful service expecting 2x traffic growth over the next 12 months. Describe the metrics you would collect (CPU, memory, storage growth, IOPS, network), assumptions about traffic patterns, headroom targets, margin, scaling options (vertical, horizontal, sharding), and how to validate the plan with load testing.

Sample Answer

Approach: Treat this as a forecasting + validation exercise. Start by collecting current baseline metrics, make conservative assumptions about growth and patterns, choose headroom/margin targets, evaluate scaling options (limits, cost, operational complexity), then validate with load and soak testing.

Metrics to collect (sample list):- CPU: % utilization per host, 95/99th percentiles, steal time- Memory: used vs cached, page faults, swap activity- Storage growth: GB/day and GB/month, retention/TTL rules- IOPS: read/write IOPS, IOPS per GB, queue depth, latency p95/p99- Disk throughput: MB/s read/write- Network: bandwidth, packets/sec, error rates- Application-level: requests/sec, RPS per partition, read/write ratio, mean/95/99 latency, error rates- DB-specific: connections, locks, tombstones, compaction times, replication lag- Ops signals: GC pause times, restart frequency, crash reports

Assumptions about traffic patterns:- Overall RPS will double in 12 months (2x linear growth), but account for seasonal spikes (e.g., +30% monthly peak)- Peak-to-average ratio (burstiness) historically ~3x; use 99th-percentile peaks for capacity decisions- Read/write mix remains similar unless product change announced- Retention and data-per-request remain constant; include planned feature changes as scenarios

Headroom and margin targets:- Target steady-state utilization: CPU <= 50–60%, memory <= 60–70% (to avoid CPU steal/GC), IOPS <= 50–60% of provisioned- Headroom for peaks: ensure capacity for 99th-percentile bursts (add extra 20–30% buffer)- Safety margin: plan for 20–30% additional unexpected growth or degraded performance during maintenance

Scaling options and trade-offs:- Vertical scaling: increase instance size. Pros: simple for stateful components; cons: finite limits, downtime risk, cost jumps.- Horizontal scaling (replication): add replica nodes for read-scaling and failover. Pros: availability; cons: state synchronization, write bottleneck if single-leader.- Sharding/partitioning: split dataset by key ranges or hash. Pros: scales writes and storage linearly; cons: complexity in routing, rebalancing, cross-shard transactions.- Storage strategies: add faster disks for hot partitions, tier cold data to cheaper storage, use IOPS-optimized volumes.- Autoscaling: more complex for stateful services—use proactive scaling with scheduled or predictive scaling rather than purely reactive.- Hybrid: combine sharding + replica sets + storage tiering for long-term 2x+ growth.

Validation with load testing:- Build representative workload from production traces (RPS, payload sizes, read/write ratio, time distribution).- Tests to run: - Ramp test: gradually increase RPS to 2x+ buffer until metrics reach thresholds. - Soak test: run sustained 48–72 hour load at expected peak to observe resource leaks, GC, compactions. - Spike test: inject sudden bursts to test throttling and degradation modes. - Failure injection: kill nodes, simulate network partition, disk latency to verify failover and recovery.- Observe key indicators: latency p95/p99, error rate, CPU/memory/IOPS saturation, replication lag.- Validate autoscaling/shard rebalancing steps in a staging environment that mirrors production topology.- Define success criteria: SLOs met (latency/error), no data loss, recovery within RTO/RPO.

Deliverables and runbook:- Capacity report with baseline, 12-month forecast (monthly granularity), recommended architecture changes, cost estimate.- Implementation plan: timeline for provisioning, migration/sharding, and fallbacks.- Runbook for scale events: when to add capacity, monitoring alerts, rollback steps.- Re-evaluate quarterly or after any product change.

This approach ensures measurable planning, explicit assumptions, operational safety margins, and validation through realistic load testing.

HardTechnical

75 practiced

You inherit an on-call rotation with high burnout and a 40% escalation rate. Propose a multi-quarter program of operational, automation, and cultural changes to reduce on-call load and burnout, improve reliability, and measure progress. Include instrumentation, runbook automation, rotation design, training, and incentives.

Sample Answer

Situation: I inherited an SRE team with high burnout and a 40% escalation rate into dev teams — frequent noisy alerts, poor runbooks, uneven rota coverage, and weak instrumentation.

Program (multi-quarter roadmap, measurable outcomes)Quarter 1 — Stabilize & Instrument- Goals: baseline metrics, quick-win noise reduction, psychological safety- Actions: - Instrument critical services: add latency, error, saturation metrics, business KPIs; centralize in Prometheus/Grafana & pager dashboard. - Define and publish SLOs and error budgets for top 10 services. - Triage alerts: label noisy/low-action alerts; set immediate debounce / suppression for clearly noisy targets. - Create “on-call health” pulse: weekly pulse metric = pages per on-call, escalation rate, MTTR, burnout survey.- Metrics: pages/on-call, escalation% baseline.

Quarter 2 — Automate & Harden- Goals: cut toil, automate common responses- Actions: - Implement runbook automation (RBA): encode common remediation playbooks as scripts/playbooks callable from alerts (e.g., restart, scale, config toggle) with guarded approvals. - Build alert routing and runbook links into alerts; enable one-click remediation in incident UI. - Replace low-value alerts with synthetic checks that reflect SLOs. - Create a “blast radius” safe sandbox for automated remediation testing.- Metrics: manual steps per incident, % incidents resolved by RBA, pages reduced.

Quarter 3 — Process & Rotation Design- Goals: fair on-call, reduce cognitive load- Actions: - Redesign rotation: primary/secondary model, predictable shifts, max 1 week on-call; limit night/weekend frequency; include follow-the-sun where possible. - Create escalation thresholds tied to error budget: if SLO healthy, higher tolerance before escalation. - Implement on-call runbook reviews & gamified drills (war games) monthly. - Add “buddy” handoff checklist and protected recovery days after major incidents.- Metrics: burnout survey NPS, escalation rate trend, on-call retention.

Quarter 4 — Culture, Training & Incentives- Goals: sustainable culture, ownership, continuous improvement- Actions: - Mandatory on-call training: tooling, incident leadership, RBA use, blameless postmortems. - Incentives: on-call stipend, explicit career credit, performance recognition for reliability improvements, rotation for engineers to contribute to runbooks/automation. - Institutionalize post-incident action tracking with SLA to close remediation work. - Share monthly reliability dashboard with execs and teams.- Metrics: MTTR, repeat incident rate, PTO usage, attrition.

Governance & Measurement (continuous)- Primary KPIs: pages per on-call, escalation rate (target <10%), MTTR, SLO compliance, burnout survey score.- Quarterly reviews: A/B rollback of automation rules, retro on runbook effectiveness.- Trade-offs: automation risks — mitigate via canary, dry-run, human-in-loop for high-risk actions.

Why this works- Instrumentation + SLOs focus effort on user impact.- Runbook automation reduces toil and speeds mean-time-to-resolution.- Rotation design and training reduce human cost and increase competence.- Cultural changes reward reliability and prevent burnout while measurable KPIs show progress.

EasyTechnical

126 practiced

Design an on-call escalation policy for a 12-engineer SRE+backend team to cover 24x7 support using PagerDuty. Include rotation cadence, primary/secondary roles, escalation steps and delays, holiday handling, handoff overlap, and how to handle long-running incidents and handoffs between shifts.

Sample Answer

Situation: We need 24x7 coverage for a 12-engineer SRE+backend team using PagerDuty.

Policy (concise, actionable):

Rotation & Cadence- Team split into 4 squads of 3 engineers. Primary on-call rotates weekly per engineer (12-week cycle).- Secondary rotates on the same cadence but offset by one week so each primary has a different secondary each rotation.- Backstop (third level) is a designated senior on-call, rotates monthly.

Primary / Secondary Roles- Primary: first responder, owns incident until resolved or formally handed off.- Secondary: support escalation within 15–30 minutes, takes over if primary unavailable.- Backstop: leadership escalation for unresolved severity-1 after SLA or conflict resolution.

Escalation Steps & Delays (PagerDuty rules)1. Alert fires → Primary receives immediate page.2. 5 min no ack → re-alert primary + page secondary.3. 15 min no ack/resolve → escalate to secondary (page + call).4. 30 min from initial without mitigation → escalate to backstop and incident commander (call + SMS).5. 60 min without containment → notify Engineering Manager and Product on-call.

Handoff & Overlap- 30-minute overlap at shift change: outgoing writes a 15–30 minute summary in a shared handoff doc (status, mitigations, next actions, risks), quick 10–15 minute voice/video sync required for active incidents.- PagerDuty status must be updated (acknowledged/unacknowledged) and ownership transferred in the incident timeline.

Holiday Handling- Pre-approved holiday swap policy: engineers can trade weeks; manager must approve 2 weeks prior.- If holiday reduces coverage, activate a contingency schedule: senior backstop covers critical weekdays; optionally use paid holiday on-call premium.

Long-running Incidents & Shift Handoffs- If incident exceeds a shift, outgoing primary must perform a formal handoff: documented timeline, current hypothesis, runbook steps taken, pending actions, and expected next check-in cadence.- Incidents >4 hours require a rotating incident commander handover every 4 hours to avoid fatigue; new commander reads handoff checklist and announces change in incident channel.- All handoffs logged in the incident timeline; the outgoing engineer remains available for 1 hour as advisor after handoff.

Best practices & tooling- Use PagerDuty escalation policies + Slack/PagerDuty integrations for automated updates.- Maintain runbooks in central repo with playbooks and postmortem templates.- Enforce “no silent escalations”: phone call for severity-1 after 15 minutes.- Regularly review rota fairness, alert noise, and runbook effectiveness in monthly retros.

This policy balances ownership, rapid escalation, overlap for safe handoffs, and fatigue management for a 12-person SRE/backend team.

Unlock Full Question Bank

Get access to hundreds of Production and Operational Awareness interview questions and detailed answers.

Join thousands of developers preparing for their dream job.