Your SRE Background and Experience Questions

Articulate your hands-on experience with systems administration, monitoring tools, automation scripts, and any incident response involvement. Be specific about technologies (e.g., Prometheus, Grafana, Kubernetes, Docker, Terraform) and concrete examples of what you've built or fixed.

EasyTechnical

67 practiced

Describe a project where you installed and configured Prometheus and Grafana for production monitoring. Include:

- A sample Prometheus scrape_config snippet

yaml

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: my-app

- One alert rule (PromQL) you created and why you chose its threshold- One Grafana dashboard panel you built and how it reduced incidents. Explain trade-offs around scrape interval, retention and label cardinality.

Sample Answer

Situation: At my last SRE role I led production monitoring for a Kubernetes service stack. I installed Prometheus (kube-prometheus-stack) and Grafana, integrated pod scraping, alertmanager, and built dashboards to reduce page noise and speed MTTR.

Prometheus scrape_config (used with additional relabeling for namespaces/ports):

yaml

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: my-app

Alert rule (PromQL) I created and rationale:

yaml

groups:
- name: my-app.rules
  rules:
  - alert: MyAppHighCPU
    expr: sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="prod",pod=~"my-app-.*"}[2m])) * 100
          / sum by (pod) (kube_pod_container_resource_limits_cpu_cores{namespace="prod",pod=~"my-app-.*"})
          > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU on {{ $labels.pod }}"

Why 85% for 5m: team SLA allowed short spikes but sustained >85% for 5 minutes correlated with request tail latency and OOM risk in load tests — threshold balances timely detection vs alert noise.

Grafana panel I built:- Single-stat + sparkline showing 95th-percentile request latency (p95) per service with an error-rate stacked bar underneath. Clicking a point opens relevant logs/traces.Impact: Reduced P1 incidents by ~30% because on-call could see rising latency before errors and remediate (scale/rollback) proactively. The panel focused on p95 (not avg) so it surfaced tail latency issues that caused customer impact.

Trade-offs considered:- Scrape interval: 15s for app metrics to detect fast degradations; 30s reduces load/TSDB size but delays detection. We used 15s for critical metrics, 60s for low-value exporters.- Retention: 15–30 days on local Prometheus for fast queries and cost control; long-term (90+ days) stored in remote TSDB (Thanos) for capacity planning.- Label cardinality: Avoided high-cardinality labels (user_id, request_id) on metrics; kept labels to service/endpoint/pod. High cardinality increases memory and WAL pressure dramatically — we converted some labels into metric labels at query time or used logs/traces for per-request details.

Result: Faster detection, fewer noisy alerts, and clearer runbooks tied to specific dashboard panels improved mean time to mitigation and stakeholder confidence.

MediumTechnical

64 practiced

A faulty change made it through your CI/CD pipeline and caused a production incident. Walk through how you would triage: identify the change, decide between rollback or patch, perform the rollback or hotfix safely, communicate with stakeholders, and run a postmortem. Include checks to ensure the rollback is safe (schema changes, rollbacks across services).

Sample Answer

Situation: A production alert shows increased error rate and customer impact after a recent deployment that passed CI/CD. I lead triage and remediation.

1) Identify the change- Immediately correlate alerts with deploy timeline (CI timestamps, Git commit hashes, deployment IDs).- Use tracing/logs and APM (e.g., Jaeger/New Relic) to map errors to service and code paths.- Run: git log / CI build metadata -> identify the exact commit and artifacts promoted.- Check recent config/infra changes (Terraform/Helm) and feature-flag flips.

2) Decide rollback vs patch- Ask: Is the failure isolated to stateless code or involves irreversible data changes (schema, migrations, long-running background jobs)?- If no DB-breaking migration and rollback is fast and tested, prefer rollback for safety. If migration ran or rollback risks data loss, prefer a targeted hotfix or feature-flag disable.

3) Safety checks before rollback/hotfix- Schema compatibility: verify migration history; if migration was applied, ensure forward/backward compatibility and presence of down migrations. If irreversible, don't rollback code that expects new schema.- Cross-service dependencies: check versions of downstream services and API contracts; use service registry or deployment manifests to ensure consistency.- Side-effects: ensure rollback won't reprocess idempotent jobs leading to duplicates.- Canary and health checks: plan to roll back in canary (10%) then progressively if metrics are good.- Prepare a rollback runbook and automated playbook (kubectl rollout undo / helm rollback / CI pipeline rollback).

4) Execute safely- Put mitigations first: disable related feature flags, scale to zero non-critical pods, or route traffic away via load balancer.- Perform canary rollback, monitor error rate, latency, logs, and business KPIs.- If stable, promote full rollback with automation; if not, revert and attempt hotfix patch.- For hotfix: branch from last-good commit, apply minimal code/data-safe fix, run smoke tests in staging, deploy canary, then full deploy.

5) Communication- Immediately notify incident channel with: impact, suspected change, remediation plan, owner, and ETA. Use concise updates every 15 minutes.- Inform stakeholders (support, product, leadership) with customer-facing impact and mitigation.- After resolution, send timeline and user impact summary.

6) Postmortem- Conduct blameless postmortem within 48-72 hours: timeline, root cause (code, CI test gaps, review gaps), detection/response metrics (MTTD/MTTR), and action items.- Actions may include: stronger pre-deploy tests (integration, contract tests), automated schema migration checks, mandatory canary for risky changes, feature-flag requirement for risky features, improved deploy gating and surge rollback automation, and runbook updates.- Assign owners, deadlines, and follow up to closure.

This approach balances speed and safety: identify precisely, prefer safe rollbacks when reversible, otherwise hotfix carefully, communicate clearly, and fix systemic gaps to prevent recurrence.

EasyTechnical

68 practiced

Describe how you would implement an alerting policy to reduce alert fatigue in a production environment. Explain how you derive alert severity from SLOs/SLIs and give an example Prometheus alert rule or pseudo-config. Include escalation levels (page, ticket, dashboard), deduping/inhibition strategies, and a brief example of thresholds you might choose.

Sample Answer

Approach: derive severity from SLO/SLI breach likelihood and impact. Use error-rate or latency SLIs vs SLO target and error budget burn rate to map to severities:- P0 (page): SLO breached or error budget exhausted and customer-facing impact (immediate paging).- P1 (ticket + page for sustained): high burn rate (>5x) or sustained 5–15m breach.- P2 (ticket + dashboard): short transient spikes that don't yet threaten SLO.- P3 (dashboard only): informational, trend or capacity warnings.

Example thresholds (for a 99.9% availability SLO):- Immediate page: availability < 99.9% over 5m OR error-budget burn-rate > 10x for 5m.- Sustained page/ticket: availability < 99.9% over 30m.- Ticket/dashboard: availability < 99.95% over 1h or latency p95 > 2s for 1h.- Dashboard only: p95 latency trending up 10% vs baseline.

Prometheus alert (pseudo):

yaml

groups:
- name: slo.rules
  rules:
  - alert: SLOAvailabilityBreach_P0
    expr: (1 - sum(rate(http_requests_errors_total[5m])) / sum(rate(http_requests_total[5m]))) < 0.999
    for: 5m
    labels:
      severity: page
      impact: customer
    annotations:
      summary: "SLO breach: availability < 99.9% (5m)"
  - alert: ErrorBudgetBurnFast_P1
    expr: (error_budget_consumed_rate[5m]) > 10
    for: 5m
    labels:
      severity: pagerticket
      impact: customer
    annotations:
      summary: "High error budget burn"

Alertmanager routing / escalation (pseudo):- severity=page -> route to on-call via pager (SMS/phone) immediately, create incident ticket.- severity=pagerticket -> page if sustained, otherwise create ticket and notify Slack.- severity=ticket -> create ticket and notify team channel.- severity=info -> post to metrics dashboard only.

Deduping / inhibition strategies:- Use Alertmanager grouping by (service, cluster, alertname) and grouping_interval to coalesce floods.- Inhibit lower-severity alerts when a higher-severity alert for same service is firing (e.g., inhibit ticket/info if page is firing).- Suppress noisy known maintenance windows via silences; automatically create silences from deployments.- Use dedupe by fingerprinting (same labels) so repeated identical alerts are collapsed.

Best practices:- Prefer SLO-driven alerts, alert on symptoms (customer-facing SLIs) not on every internal metric.- Keep pages for things that can cause customer-visible degradation quickly; everything else to ticket/dashboard.- Review alert noise metrics (alerts per contributor, time to acknowledge) and iterate thresholds and grouping.

EasyTechnical

83 practiced

Explain Infrastructure as Code (IaC) and name at least three IaC tools you've used (for example: Terraform, CloudFormation, Pulumi). For one tool provide a short example of provisioning a resource (for instance: creating an ALB or a GKE cluster) with HCL or pseudo-code and explain how you handled state, locking, and secrets.

Sample Answer

Infrastructure as Code (IaC) is the practice of defining and managing infrastructure (networks, VMs, load balancers, clusters, etc.) using declarative or programmatic configuration files so provisioning is repeatable, version-controlled, and automated. For an SRE, IaC enables reliable, auditable, and testable changes to production environments and supports automation of disaster recovery and scaling.

Tools I've used: Terraform, AWS CloudFormation, Pulumi, Ansible (for provisioning/config), and Google Deployment Manager.

Example (Terraform HCL): create an AWS ALB and target group

hcl

provider "aws" { region = "us-west-2" }

resource "aws_lb" "app_alb" {
  name               = "app-alb"
  load_balancer_type = "application"
  subnets            = var.public_subnets
  security_groups    = [aws_security_group.alb_sg.id]
}

resource "aws_lb_target_group" "app_tg" {
  name     = "app-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = var.vpc_id
}

State, locking, and secrets:- State: I store Terraform state in a remote backend (S3) with state versioning enabled so team changes are shared and recoverable.- Locking: I enable DynamoDB-based state locking (for S3 backend) to prevent concurrent apply/plan conflicts.- Secrets: I avoid embedding secrets in code; use encrypted secrets manager (AWS Secrets Manager / SSM Parameter Store) referenced at runtime and pass sensitive values via environment variables or use Vault with a Terraform provider. I also mark sensitive outputs and enable Terraform state encryption (SSE-KMS) to protect secrets in state files.

MediumTechnical

71 practiced

A node pool shows slowly increasing CPU usage across nodes over several days. Describe step-by-step how you would investigate whether the cause is a memory leak, noisy neighbor pods, misconfigured cluster-autoscaler, or mis-scheduled workloads. Include kubectl commands, Prometheus queries, log checks and isolation strategies you would run.

Sample Answer

Start by clarifying scope and gather baseline metrics (last 7–14 days) so you can correlate trends.

1) Cluster & node baseline- List node pool nodes and recent CPU:

bash

kubectl get nodes -l cloud.google.com/gke-nodepool=MYPOOL -o wide
kubectl top nodes --containers | grep MYPOOL -A 50

- Node-level Prometheus (via grafana/Prometheus): total CPU usage per node over 7d

promql

sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)

2) Is it global app increase vs single process?- Pod-level CPU trend:

promql

sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)

- Memory trend:

promql

sum(container_memory_usage_bytes) by (pod, namespace)

If CPU rises but memory grows continuously in same pod, suspect memory leak.

3) Check for memory leak signals- Per-pod memory over time; check OOM/killed events:

bash

kubectl get events --field-selector involvedObject.kind=Pod --sort-by='.lastTimestamp' -A | grep -i oom
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous

- Prometheus: monotonic increase (not reclaimed):

promql

increase(container_memory_usage_bytes[7d]) > 0

If memory never drops between restarts and growth correlates with CPU, investigate app heap/profile, get pprof or JVM heap dump.

4) Noisy neighbor (noisy pods)- Check top consumers on nodes:

promql

topk(10, sum by(pod,instance)(rate(container_cpu_usage_seconds_total[5m])))

- On node: list pods scheduled there and their limits/requests:

bash

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<node>
kubectl describe node <node>

- If one or few pods consume most CPU and co-located with others showing degraded CPU, it's noisy neighbor. Verify QoS class and resource limits/requests. Use eviction or move offending pod to confirm.

5) Mis-scheduled workloads- Look at pod topology/affinity/taints tolerations:

bash

kubectl get pod <pod> -o yaml -n <ns> | yq '.spec.affinity, .spec.nodeSelector, .spec.tolerations'

- Check if bursting or critical pods are packed onto same nodes: use Prometheus grouping by node and deployment.

6) Misconfigured cluster-autoscaler- Check autoscaler events and metrics:

bash

kubectl -n kube-system logs deploy/cluster-autoscaler

Prometheus:

promql

kube_cluster_autoscaler_events_total or cluster_autoscaler_scale_up_failures_total

- Compare desired vs actual nodes:

promql

max(kube_node_info) by (node)

If autoscaler isn’t scaling up when nodes are saturated, inspect pod pending reasons:

bash

kubectl get pods --all-namespaces | grep Pending
kubectl describe pod <pending-pod>

7) Isolation & mitigation strategies (experiments)- Reproduce: cordon a node and drain to see if problem moves:

bash

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

- Evict suspected noisy pods:

bash

kubectl delete pod <pod> -n <ns>

- Temporarily enforce QoS via limits/requests, or use PodDisruptionBudget/PodTopologies to spread workload.- Run a canary: scale a single pod with same image on another node pool and compare metrics.

8) Deep inspection- Attach profiler/tracing (pprof, async-profiler, eBPF) to suspect process.- Collect heap/CPU profiles, container logs, GC logs, thread dumps.

9) Postmortem actions- If memory leak: fix app, add container memory limit + OOM policies, add heap dumps/alerts on memory growth.- Noisy neighbor: enforce requests/limits, PodPriority, node isolation, or use Pod Topology Spread/taints.- Autoscaler: fix cloud IAM/scale-up errors, adjust scale-up delay/thresholds, or add buffer nodes.- Mis-scheduling: update affinity/anti-affinity and scheduler policies.

Throughout: correlate timestamps across Prometheus, kube-apiserver events, and pod logs to attribute cause before applying permanent fixes.

Unlock Full Question Bank

Get access to hundreds of Your SRE Background and Experience interview questions and detailed answers.

Join thousands of developers preparing for their dream job.