Staff-Level Site Reliability Engineer Interview Preparation Guide for Amazon

Site Reliability Engineer (SRE)

Amazon

Staff

7 rounds

Updated 6/18/2026

Amazon's specific SRE interview process details (exact round structure, evaluation criteria, and interview formats) were not found in available search results. This guide is based on industry-standard Staff-level SRE interview patterns, the provided job description, and general SRE interview practices. For the most current and accurate information, consult Amazon's official careers website, Levels.fyi, Blind, or recent candidate experiences.

Amazon's SRE interview process for Staff-level positions consists of an initial recruiter screening, two technical phone screens, and four onsite interview rounds. The process evaluates technical mastery in distributed systems and infrastructure at scale, operational excellence and incident response leadership, system design expertise, infrastructure automation capabilities, and alignment with Amazon Leadership Principles. Staff-level candidates are assessed not only on individual technical excellence but also on their ability to influence teams, drive operational improvements, and lead initiatives that span multiple teams.

Interview Rounds

Recruiter Screening

30 min3 focus topicsculture fit

What to Expect

Initial conversation with an Amazon recruiter to assess your background, verify Staff-level experience qualifications, gauge interest in the SRE role, and determine overall fit with Amazon. The recruiter will explore your career progression, explain the role and team, discuss compensation and logistics, and answer questions about the interview process and team structure. This round typically happens once before you move to technical interviews.

Tips & Advice

Prepare a compelling 2-3 minute summary of your SRE journey, emphasizing progression from earlier career roles to Staff-level responsibilities. Highlight the scale and complexity of systems you've operated (number of services, traffic volume, geographic regions, team influence). Quantify specific achievements: percent improvements in MTTR, automation projects that eliminated toil, systems designed to handle multi-region failover, mentoring of junior SREs. Research Amazon's SRE mission and culture beforehand. Be ready to articulate why you're interested in Amazon specifically and how this role aligns with your career goals. Discuss compensation expectations clearly and upfront. Ask thoughtful questions about the team structure, the systems you'd own, challenges the team is facing, and growth opportunities within Amazon.

Focus Topics

Amazon Leadership Principles Alignment

Familiarize yourself with Amazon's 16 Leadership Principles. Prepare brief examples showing how your work reflects these principles, especially: Ownership (taking accountability for outcomes), Customer Obsession (focusing on customer impact), Operational Excellence, and Have Backbone/Disagree and Commit.

Practice Interview

Study Questions

Scale and Systems Expertise

Concrete metrics about systems you've designed or operated: number of services/servers, traffic volume (requests per second, data processed), geographic footprint, uptime percentages achieved, team size managed or influenced, cost optimization realized.

Practice Interview

Study Questions

Career Progression to Staff Level

Clearly articulate your progression from individual contributor to Staff-level SRE, with specific roles, systems you owned, and increasing scope of influence. Explain how each position prepared you for Staff responsibilities. Emphasize organizational impact beyond individual technical contributions.

Practice Interview

Study Questions

Technical Phone Screen 1: Distributed Systems and Infrastructure Architecture

60 min5 focus topicstechnical

What to Expect

First technical phone screen conducted by a senior SRE or infrastructure engineer. The interviewer explores your deep understanding of distributed systems, large-scale infrastructure design, and architectural decision-making. Expect technical questions about system scalability, failure modes, resilience patterns, and how you've approached building reliable infrastructure in previous roles. The interviewer may ask you to explain a complex system you've designed, discuss architectural tradeoffs, or analyze a scenario involving infrastructure challenges.

Tips & Advice

Think through complex distributed system problems you've solved at your organization. Be prepared to explain architectural decisions and tradeoffs explicitly: consistency vs availability, latency vs durability, cost vs reliability, automated recovery vs manual intervention. Use specific technical terminology accurately. Draw diagrams (via whiteboard or shared document) to explain systems. When discussing infrastructure choices, explain your reasoning and what constraints you were optimizing for. If you don't know something, acknowledge it and discuss how you'd approach learning or solving it. Demonstrate architectural thinking, not just familiarity with tools. Be prepared to defend your design choices and consider alternative approaches the interviewer suggests.

Focus Topics

Load Balancing, Service Discovery, and Traffic Management

Load balancing strategies (round-robin, least connections, consistent hashing, weighted distribution), health checking mechanisms, session affinity and stateful service handling. Service discovery patterns, DNS architecture and considerations, dynamic service registration/deregistration at scale.

Practice Interview

Study Questions

Resilience Patterns and Failure Mode Analysis

Implementing resilience patterns: circuit breakers, bulkheads/compartmentalization, retry logic with exponential backoff and jitter, graceful degradation, timeout strategies. Proactive failure mode analysis: understanding potential failure scenarios, designing mitigations, chaos engineering approaches.

Practice Interview

Study Questions

AWS Services and Cloud Infrastructure Mastery

Deep proficiency with AWS services critical to SRE: EC2 and instance types, RDS and managed databases, S3 and data storage, ECS/EKS container orchestration, CloudWatch metrics and alarms, VPC networking and security, IAM and access control, Auto Scaling and load balancing, Lambda serverless compute.

Practice Interview

Study Questions

Infrastructure Automation and Infrastructure as Code (IaC)

Deep expertise in automation frameworks and IaC tools (Terraform, CloudFormation, Ansible, Puppet, Chef). Understanding of declarative vs imperative approaches, state management and idempotency, testing infrastructure changes before production, managing infrastructure drift, version control for infrastructure.

Practice Interview

Study Questions

Distributed Systems Architecture and Design Patterns

Master-level understanding of building reliable distributed systems including service-oriented architecture, microservices trade-offs, identifying and mitigating single points of failure, request routing and service discovery, eventual consistency and distributed transactions, handling network partitions.

Practice Interview

Study Questions

Technical Phone Screen 2: Observability, Monitoring, and Incident Response

60 min5 focus topicstechnical

What to Expect

Second technical phone screen with a senior SRE focused on observability, monitoring strategy, and incident response capabilities. The interviewer explores your philosophy on building observable systems, designing effective monitoring and alerting, handling production incidents, and conducting post-incident analysis. Expect deep discussions about metrics and KPIs, log aggregation and analysis, distributed tracing for complex debugging, alert design to avoid fatigue, incident management procedures, and learning from failures.

Tips & Advice

Prepare 2-3 detailed examples of significant incidents you've handled: what happened, how you diagnosed the root cause, how you resolved it, what you learned, and how you prevented recurrence. Discuss your philosophy on monitoring - what metrics matter most, how to design effective alerting, and how to avoid alert fatigue while maintaining visibility. Explain the difference between monitoring and observability. Be specific about tools and technologies you've used, but focus on the principles and strategies. Discuss how you've improved observability in previous organizations. Be prepared to explain your approach to designing monitoring for a new service. Share examples of insights you've gained from metrics that drove operational improvements.

Focus Topics

Distributed Tracing for Complex Systems

Distributed tracing concepts and trace sampling strategies, instrumenting applications for tracing, understanding request flows across microservices, identifying latency bottlenecks using traces, debugging issues across service boundaries.

Practice Interview

Study Questions

Logging, Log Analysis, and Centralized Logging Infrastructure

Structured logging practices and benefits, designing log aggregation and analysis systems, understanding common logging pitfalls, retention policies and cost management, correlating logs across services, using logs for debugging and root cause analysis.

Practice Interview

Study Questions

Observability and Monitoring Strategy Design

Principles of observability: collecting meaningful metrics, structured logs, and distributed traces. Designing comprehensive monitoring strategies that answer 'what failed and why?' Understanding RED metrics (Rate, Errors, Duration) and USE metrics (Utilization, Saturation, Errors). Instrumenting applications and infrastructure for observability. Dashboard design and visualization. Alerting strategy to catch issues while minimizing alert fatigue.

Practice Interview

Study Questions

Incident Response Framework and Management

Incident response procedures and escalation paths, incident commander role and communication protocols, severity classification and SLA management, on-call rotation design and optimization, post-incident review processes, action item tracking and verification.

Practice Interview

Study Questions

Error Budgets and Service Level Objectives (SLOs)

Defining meaningful SLOs and SLAs aligned with customer expectations and business needs. Understanding error budgets as a resource for balancing new feature development with reliability investments. Using error budgets to guide prioritization decisions and communicate tradeoffs to product teams. Monitoring SLO compliance and communicating violations.

Practice Interview

Study Questions

Onsite Round 1: System Design - Large-Scale Reliable Service

75 min5 focus topicssystem design

What to Expect

First onsite interview focused on system design expertise. You'll be presented with a real-world scenario requiring you to design a large-scale, highly available service from first principles. The interviewer will guide you through gathering requirements, identifying key challenges, proposing an initial architecture, and iterating based on constraints and feedback. You'll need to discuss infrastructure choices, data architecture decisions, failover and disaster recovery strategies, operational considerations, and how you'd monitor and operate the system. Expect deep dives into specific architectural components and explicit discussion of tradeoffs.

Tips & Advice

Use a structured approach throughout the interview: (1) Clarify requirements thoroughly - understand the scale, geographic distribution, consistency requirements, and latency requirements; (2) Identify key challenges - scalability bottlenecks, failure scenarios, operational complexity; (3) Propose initial architecture with clear diagrams showing major components; (4) Iterate based on interviewer feedback. Draw architecture diagrams showing services, databases, caching layers, load balancers, monitoring. Discuss tradeoffs explicitly and explain your reasoning - why this database over that one, why this failover strategy, what you're optimizing for. Consider multi-region deployment and cross-region failover scenarios. Discuss how you'd monitor this system and what metrics matter. Think about how your team would operate and understand this system. Be prepared to go deep on specific layers (storage tier, caching strategy, network design). Ask clarifying questions when requirements are ambiguous. It's fine to say 'there are multiple valid approaches' or 'I'd need more information.' Staff-level SREs think holistically about systems, not just the technical layer.

Focus Topics

Cost Optimization and Resource Efficiency

Making architectural decisions that optimize cost without sacrificing reliability, understanding AWS pricing models, right-sizing compute resources, choosing appropriate instance types, auto-scaling strategies for cost, spot instances and reserved instances, data storage optimization.

Practice Interview

Study Questions

Scalability, Capacity Planning, and Performance

Predicting system scalability based on load patterns, capacity planning for growth, identifying performance bottlenecks (CPU, memory, I/O, network), auto-scaling strategies, rate limiting and traffic shaping, caching strategies at various layers, CDN and edge caching usage.

Practice Interview

Study Questions

Operational Aspects of System Design

Designing systems for operational ease: monitoring instrumentation and observability hooks, debugging capabilities, configuration management, deployment and rollback procedures, graceful shutdown and connection draining, making systems understandable and operationable by humans.

Practice Interview

Study Questions

Data Architecture for Durability and Consistency

Database selection tradeoffs (SQL vs NoSQL, relational vs document vs key-value), consistency models and their implications, replication strategies, backup and recovery procedures, handling data durability across multiple regions, ensuring RPO and RTO targets.

Practice Interview

Study Questions

High-Availability System Architecture Design

Designing for availability across failure scenarios: redundancy patterns, active-active vs active-passive configurations, cross-region deployment strategies, eliminating single points of failure, quorum-based consensus decisions, automatic failover and health detection mechanisms.

Practice Interview

Study Questions

Onsite Round 2: Infrastructure Automation and Deployment Pipelines

70 min5 focus topicstechnical

What to Expect

Second onsite interview focused on infrastructure automation, Infrastructure as Code, and deployment automation. You may be asked to design an automation solution for a complex operational problem, discuss your approach to writing and testing infrastructure code, or design a CI/CD pipeline for safe deployments at scale. The interviewer assesses your ability to write quality infrastructure code, design scalable automation frameworks, and think about how automation scales across teams and organizations.

Tips & Advice

Be prepared to discuss and possibly write pseudocode or actual code for infrastructure automation. Discuss your testing philosophy for infrastructure - how do you validate IaC before production? Think about idempotency: infrastructure code should produce the same result whether run once or ten times. Discuss error handling and rollback strategies. When reviewing infrastructure code, point out both strengths and potential improvements. Discuss how you'd manage state in Terraform or CloudFormation. Talk about versioning infrastructure code and implementing CI/CD for infrastructure changes. Discuss how you scale automation across teams - standards, documentation, code reuse. Mention specific tools you've used (Terraform, Ansible, CloudFormation, etc.) but focus on concepts. Discuss testing strategies - unit tests for infrastructure code, integration tests, and validation in staging environments.

Focus Topics

Configuration Management and Avoiding Configuration Drift

Managing configuration across large fleets: configuration as code, secrets management and security, avoiding configuration drift between servers, handling configuration updates safely, immutable infrastructure vs mutable infrastructure approaches, tools like Puppet, Ansible for configuration management.

Practice Interview

Study Questions

Testing and Validation of Infrastructure Changes

Strategies for testing infrastructure before production: unit tests for IaC, integration testing, infrastructure as code linting and validation, staging environments that mirror production, chaos engineering, automated validation of deployments, detecting configuration drift.

Practice Interview

Study Questions

Kubernetes and Container Orchestration at Scale

Operating Kubernetes clusters reliably: resource requests/limits and scheduling, pod lifecycle and health checks, networking and service discovery, persistent storage, multi-cluster management and cross-cluster communication, helm charts and operators for managing applications.

Practice Interview

Study Questions

CI/CD Pipeline Design and Deployment Automation

Designing robust CI/CD systems: automated testing, security scanning, code quality checks, automated infrastructure provisioning. Deployment strategies: blue-green deployments, canary deployments, feature flags for progressive rollout, rollback procedures. Deployment validation and health checks. Pipeline orchestration and artifact management.

Practice Interview

Study Questions

Infrastructure as Code (IaC) Tools and Best Practices

Deep expertise with IaC tools (Terraform, CloudFormation, Ansible, Puppet, Chef). Declarative vs imperative approaches and their tradeoffs. State management and idempotency guarantees. Testing infrastructure code (unit tests, integration tests, linting). Version control strategies for infrastructure. Code organization and module design for reusability.

Practice Interview

Study Questions

Onsite Round 3: Complex Incident Response and Troubleshooting

70 min5 focus topicstechnical

What to Expect

Third onsite interview focused on incident response, troubleshooting complex production scenarios, and problem-solving under pressure. You'll be presented with a realistic production incident scenario involving multiple services and systems. Work through the diagnosis systematically, identify the root cause, propose resolution steps, and discuss prevention measures. The interviewer assesses your systematic approach to troubleshooting, knowledge of debugging techniques and tools, ability to communicate clearly during crisis situations, and how you extract learnings from incidents.

Tips & Advice

Approach the problem methodically rather than jumping to conclusions. Ask clarifying questions first - what's the impact, which services are affected, when did this start? Explain your thinking as you go through diagnosis. Discuss what data you'd collect: logs, metrics, traces, recent deployments, infrastructure changes. Walk through your troubleshooting hypothesis testing step-by-step. When you don't know something, say so and explain how you'd find the answer. Use diagrams to explain complex issues spanning multiple systems. Consider cascading failures and service dependencies. At the end, discuss how you'd prevent this issue in the future. Show that you stay calm, think logically through pressure, and communicate clearly about complex problems. Discuss post-incident review: what would you document, what action items would you create, how would you verify the fix?

Focus Topics

Communication, Documentation, and Post-Incident Learning

Communicating effectively during incidents: keeping stakeholders informed, clear escalation procedures, incident timelines. Post-incident documentation: timeline reconstruction, root cause analysis, prevention action items. Conducting blameless post-incident reviews and converting incident learnings into operational improvements.

Practice Interview

Study Questions

Performance Analysis and System Bottleneck Identification

Profiling applications and infrastructure to identify bottlenecks: CPU profiling, memory usage analysis, I/O and disk latency, network bandwidth and latency, database query performance, understanding cache hit rates, interpreting flame graphs and performance data.

Practice Interview

Study Questions

AWS-Specific Troubleshooting and Service Issues

Troubleshooting common AWS scenarios: RDS high CPU or connection count issues, EC2 instance connectivity and performance, network latency and routing issues, ECS/Kubernetes pod scheduling and resource issues, CloudWatch alarm misconfiguration. Understanding AWS service limits and their impact on systems.

Practice Interview

Study Questions

Systematic Troubleshooting and Root Cause Analysis

Structured approach to identifying root causes: gathering logs, metrics, and traces; forming and testing hypotheses; understanding failure propagation in distributed systems; distinguishing symptoms from underlying causes; using timeline reconstruction; Five Whys and other RCA techniques.

Practice Interview

Study Questions

Debugging Distributed Systems and Microservices

Techniques for debugging issues across multiple services: distributed tracing and trace correlation, log aggregation and pattern matching, metric analysis and anomaly detection, understanding service dependencies and call graphs, reproducing issues reliably, isolating components, using debuggers when appropriate.

Practice Interview

Study Questions

Onsite Round 4: Leadership, Influence, and Amazon Leadership Principles

60 min5 focus topicsbehavioral

What to Expect

Final onsite interview assessing leadership capabilities, organizational influence, and deep alignment with Amazon Leadership Principles. The interviewer focuses on your track record of driving technical and organizational change, mentoring and developing engineers, managing ambiguity in decision-making, and how your values align with Amazon. You'll discuss situations where you've influenced teams and organizations beyond your individual contribution, navigated difficult tradeoffs, and contributed to company success through technical leadership.

Tips & Advice

Prepare 4-5 well-structured examples using the STAR method (Situation, Task, Action, Result) that demonstrate Amazon's Leadership Principles in action. Focus especially on: Ownership (taking accountability for outcomes), Customer Obsession (prioritizing customer impact), Bias for Action (moving forward despite incomplete information), Have Backbone/Disagree and Commit (advocating for good ideas and executing once decided), and Operational Excellence. Quantify business impact when possible (cost saved, reliability improved, MTTR reduced, team velocity increased). Discuss situations where you've handled disagreement with management or peers - how did you present your case and what was the outcome? Show examples of how you've helped others succeed and grown engineers on your team. Discuss how you think about customers and the long-term implications of your decisions. Be authentic - don't try to force alignment with values you don't hold. Ask thoughtful questions about how the team embodies these principles and what challenges they're tackling.

Focus Topics

Handling Ambiguity, Making Decisions with Incomplete Information

Situations where you've operated effectively with unclear requirements or incomplete data, examples of pushing back on bad ideas or advocating for unpopular positions, balancing competing priorities and tradeoffs, learning from failures and course-correcting.

Practice Interview

Study Questions

Amazon Leadership Principle: Operational Excellence

Continuous focus on improving operational processes and practices: introducing monitoring and observability improvements, automating toil and repetitive work, improving incident response processes, implementing better change control and deployment practices, measuring and tracking operational metrics.

Practice Interview

Study Questions

Building, Mentoring, and Influencing Teams

Mentoring engineers at various levels, helping them grow careers and succeed, building trust and psychological safety, advocating for your team members, collaborating across teams to solve shared problems, influencing technical direction without direct authority, multiplying impact through others.

Practice Interview

Study Questions

Amazon Leadership Principle: Ownership

Taking full accountability for outcomes, thinking and acting on behalf of the entire company, taking on hard problems with end-to-end ownership, thinking long-term and setting high standards, making decisions that benefit Amazon even when difficult or unpopular, delivering results regardless of obstacles.

Practice Interview

Study Questions

Amazon Leadership Principle: Customer Obsession

Fundamental focus on customer needs and experience, not internal convenience. Gathering feedback from customers, understanding how your systems impact users, making decisions that prioritize customer value, thinking about long-term customer relationships.

Practice Interview

Study Questions

Frequently Asked Site Reliability Engineer (SRE) Interview Questions

Incident Management and ResponseMediumTechnical

68 practiced

Describe a reliable canary deployment strategy on Kubernetes that includes automated rollback. Explain how you would configure readiness/liveness probes, choose metrics to observe during the canary window, implement promotion and rollback triggers, and add safeguards to avoid cascading rollbacks across regions.

Sample Answer

Requirements and constraints:- Target: automated canary on Kubernetes with safe promotion/rollback and no cascading cross-region rollbacks.- Constraints: zero-downtime, automated decision based on SLOs, short canary window (e.g., 5–15m), ability to abort/pause.

High-level approach:- Use a canary controller (Argo Rollouts or Flagger) to manage weighted traffic shifts and automation hooks.- Define strict readiness/liveness probes so unhealthy pods never receive traffic.- Observe a small, representative percent of traffic (start 1–5%) and incrementally increase if metrics pass checks.

Readiness/Liveness configuration:- Liveness: conservative HTTP TCP check hitting /healthz with timeout 1s, failureThreshold 3 — kills stuck processes.- Readiness: deeper endpoint /ready that verifies downstream dependencies (DB, caches), timeout 500ms, period 5s, failureThreshold 3 — prevents routing to partially-initialized pods.

Example (Rollout with Argo Rollouts):

yaml

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: svc}
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 10m}
      - setWeight: 25
      - pause: {duration: 10m}
      - setWeight: 100
      analysis:
        templates:
        - templateName: canary-analysis

Metrics to observe in the canary window:- Primary SLO-based signals: error rate (4xx/5xx), request success ratio vs SLO (e.g., p99 latency < target).- Latency: p50/p95/p99 query latency and tail latency spikes.- Saturation/health: CPU, memory, threadpool/backpressure metrics, GC pauses.- Business metrics if available: checkout conversion, API usage.- Infrastructure signals: pod restarts, container OOMs, socket errors.

Promotion and rollback triggers:- Define analysis templates (Argo) or Prometheus alerts (Flagger) that evaluate metrics over the pause window: - Pass: error rate within SLO and latency percentiles within threshold (e.g., p95 < +20% of baseline). - Fail: error rate increase > X% absolute (e.g., +0.5–1%) or relative (e.g., >200% of baseline) OR p95 latency degradation > 50% for two consecutive windows.- Automated actions: - On pass: advance weights per steps and ultimately promote to 100%. - On fail: immediate rollback to previous ReplicaSet and create a short mitigation window (freeze further promotions).- Use consecutive-window confirmation to avoid noisy single-window flips.

Safeguards to avoid cascading rollbacks across regions:- Region isolation: run canaries independently per region/cluster; do not propagate rollbacks automatically across regions.- Quorum-based rollback: require n out of m regions to fail before global rollback or coord via central orchestration only when cross-region correlated failures detected.- Staggered rollout schedule: do region-by-region rollout with delays and manual approval threshold for global scale.- Circuit breakers and feature flags: if a failure is suspected to be feature-driven, flip feature flag for only canary group.- Rate-limited rollback: prevent automatic rollback from triggering other automated pipelines; attach a cooldown period (e.g., 30m) after rollback.- Alerting and runbook trigger: on automated rollback, create incident with context, metrics and links; require human review before redeploying.

Operational considerations:- Baseline collection: capture production baseline metrics for comparison using Prometheus/Thanos.- Alert tuning: avoid super-sensitive thresholds; use relative and absolute thresholds and require sustained deviation.- Runbooks, observability links, and automated dashboards in the analysis template to speed triage.- Post-mortem and rollback telemetry to refine thresholds and reduce false positives.

This design balances automation with safety: strict probes keep unhealthy pods out of rotation, SLO-driven metrics direct promotion/rollback, and isolation plus quorum/cooldown prevents cascading rollbacks across regions.

Infrastructure as Code and Configuration ManagementMediumTechnical

31 practiced

Compare Ansible and Terraform with respect to provisioning OS-level configuration (packages, services) and cloud resources. For common SRE tasks such as package installs, service configuration, and provisioning cloud infrastructure, explain when to use each tool and how to integrate them without causing conflicts.

Sample Answer

Short answer: Use Terraform to declare and manage cloud resources (networks, VMs, load balancers, managed DBs). Use Ansible for OS-level configuration inside those resources (package installs, services, runtime configs). Keep a clear separation of concerns, integrate via orchestration steps, and avoid race conditions by using each tool’s strengths.

Why:- Terraform is a desired-state, immutable-infrastructure tool with strong lifecycle and dependency management for cloud APIs. It tracks state, plans changes, and is ideal for provisioning resources.- Ansible is an agentless configuration-management/orchestration tool that excels at idempotent changes on running machines (apt/yum, systemd, config files, templates).

When to use which (examples):- Provision EC2/GCE instances, VPCs, IAM, managed RDS: Terraform.- Install nginx, configure /etc/nginx/nginx.conf, enable systemd service, deploy app artifacts: Ansible.- For cloud-init or image baking (Packer): consider baking common packages so Ansible surface area is smaller.

Integration pattern (recommended):1. Terraform apply: create network, instances, security groups, and output connection details (IPs, instance IDs, SSH keys).2. Inventory generation: use Terraform outputs to feed dynamic inventory (Terraform file, inventory script, or Terraform + local_file to produce Ansible inventory).3. Ansible playbooks: configure packages/services, deploy app, run health checks.4. Use orchestration tooling (CI/CD) to run steps sequentially and gated by Terraform plan/apply success.

Avoiding conflicts:- Single source of truth: don’t have both tools manage the same resource. E.g., let Terraform set instance user-data and tags; don’t have Terraform also manage package lists that Ansible changes.- Immutable attributes in Terraform: treat resources created by Terraform as “owned” — avoid external mutation that Terraform can’t see.- Use lifecycle ignore_changes for attributes you want Ansible to manage (e.g., metadata or tags) so Terraform won’t attempt to revert Ansible changes.- Idempotency & retries: ensure Ansible playbooks are idempotent and safe to re-run; use handlers for service restarts.- Locking and CI: use remote Terraform state with locking (S3+DynamoDB, GCS) and run Terraform in CI to prevent concurrent applies. Similarly, coordinate Ansible runs via orchestration (e.g., Ansible Tower/AWX).

Advanced patterns:- Use Terraform provisioners only sparingly (remote-exec, local-exec) for bootstrapping; prefer Ansible for full configuration.- Consider image baking (Packer + Terraform) to bake as much configuration as possible to reduce run-time drift.- For config drift detection, have periodic Ansible convergence jobs and Terraform plan diffs as part of CI.

This separation leverages Terraform for lifecycle of cloud infra and Ansible for mutable, detailed OS-level configuration while preventing ownership overlap and drift.

Database Selection and Trade OffsMediumTechnical

45 practiced

Compare managed relational offerings (AWS RDS Postgres, Google Cloud Spanner, Azure Cosmos DB SQL API) for a globally-distributed metadata service requiring consistent reads/writes across regions with 99.99% availability. Discuss trade-offs in latency, consistency model, operational overhead, operational tooling, and cost under expected scale.

Sample Answer

Situation/goal: We need a globally-distributed metadata service with strongly consistent reads/writes across regions and 99.99% availability. Below I compare AWS RDS (Postgres), Google Cloud Spanner, and Azure Cosmos DB (SQL API) across latency, consistency, operational overhead, tooling, and cost from an SRE perspective.

1) Latency- Spanner: Optimized for geo-distribution with TrueTime; single-digit to low-double-digit ms for regional reads/writes if using regional config, writes across regions incur Paxos/TrueTime coordination so higher but predictable latency.- Cosmos DB: Multi-region writes with configurable consistency; offers low read latency (~single-digit ms) via local replicas; multi-master writes give low write latency but conflict resolution possible.- RDS Postgres: Best for single-region low latency. Cross-region synchronous/async replication (Aurora Global DB) adds significant write latency or relaxed consistency if async.

2) Consistency model- Spanner: Strong external consistency globally — ideal for metadata requiring global serializability.- Cosmos DB: Tunable (Strong, Bounded Staleness, Session, etc.). Strong guarantees only within a single write region; multi-master strong consistency is limited; conflict handling is needed.- RDS: Strong consistency within primary region. Cross-region strong consistency requires synchronous replication which hurts latency; most managed setups are eventually consistent across regions.

3) Operational overhead & tooling- Spanner: Minimal operational burden for replication/partitioning; schema changes online; good SLA and monitoring via Cloud Console/Stackdriver; fewer operational tasks.- Cosmos DB: Managed scale, multi-region config easy in portal/ARM; need to manage RU provisioning, conflict resolution, indexing policies; good metrics and diagnostics in Azure Monitor.- RDS: Most operational work (backups, failover, read replicas) handled but you manage replicas, failover testing, schema migrations, cross-region failover playbooks; higher runbook complexity.

4) Cost under expected scale- Spanner: Higher baseline cost (nodes) but predictable at scale; cost-effective for strong global consistency and heavy write workloads.- Cosmos DB: Pay-per-throughput (RUs) can be expensive if metadata has varied spikes; multi-region write increases cost; may be cheaper for read-heavy patterns.- RDS: Lower per-VM cost for single region; cross-region replicas and high-availability add cost; at global scale, operational and replication costs can exceed managed global DBs.

Recommendation (SRE view)- If strict global strong consistency and simplified ops are primary: choose Spanner.- If you need flexible consistency, very low local read latency, and multi-master writes with careful conflict handling: consider Cosmos DB.- If workload is primarily single-region or you can tolerate eventual cross-region consistency and want SQL/Postgres ecosystem: RDS (Aurora Global DB) is appropriate but requires more operational runbooks to reach 99.99% globally.

Operational notes: validate failover scenarios, SLOs for cross-region write latencies, automate chaos/failover tests, monitor tail latencies and conflict rates (Cosmos), and model costs with realistic RU/node usage and multi-region replicas.

Incident Communication and DocumentationMediumTechnical

61 practiced

Design a quality checklist and outline an automation script (pseudocode acceptable) to validate new incident tickets for completeness: severity, impact, steps taken, owner, initial notes, and required labels for routing. Include automated reminders for missing fields and automatic labeling logic.

Sample Answer

Quality checklist (what must be present before an incident is "complete"):- Severity: mapped to agreed SLO tiers (P0/P1/P2), objective criteria listed- Impact: systems/services affected, user-visible impact, estimated user count- Steps taken: actions already executed (commands, runbooks referenced)- Owner: single on-call owner assigned (user id, rotation metadata)- Initial notes: concise timeline, first observed timestamp, alert source- Required labels for routing: service=<name>, component=<component>, region=<region>, type=(outage/degradation/maintenance), priority=<sev>

Why: enforcing these reduces mean time to acknowledge/resolve and avoids handoff friction.

Automation pseudocode (polls ticket system on create/update, enforces fields, reminders, auto-labeling):

python

# pseudocode (Python-style)
TICKET_API = TicketClient(...)
NOTIFY = NotificationClient(...)
RECHECK_DELAY = 15*60  # 15 minutes
REMINDER_BACKOFF = [15*60, 30*60, 60*60]  # seconds

REQUIRED_FIELDS = ["severity","impact","steps_taken","owner","initial_notes"]
LABEL_RULES = [
  {"if": lambda t: "database" in t.title.lower() or "db" in t.body.lower(), "add":"component:database"},
  {"if": lambda t: t.metadata.get("source")=="pagerduty", "add":"source:pagerduty"},
  {"if": lambda t: t.region in ["us-east-1","eu-west-1"], "add":f"region:{t.region}"}
]

def validate_ticket(ticket):
    missing = [f for f in REQUIRED_FIELDS if not ticket.get(f)]
    # check severity mapping
    if ticket.get("severity") and ticket.severity not in ["P0","P1","P2","P3"]:
        missing.append("severity(invalid)")
    return missing

def apply_labels(ticket):
    labels_to_add = []
    for rule in LABEL_RULES:
        if rule["if"](ticket):
            labels_to_add.append(rule["add"])
    ticket.add_labels(labels_to_add)

def handle_new_or_update(ticket):
    apply_labels(ticket)
    missing = validate_ticket(ticket)
    if not missing:
        ticket.mark("validated")
        return
    # create reminder workflow state
    for delay in REMINDER_BACKOFF:
        NOTIFY.send(to=ticket.owner or "on-call", subject="Ticket incomplete",
                    body=f"Missing fields: {missing}. Please update within {delay//60} minutes.")
        sleep(delay)
        ticket.refresh()
        missing = validate_ticket(ticket)
        if not missing:
            apply_labels(ticket)
            ticket.mark("validated")
            return
    # escalations after reminders
    NOTIFY.escalate(group="ops-leads", subject="Unresolved incomplete ticket", body=f"{ticket.id} missing {missing}")
    ticket.add_label("needs-escalation")

Key points:- Integrate with ticketing API webhooks to run handle_new_or_update on create/update (avoid polling where possible).- Use idempotent label operations and store reminder state on ticket (custom fields) to avoid duplicate messages.- Escalation after configurable retries; include audit trail of reminders.- Edge cases: owner unset → notify rotation; automated systems creating tickets → stricter rules or bypass with dedicated label; race conditions on concurrent updates handled via optimistic locking.

Container Orchestration and Kubernetes OperationsMediumSystem Design

72 practiced

Design a safe deployment plan for a microservice that requires a backward-incompatible database schema change. In your plan describe use of Kubernetes rollout strategies (canary, blue-green, rolling), feature flags, migration steps, verification, and rollback procedures to minimize user impact.

Sample Answer

Requirements & constraints:- Backward-incompatible DB schema change must not break live traffic; near-zero downtime and safe rollback required.- Kubernetes environment, CI/CD pipelines, ability to route traffic (Ingress/Service mesh), feature-flag system, monitoring/alerts.

High-level plan (stepwise):

1. Prep & compatibility layer- Add compatibility code: Deploy a version of the microservice that can read both old and new schema (expand-contract pattern). For writes, write in new format while still populating fields used by old code (dual-write) or write new columns while keeping old ones populated.- Add DB migration scripts that are idempotent and reversible where possible; include feature-flag gates to toggle use of new fields.

2. Migration staging- Run schema changes that are non-destructive first: add new columns/indexes/tables (ONLINE ALTER if supported). Avoid dropping/renaming until consumers switched.- Backfill data into new columns using a background job; run at low priority with monitoring of load.

3. Canary deployment (Kubernetes)- Create a new image that uses the new schema behind a feature flag.- Use a canary rollout: deploy 1-5% traffic to canary using Service mesh (Istio) or weighted Ingress. Monitor key SLOs (error rate, latency, DB write errors, tail latencies, CPU/mem, DB replication lag).- Verify behavior for several hours under production traffic patterns. If anomalies, rollback to previous image immediately (see rollback below).

4. Gradual ramp + verification- If canary succeeds, progressively increase weight (5→25→50%). At each stage run automated verification: - Smoke tests, synthetic transactions, data consistency checks (compare reads from old/new columns). - End-to-end user journeys and critical path synthetic checks. - DB integrity checks and monitoring dashboards.

5. Blue-Green (optional for risky cutovers)- For zero-risk switching of in-flight connections, prepare a full green environment (new version + new schema usage enabled). Switch traffic atomically in the load balancer or service mesh once verification passes. Keep blue live for quick rollback.

6. Final destructive migration- Once all traffic uses new fields and old service code removed, schedule destructive changes (DROP/RENAME) during maintenance window.- Run final cleanup migration and remove compatibility code.

Feature flags & safety gates- Gate schema-using code behind feature flags with ability to toggle per-instance or per-user.- Use kill-switch that can disable new-schema code instantly without redeploy.- Use config-managed feature flags (LaunchDarkly, Unleash, or env-config + Redis) integrated with health checks.

Verification & monitoring- Define acceptance criteria: errors < X, latency < Y, DB error rate stable, data divergence < Z.- Automated canary analyzer (Kayenta-like) to compare metrics.- Add DB-specific alerts: replication lag, migration job failures, slow queries.

Rollback procedures- App-level rollback: revert to previous image (Kubernetes rollout undo) or flip feature flag off to revert behavior if compatibility code allows.- DB rollback: non-trivial — prefer reversible migrations and keep old columns until app rollback complete. If destructive change already applied, use backups/point-in-time restore; include a runbook with steps and estimated RTO/RPO.- Practice rollbacks in staging and runbook drills quarterly.

Operational notes- Automate everything in CI/CD: migration runbooks, preflight checks, canary orchestration.- Communicate schedule to stakeholders and set on-call rotation during rollout.- Post-deployment: run postmortem and remove compatibility artifacts after N weeks once confidence established.

This plan minimizes user impact by using non-destructive schema evolution, feature flags for instant control, progressive Kubernetes rollouts for small blast radius, thorough verification, and rehearsed rollback steps.

High Availability and Disaster RecoveryHardTechnical

74 practiced

As an SRE lead with a limited budget, how would you prioritize disaster recovery investments across many services? Explain the criteria you would use, how you would quantify risk versus benefit, and how you would communicate and negotiate these priorities with product teams and executives.

Sample Answer

I’d take a risk-weighted, business-aligned approach that maximizes reliability gains per dollar and creates a clear, negotiated roadmap.

1) Criteria for prioritization- Business impact: revenue, user sessions, regulatory penalties, customer SLAs; map services to business-criticality tiers.- Technical recovery needs: RTO/RPO targets, blast radius, dependencies.- Likelihood: historical incident frequency, change-rate, code churn, deployment risk.- Cost & complexity: engineering effort, infra spend, operational overhead.- Observability: current SLI/SLO coverage and testability.

2) Quantify risk vs benefit- Estimate Annualized Loss Expectancy (ALE): ALE = impact_per_incident * probability_per_year.- For each DR candidate, estimate Risk Reduction (ΔALE) and Cost to implement (CapEx + OpEx).- Compute ROI = ΔALE / Cost and rank by risk-reduction-per-dollar. For near-term decisions, use Pareto (top 20% services cause 80% exposure).- Include non-monetary benefits (compliance, brand trust) via weighted scores.

3) Tactical investments (examples)- Low cost/high benefit: automated backups, tested runbooks, targeted observability gaps, cross-region replication for top-tier services.- Medium: automated failover, infrastructure-as-code for recovery, periodic DR drills.- High: full active-active redesign for a small set of tier-1 services only.

4) Communicate & negotiate- Tailor messaging: one-pager for execs with ALE, ROI, and proposed roadmap; heatmap for PMs/eng showing SLO breach risk and customer impact.- Propose phased MVPs with measurable milestones and risk-reduction KPIs (ΔMTTR, ΔMTTA, lower error budget burn).- Offer trade-offs: increase SLO/feature constraints vs. risk mitigation; use error budget buy-back for teams to fund reliability work.- Align incentives: embed reliability outcomes into product KPIs and shared on-call commitments.- Use data from tabletop/drills to de-risk decisions and secure recurring budget.

This approach makes DR spending transparent, measurable, and tied to business outcomes so stakeholders can prioritize rationally under budget constraints.

Incident Management and ResponseHardTechnical

68 practiced

A canary deployment in region A shows a gradually increasing error rate correlated with a new dependency call; upstream teams report no changes. The canary degrades only under peak load. Outline a prioritized action plan that minimizes customer impact while collecting evidence: include immediate mitigation (traffic shift, throttles), telemetry to enable, rollback criteria and process, and coordination steps across internal and third-party teams.

Sample Answer

Priority action plan (fast → thorough), focused on minimizing customer impact while collecting evidence.

1) Immediate mitigation (first 0–15 min)- Shift traffic away from canary: reduce canary % to 0–5% using feature-flag/orchestration (k8s/traffic-split). If autoscaling allows, drain canary pods.- Apply conservative throttles & circuit breakers on the problematic dependency call (per-user or per-service rate limits) to avoid cascade.- Enable fail-open vs fail-closed depending on business impact (prefer fail-safe that returns cached/fallback responses).- If degradation is severe under peak load, temporarily route high-risk traffic (heavy users, background jobs) to stable region B.

2) Telemetry to enable (immediate + 15–60 min)- Turn on fine-grained distributed tracing (trace sampling for canary traces) and capture correlation IDs on dependency call.- Increase logging level for the service and dependency client, capture request/response payloads (sampled) and full error stacks.- Emit metrics split by: region, canary vs baseline, endpoint, downstream-dependency, client type, and load bucket; track p50/p95/p99 latency, error rate, connection errors, timeouts, 5xx types.- Start synthetic/health checks that exercise the dependency under controlled loads to reproduce.- Collect resource metrics (CPU, memory, sockets, connection pool saturation) on canary instances.

3) Evidence collection & analysis- Correlate spikes with dependency call latencies/timeouts and upstream logs. Save raw logs/traces for vendor support (timestamps, request IDs).- Capture thread dumps, heap, connection pool stats if suspect client-side resource exhaustion.- Reproduce with controlled load tests against the upstream endpoint (with permission).

4) Rollback criteria & process- Define quantitative rollback triggers: sustained error-rate > X% (e.g., >2% user-facing errors or >SLO breach) for Y minutes (e.g., 5–10m), p99 latency > threshold, or increased customer-impact metrics (aborted transactions).- If trigger hit, roll back canary to previous version or fully disable feature via feature flag; use automated rollback pipeline if validated, else manual with runbook.- Post-rollback: verify baselines (latency/error back to normal) for 2 successive monitoring intervals before closing incident.

5) Coordination & communications- Assign Incident Commander and clear owner for dependency path. Open incident channel (Slack/Zoom) including service owners, upstream-team, third-party vendor/support, product/ops.- Share evidence bundle (traces, logs, synthetic results, timestamps, request IDs) and agreed timeline. Request upstream to validate no hidden deploys/config changes.- If third-party, open support ticket and escalate with SLA evidence; if contractual impact, involve procurement/PM.- Provide regular status updates to stakeholders and customer-facing comms team if user-visible.

6) Follow-up actions- Run a blameless postmortem: root cause, mitigations, improved SLOs, additional tests, circuit-breaker thresholds, dependency SLAs.- Harden: add canary throttling automation, dependency chaos tests, richer telemetry and automated rollback rules.

Rationale: shift traffic quickly to limit impact, collect high-fidelity traces/logs to prove root cause, use quantified rollback gates to avoid flip-flopping, and coordinate with all stakeholders to expedite resolution and preserve evidence for third-party debugging.

Infrastructure as Code and Configuration ManagementHardSystem Design

33 practiced

Design an IaC strategy for a SaaS product deployed to both AWS and GCP. Requirements: single source of truth where possible, provider-agnostic modules for common patterns, separate state per cloud/environment, secure secret handling, CI pipelines that can run provider-specific applies, and operational runbooks for rollbacks. Outline repo structure, module patterns, and key operational controls an SRE would own.

Sample Answer

Requirements recap: single source of truth where possible, provider-agnostic modules for common patterns, separate state per cloud/environment, secure secret handling, CI pipelines that can run provider-specific applies, and operational runbooks for rollbacks.

High-level approach- Use Terraform as the canonical IaC (HCL) for provider-agnostic resource definitions, plus small provider-specific wrapper modules. Use Terragrunt or a lightweight orchestration layer to manage remote state/backends per cloud/env and to compose modules.- Centralize shared logic in platform modules (network, compute, service-account, monitoring, k8s) that accept provider-agnostic inputs; implement cloud specifics behind adapters.

Repo structure (example)- infra/ - modules/ - platform/ # provider-agnostic constructs (VPC-like abstraction, IAM roles model) - storage/ # abstract storage interface - compute/ # abstract compute interface (k8s cluster, instance group) - provider-adapters/ # aws/, gcp/ — small modules mapping abstract resources -> provider resources - live/ - aws/ - staging/ - terragrunt.hcl - prod/ - gcp/ - staging/ - prod/ - global/ - dns/ - iam/ - tools/ - ci-templates/ - scripts/

Module patterns- Design modules as intent-first (e.g., module "network" exposes subnets, nat, route table) rather than cloud API-first.- Keep provider-specific code in short adapter modules that translate intent to aws_* or google_* resources.- Keep state minimal in modules; outputs are explicit contracts.- Provide examples and unit tests (terratest) for each module.

State management- Separate remote state per cloud and per environment: - AWS: S3 backend + DynamoDB for locking; unique bucket per env/region with encryption (SSE-KMS). - GCP: GCS backend + storage bucket with object versioning + Cloud KMS; use firestore/lock mechanism or use Terraform Cloud/Enterprise remote state for cross-cloud locking.- Isolate global shared state (e.g., DNS) in its own state file.

Secrets- Don’t store secrets in state or repo.- Use HashiCorp Vault as primary secret store; integrate with Terraform via Vault provider and use short-lived dynamic credentials where possible.- Use cloud KMS/Secret Manager only for cloud-specific secrets; Terraform references secrets by data sources populated at plan/apply time.- CI injects secrets via pipeline vault integrations, not pipeline env vars persisted in logs.

CI/CD and provider-specific applies- CI pipelines are declarative (e.g., GitHub Actions/GitLab CI/Jenkins). Steps: 1. Validate / fmt / module unit tests (terratest) 2. Plan in a read-only, isolated environment; store plan artifact 3. Run policy checks (OPA/Rego or Sentinel) against plan 4. If approved, pipeline triggers apply using the plan in a provider-specific runner with required cloud creds fetched from Vault.- Use separate runners/agents for AWS and GCP applies to ensure credentials never cross providers.- Support manual approvals and automated canary applies (apply to staging/canary first, monitor SLOs, then promote).

Operational runbooks and rollback- Maintain runbooks paired with each deployable unit: - Preconditions, quick rollback steps (terraform destroy/taint is last resort) - Step-by-step rollback using previously saved Terraform plan or state snapshot (state versioning + restore) - Rollback playbooks to revert to previous module version via git tag and re-apply. - Post-rollback verification checklist (health checks, SLO validation, incident logging).- Automate state snapshots before every prod apply (encrypted archive to isolated bucket).- Canary and feature-flag driven rollbacks to reduce blast radius.

Key operational controls SRE owns- Platform CI pipeline ownership (secrets, runners, approval gates)- Remote state lifecycle and access control (IAM policies, KMS keys)- Policy-as-code (Rego/OPA) for security/compliance checks- Automated drift detection and scheduled terraform plan comparisons with alerting- Monitoring/alerting tied to deploys (deploy SLOs, deployment failure alerts)- Backup/restore procedures for state and critical configs- Incident runbooks, rollback rehearsals (chaos/cutover drills)- Access reviews and least-privilege for Terraform service accounts- Cost guardrails and tagging enforcement

Trade-offs- Terraform + adapter pattern reduces duplication but needs discipline to keep adapter layer thin.- Terragrunt simplifies DRY but adds another tool to maintain.- Vault centralizes secrets but is operational overhead — mitigate with managed Vault or strong automation.

This strategy yields a single intent-driven source of truth, provider-specific isolation where required, secure secret handling, safe CI/CD applies, and operational runbooks SREs can execute during incidents.

Database Selection and Trade OffsEasyTechnical

38 practiced

Describe workloads and trade-offs for key-value stores (Redis, DynamoDB) used as primary storage versus as a cache. Discuss persistence/durability options, eviction strategies, TTL usage, memory vs disk trade-offs, and scenarios where an in-memory KV store is acceptable as primary storage versus when persistence is required.

Sample Answer

Key-value stores (Redis, DynamoDB) can be used either as caches or primary storage; each choice has workload patterns and trade-offs SREs must weigh.

Workloads & trade-offs- As cache: read-heavy, low-latency, ephemeral data (session tokens, rendered fragments). Benefits: sub-ms reads, reduced backend load. Risk: cache misses, stale data; acceptable to lose entries.- As primary store: source-of-truth data (leaderboards, user profiles if small), requires durability, backups, consistent reads/writes. Benefits: simplicity and speed; risks: data loss if durability weak, limited queryability.

Persistence / durability- Redis: RDB snapshots (point-in-time, fast recovery but potential data loss between snapshots), AOF (append-only log, configurable fsync for better durability), combination (AOF + RDB) and Redis Cluster for HA. For strict durability use AOF with fsync=always or use disk-backed DB.- DynamoDB: fully managed, durable by default (replicated across AZs/regions depending on config), strong/ eventual consistency options, built-in backups and point-in-time recovery.

Eviction strategies & TTL usage- Redis eviction policies: noeviction, allkeys-lru, volatile-lru, TTL-based (volatile-ttl), random. Use volatile policies when only some keys expire; allkeys-lru for global space limits.- TTLs: good for expiring sessions, caches; avoid relying on TTLs for critical business data. Use TTLs with leaky- bucket workloads to avoid thundering herd on expiry.

Memory vs disk trade-offs- In-memory (Redis): fastest, but memory-constrained and expensive; requires eviction or sharding. Persistence adds IO overhead and may increase recovery time.- Disk-backed (DynamoDB or Redis on Flash): cheaper storage, higher latency, throughput limits. Choose based on SLOs (p99 latency vs cost).

When in-memory KV is acceptable as primary- Acceptable: non-critical, easily recomputable data (caches warmable), low-RPO tolerance, applications requiring extreme low latency where occasional loss is tolerable (leaderboards, analytics counters with periodic reconciliation).- Not acceptable: single source of truth for financial transactions, order state, user data that cannot be reconstructed or tolerated to be lost; there use durable stores (DynamoDB with backups, RDBMS) and strong consistency.

Operational recommendations (SRE)- Define SLOs for durability and latency; choose persistence settings accordingly.- Use replication, backups, and automated failover for primary usage.- Monitor memory pressure, eviction metrics, AOF rewrite times, and restore drills.- Combine: durable DB as primary + Redis as cache with cache-aside pattern and proper TTLs/eviction to balance cost and performance.

Incident Communication and DocumentationMediumTechnical

100 practiced

Describe a verified handoff process and supporting tooling (structured checklists, chat handoff, linked tickets, read receipts) to ensure continuity when on-call shifts change. Explain how you would measure handoff reliability over time and what SLAs you might apply.

Sample Answer

Situation: As an SRE, smooth on-call handoffs prevent missed alerts and reduce MTTI/MTTR. I implement a verified handoff process combining structured checklists, chat handoff, linked tickets, and read receipts.

Process (step-by-step):- Outgoing fills a short standardized checklist in a shared runbook page (service health, degraded components, active incidents, deploys in flight, pending alerts, mitigation steps, escalation contacts). Template enforced by a bot.- Outgoing creates/updates linked tickets in the tracking system and pins them to the runbook entry.- Outgoing posts a concise handoff message in the on-call Slack channel with a link to the checklist + active ticket IDs; the bot timestamps it.- Incoming must acknowledge (react or reply) within a defined window and paste a short "I confirm" in the channel — that acknowledgement is the read receipt.- If no ack within SLA, escalation policy triggers paging of secondary on-call.- Handoff bot captures metadata: timestamps, user IDs, links; stores snapshots for audits.

Supporting tooling:- Runbook storage: Confluence/Notion/Git repo with templated checklist and history.- Scheduler & alerting: PagerDuty/Opsgenie for schedule, escalation, and automatic paging.- ChatOps: Slack/MS Teams plus a handoff bot that posts templates, enforces ack, and records read receipts.- Ticketing: Jira/GitHub linked to runbook entries; automation that links/creates issues from active alerts.- Observability: Dashboard snapshot link (Grafana) included automatically.

Measuring reliability:- Handoff completion rate = % of shifts with checklist + ack within SLA.- Mean time to acknowledge (MTTA) for incoming = time from handoff post → acknowledgement.- Missed-handoff incidents: count of alerts arriving within first X minutes after shift change where no ack existed.- Audit log coverage: % of handoffs with complete metadata (links/tickets/snapshots).- Trends and heatmaps by shift, team, person to spot weak points.

SLAs (examples, adjustable by service criticality):- Handoff posted by outgoing within 10 minutes before shift end.- Incoming acknowledgment within 15 minutes of post (critical services: 5–10 minutes).- Active incident transfer completed (ticket linked + owner assigned) before shift end; if unresolved, runbook updated within 30 minutes.- If no ack within SLA, secondary paged within 5 minutes.- Documentation updates for post-change operations within 1 hour.

Tie to SLOs:- Use missed-handoff incidents and MTTA as input to reliability reviews; set thresholds (e.g., <1 missed handoff/month, handoff completion rate >99%) and treat failures as blameless incidents driving process or tooling changes.

This combination of clear checklist, enforced chat acknowledgement, linked tickets, automated capture, and measurable SLAs provides verifiable continuity and continuous improvement.

Practice Site Reliability Engineer (SRE) questions across all topics

Additional Information

Want to create your own tailored preparation guide using our deep research?

Get Started for Free

Interview-Ready Courses

Visual-first, interactive, structured learning paths

Browse Site Reliability Engineer (SRE) jobs

AI-enriched listings across hundreds of company career pages

Explore Jobs

Staff-Level Site Reliability Engineer Interview Preparation Guide for Amazon

Interview Process Overview

Interview Rounds

Recruiter Screening

What to Expect

Tips & Advice

Focus Topics

Amazon Leadership Principles Alignment

Practice Interview

Study Questions

Scale and Systems Expertise

Practice Interview

Study Questions

Career Progression to Staff Level

Practice Interview

Study Questions

Technical Phone Screen 1: Distributed Systems and Infrastructure Architecture

What to Expect

Tips & Advice

Focus Topics

Load Balancing, Service Discovery, and Traffic Management

Practice Interview

Study Questions

Resilience Patterns and Failure Mode Analysis

Practice Interview

Study Questions

AWS Services and Cloud Infrastructure Mastery

Practice Interview

Study Questions

Infrastructure Automation and Infrastructure as Code (IaC)

Practice Interview

Study Questions

Distributed Systems Architecture and Design Patterns

Practice Interview

Study Questions

Technical Phone Screen 2: Observability, Monitoring, and Incident Response

What to Expect

Tips & Advice

Focus Topics

Distributed Tracing for Complex Systems

Practice Interview

Study Questions

Logging, Log Analysis, and Centralized Logging Infrastructure

Practice Interview

Study Questions

Observability and Monitoring Strategy Design

Practice Interview

Study Questions

Incident Response Framework and Management

Practice Interview

Study Questions

Error Budgets and Service Level Objectives (SLOs)

Practice Interview

Study Questions

Onsite Round 1: System Design - Large-Scale Reliable Service

What to Expect

Tips & Advice

Focus Topics

Cost Optimization and Resource Efficiency

Practice Interview

Study Questions

Scalability, Capacity Planning, and Performance

Practice Interview

Study Questions

Operational Aspects of System Design

Practice Interview

Study Questions

Data Architecture for Durability and Consistency

Practice Interview

Study Questions

High-Availability System Architecture Design

Practice Interview

Study Questions

Onsite Round 2: Infrastructure Automation and Deployment Pipelines

What to Expect

Tips & Advice

Focus Topics

Configuration Management and Avoiding Configuration Drift

Practice Interview

Study Questions