Apple Site Reliability Engineer (Staff Level) Interview Preparation Guide

Site Reliability Engineer (SRE)

Apple

Staff

7 rounds

Updated 6/18/2026

Apple's SRE interview process for Staff-level candidates is comprehensive and spans 6-7 weeks. The process includes an initial recruiter screening, a technical phone screen with the hiring manager, followed by a full-day virtual onsite loop consisting of 4-5 technical and behavioral rounds, and concludes with manager feedback and senior manager discussions. Apple emphasizes depth of systems knowledge, incident management expertise, and cultural alignment. The interview process is relatively unstructured compared to other tech companies, with significant variation between teams. For Staff-level SRE candidates, expect rigorous evaluation of architectural thinking, distributed systems expertise, and leadership capabilities.

Interview Rounds

Recruiter Screening

25 min3 focus topicsbehavioral

Technical Phone Screen - Hiring Manager

60 min5 focus topicsbehavioral|technical

What to Expect

60-minute phone interview with the Apple hiring manager covering technical depth, incident management experience, and behavioral assessment. The hiring manager will dig deep into your resume, focusing on specific projects where you demonstrated reliability engineering expertise. Expect questions about your most complex incidents, how you handled operational challenges, your approach to problem-solving, and how you work independently. The hiring manager assesses whether you have the technical breadth/depth and incident response maturity expected at Staff level, plus your ability to work with minimal support.

Tips & Advice

Prepare detailed narratives about 3-4 significant incidents or projects using the STAR method: Situation, Task, Action, Result. Focus on incidents where you identified root causes, drove automation solutions, improved reliability, or led post-incident reviews. Highlight your independence and decision-making at scale. For Staff level, emphasize how you elevated team practices and mentored others. Ask technical questions about the team's challenges and infrastructure. Demonstrate curiosity and strategic thinking about reliability engineering.

Focus Topics

Cross-Functional Collaboration and Influence

Examples of working effectively with software engineers, product teams, and other infrastructure teams. How you communicated reliability concerns and influenced decisions. Experience mentoring junior SREs or engineers on reliability practices.

Practice Interview

Study Questions

Automation and Infrastructure Code Development

Experience building automation frameworks, infrastructure-as-code practices, deployments, monitoring systems, and runbooks. Show examples of automation that reduced manual toil or improved reliability. Discuss your approach to testing infrastructure code.

Practice Interview

Study Questions

Independent Problem-Solving and Decision-Making

Examples of technical problems you solved with minimal guidance, how you approach debugging complex issues, and your decision-making framework when multiple solutions exist. Demonstrate comfort with ambiguity and ability to make trade-off decisions.

Practice Interview

Study Questions

Major Incident Management and Resolution

Deep dive into significant production incidents you managed: initial detection, root cause analysis, mitigation steps, permanent fixes, and post-incident process improvements. Emphasize your role in decision-making, communication, and team coordination. For Staff level, focus on incidents affecting millions of customers or critical service infrastructure.

Practice Interview

Study Questions

Systems Design and Architectural Thinking

How you think about system design at scale, trade-offs between reliability and performance, designing for observability, and improving legacy systems. Discuss examples where you influenced architectural decisions or prevented reliability problems through good design.

Practice Interview

Study Questions

Systems Internals and Linux Troubleshooting

60 min6 focus topicstechnical

What to Expect

60-minute onsite technical round focused on deep Linux systems knowledge and troubleshooting methodology. You'll face scenarios like 'diagnose why SSH is broken with only console access' or 'investigate why a system is experiencing high latency.' The interviewer will expect you to methodically use diagnostic tools (/proc, strace, tcpdump, etc.) and Linux concepts (file descriptors, inodes, system calls, memory management, virtual address space) to isolate root causes. They observe your problem-solving process, how you form and test hypotheses, and your understanding of kernel-level concepts. For Staff level, expect deep dives into complex scenarios involving multiple system layers.

Tips & Advice

Review Linux internals thoroughly: file systems (inodes, dentries, file descriptors), process management (fork, exec, signals), memory management (virtual address space, page tables, swap), I/O subsystem, and IPC mechanisms. Practice using diagnostic tools: /proc filesystem, ps, top, iostat, vmstat, strace, ltrace, tcpdump, netstat, iptables. Be comfortable reading and interpreting system state. Walk through your troubleshooting process step-by-step, explaining your hypotheses and why you're taking each action. For Staff level, show comfort with edge cases and kernel concepts. Discuss how you'd approach this systematically in production without disrupting the system.

Focus Topics

Inter-Process Communication and Networking Stack Basics

IPC mechanisms (pipes, sockets, shared memory, message queues), socket types and protocols, netlink sockets. Basic understanding of how networking integrates with process communication.

Practice Interview

Study Questions

I/O Subsystem and Performance Analysis

Understanding of disk I/O scheduling, block device operations, buffer cache, dirty pages. Using iostat, iotop, and /proc/diskstats to identify I/O bottlenecks.

Practice Interview

Study Questions

Linux Kernel Concepts and File System Architecture

Deep understanding of inode structure, file descriptors, VFS (Virtual File System), filesystem types, and how Linux manages file I/O. Understanding of filesystem journaling, mount operations, and permission model.

Practice Interview

Study Questions

Memory Management and Virtual Address Space

Virtual memory architecture, page tables, memory mapping, swap behavior, OOM killer, memory fragmentation. Understanding of malloc/free and heap management from kernel perspective.

Practice Interview

Study Questions

Advanced Diagnostic Tools and Troubleshooting Methodology

Proficiency with strace, ltrace, perf, flamegraphs, /proc filesystem parsing, and systematic troubleshooting methodology. Building hypotheses and testing them methodically. Understanding what each tool reveals about system state.

Practice Interview

Study Questions

Process Management and System Calls

Understanding of process lifecycle, fork/exec/wait system calls, signal handling, process states, zombie processes, and orphaned processes. How shell interprets commands and creates processes.

Practice Interview

Study Questions

SRE Fundamentals and Distributed Networking

60 min6 focus topicstechnical

What to Expect

60-minute onsite round focused on your understanding of SRE principles, distributed systems concepts, and networking fundamentals. Expect detailed questions about TCP, TLS, HTTP, DNS protocols; how they behave under failure conditions; load balancing strategies; and end-to-end request flows through complex systems (e.g., 'Walk me through what happens when I navigate to icloud.com'). The interviewer assesses your mental models of distributed systems, understanding of service dependencies, and knowledge of common failure modes and mitigation strategies. For Staff level, expect discussions of high-availability patterns, observability strategies, and cross-service reliability considerations.

Tips & Advice

Prepare to explain the complete request flow for a complex service (like iCloud). Be ready to discuss protocols in depth: TCP three-way handshake and connection states, TLS handshake and certificate validation, HTTP request/response cycle, DNS resolution and caching. Understand load balancing algorithms (round-robin, least connections, consistent hashing) and their trade-offs. Know common failure modes (timeouts, retries, circuit breakers, bulkheads) and mitigation patterns. For Staff level, discuss SLOs, SLAs, error budgets, and designing for high availability. Think about observability and what signals you'd need to detect problems.

Focus Topics

DNS Protocol and Name Resolution

DNS query/response format, recursive vs. iterative resolution, caching mechanisms and TTLs, DNS record types (A, AAAA, CNAME, MX, SRV), common DNS issues (NXDOMAIN, servfail), DNS security (DNSSEC). DNS at scale with multiple nameservers.

Practice Interview

Study Questions

Load Balancing Strategies and Algorithms

Load balancing algorithms (round-robin, least connections, IP hash, consistent hashing), session affinity, health checks, connection draining, and choosing appropriate strategies for different workloads. Layer 4 vs. Layer 7 load balancing.

Practice Interview

Study Questions

TLS/SSL and Secure Communication

TLS handshake process, certificate validation, certificate pinning, cipher suite selection, and TLS version negotiation. Understanding of symmetric vs. asymmetric cryptography at a high level. Troubleshooting SSL/TLS errors.

Practice Interview

Study Questions

Distributed Systems Reliability Patterns

Retry logic and exponential backoff, circuit breakers, bulkheads (isolation), timeout strategies, graceful degradation, cascading failure mitigation. Understanding when to apply each pattern and their interactions.

Practice Interview

Study Questions

HTTP Protocol and Web Request Handling

HTTP request/response structure, status codes, headers (Connection, Keep-Alive, Content-Encoding), HTTP/1.1 vs. HTTP/2 differences, chunked transfer encoding. Pipelining and connection reuse.

Practice Interview

Study Questions

TCP Protocol and Connection Management

Three-way handshake, connection states, sequence numbers, flow control, congestion control, TCP_NODELAY, SO_REUSEADDR. Understanding TIME_WAIT, FIN_WAIT states, and connection timeout behavior. Graceful vs. abrupt connection termination.

Practice Interview

Study Questions

Advanced Coding and Data Structures

60 min5 focus topicstechnical

What to Expect

60-minute onsite coding round typically featuring LeetCode medium-to-hard problems, often involving graph algorithms (BFS/DFS), tree traversal, or systems-level coding challenges. You'll write code on a whiteboard or in an IDE. Problems might include finding connected components, topological sorting, graph traversal patterns, or system-related problems like implementing a rate limiter or designing a distributed cache. The interviewer assesses your algorithmic thinking, code quality, edge case handling, and communication about your approach. For Staff level, interviewers expect clean, production-quality code and deep understanding of time/space complexity trade-offs.

Tips & Advice

Practice LeetCode medium-to-hard problems focusing on graphs, trees, and dynamic programming. Master BFS/DFS implementations and recognize patterns. Write clean, readable code with proper variable names and comments. Test your code mentally against edge cases (empty inputs, single elements, circular structures). Explain your approach before coding and discuss complexity trade-offs. For Staff level, show familiarity with systems-level concerns like memory efficiency and concurrent access patterns. Discuss optimization opportunities but prioritize correctness. Ask clarifying questions about problem requirements and constraints.

Focus Topics

Tree Algorithms and Binary Search Trees

Binary tree traversals (inorder, preorder, postorder), tree properties, balanced tree concepts, binary search tree operations, and heap operations.

Practice Interview

Study Questions

Systems-Level Coding Problems

Problems that require thinking about systems concerns: concurrency (locks, atomics), memory efficiency, resource limitations, or distributed behavior. Examples: rate limiter, LRU cache, distributed cache, task scheduler.

Practice Interview

Study Questions

Dynamic Programming and Optimization

Understanding optimal substructure and overlapping subproblems, memoization vs. tabulation, and recognizing when DP applies. Classic problems like knapsack, longest subsequence.

Practice Interview

Study Questions

Code Quality and Defensive Programming

Writing clean, readable code with proper naming conventions. Handling edge cases explicitly. Adding assertions and defensive checks. Explaining time and space complexity clearly.

Practice Interview

Study Questions

Graph Algorithms and Traversal Patterns

BFS and DFS implementations and use cases, topological sorting, finding connected components, cycle detection, shortest path algorithms (Dijkstra, Bellman-Ford). Understanding adjacency list vs. matrix representations.

Practice Interview

Study Questions

System Design - Reliability and Scalability

60 min6 focus topicssystem design

What to Expect

60-minute onsite round where you'll design a complex system focusing on reliability, scalability, and observability. You might be asked to design a distributed monitoring system, a service deployment platform, a highly available data store, or similar infrastructure. For Staff level, expect deep architectural questions about trade-offs, failure scenarios, observability design, and how to maintain reliability at scale. The interviewer probes your ability to think strategically about system design, understand SLO/SLA implications, design monitoring and alerting, plan capacity, and handle failure modes. They assess your communication, technical depth, and ability to make informed architectural decisions.

Tips & Advice

Use a structured approach: clarify requirements and constraints, discuss trade-offs, propose architecture with justification, and deeply discuss failure scenarios and mitigations. For Staff level, focus on: (1) Reliability first - design SLOs/SLAs, error budgets, and failure handling; (2) Observability - how you'd monitor, alert, and debug; (3) Scalability - handling 10x traffic growth; (4) Operational concerns - deployments, rollbacks, runbooks; (5) Team scalability - designing systems that engineers can operate. Draw diagrams showing service dependencies, data flow, and failure domains. Discuss redundancy, replication strategies, and consensus mechanisms when relevant. Show awareness of Apple's scale and use-cases (e.g., iCloud with hundreds of millions of users). Ask clarifying questions.

Focus Topics

Deployment, Rollback, and Operational Strategies

Deployment strategies (blue-green, canary, rolling), rollback procedures, and minimizing blast radius. Designing for operational simplicity and automation. Infrastructure-as-code, provisioning, and deprovisioning.

Practice Interview

Study Questions

Load Balancing and Traffic Management

Designing load balancing strategies (layer 4 vs. layer 7), handling sticky sessions, gradual traffic shifting, circuit breaker design, and managing cascading failures. Designing for graceful degradation under load.

Practice Interview

Study Questions

Observability and Monitoring System Design

Designing comprehensive monitoring and alerting systems that enable rapid problem detection and root cause analysis. Metrics (RED method: Rate, Errors, Duration), logs, traces, and their relationships. Alert design and avoiding alert fatigue. Designing dashboards and runbooks.

Practice Interview

Study Questions

Data Durability and Consistency

Understanding consistency models (strong, eventual, causal), replication and durability guarantees, trade-offs between consistency and availability, backup strategies, and disaster recovery. When to use different databases for different use-cases.

Practice Interview

Study Questions

SLO, SLA, and Error Budget Framework

Defining Service Level Objectives (SLOs) and Service Level Agreements (SLAs), understanding error budgets and their relationship to reliability targets, and making trade-off decisions based on error budgets. Examples: 99.95% availability SLO means ~22 minutes of downtime per month.

Practice Interview

Study Questions

High-Availability Architecture and Redundancy

Designing systems that tolerate component failures (active-active, active-passive, multi-region), replication strategies (master-slave, master-master, quorum-based), consensus protocols (Raft, Paxos at conceptual level), and choosing appropriate strategies for different components.

Practice Interview

Study Questions

Leadership, Cultural Fit, and Senior Manager Round

50 min5 focus topicsbehavioral

What to Expect

This final round combines elements of hiring manager feedback, senior manager discussion, and cultural assessment. It typically lasts 45-60 minutes and may involve the hiring manager providing feedback on earlier rounds and the senior manager discussing motivation, team/organizational challenges, and cultural fit. For Staff level, this round assesses your leadership impact, ability to influence across teams, mentorship capabilities, and alignment with Apple's values. The senior manager wants to understand your strategic thinking about reliability, your vision for how teams should operate, and whether you'll thrive in Apple's culture. You may discuss organizational challenges, how you'd approach them, and your leadership philosophy.

Tips & Advice

Prepare stories demonstrating leadership impact: mentoring junior SREs, influencing other teams to adopt reliability practices, driving organizational change in incident response processes, or leading a significant reliability initiative. Show genuine interest in the team's specific challenges and how you'd help. Demonstrate understanding of Apple's culture (secrecy, quality focus, integrated ecosystem, user privacy). Ask thoughtful questions about team dynamics, organizational structure, and current reliability challenges. For Staff level, discuss your philosophy on SRE practices, how you approach building reliable systems, and how you develop junior engineers. Be authentic about what excites you and what matters to you professionally.

Focus Topics

Apple Culture and Values Alignment

Understanding Apple's focus on quality, integrated ecosystem thinking, user privacy, and design excellence. How these values apply to reliability engineering. Authentic discussion of what appeals to you about Apple's culture and how you'd fit.

Practice Interview

Study Questions

Motivation and Career Goals

Genuine interest in the specific role and team. What draws you to this opportunity. How this fits into your career trajectory. What you want to accomplish in this role. Realistic expectations about the work.

Practice Interview

Study Questions

Strategic Thinking About Reliability Engineering

Your philosophy on SRE practices, how you balance reliability with business goals, thoughts on toil reduction and automation, and your vision for evolving reliability practices. Long-term thinking about infrastructure and systems evolution.

Practice Interview

Study Questions

Cross-Functional Influence and Collaboration

Examples of influencing other teams to adopt reliability practices without direct authority. Working effectively with product teams, engineering leadership, and other infrastructure teams. Communicating reliability concerns in terms other teams understand. Building consensus around reliability decisions.

Practice Interview

Study Questions

Leadership Experience and Team Development

Demonstrated mentorship of junior SREs and engineers, helping them grow and develop expertise. Leading initiatives that improved team capabilities or processes. Creating psychological safety and inclusive team environments. Examples of difficult team situations handled well.

Practice Interview

Study Questions

Frequently Asked Site Reliability Engineer (SRE) Interview Questions

Linux Troubleshooting and DiagnosticsEasyTechnical

30 practiced

You see OOM killer messages in dmesg and journalctl for a long-running service. Explain the steps you would take to determine whether the process was killed due to system OOM or a cgroup memory limit, and how to use /proc and system logs to find evidence. Mention specific files to check and what entries in dmesg look like.

Sample Answer

First, gather the kernel/journal evidence so you can tell whether the kernel global OOM killer fired or a memory cgroup limit triggered a kill.

1) Check kernel messages and journald- dmesg | egrep -i 'oom|killed process|invoked oom-killer' Example kernel lines for system OOM: "Out of memory: Kill process 1234 (myservice) score 987 or sacrifice child" "killed process 1234 (myservice) total-vm:...kB, anon-rss:...kB, file-rss:...kB"- journalctl -k --since "2025-11-22 00:00" or journalctl -u myservice to correlate timestamps and unit logs- For systemd-oomd: journalctl -t systemd-oomd or journalctl -u systemd-oomd

If you see "invoked oom-killer" / "Out of memory" and "killed process ..." it's the kernel/global OOM.

If you see cgroup-specific messages, they usually mention "memory cgroup" or the cgroup path: "memory: usage 104857600 exceeded limit 104857600, kill ..." or "Memory cgroup out of memory: Killed process 2345 (myservice) total-vm:..."

2) Inspect cgroup memory counters- For cgroup v1: /sys/fs/cgroup/memory/<cgroup>/memory.limit_in_bytes /sys/fs/cgroup/memory/<cgroup>/memory.usage_in_bytes /sys/fs/cgroup/memory/<cgroup>/memory.max_usage_in_bytes /sys/fs/cgroup/memory/<cgroup>/memory.failcnt- For cgroup v2: /sys/fs/cgroup/<unit>/memory.current /sys/fs/cgroup/<unit>/memory.max /sys/fs/cgroup/<unit>/memory.events (look for "oom" or "oom_kill")Check timestamps near the event (max_usage >= limit or memory.events showing oom) — that's evidence of a cgroup-enforced kill.

3) Correlate PID/process info via /proc- If dmesg shows PID 1234, check /proc/1234 (if still present) or use the journal to find when it exited.- Examine /proc/<pid>/status from before if you have an audit/metrics snapshot; otherwise check: /proc/<pid>/oom_score /proc/<pid>/oom_score_adjThese show how likely the kernel was to pick that process during a global OOM (higher score = more likely).

4) Additional checks- Check dmesg for stack trace vs cgroup message — global OOM often prints the oom-killer stack and scanner, cgroup kills include cgroup path.- Check container orchestrator logs (docker, kubelet): Docker may log "container killed due to memory limit" and Kubernetes events show OOMKilled in pod status.- Check /var/log/messages or journalctl -b for surrounding memory pressure logs.

Summary of decisive evidence:- Kernel global OOM: dmesg contains "Out of memory: Kill process" / "invoked oom-killer" and process oom_score high.- Cgroup OOM: dmesg/journal contains "memory cgroup" messages or shows limit exceeded; cgroup memory.* or memory.current/max show usage >= limit; memory.events or memory.failcnt increments.

Using these steps you can unambiguously determine which mechanism killed the process and collect timestamps and counters for post-incident analysis.

Incident Management and ResponseHardSystem Design

52 practiced

Design a secure, collaborative incident command tool for enterprise-scale incidents. The tool must support role assignment (IC, scribe, SMEs), timeline management, task tracking, log/trace collection snapshots, communication templates, and integrations with alerting, chat, ticketing, and storage. Outline system components, data model, integration points, availability targets, and strategies to keep the tool operational during major incidents.

Sample Answer

Requirements clarification:- Real-time collaboration (role assignment, timeline, tasks)- Capture immutable snapshots (logs/traces)- Integrations: alerting, chat, ticketing, object/storage- Security: RBAC, auditing, encryption- Must remain usable during large platform outages (degraded modes)

High-level architecture:- Web UI + Mobile clients- Backend API cluster (stateless, containerized)- Real-time pub/sub layer (Kafka + WebSocket gateway / SignalR)- Primary datastore: transactional DB (Postgres, multi-AZ, read replicas)- Event store / timeline: append-only event log (Kafka + compacted topic)- Snapshots storage: object store (S3 or equivalent) with WORM/immutable buckets- Cache/fast reads: Redis for presence, locks, optimistic concurrency- Integration workers: connector fleet to alerting, chat, ticketing, storage (k8s jobs)- Auth & IAM: centralized (OIDC), RBAC service, session service- Audit & compliance: ELK/observability, tamper-evident audit ledger (append-only)- Orchestration: k8s multi-region clusters; control plane in primary region, fallback region warm-standby

Core components & responsibilities:1. API Gateway — authentication, rate limiting, routing.2. Collaboration Service — role assignment, presence, real-time updates via pub/sub.3. Incident Service — creates Incident objects, state machine (triage → active → mitigated → closed).4. Timeline/Events Service — records timeline entries as immutable events; stores human entries and automated events from integrations.5. Task Manager — assign tasks, SLAs, reminders, escalation policies.6. Snapshot Service — collects logs/traces via integrations, stores signed manifests + pointers to object store; supports on-demand and scheduled snapshots.7. Integration Workers — reliable delivery to/from alerting, chat, ticketing; idempotent connectors with backoff and dead-letter queues.8. UI — role-aware views (IC dashboard, scribe timeline, SME task lists), offline-capable PWA for degraded connectivity.9. Observability & Health — SLO dashboards, synthetic transactions, chaos tests.

Data model (simplified):- Incident {id, title, severity, start_ts, status, primary_ic_id, scribe_id, metadata}- RoleAssignment {incident_id, user_id, role, start_ts, end_ts}- TimelineEvent {id, incident_id, timestamp, author_id, type (note, action, alert, snapshot), payload, event_seq}- Task {id, incident_id, title, owner_id, state, due_ts, sla_policy_id, related_event_id}- Snapshot {id, incident_id, created_by, type, storage_uri, hashed_manifest, signed_at}- Integration {id, type, config, status, last_sync}- AuditRecord {id, actor, action, target_type, target_id, timestamp, signature}

Integration points:- Alerting: webhooks / push -> connector ingests alerts into TimelineEvent; outgoing: create suppression/update incidents- Chat (Slack/MS Teams): bi-directional integration; post ephemeral IC channel, accept slash-commands to update tasks/timeline- Ticketing (Jira/ServiceNow): create/update incident tickets, sync status, attach snapshot URIs- Storage: S3/MinIO for snapshots; storage connectors handle upload/download with signed URLs- Monitoring/Tracing: ingest context links (trace IDs, spans), ability to pull lightweight trace snippets or export to object store- Identity: OIDC/SAML for SSO; SCIM for user provisioning

Availability targets:- API: 99.99% (four nines) global; read-only page render for 99.999% via CDN caches and precomputed state- Real-time updates: 99.9% (depends on websocket reachability); fall back to long-polling- Snapshot retrieval: 99.9% SLA on object store; preserve integrity with cross-region replication- Integration workers: eventual consistency; connectors should guarantee at-least-once with idempotency

Strategies to keep operational during major incidents:1. Minimal dependency mode: ability to operate with only API + object store. Support offline-capable PWA so IC and scribe can continue editing timeline/tasks locally; a local queue syncs when connectivity returns.2. Reduce blast radius: separate critical-path services (Incident, RoleAssignment, Timeline) into small well-tested services with independent scaling and deployment cadence.3. Multi-region active-passive with warm standby data replication (async replication of objects, logical replication for Postgres). Promote standby on failover automated with canary verification.4. Degraded UX but preserved semantics: - If chat integration fails, show "chat link unavailable"; allow paste of chat messages into timeline. - If ticketing fails, queue ticket creation in DB with retry/backoff and expose DLQ to IC.5. Circuit breakers & graceful degradation: if external integrations are failing, the system continues locally and surfaces pending integrations.6. Immutable snapshots + signed manifests stored cross-region ensure evidence is preserved even if primary region is down.7. Operational runbooks baked into product (template runbooks per incident type) and a dedicated "panic mode" that reduces features to core collaboration and evidence capture.8. Automated failover & playbooks: health checks, automatic routing to read replicas for reads, automated leader election for write-critical services.9. Chaos & runbook testing: scheduled full-scale game days to validate offline workflows, failovers, and connector recovery.10. Security under outage: keep least-privilege tokens, rotate refresh tokens, use short-lived credentials and an emergency static admin key in HSM guarded by multi-party approval for severe outages.

Security & compliance:- RBAC + attribute-based access control (role scopes per incident)- MFA & device trust for IC role changes- Field-level encryption for sensitive fields (keys in KMS)- Tamper-evident audit ledger (hash chain) and WORM storage for snapshots- Data residency and retention policies configurable per organisation

Trade-offs:- Strong consistency vs availability: prefer availability for collaboration (AP for timeline reads with event ordering guarantees via event_seq) and provide transaction boundaries for critical writes (role assignment).- Multi-region replication increases complexity but required for platform survivability.

Operational considerations:- SLOs and error budgets for integrations; provide metrics to IC about integration health- Capacity planning: reserve on-call compute capacity and pre-warmed worker pools for major incidents- Observability: per-incident metrics, disruption blast radius map, and quick diagnostics page for SREs

This design ensures secure, auditable, real-time incident coordination while allowing the system to remain usable and preserve critical evidence during large-scale outages.

High Availability and Disaster RecoveryHardTechnical

69 practiced

You are seeing frequent noisy failovers due to flapping health checks. Propose a systematic debugging approach to identify the root cause and design robust health-check and failover logic that balances fast detection with minimizing false positives.

Sample Answer

Situation: We are observing frequent failovers because health checks flap. Objective: systematically find root cause and design robust health-check + failover logic that detects real failures quickly while minimizing false positives.

Debugging approach (systematic)1. Gather evidence- Correlate timestamps from load balancer, backend logs, orchestration events (k8s liveness/readiness), metrics (CPU, mem, latency), and network device logs.- Plot health-check success/failure, response latency, request rate, and resource metrics around flaps.2. Reproduce & isolate- Try to reproduce in staging with similar load and request patterns.- Disable non-essential changes (deployments, autoscaling) to see if flaps stop.3. Hypothesis-driven checks- Network: packet loss, MTU, firewall timeouts, intermediate proxies.- Resource saturation: CPU spikes, GC pauses, thread exhaustion, connection pool limits.- Application-level: slow endpoints, intermittent dependencies (DB, auth), transient errors.- Load balancer config: aggressive timeouts, health-check frequency, retry logic.4. Deep dive- Capture traces (distributed tracing) for failing health-check requests to see call graph and latency contributors.- Check GC/threads/dumps during failures.- Verify health-check endpoint implementation: is it doing expensive work or calling downstream services?5. Short-term mitigation- Temporarily increase health-check tolerance (consecutive failures) to stabilize and buy time for root-cause fix.

Design for robust health checks and failover (principles + concrete settings)1. Health-check design- Use lightweight local checks: prefer internal readiness endpoints that validate only local critical subsystems (e.g., app process, in-process caches) rather than full end-to-end checks that call databases.- Separate liveness vs readiness: liveness = process alive; readiness = able to serve traffic.- Make checks idempotent, cheap, and deterministic.2. Detection logic: balance speed vs false positives- Use N-of-M consecutive logic (e.g., fail after 3 failures out of 5). This tolerates transient blips.- Add hysteresis: require M consecutive successes to recover (e.g., 5 successes).- Exponential backoff for rechecks after failures to avoid thrash.- Optionally use sliding-window/weighted moving average on latency/error rate and trigger only when above threshold for window (e.g., error-rate > 5% over 60s).3. Use quorum and progressive eject- For clusters behind load balancers, eject unhealthy nodes progressively (circuit breaker) rather than full cutover.- For k8s, use graceful node draining with escalating termination: mark NotReady, drain, then terminate.4. Multi-signal decisioning- Combine health-check result with other signals before failover: high CPU, high GC pause metric, or degraded upstream dependency. Use a small decision engine to reduce false positives.5. Observability & testing- Emit metrics: health_check_status, latency, failures_consecutive. Alert on trends, not single flaps.- Chaos test health checks under load and network faults.6. Example configuration (illustrative)

yaml

# Load balancer health
interval: 5s
timeout: 2s
unhealthy_threshold: 3   # fail after 3 consecutive failures
healthy_threshold: 5     # require 5 consecutive successes to mark healthy
backoff: exponential     # increases interval after detected failure
# Readiness endpoint should be cheap and local-only
readiness_path: /health/ready
liveness_path: /health/live

Trade-offs & rationale- Short intervals and low thresholds detect real outages faster but increase false positives; consecutive logic + hysteresis mitigates that.- End-to-end checks catch dependency failures but can cause flaps when downstream is flaky — prefer layered checks (local readiness + periodic end-to-end synthetic tests).- Progressive ejection and multi-signal decisioning reduce impact of misclassifications.

Resulting expectations- Fewer noisy failovers, faster detection of real issues, clearer telemetry for root-cause, and safe failure modes that avoid cascade churn.

Reliability Patterns and Fault ToleranceEasyTechnical

58 practiced

What is a retry storm (thundering herd) and why does it amplify outages? Describe three practical mitigation strategies at different levels (client, service, infrastructure) you would implement to prevent retry storms in a high-traffic API.

Linux File Systems and PermissionsEasyTechnical

75 practiced

Explain the difference between hard links and symbolic (soft) links. Provide examples using ln and ln -s, explain what happens when the original file is deleted, and describe a situation where a hard link is not possible (and why).

Sample Answer

Brief answer: hard links are additional directory entries pointing to the same inode (same file data); symbolic (soft) links are special files that store a pathname to the target.

Examples:

# create file
echo "hello" > file.txt
# hard link
ln file.txt hard.txt
# symlink
ln -s file.txt soft.txt
# show inode and link count
ls -li file.txt hard.txt soft.txt

Key differences:- Inode/link count: hard.txt and file.txt share the same inode number and link count increases; soft.txt has its own inode pointing to the pathname.- Deleting original: - rm file.txt: with hard link, data remains accessible via hard.txt because link count > 0. The inode is removed only when last link is deleted. - rm file.txt: with symlink, soft.txt becomes a broken link (points to non-existent path).- Where hard links are not possible: - Across different filesystems/mounts (inodes are unique per filesystem) — ln fails with "Invalid cross-device link". - For directories (typical users cannot create hard links to directories because it would break filesystem tree and traversal; only root and special tools/conditions can create them).Why it matters for SREs: choose symlinks for pointing across mounts or to directories; use hard links when you need atomic alternate name to same data (e.g., safe file rotation) and want data to survive deletion of one name.

Database Selection and Trade OffsHardSystem Design

32 practiced

For a global key-value store expecting 1M writes/sec, design a sharding and re-sharding strategy to minimize downtime and avoid hotspots. Include consistent hashing with virtual nodes, rebalancing triggers, controlled data movement mechanisms, client routing updates, and monitoring to detect skew and imbalance.

Sample Answer

Requirements & goals:- 1M writes/sec globally, multi-region, <1% unavailability for writes, avoid single-node hotspots, support online resharding with minimal data movement.

High-level approach:- Use consistent hashing with virtual nodes (vnodes) + weighted nodes per capacity. Clients (or local region proxies) hash keys to ring positions; each vnode maps to a physical shard.

Consistent-hashing + vnodes:- Allocate ~1024–4096 vnodes per physical shard (adjust by node capacity). Weight vnodes proportionally for heterogeneous instance sizes.- Vnode granularity lets rebalancing move small pieces of keyspace, reducing migration blast radius.

Rebalancing triggers:- Continuous monitoring of per-vnode metrics: write QPS, IOPS, CPU, disk throughput, tail latencies.- Triggers: sustained imbalance (>20% above cluster average for >5min), node addition/removal, capacity autoscale events, SLO breach or manual operator request.- Hotspot detection: heatmap (key/token → QPS) and heavy-hitter detector (top-k keys). If a small set of keys cause hotspots, use application-level sharding (split hot key ranges) or introduce “hot key shards” (cache/write-through).

Controlled data movement:- Plan rebalancing as vnode moves (not whole-node). Use a three-step handoff per vnode: 1. Mark vnode draining: new writes replicated to both source and target (write-through), reads can be served from either. 2. Stream incremental replication (changefeed/WAL or log shipping) from source to target with backpressure & rate limits; track lag. 3. Cutover when target is caught up: atomically update ring config, stop dual writes, retire source ownership for that vnode.- Use parallelism limits and token-bucket rate limiting for migration bandwidth to protect client throughput.- For sudden hot keys, use dynamic split: map hot key to a “virtual shard group” by hashing key+salt or time-window to distribute writes.

Client routing & config updates:- Clients hold a versioned ring config from a central config service (etcd/Consul) or via local proxies. Config includes vnode→shard mapping and migration state.- Use eventual consistency: clients read config periodically and subscribe to updates; support two-phase config activation (prepare + activate) so in-flight requests use old mapping until safe.- For atomicity, coordinate with a lightweight coordinator (leader per region) to publish ring versions. Clients that see migration state consult routing logic: if vnode is draining, client sends writes to both nodes or to proxy which handles dual-write until cutover.- Provide short TTL DNS or gRPC name resolver + local caching for proxies to avoid per-request coordination.

Monitoring & automation:- Metrics: per-vnode/per-shard QPS, latency P50/P95/P99, CPU, disk, replication lag, migration progress, rebalancing throughput.- Alerts: sustained imbalance, migration lag > threshold, error rate increase during migration, cache-miss spikes.- Visuals: token heatmap, vnodes-per-node histogram, hot-key dashboards, migration pipeline view.- Automated responses: autoscale new nodes and automatically schedule vnode moves respecting SLOs; canary resharding first on small subset of vnodes then scale up.- Post-mortem simulation: offline resharding simulator to estimate data movement and expected tail latencies before changes.

Trade-offs & notes:- More vnodes → finer-grained moves but larger config size and more metadata. Choose vnode count to balance move granularity and management overhead.- Dual-write increases write amplification but ensures zero-downtime cutover; use limited duration and monitor replication lag.- For extreme hotspots of single keys, consistent hashing alone fails — require application-level key splitting or dedicated hot-key stores (in-memory cache, write coalescing).

This design minimizes downtime by moving small vnode units with streaming replication and controlled rate-limiting, avoids hotspots by detecting and splitting hot keys, and ensures clients smoothly follow ring changes via versioned config and proxies.

Infrastructure Automation and ProvisioningMediumTechnical

54 practiced

Write an Ansible playbook snippet that idempotently installs Nginx, deploys a templated configuration file, and restarts the nginx systemd service only when the configuration changes. Also explain how you'd validate this playbook in CI using linting and dry-run checks before applying to production.

Linux Troubleshooting and DiagnosticsMediumTechnical

31 practiced

Explain how you would trace and debug a page-fault storm on a Linux host. Which perf events, ftrace tracepoints, and /proc counters would you inspect to identify processes generating frequent major or minor faults, and how would you isolate the offending process or mapping?

Sample Answer

Situation: A host shows very high page-fault rates (page-fault storm) causing CPU spikes and latency. I want to quickly find which process or mapping causes frequent minor or major faults and why (e.g., memory-mapped file, stack growth, swapping, or bad user code).

Approach (steps I would take)1. Quick host-level checks- vmstat 1 5 — look at “faults” column to confirm high fault rate.- cat /proc/vmstat | egrep 'pgfault|pgmajfault|pgmajfault' — pgfault and pgmajfault give system-wide counts.- free -h and /proc/meminfo — check available memory, swap pressure (SwapTotal/SwapFree), and OOM risk.

2. Find suspicious processes (aggregates)- Use pidstat -r -p ALL 1 3 or ps -eo pid,comm,min_flt,maj_flt,rss,vsz --sort=-min_flt to list processes by minflt/majflt (minflt = minor faults; majflt = major faults). Note: ps's fields map to /proc/<pid>/stat fields.- Alternatively run a short script polling /proc/*/stat: read fields 10 (minflt) and 12 (majflt) and compute deltas over 1s to find the top offenders.

3. Use perf to capture detailed events- perf stat -a -e page-faults,minor-faults,major-faults sleep 5 — system totals.- perf record -e tracepoint:exceptions:page_fault_user -a -- sleep 10 then perf script — this shows per-fault records with PID/TID and instruction pointer. You can record kernel tracepoints for kernel faults too (exceptions:page_fault_kernel).- Or perf record -e page-faults/u -a -- sleep 10 to sample user-level faults (if perf supports that event on your kernel).

4. Use ftrace/tracepoints for high-resolution tracing- echo 1 > /sys/kernel/debug/tracing/events/exceptions/page_fault_user/enable (or use trace-cmd)- tail -f /sys/kernel/debug/tracing/trace_pipe to see live events: they include PID, address and IP. This is lightweight and shows which PID and faulting virtual address are hitting frequently.- Also enable exceptions/page_fault_kernel if kernel-mode faults are suspected.

5. Map fault addresses to mappings- From perf script or trace_pipe you get PID and faulting vaddr or IP. Inspect /proc/<pid>/maps or pmap <pid> to see which mapping contains that address (file-backed vs anon), and the file path if file-backed (e.g., a huge mmap of an mmapped file).- If it's a shared library or binary, use addr2line / eu-addr2line on the executable or /proc/<pid>/exe to map IP to source line or symbol.

6. Confirm major vs minor and root cause- Major faults: check /proc/vmstat pgmajfault increases — usually indicates swap-in from disk or page-in from file. Check iostat / iotop to see increased disk IO matching pgmajfault spikes.- Minor faults: often demand-zero pages or COW, or first-touch of huge anonymous pages. Check whether the mapping is anonymous (heap/stack) vs file-backed. For first-touch in containerized environments, check memory overcommit and hugepages policies.- If faults are due to page cache reading (file-backed) and cause IO, investigate application access pattern (random vs sequential) or use readahead tuning.

7. Isolate and mitigate- Once PID and mapping are identified: reproduce with strace -e trace=mmap,munmap,brk,read? (note strace won’t show page faults but can show mapping churn) or attach gdb to inspect threads.- If mapping is a mmapped file causing heavy page-in, consider prefetching (readahead / posix_fadvise(FADV_WILLNEED)) or changing access pattern.- If COW churn from fork/exec is causing faults, consider using vfork or reduce fork-heavy design.- If swap thrashing, add memory, reduce memory usage, tune swappiness, or move workload to host with more RAM; temporarily limit process with cgroups memory.limit_in_bytes to stop system-wide impact.

Key counters and locations- /proc/vmstat: pgfault, pgmajfault, pgscan_kswapd, pgsteal- /proc/<pid>/stat: minflt (field 10), majflt (field 12) — poll deltas- /proc/<pid>/status: voluntary/involuntary context switches; VmRSS, VmSize- perf events: page-faults, minor-faults, major-faults; tracepoints: exceptions:page_fault_user, exceptions:page_fault_kernel- ftrace path: /sys/kernel/debug/tracing/trace_pipe and events under /sys/kernel/debug/tracing/events/exceptions/

Example quick commands- vmstat 1- awk '/^pgfault|^pgmajfault/ {print}' /proc/vmstat- for p in $(ls /proc | grep '^[0-9]'); do awk -v pid=$p '{getline; }' /proc/$p/stat 2>/dev/null; done # (better to script reading fields minflt/majflt)- perf record -e tracepoint:exceptions:page_fault_user -a -- sleep 5; perf script | less- echo 1 > /sys/kernel/debug/tracing/events/exceptions/page_fault_user/enable; tail -f /sys/kernel/debug/tracing/trace_pipe

Why this approach- Start broad (host counters) → narrow to processes (proc counters) → collect high-resolution evidence (perf/ftrace) → map fault addresses to mappings (/proc/<pid>/maps) → decide mitigation. This combination lets an SRE identify the exact process and mapping causing the storm and choose the right fix (code change, tuning, cgroup limits, or capacity increase).

Incident Management and ResponseMediumTechnical

99 practiced

Describe an on-call escalation and handoff process that minimizes dropped alerts during shift changes and long-running incidents. Include timing for handoffs, required documentation, shadowing practices for new on-call engineers, and tooling (ticketing, chatops, dashboards) that supports safe transitions across timezones.

Sample Answer

Situation: In my current SRE org we had gaps during shift changes and long incidents—missed alerts and duplicated work—so I led a redesign of escalation and handoff process to eliminate dropped alerts.

Approach / Process:- Handoff timing: 15-minute overlap at shift boundaries (30 minutes for long-running incidents). Outgoing provides a 5-minute asynchronous summary in chat 15 minutes before end-of-shift; live 10-minute walkthrough during overlap covering active alerts, mitigations, next actions, and pending owners.- Escalation cadence: Alerts auto-page primary; if unacknowledged in 5 minutes escalate to secondary, 10 minutes to on-call lead, 20 minutes to engineering manager. Escalation policy documented per-service in playbooks.

Required documentation:- Single-source incident ticket (ticketing system link in chatops) with: incident summary, timeline, current hypothesis, runbooks executed, mitigations applied, next steps, stakeholders, and a “handoff checklist” with explicit accept/decline buttons.- Tagging convention: severity, owner, and expected next-check time.

Shadowing & onboarding:- New on-call engineers shadow primary for 2 weeks: 1 week passive (observe alerts, read postmortems), 1 week active with buddy escalation. Graduated responsibility: start with low-severity services, weekly review sessions.

Tooling:- Ticketing (Jira) as source of truth; ChatOps (Slack with incident channel created automatically) for live communication and acknowledgement buttons; runbook links and escalation controls integrated via bot; dashboards (Grafana) with service health and alert status widgets; incident timeline (Chronicle or internal tool) auto-updated from chat and ticket events.- Cross-timezone support: handoff windows staggered so each pair has overlap during acceptable hours where possible; async-first culture—every handoff leaves a concise written summary and “must-check” markers; alert quiet-hours policies and regional failovers to prevent pager fatigue.

Safety & metrics:- KPIs: dropped-alert rate (<1%), mean time to acknowledge (MTTA), number of duplicate responders. Enforce blameless postmortems and iterate playbooks quarterly.

Why it works: combines brief live context transfer with a rigorous written single-source truth, automated escalation, graduated shadowing, and tooling that ties chat, tickets, and dashboards together—minimizing human error across shifts and timezones.

High Availability and Disaster RecoveryMediumTechnical

66 practiced

Write a health-check script in Python that performs three checks: 1) HTTP GET to /health expecting a 200, 2) a quick ping/query to a local cache or redis endpoint, and 3) ensure disk utilization is under 80 percent. The script should exit 0 for healthy and non-zero for unhealthy. Describe how to differentiate liveness vs readiness probe usage.

Sample Answer

Approach: perform three independent checks, return non-zero immediately on failure (or aggregate failures), and print structured output. Use requests for HTTP, redis client for a lightweight PING, and shutil/os for disk usage so script has minimal deps. Exit 0 if all pass.

python

#!/usr/bin/env python3
import sys
import shutil
import requests
import redis
import os
from typing import Tuple

HTTP_URL = os.getenv("HEALTH_URL", "http://localhost:8080/health")
REDIS_HOST = os.getenv("REDIS_HOST", "localhost")
REDIS_PORT = int(os.getenv("REDIS_PORT", 6379))
MAX_DISK_PCT = float(os.getenv("MAX_DISK_PCT", 80.0))

def check_http(url: str, timeout=2) -> Tuple[bool,str]:
    try:
        r = requests.get(url, timeout=timeout)
        return (r.status_code == 200, f"http_status={r.status_code}")
    except Exception as e:
        return (False, f"http_error={e}")

def check_redis(host: str, port: int, timeout=1) -> Tuple[bool,str]:
    try:
        r = redis.Redis(host=host, port=port, socket_connect_timeout=timeout, socket_timeout=timeout)
        pong = r.ping()
        return (pong is True, "redis_pong")
    except Exception as e:
        return (False, f"redis_error={e}")

def check_disk(path="/", max_pct=80.0) -> Tuple[bool,str]:
    total, used, free = shutil.disk_usage(path)
    used_pct = used / total * 100
    return (used_pct < max_pct, f"disk_used_pct={used_pct:.1f}")

def main():
    checks = []
    ok, msg = check_http(HTTP_URL)
    checks.append(("http", ok, msg))
    ok, msg = check_redis(REDIS_HOST, REDIS_PORT)
    checks.append(("redis", ok, msg))
    ok, msg = check_disk("/", MAX_DISK_PCT)
    checks.append(("disk", ok, msg))

    healthy = all(c[1] for c in checks)
    for name, ok, msg in checks:
        print(f"{name}: {'OK' if ok else 'FAIL'} - {msg}")
    sys.exit(0 if healthy else 2)

if __name__ == "__main__":
    main()

Key points / complexity:- Each check is O(1) time; network latency dominates.- Keep timeouts small to avoid slow probes.- Handle exceptions and surface errors for debugging.

Liveness vs Readiness:- Liveness = "is the process alive?" Use lightweight checks (process responsive/loop stuck). If liveness fails, container/k8s should restart the pod.- Readiness = "can this instance serve traffic?" Use dependency checks (can reach redis, DB, external APIs, disk pressure). If readiness fails, remove from load balancer but don't restart.- Map checks: HTTP health endpoint and redis connectivity are readiness-critical; a simple self-alive indicator (e.g., event loop heartbeat) is sufficient for liveness. Disk saturation might be readiness (stop traffic) unless it prevents process from functioning, then consider liveness.

Practice Site Reliability Engineer (SRE) questions across all topics

Additional Information

Want to create your own tailored preparation guide using our deep research?

Get Started for Free

Interview-Ready Courses

Visual-first, interactive, structured learning paths

Browse Site Reliability Engineer (SRE) jobs

AI-enriched listings across hundreds of company career pages

Explore Jobs

Apple Site Reliability Engineer (Staff Level) Interview Preparation Guide

Interview Process Overview

Interview Rounds

Recruiter Screening

What to Expect

Tips & Advice

Focus Topics

Communication and Professionalism

Practice Interview

Study Questions

Motivation for Apple and Role Understanding

Practice Interview

Study Questions

Professional Background and Experience Narrative

Practice Interview

Study Questions

Technical Phone Screen - Hiring Manager

What to Expect

Tips & Advice

Focus Topics

Cross-Functional Collaboration and Influence

Practice Interview

Study Questions

Automation and Infrastructure Code Development

Practice Interview

Study Questions

Independent Problem-Solving and Decision-Making

Practice Interview

Study Questions

Major Incident Management and Resolution

Practice Interview

Study Questions

Systems Design and Architectural Thinking

Practice Interview

Study Questions

Systems Internals and Linux Troubleshooting

What to Expect

Tips & Advice

Focus Topics

Inter-Process Communication and Networking Stack Basics

Practice Interview

Study Questions

I/O Subsystem and Performance Analysis

Practice Interview

Study Questions

Linux Kernel Concepts and File System Architecture

Practice Interview

Study Questions

Memory Management and Virtual Address Space

Practice Interview

Study Questions

Advanced Diagnostic Tools and Troubleshooting Methodology

Practice Interview

Study Questions

Process Management and System Calls

Practice Interview

Study Questions

SRE Fundamentals and Distributed Networking

What to Expect

Tips & Advice

Focus Topics

DNS Protocol and Name Resolution

Practice Interview

Study Questions

Load Balancing Strategies and Algorithms

Practice Interview

Study Questions

TLS/SSL and Secure Communication

Practice Interview

Study Questions

Distributed Systems Reliability Patterns

Practice Interview

Study Questions

HTTP Protocol and Web Request Handling

Practice Interview

Study Questions

TCP Protocol and Connection Management

Practice Interview

Study Questions

Advanced Coding and Data Structures