Apple Site Reliability Engineer (Senior Level) - Comprehensive Interview Preparation Guide
Apple's Site Reliability Engineer interview process for Senior-level candidates is comprehensive and spans approximately 6 months from initial application to offer. The process includes a recruiter screening phase followed by a virtual on-site with multiple technical rounds focused on systems internals, networking fundamentals, coding/algorithms, system design, and behavioral assessment. Each round includes behavioral evaluation components. The interview emphasizes depth of knowledge in distributed systems, Linux fundamentals, observability, and system design with particular focus on load balancing and reliability at scale.
Interview Rounds
Recruiter Screening
What to Expect
Initial recruiter call (typically 30 minutes) followed by a confirmation call the next day to schedule the virtual on-site. The recruiter will discuss your background, confirm interest in the SRE role, clarify expectations, and explain the interview process. This round establishes baseline communication fit and verifies that your experience aligns with Apple's senior-level SRE expectations. The second recruiter contact confirms logistics for the virtual on-site rounds scheduled for late April or a similar timeframe.
Tips & Advice
Be enthusiastic about SRE and reliability engineering specifically. Prepare a clear 2-3 minute summary of your SRE background, highlighting production system reliability improvements, incident leadership, and cross-functional collaboration. Ask clarifying questions about the team, scale of systems, and current reliability challenges. Confirm you understand the interview structure and technical requirements ahead of time.
Focus Topics
Communication and Culture Fit
Ability to communicate clearly, ask intelligent questions, and demonstrate alignment with Apple's values around quality, reliability, and user experience. Show enthusiasm for building reliable systems at scale.
Practice Interview
Study Questions
SRE Background and Experience Summary
Ability to concisely articulate your SRE career trajectory, key projects that improved reliability, and why you're interested in joining Apple's SRE organization. For Senior level, emphasize projects where you led reliability improvements, mentored engineers, or influenced architecture decisions.
Practice Interview
Study Questions
Systems Internals Deep Dive
What to Expect
Technical round (60 minutes) focused on deep Linux knowledge and system troubleshooting. Expect a realistic Linux troubleshooting scenario (e.g., SSH not working with console access). The interviewer will guide you through diagnosis using Linux tools and will probe your understanding of /proc filesystem, memory management, process management, and how shell commands are interpreted. This round assesses your foundational expertise in systems administration, ability to think through problems systematically, and depth of Linux knowledge required for production reliability work.
Tips & Advice
Before the interview, review the Linux boot process, process management, memory management (heap vs. stack, page tables), the /proc filesystem structure, and common Linux troubleshooting tools. Practice debugging a real scenario where you can't SSH into a machine - think through what you'd check first, how you'd gather information from /proc, how you'd interpret system calls with strace. Be comfortable discussing how the shell interprets commands, environment variables, and file descriptors. For Senior level, explain not just how to fix the problem but how you'd prevent it and monitor for it in production. Ask clarifying questions about the environment when given a scenario.
Focus Topics
Shell Interpretation and Command Execution
Understanding of how shells parse and execute commands, including quoting, expansions (glob, variable, command substitution), piping, redirection, and background processes. Know how environment variables are inherited, how file descriptors work, and how subshells behave.
Practice Interview
Study Questions
System Call Interface and Kernel-User Space Interaction
Understanding of what system calls are, how applications interact with the kernel, and how to trace system calls with strace. Know common system calls related to process management, file I/O, and networking. Understand the difference between user space and kernel space.
Practice Interview
Study Questions
Linux Memory Management and Virtual Memory
Understanding of physical vs. virtual memory, paging, swapping, memory mapping, and the page cache. Know how to interpret memory usage from /proc, understand OOM killer behavior, and diagnose memory-related performance issues. Understand memory isolation and how memory is allocated at the kernel level.
Practice Interview
Study Questions
Linux Process Management and /proc Filesystem
Deep understanding of how processes work in Linux, including process states, memory layouts, file descriptors, and how to inspect processes via /proc. Know how to read /proc/[pid]/status, /proc/[pid]/maps, /proc/meminfo, and interpret this information to diagnose issues. Understand process scheduling, context switching, and CPU affinity.
Practice Interview
Study Questions
Linux Troubleshooting Methodology and Tools
Systematic approach to Linux troubleshooting using strace, lsof, /proc inspection, dmesg, and other tools. Ability to narrow down where a problem exists (kernel, application, network, permissions, etc.) and use appropriate tools to investigate. Understanding of file descriptor management, socket states, and connection issues.
Practice Interview
Study Questions
SRE/Networking Deep Dive
What to Expect
Technical round (60 minutes) focused on networking protocols and distributed systems. Expect deep questions about TCP, TLS, HTTP, and DNS. You may be asked to walk through the complete request flow to a service like icloud.com, explaining each layer. The interviewer will probe your understanding of networking concepts, protocol interactions, and how these impact reliability and observability. For Senior level, expect questions about how networking issues manifest in production, how to monitor networking health, and how to design for network reliability.
Tips & Advice
Study the OSI model with deep focus on layers 3-7. Understand TCP in detail: connection states (SYN, SYN-ACK, ACK, TIME-WAIT), window size, retransmission, congestion control. Understand DNS - query flow, caching, TTL implications, A/AAAA records. Understand TLS - handshake, certificate validation, cipher suites. Understand HTTP - status codes, headers, connection management, keep-alive. Practice walking through a complete request: DNS lookup (with caching), TCP connection establishment, TLS handshake, HTTP request/response. For Senior level, discuss how each layer can fail, what metrics to monitor, and how to design systems resilient to networking issues. Be able to explain network troubleshooting tools like tcpdump, netstat, dig, curl and how you'd use them to diagnose issues. Think about load balancing implications of your networking knowledge.
Focus Topics
HTTP Protocol and Web Communication
Deep understanding of HTTP methods, status codes, headers, and connection management (HTTP/1.0 vs HTTP/1.1 keep-alive vs HTTP/2 vs HTTP/3). Understand caching headers, compression, and how these impact performance and reliability.
Practice Interview
Study Questions
Network Troubleshooting and Observability
Practical use of networking tools: tcpdump, netstat, ss, dig, nslookup, curl, wget. Understanding of metrics to monitor: packet loss, latency, connection establishment time, DNS resolution time, TLS handshake time. Knowing how to set up alerts and dashboards for network health.
Practice Interview
Study Questions
DNS Resolution and Caching
Understanding of DNS query flow, record types (A, AAAA, CNAME, MX, etc.), caching at multiple levels (resolver cache, OS cache, application-level caching), TTL implications, and DNS-related failure modes. Understand how DNS problems can cascade into application failures.
Practice Interview
Study Questions
Network Request Flow and Distributed System Communication
Ability to trace a request through all layers: DNS resolution (with caching), TCP connection establishment, TLS handshake, HTTP request, processing, and response. Understanding of how failures at each layer manifest and what signals indicate problems. For Apple services, understanding iCloud request flow or similar.
Practice Interview
Study Questions
TLS/SSL Protocol and HTTPS
Understanding of TLS handshake, certificate validation, cipher suites, and how TLS impacts latency and connection setup time. Understand certificate pinning, certificate revocation, and common TLS-related issues in production. Know how TLS 1.2 and 1.3 differ.
Practice Interview
Study Questions
TCP Protocol and Connection Management
Deep understanding of TCP including the three-way handshake, connection states (LISTEN, SYN_SENT, SYN_RECEIVED, ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, CLOSE_WAIT, TIME_WAIT, CLOSED), window size management, retransmission logic, and congestion control (slow start, congestion avoidance). Understand TIME_WAIT implications for connection reuse and ephemeral port exhaustion.
Practice Interview
Study Questions
Coding/Algorithms Assessment
What to Expect
Coding round (45-60 minutes) where you'll solve 1-2 LeetCode-style problems at Easy to Medium difficulty, typically involving data structures like graphs (BFS/DFS traversal). You'll write code in your language of choice and explain your approach. The interviewer is assessing algorithmic thinking, code quality, ability to handle edge cases, and communication while coding. For Senior level, interviewers expect clean, well-structured code and thoughtful discussion of trade-offs.
Tips & Advice
Practice LeetCode Medium problems, particularly those involving graphs and tree traversal (BFS, DFS). Be comfortable coding in your preferred language - don't attempt to code in a language you're not fluent in. Write clean, readable code with meaningful variable names. Walk through your approach before coding - ask clarifying questions about constraints (input size, etc.). Discuss time and space complexity. Handle edge cases explicitly. For Senior level, think about optimization opportunities and discuss trade-offs. Test your code mentally with sample inputs. If stuck, communicate your thinking clearly and consider simpler approaches first.
Focus Topics
Code Quality and Communication
Writing clean, readable, well-structured code with meaningful variable names and comments where necessary. Walking through your approach clearly before coding. Explaining your logic and decisions as you code. Discussing edge cases and handling them explicitly.
Practice Interview
Study Questions
Algorithm Complexity Analysis
Ability to analyze and articulate the time and space complexity of algorithms using Big O notation. Understand trade-offs between time and space. Be able to optimize algorithms and explain the improvements.
Practice Interview
Study Questions
Graph Algorithms (BFS and DFS)
Deep understanding of breadth-first search and depth-first search algorithms. Know how to implement both iteratively and recursively. Understand use cases for each approach and be able to solve problems involving graph traversal, connected components, shortest path, and tree traversal.
Practice Interview
Study Questions
Data Structures Fundamentals
Solid understanding of fundamental data structures: arrays, linked lists, stacks, queues, hash tables, trees, and heaps. Know the time/space complexity of operations and when to use each. Be comfortable implementing basic versions of these.
Practice Interview
Study Questions
System Design Round
What to Expect
System design round (60-75 minutes) where you'll design a large-scale distributed system. You may be asked to design something like a GitHub clone or similar service with focus on specific aspects like load balancing, observability, and reliability. You'll discuss architecture, components, data flows, and trade-offs. The interviewer will probe your thinking and likely ask follow-up questions about handling specific challenges. For Senior level, demonstrate deep understanding of distributed systems, ability to think through failure modes, and design for observability from the ground up.
Tips & Advice
Start by clarifying requirements and constraints - ask about scale (users, QPS, data volume), geography, consistency requirements, and what matters most (availability vs. consistency). Propose a high-level architecture with main components. For each component, discuss how it scales and where failures can occur. Design for observability from the start - what metrics, logs, and traces will you collect? Discuss load balancing strategy across components. Think about database choice and trade-offs. Discuss caching strategies. Address reliability: how do you handle component failures, how do you do deployments without downtime, what's your SLO? For Apple's focus on observability, emphasize how you'd monitor this system to understand its health and behavior. Be prepared to dive deep into one area based on interviewer's questions. Show your thinking process, don't just present a solution.
Focus Topics
Database Design and Trade-offs
Understanding relational vs. NoSQL databases and when to use each. Thinking through consistency models (strong, eventual), replication strategies, sharding, and backup/recovery. Discussing performance implications and trade-offs.
Practice Interview
Study Questions
Deployment, Rollback, and Change Management
How you'd deploy changes safely: blue-green deployments, canary deployments, staged rollouts. How you'd roll back if something goes wrong. Minimizing blast radius of changes. Coordinating changes across multiple services.
Practice Interview
Study Questions
Observability and Monitoring Design
Designing systems to be observable from the start: what metrics would you collect (latency, error rate, throughput, resource utilization)? What logs would you generate? How would you instrument requests to trace them across services? Designing alerts that indicate real problems. Understanding of SLIs, SLOs, and error budgets.
Practice Interview
Study Questions
Distributed System Architecture Design
Ability to design scalable architectures with multiple components: load balancers, API servers, databases, caches, message queues, etc. Understanding of service-oriented architecture, microservices, and when to split systems. Thinking through communication patterns between services and consistency implications.
Practice Interview
Study Questions
Handling Failure Modes and Resilience
Thinking through what can fail (server crashes, network partitions, storage failures, etc.) and how you'd handle each. Designing for graceful degradation, failover, redundancy. Understanding CAP theorem and consistency implications. Designing recovery procedures.
Practice Interview
Study Questions
Load Balancing Strategies and Techniques
Understanding of load balancing approaches (round-robin, least connections, consistent hashing, etc.) and when to use each. Understanding of load balancing at different layers (L4 vs L7). Designing systems that distribute load effectively and handle load balancer failures. Understanding sticky sessions and their implications.
Practice Interview
Study Questions
Behavioral and Leadership Interview
What to Expect
Interview round (45-60 minutes) focused on behavioral assessment, leadership, and cultural fit. Expect questions about past experiences handling incidents, making trade-offs, collaborating with teams, and influencing decisions. The interviewer (often a manager or senior engineer) will probe your approach to problem-solving, how you handle pressure, your communication style, and how you work with others. For Senior level, expect deeper questions about mentoring, project leadership, and how you balance competing priorities. This round also includes your opportunity to ask questions about the team, role, and Apple's SRE culture.
Tips & Advice
Prepare specific stories using the STAR method (Situation, Task, Action, Result) for: a major incident you handled, a reliability problem you solved, a time you collaborated effectively across teams, a time you had to make a trade-off between speed and reliability, a time you mentored someone, and a time you learned from a mistake. For Senior level, emphasize your leadership approach, how you influence teams, and how you think about technical strategy. Have concrete metrics or outcomes for your stories. Prepare thoughtful questions about Apple's SRE practices, the team's current reliability challenges, and how the role contributes to the organization. Research Apple's focus on reliability and user experience, and connect your approach to those values.
Focus Topics
Reliability Engineering Philosophy and Strategy
Your perspective on what makes systems reliable, how to approach reliability holistically, and your vision for SRE practices. For Senior level, discuss how you've influenced reliability culture in previous roles and your strategic thinking about reliability.
Practice Interview
Study Questions
Problem-Solving Approach and Learning from Failures
Describing your systematic approach to solving complex problems: how you break down unknowns, how you gather information, how you test hypotheses. Show examples of difficult problems you've solved. Discuss times you've failed and what you learned.
Practice Interview
Study Questions
Technical Mentoring and Leadership
For Senior level, describe your approach to mentoring junior engineers: how you help them grow, how you delegate, how you ensure they have learning opportunities. Show examples of engineers you've mentored and their growth. Discuss how you approach leading projects and influencing team decisions.
Practice Interview
Study Questions
Reliability Trade-offs and Decision-Making
Ability to discuss situations where you balanced competing priorities: speed to market vs. reliability, cost vs. redundancy, automation effort vs. manual work, etc. Show systematic thinking about trade-offs and willingness to make pragmatic decisions based on context.
Practice Interview
Study Questions
Cross-functional Collaboration and Communication
Examples of working effectively with development teams, product managers, and other disciplines. Ability to communicate complex technical issues to non-technical audiences. Demonstrating that you can influence decisions and drive change across teams.
Practice Interview
Study Questions
Incident Response and Post-Incident Learning
Ability to describe your approach to incident response: how you identify the problem, coordinate resolution, communicate with stakeholders, and conduct blameless post-mortems. For Senior level, discuss how you've led incident response, mentored junior engineers through incidents, and used incidents as learning opportunities. Show understanding that incidents are learning opportunities and shouldn't result in blame.
Practice Interview
Study Questions
Frequently Asked Site Reliability Engineer (SRE) Interview Questions
Design a per-user rate limiter that enforces at most R requests per rolling window of T seconds, at high request volume and for millions of distinct users. Compare at least two structural approaches (for example a fixed counter per window, a rolling log of timestamps, or a token-refill scheme) on memory per user and on how precisely each one enforces the limit at window boundaries.
Sample Answer
Direct answer
Enforcing "at most R requests per rolling T-second window" per user, at millions-of-users scale, comes down to picking how much state you keep per user and how precisely that state approximates a true rolling window. A fixed counter per window is cheapest (O(1) per user) but allows up to 2R requests to slip through right across a window boundary; a rolling log of exact timestamps is perfectly precise but costs O(R) per user; a token-refill (token-bucket) scheme and a two-counter sliding-window approximation both give O(1) per-user memory with only small, bounded imprecision near boundaries, which is why they are the usual production choice at this scale.
Structured elaboration
Three structural approaches compared
| Approach | Memory per user | Boundary precision | Notes |
|---|---|---|---|
| Fixed counter per window | O(1) (one count, one window-start timestamp) | Poor: a burst of R requests at the end of one window plus R more at the start of the next lets 2R through in a short span | Simplest to implement and reason about |
| Rolling log of timestamps | O(R) (one timestamp per allowed request in the window) | Exact: always enforces exactly R in any true rolling T-second window | Memory scales with the limit itself, not just with user count |
| Token-refill (token bucket) | O(1) (token count plus last-refill timestamp) | Good, but shapes bursts differently: it smooths sustained rate rather than exactly bounding a rolling count | Naturally supports controlled bursting up to bucket capacity |
| Two-counter sliding window | O(1) (previous window count, current window count, window start) | Good approximation: weights the previous window's count by how much of it still overlaps the current rolling window | No timestamp list, just two integers and one clock read |
Why a fixed counter's imprecision happens specifically at boundaries
If the window resets every T seconds, a user can send R requests in the last instant of one window and another R in the first instant of the next: both windows individually respect the R-per-window limit, but a true rolling T-second view sees up to 2R requests in a span far shorter than T. The rolling log fixes this by definition (it only ever counts requests actually within the trailing T seconds), at the cost of storing up to R timestamps per user. The two-counter and token-bucket schemes recover most of the precision of the rolling log at the memory cost of the fixed counter, by using the previous window's count as a fading estimate of "how many of those requests are still within the trailing T seconds," rather than discarding it entirely at the reset boundary.
Sharding for millions of users
Regardless of which per-user scheme is chosen, per-user state should be sharded by a hash of the user ID across many limiter nodes or partitions, so no single node holds all users and no single lock serializes all traffic. Route each user consistently to the same shard (consistent hashing keeps this stable as shards are added or removed) so all requests for one user hit the same counter state, and evict counters for inactive users on a time-to-live (TTL, an expiration timer after which an idle entry is dropped) so memory tracks active users rather than the full lifetime user base.
Worked example
The two-counter sliding-window approximation, concretely:
class SlidingWindowCounter:
"""
Approximate sliding-window limiter: O(1) memory per user (two counters),
O(1) time per check. Weights the previous fixed window by how much of it
still overlaps the current rolling window.
"""
def __init__(self, limit: int, window_seconds: float):
self.limit = limit
self.window = window_seconds
self.curr_window_start = 0.0
self.curr_count = 0
self.prev_count = 0
def _roll_window(self, now: float) -> None:
elapsed = now - self.curr_window_start
if elapsed >= 2 * self.window:
self.prev_count = 0
self.curr_count = 0
self.curr_window_start = now
elif elapsed >= self.window:
self.prev_count = self.curr_count
self.curr_count = 0
self.curr_window_start += self.window
def allow(self, now: float) -> bool:
self._roll_window(now)
elapsed_in_curr = now - self.curr_window_start
overlap = max(0.0, (self.window - elapsed_in_curr) / self.window)
estimated = self.prev_count * overlap + self.curr_count
if estimated + 1 > self.limit:
return False
self.curr_count += 1
return True
if __name__ == "__main__":
limiter = SlidingWindowCounter(limit=5, window_seconds=1.0)
# 5 requests at t=0.0 fill the first window
results_first = [limiter.allow(0.0) for _ in range(5)]
# a 6th request in the same window must be rejected
sixth = limiter.allow(0.05)
# at t=1.5 we are 50% into the new window; the estimate blends 50% of the
# old window's 5 requests (2.5) with 0 new ones, so 2.5 + 1 <= 5 fits
seventh = limiter.allow(1.5)
print(results_first, sixth, seventh)
Running this prints:
[True, True, True, True, True] False True
Five requests at t=0.0 fill the first one-second window exactly to the limit of 5. A sixth request at t=0.05 (still inside that same window) is rejected, since the count is already at 5. At t=1.5, half a second into the next window, the estimate blends 50% of the previous window's 5 requests (5×0.5=2.5) with the 0 requests so far in the current window: 2.5+1≤5, so the seventh request is allowed. This is the boundary smoothing a fixed counter does not give you: a fixed counter would have simply reset to 0 at t=1.0 and allowed 5 fresh requests immediately, permitting the same 2x-at-the-boundary burst described above.
Complexity
Per-user check and update: O(1) time for the fixed counter, token bucket, and two-counter sliding window; O(logR) or O(1) amortized (averaged over a sequence of operations) for the rolling log depending on whether old timestamps are pruned lazily or with a deque. Per-user memory: O(1) for the first three approaches, O(R) for the rolling log.
Edge cases
- A burst exactly at a window boundary is the scenario every design above is explicitly trying to bound; state which imprecision (if any) your chosen scheme accepts.
- Clock skew between distributed limiter nodes can make the "current time" disagree slightly across shards; keep window arithmetic tolerant of small skew rather than assuming a perfectly synchronized clock.
- A user with no prior activity needs a cold-start default (empty counters, full token bucket) rather than an error.
- Inactive users must be evicted (TTL-based) so memory does not grow without bound across millions of distinct users who each showed up once.
Trade-offs & pitfalls
The common wrong turn is presenting the rolling log as strictly "the correct one" without acknowledging its O(R)-per-user memory cost: at millions of users and even a modest R, that can dwarf the memory of the O(1) approaches by orders of magnitude, which is exactly why production rate limiters favor the token-bucket or sliding-window-counter approximation instead. A second common gap is proposing a single global lock or single-node counter for correctness: that eliminates any cross-shard race but reintroduces the exact contention problem millions of distinct users at high volume were meant to avoid; sharding by user ID sidesteps this because a fully correct answer only needs to be correct per user, not globally serialized. A third pitfall is conflating the token bucket's smoothing behavior with the sliding window's counting behavior: a token bucket happily allows a burst up to its full capacity the instant it has accumulated enough tokens, which is a different guarantee from "at most R in any rolling T-second window," and the two should not be presented as interchangeable without naming that difference.
For an e-wallet service where monetary transactions must be correct, choose an appropriate consistency model and database architecture. Compare options: distributed consensus with Paxos/Raft for multi-region strong consistency, single-region serializable RDBMS with sync replicas, and eventual consistency with compensating transactions. State your recommendation and justify the operational tradeoffs.
Sample Answer
Direct answer
Eventual consistency with compensating transactions is not viable as the primary mechanism for an e-wallet's ledger. A compensating transaction undoes a wrong debit or credit after the fact, and for money that is a regulatory and trust failure, not merely a rough edge in the user experience. That leaves a real choice between a single-region serializable relational database with synchronous replicas, and a multi-region consensus database using a protocol such as Raft or Paxos (algorithms that get a group of replicas to agree on the order of operations even if some are slow or unreachable). The deciding question is the recovery point objective (RPO, the maximum amount of recently committed data the business can tolerate losing) if an entire region is lost. If losing zero committed transactions even when a whole region disappears is a hard requirement, choose multi-region consensus despite its latency cost. If a well-rehearsed failover to a synchronous standby in a nearby region is an acceptable answer to that same question, single-region-plus-synchronous-standby is simpler to operate correctly and faster on the common path.
Structured elaboration
Single-region serializable relational database with synchronous replicas. Run the ledger under the strictest available isolation level (serializable, which prevents subtle bugs such as two concurrent withdrawals both reading a balance before either has committed and both approving), with one or more synchronous standbys nearby that must acknowledge a write before it is considered committed. A primary failure that fails over successfully loses zero committed data. The remaining exposure is a disaster that takes out the primary and all synchronous standbys together, at which point failover falls back to an asynchronous, farther-away replica, and the last few in-flight transactions can be lost. That gap is the RPO this option accepts.
Multi-region consensus (Raft or Paxos). Every write replicates to a quorum (an agreeing majority of the replica set, enough that any two possible quorums must share at least one replica) of replicas spread across regions before it is acknowledged, so losing any single region, including the one currently holding the leader, cannot lose committed data. RPO is zero by construction. The cost is that every write pays the network round trip needed to reach that quorum, bounded by the slower of the required acknowledgments, not avoidable simply by adding more replicas within the same regions, and the team now operates a genuinely different topology: region-aware placement decisions and a different mental model for reasoning about failure.
Eventual consistency with compensating transactions. Appropriate for the parts of a wallet product that are not the ledger itself, for example refreshing a cached risk score or propagating a promotional balance display, but not for the record of who owns what money. Even a well-designed saga (a multi-step transfer broken into a sequence of local transactions, each with its own compensating "undo" action, instead of one all-or-nothing global transaction) accepts a window where the system is visibly in an in-progress, not-yet-reconciled state, and for a ledger that window is exactly where double-spend and overdraft bugs live. Use this pattern only to orchestrate a multi-step transfer between two internally consistent ledgers, never as the consistency model of a ledger by itself.
Worked example
Absent any further constraint, default to single-region serializable with synchronous standbys: it is simpler to operate correctly, and a correctly operated simpler system beats a theoretically stronger but easily misconfigured one. Now quantify the alternative's cost. Light in optical fiber travels at roughly 68% of the speed of light in vacuum, about 203,900 km/s. For a route of roughly 5,900 km, a rough distance between the eastern United States and western Europe, one-way propagation alone is 5,900 / 203,900 * 1000 ~= 28.9 ms, so a round trip is ~57.9 ms before routing and serialization overhead, which typically adds further tens of milliseconds. If a regulator or a board-level continuity mandate genuinely requires that no committed transaction can be lost even if an entire region vanishes, paying roughly that extra 60 or so milliseconds of write latency, turning a 15-millisecond transfer into something closer to 75 milliseconds (15 ms + ~60 ms), is the acceptable price, and multi-region consensus is the correct call.
Trade-offs and pitfalls
A pitfall is choosing multi-region consensus reflexively because "more resilient" sounds safer, without pricing the latency into the product. A wallet transfer that now takes ten times longer is a real user-facing regression, and if the team cannot name the specific business requirement that extra time buys, that is a sign the requirement was never actually stated, just assumed.
A second pitfall is treating "serializable" as a checkbox rather than a specific property to verify. Some database engines implement serializable isolation through optimistic conflict detection, meaning transactions can abort and must be retried under contention, which changes what the application's error handling has to do, not just a configuration setting.
What would flip the recommendation further: a wallet operating in a single legal jurisdiction with no cross-region regulatory driver, and a documented, signed-off tolerance for losing the last few seconds of transactions in the vanishingly rare event of a total regional disaster. That is precisely when single-region-plus-synchronous-standby is not just acceptable but the better choice, because it is the option the team can run correctly with less operational complexity.
Two downstream services disagree about whether a transaction succeeded (service A logs success, service B logs failure). As SRE, how do you compute a reliable SLI for end-to-end success rate? Propose reconciliation rules, authoritative sources, and instrumentation changes to reduce future inconsistencies.
Sample Answer
When two services disagree about a transaction's outcome, the safest default for computing a reliable SLI is to trust whichever service is the AUTHORITATIVE source of truth for that specific outcome, and treat any disagreement itself as a signal worth investigating, not just resolving and moving on.
Structured elaboration
Reconciliation rules: identify which service is genuinely authoritative for the specific claim in question (e.g. for a payment transaction, the payment processor's own confirmation is usually more authoritative than an internal service's inferred success/failure based on an HTTP response, since the internal service could have logged "success" based on receiving a request while the actual downstream processing failed asynchronously, or vice versa); where no single service is unambiguously authoritative, define an explicit, DOCUMENTED tiebreaker (e.g. "if A and B disagree and neither is authoritative for this specific field, default to the more conservative outcome, i.e. treat as failed, since falsely counting a transaction as successful when it wasn't is worse than the reverse for this use case"). Instrumentation changes to reduce future inconsistency: propagate a shared, canonical transaction-outcome field written by whichever service IS authoritative, and have all other services read (not independently re-derive) that field for their own logging and SLI computation, rather than each service independently inferring and logging its own opinion about the same event.
Worked example
For an e-commerce checkout where service A (the order service) logs success based on receiving an acknowledgment, and service B (the payment processor integration) logs failure because the payment actually failed asynchronously after A's acknowledgment was sent: B is authoritative here, since payment success/failure is squarely B's domain of truth, and A's SLI computation should be corrected to READ B's authoritative outcome rather than relying on its own earlier, now-known-to-be-wrong acknowledgment-based inference. Going forward, A's logging is changed to write "acknowledged" (not "success") for A's own internal event, and to derive its own "transaction succeeded" SLI directly from B's authoritative confirmation, closing the specific inconsistency this incident revealed.
Trade-offs and pitfalls
Defaulting to "trust whichever service reports first" or "trust whichever reports success" (an easy but wrong shortcut under time pressure) systematically biases the SLI toward looking better than reality, since a service that reports optimistically and early will systematically override a later, more accurate failure signal from a slower but more authoritative source. It's also worth treating every discovered A/B disagreement as an OPPORTUNITY to fix the underlying instrumentation gap that allowed two services to develop divergent views of the same event in the first place, rather than treating reconciliation as a one-off patch applied after the fact each time a new disagreement surfaces.
What does psychological safety mean in the context of mentoring someone, and what concretely do you do to build it early in a mentoring relationship?
Sample Answer
Direct answer
Psychological safety, in a mentoring relationship, is a mentee's confidence that they can ask a question, admit a mistake, or push back on something without it costing them standing or opportunity. It's built through small, consistent moments early on, and it's genuinely tested the first time the mentee takes a visible risk and sees how you respond.
Concrete early actions
- Name failure modes yourself first. Mentioning a mistake you made in a similar situation signals that admitting error is normal here, not a one-way expectation.
- Model uncertainty openly. Say "I don't know, let's find out" instead of bluffing, so not-knowing reads as acceptable.
- Treat early mistakes as expected, not exceptional. React to a mistake by focusing on the fix and what it reveals, not on assigning blame.
- Be consistent between casual moments and anything formal. If private conversations are open but a formal review contradicts them, trust breaks immediately.
- Give credit publicly, give hard feedback privately. This is the pattern most people are watching for even if they never say so.
- Agree explicitly that disagreement is welcome, and actually respond well the first time it happens.
Worked example
Early in a relationship, a mentee admitted they'd made a mistake that caused some rework. The response focused entirely on understanding what happened and fixing it, walking through the reasoning openly rather than assigning blame, and treating it as a useful, expected part of learning. In the sessions that followed, the mentee started surfacing problems earlier and asking more pointed questions, rather than waiting until something couldn't be hidden.
Trade-offs and pitfalls
A common mistake is treating psychological safety as a one-time opening statement ("feel free to ask me anything") rather than an ongoing pattern that has to survive contact with a real mistake. The mentee will judge safety retrospectively, based on what actually happened the first time they took a risk, not on what was said at the start. It's also worth not confusing psychological safety with lowered standards: it's about how failure is handled and discussed, not about removing accountability for the work.
Here's a simple architecture: a single load balancer, three identical application servers behind it, and one primary database instance handling all writes. Walk through it and identify the single points of failure. For each one, what would you do about it, and what does that cost you?
Sample Answer
This architecture has three single points of failure once you look past the app tier: the load balancer, and the primary database, are both singletons that the whole request path depends on; the app-server tier looks redundant on paper (three instances) but is only actually redundant if those three instances sit in different fault domains, so it's worth confirming rather than assuming.
Walking the diagram
flowchart LR
U[Users] --> LB[Load Balancer\nSINGLE instance]
LB --> A1[App Server 1]
LB --> A2[App Server 2]
LB --> A3[App Server 3]
A1 --> DB[(Primary DB\nSINGLE writer)]
A2 --> DB
A3 --> DB
Load balancer (single instance). Every request passes through it, so its failure is a total outage regardless of how healthy the three app servers are behind it. Mitigation: run an active-active pair of LB nodes behind a floating IP or DNS-based failover, or use a managed cloud load balancer where the provider owns that redundancy. Cost: a small amount of extra infrastructure and configuration; the bigger cost is usually operational (health-check tuning, avoiding split traffic during LB failover), not dollars.
App servers (three instances, conditionally redundant). If all three run in the same rack, same availability zone, or share an underlying host, they aren't actually independent, a single power or network event takes out all three at once. Mitigation: spread them across at least two, ideally three, availability zones and confirm the LB health-checks each independently and routes around a dead one automatically. Cost: cross-AZ data transfer costs and slightly higher latency on some requests; this is usually the cheapest SPOF to fix since it's mostly a placement decision, not new infrastructure.
Primary database (single writer, no replica). This is the highest-blast-radius SPOF: if it fails, every write path is down and, depending on the failure mode, recent unreplicated data can be at risk. Mitigation: add at least one synchronous or semi-synchronous replica in another AZ with automated failover (promote-on-failure), plus continuous backups for protection against logical corruption that replication alone wouldn't catch. Cost: this is the most expensive fix of the three, both in infrastructure (a standing replica) and in write latency if replication is synchronous.
Worked example: quantifying the SPOFs
Assume, for illustration, per-component annual availability of 99.95% for the load balancer, 99.9% for each app-server instance, and 99.9% for the database, all figures pinned as inputs for this calculation, not measured facts about any real vendor.
The three-server app tier, if truly independent across fault domains, only fails when all three fail simultaneously, so its unavailability multiplies:
1−Aapp tier=(1−0.999)3=(0.001)3=10−9That's an app-tier availability of essentially 99.9999999%, negligible. But the LB and DB are each in series with the whole request path (either one being down takes the whole system down), so their unavailabilities add through multiplication of the availabilities:
Aoverall=ALB×Aapp tier×ADB≈0.9995×(1−10−9)×0.999≈0.998500(99.850%)That's about 525,600×(1−0.998500)≈788.4 minutes of downtime per year, roughly the sum of the LB's own downtime (about 262.8 min/yr at 99.95%) and the DB's own downtime (about 525.6 min/yr at 99.9%), because the redundant app tier contributes essentially nothing to the failure budget while the two singletons dominate it completely. This is the concrete version of "fix the SPOFs first": no amount of extra app-server redundancy moves that 788-minute number until the LB and DB are addressed.
Trade-offs and pitfalls
The most common wrong turn is stopping at "add more app servers," which is the SPOF that's already effectively solved in this diagram and contributes the least to the real number above; teams do this because it's the cheapest, least disruptive change, not because it's the highest-leverage one. A second pitfall is fixing the database with synchronous cross-region replication by default: it does reduce RPO to near zero, but the added write latency (and reduced availability during a partition, since a strict quorum, requiring a majority of replicas to agree before a write is accepted, can block writes when too few replicas are reachable) is often the wrong trade for a service that would have been fine with an in-region synchronous replica and async cross-region for disaster recovery only. The same "look for the singleton" walk generalizes past this exact diagram: in a streaming ingestion pipeline the SPOF is usually a single partition leader or a schema registry with no standby; in an ML-serving stack it's a lone model server or a feature store with no fallback; and at a more abstract level, any shared control plane (service discovery, config store, secrets manager) or shared cache that every downstream service depends on is a SPOF even when nobody draws it on the diagram.
Write a deploy script that releases a new build into a web root with no downtime and a fast rollback: unpack into a timestamped directory, verify its integrity, switch traffic over atomically, and prune old releases. It should run under plain /bin/sh.
Sample Answer
Direct answer
Keep every build in its own directory under releases/, serve the web root through a current symlink (a symbolic link: a tiny filesystem entry that holds a path and stands in for it, so current can point at whichever release directory is live), and deploy by building a new symlink beside the old one and renaming it over current. A rename is one atomic step (all or nothing: no other process can ever observe it half done), so requests see the old release or the new one and never a half-copied tree, and rollback is the same rename pointed at the previous directory. The script below does this in plain POSIX sh: verify a checksum, unpack into a hidden staging directory, rename it to its final release name, switch, run a health check, roll back automatically if it fails, then prune to a retention count.
The layout
/srv/www/shop/
current -> releases/20260105T101500Z (what the web server's root points at)
releases/
20260105T091000Z/
20260105T101500Z/
.deploy.lock/ (exists only while a deploy runs)
Release names are UTC timestamps (date -u +%Y%m%dT%H%M%SZ), so a plain text sort is chronological order and no extra bookkeeping file is needed. The symlink target is relative (releases/ID), so the whole tree can be moved or mounted elsewhere.
The script
#!/bin/sh
# Usage: deploy.sh deploy ARTIFACT.tar.gz (expects ARTIFACT.tar.gz.sha256 beside it)
# deploy.sh rollback
set -eu
WEBROOT=${WEBROOT:?set WEBROOT, e.g. /srv/www/shop}
RELEASES=$WEBROOT/releases
CURRENT=$WEBROOT/current
KEEP=${KEEP:-5}
HEALTH_CMD=${HEALTH_CMD:-} # runs with the live symlink as its working directory
RELEASE_ID=${RELEASE_ID:-$(date -u +%Y%m%dT%H%M%SZ)}
staging=
log() { printf '%s\n' "$*"; }
die() { printf 'deploy: %s\n' "$*" >&2; exit 1; }
case $KEEP in *[!0-9]*|0|1) die "KEEP must be an integer >= 2 (one live + one rollback target)";; esac
current_id() { basename -- "$(readlink "$CURRENT" 2>/dev/null || true)"; }
switch_to() { # build the new link beside the old one, then rename it over the old one
ln -s "releases/$1" "$WEBROOT/.current.$$"
mv -fT "$WEBROOT/.current.$$" "$CURRENT" # rename(2): readers see old or new, never neither
}
health_ok() {
[ -z "$HEALTH_CMD" ] || (cd "$CURRENT" && sh -c "$HEALTH_CMD")
}
prune() {
cur=$(current_id)
ls -1 "$RELEASES" | sort -r | awk -v keep="$KEEP" 'NR > keep' |
while IFS= read -r old; do
if [ "$old" != "$cur" ]; then rm -rf -- "${RELEASES:?}/$old"; log "pruned $old"; fi
done
}
deploy() {
artifact=${1:?usage: deploy.sh deploy ARTIFACT.tar.gz}
[ -f "$artifact" ] || die "no such artifact: $artifact"
[ ! -e "$RELEASES/$RELEASE_ID" ] || die "release $RELEASE_ID already exists"
( cd "$(dirname -- "$artifact")" && sha256sum -c "$(basename -- "$artifact").sha256" ) \
|| die "checksum mismatch, nothing was unpacked"
if tar -tzf "$artifact" | grep -Eq '(^/|(^|/)\.\.(/|$))'; then
die "archive contains absolute or ../ paths"
fi
staging=$RELEASES/.incoming-$RELEASE_ID
mkdir -p "$staging"
tar -xzf "$artifact" -C "$staging"
mv "$staging" "$RELEASES/$RELEASE_ID" # a release name only ever appears fully unpacked
staging=
prev=$(current_id)
switch_to "$RELEASE_ID"
if ! health_ok; then
if [ -n "$prev" ]; then
switch_to "$prev"
rm -rf -- "${RELEASES:?}/$RELEASE_ID" # a build that failed its check must never become a rollback target
die "health check failed, rolled back to $prev"
fi
die "health check failed on the first release (nothing to roll back to)"
fi
log "live: $RELEASE_ID (was: ${prev:-none})"
prune
}
rollback() {
cur=$(current_id)
[ -n "$cur" ] || die "nothing is live"
target=$(ls -1 "$RELEASES" | sort | awk -v cur="$cur" '$0 < cur { p = $0 } END { print p }')
[ -n "$target" ] || die "no release older than $cur"
switch_to "$target"
log "live: $target (rolled back from $cur)"
}
mkdir -p "$RELEASES"
mkdir "$WEBROOT/.deploy.lock" 2>/dev/null || die "another deploy or rollback is running"
trap 'rmdir "$WEBROOT/.deploy.lock"; if [ -n "$staging" ]; then rm -rf "$staging"; fi' EXIT
trap 'exit 1' HUP INT TERM # a signal becomes a normal exit, so the EXIT trap above still runs
case ${1:-} in
deploy) shift; deploy "$@" ;;
rollback) rollback ;;
*) die "usage: deploy.sh deploy ARTIFACT.tar.gz | rollback" ;;
esac
Reading the less obvious lines
set -eu:-emakes the script exit as soon as a command fails, and-uturns a use of an unset variable into an error instead of silently treating it as empty.${WEBROOT:?set WEBROOT, e.g. /srv/www/shop}: expands to the variable's value, but if the variable is unset or empty the shell prints the message and exits. The same form guards the delete inprune:rm -rf -- "${RELEASES:?}/$old"would otherwise collapse torm -rf -- /r1ifRELEASESwere ever empty. Executed underdash, an emptyRELEASESstops withRELEASES: parameter not set or nulland exit status 2 before anything is removed.case $KEEP in *[!0-9]*|0|1) die ...: eachcasearm lists glob patterns separated by|.*[!0-9]*is any string containing a character that is not a digit, and0and1are the two numbers that leave no rollback target. An unset or emptyKEEPnever reaches thecase, because${KEEP:-5}has already turned it into 5. Executed:abc,0and1are rejected;2,5and an emptyKEEP(which becomes 5) are accepted..current.$$:$$is the process id of the running shell, so the temporary link gets a name such as.current.23that is unique to this run (two runs cannot clash) and starts with a dot so a directory listing does not show it.rename(2): the operating system call that renames a file or link; the(2)is the manual-page section for system calls. When the destination already exists it is replaced in a single step, which is what makes the switch atomic.trap '...' EXITandtrap 'exit 1' HUP INT TERM: the first runs the quoted commands when the script ends normally, hitsdie, or is stopped byset -e. A shell killed by a signal does not necessarily run its EXIT trap (executed: dash left the lock directory behind after SIGTERM), so the second trap turns hangup, interrupt and terminate into an ordinaryexit 1, which then fires the EXIT trap and releases the lock and deletes the staging directory. A trap runs once the foreground command in progress returns, and SIGKILL or a power cut cannot be trapped at all: after one of those, check that no deploy is running and remove.deploy.lockby hand.ls -1 "$RELEASES" | sort -r | awk -v keep="$KEEP" 'NR > keep':ls -1lists one name per line,sort -rputs the newest timestamp first, andawkprints only the lines whose line numberNRis greater thankeep(-vpasses the shell'sKEEPinto awk as the variablekeep). With the three releases on disk whenpruneruns in the last deploy of the run below (r1, r2 and the new r6) andKEEP=2, the stages produce:
ls -1: r1 r2 r6
sort -r: r6 r2 r1
NR > 2: r1 <- this one is deleted; r6 and r2 stay
Why each step is there
- Integrity first.
sha256sum -cchecks the tarball against a.sha256file produced at build time (a hash computed in the build pipeline and shipped alongside, not computed on the server, which would only prove the file equals itself). A mismatch aborts before anything is unpacked. A second check rejects archives whose member paths are absolute or contain.., which would write outside the release directory. - Unpack to a hidden staging directory, then rename it. A release name under
releases/therefore only ever exists fully unpacked. A failed unpack or an interrupting signal leaves a.incoming-*directory that the traps remove; afterkill -9it can survive, and the listing ignores it either way. - Atomic switch.
ln -sfnlooks atomic but is not: it removes the old link and then creates the new one, leaving a moment with nocurrent. Creating a temporary link.current.<pid>and usingmv -fT(-Tmeans "treat the destination as a file", somvreplaces the link instead of moving the new link into the directory it points to) is a singlerename(2).-Tis in GNUmv, and it also worked in the BusyBoxmvof an Alpine container; check your target'smvbefore relying on it. - Health check after the switch, with automatic rollback.
HEALTH_CMDruns with the live symlink as its working directory, so it tests what the web server is actually serving. In production it would be something likecurl -fsS http://127.0.0.1/healthz. On failure the script switches back to the previous release, deletes the failed release directory (otherwiserollback, which picks the next older name, could later put a build that is known to be bad back in front of users) and exits 1, so the pipeline sees the failed deploy. - Retention.
prunekeeps the newestKEEPreleases and never removes the live one.KEEPmust be at least 2 so there is always a rollback target. Release names are generated by the script, so parsinglsoutput is safe here; it would not be for user-supplied names. - One deploy at a time.
mkdirof the lock directory succeeds for exactly one caller, which also works on systems withoutflock.
Executed run
Three builds (v1, v2 good, v3 whose healthz file says broken), a corrupted artifact, a path-traversal archive, a manual rollback and a final deploy with KEEP=2. Run under dash, which is /bin/sh on Debian:
#!/bin/sh
set -eu
export WEBROOT=/tmp/site KEEP=2 HEALTH_CMD='grep -qx ok healthz'
rm -rf "$WEBROOT" /tmp/build; mkdir -p "$WEBROOT" /tmp/build
build() { # build NAME HEALTH_BODY
mkdir "/tmp/build/$1"; echo "page $1" > "/tmp/build/$1/index.html"; echo "$2" > "/tmp/build/$1/healthz"
tar -czf "/tmp/build/$1.tar.gz" -C "/tmp/build/$1" .
( cd /tmp/build && sha256sum "$1.tar.gz" > "$1.tar.gz.sha256" )
}
build v1 ok; build v2 ok; build v3 broken
show() { printf 'current -> %s | releases: %s\n' "$(readlink "$WEBROOT/current")" "$(ls -1 "$WEBROOT/releases" | tr '\n' ' ')"; }
d() { sh ./deploy.sh "$@" || echo "(exit $?)"; }
echo '--- two good deploys'
RELEASE_ID=r1 d deploy /tmp/build/v1.tar.gz | grep -v ': OK'; RELEASE_ID=r2 d deploy /tmp/build/v2.tar.gz | grep -v ': OK'; show
echo '--- broken build fails its health check'
RELEASE_ID=r3 d deploy /tmp/build/v3.tar.gz | grep -v ': OK'; show; cat "$WEBROOT/current/index.html"
echo '--- corrupted artifact'
echo junk >> /tmp/build/v2.tar.gz
RELEASE_ID=r4 d deploy /tmp/build/v2.tar.gz; show
echo '--- path traversal in archive'
mkdir /tmp/build/evil; echo x > /tmp/build/evil/f
tar -czf /tmp/build/evil.tar.gz -C /tmp/build/evil --transform 's,^,../,' f
( cd /tmp/build && sha256sum evil.tar.gz > evil.tar.gz.sha256 )
RELEASE_ID=r5 d deploy /tmp/build/evil.tar.gz | grep -v ': OK'; show
echo '--- manual rollback, then a good deploy prunes to KEEP=2'
d rollback; show; cat "$WEBROOT/current/index.html"
build v6 ok; RELEASE_ID=r6 d deploy /tmp/build/v6.tar.gz | grep -v ': OK'; show
--- two good deploys
live: r1 (was: none)
live: r2 (was: r1)
current -> releases/r2 | releases: r1 r2
--- broken build fails its health check
deploy: health check failed, rolled back to r2
(exit 1)
current -> releases/r2 | releases: r1 r2
page v2
--- corrupted artifact
v2.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
deploy: checksum mismatch, nothing was unpacked
(exit 1)
current -> releases/r2 | releases: r1 r2
--- path traversal in archive
tar: Removing leading `../' from member names
deploy: archive contains absolute or ../ paths
(exit 1)
current -> releases/r2 | releases: r1 r2
--- manual rollback, then a good deploy prunes to KEEP=2
live: r1 (rolled back from r2)
current -> releases/r1 | releases: r1 r2
page v1
live: r6 (was: r1)
pruned r1
current -> releases/r6 | releases: r2 r6
Two details of the driver. grep -v ': OK' hides the v1.tar.gz: OK line that sha256sum -c prints for every artifact that verifies, purely to keep the transcript short. --transform 's,^,../,' is a GNU tar option that applies a sed-style substitution to every member name as the archive is written; here it prepends ../, so the archive holds a member named ../f (listing it with tar -tzf prints ../f), the shape the path check must reject. tar prints its own Removing leading warning while creating it.
Reading the output: the broken release r3 was live only for the duration of its health check, then current went back to r2 and the r3 directory was removed. Both bad artifacts were rejected before any unpacking. rollback moved from r2 to the next older release r1. The last deploy ended with r2 and r6 on disk: r1 was pruned and r6 is live, with r2 kept as the rollback target. Because the failed r3 was deleted, it neither takes a retention slot nor can rollback select it (had it been kept, rollback from r6 would have made the broken r3 live; this was run and confirmed). Keep the build artifact and the health-check output in the pipeline's logs if the post-mortem needs them.
Trade-offs and pitfalls
- Database migrations are not rolled back by a symlink. Rollback only undoes the code. Schema changes must be backward compatible with the previous release (expand, deploy, then contract in a later release).
- "No downtime" also depends on the server. PHP-FPM and nginx can cache the resolved path; nginx setups with PHP commonly need
$realpath_rootin thefastcgi_paramlines and an opcache reset or reload so workers stop using the old directory. Long-running processes that opened files in the old release keep them until restarted. - Shared state (uploads, sessions,
.env) lives outsidereleases/and is symlinked into each release, otherwise every deploy loses it. - If
RELEASE_IDcollides (two deploys in the same second), the script refuses rather than overwriting.
A core switch fails and the NOC receives 300 alerts within two minutes, most from devices that are merely unreachable behind it. How should the alerting be designed so the NOC sees the cause first?
Sample Answer
Direct answer
Design the alerting around dependencies, not devices. Give every alert a label that says what it depends on, make the failure of the upstream device suppress the "unreachable" alerts of everything behind it, group what remains, and page only on the cause. When the core switch fails, the network operations centre (NOC) should see one clear alert (core switch down) at the top, with the downstream unreachable devices still recorded but not notified. This is dependency-based suppression, called inhibition in Prometheus Alertmanager (the open-source alert router I use for the example).
The design, step by step
- Topology as data. Pull parent relationships from the source of truth (the inventory or CMDB, the system that records which device connects to which). Each device's reachability alert gets a
blocked_bylabel naming the upstream device whose failure would hide it. The core switch's own down alert carries its own name in the same label. - Separate cause from effect. Two alert types:
DeviceDownfor a device that fails its own health check from an independent vantage point (a place the probe runs from, other than behind that device), andDeviceUnreachablefor devices whose probes fail. Only the cause class is allowed to inhibit, and the cause alert also carries arolelabel (hererole="core") so that only devices of a role you trust as an upstream can suppress others. - Inhibition rule. A source alert (the cause) suppresses target alerts (the effects) when the labels listed in
equalmatch, per the Alertmanager configuration reference. - Group and wait. Group by
siteandblocked_by, meaning alerts that share the same value of both labels are bundled into one notification.group_wait(default 30s) is how long to wait before the first notification for a new group, giving the cause alert time to arrive before the group is sent.group_interval(default 5m) spaces later notifications andrepeat_interval(default 4h) controls reminders. - Route by severity. The cause alert pages; downstream effects appear on the dashboard and in the ticket with the incident, not in the pager.
- Redundant paths. A device dual-homed (connected to two upstream devices) to two cores is unreachable only if both are down. Emit its
blocked_byas the pair (for examplecore-pair-a) and have a rule that raises a pair-level cause alert only when both members are down. A single-valued label would otherwise suppress a real, separate failure.
Configuration and executed proof
route:
receiver: noc
group_by: [site, blocked_by]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receivers:
- name: noc
inhibit_rules:
- source_matchers:
- alertname="DeviceDown"
- role="core"
target_matchers:
- alertname="DeviceUnreachable"
equal: [blocked_by]
Reading the rule line by line. route sends everything to the noc receiver and bundles alerts by site and blocked_by (group_by), waiting 30 s before the first notification of a new group (group_wait), 5 minutes between later notifications for it (group_interval) and 4 hours between reminders (repeat_interval). inhibit_rules holds the suppression rule. source_matchers selects the alerts that are allowed to suppress: a DeviceDown alert whose role label is core. target_matchers selects the alerts that can be suppressed: any DeviceUnreachable. equal: [blocked_by] adds the condition that a suppression happens only when the source and the target carry the same value for blocked_by. So a DeviceDown for core-sw1 suppresses every DeviceUnreachable labelled blocked_by=core-sw1, but not acc-900, which is labelled blocked_by=core-sw2 and so is still reported.
I ran this on Alertmanager 0.34.1 in a container: amtool check-config reported SUCCESS. I then posted 300 alerts to /api/v2/alerts: one DeviceDown for core-sw1 (blocked_by=core-sw1), 298 DeviceUnreachable access switches with blocked_by=core-sw1, and one DeviceUnreachable (acc-900) behind a different core, core-sw2, that is still up. Result: 2 alerts not inhibited (core-sw1 and acc-900) and 298 inhibited; querying with inhibited alerts included still returned all 300, and grouping produced two groups (blocked_by=core-sw1 with 299 members, blocked_by=core-sw2 with 1). So the NOC gets 300 alerts reduced to 2 notifications, and the unrelated failure behind the healthy core was not hidden.
I then resolved the core-sw1 alert while leaving the other 298 firing: the number of non-inhibited alerts went from 2 to 299. That is the right behaviour (when the cause clears, anything still broken re-surfaces) and it means the NOC should expect a second round of alerts if devices stay down after the core returns.
Trade-offs and pitfalls
- Suppression hides symptoms by design. Keep inhibited alerts visible in the UI and the API (they were in the test) and review the dependency labels after every topology change, because a stale
blocked_byeither suppresses a real fault or lets the storm through. - The cause must be detected independently of the path it sits on. If the only monitor sits behind the core, its loss looks like unreachability for everything, so monitor the core from at least two vantage points.
- Polling gaps create ordering races: children can fail a probe before the core is flagged, which is what
group_waitcovers. Do not shorten it to speed the first page. - Keep a count of suppressed alerts on the incident so a post-incident review can see the blast radius (how many devices and users one failure touched; here 298 devices).
A security team wants to block clients from using public encrypted DNS, while users argue it protects their privacy. How do DoH and DoT change what an enterprise can see and enforce, and what would you propose?
Sample Answer
Direct answer
Encrypted DNS protects the query from everyone on the path, and that includes the enterprise. DoT (DNS over TLS) wraps DNS in TLS on its own TCP port, 853 (RFC 7858). DoH (DNS over HTTPS) carries DNS messages inside ordinary HTTPS requests (RFC 8484). Once a device talks to a public encrypted resolver, the company can no longer see the names it looks up, apply DNS-based blocking, or log queries per client. I would not fight this with a blanket ban alone. I would run an enterprise resolver that itself speaks encrypted DNS, point managed devices at it by policy (so users get privacy from the local network and the company keeps visibility on its own devices), block the unmanaged routes at the network edge, and be explicit with staff about what the company can see.
What each side sees and can enforce
| Plain DNS (UDP/TCP 53) | DoT (TCP 853) | DoH (TCP 443) | |
|---|---|---|---|
| Query names visible to the network | Yes | No (encrypted) | No (encrypted) |
| Easy to identify as DNS | Yes, by port | Yes, by port 853 | No, looks like web traffic to a firewall that only reads ports |
| Block with a firewall rule | Port rule works | One rule: deny outbound TCP 853 | Port rule would block all web traffic; needs destination lists or TLS inspection |
| Enterprise DNS filtering, logging, response policy (rewriting or refusing the answer for chosen names) | Works | Works only if devices use the enterprise resolver | Works only if devices use the enterprise resolver |
| What remains visible | Everything | Destination IP, volume, timing | Destination IP, volume, timing, and usually the server name the client announces in the TLS handshake (SNI, server name indication, normally sent unencrypted so the server knows which site is wanted) |
RFC 8484 states this plainly: filtering or inspection systems that rely on unsecured transport of DNS will not function in a DoH environment. The protocol encrypts and authenticates the server, but it does not tell the resolver who is a managed device.
The two positions are both partly right
- Security team: DNS logs and DNS firewalls (blocking known-bad domains, spotting command-and-control beaconing (malware contacting its operator's server at regular intervals), alerting on newly registered domains) only work if the queries pass through resolvers the company controls. A browser that silently sends its lookups to a public DoH service bypasses all of it.
- Users: On a coffee-shop or home network, plain DNS is readable and forgeable by anyone on the path, and encrypted DNS fixes that. The privacy gain is real against the local network and the ISP. It does not hide queries from the resolver operator, so it moves trust to a different party rather than removing it.
What I would propose
Steps 1 to 3 do the enforcing. Steps 4 to 6 make the setup smoother for clients and defensible to staff.
- Run an enterprise resolver that supports encrypted DNS. Clients on the corporate network and on VPN use it, over DoT or DoH, so the connection is protected from the LAN while the company keeps logging and filtering at the resolver. The resolver should also forward upstream over TLS. An Unbound example that passes
unbound-checkconf(the forwarder address is a documentation placeholder, which stands in for the vetted upstream you choose):
server:
interface: 127.0.0.4
tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt
forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 192.0.2.53@853#dns-upstream.example.net
Reading the block, line by line:
server:opens the global section.interface: 127.0.0.4is the address Unbound listens on for client queries (a loopback placeholder here; on a real resolver it is the address your clients are pointed at).tls-cert-bundlenames the file of trusted certificate authorities (CAs) that Unbound uses to check the upstream's TLS certificate. Without it the upstream cannot be authenticated.forward-zone:withname: "."means "the root", so every name that is not answered locally is forwarded rather than resolved by walking down from the root servers.forward-tls-upstream: yesmakes Unbound speak DoT (TLS) to the upstream instead of plain port 53.forward-addr: 192.0.2.53@853#dns-upstream.example.nethas three parts: the upstream's IP address,@853the port to connect to, and#dns-upstream.example.netthe name that the upstream's certificate must carry. The#namepart is what stops someone else answering on that IP: Unbound completes the TLS handshake only if the certificate is valid for that name.
unbound-checkconf printed unbound-checkconf: no errors in <file>, naming the file it checked. It checks syntax only; it does not contact the upstream.
- Configure managed endpoints by policy, and lock the settings. Firefox has a
DNSOverHTTPSenterprise policy withEnabled,ProviderURLandLockedkeys, so you can point it at the enterprise DoH endpoint and stop users changing it. Chrome'sDnsOverHttpsModepolicy takesoff,automatic(DoH first, with fallback to insecure) orsecure(DoH only, with no fallback, so resolution fails if the DoH server is unreachable), sosecurewith the enterprise template is the managed-device setting. An operating-system or browser setting that the user can change is not enforcement. - Block the unmanaged routes at the network edge. Deny outbound TCP 853 except from your resolvers (one firewall rule). For DoH, deny outbound connections to the public DoH providers you identify by address list or secure web gateway category, and refuse their hostnames at the enterprise resolver so a client that uses the system resolver to find them fails. A secure web gateway is a proxy that filters web traffic by site category, which is how "public DoH provider" can be a category you block. In the lab, an Unbound resolver configured with
local-zone: "dns.google." always_refuseand the same forcloudflare-dns.com.returned:
dns.google: status: REFUSED
cloudflare-dns.com: status: REFUSED
- Let clients discover the enterprise's own encrypted resolver. DDR (Discovery of Designated Resolvers, RFC 9462) lets a client that knows only the plain resolver address ask that resolver for the name
_dns.resolver.arpa.with record type SVCB (service binding, a record that says how to reach a service: target host name, protocol and port). The answer points to the same operator's encrypted endpoint. With a local-data entry in Unbound, the lab returned:
$ dig @127.0.0.1 _dns.resolver.arpa. SVCB +noall +answer
_dns.resolver.arpa. 300 IN SVCB 1 dns.corp.example. alpn="dot" port=853 ipv4hint=192.0.2.53
Read it as: priority 1, target host dns.corp.example., alpn="dot" the protocol (DoT), port=853, and ipv4hint an address to try without another lookup (all values are illustrative). The security rule is that for verified discovery the client must check that the encrypted resolver's certificate lists the IP address of the plain resolver it started from (RFC 9462 requires an iPAddress entry in the certificate's subjectAltName). The reason is that the plain query can be forged by anyone on the path, and a certificate naming that IP shows the owner of that address designated the encrypted endpoint. Advertising DDR means clients upgrade to your resolver rather than to a public one.
5. Give guests and personal devices a separate network with no promise of DNS visibility, so you are not trying to inspect devices you do not manage.
6. Write down the privacy rules. Say what the resolver logs, how long they are kept, who may query them, and that encrypted DNS is on for staff against outside observers. This turns a dispute into a documented trade-off.
Why not just block everything
Blocking 853 is cheap and reliable. Blocking DoH is a moving target: any HTTPS server can answer DNS queries, so address lists age, and TLS inspection (a gateway that decrypts and re-encrypts traffic using a certificate the company's devices trust) breaks pinned applications (apps that accept only their own expected certificate and so reject the gateway's) and adds cost and legal exposure. Block what is cheap (853, known provider hostnames and addresses), and rely on managed-endpoint policy for the rest. Some devices, such as unmanaged phones or applications that ship their own resolver, will still get through, and the answer to that gap is endpoint management, not more DNS rules.
Performance and privacy trade-offs
- Performance: A TLS or HTTPS connection costs extra round trips once, then queries reuse the connection, so steady-state latency is close to plain DNS. Enterprise resolver capacity must cover many long-lived TLS sessions. Cache hits at the enterprise resolver are faster than a cold lookup to a public service.
- Privacy: Employees' lookups are visible to the company on managed devices, which is a policy and legal decision, not just a technical one. Public encrypted DNS hides queries from the local network but hands them to the provider.
- Failure mode: With Chrome's
securemode, an outage of the enterprise resolver means no resolution at all, whileautomaticfalls back to insecure DNS. Choose consciously and monitor the resolver.
A C++ service stores millions of small objects using vector<unique_ptr<Foo>>. Explain the memory and cache implications of that layout and refactor to a cache-friendly memory layout. Provide a code sketch for a contiguous store approach and describe trade-offs for polymorphism and lifetime management.
Sample Answer
Approach
vector<unique_ptr<Foo>> gives every Foo its own separate heap allocation, so
consecutive elements in the vector are scattered across the heap, not adjacent in
memory; iterating the vector means chasing a pointer to a new, effectively random
location on every step, which defeats CPU prefetching and wastes most of each fetched
cache line on Foo objects that weren't actually requested next. The fix is to store
Foo objects by value in one contiguous buffer (a plain std::vector<Foo>) so
consecutive elements are genuinely adjacent in memory and iteration becomes a simple
linear scan.
Code
#include <cstdint>
#include <cstdio>
#include <memory>
#include <vector>
struct Foo {
int id;
double value;
char tag[24];
Foo(int i, double v) : id(i), value(v) { tag[0] = 'F'; }
};
// Refactored layout: one contiguous allocation holding all Foo objects by value.
// (Assumes call sites never need runtime polymorphism on Foo; see the trade-offs
// section below for the case where they do.)
using ContiguousStore = std::vector<Foo>;
int main() {
const int N = 8;
std::vector<std::unique_ptr<Foo>> pointerStore;
pointerStore.reserve(N);
for (int i = 0; i < N; ++i) {
pointerStore.push_back(std::make_unique<Foo>(i, i * 1.5));
}
ContiguousStore contiguousStore;
contiguousStore.reserve(N); // exactly one allocation for all N objects
for (int i = 0; i < N; ++i) {
contiguousStore.emplace_back(i, i * 1.5);
}
printf("sizeof(Foo) = %zu bytes\n", sizeof(Foo));
printf("pointer-store heap allocations for %d objects: %d\n", N, N);
printf("contiguous-store heap allocations for %d objects: 1\n", N);
bool pointerStrideConstant = true;
for (int i = 1; i < N; ++i) {
auto delta = reinterpret_cast<uintptr_t>(pointerStore[i].get()) -
reinterpret_cast<uintptr_t>(pointerStore[i - 1].get());
if (delta != sizeof(Foo)) pointerStrideConstant = false;
}
bool contiguousStrideConstant = true;
for (int i = 1; i < N; ++i) {
auto delta = reinterpret_cast<uintptr_t>(&contiguousStore[i]) -
reinterpret_cast<uintptr_t>(&contiguousStore[i - 1]);
if (delta != sizeof(Foo)) contiguousStrideConstant = false;
}
printf("pointer-store stride equals sizeof(Foo) for every element? %s\n",
pointerStrideConstant ? "yes" : "no");
printf("contiguous-store stride equals sizeof(Foo) for every element? %s\n",
contiguousStrideConstant ? "yes" : "no");
return 0;
}
Compiled with clang++ -std=c++17 -Wall -Wextra -O2 and run directly:
sizeof(Foo) = 40 bytes
pointer-store heap allocations for 8 objects: 8
contiguous-store heap allocations for 8 objects: 1
pointer-store stride equals sizeof(Foo) for every element? no
contiguous-store stride equals sizeof(Foo) for every element? yes
Key points
pointerStore.reserve(N)still performsNseparateFooallocations, one per
make_uniquecall; only the vector's own internal array of pointers is reserved up
front, not theFooobjects it points to.contiguousStore.reserve(N)performs exactly one allocation sized for allN
Fooobjects;emplace_backconstructs eachFoodirectly into that
pre-reserved memory, with no per-element heap call.- The stride check is a directly observable consequence of the layout, not an
assumption: consecutiveFooobjects in the contiguous store are exactly
sizeof(Foo)apart in memory (confirmed above), which is precisely what lets the
CPU's hardware prefetcher predict and fetch the next element before it's needed;
the pointer store's addresses have no such fixed relationship.
Complexity
Both layouts are O(N) to construct and O(N) to iterate; the difference is not
algorithmic, it's constant-factor cache behavior: the contiguous layout turns
iteration into a linear scan of one buffer (prefetcher-friendly), while the
pointer-per-object layout turns iteration into N effectively independent memory
fetches to unrelated locations.
Edge cases
- Elements moved or removed from the middle:
std::vector<Foo>reallocates and
move-constructs surviving elements on growth past capacity, or shifts elements on
erase;Fooneeds a move constructor that is at least as cheap as its copy
constructor (the compiler-generated default is fine here sinceFoohas no owned
resources) for this to stay efficient. - A
Foothat owns another heap resource (astd::stringfield, for example):
contiguity of theFooobjects themselves does not make what they point to
contiguous; avector<Foo>of objects each holding astd::stringstill chases a
pointer for that string's own heap buffer, contiguous storage only removes one level
of indirection, not every level.
Trade-offs & pitfalls
Polymorphism: vector<unique_ptr<Foo>> is a common shape specifically because it
lets Foo be a base class with virtual methods and each element be a different
concrete derived type. Flattening to vector<Foo> by value only works if Foo is a
single concrete type; if run-time polymorphism is genuinely required, the standard
alternatives are a tagged union (std::variant<DerivedA, DerivedB, ...> stored
contiguously, dispatched with std::visit, contiguous and still exhaustively type-safe,
but a fixed, closed set of types) or keeping the objects contiguous per concrete type
(one vector<DerivedA>, one vector<DerivedB>, and so on) rather than mixed.
Lifetime management: unique_ptr gave individually stable addresses, so a pointer
into pointerStore[i] stayed valid even if other elements were added or removed from
the outer vector. Elements of a vector<Foo> do not have that guarantee: any operation
that can reallocate the vector (growth past capacity, insert, erase) invalidates
existing references and pointers into it. Code that previously stored a raw Foo*
across such an operation must be audited, since it silently becomes a dangling pointer
under the new layout.
Explain the concept of an error budget and how SRE teams use it to make release decisions and gating. Provide an example policy where an error budget burn leads to pausing automated deployments and the steps that follow.
Sample Answer
An error budget is the allowable amount of unreliability for a service over a time window (1 - SLO). It quantifies acceptable user-facing errors so teams balance feature velocity against reliability. SREs use error budgets to make objective release and gating decisions: if the budget is healthy, releases proceed; if it’s depleted, riskier changes are halted until reliability improves.
Example policy (automated-deploy gating):
- Thresholds:
- Green: >50% remaining — automated canary & rolling deploys allowed.
- Yellow: 10–50% remaining — restrict deploys to low-risk changes; require SRE approval.
- Red: ≤10% remaining — pause all automated deployments; enter remediation workflow.
Red-state steps:
- Immediately pause CI/CD automated pipelines for production.
- Notify stakeholders (developers, product, SRE) and open an incident/ticket.
- Triage root causes: correlate recent deploys, alerts, error rates, and infra changes.
- Implement mitigations (rollback suspect changes, scale capacity, fix config).
- Run targeted tests and short stabilization window (e.g., 1–4 hours) while monitoring SLOs.
- When error budget recovers above Yellow threshold and risk assessment done, allow limited releases with SRE oversight; restore full automation only after sustained stability (e.g., 72 hours).
This policy makes release gating data-driven, minimizes firefighting churn, and forces prioritized investment in reliability.
Recommended Additional Resources
- Designing Data-Intensive Applications by Martin Kleppmann - comprehensive guide to distributed systems
- The Site Reliability Workbook by Google - practical SRE practices and approaches
- Linux Performance Analysis in 60,000 Milliseconds - systems performance analysis methodology
- Kubernetes in Action - container orchestration and deployment patterns
- TCP/IP Illustrated Vol. 1 by W. Richard Stevens - deep networking knowledge
- LeetCode - practice coding problems, focus on medium-difficulty graph problems
- System Design Primer GitHub repository - curated system design resources
- Production Kubernetes by Josh Rosso and Rich Lander - production-grade Kubernetes deployment
- Observability Engineering by Charity Majors et al. - modern observability approaches
- UNIX and Linux System Administration Handbook by Nemeth et al. - comprehensive Linux reference
- Incident Response and Disaster Recovery by Zurich - incident management practices
- High Performance Browser Networking by Ilya Grigorik - web performance and protocol deep-dive
- Understanding Linux Network Internals by Christian Benvenuti - kernel networking concepts
Search Results
Top 15 Apple Reliability Engineer Job Interview Questions & Answers
Question #1. Can you describe your experience with reliability engineering, particularly in the context of hardware systems? · Question #2.
Apple SRE Interview Experience (Offer) - Software Engineering - Blind
Total process took 6 months, 3 months to reply to initial application (with referral), 1 month after completing interviews to get offer, 7 rounds total.
2025 Apple Site Reliability Engineer interview question bank
A complete set of Apple Site Reliability Engineer interview questions. Contributed by recent candidates and vetted by current Apple Site ...
Apple Reliability Engineer Interview Questions - NodeFlair
Apple Reliability Engineer interview questions and answers. Free interview details posted anonymously by Apple interview candidates.
Site Reliability Engineer (SRE) Interview Preparation Guide - GitHub
A collection of questions to practice with for SRE interviews · SRE Interview Questions · Sysadmin Test Questions · Kubernetes job interview questions · DevOps ...
Apple Site Reliability Engineer Interview: Process + Questions
Prepare thoughtful questions: “What is the biggest reliability challenge your team faces right now?” “How do you measure success for an SRE here ...
This interview preparation guide was generated using AI-powered research from the sources listed above. While we strive for accuracy, we recommend verifying critical information from official company sources.
Want to create your own tailored preparation guide using our deep research?
Get Started for FreeInterview-Ready Courses
Visual-first, interactive, structured learning paths
Browse Site Reliability Engineer (SRE) jobs
AI-enriched listings across hundreds of company career pages
Explore Jobs