Airbnb Site Reliability Engineer (Entry Level) Interview Preparation Guide

Situation/goal: We need a globally-distributed metadata service with strongly consistent reads/writes across regions and 99.99% availability. Below I compare AWS RDS (Postgres), Google Cloud Spanner, and Azure Cosmos DB (SQL API) across latency, consistency, operational overhead, tooling, and cost from an SRE perspective.

1. Latency

• Spanner: Optimized for geo-distribution with TrueTime; single-digit to low-double-digit ms for regional reads/writes if using regional config, writes across regions incur Paxos/TrueTime coordination so higher but predictable latency.
• Cosmos DB: Multi-region writes with configurable consistency; offers low read latency (~single-digit ms) via local replicas; multi-master writes give low write latency but conflict resolution possible.
• RDS Postgres: Best for single-region low latency. Cross-region synchronous/async replication (Aurora Global DB) adds significant write latency or relaxed consistency if async.

2. Consistency model

• Spanner: Strong external consistency globally — ideal for metadata requiring global serializability.
• Cosmos DB: Tunable (Strong, Bounded Staleness, Session, etc.). Strong guarantees only within a single write region; multi-master strong consistency is limited; conflict handling is needed.
• RDS: Strong consistency within primary region. Cross-region strong consistency requires synchronous replication which hurts latency; most managed setups are eventually consistent across regions.

3. Operational overhead & tooling

• Spanner: Minimal operational burden for replication/partitioning; schema changes online; good SLA and monitoring via Cloud Console/Stackdriver; fewer operational tasks.
• Cosmos DB: Managed scale, multi-region config easy in portal/ARM; need to manage RU provisioning, conflict resolution, indexing policies; good metrics and diagnostics in Azure Monitor.
• RDS: Most operational work (backups, failover, read replicas) handled but you manage replicas, failover testing, schema migrations, cross-region failover playbooks; higher runbook complexity.

4. Cost under expected scale

• Spanner: Higher baseline cost (nodes) but predictable at scale; cost-effective for strong global consistency and heavy write workloads.
• Cosmos DB: Pay-per-throughput (RUs) can be expensive if metadata has varied spikes; multi-region write increases cost; may be cheaper for read-heavy patterns.
• RDS: Lower per-VM cost for single region; cross-region replicas and high-availability add cost; at global scale, operational and replication costs can exceed managed global DBs.

Recommendation (SRE view)

• If strict global strong consistency and simplified ops are primary: choose Spanner.
• If you need flexible consistency, very low local read latency, and multi-master writes with careful conflict handling: consider Cosmos DB.
• If workload is primarily single-region or you can tolerate eventual cross-region consistency and want SQL/Postgres ecosystem: RDS (Aurora Global DB) is appropriate but requires more operational runbooks to reach 99.99% globally.

Operational notes: validate failover scenarios, SLOs for cross-region write latencies, automate chaos/failover tests, monitor tail latencies and conflict rates (Cosmos), and model costs with realistic RU/node usage and multi-region replicas.

Stability: a sorting algorithm is stable if it preserves the relative order of records with equal keys. If two items A and B compare equal, A appears before B in the output iff A appeared before B in the input.

Examples:

• Stable sorts: Merge sort (stable when implemented to prefer left element on ties), Timsort (Python/Java uses it and is stable).
• Unstable sorts: QuickSort (typical in-place partitioning is unstable), HeapSort (reorders equal keys based on heap operations).

Why it matters for SRE — two realistic scenarios:

1. Multi-key log ordering: logs may be sorted by timestamp, but many entries share identical timestamps (to the ms). Stability preserves original ingestion/order of events, which is crucial to reconstruct causal sequences across services or trace request flows.
2. Incremental compaction/merge in storage: when compacting records keyed by (userID, sequence) and sorting by userID only, stability ensures existing sequence order is preserved for equal keys so app-level ordering guarantees remain intact without extra metadata.

When it doesn’t matter:

• Aggregation-only metric sorts: when you sort numeric metric buckets for display (e.g., top latency buckets) and equal values have no secondary significance, relative ordering of equals is irrelevant.

Pragmatic enforcement with an unstable built-in sort:

• Decorate with the original index (Schwartzian transform): sort by (key, original_index) so ties break by original position. Example in Python:

# make sort stable even if sort() were unstable
items = [(key, value) for key,value in items]
items_with_idx = [(key, idx, value) for idx,(key,value) in enumerate(items)]
items_with_idx.sort(key=lambda x: (x[0], x[1]))
result = [(k,v) for k,_,v in items_with_idx]

This is simple, low-overhead, and works across languages (attach index as secondary key).

Direct Answer

Clock skew is the difference between what two machines' clocks read at the same real instant. It matters because any decision that compares timestamps from different machines to decide what happened first, whether a lease is still valid, or whether a certificate is still in its valid window, quietly assumes those clocks agree, and in a real network they don't. A numerically later timestamp on one machine's clock does not reliably mean later in real time once you're comparing across machines.

Three Concrete Problems

1. Event ordering across services. If service A stamps an event with its own local clock and service B stamps a related event with its own local clock, and A's clock runs even slightly ahead of B's, an event that actually happened after (in real time) on B can end up with a numerically smaller timestamp than an earlier event on A. Anything that reconstructs what happened in what order by sorting on raw timestamps can get the sequence backwards. The same failure shows up in machine learning feature pipelines: if a feature-store write on one node happens slightly after a model-serving read on another node consumed the old value, but the writer's clock runs a bit fast, the write can carry an earlier or overlapping timestamp than the read, which corrupts any point-in-time audit of which feature value was actually used for a given prediction, if that audit trusts the raw timestamps.

2. A lease that expires early or late. Distributed locks are commonly granted as leases valid until wall-clock time T. If the holder's clock runs slow relative to the granting service's clock, the holder can believe it still owns the lease past the point the granting service has already reassigned it, expiring late from the holder's point of view and risking two nodes both acting as if they hold the resource. If the holder's clock runs fast instead, it can abandon a still-valid lease early and stop acting well before the granting service considers it expired, causing unnecessary churn.

3. A TLS (Transport Layer Security) certificate that looks valid on one node and invalid on another. Certificate validity is a wall-clock range check, evaluated locally by whichever machine happens to be doing the handshake, against the certificate's not-before and not-after dates. If one node's clock has drifted backward past the not-before date, or forward past the not-after date, that single node rejects a certificate every correctly-clocked node accepts, or the reverse, producing a confusing, node-specific TLS failure that looks like a certificate problem but is actually a clock problem.

A Concrete Trace of Why Naive Ordering Is Unsafe

Node A's clock reads 100 and Node B's clock reads 96 at the same real instant, a 4-unit skew. Event E1 happens on Node A at that instant and is stamped 100. Two time units later in real time, event E2 happens on Node B; by then B's clock reads 98 (96 plus 2), so E2 is stamped 98. Comparing the raw timestamps, 100 is greater than 98, so E1 looks like it happened after E2. But in real time, E2 actually happened after E1. Any process that orders events purely by comparing these timestamps gets the sequence exactly backwards, even though the comparison itself, 100 greater than 98, is arithmetically correct.

Why This Isn't Just a Sync-the-Clocks-Better Problem

Synchronizing physical clocks, with NTP (the Network Time Protocol) or a hardware-disciplined protocol like PTP (Precision Time Protocol), reduces the size of the skew, but it does not make comparing two independently-running clocks perfectly safe; it only shrinks the window in which the trace above can happen. The standard engineering answer for ordering that has to be correct, not just human-readable, is to stop relying on raw wall-clock comparison for that purpose and use a logical clock instead: a counter that each node increments on its own events and carries along with outgoing messages, which correctly captures which events could have influenced which others regardless of clock drift. Production distributed databases often go a step further and use a hybrid logical clock (HLC), which combines a physical-time component with a logical counter, to get correct ordering without giving up a timestamp that's still roughly readable as wall-clock time.

Trade-offs and Pitfalls

• Don't confuse clock skew, which is that two clocks disagree right now, with clock drift, the rate at which they diverge over time; skew is the instantaneous symptom you observe, drift is the ongoing cause, and a monitoring setup that only alerts on one of them will miss the other.
• A common wrong turn is assuming that running NTP means this is handled. NTP typically keeps clocks within milliseconds of each other, which is fine for human-readable log timestamps, but any nonzero skew is still a real correctness risk for anything that depends on strict ordering, so it reduces the problem rather than eliminating it.
• The lease-expiry problem specifically is best addressed by combining a conservative time-to-live with fencing at the protected resource, rejecting stale operations based on a monotonically increasing token rather than on wall-clock time at all, instead of trying to shrink clock skew to zero, which isn't achievable.

Direct answer

With a mixed panel you cannot pick one register the way you would for a single audience retelling, so the technique is to structure the story in layers instead: lead with a headline result any of the three would immediately understand, then let the Action section carry small, clearly-flagged hooks for the engineer, the product manager, and the executive, rather than switching vocabulary for one person at the expense of the other two.

Building a layered story

This still rests on the same four beats underneath, Situation, Task, Action, Result. The layering changes how you deliver those beats, not the shape itself.

Adapting a story for an executive alone usually means stripping out technical detail; adapting it for an engineer alone usually means adding it back in. In a mixed panel, fully doing either means losing one of the other two people. The fix is not a compromise register vague enough to bore all three, it is a layered story where the top layer is universally clear and each subsequent layer answers a different person's likely question.

The layering, in the order to build it:

1. Start with a result in plain outcome language everyone in the room cares about, time saved, risk removed, cost or revenue affected, users unblocked. This is the headline and it should need zero specialized vocabulary to land.
2. In Situation and Task, include the one number or constraint that matters to the business side, a deadline, a cost, a customer impact, stated briefly.
3. In the Action, include one clearly-labeled technical decision point for the engineer, specific enough that a technical listener recognizes real judgment, but framed as a decision made ("I chose X over Y because...") rather than a full technical walkthrough. A decision and its reason is understandable even without every technical detail.
4. Also in the Action or Result, include one clearly-labeled product angle, for example what you chose not to build, or how you weighed user impact against effort.
5. Close with the Result again, tying back to the opening headline so anyone who tuned out a layer they did not need still lands on the same conclusion.

The discipline is labeling each layer through pacing and phrasing, not literally narrating three separate stories aimed at three separate people.

Worked example

"Result up front: we cut our checkout failure rate by roughly half last quarter, which mattered because failed checkouts were a top complaint in support tickets. Situation: our payment flow was failing intermittently, hard enough to reproduce that two prior attempts to fix it had stalled. Task: I was asked to find and fix the root cause. Action, the technical decision point: I chose to add structured retry handling at the payment gateway boundary rather than patch individual call sites, because the failures were clustered around timeout handling in one shared layer, not scattered across the code, so a systemic fix returned more value for less risk of missing a case. The product angle: that meant delaying two smaller feature requests by about two weeks to free up the time, a trade-off I flagged to the product manager on the project before committing to it. Result: checkout failures dropped by roughly half, support tickets referencing checkout stopped being a top category, and the two delayed features shipped the following sprint."

Trade-offs and pitfalls

Avoid giving each person their own sequential mini-segment ("for the engineers in the room..."), which reads as three speeches stitched together and makes the room feel talked past rather than talked to.

Watch for over-indexing on the most senior person and dropping the technical layer entirely, which can read as evasive to the engineer, who may be the one asked afterward whether your account held up.

A fully layered story takes more preparation than a single-register one, since you have to identify in advance which decision point serves which listener. That extra prep is worth it for panel-style interviews specifically, not for every retelling of the story.

Direct answer

Serialize by writing every node exactly once (iterate the graph's key set, not a traversal frontier, so there is nothing to loop forever on) along with its adjacency list, then deserialize by rebuilding all declared nodes first and filling in adjacency second. The cycle-safety comes entirely from never using recursion or a visited-during-DFS approach to drive the writing process; a flat iteration over "all nodes I know about" has no notion of "currently in progress" to loop back into.

Structured elaboration

Why a DFS-style walk is the wrong approach here. A naive serializer that recursively follows edges to decide what to write next needs a visited set to avoid infinite recursion on a cycle, and even then it complicates the "avoid duplicating nodes" requirement (you would need to make sure a node visited via one path is not re-emitted via another). Iterating the graph's own node set sidesteps this entirely: every node is a top-level key in the input, so there is exactly one place each node's adjacency list gets written, independent of how many cycles or how much fan-in the graph contains.

Preserving node identity. Each node keeps its original id as its map key in the serialized form; deserialization uses that same id both when creating the node and when resolving every neighbor reference, so two edges pointing at the same node in the original graph point at the same reconstructed node afterward, not two independent copies.

Disconnected graphs. Because serialization iterates every key in the input dict, a node with no incoming edges (isolated, or the root of its own separate component) is still visited and written with an empty (or non-empty) adjacency list; nothing about the approach depends on reachability from a single starting point.

Worked example

import json
from typing import Dict, List, Any

def serialize(graph: Dict[Any, List[Any]]) -> str:
    '''Serialize a directed graph (possibly with cycles) to a JSON string.
    Each node is written exactly once; disconnected nodes (present as keys
    with no incoming edges) are preserved because we iterate graph.keys(),
    not a traversal frontier, so there is no traversal to loop forever on.'''
    node_ids = sorted(graph.keys(), key=str)
    payload = {
        "nodes": [str(n) for n in node_ids],
        "edges": {str(n): [str(v) for v in graph[n]] for n in node_ids},
    }
    return json.dumps(payload, separators=(",", ":"), sort_keys=True)

def deserialize(s: str) -> Dict[str, List[str]]:
    data = json.loads(s)
    nodes = [str(n) for n in data["nodes"]]
    edges = {str(k): [str(v) for v in vs] for k, vs in data["edges"].items()}
    graph: Dict[str, List[str]] = {n: edges.get(n, []) for n in nodes}
    # any node referenced only as a target gets an empty adjacency entry
    for src, nbrs in edges.items():
        for nb in nbrs:
            if nb not in graph:
                graph[nb] = []
    return graph

if __name__ == "__main__":
    # Directed graph WITH a cycle (A -> B -> C -> A) plus a disconnected node D
    g = {"A": ["B"], "B": ["C"], "C": ["A", "B"], "D": []}
    s = serialize(g)
    print("Serialized:", s)

    g2 = deserialize(s)
    print("Deserialized:", g2)

    same_edges = {k: sorted(v) for k, v in g.items()} == {k: sorted(v) for k, v in g2.items()}
    print("Round-trip preserves adjacency exactly:", same_edges)
    print("Disconnected node D preserved with empty list:", g2.get("D") == [])

    # Larger cyclic graph, confirm serialization terminates and is idempotent
    big = {str(i): [str((i + 1) % 500), str((i + 250) % 500)] for i in range(500)}
    s_big = serialize(big)
    g_big = deserialize(s_big)
    print("500-node cyclic graph round-trips:", g_big == big)
    print("Serialized length (chars) for 500-node graph:", len(s_big))

Output (actually executed with python3):

Serialized: {"edges":{"A":["B"],"B":["C"],"C":["A","B"],"D":[]},"nodes":["A","B","C","D"]}
Deserialized: {'A': ['B'], 'B': ['C'], 'C': ['A', 'B'], 'D': []}
Round-trip preserves adjacency exactly: True
Disconnected node D preserved with empty list: True
500-node cyclic graph round-trips: True
Serialized length (chars) for 500-node graph: 12581

Complexity

• Serialize: $O(V + E)$ time (visit every node once, every adjacency entry once), $O(V + E)$ output size.
• Deserialize: $O(V + E)$ time to parse and rebuild both the node set and every adjacency entry.

Edge cases

• Self-loops (a node listing itself as a neighbor): serialized and deserialized like any other edge, no special handling needed since nothing here depends on traversal state.
• Node ids that are not strings: cast to strings at serialization time so they are valid JSON object keys, and cast back only if the caller's original id type is known; the example above keeps everything as strings after a round trip, which the caller should account for if the original graph used integer ids.
• A neighbor referenced in some node's adjacency list but never itself a top-level key in the input: still gets an empty adjacency entry on deserialize, so the reconstructed graph is never missing a node that any edge points at.

Trade-offs and pitfalls

• Versioning and security when parsing. Once this format is used across service boundaries or persisted to disk, add an explicit "version" field to the payload so a future format change can be detected and migrated rather than silently misparsed, and treat json.loads output as untrusted input, cap the maximum nodes/edges accepted before deserializing fully, and validate that every neighbor id referenced actually resolves, rather than trusting the payload's internal consistency.
• Large graphs. The approach above builds the full JSON string in memory. For graphs that do not fit in memory as one object, switch to NDJSON (one node's record per line) or a streaming JSON writer that emits nodes incrementally, and a streaming parser on the read side; this keeps the same "iterate the full node set once" property without requiring the whole graph to be materialized as one string at once.
• Common mistake: trying to serialize via a DFS/BFS traversal starting from an arbitrary root, which will miss any node not reachable from that root (silently dropping disconnected components) unless the code explicitly restarts the traversal from every unvisited node, at which point it has reinvented "iterate the full node set," just with extra steps and extra cycle-safety bookkeeping that iterating the node set directly never needed in the first place.

Direct answer

Model each component's availability as a probability of being up, then combine them with two rules: components in series (every one must work for the system to work) multiply, and components in parallel (redundant copies of the same thing, any one of which is enough) combine as one minus the product of their failure probabilities. Once you have that model, "where does an extra dollar of redundancy buy the most availability" becomes a concrete comparison: compute the marginal gain in availability from each candidate investment, divide by its cost, and fund whichever option has the highest availability-per-dollar, repeating as budget allows.

The two combination rules

Series availability is always less than or equal to the least available component, because every additional required link is one more way for the chain to break. Parallel availability is always greater than or equal to the most available individual replica, because failure now requires every redundant copy to fail simultaneously, which independence makes rapidly unlikely as replica count grows.

Modeling redundant replicas across regions

For $n$ identical replicas within a region (parallel, independent failures) and $r$ regions (also parallel: the system is up if any one region is up):

where $a$ is a single replica's availability. The system-wide failure probability collapses to $(1-a)^{nr}$: the whole deployment is down only if every one of the $n \times r$ independent replicas is simultaneously down.

Worked example: where does the next dollar go

Start with a single instance at $a = 0.99$ (two nines, a not-unreasonable single-VM baseline) and no redundancy ($n=1, r=1$):

Converting to expected downtime per year (using 525,600 minutes/year, $365 \times 24 \times 60$):

Going from no redundancy to a second replica in the same region already removes over 5,200 minutes of downtime per year. Going from one region to two regions on top of that removes only another 52.5 minutes. That's the diminishing-returns pattern the model predicts: each additional nine costs roughly the same effort but buys an order of magnitude less absolute downtime reduction than the nine before it, because the failure probability being multiplied away is already small.

Deciding where the next dollar goes when both axes already have redundancy

For a system already at $n$ replicas and $r$ regions, the marginal availability gain from adding one more replica (holding regions fixed) versus adding one more region (holding replicas fixed):

At $n=3$, $r=2$, $a=0.99$: $(1-a)^{nr} = (0.01)^6 = 10^{-12}$. The bracket terms are $1-(0.01)^2 = 0.9999$ for adding a replica, versus $1-(0.01)^3 = 0.999999$ for adding a region, so $\Delta A_{\text{region}}$ is about 0.01 percent larger than $\Delta A_{\text{replica}}$ here. The pattern behind that: the bracket term is largest for whichever axis currently has fewer units ($r=2 < n=3$ in this example), so all else equal, invest in whichever axis is thinner. Then divide each $\Delta A$ by its real cost: $\text{ROI} = \Delta A / \text{cost}$, and since a new replica in an existing region is almost always cheaper than standing up an entire new region (new infrastructure, new operational surface, cross-region egress), the raw availability numbers alone usually favor adding replicas once you already have two or more regions, and the case for a third region has to come from something the pure independence model doesn't capture: correlated failure risk.

Trade-offs & pitfalls

The entire model rests on an independence assumption that gets weaker as you add replicas within the same region and stays strongest across regions: two replicas in the same region share a power grid, a network fabric, and often a control plane, so a regional outage takes both of them down together in a way the $(1-a)^n$ formula doesn't account for, since that formula assumes each replica fails independently of the others. This is precisely why real systems keep investing in additional regions even when the pure math above says another same-region replica is cheaper per nine: the region-diversity investment is buying protection against a correlated-failure mode the independence model is structurally blind to, not buying a bigger $\Delta A$ number on paper. A second pitfall is applying the series-multiplication rule to a system that has hidden shared dependencies across "independent" series stages, for example a DNS provider or a certificate authority that every stage secretly relies on; multiplying stage availabilities together understates the real risk whenever a single upstream failure can take out multiple "independent" stages simultaneously, which is exactly the class of failure that showed up in the largest real-world multi-service outages. The practical fix for both pitfalls is the same: use the independence-based formula as the starting estimate and a genuine planning tool, but treat any two components sharing physical infrastructure, a network path, or an operational dependency as a single correlated unit for the purposes of the model, not as two separate multiplicands.

CPU vs memory trade-offs balance time (latency, throughput) against space (working set, OOM risk). CPU-optimized code reduces computation by using more memory (caches, precomputed tables); memory-optimized code reduces footprint at cost of extra computation (on-the-fly recomputation, streaming).

Example (realistic for an SRE): a log-processing microservice currently loads hourly log batches into memory, indexes them for fast queries (high memory). To cut memory on small nodes, switch to a streaming approach: parse logs on demand and recompute indexes per query or use bloom filters + partial scans. This increases CPU per query (reparsing, more IO/CPU) but lowers resident memory and reduces OOMs or need to scale to larger instance types.

How to measure if the trade makes sense:

• Define SLOs: request latency p95, throughput, error rate, and cost/SLO for memory usage (OOM frequency).
• Benchmarks: run load tests comparing baseline vs streaming under representative workloads; measure CPU utilization, latency percentiles, GC churn, and memory RSS.
• Cost analysis: compare instance cost (fewer larger machines vs more smaller ones) and autoscaling behavior.
• Monitoring in canary: deploy to small % traffic with enhanced metrics and alerts on latency, CPU saturation, and tail latencies.
  Decision criteria: accept trade if SLOs remain met (p95 within limit), error budget impact is acceptable, and total cost (infrastructure + ops risk) decreases or becomes safer (fewer OOM incidents). If CPU becomes saturated and causes increased latency or noisy neighbors, revert or use hybrid (cache hot queries, evict policies).

Direct answer

Decide upfront, before launch, which leading signals would tell you the initiative is not working, review them on a fixed cadence, and treat money or time already spent as irrelevant to the go or no-go call. The only comparison that matters is the remaining cost to finish against the remaining expected benefit, discounted for how confident you actually are in that benefit.

Structured elaboration

1. Signals to watch: adoption or usage trending flat or down despite active promotion, the cost or effort to maintain the thing trending up while the benefit stays flat, the original problem it was meant to solve no longer being a real priority, or repeated 'just one more milestone' slips with no new information changing the estimate.
2. Quantify sunk versus future cost: sunk cost is a fact about the past and should never affect the decision. The decision compares only the remaining cost to finish against the remaining expected benefit.
3. Set the decision rule in advance, for example 'if adoption is not at a specific level by a specific week, we reassess,' so the call is not made emotionally in the middle of a bad week.
4. Communicate clearly: name the original goal, what is now known that was not known at launch, and the decision itself, to everyone who was told this initiative was happening.
5. Reassign ownership: hand off anything that still has value (partial infrastructure, documented learnings) to whichever team is best placed to use it, and be explicit about what is fully stopped versus paused pending different conditions.

Worked example

An engineer launches a self-service monitoring platform expected to save the team roughly 10 hours a week once fully adopted. Three months in, 40 hours have already gone into building it, and adoption sits at 2 of 8 teams. Finishing the remaining rollout is estimated at another 30 hours. The correct comparison ignores the 40 hours already spent and weighs only the remaining 30 hours of work against a benefit that, given the current adoption trend of 2 of 8 teams after three months, looks unlikely to materialize soon. The engineer pauses new build work, ships what already exists to the 2 teams using it, and reassigns the remaining backlog items to whichever team most needs them next quarter, rather than pushing the extra 30 hours to chase full adoption.

Trade-offs and pitfalls

The clearest pitfall is counting the 40 sunk hours as a reason to 'finish what we started,' which is the sunk-cost fallacy in its plainest form. A second is stopping quietly without naming the decision out loud, which erodes trust the next time this person proposes something new. A third is reacting to a single bad week instead of a sustained trend; a pre-agreed decision rule protects against both overreacting to noise and escalating commitment past the point the numbers support.

Direct answer

This is a deterministic-cause outage: the fix is to restore traffic through a path that has a valid certificate (failover to a standby LB, hot-load a backup cert, or temporarily terminate TLS elsewhere) as fast as possible, communicate on a fixed, predictable cadence while that happens, and then treat the root cause as a monitoring and automation gap rather than a one-time human mistake, since a cert that reached expiry without anyone acting means the process that was supposed to catch it failed silently well before the outage.

Immediate triage (first ~15 minutes)

• Confirm scope and cause: check LB logs for TLS handshake failures and the certificate's expiry timestamp to distinguish a cert issue from an application or network issue reporting similar symptoms (global 503s can also come from a bad deploy or DNS problem).
• Freeze churn: pause any in-flight deploys or automated config pushes so the incident isn't complicated by unrelated changes landing mid-triage.
• Pull in the right people: notify infra/network/security on-call and declare an incident commander if this is genuinely global impact.

Short-term mitigation

• Fastest path: if a hot-standby LB or path with a valid certificate exists, shift traffic to it (DNS weight change or traffic steering) rather than trying to fix the primary path under pressure.
• If the LB supports multiple certs or hot-reload, load a previously-issued backup/rollover certificate if one exists.
• If neither is available, use automated ACME (Automatic Certificate Management Environment, the protocol that lets software request and renew certificates without a human) / CA (certificate authority, the trusted party that issues certificates) tooling to issue a short-lived emergency certificate and install it, prioritizing the automated path over manual cert generation to reduce the chance of a second mistake under time pressure.
• As a last resort only, and only if policy allows, terminate TLS at a CDN or edge proxy that already has a valid cert and forward over a private/trusted link to the backend.
• Validate before declaring resolved: run synthetic TLS handshakes and smoke tests against key endpoints, not just "the dashboard looks green."

Stakeholder and customer communication

• Acknowledge within the first ~10 minutes: post to the status page and internal channels with scope, known impact, and a time for the next update, even if the next update is "still investigating."
• Update on a fixed cadence (every 15-30 minutes) regardless of whether there's new information, since silence reads worse than "no change yet" during a global outage.
• Close the loop once restored with an accurate timeline and a plain-language root cause, then follow with a fuller postmortem summary to execs, product, and customer-facing teams so they can answer customer questions consistently.

Long-term remediation and process changes

• Automate the full certificate lifecycle (issuance and renewal) end to end so a human is never the trigger for routine renewal; the emergency manual path should exist only as a fallback, not the primary mechanism.
• Alert on the action, not just the deadline: a single "certificate expires in N days" alert is not enough, because it doesn't distinguish "renewal will happen automatically before then" from "renewal has already been silently failing." A stronger design pages a human if both conditions hold: fewer than a threshold of days remain, and there has been no successful renewal within the automation's normal cycle. For example, with a 90-day certificate and automation that renews at the 30-days-remaining mark, an escalation alert at 14 days remaining plus "no successful renewal event logged in the last 16 days" catches a silently-failing automation with real margin before it becomes a repeat of this incident, rather than firing on the same schedule the (broken) automation was supposed to act on.
• Rehearse the failure: run a game day where the cert automation is deliberately disabled and confirm the escalation alert actually fires and the manual fallback actually works, rather than assuming it does.
• Track the postmortem to closure with named owners and deadlines for the automation and alerting changes, not just the incident write-up.

Trade-offs and pitfalls

• A pure "days until expiry" alert is the common mistake: it looks reasonable until you realize it fires on the same cadence the automation was already supposed to satisfy, so if the automation silently breaks right after a renewal, the alert doesn't add meaningfully more warning time than the automation's own schedule did. Tying the alert to "no successful renewal observed" closes that gap.
• Emergency manual certificate issuance under time pressure is itself a risk (wrong domain, wrong chain, wrong key usage); preferring automated emergency issuance over a fully manual process reduces that risk even during an incident.
• Communicating on a fixed cadence even with "no update" costs credibility less than going silent, but it does require someone dedicated to comms so the person fixing the issue isn't also the one writing status updates.

Direct answer

For security-sensitive decisions like authorization, default to fail closed, deny the action when the check can't complete, because the cost of a false negative (briefly blocking a legitimate user) is almost always smaller than the cost of a false positive (an unauthorized action succeeding). The exception is when failing closed itself creates a worse safety or availability problem, which is a deliberate, domain-specific judgment call, not a blanket rule.

Structured elaboration

• Default posture: authorization and permission checks fail closed. If the permission service times out, deny the request with a clear error; don't silently treat "unknown" as "allowed."
• Where fail-open is sometimes deliberate: low-stakes, availability-critical paths where an outage of the check itself would be worse than the risk it guards against, and the exposure window is bounded and monitored.
• Make the decision explicit and documented per check, not an accident of how the code happens to be written; a try/catch that swallows the exception and falls through is an accidental fail-open.
• Instrument it: a fail-closed denial caused by an internal error should be logged and alerted distinctly from a legitimate permission denial, so an outage in the auth path shows up as an incident, not just a spike in 403s.

Worked example

An admin panel calls a permissions service to check if a user can delete a record. The service times out. If the code reads let allowed = true; try { allowed = permissionCheck(); } catch (e) { logger.warn(e); } followed by if (allowed) { allowDelete(); }, the caught exception is logged but never rethrown, so allowed is left at its optimistic default of true and the timeout results in the delete going through anyway, an accidental fail-open hiding inside code that looks defensive because it has a try/catch. The fix flips the default and the exception path: let allowed = false; try { allowed = permissionCheck(); } catch (e) { return deny(); }, so any failure to get a definitive answer denies the action.

Trade-offs and pitfalls

Fail-closed can turn a partial outage of a dependency into a full outage of everything gated behind it, so it needs to be paired with making that dependency itself highly available (caching the last known-good permission, short timeouts, circuit breakers) rather than accepting cascading denial as the cost of security. Fail-open, chosen without discussion, is the classic way an authorization bug ships silently, because everything still "works" in testing.

What the interviewer probes next

Whether the candidate can articulate the asymmetry between a false allow and a false deny, rather than reciting "fail closed is always right," and whether they'd catch an accidental fail-open hiding inside ordinary-looking exception handling.