Apple Site Reliability Engineer (Mid-Level) Interview Preparation Guide 2026

Direct answer

At 500 hosts, the real question isn't 'which tool is technically capable' (both are), it's which one you want to own the operational complexity of: inventory management, secrets distribution, and failure isolation, versus building and maintaining that yourself.

Scalability and inventory management

Ansible's inventory system (static files, dynamic inventory scripts/plugins for cloud providers) is a mature, well-tested solution to 'which hosts, grouped how, with what per-group variables' -- a custom concurrent.futures + paramiko runner has to build this from scratch, and inventory management is a deceptively large surface (host groups, per-host overrides, dynamic discovery) to get right. At 500 hosts specifically, both approaches can technically execute concurrently at that scale; the difference is how much of the surrounding machinery (retry-per-host, connection pooling, structured per-host result reporting) you get for free versus have to build.

Secrets distribution

Ansible Vault (or integration with an external secrets manager via lookup plugins) has existing, audited patterns for distributing credentials to a fleet without them landing in plaintext on every target host. A custom runner needs to solve this itself -- likely reusing the secrets-handling patterns discussed elsewhere in this topic (short-lived credentials fetched per-connection rather than distributed and stored), which is very doable but is additional design and implementation surface Ansible gives you largely for free.

Retry behavior and idempotency

Ansible's module system is BUILT around idempotency as a first-class concept -- most built-in modules check current state before acting and report 'changed' vs 'ok' accordingly, which is exactly the idempotency discipline this whole topic argues for, already baked into the tool. A custom runner running an arbitrary script over SSH has NO such guarantee -- idempotency (or the lack of it) is entirely a property of whatever script you're running, and the runner itself only adds retry-on-connection-failure, not retry-safety for the underlying operation.

When the custom runner is still the right call

Despite Ansible's advantages above, a custom concurrent.futures + paramiko runner is justified when: the operation is a one-off, tightly-scoped action that doesn't fit Ansible's declarative module model well; you need custom batching/failure-isolation logic Ansible's default execution strategy doesn't offer out of the box (e.g., 'stop launching new batches immediately if failure rate exceeds 10%, but let in-flight batches finish'); or the team is already deep in a Python-based automation framework (per the language-choice discussion elsewhere in this topic) and adding Ansible as a second tool has its own onboarding/maintenance cost that isn't worth it for this one task.

Batching and failure isolation with a custom runner

If building custom: batch hosts into fixed-size groups (e.g. 50 at a time) submitted to a bounded ThreadPoolExecutor, track per-host success/failure independently (one host's connection failure must never abort or block the others), and implement a circuit-breaker-style check between batches -- if failure rate in the last batch crosses a threshold, pause and surface to an operator rather than blindly continuing through all 500 hosts and only discovering a systemic problem (bad SSH key rotation, a network partition) after every host has already failed.

Trade-offs and pitfalls

The most common mistake building a custom runner is treating the FIRST version as sufficient to reach production without the retry, secrets, and failure-isolation depth Ansible already has -- what starts as 'this is simpler than learning Ansible' quietly grows into reimplementing large parts of it, worse and less tested, exactly the scope-creep risk the custom-vs-mature-tools trade-off discussed elsewhere in this topic warns about.

Direct answer

Many "half-open" TCP sessions at a load balancer, combined with client-perceived slowness, points at connections that started but never fully progressed to normal data exchange; narrow down whether this is client-side slow start, backend slowness, an idle-timeout mismatch, a keepalive misconfiguration, or a NAT/translation issue by checking where in each connection's lifecycle it's actually getting stuck, using concrete commands and log fields rather than inference alone.

Structured elaboration

1. Define "half-open" precisely for this specific load balancer, and check its own connection-state metrics: most LBs expose a per-backend or per-listener metric or log field for connection state (e.g., an AWS NLB/ALB's active_connection_count vs new_connection_count, an HAProxy show stat field, or an nginx stub_status/access-log entry showing request duration); pull that specific field before investigating further.
2. Check backend response times directly with a command, not just a general check: curl -w '%{time_connect} %{time_starttransfer} %{time_total}\n' -o /dev/null -s http://<backend-ip>:<port>/ run directly against a backend, bypassing the LB, isolates whether the BACKEND itself is slow to respond independent of anything the LB is doing.
3. Check idle-timeout settings on the LB versus on the backend and any intermediate device, by reading the actual configured values: for example, on an AWS ALB, aws elbv2 describe-load-balancer-attributes --load-balancer-arn <arn> | grep idle_timeout; on HAProxy, the timeout client/timeout server directives in haproxy.cfg; on a Linux backend or intervening firewall, sysctl net.netfilter.nf_conntrack_tcp_timeout_established (for a stateful Linux firewall/NAT device) or the equivalent vendor CLI. A MISMATCH (the load balancer's idle timeout longer than a stateful firewall's) can leave the LB believing a connection is still valid long after an intermediate device has silently dropped its state for it.
4. Check TCP keepalive settings explicitly, on both the OS and the application/LB layer: ss -tno shows the keepalive timer value actually in effect for existing sockets (the timer:(keepalive,...) field); sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes shows the host's OS-level keepalive defaults. If the LB or backend relies on OS keepalive to detect and reap dead connections, but the configured keepalive interval is LONGER than an intermediate device's own silent idle-timeout, the LB/backend won't notice a connection is dead until long after the client has given up waiting, producing exactly this half-open-with-slowness symptom; a keepalive interval shorter than the shortest idle-timeout anywhere on the path avoids this.
5. Check for NAT translation issues specifically: if client connections pass through a NAT device, conntrack -L | grep <client-ip> (Linux conntrack) shows the NAT's own tracked state and its remaining timeout for a specific flow directly, which you can compare against what the LB believes about the same connection.
6. Simulate deliberately to isolate, with an actual command: nc -v <lb-ip> <port> to open a raw TCP connection through the same path and leave it idle, timing (with date before and after, or simply watching when the connection drops) exactly how long each device (LB, any NAT, the backend) tolerates it before silently closing; running the same test with tcpdump -n host <lb-ip> and port <port> capturing simultaneously shows precisely which side sends the FIN/RST first, or whether the connection is simply dropped with no packet at all (a silent, stateful-device timeout, not a graceful close).

Worked example

aws elbv2 describe-load-balancer-attributes shows the ALB's idle timeout configured for 300 seconds. sysctl net.netfilter.nf_conntrack_tcp_timeout_established on an intervening stateful firewall shows only 60 seconds. nc -v through the same path, held idle and captured with tcpdump, shows the connection silently stops passing traffic at almost exactly 60 seconds, with no FIN or RST visible on either side at the client's capture point, confirming the firewall drops its state silently while the ALB, unaware, continues counting the connection as established for up to 300 seconds. ss -tno on the backend shows its OS keepalive interval set to the Linux default of 7200 seconds, far too long to ever detect this. Aligning the ALB's idle timeout to be shorter than the firewall's 60-second timeout, and/or reducing tcp_keepalive_time on the backend to something well under 60 seconds, resolves the symptom.

Trade-offs & pitfalls

It's easy to focus entirely on the load balancer's own configuration and metrics without checking whether an INTERMEDIATE device has a shorter, silently enforced timeout, or whether keepalive is even configured aggressively enough to matter; pull the actual configured values (via the commands above) on every device in the path rather than assuming documented defaults apply. A hands-on nc test captured with tcpdump, rather than trusting each device's stated timeout, is what actually reveals a mismatch like this in practice, since a silently-dropped connection produces no error the LB or client would otherwise see.

A correlation ID is a single identifier generated at the edge of a request and propagated through every hop, so logs, metrics, and traces from otherwise-independent services can be joined into one end-to-end picture.

Design

1. Generate once, at the edge (the first service to see the request), and reject or ignore any incoming ID from an untrusted external source unless you specifically want to correlate with an external partner's identifier.
2. Propagate via headers (e.g. X-Request-Id or the W3C traceparent format) on every downstream call, and via message baggage for async work (queue messages, background jobs) so the ID survives a hop that isn't a synchronous HTTP call.
3. Attach it to every log line and trace span emitted while handling that request, not just at the top-level handler.
4. For third-party/legacy components you cannot instrument, enrich logs at your own ingress/egress boundary around that component (the request going in, the response coming out) so you can at least bound how much time was spent inside the black box, even without visibility inside it.
5. Query time: given an ID, a single query across the log/trace store reconstructs the full path, ordered by timestamp, across every service that touched the request.

Trade-offs and pitfalls

High-cardinality IDs are cheap to log but expensive to index if you try to make every field high-cardinality; keep the correlation ID itself indexed and unique, but avoid also indexing every other high-cardinality field (like raw user input) at the same tier, or storage cost balloons. Sampling (only tracing 1% of requests in detail) is fine for aggregate analysis but means a specific customer's specific failing request may not have full trace data; the mitigation is to force full capture whenever an error is detected on that request, regardless of the sampling decision made at the start.

Direct answer

Tier storage by how recently and how urgently data is queried, not by a single uniform policy: keep a short hot window fully indexed for fast incident response, a longer warm window with reduced indexing and cheaper storage, and a cold archive that's raw and compressed with only a thin metadata index for lookup. Index only the fields you actually search or alert on, and cut ingested volume before it ever reaches storage (dropping known-noisy events, sampling verbose debug logs, deduplicating repeats) rather than trying to compress your way out of ingesting everything at full fidelity.

Structured elaboration

Tiering by access pattern

• Hot (recent, e.g. the last several days): fully indexed and parsed, on fast storage, because this is the window most incident investigations actually query.
• Warm (weeks to a couple of months): reduced indexing, kept for less-frequent-but-still-plausible investigations, on cheaper storage.
• Cold (long tail out to the retention limit): raw, compressed, minimally indexed (just enough metadata, e.g. timestamp/service/id, to locate and rehydrate a specific slice), on the cheapest available storage.

What to index versus what to keep raw
Index only the fields actually used in searches and alerts (service, severity, request id, error code, user id) rather than full free-text message bodies by default. Indexing costs compute and storage roughly proportional to what you index, so indexing everything "just in case" is usually what makes a fully-indexed retention policy expensive in the first place. Keep the raw, unindexed payload in cheap storage so a rare deep investigation can still re-parse it on demand, even though day-to-day queries never touch it.

Reducing ingested volume before it lands anywhere

• Drop known-noisy, low-value events at the source: health-check pings, synthetic monitoring traffic that's already captured elsewhere.
• Sample verbose debug-level logs outside of active incidents, while retaining 100% of error/warn-level events; a debug log's value degrades fast once nothing is actively wrong, but an error's value doesn't.
• Deduplicate: collapse many identical repeated events within a short window into one record with a count and a first/last timestamp, rather than storing each occurrence individually.

A budget-driven decision framework
Rather than picking retention, sampling, and rollup settings independently and hoping the bill comes out reasonable, work backward from a fixed observability budget: classify data by purpose (alerting needs short-window full fidelity; long-term trend analysis can tolerate rollups; compliance may fix a minimum retention regardless of cost), then allocate the budget so the fidelity you actually rely on (recent, alert-relevant data) is protected first, and lower-value long-tail data absorbs the compression.

Worked example

Compare two policies for a service ingesting 500 GB/day of raw logs, over a 90-day retention requirement: "fully indexed for all 90 days" versus a tiered plan.

Fully indexed, 90 days, no tiering (assume indexing overhead adds 30% on top of the compressed size, and compression brings raw data to 25% of its original size, both stated assumptions used consistently below):

Tiered plan (hot: 7 days, fully indexed, same 0.25 compression and 1.30 index overhead; warm: days 8-30, still compressed 0.25 but only 30% of it indexed; cold: days 31-90, raw compressed only, no index, using a better 0.10 compression ratio typical of infrequently-touched archival data):

Under these stated assumptions, the tiered plan stores about half the volume of the fully-indexed policy (7,271 GB vs 14,625 GB) for the same 90-day retention window, purely from indexing less of the older data and compressing the coldest tier harder; it doesn't require dropping a single day of retention. The actual ratio in a real system depends entirely on the true compression and index-overhead numbers for the specific backend in use, which is why this example states its assumptions rather than presenting the savings as a general industry figure.

Trade-offs & pitfalls

• Rehydrating cold data for an investigation has a retrieval delay; make sure whoever owns compliance sign-off on the retention policy accepts that trade explicitly, rather than discovering it during an actual audit.
• Sampling debug logs outside incidents means you can't always sample-up retroactively for an incident that's already happened; if incident detection is delayed, the fine-grained debug data for the affected window may already be gone. Pair aggressive debug sampling with a short full-fidelity buffer (e.g., last 24-48 hours at 100%) to cover that gap.
• Deduplication that collapses "identical" events too aggressively can hide a subtle change (an error message that's 99% identical but has a materially different stack trace) if the dedup key isn't specific enough.
• Under-indexing to save cost can make routine investigations slower even when they're within the retention window; the goal is indexing the fields actually queried, not indexing as little as possible.

Direct answer: Since CPU and memory look normal, the problem is probably not resource saturation on your own host; it's more likely a downstream dependency slowing down, threads piling up waiting on something (which costs almost no CPU while blocked), or a specific change like a config or certificate issue. The first 30 minutes should check exactly those things in order, starting with logs and ending with runtime state.

Structured elaboration:

• 0 to 5 minutes, logs. Look for a common error or exception pattern behind the 5xx spike, and check whether it's concentrated on one endpoint or spread across all of them.
• 5 to 10 minutes, external dependency checks. Look at latency and error-rate dashboards for downstream APIs and the database directly, not just your own service's view of them. Check connection-pool metrics (checked-out connections, wait-queue depth); requests piling up waiting for a pool resource explains rising latency and errors while CPU stays flat, since blocked threads aren't burning CPU cycles.
• 10 to 20 minutes, runtime-state inspection. Take a thread dump (a snapshot of what every thread is currently doing). If a large fraction of threads are BLOCKED or WAITING at the same call site, that's the smoking gun, thread-pool exhaustion behind a slow or hung downstream call, and it explains the symptom without CPU ever needing to be elevated.
• 20 to 30 minutes, recent changes. Check deploys, config pushes, dependency version bumps, and expiring certificates or credentials around the exact start time of the spike; a routine certificate rotation is a classic cause of "everything was fine and then suddenly errors."
• Mitigation once you have a lead: if a downstream dependency is degraded, add a tight timeout and circuit breaker (a safeguard that stops sending requests to a struggling dependency once failures cross a threshold, so it gets a chance to recover instead of being hit further) so requests fail fast instead of holding a thread indefinitely; if the pool itself is undersized for this failure mode, temporarily raise its size or shed load; if it's a deploy, config, or certificate issue, roll back or fix the specific item directly.

Worked example: A thread dump shows 180 of 200 available worker threads BLOCKED in the same stack frame, waiting on a socket read to a payments service. That payments service's own latency dashboard shows its p99 (the 99th percentile latency: the slowest 1% of requests) climbed from 100ms to about 8 seconds starting at the same minute the 5xx spike began, before its own team's alerting had fired. CPU and memory looked normal because those 180 threads are idle-blocked, not computing, they're simply occupying slots so new requests can't get a thread and start timing out or erroring, which explains both symptoms (high error rate and flat CPU) at once. The immediate mitigation is a tight timeout, around 500ms, plus a circuit breaker on calls to the payments service, so blocked threads free up quickly and new requests fail fast with a clear error instead of exhausting the pool, while paging the payments-service team in parallel.

Trade-offs and pitfalls: spending the first 20 minutes staring at CPU and memory dashboards "because that's what we always check first" wastes exactly the window where a thread dump would have given the answer immediately. A fail-fast circuit breaker converts "slow and eventually erroring" into "fast and erroring immediately" for affected requests, which is better for pool health but is still a real user-facing error, so it should be communicated as a mitigation, not framed as a fix.

Two different logging models

Plain-text logs under /var/log are files written by individual daemons (or by rsyslog/syslog-ng aggregating messages from several daemons) using whatever format each program chooses; they persist as ordinary files, survive reboots by definition, and are searched with generic text tools (grep, awk, zgrep on rotated copies). journalctl queries journald, systemd's structured logging daemon, which stores entries (optionally) in a binary, indexed format with rich metadata attached to every entry (unit name, PID, UID, boot ID, priority level, and more) captured automatically at write time, not just whatever text the program chose to print.

When to prefer one over the other

Prefer journalctl when the question is about a specific systemd-managed unit (which service, across how many restarts, in what time window), because the structured metadata makes exactly that kind of filtering fast and precise without needing to know or guess a log filename or format. Prefer plain /var/log files (or an application's own dedicated log) when the software isn't managed by systemd at all, when you need a log that will survive journald's storage limits or a service like rsyslog shipping copies off-host for long-term retention (journald's in-memory or on-disk ring buffer is not unlimited and can be configured to rotate aggressively), or when a tool downstream of the logs (a log-shipping agent, a legacy parser) only understands plain text files, not the journal's binary format.

Extracting a single service's logs across boots and time ranges

journalctl -u nginx.service                      # everything logged for this unit, across all retained boots
journalctl -u nginx.service -b -1                 # only the previous boot
journalctl -u nginx.service --since "2026-09-20" --until "2026-09-21 06:00"
journalctl -u nginx.service -f                    # follow new entries live, like tail -f
journalctl -u nginx.service -p err                # only entries at error priority or worse
journalctl --list-boots                           # see which boot IDs/indices are available at all

-u <unit> filters to exactly one systemd unit regardless of which boot logged it, which is the core advantage over grepping plain files: a plain-text approach to "this service's logs across the last three boots" would require locating and concatenating however many rotated files happen to still exist, whereas journalctl -u <unit> reads across boot boundaries transparently as long as journald has retained that history. --since/--until accept natural time expressions ("1 hour ago", "yesterday", explicit timestamps), which is considerably more convenient than reconstructing a time window from log-line timestamps by hand.

Trade-offs and pitfalls

• journalctl's retained history depends entirely on journald's configured storage limits (SystemMaxUse=, time-based retention, or purely in-memory with no persistence at all on some minimal configurations); a host configured for volatile-only journal storage loses all journal history on reboot, which plain files written to persistent storage would not.
• journalctl -u <unit> only shows what that unit logged to stdout/stderr (captured automatically by systemd) or explicitly sent via the journal API; if the same application also writes its own separate log file (common with databases and web servers), the two views can disagree, and neither alone is guaranteed to be the complete picture.
• The journal's binary format means it cannot be searched with plain grep directly on the underlying files; every query must go through journalctl (or a library that understands the format), which is a real workflow difference for anyone used to piping /var/log files into arbitrary text tools.

What mmap does inside the kernel. Mapping a file does not read it. The kernel creates a VMA (virtual memory area, a record "addresses X to Y map file F from offset O with these permissions") and leaves the page-table entries empty. No disk I/O happens yet. The file's data is served through the page cache (the kernel's in-RAM copy of file contents), which is the same cache read() and write() use.

First access. The first touch of a mapped page finds no page-table entry, so the CPU raises a page fault (an exception that transfers to the kernel). The fault handler finds the VMA, then asks the file's page cache for that page:

• Page already in the page cache (minor fault): the kernel just installs a page-table entry. No disk access, microseconds or less.
• Not cached (major fault): the kernel issues a read to storage and puts the faulting thread to sleep until it completes, which is milliseconds on a spinning disk or hundreds of microseconds on an SSD.
  Then the instruction restarts and succeeds. The kernel may also map several neighbouring cached pages at once. This is called fault-around: on a read fault the kernel maps up to a window of neighbouring pages that are already in the page cache (the window defaults to 64 KiB, which is 16 pages of 4 KiB, per mm/memory.c in the kernel source). In the demo below, reading all 64 pages of a mapped file increases the minor-fault count by only 4 (47 to 51 in the printed run), not 64, which is 64 pages / 16 pages per fault (output below).

Private versus shared.

• MAP_SHARED: all processes map the same page-cache pages. A write modifies the cache page, marks it dirty, and background writeback (or msync/fsync) later reaches the file. Other processes and read() see the change immediately.
• MAP_PRIVATE: reads share the page-cache page, but the first write triggers copy-on-write: the kernel copies the page into private anonymous memory and maps the copy writable. The file and other processes never see the change.

The demo (Linux aarch64 container; the 64-page file is created by the program). It also triggers SIGBUS, the signal the kernel sends a process when it touches a mapped page that the file can no longer supply, for example a page past the end of a file that was truncated after mapping.

#define _GNU_SOURCE
#include <fcntl.h>
#include <setjmp.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <unistd.h>

static sigjmp_buf env;
static void on_bus(int sig) { (void)sig; siglongjmp(env, 1); }

static void faults(const char *label) {
    struct rusage r;
    getrusage(RUSAGE_SELF, &r);
    printf("%-34s minflt=%ld majflt=%ld\n", label, r.ru_minflt, r.ru_majflt);
}

int main(void) {
    const size_t PG = 4096, N = 64;
    faults("at start of main");
    int fd = open("data.bin", O_RDWR | O_CREAT | O_TRUNC, 0600);
    char page[4096];
    memset(page, 'A', sizeof page);
    for (size_t i = 0; i < N; i++) write(fd, page, PG);
    fsync(fd);
    faults("after creating the file");

    char *priv = mmap(NULL, N * PG, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
    char *shr  = mmap(NULL, N * PG, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    faults("after mmap (no access yet)");
    volatile char c = 0;
    for (size_t i = 0; i < N; i++) c += shr[i * PG];
    faults("after reading 64 shared pages");
    priv[0] = 'P';           /* copy-on-write of a private page */
    shr[PG] = 'S';           /* write through to the page cache */
    char b[2];
    pread(fd, b, 1, 0);  printf("file byte 0 via pread: %c (private write)\n", b[0]);
    pread(fd, b, 1, PG); printf("file byte 4096 via pread: %c (shared write)\n", b[0]);

    signal(SIGBUS, on_bus);
    ftruncate(fd, 8 * PG);   /* another process shrinks the file */
    if (sigsetjmp(env, 1) == 0) {
        c += shr[3 * PG];    printf("page 3 (still in file): ok\n");
        c += shr[40 * PG];   printf("page 40: ok (unexpected)\n");
    } else {
        printf("page 40 (past new end of file): SIGBUS\n");
    }
    return 0;
}

at start of main                   minflt=38 majflt=0
after creating the file            minflt=47 majflt=0
after mmap (no access yet)         minflt=47 majflt=0
after reading 64 shared pages      minflt=51 majflt=0
file byte 0 via pread: A (private write)
file byte 4096 via pread: S (shared write)
page 3 (still in file): ok
page 40 (past new end of file): SIGBUS

How to read the demo: faults() prints how many minor and major page faults the process has taken so far (getrusage); a minor fault is one served from memory, a major fault one that needed disk I/O. The first line, minflt=38 majflt=0, is the baseline at the start of main: faults already taken while the program was being loaded (the dynamic loader and libc touching their own pages), before any mapping exists. The absolute baseline differs between runs and environments (it is larger when the program is started through a shell wrapper, and majflt can be 1 on a cold start when the program's own pages must be read from disk), so compare differences between lines, not absolute values. The jump from 38 to 47 while creating the file comes from the first printf and the file-writing calls touching new pages for the first time. mmap itself adds no faults (47 to 47), which shows that mapping reads nothing. The loop then reads one byte from each of the 64 pages. volatile char c stops the compiler from deleting loads whose result is never used. pread reads the file through the page cache with a normal system call, so it shows what the two writes did: the private write (priv[0]) was a copy-on-write copy visible only to this process, while the shared write (shr[PG]) changed the page-cache page that pread reads. For the truncation test, ftruncate(fd, 8 * PG) stands in for another process shrinking the file to 8 pages; the read of page 3 still works, the read of page 40 raises SIGBUS, the handler runs siglongjmp, and sigsetjmp (which returns 0 the first time and 1 after the jump back) takes the else branch so the program can report it instead of dying. The majflt count does not change through the mapping and reads.
The private write to byte 0 left the file at A; the shared write to byte 4096 reached it as S. (The program had just written the file, so its pages were already in the page cache, which is why reading them caused only minor faults. Exact counts vary by a fault or two between runs.)

Failure modes a service must expect.

1. Page-in stalls. Any memory access can block on disk, so a latency-sensitive thread (an event loop, a request handler holding a lock) stalls invisibly, with no system call in the profile. Under memory pressure clean pages are evicted, so a "fast" mapped file turns slow. Mitigations: madvise(MADV_WILLNEED) (a hint asking the kernel to start reading a range into the page cache now) or MAP_POPULATE (an mmap flag that pre-faults the whole mapping up front) to prefetch, mlock (pin pages in RAM so they cannot be evicted) for a small hot set, and watching major faults (majflt in /proc/<pid>/stat or perf stat -e major-faults).
2. File truncation. If anyone shrinks the file while it is mapped, touching a page beyond the new end delivers SIGBUS, as the demo shows (and the mmap(2) man page lists). There is no error return to check, because the failure happens in a plain load or store. An I/O error while paging in also produces SIGBUS. Defences: a SIGBUS handler with sigsetjmp/siglongjmp around risky accesses (as in the demo, which is fragile in a multi-threaded service), only mapping files nobody else can truncate (a file you own and lock, or a memfd, an anonymous in-memory file, with F_SEAL_SHRINK, a seal that makes shrinking it impossible), or using pread when you need clean error codes.
3. Unmap races. If one thread calls munmap while another still uses the pointer, the second thread gets SIGSEGV, or, worse, the address range is reused by a new mapping and it silently reads or corrupts unrelated data. Tie the unmap to a reference count, or to a quiescent point, meaning a moment when no thread can still be holding the pointer (an RCU-style epoch scheme waits until every thread has passed such a point; a shutdown phase is another).
4. Durability and visibility. A write to a MAP_SHARED page is not on disk when the store instruction returns; call msync(MS_SYNC) (flush the mapped range's dirty pages to the file and wait) or fsync where it matters. Writeback, the kernel's background writing of dirty pages to disk, can also make shutdown or checkpoint latency spiky.

Recommendation: use mmap for read-mostly files that you control and can prefetch, and treat truncation and stalls as real failure paths. Use read/write with your own buffer when you need predictable latency, error codes or exact durability.

Direct answer: Availability is the fraction of time a system is usable, and "N nines" is shorthand for how close that fraction is to 100%. Going from 99.9% to 99.99% to 99.999% shrinks allowed downtime by roughly 10x at each step, and each step also costs roughly an order of magnitude more in engineering and infrastructure, because you're eliminating an entire category of failure (single-host, then single-zone, then single-region) rather than just adding more of the same redundancy.

Structured elaboration

Allowed downtime per year is derived from the availability target directly:

where 8760 is the number of hours in a 365-day year (24 x 365), and per-month downtime uses 730 hours (8760 / 12):

Plugging in each target:

Each jump divides allowed downtime by exactly 10, because each availability target divides $(1-A)$ by 10.

What actually changes architecturally between tiers

• 99.9% -> 99.99%: eliminate single points of failure inside one facility. Multi-AZ deployment, N+1 redundancy (one extra standby unit beyond what's strictly needed to handle normal load, so a single failure doesn't drop capacity below what's required) on stateful components (load balancers, databases with a standby replica), automated health-check-driven failover, and a real on-call rotation with paging. Most of the gain here comes from removing manual recovery steps: a human restarting a service takes minutes and that alone can burn the entire four-nines monthly budget.
• 99.99% -> 99.999%: eliminate the facility (zone or region) itself as a single point of failure. Multi-region active-active or hot standby (a fully-running backup kept ready to take over instantly, unlike a cold standby that would first need to be started up and warmed), automated cross-region failover (not human-triggered), synchronous or tightly-bounded-lag replication for the data that must survive a region loss, and rigorous testing of the failover path itself (chaos drills: deliberately triggering the failover in a controlled test so a broken failover path is discovered on a Tuesday afternoon, not during a real outage), because at this tier the failover mechanism is now a bigger risk to availability than the failures it's protecting against.
• Beyond 99.999%, the limiting factor usually isn't infrastructure, it's deployment risk (bad releases) and dependency risk (a vendor or DNS provider you don't control), so the remaining budget goes to progressive rollouts, fast automated rollback, and reducing the number of hard external dependencies on the critical path.

Worked example: composing a dependency chain

A request that serially depends on a load balancer (99.99%), an app tier (99.95%), and a database (99.99%) has a combined availability equal to the product of the individual availabilities, because all three must be up simultaneously:

That's roughly 99.93%, worse than any single component, which is why a system built entirely from 99.99%-rated pieces chained together does not automatically deliver 99.99% end to end. Adding a redundant standby database (parallel, either one being up is sufficient) with independent 99.99% availability changes only that term:

so the pair is effectively always up, and the chain's availability is then bounded by the weakest remaining serial link (the app tier at 99.95%), not the database.

Trade-offs & pitfalls

• Availability composes multiplicatively across a serial chain and the weakest link dominates: chasing five nines on your database while your app tier sits at three nines is wasted spend.
• Each nine costs disproportionately more: 99.9% to 99.99% is mostly process and automation (cheap-ish); 99.99% to 99.999% usually means paying for a second region and the operational overhead of keeping it truly independent (expensive, and dangerous if the failover path itself is untested).
• Downtime budgets don't distinguish planned from unplanned; a team that spends its whole error budget on deploy-related outages hasn't actually built a more resilient system, just a riskier release process.
• A very common interview trap: treating "99.99% uptime" as a promise about any single request rather than a time-integrated average. A system can meet 99.99% for the year while having a full 50-minute outage in one bad afternoon.

Direct answer

Safe production rollouts via GitOps need the PR (pull request) workflow to add exactly two things beyond a normal code-review PR: an automated, VISIBLE preview of the actual deployment effect (not just the manifest diff) and a graduated approval bar that scales with the target environment, while everything that does NOT need to slow down (linting, unit-level manifest validation, non-production environments generally) should stay genuinely fast, since developer velocity is preserved specifically by NOT applying production's rigor uniformly everywhere.

Structured elaboration

Automation added to the PR workflow. Manifest/schema validation and Helm/Kustomize rendering checks on every PR, fast and free, running regardless of target environment. For a PR touching a PRODUCTION overlay specifically: a rendered-diff comment (adapted from ordinary plan-review practice to Kubernetes manifests, showing the actual resulting object diff, not just the raw YAML text diff) and a policy-as-code evaluation (OPA/Rego) against the rendered manifest.

Approval steps added, graduated by environment. Non-production overlays: standard single-reviewer PR approval, no additional gate. Production overlays: CODEOWNERS-scoped approval from a designated group, PLUS the rendered-diff and policy checks above passing as required status checks; this is where the "safe" requirement concentrates its cost, deliberately, rather than being spread evenly across every environment's PRs.

Preserving developer velocity. The KEY design choice: none of the production-specific rigor applies to a developer's DAY-TO-DAY work in dev/staging overlays, which stay fast (standard review, no extra gates); a developer iterating rapidly in a non-production environment experiences essentially the same speed as before GitOps rigor was added anywhere, and only encounters the heavier bar at the SPECIFIC moment (a production promotion PR) where the extra rigor is actually earning its cost.

Post-merge, reconciliation and rollback. Once a production PR merges, the GitOps controller reconciles, and the SAME rollback mechanism available for any other change (Git revert plus reconciliation) is available if the newly-merged change causes a problem, itself a form of safety that does not add ANY friction to the merge process, since it is available after the fact rather than requiring extra pre-merge steps to guarantee.

Worked example

A concrete PR-workflow shape for a service with dev, staging, and production overlays:

A developer's typical day (iterating in dev, occasionally promoting to staging) experiences the FIRST two rows almost entirely, fast and low-friction; the heavier third row is encountered only at the deliberate moment of promoting to production, which is exactly the point at which the added rigor is actually buying real safety rather than merely adding process overhead to changes that do not need it.

Trade-offs and pitfalls

• Common mistake: applying the SAME heavy gate (rendered-diff review, policy evaluation, elevated approval) uniformly to every environment's PRs "for consistency." This is the single most common way a GitOps adoption ends up SLOWER than the process it replaced without a corresponding safety gain, most of a team's daily PR volume touches non-production environments, and applying production's bar there taxes exactly the changes that need it least.
• A rendered-diff comment that shows raw YAML text differences, rather than the actual resulting Kubernetes object diff, is a common, weaker substitute, the same plan-review-actionability principle applies here as to a Terraform plan: a template change can alter the rendered OUTPUT significantly while the raw template text diff looks small, or vice versa; reviewing the rendered effect, not the template source, is what actually catches what matters.
• CODEOWNERS-scoped production approval only provides real protection if the group's membership is kept current and reviewed periodically; an outdated production-approver group is a structural gap that looks like a safety control while actually being weaker than it appears.
• Post-merge rollback availability (via Git revert plus reconciliation) is a genuine safety net, but it is not a substitute for the pre-merge checks, the same caution about a revert's limits that applies to a Terraform rollback applies here too: a rollback undoes the DECLARED state, but any real-world side effect the bad change already caused (a data write, an external system notified) is not automatically undone by a Git revert alone, which is why the pre-merge gate for production still matters even with a working rollback path available afterward.

Direct answer

A delegation hands a part of the DNS tree to another set of servers: the parent zone publishes NS (name server) records at the zone cut (the point in the name tree where the parent's data ends and the child zone's data begins), naming the servers that are authoritative for the child zone. A glue record is an address record (A or AAAA) that the parent also carries for a name server whose name sits inside the delegated zone itself. It is needed to break a circle: to find ns1.example.parent a resolver must first ask the servers of example.parent, whose only server is ns1.example.parent. Without glue in the parent, a resolver that does not already have the address cached cannot reach the child zone at all.

Delegation and referral

The parent does not answer questions below the cut. It returns a referral: a response with no answer section, the child's NS records in the authority section, and the authoritative-answer (AA) flag not set (the AA flag is a bit in the reply header that a server sets only when it owns the zone the answer comes from, so a referral from the parent does not carry it). The resolver then repeats the question to one of those servers. The child zone's own NS records and address records are the authoritative copy; whatever the parent holds is a pointer.

The example, executed with BIND

The parent zone (called parent in a lab) delegates example.parent to two servers whose names are inside example.parent:

example.parent.     IN NS ns1.example.parent.
example.parent.     IN NS ns2.example.parent.
ns1.example.parent. IN A  192.0.2.1
ns2.example.parent. IN A  192.0.2.2

The last two lines are the glue. Asking the parent's server for a name below the cut without recursion (dig @127.0.0.1 +norec +noall +authority +additional www.example.parent A) returned:

example.parent.         3600  IN  NS  ns1.example.parent.
example.parent.         3600  IN  NS  ns2.example.parent.
ns1.example.parent.     3600  IN  A   192.0.2.1
ns2.example.parent.     3600  IN  A   192.0.2.2

The NS records came back in the authority section (the first two lines: who is responsible) and the glue in the additional section (the last two lines: extra records offered to save a lookup), so the resolver can query 192.0.2.1 immediately.

With the two glue lines removed, named-checkzone (BIND's zone-file checker) still reports the zone as loaded. A program's exit status is the number it returns to the shell, where 0 means success and non-zero means failure, and here it is 0. The run printed:

$ named-checkzone parent /z/parent-noglue.zone ; echo "exit=$?"
zone parent/IN: example.parent/NS 'ns1.example.parent' has no REQUIRED GLUE address records (A or AAAA)
zone parent/IN: example.parent/NS 'ns1.example.parent' (out of zone) has no addresses records (A or AAAA)
zone parent/IN: example.parent/NS 'ns2.example.parent' has no REQUIRED GLUE address records (A or AAAA)
zone parent/IN: example.parent/NS 'ns2.example.parent' (out of zone) has no addresses records (A or AAAA)
zone parent/IN: loaded serial 1
OK
exit=0

The REQUIRED GLUE lines are the ones that matter here; the checker also prints a second, (out of zone) warning per name server, and with the two glue lines restored only those second warnings remain. Every line is a warning and the run ends with OK and exit=0, so a pipeline that checks only the exit status passes a zone whose delegation cannot be followed. The referral for www.example.parent carried the same two NS records, with the header showing AUTHORITY: 2 and ADDITIONAL: 1 (the EDNS option pseudo-record only: EDNS, Extension Mechanisms for DNS, lets a message carry extra options such as a larger size limit, and dig counts that option record in the additional section), so no address records at all. With the glue present the same query showed ADDITIONAL: 3, the two glue records plus that one. The delegation is published and silently broken: that is why a warning-only check is dangerous in a pipeline that treats exit status 0 as success.

What breaks without glue

A resolver with a cold cache gets the referral, has two NS names and no addresses, and must resolve those names, which sends it back to the same parent and the same referral. It cannot make progress and returns SERVFAIL (a server failure response) to the client. Resolvers that already hold the server addresses from earlier lookups keep working until those cached records expire, so the failure looks intermittent and depends on who asks.

When glue is and is not needed

• Needed (in-domain name servers): the NS name is under the delegated zone, as above. RFC 9471 says an authoritative server must include all available glue for such names in a referral, and must set the TC (truncated) flag (a header bit meaning "this reply was cut off to fit the UDP size limit") if they do not fit, so that the client retries over TCP.
• Not needed (out-of-domain name servers): if the NS names are in a different zone, for example ns1.dnsvendor.net, the resolver finds their addresses through that other zone. Parents often still include such addresses as an optimisation, but they are not required.

Operating it

• With a registered domain, the glue is held in the parent (the TLD, top-level domain, registry), and you manage it through the registrar, which calls them host records or glue records. Changing a name server's IP means updating the glue at the registrar as well as the A record in the zone.
• Keep the child's own A records for its name servers identical to the glue. A mismatch sends different resolvers to different addresses.
• Verify with dig +norec @PARENT-SERVER child.zone NS and read the additional section.

Pitfall

Putting a domain's name servers inside the domain is legal and common, but it makes the registrar glue a hard dependency. Pairing one in-domain server with one out-of-domain server from a different provider removes the dependency and also gives provider redundancy.