Apple Site Reliability Engineer Interview Preparation Guide - Junior Level (1-2 Years Experience)

Direct answer

journalctl is the query tool for the systemd journal, the structured binary log store used by most modern Linux distributions alongside (or instead of) flat files in /var/log. The four commands below are standard, documented journalctl usage; they are shown as reference commands rather than a captured run, since this environment has no systemd journal to query against.

Structured elaboration

1) List all boots and show the last boot's ID

journalctl --list-boots
journalctl --list-boots | tail -n 1

--list-boots prints one line per system boot: a relative index (0 is the current boot, -1 the previous one, and so on), the boot ID, and its start/end timestamps. The last line of that output is the most recent boot; its second column is the boot ID you'd pass to journalctl -b <ID>.

2) Show logs for nginx.service filtered to warning and above

journalctl -u nginx.service -p warning

-p (priority) filters by syslog severity. Passing a single level like warning means "this level or anything more severe" (warning, error, critical, alert, emergency), matching "warning and above."

3) Follow logs in real time

journalctl -u nginx.service -f

-f behaves like tail -f: it prints new entries as they're written instead of exiting after the current buffer.

4) Export entries as JSON for downstream parsing

journalctl -u nginx.service -o json

-o json emits one JSON object per line, not json-pretty, which spans multiple lines per entry and is awkward to stream. One-object-per-line is what lets a downstream parser (jq, a Python script, a log shipper) read the output incrementally.

Field significance

• _SYSTEMD_UNIT identifies which systemd unit emitted the entry. On a host running many services, this is what lets you filter (journalctl -u <unit>) or, when reading exported JSON, group entries by service without relying on message text.
• _PID is the originating process ID. It matters when a unit forks multiple worker processes (a pre-fork web server, a pool of workers): grouping by _PID lets you isolate the specific worker instance a request or crash came from, which you often need when correlating with a process-level diagnostic taken from that same PID.

Worked example

Given a hypothetical JSON export like:

{"_SYSTEMD_UNIT":"nginx.service","_PID":"4213","MESSAGE":"worker process started"}
{"_SYSTEMD_UNIT":"nginx.service","_PID":"4214","MESSAGE":"worker process started"}
{"_SYSTEMD_UNIT":"nginx.service","_PID":"4213","MESSAGE":"connection reset by peer"}

Piping this through jq -r 'select(._PID=="4213") | .MESSAGE' isolates only the events from PID 4213, showing that specific worker hit the reset, not its sibling on PID 4214, a distinction the raw message text alone would not make.

Trade-offs & pitfalls

A common mistake is filtering only by unit and assuming that captures everything relevant. A crash can leave entries under kernel (an out-of-memory kill) or under systemd itself (a unit that failed to even start) that never carry _SYSTEMD_UNIT=nginx.service. When a unit-scoped query comes up empty or incomplete, widen to journalctl -k for the kernel ring buffer, or drop the unit filter for a plain time-window query, before concluding the journal has nothing.

Approach: Use the service mesh (Istio) to control traffic, introduce controlled failures, and enforce resilience policies so a canary is exercised safely and automatically rolled back if it degrades key signals.

Traffic-splitting (gradual rollout)

• Use VirtualService to split traffic between stable and canary versions and ramp from e.g. 1% → 5% → 25% → 100%.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  hosts: ["api.example.svc.cluster.local"]
  http:
  - route:
    - destination: { host: api, subset: stable }; weight: 95
    - destination: { host: api, subset: canary }; weight: 5

Fault injection (controlled testing)

• Inject latency or errors only for the canary subset to validate behavior without affecting all users.

# in VirtualService for canary route
fault:
  delay:
    percentage: 10
    fixedDelay: 200ms
  abort:
    percentage: 1
    httpStatus: 500

Retries and timeouts (prevent transient failures causing user impact)

• Configure conservative retries and per-request timeouts to mask transient upstream glitches but avoid retry storms.

retries:
  attempts: 2
  perTryTimeout: 200ms
timeout: 2s

Circuit-breakers (protect downstream services)

• DestinationRule to limit connections, requests, and enforce outlier detection so a failing canary doesn’t cascade.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
spec:
  host: api
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 50
      http:
        http1MaxPendingRequests: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Safety nets to avoid cascading failures

• Progressive ramps with automated gates: advance only if SLOs (latency, error rate, traffic success rate) remain within thresholds.
• Short, per-call timeouts + small retry counts to reduce parallelism and prevent overload.
• Outlier detection + ejection to remove misbehaving canary instances quickly.
• Rate limits and connection pools to bound resource usage.
• Observability & automation: dashboards, SLO-driven alerts, and an automated rollback job (e.g., Argo Rollouts or Flux) that reverts weights when metrics breach thresholds.
• Use shadowing/testing traffic and run chaos/fault-injection against canary only.

Why this works: mesh-level controls centralize traffic policies and resilience, enabling SREs to run realistic canary experiments, contain faults, and automate decisions based on telemetry so rollouts are safe and reversible.

Direct answer
Culture fit asks whether you already share a team's existing norms and behaviors; culture add asks what you would bring that the team does not already have. I would describe myself mostly as a culture add: I share the fundamentals a team needs to trust me (reliability, candor, respect for other people's time), but the useful thing I offer beyond that is a genuinely different working background rather than a mirror of the team that is already there.

Structured elaboration

1. Define both terms precisely before answering for yourself. Culture fit is about alignment on shared behaviors and values: does this person operate the way we already operate. Culture add is about complementary difference: does this person's background, working style, or perspective fill a gap the team doesn't currently have.
2. Explain why the distinction matters, not just define it. A team optimized purely for fit tends toward groupthink: everyone reasons the same way, so blind spots go unchallenged and the same kinds of mistakes recur. A team that only adds without any shared fit becomes uncoordinated: people can't predict each other's reasoning enough to move fast together. The healthy target is fit on a small number of load-bearing behaviors (honesty, follow-through, respect) plus deliberate add on everything else.
3. Give a genuine, specific example of your own add, not a generic trait. Vague claims ("I bring diverse perspectives") are the single most common failure mode here; a strong answer names the concrete gap and the concrete evidence.
4. Anticipate the natural follow-up: how do you know your difference is actually useful, versus just different for its own sake. The answer is to point at a specific decision, disagreement, or piece of feedback that changed because of the difference you brought, not just a credential or background fact.

Worked example
Suppose your last two teams were both product engineering teams building consumer-facing features, and the team you're interviewing for is mostly staffed by engineers with that same background. Your own prior role was on a data-platform team, closer to the systems that feed those consumer features than to the features themselves. A concrete add-story: in a past project, a product team wanted to ship a new recommendation feature quickly; because of your platform background, you asked a question the rest of the team hadn't raised (whether the upstream data pipeline's freshness guarantees actually matched what the feature's UI implied to users), which surfaced a real gap between a 24-hour batch refresh and a UI copy that said "updated just for you." The team fixed the copy and adjusted the refresh cadence before launch rather than after a user complaint. That is a genuine add: a different background produced a question the existing team composition was less likely to ask on its own, and it changed a real outcome.

Trade-offs & pitfalls
The common failure is answering only the definitional half (correctly explaining fit versus add) and then, when asked for a personal example, retreating to generic self-description ("I'm a good communicator", "I care about quality") that any candidate could say and that does not actually demonstrate difference. A second pitfall is overcorrecting into implying you don't fit at all; the strongest answers are explicit that you also share the small set of behaviors every functioning team needs, and that add is about everything on top of that baseline, not a replacement for it.

Framing

Redis Cluster shards the keyspace across nodes by hashing each key into one of 16,384 hash slots, and a multi-key operation (a transaction via MULTI/EXEC, or a Lua script touching multiple keys) is only allowed if every key involved maps to the SAME slot, because Redis Cluster doesn't do cross-node distributed transactions. If your workload needs an atomic update across keys that would naturally land on different slots (e.g. updating an order record and a separate inventory-count record together), you need a design that either forces those keys onto the same slot or achieves atomicity some other way.

Option 1: hash tags to co-locate keys on the same slot

Redis Cluster computes the hash slot from only the part of the key inside {} if present (a "hash tag"). Keys like order:{42}:status and inventory:{42}:count both hash on just 42, landing on the same slot regardless of the rest of the key, which makes them eligible for a real multi-key MULTI/EXEC transaction or a single Lua script that touches both atomically. This is the standard, low-complexity fix when the keys that need to be updated together share a natural grouping key (an order ID, a user ID).

Option 2: a Lua script with hash-tagged keys

Once co-located via hash tags, a Lua script (EVAL) that reads and writes both keys executes as a single atomic operation on the one node owning that slot, which is both atomic and typically faster than MULTI/EXEC for anything beyond the simplest read-modify-write, since the whole logic runs server-side in one round trip instead of a client issuing multiple commands within a transaction block.

Option 3: client-side two-phase commit or a saga

For keys that genuinely can't share a grouping key (truly independent entities that don't have a natural common hash-tag), you'd need either a client-driven two-phase commit (2PC: a coordinator asks each affected key/node to prepare, then commits or aborts both) or a saga (apply each step with a compensating action defined for rollback if a later step fails). Both work without cluster-level transaction support, but both add real complexity: a 2PC coordinator is another moving part that itself needs to be highly available, and a saga's compensating actions have to handle the case where the compensation itself fails partway through.

Option 4: route related data to a single node deliberately

For a workload where a bounded set of keys are consistently updated together (not one-off, structurally related), design the data model so those keys are co-located via hash tags from the start, rather than treating this as an exception to work around later. This is really the same idea as option 1, elevated to a data-modeling principle: if your access pattern needs cross-key atomicity, let that shape your key-naming scheme up front.

Recommendation for a high-throughput system

Hash tags plus a Lua script (options 1 and 2 combined) is what I'd actually recommend for the overwhelming majority of cases, weighing the three factors:

• Complexity: lowest of the options. No coordinator to build or operate, no compensating-action logic to get right, just a key-naming convention and a script.
• Latency: a single-node Lua script executes as one round trip with no cross-node coordination overhead, versus 2PC's multiple round trips (prepare, then commit, to potentially multiple nodes) or a saga's sequential steps.
• Failure modes: a hash-tagged Lua script fails atomically (the whole script succeeds or the whole thing doesn't run, standard Redis Lua semantics) with no partial-commit state to reason about; 2PC introduces the classic "coordinator crashes between prepare and commit" ambiguity that needs its own recovery protocol, and a saga can leave the system in a state that needs a compensating action to run correctly to fully unwind, which is its own source of bugs if not tested as rigorously as the forward path.

The real cost of the hash-tag approach is a data-modeling constraint: it only works when the keys that need atomicity together can be given a shared grouping key, and it also means those keys always land on the SAME node/slot, which can create a hot-shard risk if that grouping key is itself extremely popular (the same hot-key concentration risk any co-location strategy runs into). For a workload where the natural grouping (order ID, user ID) is high-cardinality and roughly evenly distributed, this risk is usually small and well worth the massive simplicity and latency win over 2PC or a saga.

When to reach for 2PC or a saga instead

Only when the keys genuinely can't be co-located (they belong to different, independently-sharded entities with no natural shared grouping key) or when the operation spans Redis and an entirely different system (e.g. Redis plus a separate SQL database), where a saga with well-tested compensating actions is often the more honest design anyway, since true cross-system atomicity is hard to achieve cheaply regardless of what Redis Cluster does or doesn't support.

Worked example

An e-commerce order-placement flow needs to atomically decrement inventory and create the order record. Both keys are named with the order's product ID as the hash tag: inventory:{prod_881}:count and a reservation entry reservation:{prod_881}:order_9931. A single Lua script checks the count, decrements it, and writes the reservation, all atomically on the one node owning that slot, at the cost of every reservation for product 881 landing on the same node, which is an acceptable trade given the product catalog is large and no single product dominates order volume enough to make that node a bottleneck.

Situation: An automation reduced operator toil but coincides with increased MTTR for ~10% of incidents. My plan is a controlled, data-driven investigation to find root cause and decide whether to keep, modify, or retire the automation.

1. Data to collect (what & how)

• Incident metadata: incident_id, timestamp, service, region, customer impact, severity, owner, automation_version, triggered_steps.
• MTTR components: detection_time, ack_time, diagnosis_time, mitigation_time, recovery_time.
• Automation telemetry: run_count, success/failure, latency, retries, error types, stack traces.
• Correlation signals: logs, traces, spans, related alerts, config changes, deploy IDs.
• User/ops actions: manual overrides, steps taken, time spent.
• Pre/post rollout baseline metrics for same window (errors/sec, latency p95/p99, service availability, alert noise).
• Example query: group_by(automation_version, incident_type) → median MTTR, 90th percentile diagnosis_time, failure rate.

2. Run controlled A/B trials (design & analysis)

• Use feature-flagged canary: split traffic/operators into two cohorts (A: automation ON, B: OFF) or time-based rollout. Ensure cohort assignment is random and large enough.
• Duration: run until statistical power reached (power analysis targeting detecting X% MTTR change at alpha=0.05, beta=0.8).
• Safety: start small (1-5% traffic), monitoring thresholds and automated rollback if error budget burn or customer-impacting metrics breach.
• Instrument additional tracing to capture causal chains.
• Analysis: compare MTTR distributions (median, p75, p90), diagnose_time contributions, incident count. Use non-parametric tests (Mann-Whitney) and bootstrap CIs. Segment by incident type and root-cause class.

3. Possible remediation options

• Tweak: adjust automation logic (timeouts, backoff, retry limits), improve error handling, add richer context/logging for faster diagnosis, reduce blast radius by scoping automation to low-risk paths.
• Partial rollback / gradual rollout: disable for specific services, regions, or incident classes where MTTR increases; keep for low-risk ones.
• Kill-switch / feature flag: immediate global disable if safety thresholds exceed.
• Compensating automation: add a quick-detection/override that preempts long-running automation that stalls recovery.
• Human-in-loop hybrid: require operator confirmation for higher-risk remediation; keep automation for routine tasks.
• Root-cause fixes: if automation reveals latent bugs, fix those underlying issues.

4. Decision criteria (keep, modify, retire)

• Quantitative thresholds:
  • If automation reduces operator toil by >= X hours/week (or Y% of incidents resolved automatically) while MTTR increase is statistically insignificant or < acceptable delta (e.g., <5% median MTTR or within error budget), keep.
  • If MTTR increases significantly for high-severity incidents (e.g., p90 MTTR increase > Z% or breaches SLO / causes >N% additional error budget burn), and remediation cannot be delivered in short-term (1-2 sprints), disable for affected slices.
  • If automation causes new incident classes or introduces regressions with customer impact and fixes are high-effort with unclear ROI, retire.
• Qualitative: operator trust, support burden, complexity added, maintainability.
• Rollout policy: prefer modify + canary → measure → expand. Use kill-switch if immediate harm.

5. Follow-up

• Post-mortem with data-backed findings, action items assigned, timeline for fixes.
• Add automated monitoring/alerts for MTTR regressions linked to automation_version.
• Document playbooks for enabling/disabling automation and operator training.

This approach minimizes risk, quantifies trade-offs between toil reduction and reliability, and yields an evidence-based decision.

Direct answer

I design the smallest test that could actually prove me wrong, write down what I expect to see before I run it, and keep the blast radius small enough that being wrong doesn't cost anything real while I find out.

Structured elaboration

Choosing the smallest falsifying experiment: not the smallest experiment that would confirm what I already believe, but the smallest one that could show my understanding is incomplete or wrong. Stating the expectation and acceptance criteria first: I write down what I expect to happen before running it, so I can't quietly reinterpret an ambiguous result afterward as agreeing with me.

Isolating blast radius: a sandbox, a lab setup, or a separate account, with a cost or scope I've deliberately bounded in advance, so a wrong understanding is cheap to discover rather than expensive.

Representative rather than toy data: using data or conditions close to the real failure pattern, not an artificially clean case that would pass regardless of whether my understanding is actually right.

Making the result reproducible: documenting the exact setup and outcome so it holds up to scrutiny, and so I can redo the check later if the underlying system changes, rather than relying on memory of what happened.

Reproducing claims instead of trusting them: if my understanding came from a vendor's or a blog's claim, I try to reproduce that specific claim myself rather than taking it as already proven.

Staged progression before it matters: an isolated experiment first, then something closer to an integration test, then one small, low-risk, production-adjacent change, rather than jumping straight from a lab result to something that matters.

Worked example

I'd read that a specific retry and backoff configuration would fix a flaky downstream call, but hadn't verified it myself. I set up a throwaway environment and replayed real traffic that reproduced the actual failure pattern, rather than a clean synthetic case. Before running anything, I wrote down the falsifiable claim: the new configuration should reduce failures without increasing load on the downstream service, not just "it'll work." I ran it isolated, checked both halves of that prediction, and both held. I rolled it out on one non-critical path first, watched it for a defined period, then extended it further once that held up too.

Trade-offs and pitfalls

The most common failure mode is designing a gentle test that only confirms the claim rather than one that could genuinely falsify it, especially when the claim came from a source you already want to trust. The other is skipping the staged rollout because the lab result felt convincing enough, and jumping straight from an isolated test to full production.

Direct Answer

A distributed lock service has to guarantee that, at any real instant, at most one live client believes it safely holds the lock, and it has to keep making that guarantee even when a client crashes mid-hold or the network delays messages. The safest way to build one is on top of something that already solves distributed agreement (a consensus protocol with a majority quorum), because a lock without a canonical ordering source can end up with two clients each convinced they're the holder.

Comparing Three Implementations

Fencing Tokens: Why a Lease Alone Isn't Safe

A lease with a time-to-live tells you how long to wait before you're allowed to consider a holder dead, but it can't stop a holder that is still alive, just delayed, from acting after its lease has actually expired. A process pause (for example, a stop-the-world garbage collection pause) or a delayed network write can make a client believe it still holds the lock long after the lock service has moved on.

Fencing tokens close this gap. Every successful lock grant returns a monotonically increasing number (the fencing token). The client includes that token with every operation it performs against the protected resource. The resource itself, not just the lock service, tracks the highest token it has ever seen and rejects any operation carrying a lower one.

sequenceDiagram
    participant A as Client A
    participant B as Client B
    participant L as Lock Service
    participant R as Protected Resource
    A->>L: Acquire lock
    L-->>A: Granted, token = 34
    Note over A: Long garbage collection pause, exceeds the lease TTL
    L->>L: Lease expires, lock released
    B->>L: Acquire lock
    L-->>B: Granted, token = 35
    B->>R: Write(token = 35)
    R-->>B: Accepted, 35 is higher than any token seen so far
    A->>R: Write(token = 34)
    R-->>A: Rejected, 34 is lower than the last accepted token 35

Concretely: suppose the lease time-to-live is configured to 10 seconds. Client A acquires the lock and gets token 34, then falls into a garbage collection pause longer than that window. The lock service, seeing no renewal, expires A's lease and grants the lock to client B with token 35. B writes to the resource with token 35; the resource records 35 as the highest token seen. A wakes up, still believing it holds the lock since it has no way to know time passed, and sends its own write carrying its old token, 34. Because 34 is lower than the 35 the resource already recorded, the resource rejects A's write. The stale holder is blocked from doing damage even though the lock service itself never directly told A it lost the lock.

Lease-Based Locking vs. Optimistic Concurrency

Not every coordination problem needs an actual lock. A lease is a pessimistic mechanism: it reserves exclusive access for a time window whether or not anyone else is actually trying to touch the resource, and it inherits all the lease-renewal and clock-skew risk described above. An alternative for many single-resource coordination problems is optimistic concurrency control: read the resource's current version, compute the intended change, and write it back with a compare-and-swap that only succeeds if the version hasn't changed since the read. This sidesteps lease renewal and clock skew entirely, since there's no time window to defend, but it means contending clients retry instead of queueing, which gets expensive under high contention.

As a rule of thumb: prefer optimistic compare-and-swap when you're coordinating a single write to a single record and can express the whole operation as one atomic check-and-set. Reach for a real lock or lease when the critical section spans multiple steps or has side effects outside the data store itself, such as kicking off a batch job or calling an external system, that can't be rolled into one atomic compare-and-swap.

Trade-offs and Pitfalls

• A lock service alone, without fencing enforced at the resource, is not actually safe: the resource has to participate in validating tokens, or a stale holder's writes go through unchallenged.
• The well-known Redlock (multi-instance Redis locking) design has been criticized for relying on synchronized clocks and not accounting for pauses like the one in the worked example; that critique is really an argument for fencing tokens as the real safety mechanism, with the lease itself only providing liveness (bounding how long you wait before treating a holder as dead), not safety.
• A lease time-to-live that's too short causes unnecessary churn under normal jitter; one that's too long makes real crashes take longer to recover from. Neither extreme removes the need for fencing tokens at the resource.
• Single-primary database row locks scale the worst across regions and don't survive a failover cleanly unless the failover procedure is specifically built to carry the lock state or invalidate it.

Direct answer
TIME_WAIT is the state the side that sent the FINAL ACK of a connection close sits in for a fixed period (commonly twice the maximum expected segment lifetime, often around 60 seconds on Linux) before the connection's resources are fully released. It exists so a delayed, duplicate packet from an old connection can't be mistaken for part of a brand-new connection reusing the same address/port pair. A half-open connection is one where only one side still believes the connection is alive; the other side has already reset, crashed, or otherwise abandoned it without a clean FIN exchange.

Structured elaboration
TIME_WAIT exists to protect two things: (1) it guarantees the final ACK the closing side sent actually gets through, by giving time to retransmit it if the peer's FIN gets retransmitted (meaning the ACK was lost); (2) it prevents a stray, delayed packet from a previous incarnation of a connection (same 4-tuple: source IP, source port, destination IP, destination port) from being delivered into a brand new connection that happens to reuse the same 4-tuple before the old segments have had time to disappear from the network.

To detect a large number of sockets stuck in TIME_WAIT on Linux, ss -tan state time-wait | wc -l (or the older netstat -ant | grep TIME_WAIT | wc -l) gives a live count; watching this metric over time distinguishes a normal, self-draining backlog from a genuine problem.

Worked example
A server that closes millions of short-lived outbound connections per hour (for instance, a service making one HTTP call per request to an upstream) can exhaust its available ephemeral source ports if TIME_WAIT sockets accumulate faster than they expire, because each TIME_WAIT socket still holds its port reserved. The two standard mitigations are: raise the number of available client-side (ephemeral) ports and/or reuse connections via keepalive/connection pooling so fewer connections churn through TIME_WAIT in the first place; and, on the SERVER side specifically, enabling SO_REUSEADDR and, where safe, tcp_tw_reuse lets a new outgoing connection reuse a TIME_WAIT 4-tuple once TCP timestamps confirm it's safe to do so, rather than waiting out the full timer.

Trade-offs & pitfalls
Disabling or drastically shortening TIME_WAIT globally (rather than tuning port ranges or reuse settings) is the wrong fix: it reintroduces the exact correctness problem TIME_WAIT was designed to prevent, stray old packets landing in a new connection and corrupting it. The safe levers are reducing HOW MANY connections churn through the state (pooling, keepalive) and widening the ephemeral port range, not shrinking the safety window itself.

Direct answer

Availability is whether the service is up and responding right now, the percentage of time requests get a correct response. Reliability is whether the service does the correct thing every time over a longer horizon, even if that means taking longer or failing loudly rather than silently. A service can be highly available (always responds) while being unreliable (frequently returns wrong or incomplete results), and vice versa.

How they're measured differently

• Availability: uptime percentage, request success rate (successful responses over total requests), and latency, all measured in real time against a rolling window.
• Reliability: job or transaction success rate over time, data-loss incidents, mean time between failures, and correctness checks like reconciliation counts, none of which are visible from a single point-in-time health check.

Worked example: an HTTP API versus a background worker

An HTTP API's job is to respond fast and stay up, so availability is the priority metric. Suppose the API calls three dependencies in sequence to serve a request: an auth service at 99.95% availability, a database at 99.9%, and a cache at 99.99%. Because a single request needs all three to succeed, the composed availability is the product of the three:

That's under three nines even though every individual dependency is at or above three nines, because failures compound across a serial chain. In annual downtime terms:

compared to a single 99.9% dependency on its own:

Chaining three otherwise-strong dependencies serially costs over 300 extra minutes of downtime a year versus just one of them alone. This is why an API-focused architect pushes hard on redundancy at each hop. To see how strong that lever is even when the underlying component is weaker, consider a hypothetical, cheaper cache tier, deliberately worse than the 99.99%-rated cache used above, where each individual replica only hits 99% availability on its own: two independent, parallel replicas of that weaker cache layer already beat any single component in the chain, the strong 99.99% cache included:

A background worker processing a queue of jobs, by contrast, doesn't need to respond within milliseconds; what matters is that every job eventually completes correctly, with no silent data loss, which is a reliability property, not an availability one. If the worker is down for ten minutes and then resumes and correctly processes every job that queued up during that window, availability took a hit but reliability didn't; if the worker stays "up" the whole time but drops or duplicates 0.01% of jobs due to a bug, availability looks perfect while reliability has quietly failed.

Trade-offs & pitfalls

Optimizing for availability alone can mask reliability problems: a service that always responds quickly, even by returning stale or wrong data rather than waiting for a correct answer, looks perfect on an uptime dashboard while silently corrupting downstream state. The practical approach is deciding, per component, which property is actually load-bearing: user-facing APIs generally prioritize availability with graceful degradation for correctness-adjacent risk, while systems of record and background processing prioritize reliability, often accepting higher latency or even temporary unavailability rather than risk an incorrect or lost write.

Direct answer

iowait is a CPU accounting category, the percentage of time a CPU was idle specifically because the only thing it had to run was blocked on I/O (input/output, typically disk). It does not directly measure disk speed, and it does not mean the whole system is I/O bound: it only registers when a CPU core has literally nothing else runnable while something on that core waits on I/O, so a busy multi-core system can show low iowait even while storage is genuinely slow, simply because other cores stay busy with unrelated work.

Structured elaboration

What iowait measures. Each CPU core's time is bucketed into categories: user, system, idle, iowait, and a few others (covered more fully alongside CPU utilization). iowait specifically means: this core was idle, and at that moment at least one process that could have run on this core was instead blocked waiting on an I/O operation to complete. It's a statement about CPU scheduling opportunity, not a direct measurement of storage latency.

What it does not measure. It does not tell you how slow the disk actually is in absolute terms; a disk could be extremely slow and iowait could still read near zero, if the system always has other runnable work to fill every core while the slow I/O is in flight (common on busy, highly concurrent multi-core boxes). Conversely, a lightly loaded single-threaded box with genuinely fast storage can show a high iowait percentage simply because there's nothing else for that one core to do while waiting, even for I/O that completes quickly in absolute terms. iowait is best read as "wasted idle CPU capacity attributable to I/O," not "how bad is my storage."

Distinguishing storage-caused iowait from other causes.

• iostat -xz 1 on the actual devices: if await and %util (or, more reliably, aqu-sz) show real saturation at the same time iowait is elevated, storage is genuinely implicated.
• vmstat 1: check the b column (processes in uninterruptible sleep, blocked, typically on I/O) alongside the wa (iowait) column. A rising b count corroborating a rising wa percentage strengthens the case that this is real I/O blocking rather than a measurement artifact.
• sar -d 1 gives historical, per-device stats you can correlate against the timing of an iowait spike after the fact.
• perf trace or strace -T -p <pid> can show you which specific syscalls (read, write, fsync) are actually blocking and for how long, useful when you need process-level attribution beyond the aggregate system-wide iowait number.

Worked example

A host shows mpstat 1 5 reporting %iowait at 35% sustained across samples. Checking iostat -xz 1 at the same moment shows the underlying NVMe device's w_await at 45ms (versus a normal sub-millisecond baseline) and aqu-sz climbing to 6, confirming the storage device itself is genuinely backed up, not just an artifact of CPU scheduling. Cross-checking vmstat 1 shows the b column at 8, meaning 8 processes are currently blocked in uninterruptible sleep, consistent with real I/O-bound stalling rather than, say, a CPU scheduling quirk. All three signals agreeing (elevated iowait, elevated device await/queue, elevated blocked-process count) is what turns "the CPU accounting category says iowait" into "the storage layer is actually the bottleneck."

Trade-offs and pitfalls

Treating iowait in isolation as a definitive storage-health metric is the main pitfall: it can understate a real storage problem on a busy multi-core box (other cores stay filled with unrelated work) and it can overstate a minor one on a lightly loaded, mostly-idle box. Always corroborate with device-level metrics (iostat) and the blocked-process count (vmstat's b column) before concluding storage is the bottleneck from iowait alone.