Spotify Site Reliability Engineer (Mid-Level) Interview Preparation Guide

Direct answer

I separate what's actually being transferred, the underlying principle, from what's incidental to the old context, the specific implementation and its defaults, and I re-verify the parts that depend on the new context's specifics rather than assuming a straight port. I check soundness by comparing the new result against an independent ground truth or the new domain's own baseline, not just against "it ran without error."

Structured elaboration

• Identify the transferable core versus the context-bound specifics. The underlying idea, an algorithm, a statistical method, a design pattern, usually carries over. The exact parameters, library defaults, and assumptions baked into the old context often don't, even when everything looks superficially the same.
• Watch for the mechanical trap. Reimplementing what looks like "the same" logic in a different toolchain can silently produce a different answer because of quiet differences in defaults: numeric precision, random seeds, how a library breaks ties, or off-by-one conventions that never mattered before because you never had to think about them.
• Watch for the conceptual trap. A method borrowed from a neighboring field brings assumptions baked into it, tuned for a particular scale, data distribution, or failure mode, that may not hold in the new one, and needs deliberate adapting rather than a straight relabel.
• Validate against something independent. A known-answer test case, an existing simpler baseline already trusted in the new domain, or a manual spot-check by someone who knows the new context well, so you're checking that the result is right, not just that it executed.
• Only trust the transfer once it holds up against the new domain's own baseline, measured on its own terms, not against the numbers you got in the old context.

Worked example

I ported a feature-engineering pipeline that had been prototyped in a small, single-machine data-analysis library over to a distributed processing toolchain meant to scale it up. I assumed the aggregation logic, grouping records and summing a value within each group, would produce identical output, since it was "the same" calculation. Before trusting it, I ran both versions on a fixed, unchanged sample and diffed the outputs directly rather than assuming a match. They disagreed slightly, and it turned out the distributed version summed floating-point numbers in a different order across its workers, which changed the result by a tiny but real amount for a few groups, and it also handled missing values differently by default than the original library had. Because I'd deliberately checked instead of trusting the port, I caught both before the new pipeline went anywhere near a real report, fixed the null handling to match intentionally, and documented the small floating-point discrepancy as expected and acceptable rather than a bug, since I understood its actual cause instead of just noticing a mismatch.

Trade-offs and pitfalls

The clearest trap is assuming "same logic, different tool" automatically means "same answer," when defaults and edge-case handling frequently differ between implementations in ways that only show up once you actually check. A close second is skipping validation because the transfer feels obvious or low-risk, which is exactly when a quiet discrepancy is most likely to go unnoticed. And carrying an assumption over from the source domain without re-examining whether it still holds, rather than deliberately adapting it, is how a borrowed method ends up quietly wrong in its new setting.

Direct answer
Acceptance criteria for a corrective action should be a specific, measurable, time-boxed statement of what 'fixed' looks like, defined before the work starts, not after. A verification plan then confirms that criterion is actually met using real data, not just confidence that the fix was implemented correctly.

Structured elaboration

• Define the metric and target explicitly. Not 'reduce timeouts' but 'reduce the service's timeout rate from its current baseline to a specific target percentage, measured over a specific window.' A vague criterion can't be verified; a specific one can.
• Set a monitoring window long enough to be meaningful. Too short a window risks declaring success on noise; too long delays knowing whether the fix worked. The right window depends on the incident's natural frequency, for example enough days to capture a representative mix of peak and off-peak traffic.
• Separate short, medium, and long-term verification. Immediately after deploying the fix: a targeted test or synthetic check confirms the mechanism works as intended. Over the following weeks: real production monitoring against the target metric confirms it holds under real conditions, not just in a controlled test. Longer term: a periodic audit or scheduled re-check confirms the improvement is durable and hasn't quietly regressed.
• Define what "success" and "failure" mean numerically in advance, including what would trigger reopening the item if the target isn't met, so there's no ambiguity or motivated reasoning once the data comes in.
• Name who signs off, so verification isn't just a self-assessment by whoever implemented the fix.

Worked example
A corrective action targets reducing a service's timeout rate from 0.5% to 0.05% within 30 days. Acceptance criteria: timeout rate, measured as a 7-day rolling average, must be at or below 0.05% for two consecutive weeks within the 30-day window, using the same monitoring dashboard and definition of 'timeout' used to measure the original 0.5% baseline. Verification plan: short-term, a synthetic load test immediately after deploy confirms the fix reduces timeout rate under simulated peak load; medium-term, the real 7-day rolling average is checked weekly against the target for the full 30 days; long-term, the metric is re-checked at 90 days to confirm it hasn't quietly crept back up as traffic patterns shift. If the 30-day window ends with the metric at 0.15%, that's a defined failure, not an ambiguous 'mostly worked,' and it triggers a re-investigation of whether the fix addressed the actual root cause or only a symptom.

Trade-offs and pitfalls
The most common mistake is defining acceptance criteria loosely enough that almost any outcome can be called success, which defeats the purpose of having criteria at all. A second is skipping the longer-term recheck: many fixes look successful in the first two weeks and then quietly regress as conditions change, and without a scheduled longer-term verification, that regression goes unnoticed until the incident recurs.

Direct answer
A full rollback reverts every service or component involved in a release back to its previous version; a partial rollback reverts only some of them, leaving others on the new version. You choose partial when only some of the changed components are actually implicated in the problem and reverting the rest would be unnecessary churn or would itself introduce risk (for example, if a later component now depends on an earlier one's new behavior).

Structured elaboration

• When full rollback makes sense: a single service deployed a bad change, or several services were released together as one coupled unit where partial reversion would leave them in an inconsistent, untested combination.
• When partial makes sense: a coordinated multi-service release where only ONE service's new version is misbehaving, and the others are backward-compatible enough to keep running against either the old or new version of the problem service.
• Risk in partial rollback: you have to be confident the services you're leaving on the new version don't assume the now-reverted service's new behavior; if they do, a partial rollback trades one outage for a different, subtler one.

Worked example
Full: three tightly coupled microservices (auth, session, and API-gateway) deploy together as one release because the API gateway's new version assumes session's new token format; a bug anywhere means reverting all three together, since a partial revert would leave a token-format mismatch between them.
Partial: a release touches a recommendations service and a completely independent notifications service in the same deploy window purely for scheduling convenience; if only recommendations regresses, rolling back just that service and leaving notifications on its (unrelated, unaffected) new version is the right call, since reverting notifications too would be pure unnecessary churn.

Trade-offs and pitfalls
Partial rollback is faster and less disruptive when it's safe, but the judgment call of "is it actually safe to leave X on the new version" is exactly where mistakes happen; teams that don't explicitly map service dependencies before a coordinated release often discover the hard way, mid-incident, that two "independent" services weren't as independent as assumed.

Investigating a heisenbug requires accepting the standard toolkit (logging, attaching a debugger) is off the table, since both perturb the exact timing you need to observe.

Non-invasive strategies, roughly in order of overhead

1. Kernel/low-level tracing (ftrace, eBPF, perf): observe scheduling and syscall timing with far less perturbation than application-level logging, since they don't run inside the process's own critical path the same way. (ftrace is Linux's built-in kernel function tracer; eBPF lets you run small sandboxed programs inside the kernel to observe events cheaply; perf is Linux's low-overhead sampling profiler; all three watch the system from outside the process rather than adding code inside it.)
2. Hardware watchpoints: a CPU-level trap that fires when a specific memory address is written, letting you catch the exact write that corrupts shared state without instrumenting the code path at all.
3. Record-and-replay (rr) (Mozilla's open-source record-and-replay debugger): capture one real occurrence (including all syscalls and nondeterministic inputs) once, then replay it deterministically as many times as needed in a full debugger session, which sidesteps the observer-effect problem entirely after the initial capture.
4. Increase trigger probability instead of adding instrumentation: run under heavier concurrent load, or add artificial scheduling pressure (deliberately delay specific operations) to make the natural race window occur more often, without touching the code path being investigated.

Confirmed generalization

The same four techniques apply whether the observing party is a human with a debugger, or an application's own logging framework: anything that runs synchronously in the critical section changes its timing. This is why four independent worked cases across roles (SRE, general engineering, systems engineering, QA) all converge on the same toolset: rr/record-and-replay, kernel-level tracing (ftrace/eBPF), perf sampling, and GDB used only against a replayed/captured session rather than the live race.

Trade-offs and pitfalls

These tools have a real learning curve and setup cost (eBPF and rr both require specific kernel/OS support and practice); the payoff is a heisenbug that would otherwise burn days of guess-and-check becomes a captured, replayable artifact you can step through as many times as needed, which is usually worth the setup cost for anything that recurs.

Direct answer

When a cache, a primary database, and a search index all hold a copy of the same entity, an update has to reach all three, and the order they are updated in determines what inconsistent intermediate state a concurrent reader might see; design the update sequence deliberately rather than firing updates to all three independently.

Structured elaboration

• The core pitfall: three independent copies means three independent chances to be out of sync at any given moment; a naive "update DB, then update cache, then update search index" sequence has a window after each step where a reader hitting a different layer sees a different, momentarily-inconsistent view of the same entity.
• An ordered update workflow: write to the primary database first (the source of truth), then propagate to the cache and search index, ideally via the same mechanism (an event derived from the database's own write, e.g., change data capture, CDC) rather than the application independently, and possibly inconsistently, updating each one itself.
• Minimizing stale reads across layers: invalidate (or update) the cache promptly after the database write; the search index, which is often inherently eventually consistent by nature (indexing takes measurable time), should be treated with a wider, explicitly acknowledged staleness tolerance rather than expected to match the database's freshness.
• Failure recovery: if the propagation to the cache or search index fails partway (the database write succeeded but the cache invalidation did not), the system needs to detect and correct that drift, either via a reconciliation job that periodically compares layers, or by treating the CDC-based propagation as the single source of truth for keeping all downstream copies in sync (so a failure there is a well-understood, monitorable gap, not a silent one).
• Trade-offs involved: driving every downstream update from the database's own change stream (rather than each application code path independently updating cache and search index) is more work to set up but eliminates entire classes of "we updated the DB and cache but forgot the search index in this one code path" bugs.

Worked example

A user profile update: the database write succeeds; a CDC-derived event then triggers both a cache invalidation and a search-index update. A reader hitting the cache immediately after the database write (before the CDC event has propagated) sees a brief stale read from the cache, bounded by the CDC pipeline's typical propagation latency (often under a second); a search query against the index during that same window might return the OLD profile data in search results for slightly longer, since indexing itself typically has its own additional latency beyond simple cache invalidation, an acceptable, well-understood difference in staleness across layers as long as it is explicit and monitored.

Trade-offs and pitfalls

Independently updating cache and search index from application code (rather than from a single CDC-derived event stream) is a common source of "we forgot this one code path" bugs, where some write paths correctly update all three layers and others silently miss one; centralizing propagation removes that class of bug at the cost of needing CDC infrastructure. Treating all downstream layers as needing the SAME freshness guarantee is usually wrong; a search index's inherent indexing latency is a different, and typically wider, staleness budget than a cache's invalidation latency, and conflating the two leads to either wasted effort over-optimizing the index or an unrealistic freshness expectation for it.

Approach: canonicalize both the configured BASE_DIR and the user-supplied path (resolving symlinks) and verify the resolved path is a descendant of BASE_DIR. Note: strict prevention of TOCTOU and symlink races requires kernel-level flags (O_NOFOLLOW) or openat-based checks; a pure Bash script is best-effort and must be paired with safer helpers for production.

#!/usr/bin/env bash
set -euo pipefail

BASE_DIR="/srv/app/data" # trusted base directory (absolute)
usage() { echo "Usage: $0 <relative-or-absolute-filename>"; exit 2; }

[ $# -eq 1 ] || usage
user_input="$1"

Resolve canonical absolute path of base (no trailing slash)

base_canon="$(realpath -m -- "$BASE_DIR")"

Resolve the user path relative to BASE_DIR if not absolute

if [[ "$user_input" != /* ]]; then
candidate="$base_canon/$user_input"
else
candidate="$user_input"
fi

Resolve symlinks and get canonical path; fail if doesn't exist

if ! target_canon="$(realpath -- "$candidate" 2>/dev/null)"; then
echo "Error: target does not exist or cannot be resolved" >&2
exit 3
fi

Ensure target is inside base (prefix check)

case "$target_canon" in
"$base_canon" | "$base_canon"/*)
;;
*)
echo "Error: path escapes base directory" >&2
exit 4
;;
esac

Safe action example: open for read-only and print first lines

Note: this still follows symlink at open time; see limitations below.

if [ -f "$target_canon" ]; then
echo "Opening: $target_canon"
head -n 10 -- "$target_canon"
else
echo "Error: not a regular file" >&2
exit 5
fi

Key points and limitations:

• We use realpath to Canonicalize (resolving .. and symlinks) and a prefix check to ensure containment.
• TOCTOU race: an attacker can swap symlinks between the time you resolve path and the time you open it. Bash and realpath can't prevent this atomically.
• Symlink / open races: opening the resolved path may still follow a different symlink if attacker replaces targets after resolution.

Mitigations:

• Prefer using low-level APIs: openat2/openat + O_NOFOLLOW | O_CLOEXEC, fstat to verify inode/device, or implement in C/Go/Python using os.open with O_NOFOLLOW and fstat comparisons.
• Use a sandboxed helper running with reduced privileges (chroot, user namespace) or run the operation as a dedicated unprivileged user with the base directory as its root.
• If possible, keep filenames immutable (only administrators can create/delete in BASE_DIR) and audit/write-protect directories.
• For critical paths, perform checks and open using the same tool that supports atomic flags (e.g., a small C helper).

Direct answer
TIME_WAIT is the state the side that sent the FINAL ACK of a connection close sits in for a fixed period (commonly twice the maximum expected segment lifetime, often around 60 seconds on Linux) before the connection's resources are fully released. It exists so a delayed, duplicate packet from an old connection can't be mistaken for part of a brand-new connection reusing the same address/port pair. A half-open connection is one where only one side still believes the connection is alive; the other side has already reset, crashed, or otherwise abandoned it without a clean FIN exchange.

Structured elaboration
TIME_WAIT exists to protect two things: (1) it guarantees the final ACK the closing side sent actually gets through, by giving time to retransmit it if the peer's FIN gets retransmitted (meaning the ACK was lost); (2) it prevents a stray, delayed packet from a previous incarnation of a connection (same 4-tuple: source IP, source port, destination IP, destination port) from being delivered into a brand new connection that happens to reuse the same 4-tuple before the old segments have had time to disappear from the network.

To detect a large number of sockets stuck in TIME_WAIT on Linux, ss -tan state time-wait | wc -l (or the older netstat -ant | grep TIME_WAIT | wc -l) gives a live count; watching this metric over time distinguishes a normal, self-draining backlog from a genuine problem.

Worked example
A server that closes millions of short-lived outbound connections per hour (for instance, a service making one HTTP call per request to an upstream) can exhaust its available ephemeral source ports if TIME_WAIT sockets accumulate faster than they expire, because each TIME_WAIT socket still holds its port reserved. The two standard mitigations are: raise the number of available client-side (ephemeral) ports and/or reuse connections via keepalive/connection pooling so fewer connections churn through TIME_WAIT in the first place; and, on the SERVER side specifically, enabling SO_REUSEADDR and, where safe, tcp_tw_reuse lets a new outgoing connection reuse a TIME_WAIT 4-tuple once TCP timestamps confirm it's safe to do so, rather than waiting out the full timer.

Trade-offs & pitfalls
Disabling or drastically shortening TIME_WAIT globally (rather than tuning port ranges or reuse settings) is the wrong fix: it reintroduces the exact correctness problem TIME_WAIT was designed to prevent, stray old packets landing in a new connection and corrupting it. The safe levers are reducing HOW MANY connections churn through the state (pooling, keepalive) and widening the ephemeral port range, not shrinking the safety window itself.

GPU scheduling in Kubernetes works through three cooperating pieces: the device plugin framework, a kubelet-side extension mechanism that advertises non-standard hardware as an allocatable resource, the resulting extended resource name a pod requests, and optional node labels used for affinity. None of it works out of the box; the cluster needs a working NVIDIA driver, a GPU-aware container runtime, and the NVIDIA device plugin running before any pod can actually get scheduled onto a GPU.

How the pieces fit together

• Node prerequisites. Every GPU node needs the NVIDIA driver installed at the kernel level, and the container runtime, containerd or CRI-O, since Docker itself was removed as a supported Kubernetes container runtime in version 1.24, configured with the NVIDIA Container Toolkit so a container can actually use the device, not just have it requested.
• Device plugin. The NVIDIA device plugin runs as a DaemonSet, one copy per GPU node, and registers with the kubelet over the device plugin API, advertising each GPU as a unit of the extended resource nvidia.com/gpu. Only after this registration does kubectl describe node show nvidia.com/gpu under that node's allocatable resources.
• Resource request. A pod requests a GPU the same way it requests CPU or memory, by naming the extended resource under resources.limits. GPUs are requested as a whole-number limit, not split fractionally the way CPU can be; a container either gets a whole GPU or none.
• Node labels. Labels such as nvidia.com/gpu.present or a GPU-model label are typically populated automatically by an NVIDIA feature-discovery component (part of the NVIDIA GPU Operator), not applied by hand; they exist so a pod can use nodeSelector or node affinity to target GPU-capable nodes, or a specific GPU generation, rather than relying on the extended-resource request alone to imply placement.

Minimal pod requesting one GPU

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: cuda-test
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    resources:
      limits:
        nvidia.com/gpu: 1

Worked example: why a GPU request can go Pending

Suppose kubectl describe node gpu-node-1 shows the following, an illustrative example rather than a live cluster:

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     4
Allocated resources:
  Resource         Requests   Limits
  --------         --------   ------
  nvidia.com/gpu   3          3

Free capacity on this node is

$4 - 3 = 1$

GPU. A new pod requesting nvidia.com/gpu: 2 cannot fit here even though the node exists and is healthy; if it is the only GPU node, or the only one with free capacity, the pod stays Pending, and kubectl describe pod on it shows an Insufficient nvidia.com/gpu scheduling event, the extended-resource equivalent of the far more common insufficient-CPU or insufficient-memory Pending cause.

Trade-offs and pitfalls

• Whole-GPU requests only: unlike CPU, a container cannot request half a GPU through the base device plugin; workloads that only need a fraction of a GPU's capacity either waste the rest or need NVIDIA's Multi-Instance GPU (MIG) partitioning on hardware that supports it, current-generation data-center GPUs, which exposes each partition as its own schedulable unit.
• Forgetting a node label, when policy requires one, while the device plugin resource is present still lets the pod schedule anywhere with free nvidia.com/gpu capacity, which is not always the intended node if a cluster mixes GPU generations.
• This is a genuinely narrow, machine-learning-infrastructure-specific mechanism; a general Kubernetes operations interview is unlikely to probe device plugins in depth, but it is a real and common question in ML-platform-focused loops.

Goal: optimize many small-file reads for build farm.
Client-side mount options:

• Use NFSv4 or v4.1 with async read caching: mount -o rsize=131072,wsize=131072,async,noatime,actimeo=1
• Enable attribute caching tuning: actimeo lower for consistency trade-off or set noac to reduce stale reads (but increases server load).
  Server-side/kernel-level:
• Increase nr_open/file-max and RPC thread pools (rpc.nfsd threads) to handle many concurrent ops: echo 64 > /proc/fs/nfsd/threads
• Use aio and increase readahead for metadata-heavy workloads: echo 128 > /sys/block/<device>/queue/read_ahead_kb
• Ensure NFS server uses XFS ext4 with tuned inode/cache settings; increase dentry cache (vm.vfs_cache_pressure) and inode cache sizes.
  Other strategies:
• Use FS cache (cachefilesd) or memcached-based caching (fs-cache) to reduce RPCs for small files.
• Consider using a distributed object cache (s3/ceph) or copy local caches on build nodes (ccache) to avoid repeated small reads.
  Trade-offs: increasing caching speeds builds but risks stale data; async and larger readahead improves throughput but can increase memory and harm consistency. Best practice: measure with iostat, nfsstat, and tune incrementally; prefer local caching layers for small files.

Direct answer

Meeting a roughly 60-second reroute target for most clients means combining health-check-driven DNS failover (to redirect clients as they re-resolve) with Anycast routing at the network layer (to redirect clients immediately, independent of DNS caching behavior), because DNS alone can't guarantee a hard bound once client-side resolver caching is accounted for.

Architecture

flowchart TD
  Client -->|Anycast IP| Edge[Anycast Edge and CDN]
  Edge --> GSLB[GSLB Health-Aware DNS]
  GSLB -->|healthy| R1[Region 1]
  GSLB -->|healthy| R2[Region 2]
  GSLB -->|healthy| R3[Region 3]
  HC[Active Health Checkers] -->|probe every 5s| R1
  HC -->|probe every 5s| R2
  HC -->|probe every 5s| R3
  HC -->|update after 3 consecutive fails| GSLB
  R1 -.fails.-> HC

GSLB (Global Server Load Balancing, shown in the diagram) is DNS that returns different regional IPs depending on which regions are currently healthy, the mechanism the DNS/load-balancer layer below relies on.

Health checks: active probes (HTTP and TCP, from multiple external vantage points) hit each region every 5 seconds, requiring 3 consecutive failures before a region is marked unhealthy, to avoid flapping on a single transient blip.

DNS/load-balancer configuration: authoritative DNS TTL for the service record is set low, 30 seconds, so clients that honor TTLs re-resolve quickly after a region is marked unhealthy. Anycast IPs are advertised from all three regions simultaneously; when a region fails, its BGP (Border Gateway Protocol: the protocol that advertises which network paths lead to a given IP address) announcement is withdrawn, which reroutes traffic to a healthy region at the network layer almost immediately, without waiting on any client's DNS cache to expire at all.

Worked example: does this hit the 60-second target?

Detection time, the health checker's confirmation that a region is actually down:

For clients relying on DNS re-resolution, the worst case adds the full TTL window on top of detection, since a client could have just refreshed its cache right before the failure:

That leaves 15 seconds of margin against the 60-second target for clients that honor the TTL correctly, covering the large majority of traffic (public recursive resolvers like major DNS providers generally respect low TTLs closely). For the remaining slice of clients behind resolvers that cache more aggressively than the stated TTL (some enterprise resolvers and certain mobile carrier networks), the Anycast BGP withdrawal is what actually gets them under the target: it operates at the network layer and doesn't depend on DNS caching behavior at all, so those clients are rerouted within the same roughly 15-second detection window, not the 45-second DNS-bound one. Combining the two mechanisms is what makes hitting the target for "most clients" (rather than only the well-behaved subset) achievable.

Long-lived connections during cutover

Neither DNS re-resolution nor an Anycast BGP withdrawal preserves an existing TCP connection or WebSocket session that was already established to the failed region; a BGP route change mid-flow actually breaks those connections rather than gracefully migrating them, since the new route doesn't carry the old connection's state. Clients holding long-lived connections need their own reconnect logic (detect the drop, re-resolve or reconnect, resume from the new region) and any in-flight request that was interrupted needs to be safely retryable, which pushes the requirement for idempotent write handling on the server side, since a client that reconnects and retries an interrupted request must not have that retry double-process the original attempt.

Trade-offs & pitfalls

A lower DNS TTL improves worst-case failover time but increases query volume against the authoritative DNS servers and, for high-traffic services, real cost; 30 seconds is a reasonable middle ground rather than pushing to something extremely aggressive like 5 seconds. Anycast gives fast, DNS-independent failover but requires BGP-level control over IP announcements, which is a meaningfully bigger operational lift than DNS alone and isn't available on every cloud platform without specific networking products. The most common mistake in this kind of design is validating the 60-second target only against health-check and DNS timers on paper, without ever measuring how real clients across different resolver populations actually behave in practice, which is the only way to know the theoretical margin actually holds up.