Lyft Site Reliability Engineer (Entry Level) - Comprehensive Interview Preparation Guide

Direct answer

A cron entry runs a script every minute that makes one curl request with hard timeouts, appends epoch latency ok to a samples file, trims the file to the last 300 seconds, computes a nearest-rank 99th percentile (p99: the latency 99% of samples are at or below) over the successful samples, and writes the result as a Prometheus text-format file that node_exporter's textfile collector picks up. (Prometheus is a monitoring system that stores numbers over time and raises alerts; node_exporter is its agent that exposes a host's metrics over HTTP; the textfile collector is the node_exporter module that reads every *.prom file in one directory and serves the lines in it as extra metrics, so a cron script can publish a number without running a server of its own.) Overlap is prevented with flock -n (a second run exits at once instead of queuing), and a hung endpoint is bounded by curl --max-time, which must be well below the 60-second cron period. One honest caveat first: one probe a minute for five minutes is only 5 samples, and with fewer than 100 samples the nearest-rank p99 is simply the maximum.

The script

#!/usr/bin/env bash
# Run from cron every minute: * * * * * /usr/local/bin/http-probe.sh
set -euo pipefail

URL=${URL:-http://127.0.0.1:8080/health}
DIR=${DIR:-/var/lib/http-probe}
OUT=${OUT:-/var/lib/node_exporter/textfile/http_probe.prom}
WINDOW=${WINDOW:-300}                 # seconds of samples to keep
MAX_TIME=${MAX_TIME:-10}              # hard cap per request, well under the 60 s cron period

mkdir -p "$DIR" "$(dirname "$OUT")"
exec 9> "$DIR/lock"
flock -n 9 || { echo "previous run still active, skipping" >&2; exit 0; }

now=$(date +%s)
read -r code secs < <(curl -sS -o /dev/null --connect-timeout 3 --max-time "$MAX_TIME" \
                           -w '%{http_code} %{time_total}\n' "$URL" 2>> "$DIR/curl.err" || true)
code=${code:-000}; secs=${secs:-$MAX_TIME}
ok=0; [[ $code == 2* ]] && ok=1
printf '%s %s %s\n' "$now" "$secs" "$ok" >> "$DIR/samples"

# keep only the last WINDOW seconds
awk -v cut=$((now - WINDOW)) '$1 > cut' "$DIR/samples" > "$DIR/samples.new"
mv "$DIR/samples.new" "$DIR/samples"

# nearest-rank p99 over the successful samples: rank = ceil(0.99 * n) = floor((99n + 99) / 100)
p99=$(awk '$3 == 1 { print $2 }' "$DIR/samples" | sort -n |
      awk '{ lat[NR] = $1 } END { if (NR) print lat[int((99 * NR + 99) / 100)] }')
read -r total good < <(awk '{ t++; g += $3 } END { print t + 0, g + 0 }' "$DIR/samples")

{
  if [[ -n $p99 ]]; then               # no successful sample: omit the series instead of writing 0
    echo "# TYPE http_probe_latency_p99_seconds gauge"
    echo "http_probe_latency_p99_seconds $p99"
  fi
  echo "# TYPE http_probe_samples gauge"
  echo "http_probe_samples $total"
  echo "# TYPE http_probe_success_ratio gauge"
  awk -v g="$good" -v t="$total" 'BEGIN { printf "http_probe_success_ratio %.3f\n", g / t }'
  echo "# TYPE http_probe_last_run_timestamp_seconds gauge"
  echo "http_probe_last_run_timestamp_seconds $now"
} > "$OUT.tmp"
mv "$OUT.tmp" "$OUT"                   # same directory, so the rename is atomic

Design points:

• Timeouts: --connect-timeout 3 bounds the TCP connect, --max-time bounds the whole request. Worst case a run lasts MAX_TIME seconds, so the next cron tick finds the lock free. A failed or timed-out probe is recorded with ok=0; the latency curl reports for it (about the timeout) is excluded from the percentile so a hang does not masquerade as a "fast failure".
• Lock: exec 9> lock; flock -n 9. -n means non-blocking. If a run is still going, the new one logs one line to stderr and exits 0. Blocking instead would queue runs behind a hung one and pile up processes.
• Rolling window: awk '$1 > cutoff' rewrites the samples file to a temp file and mvs it back. Only the lock holder writes, so there is no concurrent writer to lose data to.
• Percentile: sort the latencies numerically and take the value at rank ceil(0.99 * n), computed in integer arithmetic as floor((99n + 99) / 100) to avoid floating-point surprises. With n = 6 the rank is 6 (the maximum); you need n of at least 100 before p99 can differ from the maximum. To get a real p99 from a 5-minute window, take 20 probes per run (100 samples), or export a histogram and let the collector compute quantiles.
• Publishing: the collector reads files matching *.prom in its directory. The script writes http_probe.prom.tmp and renames it, so the collector never sees a half-written file. If there is no successful sample the p99 series is omitted rather than written as 0, because 0 would read as "very fast". Alert on absent() for that series and on time() - http_probe_last_run_timestamp_seconds growing, which also catches cron itself dying.
• Stderr from curl goes to a file; otherwise cron mails the error every minute while the endpoint is down.

Reading the dense lines

The metrics file. Each metric is a line name value, optionally preceded by a # TYPE name gauge line. A gauge is a metric that can go up and down (a latency or a ratio), unlike a counter, which only ever increases. After the seeded run below, the whole file reads:

# TYPE http_probe_latency_p99_seconds gauge
http_probe_latency_p99_seconds 0.900
# TYPE http_probe_samples gauge
http_probe_samples 6
# TYPE http_probe_success_ratio gauge
http_probe_success_ratio 1.000
# TYPE http_probe_last_run_timestamp_seconds gauge
http_probe_last_run_timestamp_seconds 1791274549

The timestamp is the run's epoch time and differs on every run. absent(x) is the Prometheus query function that returns 1 when the series x does not exist at all, which is how an alert notices that the p99 line has been omitted.

The curl line. curl -sS -o /dev/null --connect-timeout 3 --max-time "$MAX_TIME" -w '%{http_code} %{time_total}\n' "$URL": -s silences the progress meter and -S still shows errors, -o /dev/null throws the response body away, and -w prints a format after the transfer, here the status code and the total time in seconds. Executed against a local server it printed one line, 200 0.000501 (the time varies per run). read -r code secs < <(...) then splits that line on whitespace into the two variables, giving code=200 secs=0.000238 in a second run, and ok becomes 1 when $code starts with 2.

The awk one-liners. awk -v cut=150 '$1 > cut' passes a shell value into awk with -v and prints every line whose first field is greater than it, which is the 300-second window filter (on the lines 100 0.1 1, 250 0.2 1, 400 0.3 1 it keeps the last two). In the percentile stage, awk '{ lat[NR] = $1 } END { ... }' stores each input line in an array indexed by NR, the running line number; the END block runs once after the last line, when NR equals the number of lines, and picks one element.

The rank by hand. Take three successful latencies 0.9, 0.1, 0.2. Sorted numerically they are 0.1 0.2 0.9, so n = 3. The nearest rank is ceil(0.99 x 3) = ceil(2.97) = 3, the third value, 0.9. The integer form gives the same number: floor((99 x 3 + 99) / 100) = floor(396 / 100) = 3. Other sizes, computed both ways: n = 6 gives rank 6, n = 100 gives rank 99 (the second largest), n = 101 gives rank 100.

The cron schedule. * * * * * is five time fields (minute, hour, day of month, month, day of week) with a * in each, meaning every value of each: the job runs every minute.

Executed test

The test starts a small Python HTTP server in the container (/fast answers at once, /hang sleeps 30 seconds) and exercises three cases: a seeded window where old samples must be pruned and the slow sample must set p99, a hung endpoint, and an overlapping run.

The server used by the test (saved as server.py):

# server.py: /fast answers at once, /hang never answers within the probe's time cap
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/hang":
            time.sleep(30)          # simulate an endpoint that does not answer in time
        self.send_response(200)
        self.send_header("Content-Length", "2")
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, *args):   # keep the demo output clean
        pass

ThreadingHTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

exec 2>&1
set -u
chmod +x http-probe.sh
python3 server.py & sleep 1
export DIR=/tmp/probe-demo OUT=/tmp/probe-demo/out/http_probe.prom
rm -rf "$DIR"; mkdir -p "$DIR"

echo '--- seeded window: 6 samples older than 300 s, 5 recent (one slow), then one real run'
now=$(date +%s)
for age in 900 800 700 600 500 400; do echo "$((now - age)) 9.000 1"; done > "$DIR/samples"
echo "$((now - 240)) 0.100 1
$((now - 180)) 0.120 1
$((now - 120)) 0.110 1
$((now - 60)) 0.150 1
$((now - 30)) 0.900 1" >> "$DIR/samples"
wc -l < "$DIR/samples" | sed 's/^/samples before: /'
URL=http://127.0.0.1:8080/fast ./http-probe.sh; echo "exit $?"
wc -l < "$DIR/samples" | sed 's/^/samples after: /'
grep -v -e '^#' -e last_run "$OUT"

echo '--- endpoint hangs (MAX_TIME=2), empty history'
rm -rf "$DIR"; mkdir -p "$DIR"
URL=http://127.0.0.1:8080/hang MAX_TIME=2 ./http-probe.sh; echo "exit $?"
cut -d' ' -f3 "$DIR/samples" | sed 's/^/ok flag recorded: /'
grep -v -e '^#' -e last_run "$OUT"

echo '--- a second run starts while the first is still waiting on a hung endpoint'
rm -rf "$DIR"; mkdir -p "$DIR"
URL=http://127.0.0.1:8080/hang MAX_TIME=3 ./http-probe.sh & first=$!
sleep 1
URL=http://127.0.0.1:8080/fast ./http-probe.sh; echo "second run exit $?"
wait "$first"; wc -l < "$DIR/samples" | sed 's/^/samples recorded: /'
kill %1 2>/dev/null

--- seeded window: 6 samples older than 300 s, 5 recent (one slow), then one real run
samples before: 11
exit 0
samples after: 6
http_probe_latency_p99_seconds 0.900
http_probe_samples 6
http_probe_success_ratio 1.000
--- endpoint hangs (MAX_TIME=2), empty history
exit 0
ok flag recorded: 0
http_probe_samples 1
http_probe_success_ratio 0.000
--- a second run starts while the first is still waiting on a hung endpoint
previous run still active, skipping
second run exit 0
samples recorded: 1

Reading it:

• Eleven lines before the run (six older than 300 s, five recent) and six after: the six old ones were trimmed, and the new real sample joined the five recent ones. The p99 is 0.900, the slow sample, which is the maximum of 6 values, exactly as the rank formula predicts.
• With the endpoint hung and a 2-second cap, the run ended normally with exit 0, recorded one failed sample (ok flag recorded: 0) and published a success ratio of 0.000 with no p99 line.
• The second run, started while the first was waiting on the hung endpoint, printed "previous run still active, skipping" and recorded nothing; the first run later recorded its single failed sample.

Trade-offs

• A shell probe measures from one host through one network path; it complements, not replaces, a collector-side blackbox probe.
• curl.err is appended to on every failing minute and never trimmed; rotate it (logrotate) or truncate it in the script, otherwise a long outage fills the disk slowly.
• time_total includes DNS lookup and TLS handshake. Use time_starttransfer if you want time to first byte.
• Sample timestamps use the host clock; if it steps backward (NTP correction), the cutoff logic keeps stale samples a little longer, which is harmless at a 300-second window.

Direct answer

Treat this like triaging technical debt: first classify every panel by whether it maps to an actual triage decision, aggressively cut or consolidate everything that doesn't, and validate the result against real on-call engineers before calling it done, not just against your own judgment.

Approach

Step 1: audit and classify

Talk to two or three people who've actually used the dashboard during an incident. For every panel, tag it: critical (maps to a golden signal or SLO), useful for drill-down, rarely used, or dead weight with no clear owner or purpose anyone can name.

Step 2: consolidate around golden signals

Replace many single-entity panels with a small set of golden-signal panels (rate, errors, latency, saturation) plus a "top N by errors or latency" table instead of one panel per service or endpoint.

Step 3: use template variables for the long tail

Instead of one panel per service or region, add a dashboard variable and let the engineer select what to drill into. The front page stays small; context-switching happens on demand rather than by scrolling past 35 panels that don't apply to the current incident.

Step 4: redesign layout around the incident workflow

Row 1: is it broken (SLO burn, error rate, active alerts). Row 2: how broken (latency, saturation, dependency health). Row 3 and below: drill-down detail. This mirrors how someone actually works an incident, overview first, detail on demand, instead of a flat grid of 40 equally-weighted panels.

Step 5: validate, don't just ship

Run the new version past on-call for a rotation or two, or a tabletop exercise against the last few real incidents, and explicitly ask whether this panel set would have gotten them to root cause faster. Cut anything that doesn't survive that test.

Worked example

A concrete, honest instance of the consolidation pattern: suppose 15 of the 40 panels are the same latency chart repeated once per microservice. Replacing those 15 with one templated latency panel (a service selector variable) plus one "top 5 services by p99" ranked table removes 13 panels (15 down to 2) while preserving, and arguably improving, the same information, since the ranked table surfaces the worst offender automatically instead of requiring a scroll through 15 charts to spot it. The same consolidation pattern applied to a repeated error-count-per-service panel family would remove a comparable number. The exact final panel count for the full 40 depends on how much of the original sprawl is this kind of per-entity repetition versus genuinely distinct signals, which is precisely what step 1's audit is for.

Trade-offs and pitfalls

• Cutting aggressively without checking with on-call can remove a panel that's rarely used but critical for one specific failure mode, like a replication-lag panel that's silent 99% of the time but is the first thing needed during that one incident type. Validate against past incidents, not just usage frequency.
• Template variables trade a small amount of always-visible context for a click. That's usually the right trade for an overview dashboard, but a signal that's genuinely page-worthy, like overall SLO burn, should stay pinned rather than sit behind a variable.
• Common wrong turn: fixing the dashboard once without establishing an ownership or review process, which is how it reached 40 panels in the first place. Without governance, the same sprawl recurs within a year.
• Don't conflate "unused" with "unnecessary." Some panels are unused specifically because the failure mode they'd catch hasn't happened yet, not because the panel is dead weight.

Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and Cluster Autoscaler (CA) operate on three different axes: HPA changes how many pod replicas exist, VPA changes how much CPU and memory each pod requests, and CA changes how many nodes exist, reacting only to pods it cannot schedule or nodes it judges empty. The conflict case that comes up in practice is running HPA and VPA on the same workload against the same signal: VPA raising a pod's CPU request lowers that pod's CPU utilization percentage (same usage, bigger denominator), which can make HPA think load dropped and hold off scaling out right when VPA's own resizing is also making pods bigger and harder to schedule, a feedback loop that shows up as scheduling failures and node churn rather than a clean scaling curve.

How the three interact

CA is the last-mile capacity provider: it doesn't look at metrics at all, only at whether pods are stuck Pending for lack of room, or whether a node's real utilization has stayed below a threshold (the default scale-down-utilization-threshold is 50%, and a node has to stay under it for the default scale-down-unneeded-time of 10 minutes before CA considers removing it, specifically to avoid thrashing on a brief dip).

The conflict, concretely

HPA is configured to scale out at 70% CPU. VPA (in a mode that actually applies changes, not just recommends) sees real CPU usage per pod is high and raises the CPU request. Because HPA's CPU% is usage divided by request, that same usage now reads as a lower percentage against the new, larger request, so HPA either stops scaling out or, worse, in the transitional moment while pods are being resized, ends up racing VPA: HPA adds replicas for load that VPA is simultaneously deciding needs bigger pods, and the result is more, bigger pods than the workload needed, tripping the Cluster Autoscaler into rapid node adds and (once things settle) removals.

Resolving it

1. Prefer disjoint signals. Point HPA at a metric VPA's resizing doesn't move, such as requests-per-second or a custom queue-depth metric, instead of CPU percentage, so the two don't chase the same number.
2. Constrain what VPA is allowed to touch. VPA supports scoping its containerPolicies to specific resources (for example, letting it manage memory but not CPU) via controlledResources, so it can right-size the axis HPA isn't scaling on.
3. Use VPA's non-disruptive modes deliberately. VPA's update modes are Off (recommendation only, nothing applied), Initial (sets requests only at pod creation, never touches a running pod), and Recreate (evicts and re-creates the pod to apply a new recommendation); Auto still exists in older docs but was deprecated as of VPA 1.5 and is now just an alias for Recreate. A newer InPlaceOrRecreate mode (beta) attempts to resize a running pod's resources without evicting it at all, riding on Kubernetes' in-place pod resize feature (alpha in 1.27, beta in 1.33, stable/GA in 1.35), falling back to evict-and-recreate only when an in-place resize genuinely isn't possible. Running VPA in Off mode and feeding its recommendations into a review step (a dashboard, a PR) rather than letting it apply changes live sidesteps the feedback loop entirely while still getting the sizing signal.
4. Tune HPA's stabilization. behavior.scaleDown.stabilizationWindowSeconds on the autoscaling/v2 HPA object smooths out reactions to a transient metric shift instead of scaling on every sample.
5. Give Cluster Autoscaler room to be boring. Set realistic scale-up/scale-down delays, use PodDisruptionBudgets (PDB, an object that caps how many pods of a set can be unavailable at once) so CA's scale-down eviction can't take out too many replicas of the same workload at once, and use the cluster-autoscaler.kubernetes.io/safe-to-evict: "false" pod annotation for anything that must never be moved automatically.

Trade-offs

• Running HPA and VPA on the same resource axis for the same workload is the anti-pattern, not a supported combination to be tuned into stability; the fix is almost always to separate the axes or the signal, not to add more stabilization windows on top.
• InPlaceOrRecreate reduces disruption but is a newer, less battle-tested code path than plain Recreate; teams with strict availability requirements may reasonably choose the well-understood evict-and-recreate behavior over the newer in-place path until they've validated it against their own failure modes.
• VPA is generally the better fit for workloads that can't scale horizontally at all (a single-replica stateful service, a batch job), where HPA has nothing to do in the first place.

A hotfix is a fast, narrowly-scoped patch to restore correct behavior immediately; a long-term fix addresses the underlying design or process issue, typically taking longer and carrying lower risk of introducing a new problem.

When to choose which

Choose a hotfix when user impact is active and ongoing, the fix is small and well-understood (low blast radius), and a proper fix would take meaningfully longer than the acceptable time to restore service. Choose to invest in the long-term fix directly when there's no active user impact yet (a bug caught before it ships, or a near-miss), or when the "quick" fix would itself be risky/complex enough that it's not actually faster or safer than doing it right.

Risks of hotfixes

They frequently trade correctness for speed in a way that creates hidden technical debt (a special-cased branch nobody remembers the reasoning for), can mask the actual root cause (the symptom goes away, but the underlying condition that caused it is still there and can resurface differently), and sometimes introduce a new, narrower bug because they were reviewed and tested less thoroughly than a normal change under time pressure.

Communication and documentation after a hotfix

Document, at minimum: what the hotfix specifically does and does not address, why it was chosen over a full fix, and a tracked follow-up item for the durable fix with an owner and rough timeline, communicated to the team (not just left in a commit message), since an undocumented hotfix is the most common way "temporary" becomes permanent by default.

Trade-offs and pitfalls

The single biggest failure mode across roles (SRE, general engineering, systems engineering) is the same: a hotfix applied under pressure with no tracked follow-up quietly becomes the permanent state of the system, carrying its narrower risk profile forward indefinitely instead of the brief window it was meant for.

The idea in plain words

A severity framework is the written set of levels (for example SEV1 to SEV4) and the rules for choosing one, which decides who is paged and how quickly. To judge whether it works you need measures of three things: whether the label was right, whether it was chosen quickly, and whether the response that followed was effective. A single number can be gamed (declare everything low and "time to declare" looks great), so the five KPIs below are chosen to check each other. KPI means key performance indicator. MTTR is mean time to resolve. Percentile such as p90 means the value that 90% of cases are at or below. A counter-metric is a second measure placed beside a KPI to expose the way the KPI could be gamed. Under-triage means an initial severity less severe than the final one, and over-triage means more severe. MTTR is defined as a mean, but the KPIs and the worked example below report the median and p90 of resolution times instead, because a few very long incidents distort a mean; if your organisation's convention is the mean, compute and show it beside the median.

Start with KPIs 1, 2 and 5, which come from the severity history alone and show calibration fastest; the Judgment section explains why. The target values are starting points to recalibrate against your first two quarters of data. They are not industry standards.

The five KPIs

How each is instrumented

The incident tool stores a severity history (an append-only list of events, meaning entries are only ever added and never edited or deleted, each holding level, time, who and reason) plus fixed fields: impact start, detected, acknowledged, declared, mitigated, resolved, final severity and reason. Impact start is filled in from metrics during review, not from when someone noticed. The dashboard reads those fields with these panels: a trend of KPI 1 with under- and over-triage as two lines; a distribution (a histogram, which counts how many incidents fall into each range of minutes, plus the median and p90) of KPI 2 per severity; a table of SEV1 pages with their final severity for KPI 3; a box plot (a box from the 25th to the 75th percentile with a line at the median and whiskers for the spread) of resolution times per final severity for KPI 4; and a stacked bar (one bar per month, split into upgrades and downgrades) for KPI 5. Collection is a form field that the incident commander cannot skip, not a spreadsheet filled in a week later.

Worked example (hypothetical, ten incidents)

1. Correct initial severity. Initial equals final in incidents 2, 3, 6, 7, 8 and 10, which is 6 of 10 = 60%. Under-triage (initial less severe, so a bigger number than final) is incidents 1 and 5, which is 2 of 10 = 20%. Over-triage is incidents 4 and 9, which is 2 of 10 = 20%. The 60% sits below the 85% target.
2. Time to declare. Sorted: 3, 4, 4, 5, 6, 8, 9, 10, 12, 25. The median is the average of the 5th and 6th values, (6 + 8) / 2 = 7 minutes. The slowest case, 25 minutes, is incident 5, which was an under-triage, so slow declaration and wrong label occurred together.
3. SEV1 page false positives. Pages are incidents with initial severity 1: incidents 4, 7 and 9, which makes 3 pages. Final severity is lower for 4 and 9, so 2 of 3 = 66.7%. With 3 pages, a single different call moves the rate by 33 points, so a real dashboard needs a rolling window of a quarter or more before this figure means anything.
4. MTTR by final severity. Final SEV1 (incidents 1, 5, 7): 95, 120, 60, median 95 minutes. Final SEV2 (incidents 3, 4, 8, 10): sorted 70, 150, 180, 200, median (150 + 180) / 2 = 165 minutes. Final SEV3 (incidents 2, 6, 9): 40, 35, 25, median 35 minutes. Medians are used because one very long incident distorts a mean. Reading the result: the final SEV1 median of 95 minutes misses the 60 minute target, so SEV1 resolution is the slowest-to-fix gap in this set (the mean of 95, 120 and 60 is 91.7, which says the same), while the final SEV2 median of 165 minutes is inside the 4 hour (240 minute) target.
5. Reclassifications. Incidents 1, 4, 5 and 9 changed, which is 4 of 10 = 40% against a 20% target. There were 1 + 1 + 1 + 2 = 5 changes in total, 0.5 per incident. Of those, 2 were upgrades (one each in incidents 1 and 5) and 3 were downgrades (one in incident 4 and two in incident 9).

Reading the set together: a 60% correct rate, 20% under-triage and 40% reclassification point to a calibration problem, not a speed problem (the median time to declare is acceptable). The next step is a root-cause review of incidents 1 and 5, as in the severity postmortem process. The data are ten incidents, so these are illustrations of the computation, not conclusions.

Judgment

Start by instrumenting KPIs 1, 2 and 5, because they come from the severity history alone and show calibration fastest. Add 3 and 4 once there are enough incidents to read them. Review quarterly, with medians and p90, and show the executives only KPIs 1 and 2 plus the under-triage rate; the rest stay with SRE and on-call leaders.

Common mistakes

• Reporting one blended MTTR across all severities.
• Setting a false-positive target of zero.
• Using a mean on small, skewed samples.
• Letting KPI targets become individual performance goals, which pushes people to under-declare.

Direct answer

The mechanics are the same regardless of what is being changed: stand up the new version alongside the old one, shift traffic to it gradually (or all at once for blue-green), watch metrics against a defined threshold before promoting further, and have an automated way to shift traffic back if those metrics degrade. What differs by change type is what "the new version" means and which metric actually tells you it is safe, so the mechanics come first below, then how they change for a few different kinds of infrastructure change.

Structured elaboration

Traffic shifting mechanics

• Canary: route a small percentage of traffic to the new version (for example 5%), hold, then ramp (25%, 50%, 100%) with a wait window at each step.
• Blue-green: fully validate the new environment out of band, then cut traffic over in one step (DNS, load balancer listener swap, or weighted routing swap to 100%). Rollback is just swapping back.

What "the new version" and "the signal to watch" are, by change type

Stateless versus stateful

For a stateless service, this is close to mechanical: deregister the old instance from the load balancer, let connections drain, terminate it, and the new instance takes over with no coordination needed between old and new.

For a service with state, for example one backed by a database or holding session or connection state, the "new version" cannot simply replace the old one at will:

• Deregistration has to wait for graceful connection draining, not just for the health check to fail, or you drop in-flight work.
• If the state itself is what is changing (the DB-class case above), the new version has to be replication-caught-up before it sees real traffic, and you need a plan for what happens to writes that land during the cutover window.
• The old and new versions may be reading or writing the same underlying data store, so canary steps need to account for that shared state, not just for the request path.

Worked example

A concrete case: shifting 5% of production traffic on an ALB to a target group fronting instances built from a new AMI, before fully cutting over.

resource "aws_lb_target_group" "green" {
  name     = "app-green"
  port     = 443
  protocol = "HTTPS"
  vpc_id   = var.vpc_id
  health_check {
    path                = "/health"
    healthy_threshold   = 3
    unhealthy_threshold = 2
    timeout             = 2
  }
}

resource "aws_lb_listener_rule" "canary" {
  listener_arn = aws_lb_listener.app.arn
  priority     = 100

  action {
    type = "forward"
    forward {
      target_group {
        arn    = aws_lb_target_group.green.arn
        weight = 5
      }
      target_group {
        arn    = aws_lb_target_group.blue.arn
        weight = 95
      }
    }
  }

  condition {
    path_pattern { values = ["/*"] }
  }
}

Watch the green target group's 5xx rate and p95 latency for a fixed window (say 15 minutes). If it stays within the same range as blue, bump the weight to 25 and repeat; if 5xx rate or latency breaches the threshold at any step, a second terraform apply (or a CI job wrapping it) sets green's weight back to 0 and blue keeps serving everything.

flowchart TD
  A[Provision green environment via Terraform] --> B[Register in target group, run health checks]
  B --> C[Shift 5 percent traffic to green]
  C --> D{Metrics within SLO?}
  D -- Yes --> E[Ramp to 25 percent, then 50 percent]
  E --> F{Still within SLO?}
  F -- Yes --> G[Shift 100 percent traffic, decommission blue]
  F -- No --> H[Roll back: weight to 0 percent, blue keeps serving]
  D -- No --> H

Trade-offs & pitfalls

• The biggest mistake is treating every infra change like an app deploy and watching only 5xx rate. A security-group rule change will not show up as a 5xx, it shows up as connections that never complete; a DB-class change will not show up in the app's error rate until replication lag has already gotten bad. Pick the signal that matches what actually breaks for that change type.
• Canary steps for stateful changes need explicit wait-for-caught-up gates, not just a timer; ramping traffic to a replica that has not finished catching up just moves the failure into user-visible latency or stale reads.
• Automating the rollback path, not just the promotion path, is what actually buys "close to zero downtime." If pulling the plug requires someone to remember the right terraform apply under pressure, the safety only exists on paper.

Choosing SLOs for a data service you actually built starts from the specific business need the service exists to serve, then works down to the quantitative thresholds, not the other way around.

Structured elaboration

Success metrics worth defining: throughput (requests or records served per unit time, confirming the service can sustain expected load), p95/p99 latency (tail experience, since a data service's consumers often run bursty batch queries where the tail matters as much as, or more than, the median), data freshness (how current the served data is relative to its source), and error rate (proportion of requests failing outright). Thresholds should be chosen against what the CONSUMING use case actually requires: a dashboard refreshed hourly can tolerate looser freshness than a real-time alerting pipeline consuming the same underlying data service.

Worked example

For a production data service feeding both an hourly-refreshed executive dashboard and a near-real-time fraud-detection pipeline: p95 latency < 200ms (chosen based on what the fraud pipeline's own SLA requires, the tighter of the two consumers), freshness < 5 minutes (again driven by the tighter consumer, even though the dashboard alone would have tolerated much looser freshness), error rate < 0.1%. Validation during launch: a soft-launch period against a subset of real consumer traffic, checking that the chosen thresholds are both ACHIEVABLE (the service can actually sustain them under real load) and SUFFICIENT (the tightest consumer's actual needs are genuinely met, confirmed by that consumer's own downstream metrics staying healthy).

Trade-offs and pitfalls

Setting one blended threshold to satisfy the AVERAGE of all consumers' needs, rather than the TIGHTEST consumer's actual requirement, is a common mistake: it can look reasonable on paper while quietly under-serving the one consumer (often the most business-critical one) whose needs were actually the tightest. It's also worth revisiting these thresholds as new consumers are added over the service's life, since a threshold validated against the original set of consumers may become insufficient once a new, more demanding consumer starts depending on the same service without anyone re-checking whether the existing SLO still covers their needs.

Direct answer

Put a single-writer sequencer per instrument at the centre: every update for an instrument passes through one process that stamps it with a gap-free, per-instrument sequence number, which is what "strict ordering" means in practice. From the sequencer, updates go out on UDP multicast (one packet copied by the network switches to every subscriber) inside the data centre, with a second, identical A/B feed on separate network paths so one lost packet is usually recovered from the other feed with no round trip. Missed updates are recovered from retransmission servers holding a 60-second in-memory ring buffer, and from snapshots when a subscriber is too far behind. Slow subscribers never slow the sequencer: they drop and recover. The "at-most-once" requirement (downstream applies each update no more than once, even though the transport itself may redeliver it) is met on the subscriber side, by applying each sequence number at most once.

Terms

• Instrument: a tradable security, such as a stock, option, or futures contract. Every message in this system is scoped to one instrument, which is what makes the system partitionable.
• Order book: the live list of buy and sell orders waiting to trade for one instrument, ordered by price. It is what a matching engine maintains and what this system broadcasts changes to.
• Matching engine: the system that receives buy and sell orders for an instrument and pairs (matches) them into trades, updating the order book.
• Book change: one update to the order book: an order was added, modified (price or quantity changed), deleted (cancelled), or resulted in a trade (matched against another order).
• Ring buffer: a fixed-size buffer that overwrites its oldest entries once full, used here to hold the last 60 seconds of messages for replay without unbounded memory growth.
• Multicast group: a network address that, once a device subscribes to it, causes network switches to deliver a copy of every packet sent to that address, without the sender addressing each recipient individually.
• Hot standby: a second instance that runs the same computation as the primary in real time but does not publish, so it can take over instantly if the primary fails.
• A/B feed / arbitration: sending the same data twice over two independent network paths (A and B); arbitration is picking whichever copy of a given message arrives first and discarding the duplicate.

Requirements and the one that needs interpreting

• Strict ordering per instrument (not globally: ordering across instruments is not required, which is what makes the system partitionable).
• Sub-10 ms tail latency (p99, the 99th percentile) within a region.
• Replay of missed updates for up to 1 minute.
• At-most-once delivery for downstream processing, and 100,000 updates per second.

"At-most-once" normally means "may be lost, never duplicated", while "replay missed updates" means "not lost". Taken together, the sensible reading is: the transport may deliver a message more than once (A/B feeds and replays guarantee duplicates), but downstream processing must apply each update at most once, and gaps must be repairable within 60 seconds. Each subscriber keeps last_applied_seq per instrument and discards anything at or below it. I would confirm this reading with the interviewer; if they truly meant "fire and forget, never replay", the retransmission tier disappears and the design gets simpler, but the order book would silently diverge after any loss.

Components

flowchart LR
  ME[Matching engines] --> SEQ[Sequencers by instrument partition]
  SEQ --> MCA[Multicast feed A]
  SEQ --> MCB[Multicast feed B]
  SEQ --> RING[(Retransmission ring buffer 60 s)]
  SEQ --> SNAP[(Snapshot service)]
  MCA --> SUB[Subscribers]
  MCB --> SUB
  SUB -. gap request .-> RING
  SUB -. resync .-> SNAP
  MCA --> EDGE[TCP fan-out tier for remote subscribers]

1. Ingest: matching engines (which already process orders per instrument) emit book changes (add, modify, delete, trade). Each engine owns a set of instruments.
2. Sequencer: instruments are partitioned (for example by instrument id) across sequencer processes. Within a partition, one thread stamps (instrument_id, seq) and writes the message to the feeds and to the ring buffer. Single-writer means no locks and no coordination on the hot path.
3. Transport: UDP multicast, one multicast group per partition, so a subscriber that only cares about some instruments joins only those groups.
4. Retransmission servers: listen to the feed, store the last 60 seconds, answer "resend instrument X, sequences 1001 to 1010".
5. Snapshot service: maintains the full book per instrument with the sequence number it reflects, so a subscriber that joins late or falls far behind can load the book and continue from the stream.
6. Fan-out tier (TCP or WebSocket) for subscribers outside the data centre that cannot receive multicast.

Why multicast: the bandwidth arithmetic

With 64-byte messages (a compact binary format: message type, instrument id, sequence, price, quantity, side, timestamp):

• One subscriber taking the full feed: 100,000 x 64 bytes = 6.4 MB/s = 51.2 Mbit/s.
• 2,000 subscribers over unicast (a separate copy per subscriber): 51.2 Mbit/s x 2,000 = 102.4 Gbit/s leaving the publisher. That is a hardware problem and a latency problem, because the last copy is sent long after the first.
• Multicast: the publisher sends 51.2 Mbit/s once; switches replicate it. Latency is the same for every subscriber.

Averages understate market data: bursts at the open or on news can be several times the average. I would provision the feed and every subscriber path for at least 5x (500,000 updates per second, 256 Mbit/s), and treat that multiplier as an assumption to confirm with historical peak data.

Sequencing and gap detection

• Each message carries (instrument_id, seq); sequences are per instrument, gap-free, starting at 1 per trading session.
• Packets also carry a partition-level sequence (a per-instrument sequence only reveals a gap when the next message for that same instrument arrives, and if that instrument stays quiet there may be no next message for a long time) so a subscriber detects a lost packet even for an instrument that then goes quiet. A periodic heartbeat per partition carries the latest sequence for the same reason.
• Subscriber logic: receive from A and B, take whichever copy arrives first, discard the duplicate by sequence. If a gap persists beyond a short window (say 1 ms, to absorb A/B skew), request the range from a retransmission server. If the gap is older than 60 s, or the request fails, load a snapshot and replay from its sequence.

Storage for replay

60 seconds x 100,000 updates per second = 6,000,000 messages x 64 bytes = 384 MB. At the 5x burst assumption it is 1.92 GB. This is an in-memory ring buffer, indexed by (partition, seq) to an offset, on at least two retransmission servers. No disk is on the replay path. Separately, the full session is written to durable storage asynchronously for audit and regulatory record-keeping, which is off the latency path.

Latency budget for the p99 under 10 ms

These are allocations, not measurements: each hop gets a histogram and an alert on its p99, so a regression is attributable to a hop. Keeping headroom matters because a gap repair costs one round trip to a retransmission server, and it must fit within the 10 ms target for the p99 to hold. Timestamps at each hop need clock synchronization much finer than the budget; PTP (Precision Time Protocol, hardware-assisted clock sync) is the standard choice.

Backpressure

The sequencer and the multicast feed never wait for any subscriber. Backpressure from one slow reader must not become latency for thousands of others.

• A slow multicast subscriber overflows its own socket buffer (the OS's fixed-size holding area for incoming network data before the application reads it) and loses packets (the OS drops new packets once that buffer is full, since there is nowhere to put them); its gap logic recovers via retransmission, and if it is persistently slow, it is disconnected from retransmission (rate-limited per subscriber) and told to use snapshots.
• In the TCP fan-out tier, each subscriber has a bounded outbound queue. On overflow, the tier either conflates (sends the latest book state for each instrument instead of every intermediate change, acceptable for display clients) or disconnects the subscriber with a "resync from snapshot" instruction (required for trading clients that need every change).
• Retransmission servers are rate-limited per subscriber, so one broken client cannot use the replay tier to hurt everyone else's recovery.

Operational considerations

• Sequencer failover: a hot standby consumes the same inputs and computes the same sequence numbers deterministically, but does not publish. On failure, it takes over publishing from the next sequence. A paused primary is dangerous precisely because it can resume: a garbage-collection pause, a frozen VM, or a network partition can make the standby correctly promote itself while the old primary is still running, and once unpaused the old primary still believes it is the leader and keeps publishing. Fencing (a monotonically increasing leadership epoch in every packet, so subscribers ignore the old primary if it wakes up) prevents two sequencers publishing the same numbers.
• Start of day: sequences reset per session; subscribers load the opening snapshot.
• Monitoring: per-hop p99, gap rate per subscriber, retransmission requests per second, A/B arbitration stats (how often B saved you), snapshot requests.
• Capacity tests with recorded peak days replayed at 5x.

Trade-offs and pitfalls

• Kafka or a similar log broker gives replay and ordering per partition for free, but adds a disk-backed, replicated write before delivery. That is fine for downstream analytics consumers (they can subscribe to a copy) and wrong for the sub-10 ms path.
• Global ordering across instruments would force one sequencer for everything and cap throughput; per-instrument ordering is what the requirement asks for.
• Treating duplicates as the enemy at the transport: A/B feeds deliberately create duplicates; dedup belongs at the subscriber, by sequence.
• What would change the design: if subscribers were mostly across the internet (retail apps), multicast is unavailable and the design becomes a TCP/WebSocket fan-out tree with conflation, and "sub-10 ms" would have to be redefined as within our network.

Direct answer

At extreme scale you cannot rely on a single stampede defense. Combine request coalescing (only one request repopulates a hot key while others wait), background/proactive refresh before expiry, probabilistic early expiration, jittered time-to-live (TTL) values, and origin rate limiting as a last-resort backstop, coordinated so all application instances agree on who is allowed to refresh a given key at once.

Structured elaboration

• Request coalescing (singleflight): on a cache miss, the first request acquires a short-lived lock (e.g., SETNX in Redis) for that key and fetches from origin; concurrent requests for the same key either block briefly on a notification channel (pub/sub or polling with backoff) or serve a stale value if one exists. This bounds concurrent origin load per key to roughly one in-flight fetch, regardless of instance count, because the lock lives in the shared cache, not in any one process.
• Probabilistic early expiration: instead of a hard expiry, each read close to TTL end recomputes with a small, increasing probability (a common formula is $P(\text{refresh}) = e^{-\beta \cdot (t_{\text{expiry}} - t_{\text{now}}) / \delta}$ where $\delta$ is the time it took to compute the value and $\beta$ tunes aggressiveness). This spreads refreshes across many requests instead of concentrating them at the exact expiry instant.
• Jittered TTLs: add randomized jitter (e.g., base TTL plus/minus 10 to 20 percent) so keys written around the same time do not all expire in the same millisecond, which is what turns an ordinary cache miss into a correlated stampede across thousands of keys at once (a cache avalanche).
• Background/prewarmed refresh: a scheduled worker (or the request that detects "close to expiry") refreshes hot keys proactively so the TTL rarely actually lapses for high-traffic keys; this trades a small amount of continuous background load for eliminating stampede risk on the hottest paths.
• Origin rate limiting as a backstop: even with the above, cap concurrent origin requests per key (or per origin endpoint) so a defense-in-depth failure degrades gracefully into serving stale data or a fast error instead of taking the origin down.
• Orchestrating across many app instances: the lock, the "who refreshes next" decision, and the notification of waiters must live in the shared cache (Redis) or a coordination service, not in in-process state, because coalescing only works if every instance agrees on a single winner per key.

Worked example

Say a hot key normally takes 200 ms to recompute and serves 5,000 requests per second (RPS) at peak. Without any defense, if it expires with no coalescing, the next ~1,000 requests in that 200 ms window (5,000 RPS times 0.2 s) would all miss and hit the origin simultaneously; a database that comfortably serves single-digit concurrent queries per second for that expensive query falls over. With coalescing, exactly 1 request recomputes and the other ~999 either wait ~200 ms for the notification or receive the previous (slightly stale) value immediately; origin load for that key stays at 1 concurrent request regardless of RPS.

Trade-offs and pitfalls

A lock that never expires on a crashed refresher permanently blocks that key; always set a lock TTL slightly longer than the expected recompute time, plus a fallback path that lets a waiter give up and fetch directly after a bounded wait. Serving stale-while-refreshing is a correctness trade-off, not a free win: it is the right default for read-heavy, staleness-tolerant data, and the wrong default for a low-latency-but-must-be-fresh field like an account balance. Jitter alone does not help an already-hot key that is legitimately read far more than others; that is a hot-key sharding problem, not a stampede problem, and needs a different fix (splitting the key, adding a replica-backed local cache).

Raft's leader election works by giving every follower a randomized election timeout; whichever follower's timeout fires first becomes a candidate, bumps the term number, votes for itself, and asks every other node for its vote. Randomizing the timeout is what keeps split votes rare: it makes it unlikely that two followers time out at nearly the same instant and both start competing candidacies in the same term. A short timeout detects a dead leader faster but risks more split votes and RPC churn; a long timeout is calmer but leaves the cluster leaderless longer after a real crash.

Approach

Implement the state machine directly: a follower's only action is to reset its timer whenever it hears from a leader or grants a vote. When the timer fires with no such contact, it becomes a candidate, increments its term, votes for itself, and fires RequestVote RPCs at every peer in parallel. It counts granted votes as they arrive; on a majority, it becomes leader. Any RPC response or incoming request carrying a higher term makes it step down to follower immediately; this term-monotonicity is what prevents two nodes from believing they are both leader for the same term.

type Server struct {
    mu            sync.Mutex
    id            int
    peers         []int
    currentTerm   int
    votedFor      *int
    state         string // "follower", "candidate", "leader"
    votes         int
    electionTimer *Timer
}

func (s *Server) onElectionTimeout() {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.state == "leader" {
        return // leaders don't run elections; they send heartbeats instead
    }
    s.startElectionLocked()
}

func (s *Server) startElectionLocked() {
    s.state = "candidate"
    s.currentTerm++
    s.votedFor = &s.id
    s.votes = 1 // vote for self
    term := s.currentTerm
    s.resetElectionTimer()

    for _, peer := range s.peers {
        go func(peer int) {
            resp := sendRequestVote(peer, s.id, term, s.lastLogIndex(), s.lastLogTerm())
            s.mu.Lock()
            defer s.mu.Unlock()
            if resp.Term > s.currentTerm {
                s.currentTerm = resp.Term
                s.becomeFollowerLocked()
                return
            }
            if s.state != "candidate" || term != s.currentTerm {
                return // stale response from a prior term, or we already moved on
            }
            if resp.VoteGranted {
                s.votes++
                if s.votes > (len(s.peers)+1)/2 {
                    s.becomeLeaderLocked()
                }
            }
        }(peer)
    }
}

func (s *Server) resetElectionTimer() {
    d := randomDuration(electionTimeoutMin, electionTimeoutMax)
    s.electionTimer.Reset(d)
}

func (s *Server) onAppendEntries(args AppendEntriesArgs) AppendEntriesReply {
    s.mu.Lock()
    defer s.mu.Unlock()
    if args.Term >= s.currentTerm {
        s.currentTerm = args.Term
        s.becomeFollowerLocked()
        s.resetElectionTimer() // heard from a valid leader, don't start a new election
    }
    // log-matching and append logic omitted here
    return AppendEntriesReply{Term: s.currentTerm}
}

Key points

• Persist currentTerm and votedFor to stable storage before replying to any RPC. A node that crashes and restarts without doing this could grant a second vote in a term it already voted in, which is a real safety violation, not just a liveness inconvenience.
• A candidate only votes for itself and only requests votes once it becomes a candidate; a follower that already voted for someone else this term must refuse any other RequestVote for that same term.
• Majority is computed against the full cluster size n, including the leader itself, not just the peers list:

$\text{majority} = \lfloor n/2 \rfloor + 1$

Complexity

• Message complexity: one election attempt sends O(n) RequestVote RPCs, one per peer, linear in cluster size per attempt.
• State per node is O(1) beyond the log itself: a handful of integers and a timer.
• The number of election attempts before success is not bounded in the worst case, repeated collisions are theoretically possible forever, but the randomized timeout range makes the expected number of retries small in practice, since each retry independently redraws a new random timeout, making the same two nodes colliding again increasingly unlikely round over round.

Worked example: split vote and recovery, printed trace

Three nodes N1, N2, N3, currently term 0, no leader. Each node draws its election timeout independently from the same range.

Case A, no collision, the common case: N2 draws the shortest timeout and fires first. It becomes a candidate for term 1, votes for itself, and sends RequestVote(term=1) to N1 and N3. Neither has timed out or voted this term, so both grant their vote. N2 now has 3 of 3 votes, itself plus two grants, exceeding the majority threshold of 2, so N2 becomes leader for term 1. N1 and N3 reset their timers on granting the vote, so neither starts its own candidacy.

Case B, a genuine split vote: suppose instead N1, N2, and N3 all draw timeouts close enough together that all three fire before any RequestVote arrives. Each becomes a candidate for term 1, votes for itself, and requests votes from the other two. Because each node already voted for itself in term 1 before any request from a peer arrived, each rejects the other two's requests, having already voted this term. No candidate reaches a majority: N1 has 1 vote, N2 has 1 vote, N3 has 1 vote. All three time out again and start a new election for term 2, each independently drawing a fresh random timeout. Because these new timeouts are drawn independently, the chance that all three collide again is much lower than the first time, and typically one of them fires meaningfully before the others and wins term 2 uncontested.

Edge cases

• A network partition splits the cluster into a majority side and a minority side: the minority side's nodes keep timing out and calling elections forever, incrementing the term each time, but can never reach a majority, so they correctly never elect a leader. This is deliberate: it is what prevents split-brain, at the cost of the minority side being unavailable for writes until the partition heals.
• A node that crashes and restarts must reload currentTerm and votedFor from stable storage before participating again; skipping this can let it vote twice in a term it already voted in before crashing.
• A slow node, a garbage-collection pause or a disk stall, is indistinguishable from a crashed node from its peers' point of view; both simply stop sending heartbeats, and the same election timeout mechanism reacts to both identically, which can trigger an unnecessary election when a node was merely paused, not dead.
• Election timeouts are purely local, measured intervals, a node's own clock counting elapsed time, so cross-machine clock skew, which matters a great deal for wall-clock timestamp ordering elsewhere in this topic, does not matter here.

Trade-off: short vs long election timeout

• A short timeout detects a genuinely dead leader faster, a shorter unavailability window, but increases both the chance of a split vote and the RPC and CPU overhead from repeated election attempts under network jitter.
• A long timeout keeps the cluster calmer, fewer accidental elections during transient network hiccups, at the direct cost of a longer window with no leader after a real crash, since followers simply wait longer before noticing.
• In practice the timeout range is chosen relative to the network's expected round-trip time: long enough that a heartbeat reliably beats the timeout under normal jitter, short enough that a real leader loss is noticed promptly. There is no universal constant; it has to be tuned to the deployment's actual network characteristics.

stateDiagram-v2
    Follower --> Candidate: election timeout fires
    Candidate --> Candidate: split vote, retry next term
    Candidate --> Leader: receives majority of votes
    Candidate --> Follower: sees higher term or valid leader
    Leader --> Follower: discovers higher term