Lyft Site Reliability Engineer (Entry Level) - Comprehensive Interview Preparation Guide

Approach: use rsync with --link-dest to produce snapshot-style incremental backups, create a new timestamped tmp dir, rsync into it, fsync and checksum files, rotate retention by moving snapshots atomically, detect low-disk space before/after and abort without touching previous snapshots, log everything and send alerts on failure. Use flock to prevent concurrent runs.

#!/usr/bin/env bash
set -uo pipefail
IFS=$'\n\t'

# Config
SRC="/srv/data"
DEST_BASE="/backups"
TMP_DIR="${DEST_BASE}/tmp.$(date +%s)"
LOG="/var/log/backup-srvdata.log"
LOCKFILE="/var/lock/backup-srvdata.lock"
RET_DAILY=7
RET_WEEKLY=8
RET_MONTHLY=12
EMAIL="oncall@example.com"
RSYNC_OPTS="-aHAX --delete --numeric-ids --relative --info=progress2"

log() { echo "$(date -Iseconds) $*" | tee -a "$LOG"; }
alert() { log "ALERT: $*"; echo "$*" | mail -s "Backup FAILED: srvdata" "$EMAIL"; }

cleanup() {
  rc=$?
  if [[ -d "$TMP_DIR" ]]; then rm -rf -- "$TMP_DIR"; fi
  flock -u 9 2>/dev/null || true
  exit $rc
}
trap cleanup EXIT

# Ensure single instance
exec 9>"$LOCKFILE"
if ! flock -n 9; then
  log "Another backup running; exiting."
  exit 0
fi

log "Starting backup"

# Pre-check: enough free space on DEST_BASE (require >= 1G or 10% free)
MIN_BYTES=$((1 * 1024**3))
MIN_PERC=10
avail_bytes=$(df --output=avail -B1 "$DEST_BASE" | tail -1)
avail_perc=$(df --output=pcent "$DEST_BASE" | tail -1 | tr -dc '0-9')
if (( avail_bytes < MIN_BYTES || avail_perc < MIN_PERC )); then
  alert "Insufficient disk space: ${avail_bytes} bytes available, ${avail_perc}% used"
  exit 2
fi

# Determine link-dest (latest snapshot)
latest=$(ls -1d ${DEST_BASE}/daily-* 2>/dev/null | sort | tail -n1 || true)
linkopt=()
if [[ -n "$latest" ]]; then
  linkopt=(--link-dest="$latest")
  log "Using link-dest: $latest"
fi

mkdir -p "$TMP_DIR"
# Run rsync into tmp dir
if ! rsync $RSYNC_OPTS "${linkopt[@]}" "$SRC"/ "$TMP_DIR"/; then
  alert "rsync failed"
  exit 3
fi

# Sync metadata to disk (attempt to reduce corruption risk)
sync

# Integrity: generate checksums (sha256) for new snapshot
cd "$TMP_DIR" || { alert "cd failed"; exit 4; }
find . -type f -print0 | xargs -0 sha256sum > SHA256SUMS.new
# Optionally verify the files we just checksummed (quick sanity)
if ! sha256sum --check --status SHA256SUMS.new; then
  alert "Checksum verification failed immediately after backup"
  exit 5
fi

# Atomic promotion of snapshot
timestamp=$(date +%Y-%m-%d_%H%M%S)
new_snapshot="${DEST_BASE}/daily-${timestamp}"
if ! mv "$TMP_DIR" "$new_snapshot"; then
  alert "Failed to promote snapshot"
  exit 6
fi
log "Snapshot created: $new_snapshot"

# Rotate retention: daily -> weekly -> monthly
# Keep last RET_DAILY dailies, then promote oldest weekly/monthly accordingly
# Simple rotation implementation:
cd "$DEST_BASE"
# Remove old daily snapshots beyond retention
ls -1d daily-* 2>/dev/null | sort -r | awk "NR>${RET_DAILY}" | xargs -r rm -rf --

# Weekly: move every 7th daily into weekly if not present
# (Simpler: prune weekly to RET_WEEKLY most recent)
ls -1d weekly-* 2>/dev/null | sort -r | awk "NR>${RET_WEEKLY}" | xargs -r rm -rf --
ls -1d monthly-* 2>/dev/null | sort -r | awk "NR>${RET_MONTHLY}" | xargs -r rm -rf --

log "Rotation complete"

# Final check: ensure no low-disk situation occurred during backup
avail_bytes_after=$(df --output=avail -B1 "$DEST_BASE" | tail -1)
if (( avail_bytes_after < MIN_BYTES )); then
  alert "Disk low after backup: ${avail_bytes_after} bytes left"
  exit 7
fi

log "Backup successful: $new_snapshot"
exit 0

Key points / reasoning:

• rsync --link-dest creates hard-linked snapshot trees minimizing space.
• Use tmp dir + atomic mv to avoid partial snapshots overwriting history.
• Pre/post disk checks prevent starting if close to full and detect mid-run exhaustion.
• Checksums (sha256sum) verify integrity immediately; stored alongside snapshots.
• Flock prevents concurrent runs.
• Detailed logging and alerting ensures SRE visibility.
  Edge cases: interrupted mv, inode exhaustion, very large files (consider --partial-dir), NFS fcntl issues (test), permission errors. Alternatives: use borg/restic for dedup + encryption; for very large scale, use incremental block-level backup tools.

Direct answer

At extreme scale you cannot rely on a single stampede defense. Combine request coalescing (only one request repopulates a hot key while others wait), background/proactive refresh before expiry, probabilistic early expiration, jittered time-to-live (TTL) values, and origin rate limiting as a last-resort backstop, coordinated so all application instances agree on who is allowed to refresh a given key at once.

Structured elaboration

• Request coalescing (singleflight): on a cache miss, the first request acquires a short-lived lock (e.g., SETNX in Redis) for that key and fetches from origin; concurrent requests for the same key either block briefly on a notification channel (pub/sub or polling with backoff) or serve a stale value if one exists. This bounds concurrent origin load per key to roughly one in-flight fetch, regardless of instance count, because the lock lives in the shared cache, not in any one process.
• Probabilistic early expiration: instead of a hard expiry, each read close to TTL end recomputes with a small, increasing probability (a common formula is $P(\text{refresh}) = e^{-\beta \cdot (t_{\text{expiry}} - t_{\text{now}}) / \delta}$ where $\delta$ is the time it took to compute the value and $\beta$ tunes aggressiveness). This spreads refreshes across many requests instead of concentrating them at the exact expiry instant.
• Jittered TTLs: add randomized jitter (e.g., base TTL plus/minus 10 to 20 percent) so keys written around the same time do not all expire in the same millisecond, which is what turns an ordinary cache miss into a correlated stampede across thousands of keys at once (a cache avalanche).
• Background/prewarmed refresh: a scheduled worker (or the request that detects "close to expiry") refreshes hot keys proactively so the TTL rarely actually lapses for high-traffic keys; this trades a small amount of continuous background load for eliminating stampede risk on the hottest paths.
• Origin rate limiting as a backstop: even with the above, cap concurrent origin requests per key (or per origin endpoint) so a defense-in-depth failure degrades gracefully into serving stale data or a fast error instead of taking the origin down.
• Orchestrating across many app instances: the lock, the "who refreshes next" decision, and the notification of waiters must live in the shared cache (Redis) or a coordination service, not in in-process state, because coalescing only works if every instance agrees on a single winner per key.

Worked example

Say a hot key normally takes 200 ms to recompute and serves 5,000 requests per second (RPS) at peak. Without any defense, if it expires with no coalescing, the next ~1,000 requests in that 200 ms window (5,000 RPS times 0.2 s) would all miss and hit the origin simultaneously; a database that comfortably serves single-digit concurrent queries per second for that expensive query falls over. With coalescing, exactly 1 request recomputes and the other ~999 either wait ~200 ms for the notification or receive the previous (slightly stale) value immediately; origin load for that key stays at 1 concurrent request regardless of RPS.

Trade-offs and pitfalls

A lock that never expires on a crashed refresher permanently blocks that key; always set a lock TTL slightly longer than the expected recompute time, plus a fallback path that lets a waiter give up and fetch directly after a bounded wait. Serving stale-while-refreshing is a correctness trade-off, not a free win: it is the right default for read-heavy, staleness-tolerant data, and the wrong default for a low-latency-but-must-be-fresh field like an account balance. Jitter alone does not help an already-hot key that is legitimately read far more than others; that is a hot-key sharding problem, not a stampede problem, and needs a different fix (splitting the key, adding a replica-backed local cache).

Direct answer
Five Whys means repeatedly asking 'why did that happen' about the answer to the previous why, until you reach a condition that is actually fixable rather than just another symptom. It typically takes about five iterations, though the number is a rule of thumb, not a hard rule: you stop when you hit something you can change, not necessarily on the fifth why.

Structured elaboration
For the incident (a key API returned errors for 45 minutes after a deploy, affecting a fifth of users), a Five Whys chain might look like:

1. Why did the API return errors? Because the newly deployed version crashed on a specific request shape.
2. Why did it crash on that request shape? Because a null field that used to always be populated was left unhandled by new code.
3. Why was the field null? Because an upstream service started omitting it after its own recent change, and the API's input validation did not reject the malformed payload.
4. Why did input validation not catch it? Because the API's schema validation checks types but not presence of this particular field, and there is no contract test between the two services that would have caught the mismatch before deploy.
5. Why is there no contract test between these services? Because the team has no standard practice requiring consumer-driven contract tests for internal service dependencies, so this class of breaking change can slip through again.

Root cause at the fifth why: the absence of a contract-testing practice between dependent services, which let an upstream breaking change reach production undetected. Remediation: add a consumer-driven contract test between the two services that fails the upstream service's CI if it would omit a field the downstream API depends on, and, as an immediate mitigation, add explicit null-handling and a clear 400 response for the malformed field so a similar future gap fails safely instead of crashing.

Worked example
The chain above IS the worked example. The key discipline: each why answers the previous one specifically, not by restating a broader class of the same problem ('bugs happen') or jumping straight to a process indictment ('nobody tests enough'). Each step should be falsifiable, meaning someone could look at logs, code, or configuration and confirm or reject it.

Trade-offs and pitfalls
Five Whys works well for a single, mostly-linear causal chain, but it can mislead on incidents with multiple independent contributing factors, because it forces a single narrative thread and stops once any plausible chain reaches a stopping point, even if a second, unrelated factor also mattered. In this incident, if the on-call engineer's alert also fired 15 minutes late due to an unrelated threshold problem, a rigid Five Whys chain focused only on the crash would miss that second, independently-worth-fixing gap. When you suspect multiple contributing factors, pair Five Whys with a fishbone diagram or explicit causal-chain mapping so parallel factors don't get dropped.

Situation: At each shift change I own a brief, structured handoff so the incoming SRE has context, knows priorities, and can pick up immediately.

Artifacts I leave:

• Open tickets: updated ticket(s) with summary, severity, current status, next action, assigned owner, ETA, and links to related PRs/changes.
• Runbook pointers: exact runbook/playbook links and the specific step number I followed or recommend next.
• Dashboards & queries: links to relevant Grafana/Datadog dashboards, the exact LogQL/SQL/Kusto queries I used, and a short note of what to look for.
• Logs & snippets: key log excerpts, recent error rates, and the last 30–60 minutes of relevant traces.
• Deployment/context notes: recent deploys, config changes, rollback commands, and recent CI results.
• SLO/error budget status and escalation contacts (with paging order).
• Pager history and who’s already been contacted.

How I prioritize unresolved incidents:

• Triage by impact → outage affecting customers or SLOs first, then degradation, then non-customer-facing warnings.
• Consider urgency (time-to-breach), blast radius (how many users/services), and mitigation status (is a temporary workaround active?).
• Assign explicit priority (P0/P1/P2), owner, and an expected checkpoint time (e.g., 15/30/60 mins).

Communicating risks & next steps:

• I do a synchronous 5–10 minute verbal/Slack huddle when possible, covering: current priorities, active mitigations, what will escalate if nothing changes, and the one-line expected next action per ticket.
• I post a concise handoff message in the on-call channel: summary, links to artifacts, prioritized list, immediate risks, and “if X happens, do Y / page Z.”
• I note any unknowns and suggested investigations (with queries), plus the acceptance criteria for closing tickets.
• I confirm the incoming engineer acknowledges and repeats back ownership for critical items.

This process minimizes context loss, sets clear expectations, and makes follow-up actions and escalation deterministic.

Paxos vs Raft vs Gossip — short comparison and recommendation for a strongly-consistent metadata service:

Differences (conceptual):

• Paxos: Proven theoretical minimal protocol for reaching agreement in asynchronous networks. Focuses on safety; liveness requires additional assumptions. Described in terms of proposers/acceptors/learners; hard to read and implement correctly.
• Raft: Engineered for understandability and practical implementation. Provides leader election, log replication, strong consistency (linearizability) and membership change via joint consensus. Concepts: leader, followers, terms, replicated log.
• Gossip-based membership: Epidemic protocol for disseminating membership/liveness information. It’s eventually consistent, highly available, and scales well, but not suitable for strong consistency of state or leader-driven consensus.

Recommendation for metadata service:

• Choose Raft. Reasoning:
  • Strong consistency and single leader semantics map naturally to metadata operations (e.g., allocate IDs, modify namespace).
  • Built-in leader election and well-defined membership change (joint-consensus) reduce complexity of reasoning under reconfiguration.
  • Broad production use (etcd, Consul) and mature implementations/tools simplify operations.

Failure modes:

• Leader crash/partition: Raft elects new leader after election timeout; ensure tuned timeouts to avoid split-brain and unnecessary elections.
• Network partitions: Minority partition cannot make progress (safety preserved), majority continues — plan SLOs accordingly.
• Slow/disk-backed followers: Can slow replication; monitor replication lag, apply backpressure or reconfigure quorum.
• Membership change edge cases: concurrent config changes can deadlock if not using joint consensus; implement careful sequencing and monitoring.

Complexity of implementation:

• Paxos: Higher cognitive and implementation cost; subtle edge cases (multi-paxos, leader stickiness, reconfiguration) make correctness proofs and tests harder.
• Raft: Moderate complexity; clear state machine and plentiful reference implementations reduce engineering burden.
• Gossip: Simple to implement for membership only, but combining gossip for membership with separate strong-consensus store introduces integration complexity (dual failure domains).

Testing strategies (SRE-focused, practical):

• Unit tests for state machine transitions (terms, votes, log replication).
• Integration tests with failure injection: leader/network partition, packet loss, delayed messages, follower restarts.
• Chaos engineering: automated chaos (kill leader, split network) in staging and canary to validate recovery and SLOs.
• Performance tests: write/read latency under load, membership churn scenarios.
• Durability tests: disk corruptions, fsync delays, snapshot/compaction correctness.
• Observability: expose metrics (leader term, commit index, election count, replication lag), structured logs, and alerting for stuck elections or majority loss.
• Scenario coverage: reconfiguration/joint-consensus flows, simultaneous failures during leader transfer, long GC pauses.

Operational notes:

• Use battle-tested implementations (etcd, Consul, Hashicorp Raft) rather than custom Paxos.
• Harden with tight monitoring, automated failover playbooks, and well-documented recovery procedures.
• Tune timeouts to trade off failover speed vs. false elections and align with network characteristics.

Summary: For a metadata service requiring strong consistency, leader election, and membership changes, Raft provides the best balance of safety, operational clarity, available implementations, and testability.

To compute p50, p90 and p95 from minute samples over 7 days (10,080 points), filter out None/NaN, sort and pick percentiles. For SRE use, treat percentiles as inclusive (nearest-rank) or use interpolation. Handle empty input by returning None.

import math
import statistics
from typing import List, Tuple, Optional
import math

def compute_percentiles(samples: List[Optional[float]]) -> Tuple[Optional[float], Optional[float], Optional[float]]:
    """
    Returns (p50, p90, p95). Ignores None or NaN. Returns (None, None, None) if no valid samples.
    Uses linear interpolation between ranks (numpy.percentile default behavior).
    """
    # filter invalid
    vals = [v for v in samples if v is not None and not (isinstance(v, float) and math.isnan(v))]
    n = len(vals)
    if n == 0:
        return (None, None, None)
    vals.sort()
    def percentile(p):
        # p in [0,100], using interpolation (R-7 / numpy)
        k = (p/100) * (n - 1)
        lo = int(math.floor(k))
        hi = int(math.ceil(k))
        if lo == hi:
            return vals[lo]
        frac = k - lo
        return vals[lo] * (1 - frac) + vals[hi] * frac
    return (percentile(50), percentile(90), percentile(95))

Key points:

• Complexity: O(n log n) due to sort; O(n) memory.
• Edge cases: all None/NaN, single value, duplicates, percentiles at extremes.
• Streaming: use streaming quantile algorithms (e.g., t-digest, GK-algorithm) to compute approximate p95 with bounded memory and mergeability—recommended for long-running monitoring pipelines.

Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and Cluster Autoscaler (CA) operate on three different axes: HPA changes how many pod replicas exist, VPA changes how much CPU and memory each pod requests, and CA changes how many nodes exist, reacting only to pods it cannot schedule or nodes it judges empty. The conflict case that comes up in practice is running HPA and VPA on the same workload against the same signal: VPA raising a pod's CPU request lowers that pod's CPU utilization percentage (same usage, bigger denominator), which can make HPA think load dropped and hold off scaling out right when VPA's own resizing is also making pods bigger and harder to schedule, a feedback loop that shows up as scheduling failures and node churn rather than a clean scaling curve.

How the three interact

CA is the last-mile capacity provider: it doesn't look at metrics at all, only at whether pods are stuck Pending for lack of room, or whether a node's real utilization has stayed below a threshold (the default scale-down-utilization-threshold is 50%, and a node has to stay under it for the default scale-down-unneeded-time of 10 minutes before CA considers removing it, specifically to avoid thrashing on a brief dip).

The conflict, concretely

HPA is configured to scale out at 70% CPU. VPA (in a mode that actually applies changes, not just recommends) sees real CPU usage per pod is high and raises the CPU request. Because HPA's CPU% is usage divided by request, that same usage now reads as a lower percentage against the new, larger request, so HPA either stops scaling out or, worse, in the transitional moment while pods are being resized, ends up racing VPA: HPA adds replicas for load that VPA is simultaneously deciding needs bigger pods, and the result is more, bigger pods than the workload needed, tripping the Cluster Autoscaler into rapid node adds and (once things settle) removals.

Resolving it

1. Prefer disjoint signals. Point HPA at a metric VPA's resizing doesn't move, such as requests-per-second or a custom queue-depth metric, instead of CPU percentage, so the two don't chase the same number.
2. Constrain what VPA is allowed to touch. VPA supports scoping its containerPolicies to specific resources (for example, letting it manage memory but not CPU) via controlledResources, so it can right-size the axis HPA isn't scaling on.
3. Use VPA's non-disruptive modes deliberately. VPA's update modes are Off (recommendation only, nothing applied), Initial (sets requests only at pod creation, never touches a running pod), and Recreate (evicts and re-creates the pod to apply a new recommendation); Auto still exists in older docs but was deprecated as of VPA 1.5 and is now just an alias for Recreate. A newer InPlaceOrRecreate mode (beta) attempts to resize a running pod's resources without evicting it at all, riding on Kubernetes' in-place pod resize feature (alpha in 1.27, beta in 1.33, stable/GA in 1.35), falling back to evict-and-recreate only when an in-place resize genuinely isn't possible. Running VPA in Off mode and feeding its recommendations into a review step (a dashboard, a PR) rather than letting it apply changes live sidesteps the feedback loop entirely while still getting the sizing signal.
4. Tune HPA's stabilization. behavior.scaleDown.stabilizationWindowSeconds on the autoscaling/v2 HPA object smooths out reactions to a transient metric shift instead of scaling on every sample.
5. Give Cluster Autoscaler room to be boring. Set realistic scale-up/scale-down delays, use PodDisruptionBudgets (PDB, an object that caps how many pods of a set can be unavailable at once) so CA's scale-down eviction can't take out too many replicas of the same workload at once, and use the cluster-autoscaler.kubernetes.io/safe-to-evict: "false" pod annotation for anything that must never be moved automatically.

Trade-offs

• Running HPA and VPA on the same resource axis for the same workload is the anti-pattern, not a supported combination to be tuned into stability; the fix is almost always to separate the axes or the signal, not to add more stabilization windows on top.
• InPlaceOrRecreate reduces disruption but is a newer, less battle-tested code path than plain Recreate; teams with strict availability requirements may reasonably choose the well-understood evict-and-recreate behavior over the newer in-place path until they've validated it against their own failure modes.
• VPA is generally the better fit for workloads that can't scale horizontally at all (a single-replica stateful service, a batch job), where HPA has nothing to do in the first place.

Direct answer

Require every signal (metric, log line, trace span) to carry a small set of common identifiers, most importantly trace_id (or a request/correlation ID where full tracing isn't wired up yet), plus service_name and a timestamp. Build a lightweight mapping index, keyed by those identifiers, that points to where each related log or span physically lives (shard, offset, time range) rather than storing the events themselves twice. When an identifier is missing, fall back to narrowing by time window and service first, then use a scored, probabilistic match on top of that narrowed candidate set instead of searching the whole corpus.

Structured elaboration

Required identifiers per event:

• trace_id / span_id: the strong identifier, present when full distributed tracing is instrumented.
• service_name, host_or_pod_id: contextual keys usable even when tracing isn't present.
• event_timestamp: needed for the time-window fallback and for ordering.
• metric_dimensions / anomaly_id: on the metrics side, the anomaly detector emits an anomaly document carrying whatever identifiers were present on the underlying series.

Index structure, not a duplicate copy: the mapping store is a pointer index (topic/partition/offset for logs, trace/span ID for traces), not a second copy of log or trace bodies. Logs and traces stay in their existing systems (search index, trace store); the correlation index only tells the query which shard and offset to fetch.

Query flow:

1. Direct ID lookup, if the anomaly carries a trace_id: O(1) pointer lookup into the mapping store.
2. Indexed search, if not: query the log/trace index filtered by service_name and a time window around the anomaly.
3. Probabilistic fallback, if identifiers are sparse: score candidates from step 2 by temporal proximity, shared host, and payload fingerprint, and return ranked results with a visible confidence, not a silent best guess.

flowchart LR
    A[Metric anomaly] --> ID{trace_id present?}
    ID -- yes --> M[(Mapping index: id -> pointer)]
    M --> L1[Fetch log/trace by pointer]
    ID -- no --> TW[Filter: service + time window]
    TW --> SC[Score candidates: proximity, host, fingerprint]
    SC --> L2[Ranked candidates + confidence]

Maintaining the mapping: on ingest, if a strong identifier is present, upsert a bounded (capped) pointer list into the mapping store for that ID and set a TTL matching the signal's own retention. A background job prunes expired entries so the index doesn't grow unbounded past what the underlying logs and traces themselves retain.

Worked example

Index sizing. Assume the cluster produces $50{,}000$ events/sec across logs and trace spans combined:

Each mapping entry (topic, partition, offset, timestamp, service ID) is about 40 bytes:

At a 7-day retention window matching typical hot log retention:

This is the number that justifies capped pointer lists and TTL-based pruning in the design above: an unbounded, uncapped index at this event rate would grow past a terabyte inside a week even though it stores no event bodies, only pointers.

Fallback candidate-set size when the identifier is missing. With 200 services sharing the $4.32 \times 10^9$ daily events roughly evenly:

Narrowing to a $\pm 2.5$s window (5s total) around the anomaly timestamp before service-level scoring:

Filtering by service and a 5-second window before running any fuzzy scoring shrinks the search space from 21.6 million events/day for that service down to about 1,250 candidates, small enough for a scoring pass to run against on every anomaly without scanning the full day's log volume.

Trade-offs & pitfalls

Common wrong turns: treating the mapping index as a full copy of the underlying signals instead of a pointer index (this doubles storage for no correlation benefit and creates a second source of truth to keep consistent); running the fuzzy-scoring pass over the entire day's events instead of narrowing by service and time window first, which turns an O(1,250) scoring problem into an O(21,600,000) one for no accuracy gain; and silently returning the single top-scored fuzzy match without surfacing its confidence, which looks like a correlation but is actually a guess an on-call engineer has no way to sanity-check.

Direct answer

The mechanics are the same regardless of what is being changed: stand up the new version alongside the old one, shift traffic to it gradually (or all at once for blue-green), watch metrics against a defined threshold before promoting further, and have an automated way to shift traffic back if those metrics degrade. What differs by change type is what "the new version" means and which metric actually tells you it is safe, so the mechanics come first below, then how they change for a few different kinds of infrastructure change.

Structured elaboration

Traffic shifting mechanics

• Canary: route a small percentage of traffic to the new version (for example 5%), hold, then ramp (25%, 50%, 100%) with a wait window at each step.
• Blue-green: fully validate the new environment out of band, then cut traffic over in one step (DNS, load balancer listener swap, or weighted routing swap to 100%). Rollback is just swapping back.

What "the new version" and "the signal to watch" are, by change type

Stateless versus stateful

For a stateless service, this is close to mechanical: deregister the old instance from the load balancer, let connections drain, terminate it, and the new instance takes over with no coordination needed between old and new.

For a service with state, for example one backed by a database or holding session or connection state, the "new version" cannot simply replace the old one at will:

• Deregistration has to wait for graceful connection draining, not just for the health check to fail, or you drop in-flight work.
• If the state itself is what is changing (the DB-class case above), the new version has to be replication-caught-up before it sees real traffic, and you need a plan for what happens to writes that land during the cutover window.
• The old and new versions may be reading or writing the same underlying data store, so canary steps need to account for that shared state, not just for the request path.

Worked example

A concrete case: shifting 5% of production traffic on an ALB to a target group fronting instances built from a new AMI, before fully cutting over.

resource "aws_lb_target_group" "green" {
  name     = "app-green"
  port     = 443
  protocol = "HTTPS"
  vpc_id   = var.vpc_id
  health_check {
    path                = "/health"
    healthy_threshold   = 3
    unhealthy_threshold = 2
    timeout             = 2
  }
}

resource "aws_lb_listener_rule" "canary" {
  listener_arn = aws_lb_listener.app.arn
  priority     = 100

  action {
    type = "forward"
    forward {
      target_group {
        arn    = aws_lb_target_group.green.arn
        weight = 5
      }
      target_group {
        arn    = aws_lb_target_group.blue.arn
        weight = 95
      }
    }
  }

  condition {
    path_pattern { values = ["/*"] }
  }
}

Watch the green target group's 5xx rate and p95 latency for a fixed window (say 15 minutes). If it stays within the same range as blue, bump the weight to 25 and repeat; if 5xx rate or latency breaches the threshold at any step, a second terraform apply (or a CI job wrapping it) sets green's weight back to 0 and blue keeps serving everything.

flowchart TD
  A[Provision green environment via Terraform] --> B[Register in target group, run health checks]
  B --> C[Shift 5 percent traffic to green]
  C --> D{Metrics within SLO?}
  D -- Yes --> E[Ramp to 25 percent, then 50 percent]
  E --> F{Still within SLO?}
  F -- Yes --> G[Shift 100 percent traffic, decommission blue]
  F -- No --> H[Roll back: weight to 0 percent, blue keeps serving]
  D -- No --> H

Trade-offs & pitfalls

• The biggest mistake is treating every infra change like an app deploy and watching only 5xx rate. A security-group rule change will not show up as a 5xx, it shows up as connections that never complete; a DB-class change will not show up in the app's error rate until replication lag has already gotten bad. Pick the signal that matches what actually breaks for that change type.
• Canary steps for stateful changes need explicit wait-for-caught-up gates, not just a timer; ramping traffic to a replica that has not finished catching up just moves the failure into user-visible latency or stale reads.
• Automating the rollback path, not just the promotion path, is what actually buys "close to zero downtime." If pulling the plug requires someone to remember the right terraform apply under pressure, the safety only exists on paper.

Requirements / goals:

• Declare a Major Incident (MI) when impact is systemic, sustained, or regulatory/financially material and requires company-wide coordination.
• Thresholds must be objective, actionable, auditable, and minimize false positives.

Numeric & qualitative thresholds (any one triggers MI):

• Customer impact

  • = 25% of active users degraded or unable to use critical functionality for > 5 minutes

  • OR > 10% of API requests returning 5xx errors for > 10 minutes across production regions
  • Qualitative: outage of a feature marked “critical” by product (payments, auth, telemetry) irrespective of percentage.
• Revenue / business impact
  • Estimated gross revenue loss ≥ $100k/hour (configurable by product) OR projected multi-hour loss > $250k
  • Major merchant / enterprise customer SLA breach that could cause contractual penalties
• Regulatory / compliance
  • Any incident that risks PII/financial data exposure, GDPR/PCI reportable event, or legal escalation
• SLO / Error budget
  • Service-level objective breach with remaining error budget exhausted and projected 24h recovery unlikely
• Operational severity
  • Loss of multi-region control plane, inability to deploy critical fixes, or cascading failures affecting multiple services

Decision flow for declaration:

1. Detection: automated alert meets MI threshold OR pager / engineer believes impact matches qualitative trigger.
2. Triage (5 minutes): on-call runs checklist: confirm alerts, check telemetry (traffic, errors, latency), confirm blast radius (regions/customers), estimate revenue/regulatory exposure.
3. Decision point (after triage): If any objective threshold confirmed OR qualitative/regulatory trigger present → declare MI.
4. Declare: on-call hits “Declare Major” in incident system; MI state starts (timestamped).
5. Notify & mobilize (immediately): Incident Lead assigned (usually SRE Senior), War Room stood up (virtual + optional physical), responders notified.
6. Action cycles: 15-minute status cadence until resolution or recovery.

Roles & notifications:

• Immediate (within 5 mins of declaration):
  • Incident Lead (SRE Senior)
  • Engineering Manager of affected teams
  • Product Owner / PM for impacted product
  • Customer Success / Account Execs for top 10 customers
  • Security/Compliance (if data/regulatory)
  • Communications / PR
  • CTO/VP Engineering (for high revenue/regulatory incidents)
• Secondary:
  • Legal (if PII/contractual exposure)
  • Finance (if revenue impact > threshold)
  • Executive on-call escalation chain if not resolved within 30/60/120 minutes (configurable)
• External: Customer notifications per runbook templates after initial internal triage (within 30–60 minutes).

Safeguards to avoid false positives:

• Multi-signal confirmation: require at least 2 independent signals (application metrics + network/infra telemetry + synthetic checks) before auto-declare.
• Short confirmation window: automated declarations enter “pending MI” for 2–5 minutes to auto-validate metrics trend and avoid alert storms causing immediate escalate.
• Rate-limit automated declarations: backoff for known flapping alerts; require human confirmation if same service declared twice in X minutes.
• Playbook-driven checks: triage checklist includes quick sanity checks (deployments in flight, config changes, monitoring misconfig) to rule out monitoring/system false alarms.
• Canary and synthetic validation: run targeted synthetic tests against critical endpoints to confirm true customer impact.
• Post-declare audit: every MI must record evidence (metrics, logs, decision rationale) and undergo a blameless post-incident review to refine thresholds and reduce future false positives.

Operational notes / tuning:

• Thresholds should be parameterized per product/service with annual review tied to SLOs and business targets.
• Maintain a lightweight “near-MI” state for incidents that are serious but below MI thresholds; same playbooks with fewer notifications.
• Run quarterly drills to validate decision flow, notification lists, and false-positive safeguards.

This approach balances objective numeric triggers with qualitative judgment, enforces rapid but careful triage, ensures the right stakeholders are notified, and includes multiple safeguards to minimize false-positive declarations.