Lyft Site Reliability Engineer (Entry Level) - Comprehensive Interview Preparation Guide

Approach: use rsync with --link-dest to produce snapshot-style incremental backups, create a new timestamped tmp dir, rsync into it, fsync and checksum files, rotate retention by moving snapshots atomically, detect low-disk space before/after and abort without touching previous snapshots, log everything and send alerts on failure. Use flock to prevent concurrent runs.

#!/usr/bin/env bash
set -uo pipefail
IFS=$'\n\t'

# Config
SRC="/srv/data"
DEST_BASE="/backups"
TMP_DIR="${DEST_BASE}/tmp.$(date +%s)"
LOG="/var/log/backup-srvdata.log"
LOCKFILE="/var/lock/backup-srvdata.lock"
RET_DAILY=7
RET_WEEKLY=8
RET_MONTHLY=12
EMAIL="oncall@example.com"
RSYNC_OPTS="-aHAX --delete --numeric-ids --relative --info=progress2"

log() { echo "$(date -Iseconds) $*" | tee -a "$LOG"; }
alert() { log "ALERT: $*"; echo "$*" | mail -s "Backup FAILED: srvdata" "$EMAIL"; }

cleanup() {
  rc=$?
  if [[ -d "$TMP_DIR" ]]; then rm -rf -- "$TMP_DIR"; fi
  flock -u 9 2>/dev/null || true
  exit $rc
}
trap cleanup EXIT

# Ensure single instance
exec 9>"$LOCKFILE"
if ! flock -n 9; then
  log "Another backup running; exiting."
  exit 0
fi

log "Starting backup"

# Pre-check: enough free space on DEST_BASE (require >= 1G or 10% free)
MIN_BYTES=$((1 * 1024**3))
MIN_PERC=10
avail_bytes=$(df --output=avail -B1 "$DEST_BASE" | tail -1)
avail_perc=$(df --output=pcent "$DEST_BASE" | tail -1 | tr -dc '0-9')
if (( avail_bytes < MIN_BYTES || avail_perc < MIN_PERC )); then
  alert "Insufficient disk space: ${avail_bytes} bytes available, ${avail_perc}% used"
  exit 2
fi

# Determine link-dest (latest snapshot)
latest=$(ls -1d ${DEST_BASE}/daily-* 2>/dev/null | sort | tail -n1 || true)
linkopt=()
if [[ -n "$latest" ]]; then
  linkopt=(--link-dest="$latest")
  log "Using link-dest: $latest"
fi

mkdir -p "$TMP_DIR"
# Run rsync into tmp dir
if ! rsync $RSYNC_OPTS "${linkopt[@]}" "$SRC"/ "$TMP_DIR"/; then
  alert "rsync failed"
  exit 3
fi

# Sync metadata to disk (attempt to reduce corruption risk)
sync

# Integrity: generate checksums (sha256) for new snapshot
cd "$TMP_DIR" || { alert "cd failed"; exit 4; }
find . -type f -print0 | xargs -0 sha256sum > SHA256SUMS.new
# Optionally verify the files we just checksummed (quick sanity)
if ! sha256sum --check --status SHA256SUMS.new; then
  alert "Checksum verification failed immediately after backup"
  exit 5
fi

# Atomic promotion of snapshot
timestamp=$(date +%Y-%m-%d_%H%M%S)
new_snapshot="${DEST_BASE}/daily-${timestamp}"
if ! mv "$TMP_DIR" "$new_snapshot"; then
  alert "Failed to promote snapshot"
  exit 6
fi
log "Snapshot created: $new_snapshot"

# Rotate retention: daily -> weekly -> monthly
# Keep last RET_DAILY dailies, then promote oldest weekly/monthly accordingly
# Simple rotation implementation:
cd "$DEST_BASE"
# Remove old daily snapshots beyond retention
ls -1d daily-* 2>/dev/null | sort -r | awk "NR>${RET_DAILY}" | xargs -r rm -rf --

# Weekly: move every 7th daily into weekly if not present
# (Simpler: prune weekly to RET_WEEKLY most recent)
ls -1d weekly-* 2>/dev/null | sort -r | awk "NR>${RET_WEEKLY}" | xargs -r rm -rf --
ls -1d monthly-* 2>/dev/null | sort -r | awk "NR>${RET_MONTHLY}" | xargs -r rm -rf --

log "Rotation complete"

# Final check: ensure no low-disk situation occurred during backup
avail_bytes_after=$(df --output=avail -B1 "$DEST_BASE" | tail -1)
if (( avail_bytes_after < MIN_BYTES )); then
  alert "Disk low after backup: ${avail_bytes_after} bytes left"
  exit 7
fi

log "Backup successful: $new_snapshot"
exit 0

Key points / reasoning:

• rsync --link-dest creates hard-linked snapshot trees minimizing space.
• Use tmp dir + atomic mv to avoid partial snapshots overwriting history.
• Pre/post disk checks prevent starting if close to full and detect mid-run exhaustion.
• Checksums (sha256sum) verify integrity immediately; stored alongside snapshots.
• Flock prevents concurrent runs.
• Detailed logging and alerting ensures SRE visibility.
  Edge cases: interrupted mv, inode exhaustion, very large files (consider --partial-dir), NFS fcntl issues (test), permission errors. Alternatives: use borg/restic for dedup + encryption; for very large scale, use incremental block-level backup tools.

Direct answer

At extreme scale you cannot rely on a single stampede defense. Combine request coalescing (only one request repopulates a hot key while others wait), background/proactive refresh before expiry, probabilistic early expiration, jittered time-to-live (TTL) values, and origin rate limiting as a last-resort backstop, coordinated so all application instances agree on who is allowed to refresh a given key at once.

Structured elaboration

• Request coalescing (singleflight): on a cache miss, the first request acquires a short-lived lock (e.g., SETNX in Redis) for that key and fetches from origin; concurrent requests for the same key either block briefly on a notification channel (pub/sub or polling with backoff) or serve a stale value if one exists. This bounds concurrent origin load per key to roughly one in-flight fetch, regardless of instance count, because the lock lives in the shared cache, not in any one process.
• Probabilistic early expiration: instead of a hard expiry, each read close to TTL end recomputes with a small, increasing probability (a common formula is $P(\text{refresh}) = e^{-\beta \cdot (t_{\text{expiry}} - t_{\text{now}}) / \delta}$ where $\delta$ is the time it took to compute the value and $\beta$ tunes aggressiveness). This spreads refreshes across many requests instead of concentrating them at the exact expiry instant.
• Jittered TTLs: add randomized jitter (e.g., base TTL plus/minus 10 to 20 percent) so keys written around the same time do not all expire in the same millisecond, which is what turns an ordinary cache miss into a correlated stampede across thousands of keys at once (a cache avalanche).
• Background/prewarmed refresh: a scheduled worker (or the request that detects "close to expiry") refreshes hot keys proactively so the TTL rarely actually lapses for high-traffic keys; this trades a small amount of continuous background load for eliminating stampede risk on the hottest paths.
• Origin rate limiting as a backstop: even with the above, cap concurrent origin requests per key (or per origin endpoint) so a defense-in-depth failure degrades gracefully into serving stale data or a fast error instead of taking the origin down.
• Orchestrating across many app instances: the lock, the "who refreshes next" decision, and the notification of waiters must live in the shared cache (Redis) or a coordination service, not in in-process state, because coalescing only works if every instance agrees on a single winner per key.

Worked example

Say a hot key normally takes 200 ms to recompute and serves 5,000 requests per second (RPS) at peak. Without any defense, if it expires with no coalescing, the next ~1,000 requests in that 200 ms window (5,000 RPS times 0.2 s) would all miss and hit the origin simultaneously; a database that comfortably serves single-digit concurrent queries per second for that expensive query falls over. With coalescing, exactly 1 request recomputes and the other ~999 either wait ~200 ms for the notification or receive the previous (slightly stale) value immediately; origin load for that key stays at 1 concurrent request regardless of RPS.

Trade-offs and pitfalls

A lock that never expires on a crashed refresher permanently blocks that key; always set a lock TTL slightly longer than the expected recompute time, plus a fallback path that lets a waiter give up and fetch directly after a bounded wait. Serving stale-while-refreshing is a correctness trade-off, not a free win: it is the right default for read-heavy, staleness-tolerant data, and the wrong default for a low-latency-but-must-be-fresh field like an account balance. Jitter alone does not help an already-hot key that is legitimately read far more than others; that is a hot-key sharding problem, not a stampede problem, and needs a different fix (splitting the key, adding a replica-backed local cache).

Choosing SLOs for a data service you actually built starts from the specific business need the service exists to serve, then works down to the quantitative thresholds, not the other way around.

Structured elaboration

Success metrics worth defining: throughput (requests or records served per unit time, confirming the service can sustain expected load), p95/p99 latency (tail experience, since a data service's consumers often run bursty batch queries where the tail matters as much as, or more than, the median), data freshness (how current the served data is relative to its source), and error rate (proportion of requests failing outright). Thresholds should be chosen against what the CONSUMING use case actually requires: a dashboard refreshed hourly can tolerate looser freshness than a real-time alerting pipeline consuming the same underlying data service.

Worked example

For a production data service feeding both an hourly-refreshed executive dashboard and a near-real-time fraud-detection pipeline: p95 latency < 200ms (chosen based on what the fraud pipeline's own SLA requires, the tighter of the two consumers), freshness < 5 minutes (again driven by the tighter consumer, even though the dashboard alone would have tolerated much looser freshness), error rate < 0.1%. Validation during launch: a soft-launch period against a subset of real consumer traffic, checking that the chosen thresholds are both ACHIEVABLE (the service can actually sustain them under real load) and SUFFICIENT (the tightest consumer's actual needs are genuinely met, confirmed by that consumer's own downstream metrics staying healthy).

Trade-offs and pitfalls

Setting one blended threshold to satisfy the AVERAGE of all consumers' needs, rather than the TIGHTEST consumer's actual requirement, is a common mistake: it can look reasonable on paper while quietly under-serving the one consumer (often the most business-critical one) whose needs were actually the tightest. It's also worth revisiting these thresholds as new consumers are added over the service's life, since a threshold validated against the original set of consumers may become insufficient once a new, more demanding consumer starts depending on the same service without anyone re-checking whether the existing SLO still covers their needs.

Direct answer

Rebuilding trust after a major incident is less about the apology itself and more about what happens in the weeks after it, specifically whether I do exactly what I said I'd do, on the timeline I said I'd do it. I know it worked not because the client stops being upset in the moment, but because their behavior toward me changes later, they start trusting my word again in ways they'd explicitly stopped doing right after the incident.

Structured elaboration

• What to say: a direct, specific account of what happened and what impact it had on them specifically, not a generic company-wide summary. Vague language, such as saying only that "some issues" occurred, reads as evasive to someone who was personally affected and wants a real explanation.
• What to do: commit to a small number of concrete, verifiable actions rather than a broad promise to do better. Concrete commitments, a specific fix, a specific monitoring change, a specific date to report back, are things the client can actually check on later, which is exactly the point.
• Following through visibly: the trust-rebuilding work isn't the apology call, it's proactively reporting back on each commitment as it's completed, without waiting for the client to ask whether it happened. Silence after the incident, even well-intentioned silence while quietly doing the work, reads the same as not doing it.
• How to know it worked: not by the client saying it's fine now, which they may say to be polite well before they actually mean it. The real signal is a change in their behavior over time, being willing to give you the benefit of the doubt on something new, looping you in early on a related decision, or simply not bringing up the incident defensively the next time something goes slightly wrong.

Worked example

After a major incident caused a client's own downstream process to fail for several hours, I called them directly rather than sending a written update first, walked through specifically what broke on our side, exactly how it had affected their process, and what I did and didn't yet know about the root cause. I made three specific commitments on that call: a written root-cause explanation within a couple of business days, a monitoring change that would have caught this specific failure mode earlier, delivered within a set number of weeks, and a personal check-in call once that monitoring change had actually shipped, not just once it was scheduled.

I followed through on each one on the stated timeline, and proactively reported back each time rather than waiting to be asked, including the follow-up call, where I walked them through the actual monitoring dashboard so they could see the change themselves rather than taking my word for it. For a while after, our regular working relationship stayed noticeably more cautious than before the incident: the client's team double-checked details with me that they wouldn't previously have double-checked, and looped in their own leadership on decisions where they hadn't before.

I knew trust had actually come back not from anything they said directly, but from a change in that behavior: some months later, when planning a new integration, they proposed involving my team early in the design conversation, the same kind of trust they'd extended before the incident and had visibly stopped extending right after it. That, not a verbal "we're all good now," was the real signal.

Trade-offs and pitfalls

The common mistake is treating the apology conversation itself as the trust-rebuilding work, when it's really just the opening move; trust actually rebuilds, or doesn't, in what happens over the following weeks. Overpromising during that first conversation, committing to more than you can reliably deliver in the moment of wanting to make the client feel better right now, is a real trap, since a broken follow-up commitment after a trust-damaging incident does more damage than the original incident itself. The other pitfall is mistaking politeness for genuine trust recovery; a client saying the right words in a meeting is not the same evidence as a change in how they actually behave with you afterward.

Paxos vs Raft vs Gossip — short comparison and recommendation for a strongly-consistent metadata service:

Differences (conceptual):

• Paxos: Proven theoretical minimal protocol for reaching agreement in asynchronous networks. Focuses on safety; liveness requires additional assumptions. Described in terms of proposers/acceptors/learners; hard to read and implement correctly.
• Raft: Engineered for understandability and practical implementation. Provides leader election, log replication, strong consistency (linearizability) and membership change via joint consensus. Concepts: leader, followers, terms, replicated log.
• Gossip-based membership: Epidemic protocol for disseminating membership/liveness information. It’s eventually consistent, highly available, and scales well, but not suitable for strong consistency of state or leader-driven consensus.

Recommendation for metadata service:

• Choose Raft. Reasoning:
  • Strong consistency and single leader semantics map naturally to metadata operations (e.g., allocate IDs, modify namespace).
  • Built-in leader election and well-defined membership change (joint-consensus) reduce complexity of reasoning under reconfiguration.
  • Broad production use (etcd, Consul) and mature implementations/tools simplify operations.

Failure modes:

• Leader crash/partition: Raft elects new leader after election timeout; ensure tuned timeouts to avoid split-brain and unnecessary elections.
• Network partitions: Minority partition cannot make progress (safety preserved), majority continues — plan SLOs accordingly.
• Slow/disk-backed followers: Can slow replication; monitor replication lag, apply backpressure or reconfigure quorum.
• Membership change edge cases: concurrent config changes can deadlock if not using joint consensus; implement careful sequencing and monitoring.

Complexity of implementation:

• Paxos: Higher cognitive and implementation cost; subtle edge cases (multi-paxos, leader stickiness, reconfiguration) make correctness proofs and tests harder.
• Raft: Moderate complexity; clear state machine and plentiful reference implementations reduce engineering burden.
• Gossip: Simple to implement for membership only, but combining gossip for membership with separate strong-consensus store introduces integration complexity (dual failure domains).

Testing strategies (SRE-focused, practical):

• Unit tests for state machine transitions (terms, votes, log replication).
• Integration tests with failure injection: leader/network partition, packet loss, delayed messages, follower restarts.
• Chaos engineering: automated chaos (kill leader, split network) in staging and canary to validate recovery and SLOs.
• Performance tests: write/read latency under load, membership churn scenarios.
• Durability tests: disk corruptions, fsync delays, snapshot/compaction correctness.
• Observability: expose metrics (leader term, commit index, election count, replication lag), structured logs, and alerting for stuck elections or majority loss.
• Scenario coverage: reconfiguration/joint-consensus flows, simultaneous failures during leader transfer, long GC pauses.

Operational notes:

• Use battle-tested implementations (etcd, Consul, Hashicorp Raft) rather than custom Paxos.
• Harden with tight monitoring, automated failover playbooks, and well-documented recovery procedures.
• Tune timeouts to trade off failover speed vs. false elections and align with network characteristics.

Summary: For a metadata service requiring strong consistency, leader election, and membership changes, Raft provides the best balance of safety, operational clarity, available implementations, and testability.

What each layer owns

A traditional load balancer mostly handles north-south traffic, external clients reaching your services, terminating connections at the edge and routing to backends. A service mesh (Istio, Linkerd) is a dedicated layer for east-west traffic, service-to-service calls inside the system, typically implemented as a sidecar proxy (commonly Envoy) injected alongside every service instance that transparently intercepts all of that instance's inbound and outbound traffic.

Sidecar versus ingress controller

The sidecar is per-instance: it handles mutual TLS (both sides of a connection proving their identity with a certificate, not just the client verifying the server) between services, retries, per-call timeouts, fine-grained routing decisions, and emits telemetry for every single hop, without the application code knowing any of this is happening. The ingress controller is the single entry point at the edge, terminating external traffic and doing coarser routing into the mesh, often still built on a conventional load-balancing implementation. They are complementary, not competing: ingress gets traffic into the system, the mesh manages it once inside.

L7 routing and retries

Because the sidecar understands HTTP (or gRPC) at L7 (the application layer, meaning the proxy can read the actual HTTP request, not just IP/port), it can route on path, header, or version, for example sending 5 percent of traffic to a canary version based on a header, something an L4 (transport layer, routing only on IP/port) load balancer cannot do. Retries can be configured with a budget (a cap on total retry volume across the fleet) so that retrying a failing call does not itself amplify an incident by multiplying load on an already-struggling dependency.

Circuit-breaking primitives

Each sidecar independently tracks recent failures for the specific upstream instances it talks to, and stops sending requests to an instance whose failure rate crosses a threshold (often called outlier detection or ejection). This is more granular than a centrally-run health check: it happens locally, at every calling service, based on real observed call outcomes, not on a periodic external probe.

Observability improvements

Because every hop passes through the mesh, you get uniform golden-signal metrics (latency, traffic volume, error rate, and saturation) for every service pair without instrumenting each service's code individually, plus consistent propagation of distributed-tracing headers across the whole call graph.

Feeding autoscaling decisions

The mesh sees real per-request latency, error rate, and in-flight request count for every service, which are often better autoscaling signals than raw CPU: scaling on in-flight requests per instance, or on a p99 latency breach pulled from mesh metrics, reacts to actual user-facing load rather than a proxy metric that may lag it.

Latency and operational complexity trade-offs

Every hop now passes through a local sidecar proxy, adding a small amount of latency per hop, typically small on its own but compounding across a deep chain of many internal calls. Operationally, you take on a new control plane to run and upgrade, certificate rotation for mutual TLS, per-instance resource overhead (CPU and memory for every sidecar, multiplied by fleet size), a real learning curve for the team, and a new potential source of outages: the mesh itself can now be the thing that breaks. A mesh earns its cost when you have many services, need fine-grained per-route control, need mutual TLS or zero-trust between services, or want reliability primitives (retries, circuit breaking) implemented once, consistently, across every language in a polyglot fleet. For a handful of services, a conventional load balancer plus straightforward application-level retry logic is usually simpler and cheaper to operate.

Direct answer: A safe automated failover has a strict order: confirm the primary is actually down (not just unreachable from one vantage point), verify the standby is caught up enough to promote safely, promote the standby, then reroute traffic to it (load balancer target groups before DNS, since DNS propagation is slower and less reliable), and finally verify with a real smoke test before declaring success. If any safety check fails partway through, the system should stop and fall back to the old primary or escalate to a human rather than push forward on an unverified assumption.

Structured elaboration: order of operations

flowchart TD
    A[Primary reports unhealthy] --> B{Confirmed by multiple<br/>independent health checks?}
    B -->|No, single check only,<br/>possible false positive| A
    B -->|Yes, quorum of checks agree| D{Standby replication lag<br/>within acceptable RPO?}
    D -->|No, too far behind| E[Escalate to human:<br/>promoting now means data loss]
    D -->|Yes| F[Promote standby to primary]
    F --> G{Promotion succeeded<br/>and standby accepting writes?}
    G -->|No| H[Abort: old primary stays<br/>authoritative, alert on-call]
    G -->|Yes| I[Update load balancer<br/>target group to new primary]
    I --> J[Update DNS / service discovery<br/>for clients bypassing the LB]
    J --> K[Run smoke test:<br/>real read + write against new primary]
    K -->|Fails| L[Roll back: revert target group/DNS,<br/>do not resume traffic]
    K -->|Passes| M[Failover complete,<br/>declare success, notify]

Why this order specifically:

• Confirmation before action: a single failed health check from one vantage point can be a network issue on the checker's side, not the primary's. Requiring agreement from multiple independent checkers (different network paths, or a quorum-based check) is what prevents an unnecessary failover from a false positive, which is itself an availability risk (a healthy primary getting demoted).
• Replication-lag check before promotion: this is the step that directly determines your actual RPO (recovery point objective: how much data, measured in time, you can afford to lose) at failover time. If the standby is caught up to within, say, a few seconds of the primary, promoting it loses at most that much data. If it's minutes behind (e.g., during a burst of write load or a network hiccup on the replication link), promoting anyway silently commits to losing everything written since the standby's last applied transaction. This check has to gate promotion, not just log a warning.
• Load balancer before DNS: the load balancer's target group update takes effect on the next health check cycle (seconds), while DNS changes depend on client and resolver caching behavior that you don't fully control, even with a low TTL. Updating the LB first means most traffic reroutes quickly through the path you actually control; DNS updates catch the remaining clients that bypass the LB or cache DNS aggressively.
• Smoke test before declaring done: promotion succeeding and the LB/DNS pointing at the new primary doesn't guarantee the new primary is actually serving correctly, only that infrastructure changes applied. A real read and write against it (not just a health-check ping) is what confirms the failover actually restored service.

Worked example: one failover, traced with real timestamps. Say the primary in us-east-1a stops responding at T+0s. Three independent health checkers (different network vantage points) each need to agree before the quorum condition is satisfied; assume that agreement lands at T+12s. At that moment the standby's measured replication lag is 1.8 seconds against a 5-second RPO target, well inside budget, so promotion proceeds rather than escalating to a human. The promotion command runs and the standby confirms it's accepting writes by T+13.5s. The load balancer's target group updates on its next health-check cycle, landing at T+14s, and the DNS/service-discovery update for clients that bypass the load balancer follows at T+14.5s. A real smoke test, one read and one write against the newly-promoted primary, passes at T+16s, and the failover is declared complete. Total elapsed time from the primary going unresponsive to service being restored: 16 seconds, and the actual data lost is bounded by the 1.8-second lag measured at the moment of decision, which is the concrete, traced meaning of "RPO at failover time" from the bullet above, not an abstract number.

If promotion fails partway through: the critical invariant is that the system must never end up in a state with two nodes both believing they're primary, known as split-brain, where each node accepts writes independently and the two datasets silently diverge, and it must never route traffic to a target before confirming that target is actually ready. Concretely: if promotion itself fails (the standby doesn't successfully transition), the old primary should remain the traffic target and the on-call gets paged, rather than the system proceeding to update the load balancer toward a standby that isn't actually ready. If promotion succeeds but the smoke test fails, the rollback needs to revert the load balancer and DNS changes, not attempt a second promotion attempt automatically, since two consecutive automated promotion attempts in a short window is exactly the pattern that risks flip-flopping between nodes.

Trade-offs & pitfalls

• Every safety check added between "primary looks down" and "traffic moved" increases failover time; a fully unchecked failover is fast but risks promoting into data loss or acting on a false positive. This time-versus-safety trade-off should be explicit and tuned to the service's RPO/RTO targets, not left implicit.
• A common wrong turn: treating "promotion command returned success" as equivalent to "the new primary is healthy." The smoke test step exists precisely because infrastructure-level success and application-level readiness are different things.
• Automated rollback itself needs to be idempotent and safe to run from a partially-completed state; a rollback path that assumes it starts from a clean "before" state will misbehave if it's invoked after only some of promotion, LB update, and DNS update completed.

Direct answer

Shared infrastructure with strong logical isolation (separate namespaces, per-tenant auth tokens, tenant tagging, resource quotas) is usually the right default, since it pools GPU utilization across customers whose peaks rarely align, cutting cost significantly. Dedicated infrastructure per tenant is worth the extra cost when a customer's contractual or regulatory requirements demand a hard blast-radius boundary, where their data or model weights must never be reachable from another tenant's compute, even in a bug scenario.

Structured elaboration

Shared, logical isolation: pooled GPU utilization means one customer's idle hours cover another's peak, most of the cost saving in multi-tenant serving; the security posture depends entirely on the isolation layer (auth, routing, process isolation) being bug-free, since one flaw there is a cross-tenant leak.

Dedicated per-tenant: no pooling benefit, meaningfully more expensive at the same load; a compromise of the isolation layer cannot cross tenant boundaries since there's no shared compute to cross into; more fleets to patch, but any incident is contained to one tenant.

Worked example

20 customers, each needing a peak of 4 GPUs for 2 hours a day, spread through the day. Dedicated:

$\text{dedicated GPUs} = 20 \times 4 = 80$

Shared, sized to the busiest overlap window (at most 6 customers overlapping at once):

$\text{shared GPUs} = 6 \times 4 = 24$

a little over 3x fewer GPUs. That gap is exactly what a customer with hard isolation requirements, like a bank, is asking you to give up when it demands dedicated infrastructure.

Trade-offs and pitfalls

"Logical isolation" is a spectrum: container-level is weaker than VM-level, which is weaker than physically separate hardware. The mistake is treating isolation as binary instead of naming exactly which layer, network, compute, storage, or model weights, needs separation, since compliance often only demands one specific layer.

What the interviewer probes next

Tenant-scoped quotas to stop a noisy tenant from starving others, whether you'd offer a middle tier of dedicated compute with a shared control plane, and how incident response differs between the two designs.

Direct answer

The mechanics are the same regardless of what is being changed: stand up the new version alongside the old one, shift traffic to it gradually (or all at once for blue-green), watch metrics against a defined threshold before promoting further, and have an automated way to shift traffic back if those metrics degrade. What differs by change type is what "the new version" means and which metric actually tells you it is safe, so the mechanics come first below, then how they change for a few different kinds of infrastructure change.

Structured elaboration

Traffic shifting mechanics

• Canary: route a small percentage of traffic to the new version (for example 5%), hold, then ramp (25%, 50%, 100%) with a wait window at each step.
• Blue-green: fully validate the new environment out of band, then cut traffic over in one step (DNS, load balancer listener swap, or weighted routing swap to 100%). Rollback is just swapping back.

What "the new version" and "the signal to watch" are, by change type

Stateless versus stateful

For a stateless service, this is close to mechanical: deregister the old instance from the load balancer, let connections drain, terminate it, and the new instance takes over with no coordination needed between old and new.

For a service with state, for example one backed by a database or holding session or connection state, the "new version" cannot simply replace the old one at will:

• Deregistration has to wait for graceful connection draining, not just for the health check to fail, or you drop in-flight work.
• If the state itself is what is changing (the DB-class case above), the new version has to be replication-caught-up before it sees real traffic, and you need a plan for what happens to writes that land during the cutover window.
• The old and new versions may be reading or writing the same underlying data store, so canary steps need to account for that shared state, not just for the request path.

Worked example

A concrete case: shifting 5% of production traffic on an ALB to a target group fronting instances built from a new AMI, before fully cutting over.

resource "aws_lb_target_group" "green" {
  name     = "app-green"
  port     = 443
  protocol = "HTTPS"
  vpc_id   = var.vpc_id
  health_check {
    path                = "/health"
    healthy_threshold   = 3
    unhealthy_threshold = 2
    timeout             = 2
  }
}

resource "aws_lb_listener_rule" "canary" {
  listener_arn = aws_lb_listener.app.arn
  priority     = 100

  action {
    type = "forward"
    forward {
      target_group {
        arn    = aws_lb_target_group.green.arn
        weight = 5
      }
      target_group {
        arn    = aws_lb_target_group.blue.arn
        weight = 95
      }
    }
  }

  condition {
    path_pattern { values = ["/*"] }
  }
}

Watch the green target group's 5xx rate and p95 latency for a fixed window (say 15 minutes). If it stays within the same range as blue, bump the weight to 25 and repeat; if 5xx rate or latency breaches the threshold at any step, a second terraform apply (or a CI job wrapping it) sets green's weight back to 0 and blue keeps serving everything.

flowchart TD
  A[Provision green environment via Terraform] --> B[Register in target group, run health checks]
  B --> C[Shift 5 percent traffic to green]
  C --> D{Metrics within SLO?}
  D -- Yes --> E[Ramp to 25 percent, then 50 percent]
  E --> F{Still within SLO?}
  F -- Yes --> G[Shift 100 percent traffic, decommission blue]
  F -- No --> H[Roll back: weight to 0 percent, blue keeps serving]
  D -- No --> H

Trade-offs & pitfalls

• The biggest mistake is treating every infra change like an app deploy and watching only 5xx rate. A security-group rule change will not show up as a 5xx, it shows up as connections that never complete; a DB-class change will not show up in the app's error rate until replication lag has already gotten bad. Pick the signal that matches what actually breaks for that change type.
• Canary steps for stateful changes need explicit wait-for-caught-up gates, not just a timer; ramping traffic to a replica that has not finished catching up just moves the failure into user-visible latency or stale reads.
• Automating the rollback path, not just the promotion path, is what actually buys "close to zero downtime." If pulling the plug requires someone to remember the right terraform apply under pressure, the safety only exists on paper.

Start by defining measurable signals and thresholds: error-budget burn rate (EBBR) over a rolling window (e.g., 24h), current remaining error budget, and incident count/severity. Map ranges of EBBR to discrete policy actions tied to deploy controls, monitoring, and human escalation.

Concrete policy examples:

• Sev-1 (EBBR > 50% in 24h OR remaining budget < 10%): immediate safety posture
  • Pause all non-essential/deferred deployments (automated CI/CD gate)
  • Trigger on-call incident review and war room within 15 minutes
  • Enable expanded tracing and debug-level logs for affected services
  • Block automatic roll-forward of feature flags; require rollback plan
• Sev-2 (EBBR 20–50% in 24h): elevated caution
  • Require pre-merge CI checks to include new load and integration tests
  • Increase dashboarding cadence and anomaly alerts; add temporary synthetic tests
  • Restrict canary sizes and require manual approval for production rollout
  • Schedule a focused reliability review with engineering owners within 24 hours
• Sev-3 (EBBR 5–20% in 24h): watchful
  • Tighten alert thresholds, add short-term throttling or rate-limits
  • Notify product and release managers; require deployment postmortem for any failed deploys
• Normal (EBBR < 5%): normal operations, standard SLO guardrails

Automation and tooling:

• Implement policy engine that reads EBBR + SLO state and enforces CI/CD gates (e.g., pause pipelines, adjust canary config)
• Integrate with alerting/on-call tools to auto-create incidents and route to appropriate responders
• Record every enforced action for audit and postmortem correlation

Why this balances velocity and safety:

• Graduated responses preserve developer velocity when risk is low (only lightweight checks). As risk rises, controls tighten incrementally—blocking risky operations only when necessary—so teams aren’t overly constrained but systemic instability triggers immediate safety actions.
• Automating enforcement removes friction and ambiguity, enabling fast, consistent decisions and faster recovery while making the cost of reliability visible (error budget) to product and engineering stakeholders.