DoorDash Site Reliability Engineer (Junior Level) - Comprehensive Interview Preparation Guide

Direct answer

Find the shared goal underneath the surface disagreement, both sides usually want the launch to succeed, they disagree on what risk is acceptable to get there. Then convert the abstract tension into a concrete, time-boxed trade-off (what ships now versus what's deferred), with clear ownership of whatever risk gets accepted.

Framework

Reframe before negotiating. Name the actual shared objective (a successful launch) instead of letting the conversation stay framed as one function's priority against another's.

Make the trade-off concrete. Lay out a short options list showing what changes at each risk-versus-speed level, and the cost of each option. Where possible, propose a phased release, ship a reduced-risk version now, defer the rest, rather than forcing an all-or-nothing choice.

Assign ownership of the accepted risk. Whoever accepts a shortcut, for example skipping a test cycle or deferring hardening, should be named explicitly, so the decision isn't 'the team decided' with no accountability attached.

Other shapes this same tension takes. It doesn't always surface as engineering-stability-versus-speed. The identical negotiation shows up as design, performance, accessibility, and time-to-market trade-offs, for example a fully accessible, polished interaction versus a simpler version that ships on the marketing date, and as security, network, and product integration-deadline trade-offs, for example a security or network team wanting a longer hardening pass before a product integration ships, against a fixed launch date on the product side. The mechanism doesn't change across these framings: name the shared goal, make the trade-off explicit and time-boxed, and assign ownership of the risk that's accepted.

Worked example

Situation: engineering wanted an additional hardening and testing pass before a release; the business side had a customer commitment tied to a fixed date, eight weeks out.
Action: convened both sides and reframed the disagreement as 'how do we hit the date without an unacceptable stability risk', not engineering against the business. Broke the release into a smaller core scope that could pass full testing within the eight weeks, with the higher-risk pieces deferred to a fast-follow. Named engineering as the owner of the go/no-go call on stability for the core scope, and named the business side as the owner of communicating the phased scope to the customer.
Result: the reduced-risk core shipped on the committed date, and the deferred piece landed two weeks later with no incident. Because the trade-off was explicit and time-boxed rather than a vague 'we'll be a bit more careful', both sides could tell their own stakeholders exactly what was decided and why.

Trade-offs and pitfalls

• Treating this as a one-time negotiation, rather than designing a recurring mechanism such as a standing risk-versus-release framework, means the same fight repeats at every deadline.
• Splitting the difference without being explicit about what's actually being risked satisfies no one and hides the real trade-off from both sides.
• The senior version of this answer describes redesigning the choice so it isn't zero-sum, the phased release, not describing how you convinced the other side to give in.

Direct answer

Treat this like triaging technical debt: first classify every panel by whether it maps to an actual triage decision, aggressively cut or consolidate everything that doesn't, and validate the result against real on-call engineers before calling it done, not just against your own judgment.

Approach

Step 1: audit and classify

Talk to two or three people who've actually used the dashboard during an incident. For every panel, tag it: critical (maps to a golden signal or SLO), useful for drill-down, rarely used, or dead weight with no clear owner or purpose anyone can name.

Step 2: consolidate around golden signals

Replace many single-entity panels with a small set of golden-signal panels (rate, errors, latency, saturation) plus a "top N by errors or latency" table instead of one panel per service or endpoint.

Step 3: use template variables for the long tail

Instead of one panel per service or region, add a dashboard variable and let the engineer select what to drill into. The front page stays small; context-switching happens on demand rather than by scrolling past 35 panels that don't apply to the current incident.

Step 4: redesign layout around the incident workflow

Row 1: is it broken (SLO burn, error rate, active alerts). Row 2: how broken (latency, saturation, dependency health). Row 3 and below: drill-down detail. This mirrors how someone actually works an incident, overview first, detail on demand, instead of a flat grid of 40 equally-weighted panels.

Step 5: validate, don't just ship

Run the new version past on-call for a rotation or two, or a tabletop exercise against the last few real incidents, and explicitly ask whether this panel set would have gotten them to root cause faster. Cut anything that doesn't survive that test.

Worked example

A concrete, honest instance of the consolidation pattern: suppose 15 of the 40 panels are the same latency chart repeated once per microservice. Replacing those 15 with one templated latency panel (a service selector variable) plus one "top 5 services by p99" ranked table removes 13 panels (15 down to 2) while preserving, and arguably improving, the same information, since the ranked table surfaces the worst offender automatically instead of requiring a scroll through 15 charts to spot it. The same consolidation pattern applied to a repeated error-count-per-service panel family would remove a comparable number. The exact final panel count for the full 40 depends on how much of the original sprawl is this kind of per-entity repetition versus genuinely distinct signals, which is precisely what step 1's audit is for.

Trade-offs and pitfalls

• Cutting aggressively without checking with on-call can remove a panel that's rarely used but critical for one specific failure mode, like a replication-lag panel that's silent 99% of the time but is the first thing needed during that one incident type. Validate against past incidents, not just usage frequency.
• Template variables trade a small amount of always-visible context for a click. That's usually the right trade for an overview dashboard, but a signal that's genuinely page-worthy, like overall SLO burn, should stay pinned rather than sit behind a variable.
• Common wrong turn: fixing the dashboard once without establishing an ownership or review process, which is how it reached 40 panels in the first place. Without governance, the same sprawl recurs within a year.
• Don't conflate "unused" with "unnecessary." Some panels are unused specifically because the failure mode they'd catch hasn't happened yet, not because the panel is dead weight.

What right-sizing means

Right-sizing is matching a resource's provisioned capacity (a VM's vCPU/RAM, a container's CPU/memory requests and limits, or a network component's throughput/connection capacity) to what the workload actually uses, based on real historical utilization rather than a guess made at launch time. It cuts both directions: shrinking an over-provisioned resource saves cost, growing an under-provisioned one prevents throttling or saturation.

A practical process

1. Pull historical utilization for the resource over a representative window (at least 2 to 4 weeks, long enough to cover a normal weekly cycle and any monthly peak like a batch job or reporting run).
2. Look at both the average and a high percentile of peak (p95/p99, not the bare maximum, so one anomalous spike does not drive the whole decision).
3. Compare peak utilization against current provisioned capacity to see how much headroom actually exists today.
4. Pick a new size that keeps peak utilization at a safe target (commonly 60 to 80 percent depending on how spiky and business-critical the workload is), not at 100 percent.
5. Roll out gradually and validate before going broad (below).

The same process applies whether the resource is a compute VM/container or a network component: a NAT gateway or ENI (elastic network interface) is right-sized against historical throughput/connection-count metrics the same way a VM is right-sized against CPU/memory. It also applies the same way to an ML batch-training job's VM or container, just against CPU, memory, and wall-clock runtime instead of request-serving metrics; a training job that consistently uses 40 percent of a large instance's memory and finishes early is exactly as over-provisioned as a web server sitting at 20 percent CPU.

Safety buffer

The buffer is the gap deliberately left between the target utilization and 100 percent. For a spiky, latency-sensitive, or hard-to-scale-out component, size for a lower target utilization (more buffer, e.g. 60 percent) because there is less room to react to a surprise. For a smooth, easily-scaled workload with good autoscaling already in place, you can run closer to the edge (less buffer, e.g. 80 percent) because capacity can grow quickly if needed.

Validating before rolling out broadly

Change one instance, one node pool, or one non-critical NAT gateway first, and watch it under real production load for at least one full peak cycle, not just a quiet period, before touching the rest of the fleet. Watch for saturation symptoms specific to the resource (CPU throttling or OOM (out-of-memory) kills, where the operating system forcibly terminates a process for using more memory than it was allowed, for compute, connection or bandwidth limits for a NAT gateway/ENI), keep the change easy to revert, and only widen the rollout once that first change has survived a real peak cleanly.

Direct answer

A Terraform-module checklist at org scale has to cover four layers: the module's public interface (inputs and outputs), its internal implementation quality, its security posture, and its operational safety (versioning, blast radius). Each layer matters more, not less, as more teams consume the module, because a mistake in a shared module multiplies across every consumer's pull requests (PRs, proposed code changes submitted for review) instead of staying contained to one team.

Structured elaboration

Worked example

A platform team reviews a change to a shared vpc Terraform module used by 30 downstream service repositories. The change adds a new enable_flow_logs variable defaulting to true, and removes an old legacy_subnet_cidr variable outright. Applying the checklist: the new variable has a type, a description, and a sensible default, so the interface item passes. But removing legacy_subnet_cidr outright fails the backward-compatibility item, since any of the 30 consumers still passing that variable would fail their next terraform plan with an "unsupported argument" error, all at once, with no warning. The reviewer requires the author to deprecate it first (keep the variable, mark it deprecated in the description, stop using it internally) rather than remove it in the same release, and to bump the module's major version so consumers opt in on their own schedule instead of being broken automatically on their next terraform init.

Trade-offs and pitfalls

• A checklist this thorough adds real review time to every module PR; scope its strictness to modules with many consumers, a single-team internal module doesn't need the same bar
• Security scanners produce false positives on legitimate exceptions, like an intentionally public storage bucket for static assets; the checklist needs an explicit, documented suppression mechanism, not a blanket bypass
• Over-indexing on interface stability can make a module slow to evolve; balance it with a clear deprecation and major-version-bump policy rather than freezing the interface forever
• Skipping the "why it matters at scale" framing turns this into generic Terraform best practices instead of an org-scale review lens; the whole point is that consumer count changes the cost of every mistake

Direct answer

#!/usr/bin/env python3
import socket, time

CPU_SAMPLE_INTERVAL_SECONDS = 0.2

def read_cpu_times():
    with open("/proc/stat") as f:
        fields = [int(x) for x in f.readline().split()[1:]]
    idle = fields[3] + fields[4]           # idle + iowait
    return idle, sum(fields)

def cpu_percent():
    idle1, total1 = read_cpu_times()
    time.sleep(CPU_SAMPLE_INTERVAL_SECONDS)
    idle2, total2 = read_cpu_times()
    total_delta = total2 - total1
    if total_delta <= 0:
        return 0.0
    return 100.0 * (1 - (idle2 - idle1) / total_delta)

def memory_percent():
    values = {}
    with open("/proc/meminfo") as f:
        for line in f:
            key, _, rest = line.partition(":")
            values[key] = int(rest.strip().split()[0])
    total = values["MemTotal"]
    available = values.get("MemAvailable", values["MemFree"])
    return 100.0 * (total - available) / total if total > 0 else 0.0

def main():
    host = socket.gethostname()
    print(f'node_cpu_percent{{host="{host}"}} {cpu_percent():.2f}')
    print(f'node_memory_percent{{host="{host}"}} {memory_percent():.2f}')

if __name__ == "__main__":
    main()

The key design point the question is really testing: /proc/stat's CPU line is a set of
cumulative counters since boot (jiffies spent idle, in user mode, in system mode, and so on), so
a single read can never yield an instantaneous percentage, only "percent of all time since
boot." Getting a live number requires two reads separated by a short interval and computing the
delta, which is exactly what cpu_percent() does. /proc/meminfo, in contrast, already reports
an instantaneous gauge, so memory_percent() needs only one read.

Structured elaboration

CPU math

/proc/stat's first line looks like cpu user nice system idle iowait irq softirq steal guest guest_nice, all in USER_HZ jiffies (a kernel-internal time unit, typically 1/100th of a second,
though the exact value does not matter here since we only ever take a ratio of two deltas, never
an absolute duration). idle + iowait is the time the CPU was not doing useful work; utilization
is 1 - (idle_delta / total_delta) over the sampled window.

Memory math

MemAvailable (not MemFree) is the kernel's own estimate of memory that could be given to a
new process without swapping, accounting for page cache and slab memory the kernel can reclaim
on demand. Using MemTotal - MemFree instead would overstate "used" memory by counting
reclaimable cache as unavailable, which is a very common and very wrong way to compute this
metric.

Prometheus text format

node_cpu_percent{host="myhost"} 12.34 is a valid single-sample line in the Prometheus exposition
format: metric name, an optional {label="value"} block, a space, then the numeric value. This
script prints exactly two such lines, one per metric, both carrying the host label as required.

Worked example

Run against the actual /proc in a Python 3.12 container, once idle and once while a CPU-bound
loop runs in the background:

$ python3 node_metrics.py
node_cpu_percent{host="bf5d4278df38"} 0.69
node_memory_percent{host="bf5d4278df38"} 5.47

$ python3 -c "import time; e=time.time()+1.5
while time.time()<e: pass" &
$ python3 node_metrics.py
node_cpu_percent{host="bf5d4278df38"} 7.32
node_memory_percent{host="bf5d4278df38"} 5.45

CPU utilization visibly rises once the busy loop is running in the background, confirming the
two-sample delta calculation actually tracks real load rather than returning a constant.

Trade-offs and pitfalls

• Inside a container, /proc/stat reflects the host's aggregate CPU activity across every core
  and every other container sharing that kernel, not the calling container's own cgroup (Linux's kernel-level mechanism for grouping processes and capping the CPU, memory, and other resources they may use; a container runtime uses cgroups under the hood to enforce a container's resource limits) CPU
  quota; on a busy shared host this script will under- or over-represent what "this workload"
  is actually consuming. For a container-scoped number, read the cgroup v2 cpu.stat file
  (usage_usec) under the container's own cgroup instead, and divide by wall-clock time and the
  number of CPUs it is entitled to.
• The 0.2 second sample window trades responsiveness for stability: too short and a brief spike
  or scheduling jitter swings the number wildly; too long and the metric lags real load. A
  quarter-second is a reasonable middle ground for a quick scrape, but a real exporter would
  likely cache the previous sample across scrape intervals instead of sleeping inside the
  request path.
• Sleeping inside cpu_percent() means every invocation of this script blocks for at least 0.2
  seconds; that is fine for a one-off Prometheus textfile-collector script run periodically, it
  is the wrong shape for a metric endpoint that must answer many concurrent scrapes fast.
• No error handling around missing keys in /proc/meminfo; the .get("MemAvailable", ...)
  fallback protects an ancient kernel that predates MemAvailable (added in Linux 3.14), but a
  genuinely malformed or truncated /proc/meminfo would still raise KeyError on MemTotal.

Direct answer

Baking an image means pre-installing everything a server needs (OS packages, hardening, the app itself) into a reusable image with a tool like Packer, instead of configuring the server after it boots. Build the pipeline around one principle: nothing reaches production as an image that has not been baked, tested, and scanned the same way every time, and the fleet gets updated by replacing instances behind health checks and connection draining rather than patching them in place. The design has two paths through the same pipeline: the normal path (bake, test, promote through environments, canary, full rollout) and a fast path for urgent security patches that skips environment promotion but never skips the tests or the scan.

Structured elaboration

Image build and test

• CI triggers a Packer build on a base-image or application change: provision the base OS, apply hardening (CIS-style benchmarks, i.e. standardized security-configuration checklists), install the app artifact, and pull secrets via short-lived tokens rather than baking them in.
• Baked-in automated tests run as part of the same pipeline, not as a separate manual step: unit and config-validation tests during the bake, then a post-bake stage that launches the image in an isolated environment and runs integration and smoke tests against it.
• A vulnerability scan (for example Trivy or Grype against the baked image) runs in that same post-bake stage. This is a hard gate, not advisory: an image with a scan finding above the agreed severity threshold does not get published.
• On pass, the image is registered in the artifact registry tagged with its git SHA, build ID, SBOM, and the CVE baseline it passed against, so any later question of "what is actually running" and "was it scanned against what we knew at the time" has an answer.

Promotion across environments

Promotion is a pipeline gate, not a person clicking approve in a console: dev, then staging with regression tests, then a canary slice of production, each gated on the previous stage's tests and monitoring staying green.

Swapping the fleet over

• The fleet sits behind an ASG (or equivalent instance group) and a load balancer. The launch template points at the new image; an ASG instance refresh (or an equivalent rolling-replace controller) walks the fleet in batches.
• Per instance: deregister from the target group first, which starts connection draining; wait for in-flight requests to finish or the drain timeout to hit; only then terminate it. The replacement instance must pass its health check before the load balancer sends it any traffic.
• A minimum-healthy-percentage setting (for example 90%) caps how much capacity can be replacing at once, so a bad new image degrades a fraction of the fleet rather than all of it while it is still being watched.
• For workloads that carry state (a service with long-lived connections, or one with session affinity), connection draining alone is not enough: the drain window also has to respect existing session affinity, and if any part of the workload is stateful in the sense of holding data (not just connections), that has to coordinate with the data layer's own replication or failover process rather than treating the instance as freely swappable the moment its health check fails.

Fast-tracking an urgent security patch

The fast path changes how far the image travels before real traffic sees it, not whether it is tested:

• Skip the full dev-then-staging promotion chain; go straight from bake to a canary slice of production.
• Keep the bake-time tests and the vulnerability scan as hard gates; an urgent patch that has not been scanned is exactly the failure mode a patch process exists to prevent.
• Shorten, but do not remove, the canary observation window, and have the rollback path pre-verified rather than improvised, since this path is exercised under time pressure.
• Immediately backfill afterward: once the emergency patch has gone through the fast path, run it (or its base) through the normal dev and staging pipeline the following day, so the fast-tracked version does not become a permanent exception living outside the standard promotion history.

Worked example

flowchart TD
  A[Source or base image change] --> B[CI triggers Packer bake]
  B --> C[Bake time tests: hardening, vuln scan, smoke tests]
  C --> D[Publish image with SBOM and CVE tags to registry]
  D --> E[Promote through dev then staging]
  E --> F[Canary: weighted traffic on new AMI]
  F --> G{Health checks and SLOs pass?}
  G -- Yes --> H[Full fleet rollout via ASG instance refresh]
  G -- No --> I[Roll back to prior AMI, tag new image as bad]
  B -.urgent security patch.-> J[Fast path: skip dev and staging, bake plus scan only]
  J --> F

Concretely: a CVE lands in the base OS image. CI triggers a Packer bake immediately (the dashed path above). The bake produces a new image; the same automated tests and the same vulnerability scan run against it as any normal build, just without waiting for a scheduled promotion window. It goes straight to a canary slice of the fleet, monitored against the same health checks and error-rate thresholds as any other rollout, then to the full fleet via instance refresh. The following day, the same image is run through the normal dev and staging environments to confirm nothing outside the emergency scope regressed.

Trade-offs & pitfalls

• Baking images takes longer than patching in place, and that trade-off is deliberate: reproducibility and a clean rollback (revert the launch template to the previous image ID) are worth the extra build minutes.
• The most dangerous version of a "fast path" is one that quietly also skips testing or scanning under time pressure; the fast path should only ever shorten promotion, never verification.
• For stateful workloads, connection draining and health checks are necessary but not sufficient; assuming they are enough to make image replacement safe for anything holding data is a common design mistake.
• Rolling back an in-flight instance refresh needs to be a rehearsed, one-command action (point the launch template back at the previous image ID), not something improvised the first time it is needed.

Direct answer
An automated rollback orchestrator has three jobs done in a strict order: detect that the deployment is failing, pause the rollout so it stops making things worse, and then execute the rollback across every affected service in an order that respects their dependencies, followed by a health check that confirms the system actually landed in a good state and not just a different bad one.

Structured elaboration
Core components:

1. Detector: subscribes to the same metrics/health signals as the canary-analysis gate (error rate, latency, health checks) and raises a rollback INTENT, not the rollback itself.
2. Pause controller: immediately halts further traffic ramp or pod replacement so the blast radius stops growing while the system decides what to do.
3. State store: records, per deployment, the previous good artifact/version for every service touched, the order services were deployed in, and which have already completed rollback. This has to be durable (survive the orchestrator crashing mid-rollback), because a rollback that itself half-completes and then loses track of state is worse than the original failure.
4. Rollback executor: walks the dependency graph in reverse deployment order (or a computed safe order if services were deployed in parallel), issuing the rollback action per service: redeploy the previous artifact, flip the associated feature flag off, or shift traffic weight back to the stable version, depending on what mechanism that service was deployed with.
5. Post-rollback validator: re-runs the same health checks used at deploy time against the rolled-back state; a rollback that "completes" without this step can silently leave the system on a version that's ALSO broken.

Handling a partially completed rollback: the state store's per-service status (not-started / in-progress / done / failed) lets the orchestrator resume from where it left off on restart, and a service stuck in "in-progress" after a timeout should page a human rather than retry forever, since retrying a stuck rollback blind can double-apply a destructive action.

Worked example
Three services deploy together: A -> B -> C, where B depends on A's new API and C depends on B's new schema. If C's rollout fails, the safe rollback order is C, then B, then A, the reverse of the deploy order, because rolling back A first while B still expects A's new behavior would break B on a system that was working seconds earlier.

Trade-offs and pitfalls
Fully automating this across many services is powerful but risky if the dependency graph is wrong or stale; a common failure mode is an orchestrator that trusts a hand-maintained dependency list that's drifted from reality. A pragmatic middle ground many teams adopt is to automate rollback fully for a defined, lower-blast-radius subset of critical services first, and require a human confirmation step for the rest, tightening the automated scope as confidence in the detector and executor grows.

Direct answer

Disaster recovery for AI workloads splits into two different problems: recovering the training and artifact side, where large model checkpoints and feature data dominate RTO (recovery time objective: how long restoring service takes) and RPO (recovery point objective: how much data, measured in time, you could lose) because moving multi-gigabyte to multi-terabyte binary state takes real time, and recovering the serving side, which is closer to a normal stateless-service failover once the model artifact is in place. Hot, warm, and cold standby differ mainly in whether the checkpoint transfer happens continuously in the background (hot) or on demand at failover time (warm or cold), because that transfer time is usually the dominant term in RTO for this workload, not compute provisioning.

Structured elaboration

The four things that need a DR story

• Model artifacts (checkpoints): large, immutable once written, versioned; the dominant factor in RTO if not pre-replicated.
• Feature data: online (a low-latency key-value store needing near-real-time replication for serving correctness, typically via a CDC stream, change data capture: a feed of every row-level change read off the primary store's own write log and replayed onto the replica, shown as the CDC node below) and offline (batch or data-lake, tolerant of periodic snapshot replication).
• In-flight requests and training state: inference should be stateless per request and retry-safe; training needs its own frequent checkpointing so a mid-run failure doesn't lose more than one checkpoint interval of compute.
• DNS and routing failover: the same mechanics as any service, a global load balancer with active health checks, with the standard TTL and anycast trade-offs on top. Concretely: DNS-based failover depends on clients, and any resolver or CDN caching the answer along the way, picking up the new IP once the DNS record for the service is updated, but a record published with, say, a 60-second TTL (time-to-live: how long a resolver is allowed to keep serving the old, cached answer before it's required to ask again) means some fraction of clients keep sending requests to the now-dead region for up to that long, and a resolver or client that ignores the TTL and caches even longer makes it worse, which happens in practice. Anycast sidesteps that specific failure mode: the same IP address is announced from every region at once, and the network's own routing layer sends each client to whichever announcing region is currently reachable and closest, so pulling the failed region out of the announcement removes it from routing at the network layer, typically within seconds, without depending on any client or resolver noticing a DNS change at all. The trade-off is operational: anycast needs control over that network-level announcement, usually via a cloud provider's global load balancer or CDN edge rather than something most teams operate directly, while plain DNS failover is simpler to stand up but trades that simplicity for the TTL-bounded propagation delay described above.

flowchart TD
    T[Training, Region A] --> CK[Checkpoint writer, every ~5 min]
    CK --> OS[(Object storage, cross-region replicated)]
    OS --> WB[Standby Region B loads latest checkpoint]
    FS[Online feature store, Region A] --> CDC[CDC stream]
    CDC --> FSB[Feature store replica, Region B]
    GLB{Global LB health check} --> INF_A[Inference serving, Region A]
    GLB --> INF_B[Inference serving, Region B standby]
    WB --> INF_B
    FSB --> INF_B

Hot vs warm vs cold, defined by what's pre-staged

Worked example: checkpoint size and transfer time

Model: 70 billion parameters, stored at fp16, 2 bytes per parameter:

Cold standby RTO, checkpoint transferred at failover time, assuming 5 Gbps effective cross-region bandwidth, 625 MB/s:

Adding infra provisioning and model load, cold-standby RTO is easily 15 to 30-plus minutes once GPU allocation and container startup are included; the transfer time is the floor, not the total.

Warm or hot standby RTO, checkpoint already local, no transfer on the critical path, loading from local fast storage at 3 GB/s effective read and deserialize throughput:

That is roughly a 5x improvement over the cold-standby transfer time alone, 224s versus 47s, before even counting cold's additional provisioning time. Pre-replicating the checkpoint, paying the transfer cost continuously and off the critical path, is the single biggest RTO lever for this workload.

Worked example: checkpoint cadence and its RPO trade-off

Writing the 140GB checkpoint to durable object storage at 1 GB/s achievable throughput:

Checkpointing more often than roughly twice that write time risks writes overlapping or backing up, so a practical floor on full-checkpoint cadence here is around 5 minutes, giving an RPO of up to 5 minutes of lost training compute with full checkpoints only, mitigated by cheaper, more frequent optimizer-state-only incremental checkpoints between full ones.

Trade-offs & pitfalls

• Treating training-state RPO and inference-serving RPO as the same number is a common mistake: losing 5 minutes of training compute is an efficiency cost, losing 5 minutes of in-flight user requests is a correctness and trust cost. Size the DR investment to the failure mode's actual severity, not one blanket target.
• A "hot" standby that continuously replicates checkpoints but has never actually served production traffic in a drill is not verified hot, it's an assumption; the transfer pipeline can silently degrade, falling behind or corrupting a chunk, without anyone noticing until failover.
• GPU driver and container image version skew between regions is an AI-specific failure mode a generic DR checklist misses: a checkpoint that deserializes fine but produces different or broken output on a different driver or runtime version is a DR test failure that only shows up if inference is actually run against it during drills, not just a file-transfer check.
• Feature store replication lag for online features is easy to under-scope: if the online store's replication guarantee is looser than the model's sensitivity to stale features, a technically successful failover can still serve degraded predictions.

Direct answer

I run every piece of feedback through three checks before deciding whether to incorporate it or push back: how specific and testable is the claim, how strong is the evidence behind it, and how expensive and reversible is being wrong either way. Strong, specific evidence on a cheap-to-reverse decision gets incorporated quickly; vague preference on an expensive, hard-to-reverse decision gets pushed back on, or at least held for more evidence.

Structured elaboration

Evaluation criteria I weigh:

• Specificity. Is the feedback concrete enough to act on ("this function has six responsibilities and caused two of the last three bugs in this module"), or is it a vague preference ("this feels overcomplicated")? Vague feedback is not automatically wrong, but it needs a clarifying question before it can be evaluated at all.
• Evidence type. I rank convincingness roughly as: direct measurement or a reproducible failure case, above documented precedent (this exact pattern has caused a problem before), above a well-reasoned but unverified opinion, above an unsubstantiated opinion stated with confidence.
• Source's proximity to the outcome. Someone who owns the downstream consequence, such as the person who gets paged when this breaks or the user who hits the bug, usually has stronger standing on that specific claim than someone commenting from a distance, even if the distant commenter is more senior in general.
• Consistency with other signals. Does this match what other independent reviewers or past incidents have already flagged, or is it a one-off?
• Cost of being wrong either way. If incorporating the change is cheap and reversible, I lean toward incorporating even on moderate evidence. If it is expensive or hard to reverse, I require stronger evidence before either agreeing or confidently pushing back.

Types of evidence I find convincing, most to least: reproducible data such as metrics, logs, or a benchmark; a concrete failure case that demonstrates the flaw directly; a documented precedent where this exact pattern caused a problem before; a second independent reviewer agreeing unprompted. Least convincing: a stylistic preference stated with confidence, an appeal to authority alone ("I've been doing this longer") with no mechanism attached, or feedback that cannot specify what a better version would actually look like.

How I communicate the rationale to stakeholders with differing priorities. I separate the decision from the person: I state which criteria I weighed and why they landed where they did, rather than just announcing a conclusion. I invite the stakeholder to challenge the weighting itself, not just restate their original conclusion, since disagreement is often really about which criterion should dominate. Where priorities genuinely differ, for example speed versus long-term maintainability, I say so explicitly rather than pretending it is a purely factual dispute, and I name who has the authority to make that trade-off call if we cannot agree.

Worked example

A reviewer comments, "this function is too complex, refactor it." First pass: specificity is low, "too complex" alone is not actionable, so I ask what specifically makes it hard to follow. Suppose they clarify with a concrete signal: the function has six distinct responsibilities and was the site of two of the last three bugs in this module, a documented precedent. Cost of being wrong: the refactor is contained to one file and easily reversible if it goes badly. Strong evidence plus low reversibility risk means: incorporate.

Contrast that with a different reviewer on the same file who says, "I would have named this function differently." Specificity is moderate, but the evidence is pure preference with no functional impact, and the cost of the current name is negligible either way. My response: apply it as a welcome but non-blocking suggestion rather than a required change, and say so directly rather than silently ignoring it.

Trade-offs and pitfalls

Turning this into a formal scorecard for every comment makes normal discussion feel like an audit; it works best as a mental checklist, not something I table explicitly except on genuinely contested calls. A senior reviewer's terse comment can look like weak evidence when it is really just brevity from someone who trusts me to fill in the specifics, so I ask before discounting rather than assuming vague means unfounded. The biggest risk is using "insufficient evidence" as a shield to dodge feedback I simply do not like; the rubric only holds up if I apply the same bar to feedback that confirms my instincts as to feedback that contradicts them.

Direct answer

Every pod gets a routable IP address from the CNI (Container Network Interface) plugin, with no NAT (network address translation) between pods by default. A Service layers a stable virtual IP and DNS name on top of that, and kube-proxy (or an eBPF-based replacement like Cilium; eBPF lets Linux run sandboxed programs inside the kernel to process packets directly) programs the actual rewriting rules that turn a Service IP into a real pod IP. When two pods on different nodes cannot reach each other on a specific port, the fault sits in exactly one of four layers, checked in this order: the pod's own listener, the kube-proxy/Service plumbing, the CNI's cross-node routing, or a host/cloud firewall. Testing each layer directly beats guessing from symptoms.

Structured elaboration

Pod IP model

The CNI plugin assigns each pod a unique IP from the cluster's pod address range and is responsible for making that IP reachable from every node, either by programming host routes directly (an "underlay" approach, for example Calico in BGP mode, Border Gateway Protocol, the routing protocol routers use to exchange reachability information) or by encapsulating pod traffic in an overlay (VXLAN, a tunneling protocol that wraps one Ethernet frame inside a UDP packet, used by Flannel and by Calico's IP-in-IP mode).

Service virtual IPs and kube-proxy modes

A ClusterIP Service is a stable virtual IP that never corresponds to a real listener; kube-proxy rewrites traffic to it into a real pod IP and port.

Cilium and some other CNIs bypass kube-proxy entirely, attaching eBPF programs to kernel networking hooks to do the same rewriting job with lower per-packet overhead and without iptables' rule-count scaling problem.

CNI plugins in practice

• Calico: pure L3 routing via BGP (no overlay, lowest overhead, needs a BGP-capable network) or IP-in-IP/VXLAN encapsulation where BGP peering is not available.
• Flannel: a simple VXLAN overlay by default; fewer features than Calico or Cilium.
• Cilium: eBPF-based; can replace kube-proxy, adds L7-aware NetworkPolicy and traffic observability, at the cost of needing a reasonably current kernel.

flowchart LR
  A[Pod A on Node 1] -->|packet to Service ClusterIP| KP1[kube-proxy or eBPF dataplane, Node 1]
  KP1 -->|rewrite to real Pod IP| CNI1[CNI routing or overlay, Node 1]
  CNI1 -->|cross-node route or tunnel| CNI2[CNI routing or overlay, Node 2]
  CNI2 --> B[Pod B on Node 2]
  FW[Host firewall or cloud security group] -.may block specific ports.-> CNI2

Debugging cross-node connectivity, layer by layer

Reproduce and localize before changing anything:

$ kubectl get pod -o wide -n app
NAME      READY   STATUS    NODE     IP
pod-a     1/1     Running   node-1   10.244.1.7
pod-b     1/1     Running   node-2   10.244.2.9

$ kubectl exec -n app pod-a -- nc -vz 10.244.2.9 8080
nc: connect to 10.244.2.9 port 8080 (tcp) failed: Connection timed out

A timeout, not "connection refused," means the packet is being dropped somewhere in transit rather than rejected by pod-b's listener; that already rules out "the app isn't listening" and points at routing, the CNI, or a firewall.

1. Confirm the listener exists: kubectl exec -n app pod-b -- ss -tlnp should show something bound to 8080; if not, this was never a networking problem.
2. Check node-to-node routing for the pod's address range:

$ ip route get 10.244.2.9
10.244.2.9 via 10.0.1.2 dev eth0 src 10.0.1.5

A missing route, or one pointing at the wrong interface, means the CNI failed to program cross-node routing, usually because its daemonset is crashlooping:

$ kubectl get pods -n kube-system -l k8s-app=calico-node
NAME                READY   STATUS             RESTARTS
calico-node-x7k2p   0/1     CrashLoopBackOff   14

3. If the failure is specifically Service-IP traffic rather than direct pod-IP traffic, check kube-proxy's own health the same way (kubectl get pods -n kube-system -l k8s-app=kube-proxy) and inspect its programmed rules (sudo iptables -t nat -S | grep <service-name> or sudo ipvsadm -Ln, depending on mode).
4. If routing and the CNI both look healthy, check host-level and cloud firewall rules for the specific port. CNI and kube-proxy problems tend to affect all ports uniformly; a single-port failure with everything else in that layer healthy is a strong signal for a firewall rule scoped to that port.
5. Capture packets at the suspect boundary to confirm rather than infer: sudo tcpdump -n -i any host 10.244.2.9 and port 8080 on the destination node shows whether the packet arrives at the host at all, before deciding whether the problem is upstream (routing) or downstream (host firewall, pod network namespace).

Trade-offs and pitfalls

• iptables rule count grows linearly with Services and endpoints and becomes a real CPU cost past a few thousand Services; IPVS or nftables scale better, but changing kube-proxy mode cluster-wide is a disruptive change to plan and test, not to flip live.
• The distinction between "Connection timed out" and "Connection refused" is easy to skip past but changes where to look first: refused means the packet arrived and something rejected it (wrong port, no listener); timed out means the packet was dropped somewhere before that.
• MTU (maximum transmission unit) mismatches between overlay encapsulation and the underlying network are a classic cause of connections that work for small packets and hang for large ones. Confirm with a full-size, non-fragmenting ping rather than assuming a working small ping rules out MTU.