Service Discovery and Configuration Management Questions

Direct answer

Treat it as a reconciliation loop (the same shape a Kubernetes controller uses): on every tick, first observe the health of the whole fleet, then compute a budget of N minus everything currently unhealthy, and only start that many new config applies. Instances that are restarting, waiting to retry, or were already broken before you started all count against the budget, which is what makes "at most N unhealthy" hold rather than "at most N of my operations in flight". Failed applies retry with capped exponential backoff and jitter, keeping their budget slot while they wait; the rollout aborts when too many instances fail for good or when nothing has made progress for too long, and it reports progress every tick.

Approach

Two ideas carry the whole design:

1. The invariant is about the fleet, not about the orchestrator. A naive version uses a semaphore (a counter that lets at most N callers proceed at once and blocks the rest until one finishes) of size N around apply_config. That bounds your in-flight operations, but if two instances were already down when you started, you now have N + 2 unhealthy. So the budget must be computed from observed health: budget = N - unhealthy_now.
2. Observe, then act, once per tick. Rather than threads racing each other, one loop per tick (a) reads health for every instance, (b) advances each in-flight instance's state machine, (c) checks abort criteria, (d) admits new work up to the budget, (e) reports. This makes the logic deterministic and testable, and it is the same pattern whether a tick is one second in production or one step in a simulation.

Each instance moves through a small state machine:

stateDiagram-v2
  [*] --> pending
  pending --> applying: budget available
  pending --> skipped: unhealthy before rollout
  applying --> done: health_check passes
  applying --> backoff: ready_timeout expired, attempts left
  backoff --> applying: backoff elapsed, re-apply
  applying --> failed: ready_timeout expired, no attempts left

Here apply_config(instance) is taken to start the restart and return; the loop then polls health_check(instance) until it passes or ready_timeout expires. If your apply_config blocks until the restart finishes, run it in a worker thread per admitted instance and let the loop observe it the same way.

Code

The orchestrator is rolling_apply; everything under the "simulated fleet" line is a deterministic test harness (seeded random restart times of 2 to 4 ticks, instances that never recover on bad config, and instances that were sick before the rollout). The harness independently records the worst number of instances down at once, so the invariant is checked by something other than the code that enforces it.

import random
from dataclasses import dataclass

@dataclass
class Policy:
    max_unhealthy: int = 3      # N: never more than this many instances unhealthy at once
    ready_timeout: int = 6      # ticks to wait for health after an apply
    max_attempts: int = 3       # apply attempts per instance before it counts as failed
    backoff_base: int = 1       # retry delay = min(cap, base * 2**(attempt-1)) + jitter
    backoff_cap: int = 8
    max_failed: int = 2         # abort when this many instances have failed for good
    max_stall_ticks: int = 20   # abort if nothing finishes for this long

@dataclass
class Task:
    name: str
    state: str = "pending"      # pending | applying | backoff | done | failed | skipped
    attempt: int = 0
    deadline: int = 0
    wake_at: int = 0

FINAL = ("done", "failed", "skipped")

def rolling_apply(instances, health_check, apply_config, policy, rng, clock, report=print):
    """Reconciliation loop: every tick, observe health first, then act on the observation."""
    tasks = {i: Task(i) for i in instances}
    for i in instances:                         # sick before we touched it: do not restart it,
        if not health_check(i):                 # but it still uses up part of the N budget
            tasks[i].state = "skipped"
            report(f"t={clock.now:>3} {i} skipped: unhealthy before rollout")
    last_progress = clock.now
    while True:
        healthy = {i: health_check(i) for i in instances}
        unhealthy = sum(not h for h in healthy.values())

        # 1. advance in-flight work
        for t in tasks.values():
            if t.state == "applying" and healthy[t.name]:
                t.state, last_progress = "done", clock.now
            elif t.state == "applying" and clock.now >= t.deadline:
                if t.attempt >= policy.max_attempts:
                    t.state, last_progress = "failed", clock.now
                    report(f"t={clock.now:>3} {t.name} FAILED after {t.attempt} attempts")
                else:
                    delay = min(policy.backoff_cap, policy.backoff_base * 2 ** (t.attempt - 1)) + rng.randint(0, 1)
                    t.state, t.wake_at = "backoff", clock.now + delay
                    report(f"t={clock.now:>3} {t.name} not ready, retry in {delay}")
            elif t.state == "backoff" and clock.now >= t.wake_at:
                t.attempt += 1                  # still unhealthy while backing off: keeps its slot
                t.state, t.deadline = "applying", clock.now + policy.ready_timeout
                apply_config(t.name)

        # 2. abort criteria
        n_failed = sum(t.state == "failed" for t in tasks.values())
        if n_failed >= policy.max_failed:
            return "ABORTED: too many failed instances", _stats(tasks, clock)
        if clock.now - last_progress > policy.max_stall_ticks:
            return "ABORTED: no progress", _stats(tasks, clock)
        if all(t.state in FINAL for t in tasks.values()):
            return ("COMPLETE" if n_failed == 0 else "COMPLETE WITH FAILURES"), _stats(tasks, clock)

        # 3. admit new work: budget counts EVERY unhealthy instance, ours or not
        budget = policy.max_unhealthy - unhealthy
        for name, t in tasks.items():
            if budget <= 0:
                break
            if t.state == "pending":
                t.attempt, t.state, t.deadline = 1, "applying", clock.now + policy.ready_timeout
                apply_config(name)              # a restart takes it out of rotation now
                budget -= 1
        done = sum(t.state == "done" for t in tasks.values())
        report(f"t={clock.now:>3} progress {done}/{len(tasks)} done, {unhealthy} unhealthy, budget left {max(budget, 0)}")
        clock.now += 1

def _stats(tasks, clock):
    c = {"ticks": clock.now}
    for t in tasks.values():
        c[t.state] = c.get(t.state, 0) + 1
    return c

# ---------------- simulated fleet (deterministic, seeded) ----------------
class Clock:
    now = 0

class Fleet:
    """Fake instances. Independently records the worst number of instances down at once."""
    def __init__(self, n, bad, sick, rng, clock):
        self.clock, self.rng, self.bad = clock, rng, set(bad)
        self.up_at = {f"i{k:02d}": 0 for k in range(n)}
        for s in sick:
            self.up_at[s] = float("inf")
        self.worst = 0
    def health_check(self, i):
        return self.clock.now >= self.up_at[i]
    def apply_config(self, i):                  # restart: down for 2-4 ticks; bad config never recovers
        self.up_at[i] = float("inf") if i in self.bad else self.clock.now + self.rng.randint(2, 4)
        down = sum(self.clock.now < t for t in self.up_at.values())
        self.worst = max(self.worst, down)

def run(label, bad, sick, policy=Policy()):
    clock, rng = Clock(), random.Random(7)
    fleet = Fleet(12, bad, sick, rng, clock)
    events = []
    outcome, stats = rolling_apply(list(fleet.up_at), fleet.health_check, fleet.apply_config,
                                   policy, rng, clock, report=events.append)
    print(f"--- {label}: {outcome}  (worst down at once: {fleet.worst}, N = {policy.max_unhealthy})")
    print("   ", stats)
    for e in events:
        if "progress" not in e:
            print("   ", e)
    return events

ev = run("clean fleet", bad=[], sick=[])
print("    first progress lines:", *ev[:3], sep="\n    ")
run("one box sick before start, one bad box", bad=["i05"], sick=["i09"])
run("two bad boxes", bad=["i02", "i03"], sick=[])

Running it (python3 rollout.py) prints:

--- clean fleet: COMPLETE  (worst down at once: 3, N = 3)
    {'ticks': 13, 'done': 12}
    first progress lines:
    t=  0 progress 0/12 done, 0 unhealthy, budget left 0
    t=  1 progress 0/12 done, 3 unhealthy, budget left 0
    t=  2 progress 1/12 done, 2 unhealthy, budget left 0
--- one box sick before start, one bad box: COMPLETE WITH FAILURES  (worst down at once: 3, N = 3)
    {'ticks': 28, 'done': 10, 'failed': 1, 'skipped': 1}
    t=  0 i09 skipped: unhealthy before rollout
    t= 13 i05 not ready, retry in 1
    t= 20 i05 not ready, retry in 2
    t= 28 i05 FAILED after 3 attempts
--- two bad boxes: ABORTED: too many failed instances  (worst down at once: 3, N = 3)
    {'ticks': 23, 'done': 9, 'failed': 2, 'applying': 1}
    t=  6 i02 not ready, retry in 1
    t=  8 i03 not ready, retry in 1
    t= 13 i02 not ready, retry in 2
    t= 15 i03 not ready, retry in 2
    t= 21 i02 FAILED after 3 attempts
    t= 23 i03 FAILED after 3 attempts

What the three runs show:

• Clean fleet: 12 instances finish in 13 ticks and the harness never saw more than 3 down at once. At t=0 the loop observed 0 unhealthy and admitted 3 (budget left 0); at t=2 one finished and one more was admitted.
• Pre-sick box plus bad box: i09 was unhealthy before the rollout, so it is skipped (restarting a box you do not understand makes debugging harder) but it still occupies one of the 3 slots. i05's bad config never becomes healthy; it retries with backoff 1, then 2, then fails. While i05 is failing, only one slot is free, so the rollout slows to one instance at a time instead of breaking the invariant. It completes with 1 failure, below the abort threshold of 2.
• Two bad boxes: the second permanent failure trips max_failed = 2 at t=23, and the loop stops admitting work and returns ABORTED. By that point every one of the 12 instances has already been touched, matching the printed {'done': 9, 'failed': 2, 'applying': 1}: nine finished normally, i02 and i03 failed for good, and one instance is still mid-apply (state applying) when the abort fires, left exactly where its restart was, neither confirmed healthy nor failed. None are left untouched.

Key points

• Scheduling strategy: instances are admitted in list order, so the caller controls the wave shape: put a canary (a single instance updated first and watched before the rest follow, so a bad config is caught on one box instead of the whole fleet) first, then order by availability zone (AZ: an independent data center or facility with its own power and network, so one AZ at a time limits the blast radius, meaning how much of the fleet one bad change can take down). A common refinement is to admit only 1 until the canary is healthy, then open the budget to N.
• Concurrency control: the budget is recomputed every tick from observed health, so it self-corrects: a box that crashes on its own mid-rollout automatically shrinks the admission budget.
• Backoff and retry: delay = min(cap, base × 2^(attempt-1)) plus a random 0 or 1 tick of jitter (random spread so many retrying instances do not re-hit a config server in lockstep). Crucially, an instance in backoff is still unhealthy and keeps its slot; releasing the slot while it waits is the classic way to break the invariant.
• Abort criteria: (1) max_failed permanent failures, (2) no instance finished or failed for max_stall_ticks, which catches a fleet that is stuck because unhealthy instances consumed the whole budget. In production add (3) a fleet-level signal such as the service's error rate burning its SLO (service-level objective) while the rollout is running.
• Surfacing progress: the report callback emits one line per tick with done count, unhealthy count and remaining budget, plus an event for every retry and failure. Wire it to a log and a status endpoint; the final return value is the machine-readable summary.
• What abort does not do: it stops the rollout, it does not undo it. The honest next step is a second rolling pass of the previous config through the same function over the instances marked done and failed, under the same N budget, so the rollback cannot itself breach the invariant.

Complexity

Each tick is O(I) work for I instances (one health read and one state update each), so a rollout of T ticks costs O(I × T) health checks. Memory is O(I) for the task table. In the clean run (I = 12, ticks 0 through 13 inclusive) that is 12 × 14 = 168 reads in the loop plus 12 for the initial "already sick?" scan, 180 in total (counted by wrapping health_check with a counter). Real systems cap health-check cost by caching the fleet health snapshot from the registry or load balancer instead of probing every instance every tick.

Edge cases

• N = 0 or N below the number already sick: the budget is never positive, nothing is admitted, and the stall rule aborts. Validate N >= 1 up front and refuse to start if the fleet is already at or beyond N unhealthy.
• Health flaps between observation and admission: an instance that becomes unhealthy after this tick's snapshot makes the next tick's count one higher; the invariant can be exceeded for at most one tick. Shorten the tick or re-read health immediately before each apply_config if that window matters.
• health_check itself fails or times out: treat "unknown" as unhealthy. Counting it as healthy would let the loop admit work into a fleet it cannot see.
• A healthy-looking bad config: health checks pass but business errors rise. Only a fleet-level SLO abort criterion catches that, which is why it belongs in production abort rules.
• Two orchestrators on the same fleet: both compute budget from the same observed health and each admits N. Take a lease-based lock (a lock that expires if the holder dies) on the service before starting.

Trade-offs and pitfalls

• Semaphore-only designs bound operations, not unhealthy instances; they pass unit tests and fail on the first day a box is already down.
• Small N is safe but slow: at N = 3 with restarts taking 2 to 4 ticks, 12 instances took 13 ticks; 1,200 instances would take roughly 100 times as long. Set N as a percentage of fleet size, and check it against spare capacity by dividing, not subtracting: if the fleet runs at 70% average utilization and N takes 10% of it offline at once, the remaining 90% of instances must carry that same load, so their utilization becomes 70% / 90% ≈ 77.8%, still under 100% and safe. Taking 40% offline at once leaves only 60% of instances carrying that load: 70% / 60% ≈ 116.7%, over capacity, and requests queue or fail. In general, keep N (as a fraction of the fleet) below 1 - utilization, here below 30%, with margin left over for the retries and stragglers the rollout itself creates.
• Retrying a deterministic failure only wastes time: a config that fails to parse will fail three times. Classify failures (retryable: timeout, dependency not ready; non-retryable: validation error) and fail fast on the second kind.

Direct answer

Use Vault as the single source of truth, have every pod authenticate as itself using its Kubernetes service account token, and deliver secrets as files on an in-memory volume written by a node-level agent (the Secrets Store CSI Driver with the Vault provider), with rotation enabled. The application re-reads the file when it changes, which is what makes rotation work without a restart. Prefer short-lived dynamic credentials (Vault generates a database user per workload with a lease) over long-lived static secrets. Keep Vault Agent sidecars for the minority of workloads that need templating or per-pod lease renewal, and do not use plain Kubernetes Secrets as the source of truth for sensitive material.

Terms

• Vault: HashiCorp's secrets manager. It stores secrets, issues dynamic secrets (credentials created on demand with an expiry), and writes an audit log of every request.
• Lease / TTL: a dynamic secret comes with a time-to-live. The holder must renew it before expiry or get a new one; Vault revokes it afterwards.
• Least privilege: each workload can read only the secrets it needs, and nothing else.
• Kubernetes auth method: a pod presents its service account token (a signed JWT, JSON Web Token) to Vault; Vault verifies it with the Kubernetes API and maps the service account to a Vault policy.
• CSI (Container Storage Interface) driver: a node-level plugin that mounts volumes into pods. The Secrets Store CSI Driver mounts secrets fetched from an external store as files.
• DaemonSet: a Kubernetes workload type that runs exactly one copy of a pod on every node (or every matching node), unlike a Deployment which runs a chosen number of copies wherever the scheduler likes. It is the standard way to run one node-level agent per machine.
• Audience-bound token: a service account token that is only valid for a declared recipient (Vault, here); even if it leaked, another service could not use it to authenticate as this pod to something else.
• Alpha (Kubernetes feature stage): an early, experimental stage of a feature. It can change behaviour or be removed between versions with no long-term compatibility guarantee, and many production platforms disable alpha features by policy.
• Rotation: replacing a credential with a new one while the old one is still valid for an overlap period, so nothing breaks mid-swap.

The four options, compared

The design

flowchart LR
  Pod[App pod with service account] -->|mounts in-memory volume| CSI[CSI driver on node]
  CSI -->|pod's own SA token| Prov[Vault provider on node]
  Prov -->|Kubernetes auth login| Vault[Vault cluster]
  Vault -->|verify token| K8s[Kubernetes API]
  Vault -->|policy per service account| Prov
  Vault --> Audit[Audit log sink]
  Prov -->|write files, re-fetch on rotation| Pod

1. Identity. Every workload gets its own service account. The CSI driver passes the pod's projected service account token (a short-lived, auto-rotated token Kubernetes mounts into the pod for exactly this purpose, restricted to Vault as its audience) to the provider. Vault's Kubernetes auth role maps namespace=payments, sa=payments-api to a policy that allows only database/creds/payments-ro (the database secrets engine, generating a dynamic, read-only credential named payments-ro) and kv/data/payments/* (everything under the payments folder of the plain key-value secrets engine).
2. Delivery. Secrets land as files in a tmpfs (memory-backed) volume, never on node disk and never in environment variables (env vars leak into crash dumps and child processes: by default, every process an application spawns inherits a copy of its parent's environment, so a secret sitting in an env var ends up in tools and scripts that have no reason to see it, and cannot change without a restart).
3. Rotation without restarts. Three things need to work together:
   • Delivery side: the CSI driver re-fetches on its poll interval and rewrites the file when Vault returns a new value.
   • Application side: the app watches the file (inotify, the Linux file-change notification API, or re-reading on a timer) and swaps its connection pool or client credentials in place. This is the part teams forget: an app that reads a password once at startup will keep using the old one until it is revoked, and then fail. Provide a shared library that does "watch file, rebuild client, drain old connections" so each team does not reimplement it.
   • Overlap: rotate with two valid credentials at once (Vault's database engine issues a new user while the old lease is still valid; static secrets should support "current" and "previous"), so a pod that has not yet re-read the file keeps working.
4. Node-level caching. Whatever caches on the node (the provider itself, or a node-local Vault Agent used as a caching proxy in front of Vault) must key entries by identity plus path, not by path alone, so two pods with different service accounts on the same node never share an entry. Pods of the same service on the same node can share, which is where the cache saves Vault traffic. A cache keyed only by secret path would let a pod read a secret another pod on the node fetched: a confused-deputy bug, where the agent's own broad access is used on behalf of a caller who should not have it.
5. Audit. Vault's audit device (a pluggable log sink Vault writes every request to, typically shipped into your normal log pipeline) logs every login and read with the requesting identity. Because each pod authenticates as its own service account, the log answers "which workload read this secret", not just "the node agent did".

Worked example: load and footprint

Assume 5,000 containers, each using 3 dynamic secrets with a 1-hour TTL, renewed at two-thirds of the TTL (40 minutes = 2,400 s).

• Vault renewal/issue rate: 5,000 × 3 / 2,400 s = 6.25 requests per second in steady state. That is modest; the danger is not the average but a thundering herd (a mass of clients all making the same request at the same instant, turning a trivial average load into a brief spike large enough to overwhelm the target) when a node pool restarts and hundreds of pods log in at once. Add jitter to renewal times and rate-limit logins per node.
• Footprint, using an assumed budget of 64 MiB per Vault Agent sidecar: 5,000 × 64 MiB = 320,000 MiB, about 312.5 GiB of cluster memory just for sidecars. With 200 nodes and an assumed 128 MiB per node agent: 200 × 128 MiB = 25,600 MiB, about 25 GiB. Measure the real figures on your workloads; the point is the scaling shape (per pod vs per node), which is why the node agent is the default here.
• Latency: reads come from a local file, so the hot path never touches the network. Vault is only on the refresh path.

Trade-offs and what would change the choice

• Node agent risk: a larger blast radius per node. A compromised node agent can request secrets for any pod scheduled on that node (it holds their tokens). Mitigate with node isolation for sensitive workloads (dedicated node pools for payments) and short token lifetimes.
• Alpha rotation in the CSI driver. If your platform policy forbids alpha features, use Vault Agent sidecars for workloads that must rotate, and the CSI driver only for secrets that change rarely.
• Vault availability becomes critical. Run it highly available, and make sure a Vault outage only blocks new secrets, not running pods: files already on the volume keep working until their credentials expire, so choose TTLs longer than your realistic Vault recovery time.
• Plain Kubernetes Secrets are acceptable for low-sensitivity, rarely-rotated values if encryption at rest and tight RBAC are configured. They are not acceptable as the only protection for database or payment credentials.
• When sidecars win: workloads needing complex templates (rendering one full config file that weaves together several secrets and static values, not just dropping each secret in its own file), or per-pod lease renewal for dynamic credentials that must be revoked the moment the pod dies.

Direct answer

Service discovery is how one service finds the current network addresses of another when instances come and go all the time (autoscaling, deploys, crashes). A service registry keeps the list of live instances. In client-side discovery, the calling service asks the registry for the list and picks an instance itself, so the client does the load balancing. In server-side (proxy) discovery, the caller sends the request to one stable address (a load balancer or proxy), and that component looks up the instances and picks one. My default is server-side through the platform (a Kubernetes Service or a load balancer) because it keeps clients simple and language-independent; I switch to client-side when the extra hop or the lack of per-request control actually costs something.

Why it is needed

In a static world you would hard-code payments = 10.0.1.5. In a microservice fleet, instances get new IPs on every deploy and the count changes with load, so addresses must be looked up at runtime, and the lookup must only return instances that are healthy (able to serve).

The two approaches

Client-side discovery

1. Each payments instance registers itself (or is registered by the platform) in the registry.
2. The checkout service queries the registry (or a local cache of it) for payments and gets a list, such as [10.0.1.5, 10.0.1.9, 10.0.2.4].
3. checkout's own library picks one (round robin: cycle through the list in order; or least outstanding requests: send to whichever instance currently has the fewest in-flight calls) and connects directly.

Server-side (proxy) discovery

1. Instances register the same way.
2. checkout calls one stable name, such as payments.default.svc.cluster.local or a load-balancer address.
3. The load balancer or proxy (which watches the registry) chooses an instance and forwards the request.

Who does what

On health checks: the registry usually learns health by probing instances (an HTTP call to /health every few seconds) or by requiring heartbeats. The client or proxy adds a second, faster signal: passive health checking, which stops sending to an instance that just returned several errors in a row, without waiting for the next probe.

Real-world tools

• DNS: a service name resolves to several A records (IP addresses) and the client picks one. That is a simple form of client-side discovery, but with weak health handling and caching delays (clients keep an answer until its TTL, time-to-live, expires, and some ignore TTLs).
• Consul: supports both. Clients can query its HTTP API or its DNS interface (payments.service.consul, returning only healthy instances) and pick an instance themselves: client-side. Or traffic goes through Consul's service mesh, where a sidecar proxy (a helper process running next to each service) does the choosing: server-side, but on the same host.
• Kubernetes Service: server-side. A normal Service gets one stable virtual IP (the ClusterIP); kube-proxy (the Kubernetes component that runs on every node and keeps its routing rules in sync with the cluster) programs rules, either iptables (the Linux kernel's built-in packet-filtering and NAT table) or IPVS (the Linux kernel's built-in load balancer), that spread connections across the pods (a pod is Kubernetes' smallest deployable unit, one or more containers scheduled together) that are passing their readiness probe (a periodic check, such as an HTTP call, that decides whether a pod is currently able to serve traffic; failing it removes the pod from the Service without killing it). A headless Service (no ClusterIP) instead returns the pod IPs through DNS, which is client-side, and is what gRPC (a high-performance RPC framework built on HTTP/2, which multiplexes many concurrent requests over a single long-lived TCP connection instead of opening a new one per call) clients often use to balance per request.
• Cloud load balancers (for example, an AWS Application Load Balancer with a target group): server-side, with the cloud provider registering instances.
• Client-side libraries: gRPC's built-in load-balancing policies, Spring Cloud LoadBalancer (which replaced the Netflix Ribbon library that is no longer developed).

Worked example

checkout runs 200 instances and calls payments, which runs 50.

• Client-side: every checkout instance may keep connections to every payments instance: 200 × 50 = 10,000 connections. When payments instance 17 starts timing out, each checkout instance notices within its own next few requests and stops using it, without waiting for the registry. But the balancing code has to exist, and be kept correct, in every language checkout-like services are written in.
• Server-side with a Kubernetes Service: checkout knows one address. Connections are spread by kube-proxy, which balances per connection, not per request: it picks a pod once, when a TCP connection is first opened, and every request sent over that connection afterward goes to that same pod. Since gRPC keeps one HTTP/2 connection open and multiplexes many requests over it (see above), a checkout instance that opens one long-lived HTTP/2 connection sends all of its requests, forever, to whichever single pod answered first, which is a classic cause of uneven load for gRPC. Fixes: a headless Service plus client-side balancing, or a proxy that balances per request.

When to choose each

• Choose server-side when you have many languages, want one place to change routing, or run on a platform that already provides it (Kubernetes, cloud load balancers). This is the right default for most teams.
• Choose client-side when the extra hop matters (very latency-sensitive internal calls), when you need per-request balancing of long-lived connections (gRPC), or when you want each caller to react to failures instantly.
• A service mesh blends them: the logic lives in a proxy (server-side style, one implementation for all languages), but the proxy sits next to each client (client-side style, no central hop). It costs you operating the mesh.

Pitfalls

• Registering an instance as soon as the process starts, before it can serve (use readiness, not liveness, to decide when it receives traffic: a liveness probe only answers "is this process stuck and needs restarting", it says nothing about whether the process is ready to handle requests right now).
• Clients that cache the instance list with no refresh, so they keep calling instances that were removed.
• Treating the registry as the only health signal: a registry probe every 10 s means up to 10 s of requests to a dead instance unless the caller also reacts to errors.

Direct answer

DNS-based service discovery means clients find a service by resolving a name such as orders.internal.example.com to the IP addresses of its healthy instances. Because every resolver and client caches the answer for the record's TTL (time-to-live), the TTL sets how long clients may keep sending traffic to an address you have already removed: it is a floor on failover time and a limit on how quickly load shifts. For a service record that must fail over, I would pick 60 seconds. The extra query load is small and a 300-second TTL means up to five minutes of errors after an instance or region dies. But a 60-second TTL only holds if clients actually honour it, so the rest of the job is making sure they do.

How DNS-based discovery works

1. Instances register (directly, or via a health-checking DNS provider such as Consul's DNS interface, Kubernetes' CoreDNS, or a cloud DNS service with health checks).
2. A client asks its local stub resolver (the small resolver library in the OS or language runtime), which asks a recursive resolver (the caching server, e.g. the node-local cache or the VPC (Virtual Private Cloud) resolver), which asks the authoritative server (the one that actually owns the zone, meaning the portion of the DNS namespace, such as internal.example.com, that this server holds the real records for).
3. The answer, say three A records (address records, each mapping the name to one IPv4 address), is cached at every layer for the TTL. The client picks one address, usually the first, and connects.

Nothing tells a client that an address went bad: it only learns on its next lookup after the cached copy expires.

How caching and TTLs affect failover

Worst-case time for all clients to stop using a dead address is roughly:

failover time ≈ health-check detection time + TTL (+ any extra caching the client adds)

With a health check that marks an instance down after 3 failures at 10-second intervals (30 seconds):

The 60-second TTL costs about 26 more queries per second, which any resolver absorbs trivially, and cuts the worst-case failover from 5.5 minutes to 1.5 minutes. Query volume scales linearly with 1/TTL, so going lower (5 seconds) is 400 queries per second for the same fleet and starts to matter for resolvers and for paid public DNS.

How caching affects load distribution

DNS balances resolvers, not requests. If one large recursive resolver (a shared NAT, meaning a network-address-translation gateway that many separate devices sit behind and also route their DNS queries through, or a corporate DNS server) serves 500 clients, all 500 may receive the same cached answer in the same order and pile onto the first IP. Longer TTLs make this worse, because a skewed answer persists longer; round-robin record ordering, returning a random subset per response, or client-side random selection among all returned addresses spreads it out. Adding capacity has the same lag as removing it: new instances receive traffic only as caches expire.

What else must be true for the 60-second choice to hold

1. Clients must honour the TTL. Language runtimes add their own caches. The JVM's positive cache (a positive cache remembers successful lookups; the negative cache below remembers failures) is governed by the networkaddress.cache.ttl security property, whose default is implementation-specific; set it explicitly (for example to 60) instead of trusting it. Some resolvers also enforce a minimum TTL that silently raises yours.
2. Clients must re-resolve at all. A connection pool or an HTTP/2 keep-alive connection (reusing one already-open connection for many requests instead of reconnecting for each one) opened to an IP will keep using that IP for hours, regardless of DNS TTL. Cap connection lifetime (e.g. recycle connections every few minutes) and re-resolve on reconnect, or failover never happens for long-lived clients.
3. DNS must only return healthy addresses. The TTL is pointless if the authoritative server keeps handing out the dead IP; the records must be driven by health checks.
4. Negative caching. If a lookup returns "no such name" (for example during a brief deregister-everything incident), resolvers cache that negative answer for a time taken from the zone's SOA (start of authority) record, per RFC 2308, and Java's networkaddress.cache.negative.ttl defaults to 10 seconds. A long negative TTL turns a 5-second blip into minutes of "host not found". Keep it short for service zones.
5. Clients must retry on another address. Even with perfect TTLs, requests fail during the detection window. Returning several records and having clients try the next address on connection failure turns an outage into a latency bump.
6. Lower TTLs before planned changes. A TTL only shortens after the old one expires, so for a planned migration drop the TTL a full old-TTL period ahead of time.

When I would pick 300 seconds instead

For records that rarely change and front something that fails over by other means (a load balancer's stable virtual IP, an address the load balancer itself keeps fixed while swapping which physical instance answers behind it, or an anycast address, the same IP announced from multiple physical locations so network routing sends each client to the nearest live one and simply stops routing to a failed location, no DNS change needed), 300 seconds or more is fine: the address itself never dies, so failover does not depend on DNS, and a longer TTL cuts query volume and shields you from brief DNS outages.

Trade-offs and pitfalls

• A very low TTL (under 10 seconds) makes DNS itself a hard dependency: if the resolver hiccups, every client fails its lookups at once.
• DNS offers no per-request load awareness and no health-based retries; for east-west traffic (traffic between your own services inside the system, as opposed to north-south traffic coming in from outside clients) between microservices that need fast failover, client-side load balancing (the calling service itself holds the instance list and picks one per request, no DNS or proxy hop involved) or a service mesh (a proxy next to each service that receives endpoint updates by streaming) reacts in seconds instead of a TTL period.
• Assuming "TTL 60 means failover in 60 seconds" ignores detection time, runtime caches and pooled connections, which is where most real incidents come from.

Direct answer

I would not run this as a tool bake-off. I would first write down what the company needs discovery to do (the requirements every team can agree on), score the options against those with weights the teams help set, and then commit to a layered standard: one authoritative service registry as the source of truth, DNS as the universal read interface so every language and legacy system works on day one, and a service mesh as an opt-in layer for teams that need mutual TLS and traffic shaping. Migration runs in reversible phases (dual-register, shadow, cut over by tier, decommission) led by a pilot with the most sceptical team, with success measured and published before anything is mandated.

Terms first

• Service discovery: how a caller finds the current network address of a service it wants to talk to.
• Service registry: the database of which instances of each service are alive and where (Consul is a common example; Kubernetes keeps its own).
• Service mesh: a proxy (a sidecar) runs next to every instance and handles discovery, retries, encryption and routing, programmed by a central control plane. Istio and Linkerd are examples.
• mTLS (mutual TLS): both sides of a connection prove their identity with certificates, not just the server.
• Vendor lock-in: when leaving a tool later would cost more than it is worth, usually because application code or data formats depend on it.

1. Evaluate: requirements first, then options

Run short interviews with each camp and turn their preferences into requirements. "We like DNS" usually means "it works with every client and needs no library"; "we want the mesh" usually means "we need mTLS and canary routing". Requirements, not tools, get weights.

The operational side matters as much as the technical: who is on call for it, how upgrades happen, what happens when it is down (does traffic keep flowing on cached data, or does everything stop?), and whether the team that built the custom mesh is still staffed to maintain it. A custom mesh scores well on features and badly on bus factor (how many key people could leave or be unavailable before nobody left understands the system).

Turning that into a decision: score each cell 0 (worst) to 10 (best), consistent with the qualitative read above, multiply by its criterion's weight, and sum:

Consul wins clearly, 7.65 against 5.85 and 5.80, which is why it becomes the source of truth below rather than a coin flip between three close options. DNS and the mesh land near each other for opposite reasons: DNS wins on universal compatibility and zero lock-in but loses hard on security and failover; the mesh wins on security and failover but loses hard on operational burden and lock-in. That is exactly why the recommendation layers them instead of picking one outright: Consul underneath for its lead on health-awareness and multi-datacenter support, DNS names as the interface every client already speaks, and the mesh opt-in only where a team's own security or failover need would flip its individual score.

Recommendation for a typical mixed estate: Consul (or the platform's native registry) as source of truth, DNS names as the contract applications code against, mesh opt-in. What would flip it: if more than roughly 80% of workloads already run on Kubernetes and a regulator requires mTLS everywhere, standardise on a supported mesh as the default instead, and treat DNS as the legacy bridge.

2. Migration plan

(Two terms used below: tier-1 means the most critical, highest-traffic services, where an outage matters most; the long tail means the large number of smaller, lower-priority services that are slow to migrate voluntarily.)

1. Contract first. Publish the naming scheme (<service>.service.<dc>.<domain>), health-check requirements, and the rule that application code references names, never tool-specific APIs. This is the lock-in defence: if apps only know DNS names, the registry behind them can be swapped.
2. Dual registration. Services register in both old and new systems, automatically from the deploy pipeline, so nobody hand-edits two places. Nothing reads the new system yet.
3. Shadow reads and diffing. A job resolves every service through both systems and reports mismatches. Cutover of a service is blocked until its diff has been clean for a week.
4. Cut over by tier. Start with internal tools, then non-critical services, then tier-1. Each cutover is a config flag per caller, so rollback is a flag flip, not a redeploy.
5. Decommission the old path per service once its traffic on the old path is zero for 30 days, with a published date.

Worked example: sizing the timeline

Say 300 services owned by 40 teams. The pilot (weeks 1 to 6) moves 10 services with the platform team doing the work. After that, a paved-road script means a team can move one service in about half a day, and the platform team can support about 15 cutovers per week without degrading review quality.

• Remaining after pilot: 300 - 10 = 290 services.
• At 15 per week: 290 / 15 = 19.3, so 20 weeks of rollout.
• Total: 6 + 20 = 26 weeks, roughly two quarters, plus the 30-day decommission tail.

State it as a plan with that arithmetic, so leadership can see what adding or removing platform capacity does to the date.

3. Addressing the specific concerns

• Operational burden: the platform team owns the registry cluster, its upgrades and on-call, with a published SLO. Application teams own only their health checks. Quantify it: a 5-server consensus cluster per datacenter, which tolerates the loss of 2 servers (a write only needs a majority, 3 of the 5, to agree before it commits; losing any 2 still leaves 3, a majority, so the cluster keeps accepting writes, while losing 3 would leave only 2, which can no longer form one).
• Vendor lock-in: the primary defence is contractual, not technical: apps depend on DNS names and standard health endpoints, not on a client SDK, so the registry behind those names can be swapped without touching application code. As a secondary, deeper defence inside the mesh itself, it also uses open standards (Envoy's xDS configuration APIs, a standard protocol for pushing routing config to sidecar proxies, so the proxies do not have to change even if the control plane does; and SPIFFE identities, a vendor-neutral standard for verifiable service identity, so certificates are not tied to one mesh vendor) so even the mesh's control plane is replaceable. Write down the exit plan now, with its cost.
• "Our custom mesh works fine": respect the investment. Offer to make it a candidate for the opt-in mesh layer if it passes the same criteria, and ask its owners to lead the mesh working group. Most resistance is about losing ownership, not about technology.

4. Gaining buy-in

• Publish the decision as an RFC (request for comments) with the weighted scoring, and let teams challenge the weights before the scores. Changing a weight is a cheap, visible concession.
• Pilot with the loudest sceptic. If their pain is solved, that is the most credible endorsement you can get.
• Make the paved road (the supported, recommended way of doing something, made deliberately the easiest path) cheaper than the old road: generated configs, dashboards and alerts for free on the new platform; the old one gets security fixes only.
• Measure and publish: services migrated, lookup failure rate, time to failover, incidents attributable to discovery, before and after.
• Escalate last. Executive mandate is for the long tail after the default is clearly better, not for the opening move.

Pitfalls

• Choosing by committee vote instead of weighted requirements: the loudest team wins, and the others disengage.
• Big-bang cutover: no rollback, and one bad week kills trust in the platform for years.
• Letting apps import the registry's client library "just for one feature": that is how lock-in re-enters.
• Forgetting non-Kubernetes workloads (VMs, databases, batch jobs): DNS is what keeps them in the plan.