Service Discovery and Configuration Management Questions

Direct answer

etcd, ZooKeeper and Consul's server cluster are CP systems: they use a consensus protocol (an algorithm, such as Raft or ZAB, that gets every replica to agree on one ordered history of writes, normally by electing a single leader that orders and replicates each write before it counts as committed, which is why "no leader" means "no writes" in these systems), so during a network partition the side without a majority stops accepting writes rather than risk disagreeing. Eureka is AP: every server accepts registrations and keeps answering during a partition, even if the answer is stale. Operationally, CP means "correct or unavailable" and AP means "always answers, possibly wrong". My default is a CP store for configuration, locks and leader election, and discovery clients that tolerate staleness (cache the last good answer, retry on a different instance), because for routing a slightly old list beats no list.

CP and AP in plain terms

The CAP theorem says that when a network partition splits a cluster (some nodes cannot talk to others), a replicated system must choose between consistency (every read sees the latest write, or fails) and availability (every request gets an answer). CP systems choose consistency, AP systems choose availability. When there is no partition, both behave well; the choice only shows during failures, which is exactly when you are debugging at 3 a.m.

Quorum is the majority needed to make a decision: 3 of 5 nodes, 2 of 3. A 5-node CP cluster survives 2 node failures; a partition that splits it 3 and 2 leaves the 3-node side working and the 2-node side unable to write.

The four systems side by side

The table below is dense; for the CP-versus-AP question itself, the first three rows (Replication, CAP stance, Reads) carry the answer. The remaining rows (health/liveness through operational load) are what you'd use to justify picking one system over another, not to argue CP versus AP.

What CP vs AP means during a partition

ZooKeeper during leader election. When the leader dies or is cut off, the ensemble cannot accept writes until a new leader is elected by a quorum. Clients connected to servers on the minority side get disconnected and, if the session times out, their ephemeral nodes (the entries that represent "this instance is alive") are deleted. A discovery design built on ephemeral nodes can therefore remove healthy instances because of a partition between the instances and ZooKeeper, not because the instances died.

Eureka self-preservation. Eureka expects heartbeats at a steady rate. If renewals fall below a threshold (by default, when more than 15% of registered instances are overdue), the server assumes the problem is the network, not mass instance death, and stops evicting anyone. The registry then serves dead instances until renewals recover. That is the AP trade: during a real mass failure, clients keep receiving addresses that no longer answer.

etcd. The minority side refuses writes and, with default linearizable reads, refuses reads too. A Kubernetes control plane (the components that make scheduling and management decisions, such as the API server and scheduler) on the minority side cannot schedule anything, but the workloads already running keep running, because the data plane (the running workloads themselves and the networking that serves their live traffic) does not depend on etcd for every request.

Consul. Writes to the catalog need the Raft leader. Clients that use stale reads (and Consul's DNS interface allows stale reads by default) keep getting answers from any server, even with no leader, so discovery continues while registration of new instances pauses.

Worked example: when does Eureka stop evicting?

100 instances, heartbeat every 30 s, so the server expects

and with the default 85% threshold it enters self-preservation below

(Eureka's exact computation of the expected count has more detail, but this is the shape.) If a switch fault cuts off 20 instances, renewals drop to 80 × 2 = 160, below 170, so nothing is evicted. If those 20 instances were genuinely dead, a client choosing uniformly at random hits a dead one 20% of the time until they return or an operator acts. The client-side mitigation is a short connect timeout plus a retry on a different instance: with independent random picks, two failures in a row happen 0.2 × 0.2 = 4% of the time. The same fault on ZooKeeper would have taken the other path: those 20 instances' sessions expire and they vanish from the registry, which is correct if they are dead and a self-inflicted outage if only their link to ZooKeeper failed.

Failure modes to name in an interview

• etcd: slow disk (fsync latency) causes leader elections; storage quota reached puts the cluster into alarm mode where it rejects writes until you compact (discard old key revisions no longer needed) and defragment; large watch fan-out on one prefix loads the leader.
• ZooKeeper: long JVM garbage-collection pauses look like dead servers or expired sessions; a "herd effect" when many clients watch one node and all wake up at once.
• Consul: gossip misconfiguration (encryption keys, ports blocked) causing agents to flap between alive and failed; a lost quorum of servers blocks all registrations; a very large catalog makes blocking queries on the whole service list expensive.
• Eureka: stale entries during self-preservation; up to roughly 90 seconds, three heartbeat intervals at the default 30-second heartbeat, for a change to reach every client (Spring Cloud Netflix's Eureka documentation: "a service is not available for discovery by clients until the instance, the server, and the client all have the same metadata in their local cache (so it could take 3 heartbeats)"), and longer still when an intermediate server's own response-cache refresh cycle adds another hop on top.

Recommendation and what would flip it

• Running on Kubernetes: use the platform's own discovery (Services backed by etcd) and do not add another registry unless you span clusters.
• Many datacenters, VMs and containers mixed, want DNS-based discovery and health checks: Consul.
• New design choosing ZooKeeper: rarely justified now; the main reason is an existing system that depends on it. Kafka, its best-known user, removed ZooKeeper mode in Kafka 4.0 in favour of its own Raft-based controller.
• Eureka: fine inside an existing Spring Cloud estate where clients already retry and tolerate staleness; I would not introduce it into a new, non-JVM system.

Pitfalls

• Putting service discovery on the critical path of every request instead of caching locally, so a registry outage becomes a full outage.
• Treating a ZooKeeper one-time watch as a change stream and silently missing updates.
• Using an even number of consensus nodes: 4 nodes need 3 for quorum and still tolerate only 1 failure, the same as 3 nodes but with more to go wrong.
• Assuming "CP" means "always correct for clients": ZooKeeper reads can still be stale without sync(), and Consul's default mode has a brief stale window during leader changes.

Direct answer

Treat it as a reconciliation loop (the same shape a Kubernetes controller uses): on every tick, first observe the health of the whole fleet, then compute a budget of N minus everything currently unhealthy, and only start that many new config applies. Instances that are restarting, waiting to retry, or were already broken before you started all count against the budget, which is what makes "at most N unhealthy" hold rather than "at most N of my operations in flight". Failed applies retry with capped exponential backoff and jitter, keeping their budget slot while they wait; the rollout aborts when too many instances fail for good or when nothing has made progress for too long, and it reports progress every tick.

Approach

Two ideas carry the whole design:

1. The invariant is about the fleet, not about the orchestrator. A naive version uses a semaphore (a counter that lets at most N callers proceed at once and blocks the rest until one finishes) of size N around apply_config. That bounds your in-flight operations, but if two instances were already down when you started, you now have N + 2 unhealthy. So the budget must be computed from observed health: budget = N - unhealthy_now.
2. Observe, then act, once per tick. Rather than threads racing each other, one loop per tick (a) reads health for every instance, (b) advances each in-flight instance's state machine, (c) checks abort criteria, (d) admits new work up to the budget, (e) reports. This makes the logic deterministic and testable, and it is the same pattern whether a tick is one second in production or one step in a simulation.

Each instance moves through a small state machine:

stateDiagram-v2
  [*] --> pending
  pending --> applying: budget available
  pending --> skipped: unhealthy before rollout
  applying --> done: health_check passes
  applying --> backoff: ready_timeout expired, attempts left
  backoff --> applying: backoff elapsed, re-apply
  applying --> failed: ready_timeout expired, no attempts left

Here apply_config(instance) is taken to start the restart and return; the loop then polls health_check(instance) until it passes or ready_timeout expires. If your apply_config blocks until the restart finishes, run it in a worker thread per admitted instance and let the loop observe it the same way.

Code

The orchestrator is rolling_apply; everything under the "simulated fleet" line is a deterministic test harness (seeded random restart times of 2 to 4 ticks, instances that never recover on bad config, and instances that were sick before the rollout). The harness independently records the worst number of instances down at once, so the invariant is checked by something other than the code that enforces it.

import random
from dataclasses import dataclass

@dataclass
class Policy:
    max_unhealthy: int = 3      # N: never more than this many instances unhealthy at once
    ready_timeout: int = 6      # ticks to wait for health after an apply
    max_attempts: int = 3       # apply attempts per instance before it counts as failed
    backoff_base: int = 1       # retry delay = min(cap, base * 2**(attempt-1)) + jitter
    backoff_cap: int = 8
    max_failed: int = 2         # abort when this many instances have failed for good
    max_stall_ticks: int = 20   # abort if nothing finishes for this long

@dataclass
class Task:
    name: str
    state: str = "pending"      # pending | applying | backoff | done | failed | skipped
    attempt: int = 0
    deadline: int = 0
    wake_at: int = 0

FINAL = ("done", "failed", "skipped")

def rolling_apply(instances, health_check, apply_config, policy, rng, clock, report=print):
    """Reconciliation loop: every tick, observe health first, then act on the observation."""
    tasks = {i: Task(i) for i in instances}
    for i in instances:                         # sick before we touched it: do not restart it,
        if not health_check(i):                 # but it still uses up part of the N budget
            tasks[i].state = "skipped"
            report(f"t={clock.now:>3} {i} skipped: unhealthy before rollout")
    last_progress = clock.now
    while True:
        healthy = {i: health_check(i) for i in instances}
        unhealthy = sum(not h for h in healthy.values())

        # 1. advance in-flight work
        for t in tasks.values():
            if t.state == "applying" and healthy[t.name]:
                t.state, last_progress = "done", clock.now
            elif t.state == "applying" and clock.now >= t.deadline:
                if t.attempt >= policy.max_attempts:
                    t.state, last_progress = "failed", clock.now
                    report(f"t={clock.now:>3} {t.name} FAILED after {t.attempt} attempts")
                else:
                    delay = min(policy.backoff_cap, policy.backoff_base * 2 ** (t.attempt - 1)) + rng.randint(0, 1)
                    t.state, t.wake_at = "backoff", clock.now + delay
                    report(f"t={clock.now:>3} {t.name} not ready, retry in {delay}")
            elif t.state == "backoff" and clock.now >= t.wake_at:
                t.attempt += 1                  # still unhealthy while backing off: keeps its slot
                t.state, t.deadline = "applying", clock.now + policy.ready_timeout
                apply_config(t.name)

        # 2. abort criteria
        n_failed = sum(t.state == "failed" for t in tasks.values())
        if n_failed >= policy.max_failed:
            return "ABORTED: too many failed instances", _stats(tasks, clock)
        if clock.now - last_progress > policy.max_stall_ticks:
            return "ABORTED: no progress", _stats(tasks, clock)
        if all(t.state in FINAL for t in tasks.values()):
            return ("COMPLETE" if n_failed == 0 else "COMPLETE WITH FAILURES"), _stats(tasks, clock)

        # 3. admit new work: budget counts EVERY unhealthy instance, ours or not
        budget = policy.max_unhealthy - unhealthy
        for name, t in tasks.items():
            if budget <= 0:
                break
            if t.state == "pending":
                t.attempt, t.state, t.deadline = 1, "applying", clock.now + policy.ready_timeout
                apply_config(name)              # a restart takes it out of rotation now
                budget -= 1
        done = sum(t.state == "done" for t in tasks.values())
        report(f"t={clock.now:>3} progress {done}/{len(tasks)} done, {unhealthy} unhealthy, budget left {max(budget, 0)}")
        clock.now += 1

def _stats(tasks, clock):
    c = {"ticks": clock.now}
    for t in tasks.values():
        c[t.state] = c.get(t.state, 0) + 1
    return c

# ---------------- simulated fleet (deterministic, seeded) ----------------
class Clock:
    now = 0

class Fleet:
    """Fake instances. Independently records the worst number of instances down at once."""
    def __init__(self, n, bad, sick, rng, clock):
        self.clock, self.rng, self.bad = clock, rng, set(bad)
        self.up_at = {f"i{k:02d}": 0 for k in range(n)}
        for s in sick:
            self.up_at[s] = float("inf")
        self.worst = 0
    def health_check(self, i):
        return self.clock.now >= self.up_at[i]
    def apply_config(self, i):                  # restart: down for 2-4 ticks; bad config never recovers
        self.up_at[i] = float("inf") if i in self.bad else self.clock.now + self.rng.randint(2, 4)
        down = sum(self.clock.now < t for t in self.up_at.values())
        self.worst = max(self.worst, down)

def run(label, bad, sick, policy=Policy()):
    clock, rng = Clock(), random.Random(7)
    fleet = Fleet(12, bad, sick, rng, clock)
    events = []
    outcome, stats = rolling_apply(list(fleet.up_at), fleet.health_check, fleet.apply_config,
                                   policy, rng, clock, report=events.append)
    print(f"--- {label}: {outcome}  (worst down at once: {fleet.worst}, N = {policy.max_unhealthy})")
    print("   ", stats)
    for e in events:
        if "progress" not in e:
            print("   ", e)
    return events

ev = run("clean fleet", bad=[], sick=[])
print("    first progress lines:", *ev[:3], sep="\n    ")
run("one box sick before start, one bad box", bad=["i05"], sick=["i09"])
run("two bad boxes", bad=["i02", "i03"], sick=[])

Running it (python3 rollout.py) prints:

--- clean fleet: COMPLETE  (worst down at once: 3, N = 3)
    {'ticks': 13, 'done': 12}
    first progress lines:
    t=  0 progress 0/12 done, 0 unhealthy, budget left 0
    t=  1 progress 0/12 done, 3 unhealthy, budget left 0
    t=  2 progress 1/12 done, 2 unhealthy, budget left 0
--- one box sick before start, one bad box: COMPLETE WITH FAILURES  (worst down at once: 3, N = 3)
    {'ticks': 28, 'done': 10, 'failed': 1, 'skipped': 1}
    t=  0 i09 skipped: unhealthy before rollout
    t= 13 i05 not ready, retry in 1
    t= 20 i05 not ready, retry in 2
    t= 28 i05 FAILED after 3 attempts
--- two bad boxes: ABORTED: too many failed instances  (worst down at once: 3, N = 3)
    {'ticks': 23, 'done': 9, 'failed': 2, 'applying': 1}
    t=  6 i02 not ready, retry in 1
    t=  8 i03 not ready, retry in 1
    t= 13 i02 not ready, retry in 2
    t= 15 i03 not ready, retry in 2
    t= 21 i02 FAILED after 3 attempts
    t= 23 i03 FAILED after 3 attempts

What the three runs show:

• Clean fleet: 12 instances finish in 13 ticks and the harness never saw more than 3 down at once. At t=0 the loop observed 0 unhealthy and admitted 3 (budget left 0); at t=2 one finished and one more was admitted.
• Pre-sick box plus bad box: i09 was unhealthy before the rollout, so it is skipped (restarting a box you do not understand makes debugging harder) but it still occupies one of the 3 slots. i05's bad config never becomes healthy; it retries with backoff 1, then 2, then fails. While i05 is failing, only one slot is free, so the rollout slows to one instance at a time instead of breaking the invariant. It completes with 1 failure, below the abort threshold of 2.
• Two bad boxes: the second permanent failure trips max_failed = 2 at t=23, and the loop stops admitting work and returns ABORTED. By that point every one of the 12 instances has already been touched, matching the printed {'done': 9, 'failed': 2, 'applying': 1}: nine finished normally, i02 and i03 failed for good, and one instance is still mid-apply (state applying) when the abort fires, left exactly where its restart was, neither confirmed healthy nor failed. None are left untouched.

Key points

• Scheduling strategy: instances are admitted in list order, so the caller controls the wave shape: put a canary (a single instance updated first and watched before the rest follow, so a bad config is caught on one box instead of the whole fleet) first, then order by availability zone (AZ: an independent data center or facility with its own power and network, so one AZ at a time limits the blast radius, meaning how much of the fleet one bad change can take down). A common refinement is to admit only 1 until the canary is healthy, then open the budget to N.
• Concurrency control: the budget is recomputed every tick from observed health, so it self-corrects: a box that crashes on its own mid-rollout automatically shrinks the admission budget.
• Backoff and retry: delay = min(cap, base × 2^(attempt-1)) plus a random 0 or 1 tick of jitter (random spread so many retrying instances do not re-hit a config server in lockstep). Crucially, an instance in backoff is still unhealthy and keeps its slot; releasing the slot while it waits is the classic way to break the invariant.
• Abort criteria: (1) max_failed permanent failures, (2) no instance finished or failed for max_stall_ticks, which catches a fleet that is stuck because unhealthy instances consumed the whole budget. In production add (3) a fleet-level signal such as the service's error rate burning its SLO (service-level objective) while the rollout is running.
• Surfacing progress: the report callback emits one line per tick with done count, unhealthy count and remaining budget, plus an event for every retry and failure. Wire it to a log and a status endpoint; the final return value is the machine-readable summary.
• What abort does not do: it stops the rollout, it does not undo it. The honest next step is a second rolling pass of the previous config through the same function over the instances marked done and failed, under the same N budget, so the rollback cannot itself breach the invariant.

Complexity

Each tick is O(I) work for I instances (one health read and one state update each), so a rollout of T ticks costs O(I × T) health checks. Memory is O(I) for the task table. In the clean run (I = 12, ticks 0 through 13 inclusive) that is 12 × 14 = 168 reads in the loop plus 12 for the initial "already sick?" scan, 180 in total (counted by wrapping health_check with a counter). Real systems cap health-check cost by caching the fleet health snapshot from the registry or load balancer instead of probing every instance every tick.

Edge cases

• N = 0 or N below the number already sick: the budget is never positive, nothing is admitted, and the stall rule aborts. Validate N >= 1 up front and refuse to start if the fleet is already at or beyond N unhealthy.
• Health flaps between observation and admission: an instance that becomes unhealthy after this tick's snapshot makes the next tick's count one higher; the invariant can be exceeded for at most one tick. Shorten the tick or re-read health immediately before each apply_config if that window matters.
• health_check itself fails or times out: treat "unknown" as unhealthy. Counting it as healthy would let the loop admit work into a fleet it cannot see.
• A healthy-looking bad config: health checks pass but business errors rise. Only a fleet-level SLO abort criterion catches that, which is why it belongs in production abort rules.
• Two orchestrators on the same fleet: both compute budget from the same observed health and each admits N. Take a lease-based lock (a lock that expires if the holder dies) on the service before starting.

Trade-offs and pitfalls

• Semaphore-only designs bound operations, not unhealthy instances; they pass unit tests and fail on the first day a box is already down.
• Small N is safe but slow: at N = 3 with restarts taking 2 to 4 ticks, 12 instances took 13 ticks; 1,200 instances would take roughly 100 times as long. Set N as a percentage of fleet size, and check it against spare capacity by dividing, not subtracting: if the fleet runs at 70% average utilization and N takes 10% of it offline at once, the remaining 90% of instances must carry that same load, so their utilization becomes 70% / 90% ≈ 77.8%, still under 100% and safe. Taking 40% offline at once leaves only 60% of instances carrying that load: 70% / 60% ≈ 116.7%, over capacity, and requests queue or fail. In general, keep N (as a fraction of the fleet) below 1 - utilization, here below 30%, with margin left over for the retries and stragglers the rollout itself creates.
• Retrying a deterministic failure only wastes time: a config that fails to parse will fail three times. Classify failures (retryable: timeout, dependency not ready; non-retryable: validation error) and fail fast on the second kind.

Direct answer

For about 200 services and 1,000 instances, I would choose Consul: it already is a service registry (catalog, health checks, DNS interface, watch-style queries), whereas etcd and ZooKeeper (distributed key-value stores built for coordination tasks like leader election and locking, not purpose-built service registries) are general coordination stores on which you would have to build registration, health checking and a lookup API yourself. The design is: 3 or 5 Consul servers forming a Raft cluster that holds the catalog; a Consul client agent on every host that registers local instances and runs their health checks locally; and clients discovering endpoints either through DNS (zero code change) or through blocking queries in a shared client library that caches healthy endpoints and load-balances client-side.

Terms

• Service registry: a directory mapping a service name (orders) to its live instances (10.1.4.7:8080, ...).
• Client-side vs server-side discovery: in client-side discovery the caller looks up instances and picks one itself; in server-side discovery the caller hits a load balancer or proxy that does the lookup.
• Raft and quorum: the servers replicate every write using the Raft consensus protocol; a write commits once a majority (floor(n/2) + 1) has it. 3 servers survive 1 failure; 5 survive 2.
• Gossip: Consul agents also exchange membership information peer to peer, using a protocol based on SWIM. In plain terms: each agent periodically picks a random peer and pings it directly; if there is no reply, it asks a few other random peers to ping that same target on its behalf, in case the problem is just the direct path between prober and target; if those also fail, the agent marks the target suspect and mentions that suspicion in its own next round of pings to other random peers, who do the same, so the news spreads through the cluster the way a rumor spreads through a crowd (hence "infection-style") without any node pinging a central server. This detects dead nodes without every node pinging the servers.
• Blocking query: an HTTP request that includes the last index the client saw; the server holds it open until the data changes (or a timeout passes), then answers. It gives near-instant change notification with ordinary HTTP.

Components

flowchart LR
  subgraph Host[Each host]
    Svc[Service instance] -->|register on start| Agent[Consul client agent]
    Agent -->|run health check locally| Svc
  end
  Agent -->|state changes, anti-entropy sync| Servers[3 or 5 Consul servers, Raft]
  Caller[Calling service with client library] -->|blocking query: healthy instances of orders| Servers
  Caller -->|or DNS: orders.service.consul| Agent
  Caller -->|direct connection| Svc

1. Servers (3, spread across availability zones): hold the catalog and key-value store, replicate with Raft. At 1,000 instances, 3 servers are enough; move to 5 when you want to survive a zone loss plus one more failure, or a failure during maintenance.
2. Client agents (one per host): accept registrations from local instances, run their health checks, and sync to the servers. That sync is called anti-entropy: the agent periodically reconciles its local view of what should be registered against the servers' catalog, so a missed update from a blip or a restart is corrected on the next pass instead of drifting forever.
3. Client library / DNS: the lookup path for callers.
4. Deploy integration: the deploy system or the instance's startup script registers, and graceful shutdown deregisters before the process stops.

APIs

Registration (instance to local agent):

PUT /v1/agent/service/register
{ "Name": "orders", "ID": "orders-7f3a", "Address": "10.1.4.7", "Port": 8080,
  "Tags": ["v2", "zone-a"], "Meta": {"version": "2.14.0"},
  "Check": { "HTTP": "http://10.1.4.7:8080/healthz", "Interval": "10s", "Timeout": "2s",
             "DeregisterCriticalServiceAfter": "5m" } }
PUT /v1/agent/service/deregister/orders-7f3a      (on graceful shutdown)

Reading the Check block: Consul calls the HTTP URL every Interval, allows Timeout for a reply, and if it stays failing for DeregisterCriticalServiceAfter, removes the registration entirely rather than just marking it unhealthy forever.

Discovery (caller):

GET /v1/health/service/orders?passing=true&index=<last_index>&wait=60s

dig @127.0.0.1 -p 8600 orders.service.consul SRV

The first returns only instances whose checks pass, and blocks until that list changes. The second uses Consul's DNS interface (port 8600 by default), which returns healthy instances and works for any language that can resolve a hostname.

Health-check model: active pings vs TTL vs passive

Choice: active HTTP checks as the default, TTL checks for workers without a port, and passive outlier ejection in the client library as a second layer, because it catches the gap between "health check passes" and "real requests fail". Checks run on the local agent, so they do not travel across the network, and Consul syncs status changes to the servers rather than every check result, which keeps server load independent of the check interval.

DeregisterCriticalServiceAfter removes instances that have been failing for 5 minutes, so crashed instances that never deregistered do not accumulate.

How clients discover endpoints

• Default: DNS. Any service can call orders.service.consul. Keep the DNS TTL short (a few seconds) so changes propagate, and remember that DNS gives addresses but not ports unless the client asks for SRV records (DNS service records, which carry a port and a hostname).
• Preferred for services owned by the platform: client library using blocking queries, holding the healthy list in memory, balancing across it (round robin, or least-requests: send the next call to whichever healthy instance currently has the fewest outstanding calls), with passive ejection and a stale-on-error fallback: if Consul is unreachable, keep using the last known list rather than failing every request.

Worked example: is this load a problem?

• Watches: 1,000 instances × about 5 dependencies each = 5,000 blocking queries held open. Blocking queries are designed for this; each is an idle held connection until something changes.
• A rolling deploy of a 20-instance orders service produces about 40 catalog changes (each old instance deregistered, each new one registered). If 50 caller instances watch orders, each change wakes 50 watchers: 40 × 50 = 2,000 responses. At roughly 20 KB per response (20 instances with tags and metadata), that is 2,000 × 20 KB = 40,000 KB, about 40 MB spread across the deploy window. Trivial for three servers.
• Registration writes: a few per second at most, even during a busy deploy hour. Raft write throughput is not the constraint at this size; operational simplicity is.

Justifying Consul operationally (vs etcd and ZooKeeper)

What would change my choice: if everything runs on Kubernetes, I would not add Consul at all: Kubernetes Services (a stable name for a group of pods), EndpointSlices (the live, auto-updated list of those pods' actual IPs) and cluster DNS already are a registry, with a readiness probe (Kubernetes' own periodic "can this pod take traffic yet" check) standing in for the health check. If the platform already runs etcd for other coordination and the team is Go-heavy, etcd with leases is reasonable. ZooKeeper only makes sense if it is already operated for Kafka-era or Hadoop-era systems (older big-data tools such as Apache Kafka and Hadoop that were built assuming ZooKeeper was available) and the team knows it well.

Pitfalls

• Health check says healthy, dependency is dead. Keep /healthz shallow (the process can serve) and rely on passive ejection for deep failures; a deep check that pings the database can take every instance out at once when the database blips.
• No graceful deregistration. Instances killed without deregistering keep receiving traffic until their check fails. Deregister, wait for callers to notice, then stop.
• Treating the registry as always available. Callers must cache and serve stale on error, or a registry outage becomes a whole-platform outage.

Direct answer

A secret in a config file or a Git repository is readable by everyone who can read that file or clone that repo, today and forever: Git keeps every past version, so deleting the line later does not remove it. It also cannot be rotated without a code change, and nobody can tell who read it. The secure approach is to keep secrets in a dedicated secrets store, have each container prove its identity to fetch only the secrets it needs at runtime, prefer short-lived credentials that expire on their own, log every access, and rotate by overlapping old and new rather than swapping in place.

Why plain files and version control are the wrong place

1. History is permanent. A commit that removes a password leaves the previous commit intact. Every clone, fork, CI cache and backup has a copy.
2. Access is far too broad. Everyone with read access to the repo (often the whole engineering org, plus CI systems and contractors) can read production credentials. Access control on secrets should be per service, not per repository.
3. No audit trail. You cannot answer "who read the payments database password last month?" for a file.
4. Rotation requires a deploy. Changing the secret means a commit, a review and a rollout, so in practice it never gets rotated.
5. It leaks sideways. Config files get printed in debug logs, attached to tickets, and baked into container images, where anyone who pulls the image can read them with docker history or by unpacking the layers.

Executed demonstration: deleting it does not remove it

set -e
rm -rf /tmp/leak-demo && mkdir /tmp/leak-demo && cd /tmp/leak-demo
git init -q
git config user.email dev@example.com; git config user.name dev
printf 'db_host: db.internal\ndb_password: hunter2-prod\n' > config.yaml
git add config.yaml && git commit -qm "add config"
printf 'db_host: db.internal\ndb_password: ${DB_PASSWORD}\n' > config.yaml
git commit -qam "remove password from config"
echo "current file:"; cat config.yaml
echo "search all history:"; git log -p --all -S 'hunter2' --format='commit: %s' | grep -E '^commit|hunter2'

Output (git 2.54):

current file:
db_host: db.internal
db_password: ${DB_PASSWORD}
search all history:
commit: remove password from config
-db_password: hunter2-prod
commit: add config
+db_password: hunter2-prod

The working file is clean, yet one git log -S (search history for a string) recovers the password. The only real fix once a secret has been committed is to rotate it; rewriting history is cleanup, not remediation, because copies already exist elsewhere.

A secure approach for containerized microservices

Where secrets live

A secrets manager (HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault) stores secrets encrypted, controls access per identity, versions them and logs every read. Config files and repos hold only a reference (for example "the secret at path db/payments"), never the value.

How a container authenticates to the store

This is the "secret zero" problem: you need a credential to fetch credentials. The answer is to use an identity the platform already gives the workload, instead of a password:

• In Kubernetes, the pod's service account token (a signed token the cluster issues to each pod) is presented to the store, which checks it with the cluster and maps the service account to a policy, such as "the payments service account may read db/payments only".
• On a cloud platform, the workload's cloud identity (an instance or task role) plays the same part.

Each service gets least privilege: only the secrets it needs.

Short-lived versus long-lived secrets

Default to short-lived wherever the target system supports it (databases, cloud credentials, internal certificates). Use long-lived only where a third party forces it, and put those on a rotation schedule.

How the secret reaches the process

Prefer a file on an in-memory volume, written by an agent or sidecar (a helper container in the same pod) that renews it, over environment variables. Environment variables are read once at process start, show up in crash dumps (a snapshot of a failed process's memory written to disk, which can include its environment) and /proc (the Linux virtual filesystem that exposes a running process's environment to anything with permission to read it, such as /proc/<pid>/environ), and are not refreshed when the secret changes. Kubernetes Secret objects are only base64-encoded by default (encoding, not encryption), so enable encryption at rest (encrypting the data as etcd, Kubernetes's own backing store, writes it to disk, so a copy of the disk or an etcd backup doesn't hand over the plaintext) for them or fetch directly from the external store.

Audit logging

Log every secret access: which identity, which secret path and version, when, from where, allowed or denied. Never log the secret value itself. Alert on unusual patterns: a service reading a secret it has never read, or reads from an unexpected network.

Revoking and rotating safely

Rotation fails when old and new cannot both work for a moment. The safe order:

1. Create the new secret and make the receiving side accept it (add a second API key, create the new DB password) while the old one still works.
2. Let consumers pick up the new version (the agent refreshes the file; the app reloads).
3. Confirm from logs that nothing uses the old one any more.
4. Revoke the old one.

For an emergency (a leak), skip the wait: revoke immediately and accept a short burst of errors, because an attacker holding a valid credential is worse than a brief outage.

Worked example

A payments service runs 40 pods and needs a PostgreSQL password and a card-processor API key.

• Database: the secrets store holds its own admin credential to the database and, through its database secrets engine (a plugin that runs CREATE USER / GRANT statements on request using that stored admin access), creates a fresh, least-privilege user per pod with a 1-hour TTL, renewed while the pod lives. A pod killed at 10:00 leaves a credential that dies by 11:00 at the latest, without anyone acting.
• API key: stored once at payments/processor-key, readable only by the payments service account. Rotated every 90 days (a policy choice, not a standard) using the four steps above, with the processor holding both keys during the overlap.
• Audit: 40 pods × 1 renewal every 30 minutes = 80 database-credential events per hour. A read of payments/processor-key by the search service account is denied and raises an alert.

Pitfalls

• Committing .env files. Add them to .gitignore and turn on secret scanning (an automated check that looks for patterns like API keys and private keys in a diff before it is accepted) in the repo host and in CI to block pushes that contain keys.
• Secrets in images. Anything COPY'd or set with ENV during build is in a layer forever. Inject at runtime.
• One shared "super" credential for all services: one leak compromises everything, and you cannot revoke it without breaking everyone.
• Revoking without killing sessions. A database connection opened with the old password stays logged in after the password changes, so revocation must also close those sessions.

Direct answer

This targets Consul KV (Consul is a service-discovery and configuration tool; its KV store is a simple key/value database, reachable over HTTP, that supports versioning and atomic multi-key transactions). Read the key and remember its ModifyIndex (a number that goes up every time the key changes; Consul takes it from the Raft log index, the position of that write in the ordered, replicated log its consensus protocol keeps across servers, so two writers can never be given the same index). Write the new value with a compare-and-swap (CAS) that succeeds only if the index is still the one you read, and write the audit record (who, when, change-id, old and new value) in the same Consul transaction, so the change and its history commit together or not at all. Then run the health checks; if they fail, roll back with a second CAS that is conditioned on the index your own write produced. That condition is the key to minimising races: the rollback restores the old value only if nobody has changed the key since you did, and if someone has, the tool refuses and pages a human instead of overwriting their change.

Approach

1. Snapshot: GET /v1/kv/<key> returns the value (base64-encoded, since Consul stores values as raw bytes and base64 turns arbitrary bytes into plain text that survives being embedded in JSON) and its ModifyIndex. A missing key is treated as index 0.
2. Atomic swap plus audit: one PUT /v1/txn with three operations: cas on the config key with Index = the snapshot index, set of an audit record under <key>.history/<change-id>, and get of the key to read back the new ModifyIndex. Consul transactions are all-or-nothing, so either the value changes and the audit record exists, or neither happens; a lost CAS returns HTTP 409 and the transaction rolls back.
3. Detect failure: call health_check() several times (a soak period, so one lucky pass does not count). All passes means committed.
4. Guarded rollback: another transaction that does cas back to the old value with Index = the index from step 2, plus its own audit record naming the change it reverts. If the key did not exist before, the rollback is a delete-cas instead (a conditional delete: it removes the key only if its ModifyIndex still matches the one given, so it cannot delete something someone else has since written).

Why CAS instead of a lock: a lock (a Consul session: a lease you acquire and must keep renewing, which Consul releases automatically once you stop) protects only writers that also take the lock; an operator's plain consul kv put ignores it. CAS is enforced by the server on the key itself, so even a writer that does not know about this tool causes our stale write or stale rollback to fail rather than silently clobbering it.

Code

Standard library only. It reads the agent address from CONSUL_HTTP_ADDR (default http://127.0.0.1:8500). The __main__ section is a demo against a throwaway key, run here against a local Consul 2.0.4 agent in -dev mode (a single in-memory server for testing).

import base64, json, os, time, uuid, getpass, datetime
import urllib.request, urllib.error

CONSUL = os.environ.get("CONSUL_HTTP_ADDR", "http://127.0.0.1:8500")

def _http(method, path, body=None):
    data = None if body is None else json.dumps(body).encode()
    req = urllib.request.Request(CONSUL + path, data=data, method=method)
    try:
        with urllib.request.urlopen(req) as r:
            return r.status, json.loads(r.read() or b"null")
    except urllib.error.HTTPError as e:  # txn returns 409 when it rolls back
        return e.code, json.loads(e.read() or b"null")

def b64(s): return base64.b64encode(s.encode()).decode()

def put_raw(key, value):
    """Plain, unconditional write: what an operator or another tool might do."""
    req = urllib.request.Request(f"{CONSUL}/v1/kv/{key}", data=value.encode(), method="PUT")
    urllib.request.urlopen(req).read()

def read(key):
    """Return (value, ModifyIndex); (None, 0) if the key does not exist."""
    status, body = _http("GET", f"/v1/kv/{key}")
    if status == 404:
        return None, 0
    e = body[0]
    val = base64.b64decode(e["Value"]).decode() if e["Value"] else ""
    return val, e["ModifyIndex"]

def _cas_with_audit(key, new_value, expected_index, meta):
    """ONE Consul transaction: CAS the config key AND append the audit record.
    Both commit or neither does. Returns the key's new ModifyIndex, or None if the CAS lost."""
    ops = [
        {"KV": {"Verb": "cas", "Key": key, "Value": b64(new_value), "Index": expected_index}},
        {"KV": {"Verb": "set", "Key": f"{key}.history/{meta['change_id']}",
                "Value": b64(json.dumps(meta))}},
        {"KV": {"Verb": "get", "Key": key}},   # read back our own write's index, atomically
    ]
    status, body = _http("PUT", "/v1/txn", ops)
    if status == 409:
        return None
    if status != 200:
        raise RuntimeError(f"txn failed: {status} {body}")
    return body["Results"][2]["KV"]["ModifyIndex"]

def safe_update(key, new_value, health_check, who=None, checks=3, interval_s=0.0):
    who = who or getpass.getuser()
    old_value, old_index = read(key)                       # 1. snapshot
    change_id = str(uuid.uuid4())
    meta = {"change_id": change_id, "who": who, "action": "update",
            "when": datetime.datetime.now(datetime.timezone.utc).isoformat(),
            "prev_value": old_value, "prev_index": old_index, "new_value": new_value}
    my_index = _cas_with_audit(key, new_value, old_index, meta)   # 2. atomic swap
    if my_index is None:
        return "CONFLICT: key changed since read; nothing written"
    for _ in range(checks):                                # 3. soak: N consecutive passes
        if not health_check():
            break
        time.sleep(interval_s)
    else:
        return "COMMITTED"
    # 4. rollback, but ONLY if the key still holds OUR write (CAS on my_index)
    rb = {"change_id": str(uuid.uuid4()), "who": who, "action": "rollback",
          "rolls_back": change_id,
          "when": datetime.datetime.now(datetime.timezone.utc).isoformat(),
          "prev_value": new_value, "prev_index": my_index, "new_value": old_value}
    if old_value is None:   # key did not exist before: delete it, guarded by our index
        status, _ = _http("PUT", "/v1/txn", [
            {"KV": {"Verb": "delete-cas", "Key": key, "Index": my_index}},
            {"KV": {"Verb": "set", "Key": f"{key}.history/{rb['change_id']}", "Value": b64(json.dumps(rb))}}])
        ok = status == 200
    else:
        ok = _cas_with_audit(key, old_value, my_index, rb) is not None
    return "ROLLED_BACK" if ok else "ROLLBACK_REFUSED: someone wrote after us; page a human"

if __name__ == "__main__":
    KEY = "demo/orders/timeout_ms"
    _http("DELETE", f"/v1/kv/{KEY}?recurse=true")
    _http("DELETE", f"/v1/kv/{KEY}.history?recurse=true")
    put_raw(KEY, "800")

    print("A healthy change :", safe_update(KEY, "900", lambda: True, who="alice"), "| value =", read(KEY)[0])
    print("B failing change :", safe_update(KEY, "50", lambda: False, who="bob"), "| value =", read(KEY)[0])

    # C: a concurrent writer lands between our swap and our health verdict
    def check_while_someone_else_writes():
        put_raw(KEY, "1200")
        return False
    print("C racing change  :", safe_update(KEY, "60", check_while_someone_else_writes, who="carol"), "| value =", read(KEY)[0])

    # D: stale read. Snapshot the index, let another writer win, then try to CAS
    _, stale = read(KEY)
    put_raw(KEY, "1300")
    print("D stale CAS      :", "wrote" if _cas_with_audit(KEY, "70", stale, {"change_id": "d"}) else "rejected", "| value =", read(KEY)[0])

    _, hist = _http("GET", f"/v1/kv/{KEY}.history?recurse=true")
    rows = sorted((json.loads(base64.b64decode(h["Value"])) for h in hist), key=lambda m: m.get("when", ""))
    print("audit trail      :", [(m.get("who"), m.get("action"), m.get("prev_value"), "->", m.get("new_value")) for m in rows])

Output of python3 config_cas.py:

A healthy change : COMMITTED | value = 900
B failing change : ROLLED_BACK | value = 900
C racing change  : ROLLBACK_REFUSED: someone wrote after us; page a human | value = 1200
D stale CAS      : rejected | value = 1300
audit trail      : [('alice', 'update', '800', '->', '900'), ('bob', 'update', '900', '->', '50'), ('bob', 'rollback', '50', '->', '900'), ('carol', 'update', '900', '->', '60')]

What each case proves:

• A: normal path. Value 800 becomes 900, health checks pass, audit record written.
• B: health checks fail, so the tool rolls back 50 to 900 using a CAS on its own write's index, and the rollback gets its own audit record.
• C: another writer changes the key to 1200 after our write but before our health verdict. Our rollback CAS fails (the index no longer matches), so the tool leaves 1200 in place and reports ROLLBACK_REFUSED. Blindly writing 900 back would have destroyed someone else's change.
• D: the classic stale read. A CAS with an index read before someone else's write is rejected by Consul, and the value 1300 survives.
• Audit trail: notice the unconditional writes in C and D (1200, 1300) are missing from the history. The tool can only audit writes that go through it; in production, use Consul ACLs (access control lists, Consul's token-based permissions) so that only the tool's token can write under the config prefix.

Key points

• Atomicity: the value and its audit record are written in one transaction, so there is never a change without history or history without a change.
• Race minimisation: every write is conditional on an index observed by this tool, so there is no read-modify-write window in which another writer can be silently lost. The remaining window (a writer landing between our CAS and our rollback) is detected, not ignored.
• Version metadata: change_id (a UUID, a long randomly generated identifier, so two audit records never collide), who, when (UTC ISO-8601, a standard sortable date-time format such as 2026-09-28T10:15:00Z), and the previous value and index. Rollback records point at the change they revert with rolls_back, so the history reads as a chain.
• Failure detection: several consecutive health passes, not one; the soak length (checks × interval_s) should cover at least one config-reload cycle of the consuming services, otherwise you declare success before anyone has read the new value.

Complexity

Each update is a constant number of HTTP round trips: 1 read, 1 transaction, checks health calls, and at most 1 rollback transaction. History grows by one small key per change (plus one per rollback); prune old records with a retention job, since each Consul value is capped at 512 KB and very large KV trees slow down snapshots.

Edge cases

• Key absent before the update: CAS with index 0 means "create only if missing"; rollback deletes with delete-cas guarded by our index.
• Consul unavailable during rollback: the new value stays live and the tool must exit non-zero and page; retrying the guarded rollback is safe because it is conditional.
• Consumers read between write and rollback: services may have already acted on the bad value for the soak duration. Keep the soak short and make the config backward compatible where possible.
• The health check is flaky: a false failure triggers an unnecessary rollback; since the rollback is itself audited and conditional, the cost is a wasted change, not corruption.
• Values over 512 KB: rejected by Consul; store large configs elsewhere and keep a pointer plus checksum in KV.

Trade-offs and pitfalls

• Read, compare, then write in two requests instead of server-side CAS: a race window where two tools both "win".
• Unconditional rollback (just writing the old value back) is the most common bug: it reverts other people's changes made during your soak.
• Storing history in the same key (a JSON blob with all versions) makes every write larger and every CAS contend on one key; separate history keys avoid that.
• Using Flags (Consul's opaque 64-bit number on each entry) to hold a version is tempting but cannot carry who/when; keep it for a compact schema version at most.