Service Discovery and Configuration Management Questions

Direct answer

A secret in a config file or a Git repository is readable by everyone who can read that file or clone that repo, today and forever: Git keeps every past version, so deleting the line later does not remove it. It also cannot be rotated without a code change, and nobody can tell who read it. The secure approach is to keep secrets in a dedicated secrets store, have each container prove its identity to fetch only the secrets it needs at runtime, prefer short-lived credentials that expire on their own, log every access, and rotate by overlapping old and new rather than swapping in place.

Why plain files and version control are the wrong place

1. History is permanent. A commit that removes a password leaves the previous commit intact. Every clone, fork, CI cache and backup has a copy.
2. Access is far too broad. Everyone with read access to the repo (often the whole engineering org, plus CI systems and contractors) can read production credentials. Access control on secrets should be per service, not per repository.
3. No audit trail. You cannot answer "who read the payments database password last month?" for a file.
4. Rotation requires a deploy. Changing the secret means a commit, a review and a rollout, so in practice it never gets rotated.
5. It leaks sideways. Config files get printed in debug logs, attached to tickets, and baked into container images, where anyone who pulls the image can read them with docker history or by unpacking the layers.

Executed demonstration: deleting it does not remove it

set -e
rm -rf /tmp/leak-demo && mkdir /tmp/leak-demo && cd /tmp/leak-demo
git init -q
git config user.email dev@example.com; git config user.name dev
printf 'db_host: db.internal\ndb_password: hunter2-prod\n' > config.yaml
git add config.yaml && git commit -qm "add config"
printf 'db_host: db.internal\ndb_password: ${DB_PASSWORD}\n' > config.yaml
git commit -qam "remove password from config"
echo "current file:"; cat config.yaml
echo "search all history:"; git log -p --all -S 'hunter2' --format='commit: %s' | grep -E '^commit|hunter2'

Output (git 2.54):

current file:
db_host: db.internal
db_password: ${DB_PASSWORD}
search all history:
commit: remove password from config
-db_password: hunter2-prod
commit: add config
+db_password: hunter2-prod

The working file is clean, yet one git log -S (search history for a string) recovers the password. The only real fix once a secret has been committed is to rotate it; rewriting history is cleanup, not remediation, because copies already exist elsewhere.

A secure approach for containerized microservices

Where secrets live

A secrets manager (HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault) stores secrets encrypted, controls access per identity, versions them and logs every read. Config files and repos hold only a reference (for example "the secret at path db/payments"), never the value.

How a container authenticates to the store

This is the "secret zero" problem: you need a credential to fetch credentials. The answer is to use an identity the platform already gives the workload, instead of a password:

• In Kubernetes, the pod's service account token (a signed token the cluster issues to each pod) is presented to the store, which checks it with the cluster and maps the service account to a policy, such as "the payments service account may read db/payments only".
• On a cloud platform, the workload's cloud identity (an instance or task role) plays the same part.

Each service gets least privilege: only the secrets it needs.

Short-lived versus long-lived secrets

Default to short-lived wherever the target system supports it (databases, cloud credentials, internal certificates). Use long-lived only where a third party forces it, and put those on a rotation schedule.

How the secret reaches the process

Prefer a file on an in-memory volume, written by an agent or sidecar (a helper container in the same pod) that renews it, over environment variables. Environment variables are read once at process start, show up in crash dumps (a snapshot of a failed process's memory written to disk, which can include its environment) and /proc (the Linux virtual filesystem that exposes a running process's environment to anything with permission to read it, such as /proc/<pid>/environ), and are not refreshed when the secret changes. Kubernetes Secret objects are only base64-encoded by default (encoding, not encryption), so enable encryption at rest (encrypting the data as etcd, Kubernetes's own backing store, writes it to disk, so a copy of the disk or an etcd backup doesn't hand over the plaintext) for them or fetch directly from the external store.

Audit logging

Log every secret access: which identity, which secret path and version, when, from where, allowed or denied. Never log the secret value itself. Alert on unusual patterns: a service reading a secret it has never read, or reads from an unexpected network.

Revoking and rotating safely

Rotation fails when old and new cannot both work for a moment. The safe order:

1. Create the new secret and make the receiving side accept it (add a second API key, create the new DB password) while the old one still works.
2. Let consumers pick up the new version (the agent refreshes the file; the app reloads).
3. Confirm from logs that nothing uses the old one any more.
4. Revoke the old one.

For an emergency (a leak), skip the wait: revoke immediately and accept a short burst of errors, because an attacker holding a valid credential is worse than a brief outage.

Worked example

A payments service runs 40 pods and needs a PostgreSQL password and a card-processor API key.

• Database: the secrets store holds its own admin credential to the database and, through its database secrets engine (a plugin that runs CREATE USER / GRANT statements on request using that stored admin access), creates a fresh, least-privilege user per pod with a 1-hour TTL, renewed while the pod lives. A pod killed at 10:00 leaves a credential that dies by 11:00 at the latest, without anyone acting.
• API key: stored once at payments/processor-key, readable only by the payments service account. Rotated every 90 days (a policy choice, not a standard) using the four steps above, with the processor holding both keys during the overlap.
• Audit: 40 pods × 1 renewal every 30 minutes = 80 database-credential events per hour. A read of payments/processor-key by the search service account is denied and raises an alert.

Pitfalls

• Committing .env files. Add them to .gitignore and turn on secret scanning (an automated check that looks for patterns like API keys and private keys in a diff before it is accepted) in the repo host and in CI to block pushes that contain keys.
• Secrets in images. Anything COPY'd or set with ENV during build is in a layer forever. Inject at runtime.
• One shared "super" credential for all services: one leak compromises everything, and you cannot revoke it without breaking everyone.
• Revoking without killing sessions. A database connection opened with the old password stays logged in after the password changes, so revocation must also close those sessions.

Direct answer

Use Vault as the single source of truth, have every pod authenticate as itself using its Kubernetes service account token, and deliver secrets as files on an in-memory volume written by a node-level agent (the Secrets Store CSI Driver with the Vault provider), with rotation enabled. The application re-reads the file when it changes, which is what makes rotation work without a restart. Prefer short-lived dynamic credentials (Vault generates a database user per workload with a lease) over long-lived static secrets. Keep Vault Agent sidecars for the minority of workloads that need templating or per-pod lease renewal, and do not use plain Kubernetes Secrets as the source of truth for sensitive material.

Terms

• Vault: HashiCorp's secrets manager. It stores secrets, issues dynamic secrets (credentials created on demand with an expiry), and writes an audit log of every request.
• Lease / TTL: a dynamic secret comes with a time-to-live. The holder must renew it before expiry or get a new one; Vault revokes it afterwards.
• Least privilege: each workload can read only the secrets it needs, and nothing else.
• Kubernetes auth method: a pod presents its service account token (a signed JWT, JSON Web Token) to Vault; Vault verifies it with the Kubernetes API and maps the service account to a Vault policy.
• CSI (Container Storage Interface) driver: a node-level plugin that mounts volumes into pods. The Secrets Store CSI Driver mounts secrets fetched from an external store as files.
• DaemonSet: a Kubernetes workload type that runs exactly one copy of a pod on every node (or every matching node), unlike a Deployment which runs a chosen number of copies wherever the scheduler likes. It is the standard way to run one node-level agent per machine.
• Audience-bound token: a service account token that is only valid for a declared recipient (Vault, here); even if it leaked, another service could not use it to authenticate as this pod to something else.
• Alpha (Kubernetes feature stage): an early, experimental stage of a feature. It can change behaviour or be removed between versions with no long-term compatibility guarantee, and many production platforms disable alpha features by policy.
• Rotation: replacing a credential with a new one while the old one is still valid for an overlap period, so nothing breaks mid-swap.

The four options, compared

The design

flowchart LR
  Pod[App pod with service account] -->|mounts in-memory volume| CSI[CSI driver on node]
  CSI -->|pod's own SA token| Prov[Vault provider on node]
  Prov -->|Kubernetes auth login| Vault[Vault cluster]
  Vault -->|verify token| K8s[Kubernetes API]
  Vault -->|policy per service account| Prov
  Vault --> Audit[Audit log sink]
  Prov -->|write files, re-fetch on rotation| Pod

1. Identity. Every workload gets its own service account. The CSI driver passes the pod's projected service account token (a short-lived, auto-rotated token Kubernetes mounts into the pod for exactly this purpose, restricted to Vault as its audience) to the provider. Vault's Kubernetes auth role maps namespace=payments, sa=payments-api to a policy that allows only database/creds/payments-ro (the database secrets engine, generating a dynamic, read-only credential named payments-ro) and kv/data/payments/* (everything under the payments folder of the plain key-value secrets engine).
2. Delivery. Secrets land as files in a tmpfs (memory-backed) volume, never on node disk and never in environment variables (env vars leak into crash dumps and child processes: by default, every process an application spawns inherits a copy of its parent's environment, so a secret sitting in an env var ends up in tools and scripts that have no reason to see it, and cannot change without a restart).
3. Rotation without restarts. Three things need to work together:
   • Delivery side: the CSI driver re-fetches on its poll interval and rewrites the file when Vault returns a new value.
   • Application side: the app watches the file (inotify, the Linux file-change notification API, or re-reading on a timer) and swaps its connection pool or client credentials in place. This is the part teams forget: an app that reads a password once at startup will keep using the old one until it is revoked, and then fail. Provide a shared library that does "watch file, rebuild client, drain old connections" so each team does not reimplement it.
   • Overlap: rotate with two valid credentials at once (Vault's database engine issues a new user while the old lease is still valid; static secrets should support "current" and "previous"), so a pod that has not yet re-read the file keeps working.
4. Node-level caching. Whatever caches on the node (the provider itself, or a node-local Vault Agent used as a caching proxy in front of Vault) must key entries by identity plus path, not by path alone, so two pods with different service accounts on the same node never share an entry. Pods of the same service on the same node can share, which is where the cache saves Vault traffic. A cache keyed only by secret path would let a pod read a secret another pod on the node fetched: a confused-deputy bug, where the agent's own broad access is used on behalf of a caller who should not have it.
5. Audit. Vault's audit device (a pluggable log sink Vault writes every request to, typically shipped into your normal log pipeline) logs every login and read with the requesting identity. Because each pod authenticates as its own service account, the log answers "which workload read this secret", not just "the node agent did".

Worked example: load and footprint

Assume 5,000 containers, each using 3 dynamic secrets with a 1-hour TTL, renewed at two-thirds of the TTL (40 minutes = 2,400 s).

• Vault renewal/issue rate: 5,000 × 3 / 2,400 s = 6.25 requests per second in steady state. That is modest; the danger is not the average but a thundering herd (a mass of clients all making the same request at the same instant, turning a trivial average load into a brief spike large enough to overwhelm the target) when a node pool restarts and hundreds of pods log in at once. Add jitter to renewal times and rate-limit logins per node.
• Footprint, using an assumed budget of 64 MiB per Vault Agent sidecar: 5,000 × 64 MiB = 320,000 MiB, about 312.5 GiB of cluster memory just for sidecars. With 200 nodes and an assumed 128 MiB per node agent: 200 × 128 MiB = 25,600 MiB, about 25 GiB. Measure the real figures on your workloads; the point is the scaling shape (per pod vs per node), which is why the node agent is the default here.
• Latency: reads come from a local file, so the hot path never touches the network. Vault is only on the refresh path.

Trade-offs and what would change the choice

• Node agent risk: a larger blast radius per node. A compromised node agent can request secrets for any pod scheduled on that node (it holds their tokens). Mitigate with node isolation for sensitive workloads (dedicated node pools for payments) and short token lifetimes.
• Alpha rotation in the CSI driver. If your platform policy forbids alpha features, use Vault Agent sidecars for workloads that must rotate, and the CSI driver only for secrets that change rarely.
• Vault availability becomes critical. Run it highly available, and make sure a Vault outage only blocks new secrets, not running pods: files already on the volume keep working until their credentials expire, so choose TTLs longer than your realistic Vault recovery time.
• Plain Kubernetes Secrets are acceptable for low-sensitivity, rarely-rotated values if encryption at rest and tight RBAC are configured. They are not acceptable as the only protection for database or payment credentials.
• When sidecars win: workloads needing complex templates (rendering one full config file that weaves together several secrets and static values, not just dropping each secret in its own file), or per-pod lease renewal for dynamic credentials that must be revoked the moment the pod dies.

Direct answer

Zero-downtime rotation rests on one rule: there must always be a moment when both the old and the new secret are accepted, and the old one is retired only after you have evidence nobody still uses it. So every rotation runs in four ordered phases: make the new secret acceptable to whoever verifies it, distribute it to every consumer, confirm the consumers have switched, and only then revoke the old one. I would push most of the fleet onto short-lived, per-instance credentials issued by a secrets manager (such as HashiCorp Vault) so that "rotation" becomes routine expiry, and run the four-phase process only for the long-lived secrets that cannot be made dynamic (third-party API keys, signing keys, a certificate authority or CA).

Terms used below

• Secret: anything that grants access: a database password, an API key, a TLS private key, a token-signing key.
• Short-lived (dynamic) credential: a credential minted on request for one workload, with a built-in expiry (its lease or TTL, time-to-live). Vault's database secrets engine, for example, creates a new database user per lease and drops it when the lease ends.
• Dual-valid window: the period when both old and new secrets work. Its length is the core design parameter.
• Key version header: a label sent alongside a signed or encrypted message that says which key produced it, such as the kid (key ID) header in a JSON Web Token (JWT). It lets a verifier hold several keys at once and pick the right one.

The rotation state machine

stateDiagram-v2
    [*] --> Staged: generate v2, store encrypted
    Staged --> Accepted: verifier trusts v1 AND v2
    Accepted --> Distributed: consumers fetch v2
    Distributed --> Verified: telemetry shows no v1 use
    Verified --> Revoked: v1 disabled at verifier
    Revoked --> [*]
    Distributed --> Accepted: rollback (consumers back to v1)

Each arrow is a gate with a check, not a timer alone. Rollback is possible right up to Revoked, and never after it, which is why the evidence gate before revocation matters most.

1. Distributing new secrets

• Pull, never bake. The core idea is that the workload's own platform identity, something the platform already hands it for free, doubles as its credential to the secrets store, so nothing has to be pre-shared. Two common implementations: a Kubernetes service-account token (a signed token the cluster issues to each pod) presented through Vault's Kubernetes auth method (Vault's plugin that verifies that token with the cluster), or a cloud workload identity (the equivalent for a VM or serverless task). Containers authenticate this way and fetch secrets at runtime. Nothing lives in the image or in environment variables baked into a deployment spec.
• A local agent per pod. A sidecar (a second container that runs alongside the application container in the same pod, sharing its network and storage) or init container (a container that runs once before the main container starts, then exits), such as Vault Agent, the Secrets Store CSI Driver (a Kubernetes storage plugin that mounts secrets from an external store as files), or a mesh agent (the local proxy a service mesh, a layer that manages service-to-service networking and certificates for every workload, installs next to each one) for certificates, fetches, renews and writes secrets to an in-memory volume (tmpfs, so they never touch disk) and re-renders the file when a new version appears. The application watches the file and reloads.
• Environment variables are the trap. A process reads its environment once at start. Kubernetes updates Secrets mounted as volumes eventually, but does not update values injected as environment variables or files mounted with subPath. Those require a restart, which turns every rotation into a fleet-wide rolling restart.
• Stagger the fetch. Consumers refresh on their own schedule with jitter (random offset) rather than all at once when a new version lands.

2. Coordinating dependent services

Order matters, and the rule is: the side that verifies changes first, the side that presents changes second.

3. Guaranteeing minimal exposure to old secrets

"Exposure" means the time an old secret remains usable after we decided to replace it. The window must cover the slowest consumer, and nothing more:

where T_refresh is the longest time for a consumer to pick up the new version, T_conn is the longest a pooled connection opened with the old secret stays open, and T_margin is a buffer added on top of the other two so the revocation decision does not hinge on borderline telemetry.

Key-version headers in practice

Anything signed or encrypted should say which key produced it, so verifiers and decryptors can hold several keys during a rotation:

• Tokens: the kid in the JWT header selects the verification key from the key set.
• Encrypted data at rest: store the key version next to each ciphertext (for example a v2: prefix), so records written before the rotation still decrypt with v1 while new writes use v2, and a background job re-encrypts old records before v1 is retired. Vault's transit engine formats its ciphertexts this way.
• Webhooks and service-to-service signatures: send a header naming the key version, and let the receiver accept the current and previous version during the window.

Without a version label, a verifier has to try every key it holds, and it cannot tell you from logs who is still using the old one, which removes the evidence the revocation gate depends on.

4. Auditing the rotation

• Store-side audit: every read of every secret version, by which identity, from where. In Vault, audit devices (file, syslog, socket) log every request; Vault refuses to serve requests if it cannot write to at least one enabled audit device, so auditing cannot be silently lost.
• Usage-side evidence: the verifier's logs of which key version was presented (DB login user, API key ID, kid of verified tokens). This is the gate for revocation: "zero v1 uses in the last N minutes across all verifiers".
• Rotation record: who or what triggered it, version numbers, the time each phase gate passed, and whether it rolled back. Alert when a secret passes its maximum age without rotating.
• Never log the value, only version IDs or a short hash.

5. Race conditions and how each is closed

1. Consumer ahead of verifier. A pod fetches v2 before the database accepts it and fails to log in. Closed by ordering: the verifier accepts v2 before v2 is readable by consumers (the Staged to Accepted gate).
2. Two rotators at once. A scheduled rotation and a manual one both generate a "v2", one overwrites the other, and half the fleet holds a secret that no longer exists. Closed by a compare-and-set write (a write that only succeeds if the stored value still matches what was last read, so two writers racing to change it cannot silently clobber each other) on the version number (write v3 only if the current version is still v2) or a lock.
3. Revoke before drain. v1 is disabled while pooled connections or cached tokens still use it. Closed by the telemetry gate plus the connection-lifetime term in the window formula.
4. Rollback re-enables a revoked secret. Rolling back after revocation would resurrect a secret you may be rotating because it leaked. Closed by making rollback only possible before Revoked; after it, roll forward to v3.
5. Restart storm. A new version triggers every pod to restart or re-fetch at once, overloading the secrets store. Closed by jittered refresh and by avoiding restart-based delivery.

Worked example: 5,000 containers on one PostgreSQL cluster

Inputs (illustrative, chosen to show the arithmetic): consumers poll for a new version every 5 minutes; the connection pool recycles connections at most every 30 minutes; margin 10 minutes.

So the old password is revoked no earlier than 45 minutes after distribution, and only if the database's login telemetry shows zero v1 logins for the last 10 minutes. If you shortened the pool's maximum connection lifetime to 10 minutes, the window drops to 25 minutes: connection lifetime, not secret distribution, is usually the dominant term.

Load on the secrets store if the fleet moves to dynamic credentials with a 1-hour lease renewed at half-life (a common convention: renew once half the lease's time remains, so one missed renewal still leaves margin before expiry), that is, every 30 minutes, 1,800 s:

That is trivial. The risk is the burst: if all 5,000 pods restart within 60 seconds after an outage, that is 5000 / 60 ≈ 83 requests/s of new credential creation, and each creation is also a CREATE ROLE on the database. Plan for the burst: rate-limit creation, jitter start-up, and size the database's connection limits for the extra users briefly alive during overlap.

Trade-offs and pitfalls

• Dynamic credentials vs a shared rotated secret. Dynamic per-instance credentials give the smallest blast radius (how much is exposed when one thing goes wrong) (one leaked credential is one pod for one lease) and make revocation per instance, but they make the secrets store a hard dependency for every start-up. Mitigate with long enough leases to ride out a short store outage and a highly available store. I would still choose them for anything we control.
• Short TTLs are not free. A 5-minute TTL means a 5-minute store outage breaks everyone. Pick TTLs that exceed your store's realistic recovery time.
• In-place rotation (one credential, changed in place) always has a gap, because the change and the consumers' reload cannot be simultaneous. Use two credentials or dynamic ones.
• "Revocation" that only stops new logins is incomplete. Kill existing sessions and invalidate caches, or a leaked credential keeps working through an open connection.
• Emergency rotation is a different mode. When a secret has leaked, you shrink the dual-valid window deliberately and accept some errors, because leaving the old secret alive is worse than a brief outage. Decide that policy before the incident.