Distributed Systems Security and Trust Questions

Direct answer

Give every workload a cryptographic identity issued by its own region's certificate authority after the platform attests what the workload is, and make every credential short-lived so rotation is continuous rather than an event. Authentication happens with mutual TLS (both sides present certificates) using those identities; authorization is a default-deny policy keyed on the caller's identity, evaluated locally. The design principle that answers "what if the identity provider is down" is: verification never needs the issuer online, and issuance is regional with enough credential lifetime left to ride out an outage.

Terms

• Workload identity: a name for a running service, for example the SPIFFE ID spiffe://prod.eu.example/ns/payments/sa/api (SPIFFE, the Secure Production Identity Framework for Everyone, is an open standard for naming workloads and issuing them certificates, called SVIDs, short for SPIFFE Verifiable Identity Document: the actual certificate or token a workload presents).
• Attestation: proving what a workload is from facts the platform controls (its Kubernetes service account, node, image) rather than from something the workload says.
• Certificate chain: a certificate is signed by the authority above it (its issuer); that issuer's own certificate is signed by the authority above it, and so on up to a root. A workload's certificate, at the bottom of that chain, is called a leaf certificate. To trust a leaf, a verifier walks the chain up to a root it already has in its trust bundle; if it has never been told about one of the intermediates in between, it cannot validate anything that intermediate signed, no matter how correctly the leaf itself was issued. That is what "chained to a CA" means, and it is why the rotation order below matters.
• Trust domain / trust boundary: the set of workloads that share a root of trust; crossing a boundary requires explicit federation: the two sides must each deliberately configure trust for specific identities in the other domain, rather than automatically trusting everything the other domain issues.
• Trust bundle: the set of CA (certificate authority) certificates a verifier accepts.

Architecture

flowchart TB
  ROOT[Offline root CA] --> ICA1[Intermediate CA: region EU]
  ROOT --> ICA2[Intermediate CA: region US]
  ICA1 --> AG1[Node agents EU clusters]
  ICA2 --> AG2[Node agents US clusters]
  AG1 --> W1[Workload certs, 24h]
  AG2 --> W2[Workload certs, 24h]
  PAP[Policy repo] --> PE1[Local policy check EU]
  PAP --> PE2[Local policy check US]

Token and credential issuance

1. A node agent (a small process the platform runs on every host, distinct from the application itself) attests the workload using facts it can verify locally: its Kubernetes service account, its namespace (a Kubernetes grouping boundary that scopes and isolates related workloads), and its image digest (a cryptographic hash identifying the exact container image build, so a tampered or swapped image gets a different identity). It then requests a certificate from the regional intermediate CA.
2. The workload receives an X.509 certificate (X.509 is the standard format essentially every TLS certificate uses) valid for 24 hours, with its SPIFFE ID in the subject alternative name, a field in the certificate that states which identity or identities it is valid for.
3. For calls that also need application-level claims (user on whose behalf, audience), a regional token service issues JWTs (JSON Web Tokens: compact, signed tokens that carry a set of claims, such as who the caller is and who the token is for) valid for 5 minutes with an audience claim, signed with a key whose public half is distributed alongside the trust bundle.

Short-lived credentials and outage tolerance, with numbers

Agents renew a certificate when 50% of its lifetime remains. The worst case for an issuer outage is that it starts at the moment renewal was due, so the tolerance is the remaining lifetime at the renewal point:

Commit to 24 hours for certificates. Twelve hours of outage tolerance is long enough to fix a broken CA across a night, and a day is a tolerable exposure window when combined with the other controls. Load is trivial: 50,000 workloads renewing every 12 hours is 50,000 / 43,200 s, about 1.2 issuances per second. The JWT layer stays at 5 minutes because it carries user context, where faster expiry matters more.

Certificate and key rotation

• Leaf certificates rotate continuously by construction.
• Intermediate CAs rotate every few months by overlap: add the new intermediate to trust bundles first, wait until every verifier has the updated bundle, then start issuing from it, and remove the old one only after the last certificate it signed has expired (24 hours plus margin). Reversing that order causes an outage because verifiers reject certificates chained to a CA they have not been told about.
• The root is kept offline in an HSM (hardware security module) and used only to sign intermediates. Root rotation is the same add-then-switch-then-remove sequence stretched over weeks.
• JWT signing keys rotate the same way, publishing the new public key before signing with it.

Trust boundaries

• One trust domain per region (or per environment), not one global domain. A compromised EU intermediate lets an attacker mint EU identities only.
• Cross-region calls use federation: US verifiers are explicitly configured to accept the EU bundle, for specific identities. The US payments service accepts spiffe://prod.eu.example/ns/checkout/sa/api and nothing else from EU.
• Staging and production never share a root. This is the most common real-world boundary failure.

Least-privilege authorization

Policies are written against identities, not network addresses: "payments accepts calls from checkout on POST /charges; everything else is denied". Policies live in version control, are compiled and pushed to a local enforcement point in every workload (a sidecar, a helper process deployed alongside the application container, or a library linked into the application itself), and are evaluated without a network call. New services start with zero inbound permissions and receive them through review. Every allow and deny is logged with caller identity and policy version.

Fallback when the identity provider is down

Walk through what depends on the issuer:

What we deliberately do not do: fall back to a long-lived static shared secret, or disable mTLS. Both turn an availability incident into a security incident. If the outage outlasts the 12-hour tolerance, the controlled response is an emergency extension: issue longer-lived certificates from a standby intermediate held for this purpose, logged and time-boxed.

Operational practices for key management

• Private keys for leaf certs are generated on the host and never leave it; only CSRs (certificate signing requests) travel.
• CA keys live in HSMs or a cloud KMS (key management service); root operations require two people (dual control) and are recorded.
• Monitor: certificate time-to-expiry across the fleet (alert when any workload is under 25% of lifetime), issuance error rates, and bundle version per verifier.
• Rehearse a compromised intermediate: revoke it by removing it from bundles, re-issue the region's workloads from a fresh intermediate, and measure how long that takes. That measured time is your real recovery objective.

Trade-offs and pitfalls

• Short lifetimes replace revocation lists. Certificate revocation lists and OCSP (Online Certificate Status Protocol) checks are rarely reliable at mesh scale (the scale of a service mesh, the network of many services all calling each other over these short-lived certificates); expiry is. The cost is issuer availability, which the table above quantifies.
• Regional isolation costs federation configuration. It is worth it: it bounds a CA compromise to one region.
• Clock skew breaks short-lived credentials. Allow a small skew tolerance (a minute or two) and monitor time sync.
• Common wrong turn: authorizing by IP or namespace label. Those change with every reschedule; identities do not.

Direct answer

Use opaque tokens at the edge and short-lived, audience-scoped JWTs inside the mesh (the network of proxies that carries and secures service-to-service traffic across the estate), minted by a central token service through token exchange at each trust hop. The edge gateway turns the user's external token into an internal token that carries only who the user is and which service it is meant for, not every permission they hold. Downstream services authenticate the calling service with its workload identity (for example mTLS, mutual TLS where both sides present certificates), validate the internal token locally, and ask a policy decision point for fine-grained decisions, with a local sidecar cache (a small helper process deployed alongside each service instance, sharing its network so it can intercept and answer calls without a remote hop) to keep that off the latency critical path. Revocation is handled by keeping internal tokens to a few minutes and pushing revocation events for the rare cases that cannot wait.

Vocabulary first

• JWT (JSON Web Token): a signed, self-contained token. Any service holding the issuer's public key can verify it without a network call. Downside: once issued it is valid until it expires.
• Opaque token: a random string meaningful only to the issuer. Verifying it means calling the issuer (introspection, RFC 7662). Revocation is instant, but every check is a network call.
• Token exchange (RFC 8693): an OAuth 2.0 grant where a service presents a token it holds and receives a new token for a different audience, typically with narrower scope.
• PEP / PDP / PAP / PIP: the policy enforcement point (code that blocks or allows the request), policy decision point (evaluates the policy), policy administration point (where policies are authored and versioned) and policy information point (the source of attributes such as team membership).

Architecture

flowchart LR
  U[Client] -->|opaque token| GW[Edge gateway]
  GW -->|exchange| TS[Token service]
  TS -->|internal JWT aud=orders| GW
  GW -->|JWT + mTLS| O[Orders service]
  O -->|exchange for aud=payments| TS
  O -->|new JWT + mTLS| P[Payments service]
  O -.->|check| SC1[Policy sidecar]
  P -.->|check| SC2[Policy sidecar]
  PAP[Policy admin and store] -->|bundles| SC1
  PAP -->|bundles| SC2

1. Token formats: commit per boundary

Introspecting an opaque token on every internal hop does not scale: 20,000 requests per second fanning out through an average of 6 internal hops is 120,000 introspection calls per second against one service, which becomes the platform's single point of failure.

2. Token exchange for internal calls

Forwarding the user's original token unchanged through the call chain is the common anti-pattern: every downstream service receives a token that is valid at every other service (a replayable bearer credential: whoever holds the bytes can use it, with no extra proof that they are the party it was issued to), and a compromised low-value service can replay it against a high-value one.

Instead, each hop exchanges:

1. Orders receives a JWT with aud: orders.
2. To call Payments it presents that token plus its own workload identity to the token service and requests aud: payments.
3. The token service checks policy ("may orders call payments on behalf of a user?") and returns a JWT with aud: payments, the same sub (the user), and an act (actor) claim recording that orders is the intermediary. RFC 8693 defines act for exactly this delegation chain.

Payments rejects any token whose aud is not payments, so a token stolen from Orders is useless against Payments. Cost: one extra call per cross-service hop, which you cache: the exchanged token is reusable for its lifetime for the same user and audience.

3. Refresh and revocation

• Internal tokens are not refreshed; they are re-minted. They live 2 to 5 minutes, so the worst-case window in which a revoked user keeps access internally is that lifetime.
• Refresh tokens exist only between the client and the authorization server, are rotated on every use (a reused refresh token signals theft and revokes the whole family), and never enter the mesh.
• Urgent revocation (account takeover, fired employee): the auth server publishes a revocation event (sub or session ID plus a timestamp) on a stream; sidecars and gateways keep a small in-memory deny set and reject any token for that subject issued before the revocation time. The set stays small because entries can be dropped once every token issued before them has expired.

4. Minimizing token bloat

Putting a user's permissions in the token is the fastest way to break the platform. Measured by encoding real JWTs (ES256: an elliptic-curve signing algorithm producing a compact, fixed 64-byte signature, versus RS256, an RSA signature that runs 256 bytes for a 2048-bit key; compact JSON): jwt_len builds the same three parts a real token has (a header naming the algorithm and key, a claim set, and a fixed-length signature), base64url-encodes each, and adds them up, once with a bare claim set and once with a growing list of permission strings appended to it, to measure how token size scales with how many permissions ride inside it.

import base64, json

def b64url(b):
    return base64.urlsafe_b64encode(b).rstrip(b"=")

def jwt_len(claims):
    header = {"alg": "ES256", "kid": "2026-09-a", "typ": "JWT"}
    h = b64url(json.dumps(header, separators=(",", ":")).encode())
    p = b64url(json.dumps(claims, separators=(",", ":")).encode())
    sig = b64url(bytes(64))  # an ES256 signature is always 64 bytes (r || s)
    return len(h) + 1 + len(p) + 1 + len(sig)

base = {"iss": "https://auth.internal", "sub": "user:48213", "aud": "orders-api",
        "exp": 1790000000, "iat": 1789999100, "jti": "8f14e45fceea167a"}
print("minimal claims:", jwt_len(base), "bytes")
for n in (50, 200, 500):
    fat = dict(base, perms=[f"tenant-{i:04d}:orders:read" for i in range(n)])
    print(f"{n} permission strings:", jwt_len(fat), "bytes")

minimal claims: 319 bytes
50 permission strings: 2066 bytes
200 permission strings: 7266 bytes
500 permission strings: 17666 bytes

Each permission string like tenant-0004:orders:read is 23 raw characters; as one element of a JSON array that is 26 bytes (23 plus 2 quote bytes plus a comma), and base64url encoding then expands every 3 raw bytes into 4 encoded characters, a 4/3 blow-up, so each permission costs about 26 x 4/3 ~= 34.7 bytes in the final token. That matches the numbers above exactly: 200 permissions minus 50 is 7266 - 2066 = 5200 bytes over 150 permissions, 5200 / 150 ~= 34.7 bytes each. nginx's default buffer for one large request header line is 8 KB, so a user with a few hundred permissions starts getting unexplained 4xx errors at the proxy, and a 7 KB header repeated across 6 hops is 42 KB of overhead per request. The fix is structural, not compression: keep identity claims (sub, aud, exp, iat, jti, tenant, maybe a coarse role) in the token and resolve permissions at decision time from the PIP. Using ES256 rather than RS256 also helps: its signature is 64 bytes versus 256 bytes for a 2048-bit RSA key.

5. Fine-grained authorization downstream

Three options, and where each fits:

Recommendation: middleware for coarse scope checks, a sidecar PDP for most fine-grained rules, and a central relationship service only for resource-level sharing graphs. Every decision is logged with the policy version that produced it so an auditor can reconstruct why access was granted.

Trade-offs and pitfalls

• JWTs trade revocation for scale. The honest statement of the design is "internal access can outlive revocation by at most the internal token lifetime", and that lifetime is the knob.
• Token exchange adds a dependency. If the token service is down, cross-service calls fail. Run it per region, cache exchanged tokens, and treat it with the same SLO (service-level objective) as the edge.
• Do not let services mint their own internal tokens. Only the token service holds signing keys; services only verify.
• Policy drift: sidecars running stale bundles make inconsistent decisions. Export the bundle version each sidecar is running and alert when any lags the published version beyond a threshold.
• Common wrong turn: using the same token for user identity and service identity. Keep them separate: the certificate says which service, the token says which user.

Direct answer

An opaque token is a random string that means nothing by itself; a service must ask the authorisation server "is this valid, and for whom?" (token introspection, standardised in RFC 7662). A JWT (JSON Web Token) carries its claims (the payload's key-value assertions about the token, such as who it identifies and when it expires) and a signature, so any service can validate it locally. Under network partitions the difference becomes a choice between consistency and availability of the auth decision: opaque tokens give you accurate, immediately revocable decisions but fail (or go stale) when the introspection endpoint is unreachable; JWTs keep working through a partition but keep honouring a revoked token until it expires or a denylist reaches that node. My recommendation for a global microservices estate: opaque tokens at the external trust boundary, introspected at the regional gateway, exchanged for short-lived JWTs inside, with the partition behaviour chosen per route from its SLO (service-level objective) and risk.

Side-by-side on the five named axes

Proposed architecture for each approach

Opaque-token architecture

• A token service with a replicated token store in every region (not one central region), so introspection is an in-region call. Tokens are created with a region-independent ID and the store is replicated asynchronously; revocations replicate the same way, with priority.
• Introspection happens once at the edge gateway (the proxy at the boundary between the open internet and your internal services, which terminates and inspects every incoming request), not in every microservice; results are cached for a short TTL (for example 10 seconds).
• Revocation is a delete in the store plus a purge message to gateway caches.

JWT architecture

• Access tokens with a 5-minute lifetime, signed by keys published through a JWKS endpoint; refresh tokens (long-lived credentials a client exchanges for a fresh access token, so it does not have to make the user log in again) held server-side.
• A revocation stream pushing token IDs and per-user "revoked before" timestamps to every service's in-memory denylist.
• Services validate locally; no runtime dependency on the auth server except key refresh and the stream.

The hybrid I would ship (the "phantom token" pattern)

sequenceDiagram
  participant C as External client
  participant GW as Regional gateway
  participant AS as Regional token service
  participant S as Internal service
  C->>GW: request + opaque token
  GW->>AS: introspect (cache miss only)
  AS-->>GW: active, subject, scopes
  GW->>GW: mint 60 s internal JWT
  GW->>S: request + internal JWT
  S->>S: verify signature locally

The outside world holds an opaque token: nothing leaks if it is logged, and it is revocable at once. The gateway's introspection call above gets back whether the token is active, its subject (whose token it is) and its scopes (the specific permissions it grants), and mints a JWT carrying only those. Inside, services get a JWT that lives 60 seconds, so revocation reaches internal calls within one minute without any denylist, and internal hops never block on the auth server.

Worked example: what a partition does

A request path spans three internal hops, and the EU region loses connectivity to the US for 20 minutes. The token service is replicated in both regions.

• All-JWT (5-minute access tokens): EU keeps serving. A token revoked in the US at minute 2 of the partition is honoured in EU until its exp passes, at most 5 minutes after it was issued, because the revocation stream cannot cross the partition. Worst-case exposure: 5 minutes.
• All-opaque with central introspection in the US: every EU cache miss fails. With a 10-second cache, EU effectively goes down for authenticated traffic within about 10 seconds of the partition starting, unless you fail open.
• Hybrid with regional token stores: EU introspects locally, so it stays up. A US-side revocation does not reach the EU store until the link heals, so exposure is the partition duration for that token on EU gateways. Mitigation: EU gateways notice replication lag (heartbeat sequence stops advancing: each region's token store publishes a steadily incrementing counter to the others on a fixed interval, and a store that has fallen out of contact stops advancing its counter) and, for high-risk scopes, reduce cache TTL to zero and require step-up (re-authentication) for new sessions, while low-risk reads continue.

The point to make out loud: no design gives both perfect revocation and full availability during a partition. You choose where to sit per route, and you choose it before the incident.

Guidelines for SREs: when to prefer which

Two terms recur in this table: fail closed means denying a request when the system cannot get a fresh answer (safer, less available), and fail open means allowing it anyway and accepting some risk (more available, less safe).

Monitoring that proves the choice is working: introspection p99 and error rate per region, cache hit rate, revocation propagation lag per region (with a synthetic revoked-token canary), and the count of requests served under fail-open.

Trade-offs and pitfalls

• Introspecting at every microservice multiplies latency and load by the number of hops; do it once at the edge.
• Long cache TTLs on introspection quietly turn opaque tokens into JWTs with worse performance; keep TTLs short and deliberate.
• Putting sensitive claims in JWTs (email, roles, internal IDs) exposes them to anyone holding the token, since JWT payloads are only encoded, not encrypted.
• Forgetting key rotation for JWTs: services must refresh the JWKS and accept both old and new keys during rotation, or a rotation becomes an outage.

Direct answer

Treat the audit pipeline as three tiers joined by a durable log: a local, disk-buffered collector in every region, a partitioned append-only log (Kafka-style: Kafka is a distributed log system where writes are split into ordered partitions, each an append-only sequence that many readers can replay from the beginning) keyed by entity ID so each entity's events stay in order, and tiered storage: a hot search index for recent investigations and immutable columnar files (data stored column by column rather than row by row, which lets an analytical query read only the columns it needs and compresses far better than row-oriented storage) in write-once object storage for the seven-year archive. Integrity comes from a hash chain per log partition plus a signed checkpoint every minute, written to storage nobody (including administrators) can rewrite. Regions keep accepting events through a network partition and forward them when the link heals, because an audit system that drops events during an outage is failing exactly when it matters.

Requirements and the load you are designing for

At 100k events per second, assume an average event of 1 KiB (a JSON record with actor, action, resource, source IP and a few tags). That assumption drives everything, so say it out loud in the interview.

(2,557 days is 7 years including two leap days.) Sustained ingest is about 102 MB/s in total, roughly 34 MB/s per region if traffic splits evenly. Audit JSON compresses well in a columnar format; if you assume a 5:1 ratio the archive is about 4.5 PB, at 8:1 about 2.8 PB. Measure your own ratio on a day of real data before committing a budget, because this is the single largest cost line.

Ingestion: never lose an event at the edge

• Emit at the source with a schema. Each of the 200 services writes events through a small shared library that stamps an event ID (a UUID, used later for de-duplication), the entity ID (user, tenant, resource), a producer sequence number, and a hybrid logical clock timestamp (a timestamp that combines wall-clock time with a counter so events stay causally ordered even when clocks drift slightly).
• Local agent with a disk buffer. A node-level collector, any of the standard off-the-shelf log-shipping agents (Vector, Fluent Bit or an OpenTelemetry Collector; the choice between them matters far less than getting the "buffer to disk before acknowledging" behaviour right) accepts events over a local socket and persists them to disk before acknowledging. If the regional log is unreachable, the agent keeps buffering.
• At-least-once, then de-duplicate. At-least-once delivery means an event may arrive twice but is never silently dropped. The storage layer drops duplicates by event ID. Exactly-once across 200 services and three regions is not worth its coordination cost here; idempotent storage gives the same observable result.

Transport: a partitioned log per region

Each region runs its own log cluster. The partition key is the entity ID, so all events for user:42 land on one partition and are stored in the order the partition received them. That is what gives "per-entity ordered history" cheaply. Two caveats worth naming:

• Ordering is per entity per region. If user:42 acts in the US and EU in the same second, merge the two streams at query time by hybrid logical clock timestamp, then by region, then by sequence. Do not pretend a global total order exists; build one only if a regulator demands it, and then with a sequencer per entity home region.
• A hot entity (a service account doing 5,000 actions per second) can overload one partition. Detect it and sub-partition that entity by entity_id + bucket, recording the bucket so the query layer knows to merge.

Cross-region replication (for example Kafka MirrorMaker 2, a Kafka component that continuously copies topics from one cluster to another) copies each regional log to a second region asynchronously so a regional loss does not lose history.

flowchart LR
  S[200 services] --> A[Node agent with disk buffer]
  A --> K[Regional log keyed by entity ID]
  K --> C[Chainer: hash chain + minute checkpoint]
  C --> W[(WORM object storage, Parquet)]
  C --> I[(Hot search index, 90 days)]
  C --> N[Signing service, HSM-backed key]
  K -. async replication .-> R[Log in second region]

Storage choices

Reconciling the log-tier number with cross-region replication. "186 TB" (8.85 TB/day x 7 days x replication factor 3, rounded) is the storage for each region's primary log only, summed across all three regions. The cross-region replica described above (MirrorMaker copying each region's log to a second region) is a separate, additional set of topics living in another region's cluster, and if that replica is kept at the same replication factor 3 and the same 7-day retention as the primary, it costs the same again: another 8.85 TB/day x 7 days x 3 is about another 186 TB. So the log tier's real fleet-wide total is closer to 372 TB, not 186 TB. If that is too much, the cheaper option is to hold the replica at replication factor 1 (it is a DR copy of data the primary region also still has, not the only copy) or a shorter retention than 7 days, since its job is "replace a lost region," not "serve replays."

WORM (write once, read many) means the storage rejects overwrites and deletes until a retention date passes. On AWS this is S3 Object Lock in compliance mode, in which no user, including the account root user, can delete or shorten retention on a locked object version. That property is what makes the archive defensible, not just convenient.

Cryptographic integrity controls

The threat is an insider or attacker who edits or deletes audit rows to hide what they did. Controls, in order:

1. Hash chain per partition. A single "chainer" consumer per partition computes h_n = SHA-256(h_{n-1} || canonical_event_n), where || means concatenation (joining the two values end to end into one string before hashing) and canonical_event_n is the event serialised the same fixed way every time (so the same event always hashes to the same bytes), and stores h_n with the event. Changing, removing or reordering any event breaks every later hash.
2. Signed checkpoints. Every minute the chainer builds a Merkle root (a single hash summarising all events in that minute, with short proofs that any one event is included) over the partition heads and has it signed by a key held in an HSM (hardware security module) or a cloud KMS (key management service). The chainer never sees the private key; it can only request signatures, and every request is itself logged.
3. Anchor outside the blast radius. Write each signed checkpoint to the WORM bucket in a separate account owned by the security team, and optionally publish the daily root to an external party (a partner or a public timestamping service). Now tampering requires compromising the pipeline, the signing key and the separate account.
4. Verify continuously. A verifier job re-reads a random sample of archived files daily, recomputes the chain and compares it with the signed checkpoints. Integrity nobody checks is only a claim.

Resilience to partitions

A region that loses its link keeps working because every hop buffers: agent disk, then the regional log. Size the buffers for the longest outage you will tolerate. At about 34 MB/s per region, a 24-hour isolation means roughly 2.9 TB of backlog, which fits comfortably in a 7-day log. When the link heals, replication catches up; the hash chains are per region, so no cross-region coordination is needed to keep integrity.

How SREs operate restores and audit queries

• Audit query path. Investigators query the hot tier by entity and time range. For older data, the query engine prunes Parquet partitions by date and entity bucket, so a query for one user over one month reads a small slice, not petabytes. Every audit query is itself audit-logged (who looked at whose history).
• Restores. A restore is "re-hydrate a time range from the archive into a scratch index". The runbook: pick the range, copy Parquet files from WORM storage (reads are allowed; the lock only blocks writes), verify the hash chain and checkpoint signatures before loading, then index. Refuse to load a range that fails verification and page the security on-call.
• Drills. Run a restore drill monthly on a random historical day, and track two SLOs (service-level objectives): end-to-end ingest lag (for example p99, the 99th percentile, under 60 seconds from emit to hot index) and restore time for one day of one region.
• Key and retention operations. Rotate the checkpoint signing key yearly and keep old public keys for the full seven years, because a 2026 checkpoint must still verify in 2033. Expired data is removed by a lifecycle policy (an automated storage rule that deletes or moves objects once they cross an age threshold, so nobody has to remember to do it) after the lock lapses, never by hand, and legal holds override deletion.

Trade-offs and pitfalls

• Per-entity order vs throughput. Keying by entity costs you hot-partition handling. Keying by random ID balances load but forces a sort for every entity query; for an audit system, the entity key is the right default.
• Signing every event is tempting and wasteful: 100k signatures per second through an HSM is expensive and slow. Chain and sign checkpoints instead; tamper evidence is equivalent.
• Hot index is not the record. The index can be rebuilt and may be edited by operators; only the WORM archive plus signed checkpoints is authoritative. Say this explicitly or auditors will question every search result.
• Common miss: forgetting that the pipeline's own admin actions (retention changes, key rotation, replays) are audit events too, and must flow through the same pipeline.

Direct answer

Rotate in three phases driven by two numbers: C, the longest time a verifier can go without refreshing its key set, and T, the maximum token lifetime. First publish the new public key alongside the old one and wait at least C until telemetry shows verifiers have it. Then switch signing to the new key. Then keep the old public key published for at least T (plus clock skew) so every token signed before the switch can still verify, and only then remove it and destroy its private key. Every token carries a kid (key ID) so verifiers pick the right key, and the whole rollout is watched through per-kid verification metrics, with rollback possible at every phase until the old private key is destroyed.

Terms

• Asymmetric signing: the issuer signs with a private key; verifiers check with the public key. Only the issuer needs protecting; verifiers can be many.
• JWKS (JSON Web Key Set): a published JSON document listing current public keys, each with a kid. Tokens name the kid in their header (RFC 7517 defines the format).
• Grace window: the bounded period after rotation when tokens signed by the old key still verify.

Setup and numbers

• 1,000 services, assume 20 instances each: 20,000 verifiers.
• 200,000 tokens in flight, maximum lifetime T = 60 minutes, so the issuer mints about 200,000 / 3,600 s, about 56 tokens per second.
• Verifiers cache the JWKS for C = 15 minutes. Refresh load is 20,000 / 900 s, about 22 requests per second: trivial, especially if the JWKS is a static file on a CDN (content delivery network).
• Allowed clock skew between issuer and verifiers: s = 5 minutes.

The protocol

sequenceDiagram
  participant I as Issuer
  participant J as JWKS endpoint
  participant V as Verifiers
  I->>J: Phase 1 publish K1 and K2, still sign with K1
  V->>J: refresh within C
  Note over V: telemetry shows K2 cached on most verifiers
  I->>I: Phase 2 sign new tokens with K2
  Note over V: K1 tokens still verify, grace window T plus skew
  I->>J: Phase 3 remove K1
  V->>J: refresh, K1 gone
  I->>I: destroy K1 private key

Phase 1: pre-publish (duration at least C, ends on telemetry).
The JWKS now lists old key K1 and new key K2; the issuer still signs only with K1. Nothing verifies against K2 yet, so this phase cannot break anything. It ends when (a) at least C = 15 minutes have passed and (b) at least 99.9% of verifiers report K2 in their cached key set. Verifiers export a gauge (a metric that reports the current value, unlike a counter that only ever goes up) of the kid values they hold, so (b) is measured, not assumed.

Phase 2: switch signing (grace window at least T + s).
The issuer signs every new token with K2. A verifier that somehow missed the refresh sees an unknown kid and must refetch the JWKS immediately rather than reject. Verifiers accept K1 and K2 throughout. Old K1 tokens drain naturally: the last one issued just before the switch, say at 09:00:00, has exp (expiry) set to 10:00:00, 60 minutes later. Now add the skew: a verifier whose clock runs up to 5 minutes slow could, at the issuer's real time of 10:03, still read its own clock as 09:58 and correctly consider that token not yet expired. If K1 had already been removed from the JWKS at 10:00, that verifier would fail to find the key for a token it is right to still trust. So K1 must stay published for at least T + s, 65 minutes, not just T.

Phase 3: retire.
Remove K1 from the JWKS. Within another C = 15 minutes every verifier has dropped it. Confirm that K1 verification counts are zero, then destroy the K1 private key.

Minimum end-to-end time: 15 + 65 + 15 = 95 minutes. In practice run it over a day (pre-publish for hours, not minutes) because extra waiting costs nothing and absorbs stragglers.

How verifiers must behave

1. Select the key by kid; never try every key (that turns an unknown-kid attack into many signature checks).
2. On unknown kid, refetch at most once per 30 seconds per instance. Rate limiting matters: an attacker spraying tokens with random kid values could otherwise make 20,000 instances hammer the endpoint. With the limit, the worst case is 20,000 / 30, about 667 requests per second, which a CDN-served static file absorbs.
3. If the JWKS endpoint is unreachable, keep using the cached set (with an alert) rather than failing all verification.
4. Pin the algorithm per key (from the JWKS entry), never from the token header, to block algorithm-substitution attacks: a forged token can set its own header to claim any algorithm it likes: the classic version names a symmetric algorithm (HMAC) instead of the real asymmetric one, then "signs" the token using the issuer's own public key, which anyone can read from the JWKS, as if it were a shared HMAC secret. A verifier that blindly trusts whatever algorithm the header names will hash the token with that same public value and get a matching signature, accepting a forgery. Pinning the algorithm to what the JWKS says that kid actually uses closes that off.

Monitoring

Rollback

• During phase 1: remove K2 from the JWKS. No token was ever signed with it, so nothing breaks.
• During phase 2: switch signing back to K1, which is still published and whose private key still exists. Tokens already signed with K2 remain valid until they expire because K2 stays published. This is why the old private key is not destroyed until phase 3 has succeeded.
• After phase 3: rolling back means a new rotation, not a revert.

Compromised key: a different protocol

If K1 is rotated because it leaked, the grace window is the vulnerability: an attacker can mint K1 tokens for as long as verifiers accept K1. So remove K1 from the JWKS immediately, push a "refresh now" signal to verifiers instead of waiting for C, and accept that all 200,000 in-flight tokens become invalid. That forces re-authentication: if clients retry within 5 minutes, the issuer must absorb 200,000 / 300 s, about 667 issuances per second, about 12 times the steady 56 per second. Size the issuer, and give clients jittered retries (retries with a randomised delay, so thousands of clients invalidated at once don't all retry in the same instant and overwhelm the issuer), for that event before you need it.

Trade-offs and pitfalls

• Shorter T shortens every rotation and every compromise window, but raises issuance load and dependence on the issuer.
• Longer C reduces endpoint load but lengthens phase 1 and delays compromise response; push-based refresh (the issuer actively notifying verifiers the moment a key changes, instead of verifiers only finding out whenever their own polling interval C next comes around) gives you both.
• Double-signing (issuing each token under both keys) is only needed when verifiers cannot fetch key sets dynamically; with kid-based JWKS it adds token size for no gain.
• Common wrong turn: switching signing and publishing the new key in one step. For up to C minutes, some verifiers reject every new token.