Identity, Authentication, and Access Management Questions

Direct answer

Give each pipeline stage its own narrowly scoped, short-lived credential, minted just before that stage runs and expiring shortly after, obtained through workload identity federation rather than a stored static secret; inject any further secrets into the job's runtime environment, never its source or logs, through a dedicated secrets manager or key-management service; and treat the build artifact itself as something that must be cryptographically signed and later verified before deployment, not merely produced.

Structured elaboration

Ephemeral credentials per job. Rather than storing a long-lived cloud credential inside the CI/CD (continuous integration and continuous delivery) system's own secret store, have the build agent present its pipeline's own OpenID Connect (OIDC) identity token, which most modern CI/CD platforms can mint fresh per job, to a workload identity federation endpoint, which exchanges it for a short-lived, narrowly scoped cloud credential valid only for that specific stage, typically for minutes. Scope each credential to exactly what its stage needs: a build stage gets read access to source and write access to an artifact repository, a deploy stage gets write access to the target environment and nothing else, so a compromised build stage can never also deploy to production. This is also where developer single sign-on (SSO) and cross-account role assumption fit together: a human developer triggering a pipeline authenticates through the organization's own SSO, while the pipeline's execution identity separately assumes a distinct, narrowly scoped role per target account, for example a separate role for a build account, a staging account, and a production account, so a developer's own broad SSO identity is never what actually executes a cloud action; the pipeline's own per-stage assumed role is.

Injecting secrets without leaking them. Pull any additional secrets, a database password, a third-party API key, from a dedicated secrets manager or a key-management-integrated vault at job start, inject them as environment variables or mounted files scoped to that job's own process, and never let them reach build logs. Most CI/CD platforms automatically mask values sourced through their own secret-reference mechanism, but that masking only covers the original value; it does nothing for a script that prints a transformed or derived copy of the same secret, which is the common, unglamorous way a properly masked secret leaks anyway. Where the organization already runs a secrets manager such as HashiCorp Vault, or a cloud key-management service (KMS) for its encryption keys, integrate the pipeline against that same system rather than standing up a second, parallel secret store, so there is exactly one place secrets are issued, rotated, and audited from.

Signing and verifying build artifacts. Sign every build artifact, a container image or a package, with a key the build stage alone can access, ideally itself short-lived or tied to that specific pipeline run's identity rather than a long-lived signing key shared across every build, and attach the resulting signature plus provenance metadata, which source commit, which pipeline run, which build environment produced it, as an attestation. The deploy stage verifies both the signature and the attestation before deploying anything, refusing an artifact that wasn't produced by this exact pipeline or whose signature doesn't match, which closes the gap where a malicious or simply mismatched artifact could otherwise be substituted between the build and deploy stages.

Preventing credential leakage. The ephemeral-credential design above is itself the primary defense, since a credential scoped to one stage and a few minutes has almost nothing left to leak once it expires; scrubbing build logs for anything resembling a token or key pattern is a reasonable second layer; and treating the build environment itself as untrusted between runs, tearing it down and recreating it fresh for each job rather than reusing a long-lived build machine, prevents a credential from a previous run lingering on disk into the next one.

Approval gates and commit-signature verification. Require any deployment to a production-target stage to pass through an explicit approval gate that checks a policy condition, for example requiring the triggering commit to carry a signature from a verified, authorized developer key, confirming the change actually originated from someone authorized rather than an unsigned or spoofed commit, before the production-scoped ephemeral credential is ever issued. The approval gate and the credential-issuance step should be the same control point, not two independent checks that could disagree: a failed approval should mean the credential is never minted at all, not merely that a warning gets logged somewhere alongside it.

flowchart LR
  Dev["Developer (SSO login)"] --> Trigger["Pipeline trigger (signed commit)"]
  Trigger --> Build["Build stage: ephemeral cred, source read + artifact write"]
  Build --> Sign["Sign artifact + attestation"]
  Sign --> Gate["Approval gate: verify commit signature"]
  Gate --> Deploy["Deploy stage: ephemeral cred, target-account write only"]
  Vault["Secrets manager / KMS"] -. injects secrets .-> Build
  Vault -. injects secrets .-> Deploy

Worked example

A container-image pipeline: the build stage's job token is exchanged, via workload identity federation, for a credential scoped to read the source repository and push to the artifact registry, valid 15 minutes. Once the image builds, the pipeline signs it with a key scoped to that specific run and attaches provenance attesting the source commit and build environment. Before deployment, the approval gate checks that the triggering commit carries a verified signature from an authorized developer; if it does not, the pipeline halts and no production credential is ever requested. On approval, the deploy stage's own job token is exchanged for a separate, narrower credential scoped only to the production account's deployment role, which first verifies the image's signature and attestation match this exact pipeline run before pulling and deploying it.

Trade-offs and pitfalls

The most damaging pitfall is a build script that correctly fetches a secret through the platform's masking mechanism, then prints a transformed or derived version of it, a base64-encoded copy, a substring, that the masking doesn't recognize and leaks into logs anyway; masking helps, but it is not a substitute for disciplined script behavior. A second is granting the deploy stage a broad, standing credential "to keep things simple," which defeats the entire point of stage-scoped ephemeral credentials the moment any single stage is compromised. A third is treating artifact signing as a checkbox without actually verifying the signature and attestation at deploy time, which leaves signing providing an audit trail after an incident rather than any actual protection before one. The core trade-off: this design carries real setup cost, workload identity federation configuration per environment, a signing and attestation pipeline, an approval gate wired to commit-signature verification, which is proportionate for a production deployment pipeline but likely excessive for a low-stakes internal tool, where a simpler secrets-manager-only approach may be the right, deliberately less rigorous choice.

Direct answer

For most organizations, the right default is active-active: every region runs a full, locally-writable copy of the identity provider (IdP, the service that authenticates users and issues tokens) and directory service, with each identity's canonical record "homed" in one region to avoid write conflicts, and global DNS routing sending each client to its nearest healthy region. Active-passive (one primary region takes all writes, others are cold or read-only standbys) is only the better choice when the directory cannot tolerate any risk of a stale or conflicting write, such as a single break-glass emergency-access store, and a slower, human-verified failover is acceptable. The two hard constraints in this question, sub-5-second local authentication and surviving a full region loss, both point toward active-active plus stateless session validation, because a passive standby cannot serve local reads while it is cold and its promotion time directly becomes your outage window.

Structured elaboration

The topology below is the shape the rest of this answer argues for: three regions, each a full read/write replica, reached through geo/latency-based DNS, with a thin cross-region layer carrying only home-region writes and a replicated revocation list (explained in the sections that follow).

flowchart TB
  Client[Client]
  DNS[Geo/latency-based DNS]
  Client --> DNS
  DNS --> R1
  DNS --> R2
  DNS --> R3
  subgraph R1[us-east region]
    IdP1[IdP + directory replica]
  end
  subgraph R2[eu-west region]
    IdP2[IdP + directory replica]
  end
  subgraph R3[ap-southeast region]
    IdP3[IdP + directory replica]
  end
  IdP1 <-.->|home-region writes + minimized cross-region replication| IdP2
  IdP2 <-.->|home-region writes + minimized cross-region replication| IdP3
  IdP1 <-.->|home-region writes + minimized cross-region replication| IdP3
  RevList[(Replicated revocation list)]
  IdP1 --- RevList
  IdP2 --- RevList
  IdP3 --- RevList

Active-active vs. active-passive. Active-active means two or more regions each accept live authentication traffic and directory writes simultaneously. To avoid the classic multi-master problem (two regions independently updating the same user record and disagreeing), the practical pattern is "multi-master infrastructure, single-writer-per-record": each identity has a home region that owns writes to that specific record (password changes, attribute updates), while every region can serve reads and validate tokens for any identity. This gets you local low-latency authentication everywhere without needing a general conflict-resolution algorithm for the common case. Active-passive instead designates one region as the sole writer; other regions replicate asynchronously and only start accepting writes after a manual or automated promotion. Its main advantage is a simpler consistency story (there is only ever one writer, so there is no reconciliation logic to get wrong); its cost is that failover has a real recovery time (the time to detect the primary is down and promote a replica, often called RTO, recovery time objective), during which no new writes anywhere in the world are possible, and any user whose local replica lagged the primary may briefly authenticate against stale data.

Consistency trade-offs. This is a direct instance of the CAP trade-off (a system split across a network Partition must choose between Consistency and Availability for the affected data): when the link between regions is down, active-active must decide whether to keep serving local authentication with a possibly-stale replica (available, eventually consistent) or to refuse requests until the replica is confirmed current (consistent, less available). For identity systems specifically, the right answer is not the same for every write:

• Authentication reads (does this password/hash match, what groups is this user in) are the hot path and should be served locally with bounded staleness, typically single-digit seconds. A local read that is a few seconds stale is a rounding error against a 5-second latency budget and is what makes the budget achievable at all.
• Security-critical revocations (disable an account, kill a session, revoke a privilege) are the one class of write that should propagate synchronously to at least a quorum of regions, or be enforced through a separately-replicated, low-latency revocation/negative cache, precisely because an eventually-consistent disable command creates a window where a compromised account still authenticates successfully somewhere in the world.

Session state handling. There are two designs. A stateful session store (a session ID that maps to server-side state) must itself be replicated multi-region, which re-imports the entire consistency problem one layer up and adds a network hop to every request. A stateless session (a signed token, containing identity claims and an expiry, that any region can verify locally using a shared or per-region-replicated signing key) avoids that hop entirely: any region can validate any token issued anywhere, including one issued moments before the client's home region went down. The remaining gap is revocation: a stateless token is valid until it expires even if the underlying account was just disabled. The fix is to pair stateless tokens with a small, fast-replicating revocation list (a negative cache keyed by token ID or user ID) so the common case (99%+ of requests) is a local, stateless verification, and only the rare revoked case needs the cross-region signal to have arrived.

DNS/routing strategies. Route clients to the nearest healthy region using latency-based or geo-proximity DNS routing (or an anycast IP announced identically from every region, which lets the network layer itself route to the nearest point of presence without relying on DNS caching behavior at all). Health-checked failover records remove a region from rotation automatically once it stops passing checks. The design tension is DNS time-to-live (TTL, how long resolvers are allowed to cache an answer before re-querying): a long TTL (minutes to hours) means fewer DNS queries and better client-side caching, but a dead region stays in rotation for that whole window after it fails; a short TTL (30 to 60 seconds) speeds up failover at the cost of more DNS traffic and less caching upstream, and even then, some resolvers and corporate networks ignore TTLs and cache longer, so DNS failover alone is not a hard guarantee, only a fast default path.

Data residency constraints. Some jurisdictions (the EU under GDPR, the General Data Protection Regulation, and various national data-localization laws) require that a specific person's personal data, or its authoritative copy, physically stay within that jurisdiction. This directly shapes which regions can be "home" for which identities: an EU user's canonical record must be homed in an EU region, and you cannot casually replicate the full record to every region "for availability" without violating residency. The resolution is to replicate only what cross-region authentication actually needs (a minimized identity assertion: subject ID, a few claims, a public key or hash sufficient to validate the user elsewhere) globally, while keeping the full attribute set durably stored only in the home region(s). This turns the architecture from "one global directory" into "federated regional directories plus a deliberately thin, minimized cross-region layer," which is a real cost (some data literally cannot follow the user to whichever region is fastest) but is not optional where the law applies.

Worked example

Take three regions: us-east, eu-west, ap-southeast, each running a full IdP and directory replica, active-active, with per-identity home regions (an EU-domiciled user is homed in eu-west for residency). Authentication is a local directory lookup plus a signature check, both served from the nearest region, so the within-region path (tens of milliseconds for a lookup and a cryptographic signature check) is comfortably inside the 5-second budget with wide margin even before accounting for network transit.

Now size the failover path with the parameters you would actually configure, and derive the numbers rather than assert them:

• Health checks run every 10 seconds, and a region is marked unhealthy after 2 consecutive failed checks.
• DNS record TTL is set to 30 seconds.

Detection time is bounded by (checks needed - 1) x interval + one more check to fail = 1 x 10s + 10s = 20 seconds worst case for the check itself to observe the failure twice, plus up to one more health-check interval before the monitoring system reacts, giving a detection window of roughly 20 to 30 seconds. Once the unhealthy region is pulled from the DNS answer, a resolver that cached the old answer at the worst possible moment (just before the outage) holds it for up to the full 30-second TTL before re-querying. Adding detection and propagation conservatively (worst case, not typical case) gives roughly 20 to 60 seconds before all new login attempts are routed only to healthy regions. That is your realistic recovery time for new authentications, an explicit function of the two numbers you chose (check interval, TTL), not a measured result, and it is the number to defend or tighten in a design review, not "sub-5-second," because 5 seconds is the local-latency budget for a healthy region, not the cross-region failover budget.

Sessions that were active against the now-dead region are unaffected during that whole window, because the stateless-token design means us-east or ap-southeast can validate a token the dead region issued without ever calling back to it; only brand-new logins are impacted, and only until DNS reroutes them.

Trade-offs and pitfalls

• Naive multi-master is a trap. If every region can write every attribute of every record without a home-region rule, you get silent conflict resolution (commonly last-writer-wins by timestamp), and a clock skew or a delayed replication event can un-revoke a privilege that was correctly revoked moments earlier. Single-writer-per-record is what makes active-active safe, not incidental.
• DNS TTL is a lower bound, not a guarantee. Client OS resolvers, corporate DNS forwarders, and some ISPs cache longer than the TTL you set. Treat DNS-based failover as the fast common path and pair it with client-side retry-on-failure logic (try the configured endpoint, fall back to a documented alternate) for the tail.
• Data residency can bite you at the log layer, not just the directory. Authentication logs and audit trails frequently contain the same personal data subject to residency rules as the directory record itself; a design that carefully homes directory data correctly but ships all authentication logs to one global logging region can reintroduce the same violation one layer removed.
• Active-passive is not simply "worse." It is the right, deliberate choice when correctness must dominate availability, such as a small, rarely-used break-glass identity store where a brief outage during a true regional disaster is acceptable but a split-brain (two regions both believing they are the authoritative break-glass store) is not. The pitfall is defaulting to active-passive for the whole IdP out of caution and then failing the 5-second local-latency requirement for ordinary users during any single-region slowdown, not just a full outage.

Direct answer

Implement against SCIM 2.0's two core resource types, Users and Groups (per RFC 7643 and RFC 7644, System for Cross-domain Identity Management), treat every incoming event as idempotent against an external identifier the identity provider (IdP) supplies, and treat a SCIM delete as a deactivation internally rather than an immediate hard delete, so a provisioning mistake or a bad push from the identity provider can never silently destroy an account's history.

Structured elaboration

Which SCIM endpoints. /Users creates, reads, updates, and deactivates a user resource carrying a standard schema (userName, name, emails, an active flag, and an externalId used to correlate the resource with the identity provider's own record). /Groups manages group resources, typically carrying a members list referencing user resource IDs, which is how group-membership-driven role assignment gets pushed down from the identity provider. Support filtering (GET /Users?filter=userName eq "x") since the identity provider looks up existing resources before creating new ones, and support PATCH for partial updates: RFC 7644 explicitly calls out PATCH support for Groups because a full-resource PUT to add or remove one member from a large group is wasteful.

Attribute mapping. Map identity-provider attributes to the internal user model through an explicit, centrally maintained mapping configuration, not bespoke per-integration code. Always key identity correlation on externalId, the identity provider's immutable identifier, never on userName or email, since both can change (an email changes on marriage, a username gets normalized) and using either as the join key silently orphans or duplicates the account. Store only the attributes actually consumed downstream, not the full schema by default.

Handling create, update, and delete events. Create is idempotent on externalId: a duplicate create request for an identifier that already exists should update the existing resource rather than error or create a second one. Update, whether a full PUT or a partial PATCH, applies to the resource matched by externalId; PATCH is preferred for group-membership changes at scale, since adding one member to a group of thousands via PATCH avoids retransmitting the entire membership list. Delete is intercepted and translated into a soft deactivation rather than a hard row delete, described below.

Idempotency and retry semantics. Every write must be safe to retry: sending the same request twice should produce the same end state, never a duplicate or an error, since identity providers retry on any ambiguous response such as a timeout or a 5xx status. Return the SCIM-correct status codes so the identity provider's own retry logic behaves sensibly, for example a 409 Conflict for a genuine duplicate-with-different-payload case rather than a generic server error that triggers an unbounded retry storm. A full-directory synchronization should be resumable and paginated, using SCIM's startIndex and count parameters, rather than one all-or-nothing transaction, so a failure partway through a ten-thousand-user sync doesn't force a restart from zero.

Reconciliation to correct drift. A periodic, for example nightly, diff between the identity provider's actual directory state and the locally provisioned state catches what event-driven push alone misses: a connector that silently stops firing for one customer, a dropped delete event from a network blip, or an out-of-band manual change on either side. Reconciliation should surface the diff for review before auto-correcting anything, since blindly trusting the identity provider's state as always correct can itself wrongly deprovision someone if it is actually the identity-provider-side synchronization that broke.

Safe deprovisioning. Translate a SCIM delete, or an update setting active: false, into an immediate access suspension (revoke active sessions and tokens, block new logins) while retaining the account record and its full audit history for a defined retention window, for example 90 days, before any hard delete actually happens. This protects against both an accidental delete event and a legitimate offboarding that later needs its audit trail for a post-departure investigation or a legal hold. Treating a delete event as an immediate hard delete is the single most common way this kind of integration causes real, hard-to-reverse damage.

Worked example

The identity provider sends PATCH /Users/{id} for a departing employee, external identifier hr-00456:

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    { "op": "replace", "path": "active", "value": false }
  ]
}

The internal handler: revokes all active sessions and tokens for the user matched by externalId = hr-00456 immediately, marks the account inactive, and schedules a hard delete for 90 days out rather than performing one now. A nightly reconciliation job separately diffs the identity provider's live group membership against the locally provisioned state and flags one user whose group membership drifted after a manual, out-of-band change on the identity-provider side, surfacing it for review rather than silently correcting it.

Trade-offs and pitfalls

The most damaging pitfall is honoring a SCIM delete as an immediate hard delete, which destroys the audit trail an offboarding or incident investigation may need later and cannot be undone. A close second is keying identity correlation on userName or email instead of the identity provider's externalId, which silently produces duplicate or orphaned accounts the moment either value changes upstream. Using full-resource PUT for every group-membership change at scale, instead of PATCH, adds unnecessary payload size and lock contention on large groups. The core trade-off in reconciliation design is between auto-correcting drift immediately, which fixes problems fast but risks trusting a broken identity-provider-side state, and surfacing the diff for manual approval first, which is safer but slower; for anything touching deprovisioning specifically, the safer, slower option is the right default.

Direct answer

OAuth 2.0 is an authorization framework: it lets a user grant a third-party application scoped access to an API or resource without handing over a password. On its own it has no standard concept of "who logged in," only "what access was granted." OpenID Connect (OIDC) is a thin identity layer built on top of OAuth 2.0 that adds a standardized token proving who authenticated, not just what the app can now touch. SAML (Security Assertion Markup Language) is an older, XML-based protocol built specifically for browser-based single sign-on (SSO), most often used to federate identity into enterprise web applications. All three answer "who is this, and can we trust that claim across a network boundary," but they target different client types and eras of the web.

Structured elaboration

Criteria for choosing between them:

• Building a login experience for a modern web or mobile app that also needs to call an API on the user's behalf: use OIDC. It sits on top of OAuth 2.0, so you get delegated authorization and a verified identity from the same flow.
• Only need delegated API access, with no identity concept for the calling app itself (a backend job reading a user's calendar): plain OAuth 2.0 is sufficient and simpler.
• Integrating with an enterprise's existing identity provider, and that provider or the target application only speaks SAML: you use SAML even though it is heavier to implement than a JWT-based approach, because the counterpart has no OIDC endpoint to talk to.
• Mobile deployment pushes the decision toward OIDC: SAML's browser-redirect-and-XML-post pattern is awkward inside a native app, while OIDC's authorization code flow was purpose-built for exactly that client type.
• Enterprise SSO deployment sometimes forces SAML regardless of preference: large enterprise buyers frequently standardize on a SAML identity provider for audit and compliance reasons, and the vendor's application may expose only a SAML integration point.

Worked example

Trace "Log in with Google," which demonstrates both the OAuth delegation layer and the OIDC identity layer built on top of it:

1. The user clicks "Log in with Google"; the app redirects to Google's authorization endpoint requesting scope=openid email profile.
2. Google authenticates the user through its own login screen and the user consents to the requested scopes.
3. Google redirects back to the app with a short-lived authorization code.
4. The app exchanges that code at Google's token endpoint for an ID token (the OIDC-specific artifact, a JWT) and an access token (the underlying OAuth artifact).
5. The app validates the ID token's signature and claims (iss equals Google's issuer, aud equals the app's own client id, exp has not passed) to learn who logged in, from the sub and email claims.
6. If the app also wants to read the user's Google Calendar, it uses the separate access token for that call. That is the original OAuth layer doing its job, distinct from step 5.

This split is exactly why using bare OAuth to implement login was a historical mistake, before OIDC existed: an access token alone does not certify identity (it authorizes calls to a specific API, and it isn't required to be a verifiable, self-contained token at all), so an app inspecting only an access token to decide "who is logged in" could be fooled by a token that was legitimately issued, just for a different purpose or audience. OIDC's ID token exists specifically to close that gap.

Trade-offs and pitfalls

• SAML assertions are XML-based and require careful canonicalization and signature validation. Implementing that by hand is a well-known source of signature-wrapping vulnerabilities; always use a maintained library rather than parsing and verifying the XML yourself.
• Treating an OAuth access token as proof of identity, instead of using OIDC's ID token, is the single most common protocol-selection mistake in this space; it works in testing and fails once a token issued for a different audience gets presented to the wrong service.
• Bridging protocols (a SAML-only enterprise identity provider fronting an OIDC-only application, or the reverse) is a common real integration need, but it adds an extra hop and an extra trust boundary; treat that bridge as its own design problem rather than assuming one protocol trivially substitutes for the other.

Direct answer

To hit 1,000,000 authorization checks per second, compile policies into an efficient in-process representation, for example Rego compiled to WebAssembly (Wasm), so each check never leaves the process, cache both raw attributes and full decisions with explicit freshness bounds, and scale horizontally by embedding the evaluator as a sidecar next to each calling service rather than routing every check through one centralized decision point. This design deliberately accepts bounded staleness (attributes and cached decisions can lag their source of truth by a tunable window) in exchange for the throughput a synchronous 1M-checks-per-second workload requires; correctness is then validated with a differential test harness that compares the fast path against a reference, always-correct evaluator.

Structured elaboration

Policy language

• Open Policy Agent's Rego is the strongest default: it separates policy authoring from the evaluator's implementation language, has a mature WebAssembly compilation target that turns a policy into a portable, sandboxed bytecode module embeddable directly in the evaluating process, and has first-class support for partial evaluation (below).
• Alternative: a narrower, purpose-built domain-specific language compiled directly to a native decision tree, trading Rego's generality for a smaller, more predictable per-check cost. Worth it only once profiling shows the evaluator itself, not attribute I/O, is the bottleneck.

Attribute retrieval and caching strategies

• Attributes split into two latency classes: request-local (already present on the incoming call, free) and remote (user/resource/environment attributes living in another system).
• Cache remote attributes locally on each evaluator node with a short time-to-live (TTL), plus push-invalidation for attributes where staleness has real consequences (a just-revoked project membership).
• Cache full decisions too, keyed by a canonical hash of the request's relevant attributes, since a real workload repeats the same (subject, resource-class, action) tuple often; this is the highest-leverage cache, since it skips policy evaluation entirely, not just remote attribute lookup.

Policy compilation and pre-evaluation techniques

• Compile the policy bundle ahead of time (Rego to Wasm, or a custom DSL to bytecode) rather than interpreting source text per request; a policy change is a build-and-deploy-a-new-bundle event, not a live text reload.
• Partial evaluation: where some attributes are known in advance (an endpoint where resource.type is always "invoice"), pre-specialize the compiled policy against those known values so the runtime evaluator has fewer branches per check.

Consistency vs freshness trade-offs

• Checking every attribute against a strongly consistent source of truth on every request is not achievable within a synchronous latency budget at this scale; the design deliberately trades strict consistency for bounded staleness, a decision may be computed against attributes up to the cache's TTL old.
• Make the staleness window asymmetric by attribute sensitivity: fast-changing, high-consequence attributes get push-invalidation and a near-zero TTL fallback; slow-changing, low-consequence attributes get a longer TTL, since staleness there is a minor annoyance, not a security incident.

Horizontal scaling

• Deploy the compiled evaluator as a sidecar (or in-process library) next to each calling service, not as one centralized policy decision point (PDP) cluster every request round-trips to; this removes the network hop from the hot path entirely and lets throughput scale linearly with the calling service's own replica count, a resource already scaled to meet its own traffic.
• A small central control plane distributes compiled bundles and cache-invalidation events to every sidecar, but stays off the per-request path.

flowchart LR
    Req[Authorization Check]
    Cache[(Attribute + Decision Cache)]
    Compiler[Policy Compiler]
    Bundle[(Compiled Policy Bundle)]
    PIP[Attribute Sources: PIP]
    Eval[In-Process Evaluator]

    Req --> Cache
    Cache -->|hit| Req
    Cache -->|miss| Eval
    Eval --> Bundle
    Eval --> PIP
    Compiler --> Bundle
    PIP -->|push update| Cache

Testing correctness and performance under load

• Correctness: maintain a straightforward, unoptimized reference evaluator (no caching, no partial evaluation) that is trivially correct by construction. Run the same corpus of (policy, request) pairs through both the reference evaluator and the compiled fast path and assert identical decisions; any divergence is a caching or compilation bug caught before production.
• Performance: load-test the compiled evaluator in isolation (no network) to find its true per-core ceiling, then load-test end-to-end with a realistic, non-100% attribute-cache hit rate, since a 100% hit rate in a test would hide the cost of a real cache-miss rate.

Worked example

Capacity arithmetic for reaching the 1,000,000 checks/sec target. The per-node throughput and cache-miss rate below are stated design assumptions (labeled as such), not measurements; only the arithmetic that follows is the claim:

target_checks_per_sec = 1_000_000
per_node_checks_per_sec = 50_000   # assumption: one evaluator instance, warm in-process cache
cache_miss_rate = 0.01             # assumption: 1% of checks need the shared attribute store

nodes_needed_no_headroom = target_checks_per_sec / per_node_checks_per_sec
nodes_with_headroom = nodes_needed_no_headroom + 2   # N+2 for rolling deploys / node loss
fallback_checks_per_sec = target_checks_per_sec * cache_miss_rate

print(f"target: {target_checks_per_sec:,} checks/sec")
print(f"nodes needed at {per_node_checks_per_sec:,}/sec/node, no headroom: {nodes_needed_no_headroom:.1f}")
print(f"nodes with +2 headroom: {nodes_with_headroom:.1f} -> round up to {int(nodes_with_headroom) + 1}")
print(f"at {cache_miss_rate:.0%} local-cache-miss rate, the shared attribute store must sustain: {fallback_checks_per_sec:,.0f} lookups/sec")

Output (actually run):

target: 1,000,000 checks/sec
nodes needed at 50,000/sec/node, no headroom: 20.0
nodes with +2 headroom: 22.0 -> round up to 23
at 1% local-cache-miss rate, the shared attribute store must sustain: 10,000 lookups/sec

Twenty evaluator nodes cover the raw target with no margin; twenty-three gives standard rolling-deploy headroom. The shared attribute store, the one piece of this design that is NOT embarrassingly parallel per calling service, only needs to sustain 10,000 lookups/sec at a 1% local-cache-miss rate, two orders of magnitude below the raw target, which is exactly why the local-cache-hit path, not the shared store, has to carry the bulk of the throughput.

Trade-offs and pitfalls

• The core trade-off, bounded staleness, is also the biggest risk: a revoked permission that has not propagated yet is a live security gap for the length of the TTL or propagation lag. Push-invalidation for sensitive attributes reduces, never eliminates, this window, so it must be sized and monitored, not assumed away.
• Compiling policy to Wasm or bytecode adds a build/deploy pipeline for policy changes that a live-interpreted system didn't need; a policy fix becomes "compile, differentially test, deploy a new bundle" rather than "edit a file", slower by design, but exactly what makes the hot path fast and safe.
• A decision cache keyed on a hash of "relevant" attributes is only as good as that definition: omit an attribute the policy actually depends on and the cache serves wrong decisions that look like correct hits. Differential testing must include cache-key coverage, not just evaluator-logic coverage.
• Sidecar-per-service scaling avoids a central bottleneck but multiplies operational surface: every sidecar needs bundle updates and cache warm-up, and a bad bundle rollout becomes a fleet-wide problem instead of a single service's problem.