Zero Trust, Segmentation, and Service-to-Service Security Questions

Direct answer

Design privileged access as a request, approve, elevate, record, and expire pipeline: nobody holds standing privileged access. They request exactly the scope needed for a task, get it approved, receive a time-limited grant with the session recorded, and the grant is automatically revoked when the window ends, across both cloud and on-premises resources, with a separate emergency path for when the normal flow itself is unavailable.

Structured elaboration

• Approval workflow: the requester specifies the resource, the specific privileged action, and a business justification tied to a ticket. The approver should generally not be the requester, separation of duties, and for lower-risk, well-understood requests, approval can be automated against policy, keeping human review focused on higher-risk or unusual cases.
• Time-limited elevation: once approved, the system grants a credential, a temporary cloud role or an on-premises privileged group membership, with an explicit, short expiration matched to the task, minutes to a few hours for most operational work, not days.
• Session recording: while the elevated credential is active, the session is recorded, keystrokes or commands for a shell session, API call logs for programmatic access, so what was actually done with the privilege is reviewable independent of what was merely authorized.
• Emergency break-glass path: a separate, tightly controlled mechanism for when the normal approval system itself is unavailable, such as during a major incident affecting the identity provider, or when there is no time to wait for approval. Break-glass credentials are typically pre-provisioned and sealed, so any use immediately triggers a high-priority alert, and every use is treated as a mandatory post-incident review regardless of outcome, precisely because it bypasses the normal controls.
• Automated deprovisioning across cloud and on-premises: the expiration has to actually revoke access, not just remind someone to do it manually. For cloud resources this generally means automatic role or session expiry native to the cloud identity system; for on-premises resources, such as an Active Directory group membership or a local admin grant, it means an automated job or workflow engine that removes the grant at the exact expiration time, rather than depending on a human running a cleanup script.

Worked example

An engineer needs temporary root access to an on-premises database server to apply an emergency patch. She requests root access to that specific server for two hours, citing the incident ticket. An on-call lead approves it within minutes (the approval workflow). The system grants a temporary local admin credential valid for two hours and starts session recording on that host for the duration (time-limited elevation and session recording). If the identity system itself were down during a wider outage, she would instead use a break-glass credential from a sealed vault, whose use immediately pages the security team regardless of whether the incident is legitimate (the emergency path). At the two-hour mark, an automated on-premises workflow removes her from the local admin group whether or not she has finished; if she needs more time, she submits a new request rather than the grant silently persisting (automated deprovisioning).

Trade-offs and pitfalls

The biggest operational risk is inconsistent enforcement of expiration between cloud and on-premises. Cloud identity systems usually expire credentials natively and reliably, but on-premises deprovisioning often depends on a custom automation job that can silently fail, leaving stale privileged access behind exactly where it is hardest to notice; that job's own health needs to be monitored as carefully as the privileged access it manages.

Treat this as a staged, reversible traffic migration, not a single cutover event: run both access paths in parallel, shift a small amount of real traffic to the new path first, watch specific health signals, and only fully cut over once those signals hold steady, with an explicit, pre-agreed rollback trigger rather than a judgment call made under pressure during the cutover itself.

Staging

Before touching real users, validate the Zero Trust Network Access (ZTNA) path end to end in a non-production or shadow-traffic mode, confirming identity-aware access control, performance, and every legitimate access pattern the VPN currently supports, including less obvious ones like a scheduled batch job or a support tool that authenticates differently than an interactive user.

Canary traffic

Shift a small, low-risk slice of real traffic to the ZTNA path first, a specific user group or access pattern, while the majority stays on the VPN, so a problem affects a bounded, known population rather than everyone at once.

Monitoring indicators that would make you halt

Authentication failure rate on the new path rising above its normal baseline; latency or error rate on the application rising for canary traffic relative to a VPN-routed control group; a spike in access-denied events for users who should legitimately have access, a sign the policy migration missed an entitlement; and any rise in support-ticket volume tied to the canary population.

Rollback criteria

Define observable thresholds for each of the above BEFORE the cutover starts, for example "canary authentication failure rate clearly exceeding its own recent baseline for more than a few minutes" or "any single canary user unable to complete a critical workflow," rather than "we'll know it if we see it," so the decision to roll back is fast and doesn't require re-litigating what counts as bad in the middle of an incident.

Coordination with the owning team

The application team, not just the security or network team running the migration, needs to be on the call during the cutover window, since they can quickly tell whether an odd signal is a real regression or an unrelated, coincidental issue, and they own communicating with end users if something needs to roll back.

Worked example

Cutting over a revenue-generating checkout-support tool from VPN to ZTNA: route a small percentage of eligible users, chosen through an existing feature-flag mechanism rather than by network topology so it works cleanly with ZTNA's identity-based routing, to the ZTNA path for one business day while the rest stay on VPN as a control group. Compare authentication failure rate and support-ticket rate between the two groups over that day, and only expand the canary population, then eventually cut over everyone, once it shows no elevated failure or ticket rate relative to the VPN control group across the full comparison window, reducing risk at each step rather than committing the whole user base at once.

Trade-offs and pitfalls

Running both paths in parallel costs real engineering and operational effort, maintaining two access mechanisms at once, and it's tempting to shorten that window to save cost; resist that, since the parallel period is exactly what makes the migration reversible without user-visible downtime. Zero downtime for the cutover itself also doesn't mean zero risk overall: the parallel period has its own ongoing risk, since two systems' worth of access-control surface, VPN credentials and ZTNA policy, are both live at once, which is itself something to monitor for drift or a forgotten access path left open on the old system after the cutover completes.

Anchor the whole design on one identity abstraction that survives a hop between clouds, keep policy authoring centralized while enforcement stays resilient to a control-plane outage, and route every cross-cloud call through mutually authenticated gateways that carry a tenant claim end to end.

Tenant mapping

Mint a stable, globally unique tenant ID once, at signup, and propagate it as a signed claim through every hop and every cloud, rather than re-deriving tenant identity separately in each cloud's native identity system. AWS IAM (Identity and Access Management) and a different cloud's native identity service don't share semantics, so a cloud-agnostic workload identity standard, such as SPIFFE (Secure Production Identity Framework For Everyone), issuing a per-tenant, per-service identity keeps the mapping consistent regardless of which cloud a service happens to run in.

Control-plane versus data-plane separation

The control plane, where policy is authored and identity is issued (for example, a central service-mesh control plane or a policy engine like OPA, Open Policy Agent, holding the rules), can stay centralized since it isn't customer data. The data plane, the actual proxies and gateways moving traffic in each region and cloud, must keep enforcing the last known-good policy even through a brief loss of connectivity to the control plane (fail-static, not fail-open). For GDPR (the EU's General Data Protection Regulation), this split matters because data-plane telemetry that contains personal data has to respect residency rules on where it's processed and stored, even though the policy definitions governing it can live centrally.

Centralized compliance logging

Ship audit and access logs from every cloud into one compliance-grade log store, ideally immutable (write-once, read-many), retained per PCI-DSS's (Payment Card Industry Data Security Standard) required window, and tagged consistently with the same tenant ID everywhere. That consistent tagging is what makes a GDPR data-subject request resolvable: querying by tenant ID across the single compliance store finds every cloud and service touched, which no single cloud's own logs would show on their own.

Cross-cloud secure communication

Cross-cloud calls terminate at a per-cloud egress and ingress gateway, mutually authenticated with mTLS (mutual TLS) against a shared trust root, or cross-signed intermediate certificate authorities so an identity issued in one cloud verifies cleanly in the other, tunneled over a private interconnect between the clouds rather than the public internet where that option exists.

Worked example

Tenant "Acme" (tenant_id acme-042) runs its order-service in one cloud and its analytics-service in another cloud's EU region, to keep EU customer data in-region. A call from order-service to analytics-service crosses the cloud boundary through the paired gateways: the origin-side gateway attaches Acme's signed tenant claim and forwards over the private interconnect, and the destination-side gateway verifies that claim's signature against the shared trust root before admitting the call. Both gateways log the call tagged acme-042 to the shared compliance store, so a later GDPR erasure request for one of Acme's customers can be resolved by querying that one store by tenant ID, instead of manually correlating separate logs from two different clouds.

Trade-offs and pitfalls

Centralizing identity and policy authoring creates a new critical dependency: WAN reachability to the control plane, which is why the data planes must be able to keep enforcing cached policy through a brief outage. Cross-signing certificate authorities between two different cloud-native identity systems is genuinely hard to operate and is often the real blocker multicloud migrations run into. PCI-DSS and GDPR pull in different directions on retention: PCI wants specific access logs kept for a defined window, GDPR wants personal data minimized and not retained longer than necessary, so the compliance log store needs separate retention rules for PCI-relevant records and personal-data records rather than one blanket policy.

Direct answer

At millions of authorization checks per second with a single-digit-millisecond latency budget, a per-request network round trip to a central decision point is off the table. The architecture has to push policy evaluation out to where the request already is, in-process or same-host, and treat the central policy decision point (PDP) as a policy distributor and source of truth rather than a per-request decision maker.

Structured elaboration

• Caching strategy: cache the compiled policy itself at each enforcement point, not just individual decisions, so every request is evaluated fully in-process, sub-millisecond, against a locally held policy set. No network hop sits on the request's critical path at all.
• Policy distribution: the central PDP computes and pushes policy changes out to every policy enforcement point (PEP) through a publish-subscribe or streaming mechanism rather than PEPs polling for updates, so updates propagate without adding per-request cost. Distribution should send deltas once a PEP already has a baseline, not the full policy set every time, to keep propagation fast as the policy set grows.
• Consistency trade-offs: this is an eventually-consistent system by design, there will always be a short window where different PEPs run slightly different policy versions during a rollout. You choose the size of that window against how urgently a change needs to take effect everywhere, and for something as urgent as revoking a compromised credential, a separate, higher-priority fast path (or a short-lived local cache specifically for revocation status) is usually layered on top of the bulk distribution mechanism, precisely because eventual consistency is not acceptable for that one case.
• Fault tolerance: each PEP keeps serving decisions from its last-known-good local policy if it loses contact with the distribution mechanism, failing static rather than failing open or failing entirely, and the distribution mechanism itself needs redundancy so one central PDP instance being down only delays the next update rather than stopping policy from ever reaching PEPs.
• Rolling out updates without violating the latency budget: since evaluation already happens locally and in-process, a policy update itself never touches per-request latency. The risk is only in how the update is applied at each PEP, so it should be deployed as an atomic swap to a newly compiled ruleset, not a live in-place edit of the structure currently being evaluated concurrently.

Worked example

As a capacity-planning illustration, not a benchmark claim, suppose the system must sustain 2,000,000 authorization checks per second in aggregate. If policy evaluation happens locally in-process at each of 500 service instances, each instance only needs to sustain 2,000,000 divided by 500, or 4,000 checks per second locally, comfortably within what an in-memory rule evaluation does in well under a millisecond, with no network call involved at all. Contrast that with routing all 2,000,000 checks per second to a central PDP cluster: even a well-provisioned cluster doing 50,000 evaluations per second per node would need 2,000,000 divided by 50,000, or 40 PDP replicas, just to keep up, and every one of those checks would still pay a network round trip, exactly the latency and infrastructure cost the local-cache design avoids.

Trade-offs and pitfalls

The biggest mistake is treating "a central PDP" and "a central point of decision for every request" as the same thing; at this scale they must not be. The second is under-investing in the revocation fast path and discovering during an incident that a compromised credential's access takes as long to stop working everywhere as your slowest, most conservative policy-propagation setting.

Direct answer

That claim does not hold up: zero trust reduces the frequency and blast radius of breaches, it does not eliminate them, because it still depends on identities, credentials, and policy that can themselves be compromised or simply wrong. If an attacker obtains a legitimate, currently-valid identity, a phished token, a stolen service-account key, a compromised build pipeline, every zero-trust check will honor that identity exactly as it should, because cryptographically it is authorized.

Structured elaboration

Where the gaps remain:

• Identity-based access is only as strong as identity issuance and lifecycle management. Credential theft, session-token replay, or a compromised identity provider defeats it at the root, since everything downstream trusts that identity.
• Mutual authentication between services proves which service is talking, not that the service's logic or the human behind a request is behaving correctly. A legitimate service with a compromised dependency can still make destructive calls using its own valid identity.
• Policy enforcement points are only correct if the policy behind them is complete and current. A gap nobody thought to write, an overly broad default scope, a stale rule left over from an old integration, is not something the architecture closes automatically; a human still has to author the policy correctly, and human authoring is fallible.
• Zero trust also does not protect against fully authorized insider misuse or a supply-chain compromise inside code the identity is entitled to run: the request looks legitimate at every checkpoint because, by the rules of the system, it is.

Migration friction, moving from a permissive environment to zero trust:

• Developer friction: engineers used to broad, standing access (a shared service account with wide database permissions, SSH (secure shell) access anywhere) now hit explicit denials for previously-invisible dependencies, which slows delivery until the missing flows are identified and granted. The common failure mode is routing around the friction with overly broad, "temporary" grants that never get revoked, quietly recreating the old permissive model.
• Trust bootstrapping: early in a migration, new identity and policy infrastructure has to be trusted by systems that have no independent way yet to verify it. A new policy decision point typically has to run in shadow mode against production traffic before anyone is comfortable making it the sole authority, and the very first workloads onboarded often have nothing established yet to authenticate their own dependencies against, which is why pilots usually start with a small, self-contained set of services and a manually managed root of trust before automation exists.

Worked example

A payroll service uses short-lived, cryptographically issued service identities, and every request is authorized per call by a policy engine, a textbook zero-trust setup. An attacker compromises the build pipeline's deployment credentials, a supply-chain attack rather than a network attack, and pushes a malicious build that, once deployed, carries the payroll service's own legitimate identity. Every request that malicious build makes to the database is mutually authenticated, matches the policy that the payroll service is supposed to read and write payroll records, and passes every zero-trust check, because the compromise happened upstream of all of them, in the build pipeline, not in the network or the request path. Zero trust here limits what the attacker can do, only what the payroll service's identity is scoped to touch, but it does not prevent the breach, that requires supply-chain controls entirely outside the access model.

Trade-offs and pitfalls

The common wrong turn is treating zero trust as a project with an end state, "we're zero trust now, we're safe," rather than one layer of defense-in-depth that still needs supply-chain security, credential hygiene, detection and response, and correctly authored policy behind it. The friction and bootstrapping costs above are real and frequently underestimated in migration timelines.