Zero Trust, Segmentation, and Service-to-Service Security Questions

Design a layered telemetry pipeline that collects from every enforcement point, not just one, centralizes it so cross-segment activity is visible in one place, and runs detection logic that specifically looks for traffic crossing a segment boundary the policy never intended to allow.

Log and telemetry sources

Network flow logs at each segment boundary; enforcement-point decision logs, both allow and deny, from firewalls, security groups, and service mesh sidecars; DNS query logs; endpoint-level logs from hosts or workloads (especially useful once a host is flagged as involved in a cross-segment connection); identity-provider authentication logs, since unusual login patterns often precede lateral movement via stolen-but-valid credentials; and cloud or orchestrator audit logs, since a segmentation policy CHANGE is itself a common lateral-movement enabler if an attacker gets administrative access.

Collector placement

Place lightweight collectors close to each enforcement point, per segment, per cluster, per VPC, so the decision is captured at the moment and location it's actually made, rather than reconstructed later from a distance. Forward everything to a central aggregation layer, whether a full security information and event management (SIEM) pipeline or simply a centralized log store with a query layer, so an analyst or automated detection can correlate events ACROSS segments, which a purely local, per-segment view can't do.

Detection logic for lateral movement across segments

The core rule is a comparison between OBSERVED cross-segment traffic and the DECLARED policy: any connection between two segments the policy doesn't explicitly allow is, by definition, either a misconfiguration or an intrusion, and both deserve investigation. Concretely: alert on any successful connection crossing a segment boundary with no matching allow rule (this should be rare to nonexistent if enforcement and detection both work, so its mere presence is high signal); alert on a spike in DENIED cross-segment attempts from a single source (probing behavior); and alert on any segmentation policy change that widens access, correlated with who made it and whether it matches a recorded, approved change.

Worked example

The "payments" and "public-web" segments have default-deny between them except one allowed path, public-web to payments on its designated charge endpoint only. If flow logs show a connection from a public-web host to payments on a different port or destination than that one allowed path, that's a direct violation of declared policy, an unambiguous, high-priority alert, versus a denied-attempt log entry on the same pair, already blocked by the enforcement point, which is lower urgency but still worth trending, since a rising rate of denied attempts against the same target can indicate active probing before a successful breach.

Trade-offs and pitfalls

Collecting everything everywhere generates a large volume of low-value ALLOWED-traffic logs; most designs sample or reduce detail on allowed traffic while keeping full detail on denials and cross-segment activity, since that's where the signal density is highest. If the audit log for policy changes itself isn't centrally collected and protected, an attacker with control-plane access can quietly widen policy and erase the evidence, so that log source needs the same, or stronger, protection as the segmentation controls it's monitoring.

Design the enforcement point (the PEP, Policy Enforcement Point, meaning the sidecar or gateway that actually allows or denies each call) to fail static, never fail open: it keeps honoring the last known-good decision it already has, denies anything it has no cached decision for, and never quietly lets new traffic through just because it can't reach the decision service (the PDP, Policy Decision Point).

Avoiding default-open

The PEP's own configuration must default to deny when it can't reach the PDP or when a cached decision has fully expired with nothing to fall back on. Many systems ship a "fail open on error" default for ease of onboarding, which is exactly backwards for a security control; that default has to be deliberately overridden.

Detecting the outage

The PEP runs short-interval health checks against the PDP, or watches the error rate on live decision calls, and trips a circuit breaker once a failure threshold is crossed, for example three consecutive failed calls. Tripping the breaker switches the PEP into cached-fallback mode immediately, rather than retrying synchronously on every request, which would both slow down every single call with a doomed real-time attempt and add load to an already-struggling PDP.

Safe cached-decision fallback

Each PEP keeps a local cache of the most recent decision per identity and route, each entry with an issuance time and TTL (time-to-live). While the PDP is unreachable, the PEP keeps honoring cached decisions until their TTL expires, a deliberately short, bounded staleness window, and denies anything with no cached decision at all, since there's nothing safe to fall back to for an identity or route it has never evaluated before.

sequenceDiagram
    participant PEP as Enforcement point (PEP)
    participant PDP as Policy decision point (PDP)
    PEP->>PDP: Decision request
    PDP-->>PEP: Signed decision (cached, TTL)
    Note over PEP: PDP becomes unreachable
    PEP->>PDP: Health check
    PDP--xPEP: No response (x3)
    PEP->>PEP: Trip circuit breaker, enter cached-fallback mode
    PEP->>PEP: Honor cached decisions until TTL expiry
    PEP->>PEP: Deny any identity with no cached decision
    Note over PEP,PDP: PDP restored
    PEP->>PDP: Resume live decision calls

Testing without locking out real users

Run this as a controlled exercise: block the PDP from a canary subset of PEPs, not all of them, during a low-traffic window; confirm cached decisions keep being honored, confirm new-identity requests are denied as designed, and confirm alerts fire; then restore connectivity and confirm the PEP reconciles cleanly back to live decisions. Repeat this on a schedule, for example quarterly, rather than trusting it once at launch, since software upgrades to either side can silently change fallback behavior.

Worked example (illustrating the mechanism, not a reported incident)

Suppose a PDP cluster becomes unreachable at a given moment, and PEPs detect the failure ten seconds later after three consecutive failed calls, tripping the breaker into cached mode. A user whose decision was cached shortly before, with a ten-minute TTL, keeps working until that cache entry expires ten minutes after it was issued; if the PDP is still down at that point, the user's next request is denied and they're prompted to retry, a visible but intentionally safe failure rather than a silent grant. A brand-new user trying to authenticate for the first time during the outage has no cache entry and is denied immediately. Once the PDP is restored, PEPs resume live calls, and the whole incident's designed bound was "degraded new access for a limited window, zero unauthorized access."

Trade-offs and pitfalls

Longer cache TTLs improve availability during a PDP outage but widen the window a genuinely revoked identity, for example a fired employee or a rotated compromised key, keeps working purely because its cache entry hasn't expired yet; size the TTL against how much of that risk the organization is actually willing to accept, not just for convenience. A common pitfall is testing failover against a single PEP instance and assuming it generalizes: distributed caches can behave inconsistently under a partial network partition, where some PEPs still see the PDP as reachable and others don't, producing uneven enforcement across the fleet unless that scenario is tested too.

Enforce identity cryptographically on every hop, but treat device posture as a signed, cacheable assertion rather than a live call, because posture checks are too expensive to perform synchronously on every request and identity verification already gives you a cheap, local check.

Where enforcement points sit

• Network layer: a coarse allow-list (for example, Kubernetes NetworkPolicy or cloud security groups) that limits which services can reach which ports at all. This is a backstop, not identity-aware.
• Host layer: a posture agent on the workload or endpoint that continuously evaluates health (patch level, endpoint detection status, configuration drift) and issues a short-lived, signed posture assertion.
• Service mesh (sidecar) layer: where the actual per-request decision is made. The sidecar verifies the caller's mTLS certificate (a cryptographic service identity, commonly issued through a workload-identity standard like SPIFFE, the Secure Production Identity Framework For Everyone) and checks the posture assertion attached to the call.

Evaluating and caching identity and posture

Identity verification is essentially free per request: it happens as part of the mTLS handshake against a locally cached trust root, with no external call needed. Posture is the expensive one, so the posture agent issues a signed assertion with a short time-to-live, for example five minutes, and the sidecar validates that assertion's signature and expiry locally on every request, only reaching back out to the posture service when the assertion is near expiry. That bounds how stale a posture decision can be to the TTL window while keeping the steady-state cost of each request purely local.

sequenceDiagram
    participant Caller as Caller service sidecar
    participant Posture as Host posture agent
    participant Callee as Callee service sidecar
    Posture->>Caller: Signed posture token (5 min TTL)
    Caller->>Callee: Request + mTLS cert + posture token
    Callee->>Callee: Verify mTLS chain locally (no network call)
    Callee->>Callee: Verify posture token signature and expiry locally
    alt token valid and unexpired
        Callee-->>Caller: Allow
    else token expired and posture service reachable
        Callee->>Posture: Refresh posture check
        Posture-->>Callee: New token or deny
    else token expired and posture service unreachable
        Callee-->>Caller: Deny (fail closed after grace window)
    end

What happens when identity or posture is unreachable

These two failure modes are not the same risk. Identity verification keeps working through a brief outage of the identity-issuing service, because existing certificates are already cached and valid until they expire, and a well-run workload-identity system rotates certificates well ahead of expiry. Posture assertions are the real exposure: if the posture service goes down, treat it with a tiered response rather than one blanket rule. High-sensitivity calls (for example, anything touching payment or admin actions) fail closed immediately once their assertion expires. Lower-risk internal reads get a short, explicitly bounded grace period, for example one additional TTL cycle, so a brief blip doesn't cause a wide outage, but the maximum staleness this can ever produce (in this example, roughly ten minutes: the original five-minute TTL plus one five-minute grace extension) is a deliberate, logged, alertable decision, not a silent default.

Trade-offs and pitfalls

Longer TTLs improve resilience to posture-service outages but widen the window a device that has actually gone out of compliance keeps being treated as trusted. The most common pitfall is coupling posture too tightly to identity, so an outage in one legacy posture system becomes a single point of failure for every service call across the whole mesh; put a circuit breaker around the posture check specifically so its failure degrades gracefully instead of taking down the identity layer with it. The other pitfall is a fail-open default: if posture verification fails and the system silently allows the call anyway, you've built exactly the flat-trust model zero trust was supposed to replace, just with more steps.

Migrate in observe-then-enforce phases, never straight to enforcement, because the biggest risk in this kind of migration isn't writing the wrong policy, it's not knowing about a dependency until the policy that blocks it goes live.

Migration plan

flowchart LR
    A[Discover: passive traffic mapping] --> B[Design target segmentation model]
    B --> C[Roll out policies in audit/log mode]
    C --> D[Cutover dev namespace]
    D --> E[Cutover staging]
    E --> F[Canary slice of production]
    F --> G[Full production enforce]
    G --> H[Continuous verification job]
    C -.rollback.-> R[Revert via GitOps]
    D -.rollback.-> R
    E -.rollback.-> R
    F -.rollback.-> R
    G -.rollback.-> R

1. Discover: run passive traffic observation (mesh telemetry, flow logs, or an eBPF-based tool) for a real baseline period before writing any policy. Never rely on documentation alone: undocumented calls are exactly what a migration like this breaks.
2. Design the target model: production and non-production get separate namespaces (or clusters) with an explicit authorization matrix of which services in which environment may call which.
3. Audit-mode rollout: deploy the new NetworkPolicies and mesh authorization policies in a permissive, log-only mode where supported, so you see what WOULD be denied without actually denying it yet.
4. Staged cutover, lowest risk first: dev namespace, then staging, then a small canary slice of production, watching error rates at each step before moving on.
5. Add mesh identity (mTLS) for east-west once network-level segmentation is stable, as a second, independent enforcement layer.
6. Continuous verification: keep a scheduled synthetic test (a small client pod that attempts both allowed and disallowed calls on a recurring basis) running permanently, not just as a one-time launch gate.

Rollback plan

Manage policies through GitOps (a workflow where the desired state is a version-controlled manifest that a controller continuously reconciles against) so rollback is a revert of the last commit, applied automatically. Keep a pre-tested "break-glass" wide-open policy that operators can apply directly, with mandatory logging and required approval, for the rare case where waiting on GitOps reconciliation is too slow during an active incident; that path should be alarmed on and time-boxed so it can't quietly become the new normal.

Worked example

Suppose the discovery phase's two-week observation window turns up three legacy scheduled jobs in an unrelated namespace that call a production database service directly, with no clear owner and no entry in the architecture docs. That's the exact kind of finding this phase exists to catch: had the team skipped straight to a deny-all policy, those jobs would have failed silently at 2 a.m. with no immediate connection back to "we just rolled out network segmentation."

Trade-offs and pitfalls

The audit-first approach adds real calendar time before the "real" cutover happens, which is a legitimate cost against the safety it buys. Rollback is easy for the network-policy layer itself, but harder once application code has started depending on the new state, for example if client libraries quietly stop doing their own authentication because they now assume mTLS is always present; rolling back the mesh halfway can leave both layers of auth disabled at once, a regression that's easy to miss because nothing errors loudly. The most common pitfall is letting the log-only phase run indefinitely because "everything looks fine": set a hard date to force the decision to either enforce or explicitly re-scope, or the migration quietly stalls at zero actual enforcement.

Direct answer

A zero-trust deployment is assembled from a small set of cooperating building blocks: an identity provider that establishes who or what is asking, a policy decision point and policy enforcement point that decide and apply access, microsegmentation and a service mesh that constrain what can talk to what, an API gateway that fronts requests at the edge, and telemetry that makes every one of those decisions auditable. None of them is "the" zero-trust system on its own; each fails in a specific, predictable way if it is missing or misconfigured.

Structured elaboration

Worked example

A team deploys a PDP and, to avoid outages, sets its behavior on timeout to "allow" rather than "deny." During a ten-minute PDP incident, every request that would normally be checked, at a steady rate of roughly 50,000 requests per minute, is instead let through unauthenticated: 10 minutes multiplied by 50,000 requests per minute is 500,000 unauthenticated requests waved through in that window. Choosing availability over safety at just one component silently disabled zero trust for the entire outage, even though every other component was configured correctly.

Trade-offs and pitfalls

The biggest practical risk is inconsistent fail-open versus fail-closed behavior across these components: mixing philosophies (some fail safe, some fail available) undermines the guarantees of the whole system even when each piece looks correct in isolation. The second most common risk is an unnoticed path that bypasses the PEP or gateway entirely, which is why telemetry across all of them, not just the "security" pieces, matters as much as the components themselves.