Zero Trust, Segmentation, and Service-to-Service Security Questions

Approach

Start from Kubernetes's actual default: with no NetworkPolicy objects at all, every pod can reach every other pod in the cluster, a flat network. The standard pattern is default-deny-first: apply a policy that blocks everything, then add narrow allow rules for only the traffic you intend. The checkout namespace example below shows the basic pattern; the payments namespace example after it shows a stricter variant with deny-all egress except one destination and cross-namespace ingress from a single named caller.

Code

# 1. Deny all ingress and egress by default in the checkout namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: checkout
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# 2. Explicitly allow frontend -> backend on port 8080
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: checkout
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

# Stricter payments-namespace variant:
# deny-all egress except one destination, cross-namespace ingress from one named caller only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-allow-ingress-from-checkout
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: checkout   # auto-applied label since Kubernetes 1.21
          podSelector:
            matchLabels:
              app: order-service
      ports:
        - protocol: TCP
          port: 8443
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-allow-egress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.50.0.0/24   # payment processor network range (CIDR: a block of IP addresses), illustrative only
      ports:
        - protocol: TCP
          port: 443
    - to:
        - namespaceSelector: {}   # allow DNS resolution cluster-wide, or scope tighter to kube-system in production
      ports:
        - protocol: UDP
          port: 53

Key points

• Combining namespaceSelector and podSelector inside the same from entry is an AND: the source must be in that namespace AND carry that pod label. Separate entries in the from list are OR'd together.
• Without the DNS egress rule, a deny-all-egress policy also blocks the pod from resolving service names, a very common way this pattern accidentally breaks the very service it was meant to protect.
• Not all CNI plugins (Container Network Interface, the pluggable layer that actually implements pod networking) enforce NetworkPolicy. Basic Flannel in its default overlay mode historically does not enforce it at all: the object is accepted by the API server but silently has no effect on the network. Calico, Cilium, and several cloud-managed CNIs in policy-enabled mode do enforce it, and Cilium (built on eBPF) can additionally enforce Layer 7 rules beyond what the core NetworkPolicy API supports.

Complexity

There's no algorithmic complexity to analyze since this is declarative configuration, but the operational cost is real: the CNI evaluates every applicable policy's rules against every connection, so more granular policies mean more per-connection evaluation work in whichever datapath the CNI uses (iptables rule chains, eBPF programs, or similar), and more surface area to keep consistent as services multiply.

Edge cases

• A pod with hostNetwork: true is not subject to NetworkPolicy at all.
• Policies from multiple objects that target the same pod are additive (a union of all matching rules), never a first-match short-circuit.
• If a new pod is added without the expected label, a deny-all baseline silently blocks it entirely rather than failing loudly, so label discipline needs its own check.
• The single biggest risk in this whole pattern: a policy that applies cleanly (kubectl apply succeeds) gives false confidence if the underlying CNI doesn't actually enforce NetworkPolicy, so always verify enforcement against the specific CNI in use, not just that the manifest was accepted.

Direct answer: a JSON Web Token (JWT) is self-contained and verified locally with a signature check, fast, no network call needed, but hard to revoke early. An opaque token is a meaningless random string that must be looked up (introspected) against the issuing server on every use, slower and an added dependency, but instantly revocable by deleting it server-side.

Local verification versus introspection: a JWT's verifier checks its cryptographic signature against the issuer's public key locally, no network call per request. An opaque token carries no embedded meaning, so the verifier must call the issuing server's introspection endpoint to ask whether it is still valid, on every use.

Revocation: a JWT is hard to revoke before its natural expiry, since any verifier holding the issuer's public key validates it independently, options are keeping expiry very short, or maintaining a deny-list every verifier must also check, which erodes the "no network call" advantage. An opaque token is trivially revocable, delete or invalidate the record in the issuer's store and every future introspection call immediately reports it invalid.

Size and transport: a JWT is bigger, a header, a payload of claims, and a signature, all base64-encoded, and this adds up if it rides along on many internal calls. An opaque token is small, just an identifier, cheap to pass around.

When to prefer each: prefer JWTs when many verifiers need to check tokens fast and independently, without a hot-path dependency on a central auth server, and when the token's lifetime can be kept short enough that limited revocability is not a real risk. Prefer opaque tokens with introspection when you need to instantly kill a specific session or credential, a compromised account, a departed employee's access, or when you do not want token contents exposed to anyone who can read the token in transit or in logs.

Common JWT validation mistakes a reviewer should look for:

• Algorithm confusion: trusting the alg field the token itself claims instead of pinning the expected algorithm, this is how a server ends up accepting a token signed with none, or gets tricked into verifying a forged token using its OWN public key as if it were a shared secret.
• Missing audience (aud) check: without verifying the token was issued FOR this specific service, a valid token minted for one internal service can be replayed against a different service that trusts the same issuer.
• Missing or ignored expiry (exp) check: a technically expired token is still accepted because the timestamp claim is never checked.
• Missing or mismatched issuer (iss) check when multiple issuers are trusted: a token from a lower-trust issuer gets accepted where only a specific higher-trust issuer should be.

Worked example: a reviewer should specifically confirm the code hardcodes an expected algorithm, for example explicitly requiring RS256 rather than reading alg from the token, explicitly compares the aud claim against an expected value, and explicitly checks exp, rather than trusting a library's defaults without confirming what those defaults enforce. Library defaults on this have changed across versions and ecosystems, so "we used a JWT library" is not itself something you can verify without reading the actual validation call.

Trade-offs & pitfalls: teams sometimes treat "JWT" and "secure" as synonyms, an improperly validated JWT is not safer than an opaque token, it is often worse, because it looks cryptographically strong while actually accepting forged tokens. Long-lived JWTs adopted because "introspection was too slow" reintroduce the exact revocation problem opaque tokens exist to solve.

Use Open Policy Agent (OPA), a general-purpose policy engine, with its policy language Rego to compute a single allow decision as the AND of three independent checks against the incoming request: the caller's JSON Web Token (JWT, a compact, signed way of encoding claims about who someone is), an access-control list, and a device posture score. Each clause enforces a distinct part of the zero trust story: who is calling, what they're allowed to do, and whether the thing presenting that identity is healthy enough to be trusted with it.

Approach

Model the request as input (the caller's JWT claims, the target service name, and the device posture) and keep the access lists and posture thresholds in a separate data document, so a security team can update who's allowed without touching the policy logic itself.

Code (Rego v1)

package service.authz

import rego.v1

default allow := false

allow if {
	input.jwt.aud == input.target_service
	input.jwt.role in data.access_lists[input.target_service]
	input.device.posture_score >= data.posture_thresholds[input.target_service]
}

Control data (data.json):

{
  "access_lists": {
    "billing-service": ["payments-writer", "payments-admin"]
  },
  "posture_thresholds": {
    "billing-service": 70
  }
}

A request that should be allowed (input.json):

{
  "jwt": {"aud": "billing-service", "role": "payments-writer"},
  "target_service": "billing-service",
  "device": {"posture_score": 82}
}

Run it:

opa eval -d policy.rego -d data.json -i input.json "data.service.authz.allow" --format pretty

Output: true

Change only posture_score to 55 in the input (below the 70 threshold) and re-run the same command: the output flips to false, and the same happens if jwt.aud is set to a different service than target_service.

Key points: what each clause protects against

• input.jwt.aud == input.target_service: protects against a token issued for one service being replayed against a different one. Without an audience check, a token that's valid but meant for the inventory service could be presented to the billing service and pass identity verification even though it was never intended for that call.
• input.jwt.role in data.access_lists[...]: protects against a caller who is correctly authenticated but not authorized for this specific service. Identity (who you are) is kept separate from entitlement (what you're allowed to do), which is the least-privilege half of the model.
• the posture check: protects against a compromised or non-compliant device or workload using otherwise-valid credentials. Even a correctly-scoped, correctly-authenticated caller shouldn't get access if the thing presenting that identity fails a health check. This is the "continuous verification" piece of zero trust: trust is re-evaluated against the current state of the caller, not granted once and assumed to hold.

Complexity and edge cases

Evaluation is a constant number of map lookups per request against the loaded data document, so the cost that actually scales is the size of data and how often it's refreshed, not the policy logic itself. Edge cases worth testing: a missing key anywhere in the lookup chain should resolve to undefined and therefore deny, matching the default allow := false; a request with posture_score entirely absent should fail closed rather than being treated as passing; and a role that exists on some OTHER service's access list but not this one's should still deny, since the lookup is keyed strictly by target_service.

Trade-offs and pitfalls

Hardcoded, hand-maintained access lists don't scale as the number of services grows; production setups usually generate data from a service catalog or a CI pipeline rather than editing it by hand. If this policy runs behind a centralized Policy Decision Point (PDP, the component that computes authorization decisions) rather than as a local OPA sidecar, add a timeout and an explicit fail-closed or short-TTL-cached behavior for when the PDP is unreachable, since otherwise an identity or posture system outage silently becomes an outage of all service traffic.

Direct answer: a service mesh, a dedicated infrastructure layer managing how services in a distributed system talk to each other, gives every service automatic mutual authentication, encryption, and traffic policy enforcement without each team writing that logic into its own application code, plus built-in visibility into every call the mesh handles.

Mutual TLS: the mesh automatically encrypts and mutually authenticates connections between services, both sides proving who they are, not just one, issuing and rotating the certificates behind the scenes, so application code never has to implement Transport Layer Security (TLS) handling itself.

Service identity: every service gets a cryptographic identity managed by the mesh, often following the SPIFFE standard for issuing workload identities, which is what mutual authentication actually checks, rather than trusting a service based on its network location or IP address.

Traffic policy enforcement: the mesh can enforce fine-grained rules about who is allowed to call whom, for example "service A may call service B's read endpoint but not its write endpoint," consistently across every service, without each one implementing its own authorization logic.

Observability: because every call already passes through the mesh's proxies, it captures metrics, logs, and distributed traces for every hop automatically, giving visibility into latency, error rates, and call patterns across the whole system without instrumenting each service's code individually.

What moves off individual services: teams no longer need to implement their own certificate handling, encryption, retry and circuit-breaking logic, or authorization checks in each service's own language, that logic lives once, in the mesh's shared infrastructure, instead of being reimplemented, and potentially reimplemented inconsistently, by every team.

Worked example (a scenario where a mesh adds more complexity than it is worth): a small startup with 8 services, all owned by one team, low compliance requirements, and a strong preference for moving fast, is a case where a full service mesh is probably not worth it. Running and upgrading a mesh control plane (the central component that configures and manages the mesh across all services), understanding sidecar-based debugging (a sidecar is a small helper proxy container that runs alongside each service and intercepts its network traffic), and troubleshooting a new network layer is real ongoing work; a lighter-weight approach, a shared authentication library, or a single API gateway, addresses that scale and risk profile with far less operational overhead.

Trade-offs & pitfalls: a mesh does not replace application-level authorization entirely, it enforces service-to-service identity and coarse traffic rules well, but fine-grained business logic about who can see what still usually lives in the application. Adopting a mesh is also not free just because it is "more secure," the sidecar proxies, control plane, and certificate infrastructure are new systems that need their own operational ownership.

Direct answer

Represent each microsegmentation rule as a structured record, source, destination, port or protocol, action, and priority, rather than free text. Detect a conflict by checking for overlapping match conditions with disagreeing outcomes, and resolve it with an explicit, documented precedence scheme, most commonly most-specific-wins with a small explicit-override mechanism for genuine exceptions, rather than leaving resolution to whichever rule happens to be evaluated first.

Structured elaboration

Data structure: represent each rule as {source, destination, port/protocol, action (allow or deny), priority, owner}, where source and destination are ranges or sets, such as a service identity or an IP block. Group rules by destination in an index, since "what can reach this resource" is usually the more useful question to ask, so checking a specific destination doesn't require scanning the entire rule set.

Algorithm to detect a conflict: two rules conflict when their source ranges overlap, their destination ranges overlap, their port or protocol scopes overlap, and their actions disagree for that overlapping space. This is fundamentally a multi-dimensional interval-overlap check, similar to detecting overlapping time ranges but generalized across source, destination, and port. A practical approach: for each new or changed rule, compare it only against existing rules sharing the same destination, using the index above rather than the whole rule set, check whether source ranges intersect and port scopes intersect, and if both do while actions differ, flag it for resolution instead of silently accepting whichever rule was added or evaluated last.

Resolving a detected conflict, three approaches:

• Priority order: each rule carries an explicit numeric priority, and the higher-priority rule wins regardless of specificity. Simple to reason about, but requires someone to correctly assign and maintain priorities as rules accumulate, which can itself drift into disorder.
• Most-specific-wins: the rule matching the narrowest scope wins, for example a rule naming one exact service pair beats a rule naming an entire team's services. This tends to match human intuition, that a specific exception should override a general policy, and needs less manual priority bookkeeping, but requires a clear, computable definition of "more specific," typically narrower source first, then narrower destination, then narrower port scope, so it's unambiguous when two rules are each specific along different dimensions.
• Explicit override: a small, tightly controlled set of rules marked to take precedence over the normal resolution order entirely, used sparingly for genuine exceptions, such as a security team's emergency block that must always win over any other team's allow rule. This should require its own approval process precisely because it breaks the otherwise predictable resolution logic.

A default-deny environment should generally favor most-specific-wins as the default resolution, since it naturally lets a narrow, deliberate allow exception coexist with a broad deny, reserving explicit override for a small number of genuinely special cases and priority order as the tiebreaker only when two rules are equally specific.

Worked example

A default policy denies all traffic between teams: source is Team A, destination is Team B, action deny, the broad rule. A specific integration needs Team A's checkout service to reach Team B's inventory service: source checkout-service, destination inventory-service, action allow, the narrow rule. These two rules overlap, checkout-service is a subset of Team A and inventory-service is a subset of Team B, and disagree in action, a detected conflict. Under most-specific-wins, the narrower rule, naming two exact services, beats the broader team-level rule, so that one integration is allowed while everything else between the two teams stays denied, the intended outcome, without anyone needing to hand-assign relative priorities between the two rules.

Trade-offs and pitfalls

The hardest part in practice is not detecting an overlap, it's defining "more specific" unambiguously when two rules are specific along different dimensions, one narrowing source, the other narrowing port range. An unambiguous, documented tiebreaking order has to exist before the algorithm can be applied consistently, otherwise different reviewers will disagree about which rule should win.