Kubernetes Architecture, Operations, and Troubleshooting Questions

Namespaces partition a single cluster into virtual clusters, mainly for scoping quotas, policies, and access control, not for hard security isolation on their own (that also needs NetworkPolicy). Role-based access control (RBAC) governs who, or what workload, can perform which verbs (get, list, create, update, delete) against which resources. In RBAC, Role and ClusterRole define the what (a set of permissions); RoleBinding and ClusterRoleBinding define the who and attach that permission set to subjects (users, groups, or ServiceAccounts). A ServiceAccount is the identity a pod uses to talk to the API server. Least-privilege in a multi-team cluster comes from combining namespace-per-team boundaries with narrowly scoped Roles and per-workload ServiceAccounts, not from handing out ClusterRoles.

The four RBAC objects

The mechanic most people miss: a RoleBinding can reference a ClusterRole. When it does, the ClusterRole's permissions are limited to the RoleBinding's own namespace. That is exactly how the built-in view, edit, and admin ClusterRoles are meant to be consumed: define the permission set once as a ClusterRole, then reuse it per-namespace via ordinary RoleBindings instead of writing a near-identical Role in every namespace. cluster-admin and ClusterRoleBinding are the only combination that actually grants cluster-wide, unrestricted access; everything else can be scoped down to a namespace even when it starts from a ClusterRole.

ServiceAccounts and workload identity

Every pod runs as a ServiceAccount (the default one in its namespace if you don't set one). Since Kubernetes 1.24, ServiceAccounts no longer get a long-lived Secret-based token auto-created and auto-mounted; tokens are now short-lived, audience-bound, and obtained through the TokenRequest API, delivered to the pod as a projected volume that Kubernetes refreshes automatically. Practically this means: don't design around "the ServiceAccount's Secret," and if you need a long-lived token for an out-of-cluster consumer you must request one explicitly.

Least-privilege patterns across teams and environments

• Namespace per team-per-environment (team-a-dev, team-a-prod), each with its own Roles rather than one shared namespace with broad access.
• One ServiceAccount per workload (not one shared "app" account per namespace), so a compromised pod's blast radius is exactly that workload's permissions.
• Write Roles with explicit verb and resource lists; avoid resources: ["*"] or verbs: ["*"]. A Role that grants everything is functionally a ClusterAdmin-lite and defeats the point.
• Reuse the built-in view/edit ClusterRoles via namespaced RoleBindings for read-only or standard app-team access instead of hand-rolling near-duplicates.
• Enforce the guardrails you can't trust humans to remember with a validating admission webhook, for example OPA (Open Policy Agent) Gatekeeper or Kyverno, to reject wildcard verbs or overly broad bindings at creation time rather than catching them in an audit later.
• Keep this distinct from pod-level security: RBAC controls API access; it says nothing about what a container can do on the node (privileged mode, host mounts). Since PodSecurityPolicy was removed in Kubernetes 1.25, that separate concern is covered by Pod Security Admission (PSA), not RBAC.

Worked example

A CI runner needs to deploy to team-a-dev only, never to team-a-prod.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: team-a-dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: team-a-dev
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: team-a-dev
subjects:
- kind: ServiceAccount
  name: ci-deployer
  namespace: team-a-dev
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

This ServiceAccount can get/update/patch Deployments in team-a-dev and nothing else: no Secrets, no other namespaces, no cluster-scoped objects. Production deploys use a completely separate ServiceAccount and Role bound only in team-a-prod, typically gated behind a different pipeline stage with its own approval step.

Trade-offs and pitfalls

• ClusterRoleBinding sprawl is the most common real-world RBAC failure: it's easy to grant a ClusterRole via ClusterRoleBinding "to unblock someone quickly" and never revisit it. Prefer the namespaced-RoleBinding-to-a-ClusterRole pattern above instead.
• Wildcard verbs or resources ("*") are the second most common failure; they're invisible in a quick review of the RoleBinding but grant far more than intended.
• Audit what's actually bound, not just what you intended to bind; the kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> command (or the community kubectl-who-can plugin) will tell you the effective permissions for a subject, which is the only reliable way to catch drift.
• Namespaces are a policy and quota boundary, not a hard security boundary by themselves; don't rely on RBAC alone to stop a compromised pod from reaching another namespace's pods over the network. That needs NetworkPolicy in addition.

A pod's image pull uses the first credential source it finds, in order: pull secrets listed directly on the Pod spec, then pull secrets attached to the Pod's ServiceAccount, then whatever credentials the node itself can supply. If pods across several namespaces all start failing at once, the fault is almost never a per-namespace object (Secrets and ServiceAccounts are namespace-scoped, so a typo in one namespace would not explain a cluster-wide outage); look one level below namespaces, at the registry, the node credential mechanism, or a change that touched every namespace at once (a controller that syncs secrets, an expired shared token, or a registry-side block).

How credential resolution actually works

1. Pod-level imagePullSecrets. If pod.spec.imagePullSecrets lists a Secret, the kubelet uses it. The Secret must be of type kubernetes.io/dockerconfigjson (a JSON blob shaped like Docker's ~/.docker/config.json, holding registry hostname to username/password or token) and must live in the same namespace as the Pod, since Secrets are namespace-scoped objects.
2. ServiceAccount-attached secrets. If the Pod does not name its own imagePullSecrets, Kubernetes checks the ServiceAccount the Pod runs as (pod.spec.serviceAccountName, defaulting to default). If that ServiceAccount has imagePullSecrets configured, an admission controller copies them onto the Pod at creation time. This is why teams standardize on patching the namespace's default ServiceAccount rather than repeating the same Secret on every Pod spec.
3. Node-level credentials, last. If neither of the above supplies a credential, the kubelet falls back to whatever it can resolve on the node itself. On a modern cluster this is the kubelet's credential provider plugin mechanism (--image-credential-provider-config / --image-credential-provider-bin-dir, generally available since Kubernetes 1.26): a small binary such as an ECR- or GCR-flavored credential provider that exchanges the node's cloud identity (an AWS IAM role, a GCP service account, workload identity) for a short-lived registry token, with no Secret object involved at all. This is a currency point worth being precise about: the older mental model of the kubelet reading a static /etc/docker/config.json on the node belongs to the dockershim era (dockershim was removed in Kubernetes 1.24, alongside direct Docker-socket calls); today the kubelet talks to containerd or CRI-O over the Container Runtime Interface (CRI), and credentials are resolved by the kubelet itself (via credential providers, or a static credential file for private on-prem registries) before it hands an authenticated pull request to the runtime, not by the runtime reading a Docker-specific file.

Debugging pods failing across multiple namespaces

1. Confirm the blast radius and the error shape first:

kubectl get pods -A --field-selector=status.phase!=Running | grep -E 'ErrImagePull|ImagePullBackOff'
kubectl describe pod <pod> -n <ns>

Look for the actual reason string, not just the backoff state, for example:

Warning  Failed   12s   kubelet  Failed to pull image "registry.example.com/app:v3":
  rpc error: code = Unknown desc = failed to authorize: 401 Unauthorized

401 Unauthorized points at a credential problem specifically, as opposed to manifest unknown (wrong tag) or a network timeout (egress/firewall problem to the registry).
2. Check whether the affected namespaces share a common ServiceAccount pattern or a common Secret sync mechanism (many teams push the same dockerconfigjson Secret into every namespace via a controller such as kubernetes-reflector or an external-secrets operator); a bug or expiry in that shared pipeline explains a simultaneous multi-namespace break far better than coincidental per-namespace misconfiguration.
3. Decode and inspect one representative Secret:

kubectl get secret regcred -n <ns> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Check the token/password is not expired and the registry hostname key matches the image reference exactly (a mismatched hostname, for example a registry alias versus its canonical DNS name, is a common silent failure).
4. If node-level credentials are in play (no imagePullSecrets anywhere), check the node's credential provider instead of any namespace object:

kubectl logs -n kube-system <credential-provider or kubelet-adjacent pod, if run as one>
journalctl -u kubelet -n 200 | grep -i credential

A cluster-wide, sudden break here usually means the node's cloud identity lost registry permission (an IAM policy or role binding changed) rather than anything Kubernetes-side.
5. Rule out the registry itself: a registry-side rate limit, an account suspension, or a maintenance window will produce 401/403/429 responses to every caller regardless of namespace, which looks identical to a credential problem from the cluster's point of view.

Rotating credentials safely

Because the kubelet re-resolves credentials on every pull rather than caching them for a pod's lifetime, rotating the Secret's contents does not require restarting already-running pods; it only affects the next pull (a new pod, a restart after a crash, or a rolling update). The safe rotation sequence is: create the new Secret (or kubectl create secret docker-registry ... --dry-run=client -o yaml | kubectl apply -f - to update in place), verify a fresh pull succeeds in a canary namespace, then let the old credential expire on the registry side. For node-level (cloud IAM) credentials, rotation is handled entirely outside Kubernetes by the cloud provider's short-lived token issuance, which is one real advantage of workload-identity-based node credentials over static Secrets: there is no rotation to schedule at all.

Trade-offs and pitfalls

• Node-level credentials (cloud IAM / workload identity) eliminate Secret management and rotation entirely, but only work for registries the cloud provider integrates with (its own container registry); a private on-prem or third-party registry still needs an explicit imagePullSecrets chain.
• A common wrong turn: assuming a Secret created in one namespace is visible to Pods in another. It is not. Teams either duplicate the Secret per namespace or run a sync controller; if that sync controller silently fails, only namespaces created after the failure are missing the Secret, producing a confusing "some namespaces work, some don't" pattern that looks like a Kubernetes bug but is a sync-pipeline gap.
• Patching only the default ServiceAccount is a common oversight when workloads intentionally use a non-default ServiceAccount; the pull secret has to be attached to whichever ServiceAccount the failing Pods actually reference.
• Restarting kubelet or recreating pods "to fix it" without checking the actual 401/403 reason wastes an incident cycle; the reason string in the pod event almost always tells you which of the three resolution layers to look at first.

Direct answer

Gatekeeper enforces policy as a validating admission webhook: a ConstraintTemplate defines reusable Rego (OPA's policy language) logic and the CRD (Custom Resource Definition) shape it is configured with, and a Constraint is an instance of that template scoped to specific kinds and namespaces. For "deny privileged pods or pods missing resource limits," the template checks spec.containers[].securityContext.privileged and resources.limits.cpu/resources.limits.memory across every container and returns one violation message per offending container. The safe way to ship it is audit-only first, then targeted enforcement, never enforce cluster-wide on day one against an unaudited cluster.

Structured elaboration

ConstraintTemplate: the reusable Rego logic

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredpodsecurityandresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredPodSecurityAndResources
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredpodsecurityandresources

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          c.securityContext.privileged == true
          msg := sprintf("container '%v' is privileged", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.cpu
          msg := sprintf("container '%v' is missing resources.limits.cpu", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.memory
          msg := sprintf("container '%v' is missing resources.limits.memory", [c.name])
        }

This is Rego v0 syntax, still Gatekeeper's default. Gatekeeper 3.19 and later also supports opt-in Rego v1, which requires an explicit if before each rule body, but v0 remains what ships by default and what most existing ConstraintTemplates use.

Constraint: applying the template

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredPodSecurityAndResources
metadata:
  name: deny-privileged-or-no-limits
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]

enforcementAction: dryrun records violations without blocking anything, the correct starting state for any new policy on an existing cluster.

Validating vs mutating admission, and when to reach for either over OpenAPI schema validation

Gatekeeper is a validating admission webhook: it can accept or reject an object but cannot change it. A mutating admission webhook runs earlier in the chain and can rewrite the object before it is persisted, for example a sidecar injector adding a container, or a default value filled in. Kubernetes ships built-in admission controllers doing exactly these two jobs without any webhook at all, and they are the closest analogues to what Gatekeeper does here: LimitRanger (mutating; injects default resource requests and limits when a pod omits them) and ResourceQuota (validating; rejects a request that would exceed a namespace's aggregate quota).

The choice between a webhook and plain OpenAPI schema validation on a CRD comes down to what the rule needs to know:

• If the rule is fully expressible as a shape constraint on one object in isolation (a field must be one of an enum, a string must match a pattern, a number must sit in a range), OpenAPI schema validation on the CRD costs nothing at admission time and needs no separate service running.
• Reach for a webhook only when the rule needs something schema validation cannot express: cross-field logic (a field is required only if another field has a certain value), cross-object lookups (checking a Secret exists, checking sibling objects against a quota), or a policy that must apply uniformly across many unrelated resource kinds, exactly the "any Pod, any namespace" shape of the privileged/no-limits policy here.

Safe rollout plan

1. Deploy in enforcementAction: dryrun; let it run against real traffic for a full deploy cycle while pulling violations from Gatekeeper's audit results.
2. Share violations with owning teams with the exact fix needed (add resources.limits.cpu/memory, remove privileged: true), not just "you're non-compliant."
3. Flip to enforcementAction: deny first in one low-risk namespace, watch for unexpected rejections, then expand namespace by namespace.
4. Keep excludedNamespaces narrow and explicit, system namespaces only; a broad exclusion list defeats the point of a cluster-wide policy.

Worked example

A pod with two containers: one declares resources.limits: {cpu: "500m", memory: "256Mi"} and passes cleanly; the other has no resources block at all. The template's second and third rules each fire once for the second container, producing two separate violation messages ("missing resources.limits.cpu" and "missing resources.limits.memory"). That per-container, per-field granularity is what makes the audit output actionable rather than a single opaque "pod rejected."

Trade-offs and pitfalls

• Rego policy is powerful but opaque to most application developers; ship it with plain-language violation messages as above, and do not expect teams to read Rego to understand why they were blocked.
• excludedNamespaces is a blunt instrument. Overusing it to unblock a team quickly erodes the policy's coverage silently; track exclusions the same way you would track a firewall exception.
• A validating webhook adds a synchronous hop to every matched request. Keep the Rego evaluation cheap, no external calls, so it does not become the very apiserver latency problem it would otherwise be diagnosing.

Direct answer

A comprehensive Kubernetes cluster security strategy layers four kinds of control: what is allowed to run (admission control plus the current Pod Security Standards, which replaced the now-removed PodSecurityPolicy), what is allowed to run only if it is provably yours (image signing with sigstore/cosign), what is allowed to talk to what (NetworkPolicy segmentation), and who is allowed to do what (RBAC, role-based access control, least privilege, plus encrypted, rotated secrets). None substitutes for another; the design work is sequencing the rollout so each layer is validated in audit mode before it can reject a real deployment.

Structured elaboration

Admission control: OPA Gatekeeper plus Pod Security Standards

Use Gatekeeper for custom, org-specific rules (image provenance, required labels, naming conventions) and the built-in Pod Security Admission controller for the baseline hardening levels (privileged, baseline, restricted) defined by the Pod Security Standards. This split matters because PodSecurityPolicy, the older mechanism for this same job, was deprecated in Kubernetes 1.21 and fully removed in 1.25; a runbook or an older interview question that still describes writing a PodSecurityPolicy object is describing something that no longer exists in a supported cluster. The current mechanism is a namespace label:

kubectl label namespace payments pod-security.kubernetes.io/enforce=restricted

with audit and warn variants of the same label for a non-blocking rollout, mirroring Gatekeeper's own dry-run-then-enforce pattern.

Image signing and verification (sigstore/cosign)

CI signs each image at build time (cosign sign), ideally keyless and backed by the transparency log (Rekor) rather than a long-lived private key that itself has to be rotated and protected. Gatekeeper, or a purpose-built admission webhook, verifies the signature and, where available, checks provenance or SBOM (software bill of materials) attestations before allowing the image to run. Cache verification results at the admission layer; re-verifying every pull on every pod restart adds latency to something that should not change between restarts of the same image digest.

Network segmentation with NetworkPolicy

Default-deny per namespace, then explicit allow rules per service-to-service relationship:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

This is enforced by the CNI (Container Network Interface) plugin, not the API server. A cluster on a CNI without NetworkPolicy support will accept this object but silently not enforce it, a common false sense of security worth checking explicitly: kubectl get networkpolicy succeeding proves the object exists, not that anything is actually blocking traffic.

RBAC hardening

Least privilege means a Role/RoleBinding scoped per namespace and per workload's own ServiceAccount, never a shared default ServiceAccount with broad permissions, and never a wildcard verb or resource in a Role used by application workloads. Periodic access reviews, who can actually reach cluster-admin, directly or through a chain of bindings, catch privilege creep that accumulates silently over time.

Secrets: encryption and rotation

Encrypt etcd at rest using the KMS (Key Management Service) encryption provider, specifically KMS v2, the recommended provider since it went stable in Kubernetes 1.29 (KMS v1 was deprecated in 1.28 and is disabled by default starting in 1.29). KMS v2 caches a data-encryption key locally after one call to the external KMS plugin, instead of calling out on every secret read, removing what used to be a real apiserver-latency cost of encryption at rest. Layer short-lived, automatically rotated credentials (a secrets manager issuing dynamic credentials, or a cloud provider's workload-identity federation mapping a ServiceAccount to a cloud role) on top, so long-lived static secrets are the exception rather than the default.

Multi-tenant blast-radius controls, the absorbed noisy-neighbor angle

At the Kubernetes object level, not the platform-availability layer, ResourceQuota and LimitRange are what actually stop one tenant's namespace from starving another's: ResourceQuota caps aggregate CPU, memory, and object counts per namespace, and LimitRange sets default and maximum per-container requests so a single unbounded pod cannot consume a quota's entire budget alone. This is deliberately narrow: quota and limit-range mechanics are this topic's ground; broader noisy-neighbor blast-radius design (autoscaling interaction, billing isolation) belongs to availability and disaster-recovery planning, not to a cluster security strategy.

Worked example (rollout sequencing, not a fabricated metric)

A four-phase rollout, each phase gated on the previous one running clean in audit mode for a full deploy cycle:

1. Baseline: enable audit logging, run Gatekeeper in dryrun, apply pod-security.kubernetes.io/warn=restricted everywhere (visible, non-blocking).
2. Enforce cheap wins: require resource limits and non-privileged containers cluster-wide (few legitimate exceptions), enable KMS v2 encryption for new secrets.
3. Enforce identity-sensitive controls: require image signature verification in production namespaces, flip Pod Security to enforce=restricted namespace by namespace as owners confirm compatibility.
4. Enforce network segmentation last, since it carries the highest blast radius for a mistake: roll out default-deny NetworkPolicies namespace by namespace, watching connection-refused rates as the signal to fix an allow-rule gap before moving to the next namespace.

Trade-offs and pitfalls

• PodSecurityPolicy no longer exists in any supported Kubernetes version; if an older reference or absorbed legacy question still frames this control that way, translate it to the Pod Security Standards labels above rather than reproducing a removed API object.
• Signature verification and default-deny NetworkPolicy are the two highest-friction controls for developers; sequence them last, after cheaper wins have built trust in the rollout process.
• KMS v1 is disabled by default in current Kubernetes; a cluster still relying on it is itself a finding, not a stable baseline to build on.

A ConfigMap and a Secret are both key-value objects for feeding configuration into a pod without baking it into the image, but a Secret is meant for sensitive values and Kubernetes handles it slightly differently. Neither is encrypted by default: Secret values are only base64-encoded in etcd (the cluster's data store), which is trivially reversible, not encryption. Treat 'it's a Secret' as an access-control and audit boundary, not as cryptographic protection, unless encryption at rest has been explicitly turned on.

ConfigMap vs Secret

Two ways to expose a Secret to a pod

Environment variable:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: my-secret
        key: db-password

Mounted file:

volumes:
  - name: secret-vol
    secret:
      secretName: my-secret
containers:
  - name: app
    volumeMounts:
      - name: secret-vol
        mountPath: /etc/creds
        readOnly: true

Worked example: what actually happens when a Secret's value changes

Trace what happens after my-secret's db-password key is updated, depending on how it was exposed:

• Env var: the running pod keeps using the old value until it is restarted or replaced. Environment variables are read once at container start and never update in place.
• Mounted file, no subPath: the file at /etc/creds/db-password does eventually update, once the kubelet's periodic sync catches up (on the order of a minute, not instantly), but the running process only sees the change if it re-reads the file itself, since nothing forces that.
• Mounted file with subPath: it never updates at all, because a subPath mount is bound to a specific file version at mount time.
  Neither the env-var path nor the volume path forces a pod restart on a config change. The common pattern to actually guarantee a fresh pod is to hash the ConfigMap or Secret's content into a pod template annotation (Helm's checksum/config annotation is the usual form); a content change then produces a different pod template hash, which forces a real rollout instead of relying on an in-place file update the application may not even notice.

Trade-offs and pitfalls

• Base64 is not encryption; anyone with read access to Secret objects, or to etcd's data files directly, can decode it in one command. Enabling encryption at rest (an EncryptionConfiguration backed by a Key Management Service, KMS, provider, the current recommended approach since the older static-key KMS v1 API was deprecated as of Kubernetes 1.28) protects the etcd-at-rest copy; Role-Based Access Control (RBAC) is what actually protects who can read the Secret object in the first place, and the two are not substitutes for each other.
• Environment variables are easy to leak: they show up when describing a running pod, in crash dumps, and in some logging frameworks that log the process environment; prefer mounted files for anything sensitive when the application can read from a file path instead.
• For continuous integration and continuous delivery (CI/CD) pipelines, never let plaintext secrets sit in pipeline configuration or version control; use the pipeline platform's own secret store, scope credentials as narrowly and as short-lived as possible, and prefer pulling secrets at deploy time from an external manager (HashiCorp Vault, a cloud provider's secrets manager, or a Secrets Store CSI driver) over baking them into a committed manifest, even one covered by a .gitignore entry.