Kubernetes Architecture, Operations, and Troubleshooting Questions

A ConfigMap and a Secret are both key-value objects for feeding configuration into a pod without baking it into the image, but a Secret is meant for sensitive values and Kubernetes handles it slightly differently. Neither is encrypted by default: Secret values are only base64-encoded in etcd (the cluster's data store), which is trivially reversible, not encryption. Treat 'it's a Secret' as an access-control and audit boundary, not as cryptographic protection, unless encryption at rest has been explicitly turned on.

ConfigMap vs Secret

Two ways to expose a Secret to a pod

Environment variable:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: my-secret
        key: db-password

Mounted file:

volumes:
  - name: secret-vol
    secret:
      secretName: my-secret
containers:
  - name: app
    volumeMounts:
      - name: secret-vol
        mountPath: /etc/creds
        readOnly: true

Worked example: what actually happens when a Secret's value changes

Trace what happens after my-secret's db-password key is updated, depending on how it was exposed:

• Env var: the running pod keeps using the old value until it is restarted or replaced. Environment variables are read once at container start and never update in place.
• Mounted file, no subPath: the file at /etc/creds/db-password does eventually update, once the kubelet's periodic sync catches up (on the order of a minute, not instantly), but the running process only sees the change if it re-reads the file itself, since nothing forces that.
• Mounted file with subPath: it never updates at all, because a subPath mount is bound to a specific file version at mount time.
  Neither the env-var path nor the volume path forces a pod restart on a config change. The common pattern to actually guarantee a fresh pod is to hash the ConfigMap or Secret's content into a pod template annotation (Helm's checksum/config annotation is the usual form); a content change then produces a different pod template hash, which forces a real rollout instead of relying on an in-place file update the application may not even notice.

Trade-offs and pitfalls

• Base64 is not encryption; anyone with read access to Secret objects, or to etcd's data files directly, can decode it in one command. Enabling encryption at rest (an EncryptionConfiguration backed by a Key Management Service, KMS, provider, the current recommended approach since the older static-key KMS v1 API was deprecated as of Kubernetes 1.28) protects the etcd-at-rest copy; Role-Based Access Control (RBAC) is what actually protects who can read the Secret object in the first place, and the two are not substitutes for each other.
• Environment variables are easy to leak: they show up when describing a running pod, in crash dumps, and in some logging frameworks that log the process environment; prefer mounted files for anything sensitive when the application can read from a file path instead.
• For continuous integration and continuous delivery (CI/CD) pipelines, never let plaintext secrets sit in pipeline configuration or version control; use the pipeline platform's own secret store, scope credentials as narrowly and as short-lived as possible, and prefer pulling secrets at deploy time from an external manager (HashiCorp Vault, a cloud provider's secrets manager, or a Secrets Store CSI driver) over baking them into a committed manifest, even one covered by a .gitignore entry.

Direct answer

A comprehensive Kubernetes cluster security strategy layers four kinds of control: what is allowed to run (admission control plus the current Pod Security Standards, which replaced the now-removed PodSecurityPolicy), what is allowed to run only if it is provably yours (image signing with sigstore/cosign), what is allowed to talk to what (NetworkPolicy segmentation), and who is allowed to do what (RBAC, role-based access control, least privilege, plus encrypted, rotated secrets). None substitutes for another; the design work is sequencing the rollout so each layer is validated in audit mode before it can reject a real deployment.

Structured elaboration

Admission control: OPA Gatekeeper plus Pod Security Standards

Use Gatekeeper for custom, org-specific rules (image provenance, required labels, naming conventions) and the built-in Pod Security Admission controller for the baseline hardening levels (privileged, baseline, restricted) defined by the Pod Security Standards. This split matters because PodSecurityPolicy, the older mechanism for this same job, was deprecated in Kubernetes 1.21 and fully removed in 1.25; a runbook or an older interview question that still describes writing a PodSecurityPolicy object is describing something that no longer exists in a supported cluster. The current mechanism is a namespace label:

kubectl label namespace payments pod-security.kubernetes.io/enforce=restricted

with audit and warn variants of the same label for a non-blocking rollout, mirroring Gatekeeper's own dry-run-then-enforce pattern.

Image signing and verification (sigstore/cosign)

CI signs each image at build time (cosign sign), ideally keyless and backed by the transparency log (Rekor) rather than a long-lived private key that itself has to be rotated and protected. Gatekeeper, or a purpose-built admission webhook, verifies the signature and, where available, checks provenance or SBOM (software bill of materials) attestations before allowing the image to run. Cache verification results at the admission layer; re-verifying every pull on every pod restart adds latency to something that should not change between restarts of the same image digest.

Network segmentation with NetworkPolicy

Default-deny per namespace, then explicit allow rules per service-to-service relationship:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

This is enforced by the CNI (Container Network Interface) plugin, not the API server. A cluster on a CNI without NetworkPolicy support will accept this object but silently not enforce it, a common false sense of security worth checking explicitly: kubectl get networkpolicy succeeding proves the object exists, not that anything is actually blocking traffic.

RBAC hardening

Least privilege means a Role/RoleBinding scoped per namespace and per workload's own ServiceAccount, never a shared default ServiceAccount with broad permissions, and never a wildcard verb or resource in a Role used by application workloads. Periodic access reviews, who can actually reach cluster-admin, directly or through a chain of bindings, catch privilege creep that accumulates silently over time.

Secrets: encryption and rotation

Encrypt etcd at rest using the KMS (Key Management Service) encryption provider, specifically KMS v2, the recommended provider since it went stable in Kubernetes 1.29 (KMS v1 was deprecated in 1.28 and is disabled by default starting in 1.29). KMS v2 caches a data-encryption key locally after one call to the external KMS plugin, instead of calling out on every secret read, removing what used to be a real apiserver-latency cost of encryption at rest. Layer short-lived, automatically rotated credentials (a secrets manager issuing dynamic credentials, or a cloud provider's workload-identity federation mapping a ServiceAccount to a cloud role) on top, so long-lived static secrets are the exception rather than the default.

Multi-tenant blast-radius controls, the absorbed noisy-neighbor angle

At the Kubernetes object level, not the platform-availability layer, ResourceQuota and LimitRange are what actually stop one tenant's namespace from starving another's: ResourceQuota caps aggregate CPU, memory, and object counts per namespace, and LimitRange sets default and maximum per-container requests so a single unbounded pod cannot consume a quota's entire budget alone. This is deliberately narrow: quota and limit-range mechanics are this topic's ground; broader noisy-neighbor blast-radius design (autoscaling interaction, billing isolation) belongs to availability and disaster-recovery planning, not to a cluster security strategy.

Worked example (rollout sequencing, not a fabricated metric)

A four-phase rollout, each phase gated on the previous one running clean in audit mode for a full deploy cycle:

1. Baseline: enable audit logging, run Gatekeeper in dryrun, apply pod-security.kubernetes.io/warn=restricted everywhere (visible, non-blocking).
2. Enforce cheap wins: require resource limits and non-privileged containers cluster-wide (few legitimate exceptions), enable KMS v2 encryption for new secrets.
3. Enforce identity-sensitive controls: require image signature verification in production namespaces, flip Pod Security to enforce=restricted namespace by namespace as owners confirm compatibility.
4. Enforce network segmentation last, since it carries the highest blast radius for a mistake: roll out default-deny NetworkPolicies namespace by namespace, watching connection-refused rates as the signal to fix an allow-rule gap before moving to the next namespace.

Trade-offs and pitfalls

• PodSecurityPolicy no longer exists in any supported Kubernetes version; if an older reference or absorbed legacy question still frames this control that way, translate it to the Pod Security Standards labels above rather than reproducing a removed API object.
• Signature verification and default-deny NetworkPolicy are the two highest-friction controls for developers; sequence them last, after cheaper wins have built trust in the rollout process.
• KMS v1 is disabled by default in current Kubernetes; a cluster still relying on it is itself a finding, not a stable baseline to build on.

Native Kubernetes Secrets are the weakest link in the chain unless layered with etcd encryption at rest and strict RBAC (Role-Based Access Control), and for anything genuinely sensitive in production the stronger pattern is a dynamic-secrets store like HashiCorp Vault, or a cloud KMS (Key Management Service)-backed secret manager, surfaced into Pods through an ExternalSecrets operator or the Secrets Store CSI (Container Storage Interface) driver rather than stored as native Secret objects at rest.

Comparison

Key rotation: a worked example

Say a database credential is consumed by 50 replica Pods through Vault's database secrets engine rather than a static password.

1. Configure Vault's database engine with a lease TTL, for example 1 hour, and a max TTL, for example 24 hours.
2. Each Pod's sidecar or CSI driver requests a lease-backed credential on startup and again before the lease expires; Vault generates a genuinely new database user/password pair per lease rather than handing out one shared static credential.
3. Consumption method changes what "rotation" actually delivers. If the credential is mounted as a projected volume file via the Secrets Store CSI driver, the driver's periodic sync picks up the renewed value and the file updates in place; an application that re-reads the file (rather than caching it in memory indefinitely) picks up the new credential without a restart. If instead the credential is injected as an environment variable, it is fixed at container start and never updates on its own. Rotating a credential consumed this way requires a rolling Pod restart to actually take effect, which is the single most common reason a "rotation" silently fails to protect anything: the store issued a new credential, but every running Pod kept using the old one until restarted.
4. At the max TTL, Vault revokes the lease outright; any Pod that hasn't picked up a renewed credential by then loses access, which is the intended forcing function, not a bug.

Cross-environment promotion pattern

Dev, staging, and prod should each read from environment-scoped secret paths (separate Vault namespaces/mounts, or separate cloud-secret-manager projects/resource groups), never the same secret value copied across environments. A promotion pipeline should promote the reference (an ExternalSecret object pointing at secret/prod/db-password) as part of the manifest, while the actual value it resolves to differs per environment because it lives in a separate, environment-scoped backend location. This also means a compromised staging credential can never be the same value as the production one, since they were never the same secret to begin with, not just separately labeled copies of it.

Avoiding leakage in Git-backed CI/CD pipelines

• Never commit a plaintext Secret manifest, ever, including "temporarily" during testing; a Git history rewrite does not reliably remove a secret that's already been cloned or cached elsewhere.
• For GitOps workflows (ArgoCD, Flux) specifically, this is why Sealed Secrets or an ExternalSecrets-style reference exists: the reconciler needs something committed to Git describing the secret, and that something must not be the plaintext value.
• Use the CI platform's own masked secret storage (GitHub Actions secrets, GitLab CI variables, and equivalents) for pipeline-time credentials, and avoid echoing them into build logs, where masking can fail on transformed or partial values.
• Run a pre-commit or PR-gate secret scanner (a gitleaks/truffleHog-class tool) so an accidental plaintext secret is caught before merge, not after.
• Inject pipeline-time credentials as short-lived tokens (a CI-specific Vault role, or a cloud workload-identity federation token) rather than a long-lived static credential baked into pipeline configuration.

Mounting practices

Prefer in-memory or CSI-driven mounts over writing secrets to a persistent, world-readable location on disk; where a file mount is unavoidable, restrict file permissions and avoid logging the file's path in a way that invites accidental cat-ing during debugging.

Auditing

Centralize Vault audit-device logs, cloud KMS key-access logs, and API server audit logs into one place (a SIEM or log aggregation system) and alert specifically on abnormal secret-access patterns (an unfamiliar identity requesting a production database credential, a spike in credential requests outside a deploy window), not just on the existence of access events, which are otherwise too frequent to review manually.

Trade-offs and pitfalls

• Native Secrets with etcd encryption at rest is acceptable for genuinely low-sensitivity configuration, but treating it as sufficient for production database or API credentials skips rotation, leasing, and fine-grained audit entirely.
• Sealed Secrets solves the Git-storage problem specifically; it does not solve in-cluster exposure, since the decrypted value becomes an ordinary Secret object once unsealed, same runtime exposure as the native case.
• The single most common operational miss is rotating the value at the secret store while forgetting that environment-variable consumption needs an explicit Pod restart to actually pick it up; verify the consumption method before declaring a rotation complete.

Direct answer

Gatekeeper enforces policy as a validating admission webhook: a ConstraintTemplate defines reusable Rego (OPA's policy language) logic and the CRD (Custom Resource Definition) shape it is configured with, and a Constraint is an instance of that template scoped to specific kinds and namespaces. For "deny privileged pods or pods missing resource limits," the template checks spec.containers[].securityContext.privileged and resources.limits.cpu/resources.limits.memory across every container and returns one violation message per offending container. The safe way to ship it is audit-only first, then targeted enforcement, never enforce cluster-wide on day one against an unaudited cluster.

Structured elaboration

ConstraintTemplate: the reusable Rego logic

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredpodsecurityandresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredPodSecurityAndResources
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredpodsecurityandresources

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          c.securityContext.privileged == true
          msg := sprintf("container '%v' is privileged", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.cpu
          msg := sprintf("container '%v' is missing resources.limits.cpu", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.memory
          msg := sprintf("container '%v' is missing resources.limits.memory", [c.name])
        }

This is Rego v0 syntax, still Gatekeeper's default. Gatekeeper 3.19 and later also supports opt-in Rego v1, which requires an explicit if before each rule body, but v0 remains what ships by default and what most existing ConstraintTemplates use.

Constraint: applying the template

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredPodSecurityAndResources
metadata:
  name: deny-privileged-or-no-limits
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]

enforcementAction: dryrun records violations without blocking anything, the correct starting state for any new policy on an existing cluster.

Validating vs mutating admission, and when to reach for either over OpenAPI schema validation

Gatekeeper is a validating admission webhook: it can accept or reject an object but cannot change it. A mutating admission webhook runs earlier in the chain and can rewrite the object before it is persisted, for example a sidecar injector adding a container, or a default value filled in. Kubernetes ships built-in admission controllers doing exactly these two jobs without any webhook at all, and they are the closest analogues to what Gatekeeper does here: LimitRanger (mutating; injects default resource requests and limits when a pod omits them) and ResourceQuota (validating; rejects a request that would exceed a namespace's aggregate quota).

The choice between a webhook and plain OpenAPI schema validation on a CRD comes down to what the rule needs to know:

• If the rule is fully expressible as a shape constraint on one object in isolation (a field must be one of an enum, a string must match a pattern, a number must sit in a range), OpenAPI schema validation on the CRD costs nothing at admission time and needs no separate service running.
• Reach for a webhook only when the rule needs something schema validation cannot express: cross-field logic (a field is required only if another field has a certain value), cross-object lookups (checking a Secret exists, checking sibling objects against a quota), or a policy that must apply uniformly across many unrelated resource kinds, exactly the "any Pod, any namespace" shape of the privileged/no-limits policy here.

Safe rollout plan

1. Deploy in enforcementAction: dryrun; let it run against real traffic for a full deploy cycle while pulling violations from Gatekeeper's audit results.
2. Share violations with owning teams with the exact fix needed (add resources.limits.cpu/memory, remove privileged: true), not just "you're non-compliant."
3. Flip to enforcementAction: deny first in one low-risk namespace, watch for unexpected rejections, then expand namespace by namespace.
4. Keep excludedNamespaces narrow and explicit, system namespaces only; a broad exclusion list defeats the point of a cluster-wide policy.

Worked example

A pod with two containers: one declares resources.limits: {cpu: "500m", memory: "256Mi"} and passes cleanly; the other has no resources block at all. The template's second and third rules each fire once for the second container, producing two separate violation messages ("missing resources.limits.cpu" and "missing resources.limits.memory"). That per-container, per-field granularity is what makes the audit output actionable rather than a single opaque "pod rejected."

Trade-offs and pitfalls

• Rego policy is powerful but opaque to most application developers; ship it with plain-language violation messages as above, and do not expect teams to read Rego to understand why they were blocked.
• excludedNamespaces is a blunt instrument. Overusing it to unblock a team quickly erodes the policy's coverage silently; track exclusions the same way you would track a firewall exception.
• A validating webhook adds a synchronous hop to every matched request. Keep the Rego evaluation cheap, no external calls, so it does not become the very apiserver latency problem it would otherwise be diagnosing.

Namespaces partition a single cluster into virtual clusters, mainly for scoping quotas, policies, and access control, not for hard security isolation on their own (that also needs NetworkPolicy). Role-based access control (RBAC) governs who, or what workload, can perform which verbs (get, list, create, update, delete) against which resources. In RBAC, Role and ClusterRole define the what (a set of permissions); RoleBinding and ClusterRoleBinding define the who and attach that permission set to subjects (users, groups, or ServiceAccounts). A ServiceAccount is the identity a pod uses to talk to the API server. Least-privilege in a multi-team cluster comes from combining namespace-per-team boundaries with narrowly scoped Roles and per-workload ServiceAccounts, not from handing out ClusterRoles.

The four RBAC objects

The mechanic most people miss: a RoleBinding can reference a ClusterRole. When it does, the ClusterRole's permissions are limited to the RoleBinding's own namespace. That is exactly how the built-in view, edit, and admin ClusterRoles are meant to be consumed: define the permission set once as a ClusterRole, then reuse it per-namespace via ordinary RoleBindings instead of writing a near-identical Role in every namespace. cluster-admin and ClusterRoleBinding are the only combination that actually grants cluster-wide, unrestricted access; everything else can be scoped down to a namespace even when it starts from a ClusterRole.

ServiceAccounts and workload identity

Every pod runs as a ServiceAccount (the default one in its namespace if you don't set one). Since Kubernetes 1.24, ServiceAccounts no longer get a long-lived Secret-based token auto-created and auto-mounted; tokens are now short-lived, audience-bound, and obtained through the TokenRequest API, delivered to the pod as a projected volume that Kubernetes refreshes automatically. Practically this means: don't design around "the ServiceAccount's Secret," and if you need a long-lived token for an out-of-cluster consumer you must request one explicitly.

Least-privilege patterns across teams and environments

• Namespace per team-per-environment (team-a-dev, team-a-prod), each with its own Roles rather than one shared namespace with broad access.
• One ServiceAccount per workload (not one shared "app" account per namespace), so a compromised pod's blast radius is exactly that workload's permissions.
• Write Roles with explicit verb and resource lists; avoid resources: ["*"] or verbs: ["*"]. A Role that grants everything is functionally a ClusterAdmin-lite and defeats the point.
• Reuse the built-in view/edit ClusterRoles via namespaced RoleBindings for read-only or standard app-team access instead of hand-rolling near-duplicates.
• Enforce the guardrails you can't trust humans to remember with a validating admission webhook, for example OPA (Open Policy Agent) Gatekeeper or Kyverno, to reject wildcard verbs or overly broad bindings at creation time rather than catching them in an audit later.
• Keep this distinct from pod-level security: RBAC controls API access; it says nothing about what a container can do on the node (privileged mode, host mounts). Since PodSecurityPolicy was removed in Kubernetes 1.25, that separate concern is covered by Pod Security Admission (PSA), not RBAC.

Worked example

A CI runner needs to deploy to team-a-dev only, never to team-a-prod.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: team-a-dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: team-a-dev
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: team-a-dev
subjects:
- kind: ServiceAccount
  name: ci-deployer
  namespace: team-a-dev
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

This ServiceAccount can get/update/patch Deployments in team-a-dev and nothing else: no Secrets, no other namespaces, no cluster-scoped objects. Production deploys use a completely separate ServiceAccount and Role bound only in team-a-prod, typically gated behind a different pipeline stage with its own approval step.

Trade-offs and pitfalls

• ClusterRoleBinding sprawl is the most common real-world RBAC failure: it's easy to grant a ClusterRole via ClusterRoleBinding "to unblock someone quickly" and never revisit it. Prefer the namespaced-RoleBinding-to-a-ClusterRole pattern above instead.
• Wildcard verbs or resources ("*") are the second most common failure; they're invisible in a quick review of the RoleBinding but grant far more than intended.
• Audit what's actually bound, not just what you intended to bind; the kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> command (or the community kubectl-who-can plugin) will tell you the effective permissions for a subject, which is the only reliable way to catch drift.
• Namespaces are a policy and quota boundary, not a hard security boundary by themselves; don't rely on RBAC alone to stop a compromised pod from reaching another namespace's pods over the network. That needs NetworkPolicy in addition.