Kubernetes Architecture, Operations, and Troubleshooting Questions

A PersistentVolume (PV) is a cluster-scoped object representing an actual piece of storage (a cloud disk, an NFS share, and so on); a PersistentVolumeClaim (PVC) is a namespaced request for storage that a Pod actually references. Kubernetes exists to decouple the two: application manifests ask for storage by claim, never by naming a specific disk.

Binding

• Static: an admin pre-creates PVs ahead of time. When a PVC is created, the control plane looks for an existing PV whose capacity, access mode, and (if specified) StorageClass match, and binds them; the PV's status.phase becomes Bound.
• Dynamic: if the PVC references a StorageClass and no matching PV exists yet, the StorageClass's provisioner (a CSI, Container Storage Interface, driver) creates a new PV on demand and binds it automatically. This is the default path in most modern clusters and needs no manual admin step per volume.

accessModes and volumeMode: the other fields people gloss over

volumeMode is a separate field from accessModes:

• Filesystem (default): the volume is formatted with a filesystem and mounted as a directory the container sees normally.
• Block: the volume is exposed to the container as a raw block device with no filesystem at all; used by workloads (some databases, specialized storage software) that manage their own on-disk format and want direct block access rather than going through a filesystem layer.

reclaimPolicy

Set on the PV, this decides what happens to the underlying storage once its PVC is deleted:

• Delete: the underlying storage is destroyed. The default for dynamically provisioned volumes.
• Retain: the storage and its data are kept; an administrator has to manually inspect and clean it up. Preferred for anything holding data you can't regenerate.
• Recycle is deprecated and should be treated as unavailable on a current cluster.

StorageClass and dynamic provisioning

A StorageClass names a provisioner and its parameters (disk type, IOPS, encryption, and similar) plus a reclaimPolicy. A PVC that references a StorageClass triggers that provisioner to create a matching PV automatically, which is what makes dynamic provisioning possible at all: without a StorageClass, every PV needs a human to create it ahead of time.

When PVCs are preferable to ephemeral volumes

Use a PVC whenever data has to survive something the Pod itself doesn't survive: a restart, a reschedule to a different node, or a node failure. Concretely: databases, message queue storage, anything with an SLO (service-level objective) tied to data durability. Ephemeral volumes (emptyDir, ephemeral CSI volumes) are the right choice when data loss on Pod termination is acceptable, for scratch space, caches, or stateless replicas that regenerate their working data on startup.

Worked example

A stateful application needs one 20Gi read-write volume that must survive a Pod reschedule:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes: ["ReadWriteOnce"]
  volumeMode: Filesystem
  resources:
    requests:
      storage: 20Gi
  storageClassName: fast-ssd

The PVC binds to a dynamically provisioned 20Gi PV from the fast-ssd StorageClass; if the Pod is rescheduled (say, after a node failure), the same PVC reattaches to the same underlying data, whereas an emptyDir volume would have been created empty on the new node with the old data simply gone.

Trade-offs and pitfalls

• A PVC that stays Pending usually means either no PV matches it (static case) or the StorageClass's provisioner can't satisfy the request (wrong zone under WaitForFirstConsumer, quota exhausted, or a typo'd StorageClass name); check kubectl describe pvc for the actual event before guessing.
• Resizing a PVC safely requires the StorageClass to have allowVolumeExpansion: true; even then, some volume types only support growing, never shrinking, and some require a Pod restart to pick up the new size at the filesystem level.
• Plain ReadWriteOnce's same-node multi-Pod loophole is a common source of confusion; if the real requirement is "exactly one Pod, full stop," ReadWriteOncePod is the access mode that actually guarantees that on a current CSI-backed cluster.

A pod moves through Pending (accepted but not yet running), ContainerCreating (scheduled, kubelet is pulling images and mounting volumes), Running (at least one container is up), and finally either Succeeded or Failed for a pod that's meant to terminate, or a restart loop such as CrashLoopBackOff for one that keeps failing and coming back. Unknown is different in kind from the rest: it means the API server has simply lost contact with the node, not that anything about the pod itself is known to be wrong.

States, what each implies, and where to look

A sample of what the Events section actually looks like for a crash-looping pod:

$ kubectl describe pod worker-6b9f-x2z4p -n prod
...
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
Ready:          False
Restart Count:  6
Events:
  Type     Reason   Age                From     Message
  ----     ------   ----               ----     -------
  Warning  BackOff  38s (x5 over 90s)  kubelet  Back-off restarting failed container

OOMKilled here means the kernel's out-of-memory killer terminated the container because it exceeded its memory limit, not that the application itself crashed; that distinction changes the fix (raise the memory limit or reduce the footprint, versus debug application logic).

Graceful termination, and how it relates to Failed vs a clean stop

When a pod is deleted (including during a rolling update), Kubernetes first removes it from Service endpoints so it stops receiving new traffic, then sends SIGTERM to the container, runs any preStop hook first if one is defined, and waits up to terminationGracePeriodSeconds (30 seconds by default) before sending SIGKILL. A container that ignores SIGTERM and needs the full grace period before being force-killed can look, from the outside, like a slow or stuck termination rather than a clean stop; tuning the grace period and handling SIGTERM in the application are what make a shutdown graceful instead of abrupt.

Trade-offs and pitfalls

• Running is frequently mistaken for "healthy." A container can be Running while its readiness probe fails continuously, meaning it's alive but not receiving traffic, which shows up in status.conditions (Ready: False), not in the pod phase.
• Failed versus CrashLoopBackOff is really about restartPolicy: with Always (the Deployment default) a failing container becomes CrashLoopBackOff because Kubernetes keeps retrying with backoff; with Never or OnFailure in the wrong combination a similar failure surfaces as Failed instead, which changes which command shows you the useful evidence (--previous logs only apply once a restart has actually happened).

A ConfigMap and a Secret are both key-value objects for feeding configuration into a pod without baking it into the image, but a Secret is meant for sensitive values and Kubernetes handles it slightly differently. Neither is encrypted by default: Secret values are only base64-encoded in etcd (the cluster's data store), which is trivially reversible, not encryption. Treat 'it's a Secret' as an access-control and audit boundary, not as cryptographic protection, unless encryption at rest has been explicitly turned on.

ConfigMap vs Secret

Two ways to expose a Secret to a pod

Environment variable:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: my-secret
        key: db-password

Mounted file:

volumes:
  - name: secret-vol
    secret:
      secretName: my-secret
containers:
  - name: app
    volumeMounts:
      - name: secret-vol
        mountPath: /etc/creds
        readOnly: true

Worked example: what actually happens when a Secret's value changes

Trace what happens after my-secret's db-password key is updated, depending on how it was exposed:

• Env var: the running pod keeps using the old value until it is restarted or replaced. Environment variables are read once at container start and never update in place.
• Mounted file, no subPath: the file at /etc/creds/db-password does eventually update, once the kubelet's periodic sync catches up (on the order of a minute, not instantly), but the running process only sees the change if it re-reads the file itself, since nothing forces that.
• Mounted file with subPath: it never updates at all, because a subPath mount is bound to a specific file version at mount time.
  Neither the env-var path nor the volume path forces a pod restart on a config change. The common pattern to actually guarantee a fresh pod is to hash the ConfigMap or Secret's content into a pod template annotation (Helm's checksum/config annotation is the usual form); a content change then produces a different pod template hash, which forces a real rollout instead of relying on an in-place file update the application may not even notice.

Trade-offs and pitfalls

• Base64 is not encryption; anyone with read access to Secret objects, or to etcd's data files directly, can decode it in one command. Enabling encryption at rest (an EncryptionConfiguration backed by a Key Management Service, KMS, provider, the current recommended approach since the older static-key KMS v1 API was deprecated as of Kubernetes 1.28) protects the etcd-at-rest copy; Role-Based Access Control (RBAC) is what actually protects who can read the Secret object in the first place, and the two are not substitutes for each other.
• Environment variables are easy to leak: they show up when describing a running pod, in crash dumps, and in some logging frameworks that log the process environment; prefer mounted files for anything sensitive when the application can read from a file path instead.
• For continuous integration and continuous delivery (CI/CD) pipelines, never let plaintext secrets sit in pipeline configuration or version control; use the pipeline platform's own secret store, scope credentials as narrowly and as short-lived as possible, and prefer pulling secrets at deploy time from an external manager (HashiCorp Vault, a cloud provider's secrets manager, or a Secrets Store CSI driver) over baking them into a committed manifest, even one covered by a .gitignore entry.

Pods are the runtime unit; everything else here is a controller deciding how many pods to run and how to replace them. A ReplicaSet just keeps N identical pods alive. A Deployment wraps a ReplicaSet with declarative rolling updates and rollback history. A StatefulSet keeps stable per-replica identity and storage for workloads that care which replica they are. A DaemonSet guarantees exactly one pod per matching node. Job and CronJob run pods to completion, once or on a schedule, rather than keeping them running indefinitely.

Comparison

Worked example: rolling-update arithmetic

For a Deployment with replicas: 10, maxSurge: 25%, and maxUnavailable: 25%, Kubernetes rounds surge up and unavailable down:
$\text{surge} = \lceil 10 \times 0.25 \rceil = 3$
$\text{unavailable} = \lfloor 10 \times 0.25 \rfloor = 2$
So at the busiest point of the rollout the Deployment can briefly run up to 13 pods (10 plus 3 surge) while guaranteeing at least 8 stay available (10 minus 2 unavailable). This asymmetric rounding is why setting surge and unavailable to the same percentage still lets the total pod count grow slightly during a rollout instead of staying flat.

DaemonSets interact with node taints in a way that is easy to miss: they commonly carry an explicit toleration for the control-plane taint (conventionally node-role.kubernetes.io/control-plane:NoSchedule) so node-level agents like a CNI plugin or kube-proxy itself still run on control-plane nodes that ordinary application pods are excluded from. That toleration is a manifest choice the DaemonSet author makes, not automatic behavior.

Trade-offs and pitfalls

• Choosing Deployment for a workload that actually needs stable identity (a small quorum service, a primary-replica database) forces the team to rebuild that identity logic in application code; StatefulSet gives it for free, at the cost of slower, ordered rollouts.
• A stuck rollout is usually a readiness-probe failure hiding behind the surge and unavailable math: if new pods never pass readiness, the Deployment can stall indefinitely partway through, with old and new pods coexisting. Check pod events and readiness status before assuming a controller bug.
• A PodDisruptionBudget set too strictly (for example, minAvailable equal to the full replica count) can make a Deployment's own rolling update unable to proceed, because the update's own temporary unavailability trips the budget meant to protect against unrelated disruptions.
• DaemonSets bypass normal replica-count thinking: adding nodes silently adds pods and cost. Forgetting a DaemonSet exists across a large node pool is a common source of untracked resource consumption.

CrashLoopBackOff means the container keeps exiting and the kubelet is backing off its restart attempts with exponential delay (10s, 20s, 40s, up to a 5-minute cap). It is a symptom, not a root cause: something inside the container is exiting, or something outside it is killing the container, and the job is to figure out which.

Step-by-step diagnosis

1. Describe the pod first. This shows the Events list and the container's last termination reason before you even look at logs.

kubectl describe pod <pod> -n <ns>

2. Read current and previous logs. The -p (previous) flag is what most people forget, and it's the only way to see output from the crashed instance once it has already restarted.

kubectl logs <pod> -c <container> -n <ns>
kubectl logs <pod> -c <container> -p -n <ns>

3. Check the exit code, visible in describe's Last State block.

4. Separate "app crashed" from "kubelet killed it." A liveness probe failure produces its own event and a different signature than an application-level crash:

Containers:
  app:
    State:      Waiting
      Reason:   CrashLoopBackOff
    Last State: Terminated
      Reason:   Error
      Exit Code: 1
    Restart Count: 5
Events:
  Warning  Unhealthy  90s (x3 over 3m)   kubelet  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": dial tcp 10.1.2.3:8080: connect: connection refused
  Warning  BackOff    30s (x12 over 6m)  kubelet  Back-off restarting failed container

If the events show Liveness probe failed right before the restart, the app may actually be starting fine but too slowly, or the probe is pointed at the wrong port or path, and kubelet is killing a container that would otherwise have recovered.
5. Check image and config next, since a large share of CrashLoopBackOff cases are not code bugs: wrong environment variable, a ConfigMap/Secret key that doesn't exist, a missing volume mount, or an image tag that doesn't match what's expected.

Reproduce and test a fix safely

• Recreate the same image/env in a disposable pod without a restart policy fighting you:

kubectl run debug --image=<image> --restart=Never -n dev --env="KEY=value" --command -- sleep 3600
kubectl exec -it debug -n dev -- /bin/sh

• If you suspect the liveness probe is the culprit, don't leave it disabled permanently; temporarily loosen initialDelaySeconds or failureThreshold in a copy of the manifest in a non-prod namespace, confirm the app actually stabilizes, then fix the probe config for real rather than shipping it disabled.
• Once you have a fix, patch and roll it out normally rather than editing the live object by hand:

kubectl set image deploy/<d> <c>=<image>:<new-tag> -n <ns>
kubectl rollout status deploy/<d> -n <ns>

• Capture the failing pod's YAML before you touch anything (kubectl get pod <pod> -o yaml), since the evidence disappears once the pod is replaced.

Trade-offs and pitfalls

• Don't "fix" a crash loop by disabling the liveness probe or setting restartPolicy more leniently; that hides the symptom and turns a visible failure into a silent one.
• Exit code 137 is a strong hint but not proof of OOMKilled; always confirm via the Reason field in describe, since a probe-triggered kill or an external kill -9 produces the same code.
• If a pod never produces logs at all before crashing, the failure is likely before your application's logging even initializes (missing dependency, bad entrypoint, immediate segfault); at that point kubectl logs -p is empty and you have to fall back to the exit code, describe events, and possibly running the image manually to see stdout that never reached the log driver.