Kubernetes Architecture, Operations, and Troubleshooting Questions

Direct answer

Online PersistentVolumeClaim (PVC) expansion needs three things to line up: the StorageClass must set allowVolumeExpansion: true, the CSI (Container Storage Interface) driver must implement the ControllerExpandVolume and NodeExpandVolume calls, and the filesystem plus kubelet must support growing in place (resize2fs for ext4, xfs_growfs for xfs). When all three hold, resizing from 100Gi to 200Gi is a one-line patch to the PVC with no pod restart. When the node-side piece is missing, the pod needs a restart to pick up the new size, which is the only source of downtime in the whole process.

Structured elaboration

Preconditions

$ kubectl get storageclass fast-ssd -o jsonpath='{.allowVolumeExpansion}'
true

If this is empty or false, expansion is rejected at the PVC level before anything else matters. Patching allowVolumeExpansion onto an existing StorageClass is safe; it only affects future resize requests, not past provisioning.

Steps to go from 100Gi to 200Gi

$ kubectl patch pvc data-pvc -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
persistentvolumeclaim/data-pvc patched

Kubernetes then drives two calls against the CSI driver in sequence:

1. ControllerExpandVolume, issued by the external-resizer sidecar, grows the underlying block device (or cloud volume) and updates the PersistentVolume's recorded capacity.
2. NodeExpandVolume, issued by the kubelet on whichever node has the volume mounted, grows the filesystem to fill the new device size.

Watch the PVC's events to see this happen:

$ kubectl describe pvc data-pvc
Events:
  Type    Reason                     Message
  Normal  Resizing                   External resizer is resizing volume
  Normal  FileSystemResizeSuccessful MountVolume.NodeExpandVolume succeeded for volume "pvc-..."

If the CSI driver only supports offline expansion, the event sequence instead pauses at FileSystemResizePending. The fix is to let the pod restart (respecting any PodDisruptionBudget) so the kubelet can run NodeExpandVolume against the volume once it is unmounted and remounted.

Minimizing downtime for the offline case

For a single-replica stateful workload the restart is unavoidable but brief. For a replicated one, cordon and evict one replica at a time so the others keep serving, exactly the pattern a PodDisruptionBudget protects during any voluntary disruption, resize included.

Worked example

A 100Gi PVC backed by a CSI driver that supports online expansion is patched to 200Gi. ControllerExpandVolume grows the cloud volume to 200Gi and the PersistentVolume object's capacity updates to match; NodeExpandVolume then runs resize2fs (or the xfs equivalent) against the already-mounted filesystem, and kubectl describe pvc shows FileSystemResizeSuccessful without the pod ever restarting. If the same driver only supported offline expansion, the sequence would stop at FileSystemResizePending until the pod is deleted and rescheduled, at which point the kubelet completes the resize during the new mount.

Trade-offs and pitfalls

• PVCs can only be resized upward; shrinking is not supported the same way.
• Not every CSI driver implements online (in-place) expansion. Check the specific driver's documentation rather than assuming; "supports expansion" and "supports expansion without a restart" are different claims.
• Resize during low traffic and keep a snapshot beforehand. The resize itself does not touch existing data, but any manual filesystem operation carries risk if interrupted.

Run the database through a purpose-built operator (a controller that encodes that specific database's operational knowledge as a CRD, Custom Resource Definition) rather than a hand-rolled StatefulSet, back it with topology-aware block storage, protect quorum with a PodDisruptionBudget (PDB) and anti-affinity, and treat backup/restore and failure recovery as first-class design requirements, not afterthoughts. The hardest part is not storage or scaling, it's guaranteeing zero data loss without a split-brain during failover, which needs an explicit fencing mechanism the platform doesn't give you for free.

Requirements first

Pin down RPO/RTO (Recovery Point/Time Objective) targets, expected throughput, latency budget, number of Availability Zones (AZs), and whether managed-DB-as-a-service is politically or contractually off the table; these decide almost everything below.

Three-way comparison: plain StatefulSet vs operator vs managed DB

For production, an operator is almost always the right middle ground unless a managed service is available and acceptable; a bare StatefulSet without an operator means re-implementing failover, backup orchestration, and safe upgrades yourself, which is exactly the hard part.

Concrete examples of DB-specific operators: CloudNativePG or the Zalando operator for PostgreSQL, Strimzi for Kafka, ECK (Elastic Cloud on Kubernetes) for Elasticsearch. Worth noting for currency: Kafka 4.0 (released March 2025) dropped ZooKeeper support entirely and runs KRaft-only, so a current Strimzi-managed Kafka cluster is one StatefulSet-backed topology (the brokers, using the KRaft consensus protocol for metadata) rather than two separate stateful systems (brokers plus a ZooKeeper ensemble) as older deployments required.

Architecture

flowchart TB
    OP[Database Operator / CRD controller]
    SS[StatefulSet: ordered, stable Pod identities]
    PDB[PodDisruptionBudget: protects quorum]
    SC[Topology-aware StorageClass, CSI provisioner]
    BK[Backup: snapshots + WAL/binlog shipping to object storage]
    OP --> SS
    OP --> PDB
    SS --> SC
    OP --> BK
    SS -->|anti-affinity| AZ1[AZ 1: primary]
    SS -->|anti-affinity| AZ2[AZ 2: sync replica]
    SS -->|anti-affinity| AZ3[AZ 3: async replica / DR]

Storage layer

• CSI (Container Storage Interface)-backed StorageClass with volumeBindingMode: WaitForFirstConsumer, so the volume is provisioned in the same zone the scheduler actually places the Pod, avoiding an unschedulable Pod stuck waiting on a volume in the wrong zone.
• reclaimPolicy: Retain on the StorageClass: an accidental PVC (PersistentVolumeClaim) deletion must not silently destroy the primary's data; recovery should require a deliberate, human step.
• Block storage versus a distributed filesystem, explicitly. Zonal block storage (EBS/PD/Azure Disk-style volumes) gives the lowest latency per replica but is single-zone and single-writer; the database's own replication protocol is what provides cross-zone durability, not the storage layer. A distributed filesystem or storage system (Ceph, Portworx, and similar) instead replicates data itself across nodes/zones at the storage layer, trading some latency and operational complexity for a storage layer that doesn't rely entirely on the database's own replication being correctly configured. For a workload with a mature, well-tested internal replication protocol (most production RDBMS and Kafka), zonal block storage plus database-level replication is usually the better latency/complexity trade-off; a distributed storage layer is more attractive for workloads with weak or no built-in replication.

The zero-data-loss and fencing problem

This is the part a plain StatefulSet does not solve for you. If a primary becomes unreachable but is not actually dead (a network partition, not a crash), promoting a replica while the old primary is still accepting writes creates two primaries accepting conflicting writes, a split-brain, which is a correctness failure, not just an availability one. A correct design needs explicit fencing: before a new primary is promoted, the old one must be provably prevented from accepting further writes, whether by a storage-level fence (revoking its volume attachment), a network-level fence (an admission gate the operator controls), or a consensus-based lease/term mechanism where a promoted replica only becomes writable once it holds a lease the old primary cannot renew. Most mature operators build this in (Kafka's KRaft controller quorum and Postgres operators using a distributed lock like Patroni's approach are both fencing-aware); a hand-rolled StatefulSet failover script that just runs a promote command is not safe against a network partition unless it independently solves this same problem.

PodDisruptionBudget and placement

• PodDisruptionBudget with minAvailable set so voluntary disruptions (node drains, cluster upgrades) can never take out enough replicas to lose quorum, for example minAvailable: 2 on a 3-replica set.
• Required (hard) pod anti-affinity across zones so replicas never land on the same node or, ideally, the same zone, or a single zone failure takes out more than one replica at once.
• Readiness probes gate traffic during recovery; liveness probes tuned to the database's actual health semantics, not a generic TCP check.

3-AZ replication topology

A common shape for regional durability without full cross-region cost: synchronous replication between the primary and one replica in a second AZ (so a confirmed write survives losing either AZ, at the cost of added write latency for the round trip), plus asynchronous replication to a third AZ or a separate region for disaster recovery, accepting a small, bounded RPO there in exchange for avoiding synchronous replication's latency cost across a third leg. The trade-off is explicit: every synchronous replica added tightens RPO but adds write latency; asynchronous replicas protect against a wider blast radius (a whole-region event) but carry a real, non-zero RPO if the primary is lost before the async replica catches up.

Backup and restore

• Application-consistent snapshots via CSI Volume Snapshots for fast full backups.
• Continuous WAL (write-ahead log) or binlog shipping for point-in-time recovery, usually built into the operator.
• Periodic backups to durable object storage (S3/GCS/Azure Blob-equivalent) with encryption and a lifecycle/retention policy.
• Restore drills on a schedule, not just documented in a runbook; an untested restore path is not a real recovery capability.

Handling node failures and scaling

• Node failure: the operator promotes a healthy replica (using the fencing mechanism above), Kubernetes reschedules the failed Pod elsewhere, and the storage layer either reattaches the same volume (single-zone block storage on a healthy node) or the operator rebuilds a fresh replica from backup/replication if the volume itself is unavailable.
• Vertical scaling: usually requires a coordinated, operator-sequenced resize or a planned failover, not a live in-place change.
• Horizontal scaling: read scaling via additional replicas behind a read-only Service; write scaling (if needed at all) requires sharding at the application or a purpose-built layer (e.g. Citus for Postgres, Vitess for MySQL), which is a materially different architecture, not a StatefulSet replica-count change.

Trade-offs and pitfalls

• Treating "add a PodDisruptionBudget and anti-affinity" as sufficient without addressing fencing is the most dangerous gap: it protects against voluntary disruption and Pod placement risk but says nothing about split-brain during an involuntary, ambiguous failure like a network partition.
• Choosing a distributed storage layer for its own replication when the database already replicates well is redundant complexity and extra latency for no real durability gain; verify what layer is actually providing the durability guarantee you're relying on.
• An untested backup is not a backup; the failure mode where a restore procedure is discovered to be broken happens exactly once, during a real incident.
• Running a fully DIY StatefulSet without an operator is a legitimate choice only for genuinely low-stakes or learning environments; for anything production-grade, the missing failover and fencing logic is a correctness gap, not just an operational inconvenience.

A PersistentVolume (PV) is a cluster-scoped object representing an actual piece of storage (a cloud disk, an NFS share, and so on); a PersistentVolumeClaim (PVC) is a namespaced request for storage that a Pod actually references. Kubernetes exists to decouple the two: application manifests ask for storage by claim, never by naming a specific disk.

Binding

• Static: an admin pre-creates PVs ahead of time. When a PVC is created, the control plane looks for an existing PV whose capacity, access mode, and (if specified) StorageClass match, and binds them; the PV's status.phase becomes Bound.
• Dynamic: if the PVC references a StorageClass and no matching PV exists yet, the StorageClass's provisioner (a CSI, Container Storage Interface, driver) creates a new PV on demand and binds it automatically. This is the default path in most modern clusters and needs no manual admin step per volume.

accessModes and volumeMode: the other fields people gloss over

volumeMode is a separate field from accessModes:

• Filesystem (default): the volume is formatted with a filesystem and mounted as a directory the container sees normally.
• Block: the volume is exposed to the container as a raw block device with no filesystem at all; used by workloads (some databases, specialized storage software) that manage their own on-disk format and want direct block access rather than going through a filesystem layer.

reclaimPolicy

Set on the PV, this decides what happens to the underlying storage once its PVC is deleted:

• Delete: the underlying storage is destroyed. The default for dynamically provisioned volumes.
• Retain: the storage and its data are kept; an administrator has to manually inspect and clean it up. Preferred for anything holding data you can't regenerate.
• Recycle is deprecated and should be treated as unavailable on a current cluster.

StorageClass and dynamic provisioning

A StorageClass names a provisioner and its parameters (disk type, IOPS, encryption, and similar) plus a reclaimPolicy. A PVC that references a StorageClass triggers that provisioner to create a matching PV automatically, which is what makes dynamic provisioning possible at all: without a StorageClass, every PV needs a human to create it ahead of time.

When PVCs are preferable to ephemeral volumes

Use a PVC whenever data has to survive something the Pod itself doesn't survive: a restart, a reschedule to a different node, or a node failure. Concretely: databases, message queue storage, anything with an SLO (service-level objective) tied to data durability. Ephemeral volumes (emptyDir, ephemeral CSI volumes) are the right choice when data loss on Pod termination is acceptable, for scratch space, caches, or stateless replicas that regenerate their working data on startup.

Worked example

A stateful application needs one 20Gi read-write volume that must survive a Pod reschedule:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes: ["ReadWriteOnce"]
  volumeMode: Filesystem
  resources:
    requests:
      storage: 20Gi
  storageClassName: fast-ssd

The PVC binds to a dynamically provisioned 20Gi PV from the fast-ssd StorageClass; if the Pod is rescheduled (say, after a node failure), the same PVC reattaches to the same underlying data, whereas an emptyDir volume would have been created empty on the new node with the old data simply gone.

Trade-offs and pitfalls

• A PVC that stays Pending usually means either no PV matches it (static case) or the StorageClass's provisioner can't satisfy the request (wrong zone under WaitForFirstConsumer, quota exhausted, or a typo'd StorageClass name); check kubectl describe pvc for the actual event before guessing.
• Resizing a PVC safely requires the StorageClass to have allowVolumeExpansion: true; even then, some volume types only support growing, never shrinking, and some require a Pod restart to pick up the new size at the filesystem level.
• Plain ReadWriteOnce's same-node multi-Pod loophole is a common source of confusion; if the real requirement is "exactly one Pod, full stop," ReadWriteOncePod is the access mode that actually guarantees that on a current CSI-backed cluster.

A StorageClass is a cluster-scoped object that tells Kubernetes how to dynamically create storage for a PersistentVolumeClaim (PVC, a namespaced request for storage): which driver provisions it, what parameters to pass that driver, and what happens to the underlying volume when its claim is deleted.

The important fields

How the CSI model makes this work

The provisioner field names a CSI driver, the standard interface Kubernetes uses so any storage vendor can plug in a compliant driver without Kubernetes needing vendor-specific code built in. In practice, a CSI driver ships as a controller-side component (which handles create/delete/attach requests, typically via sidecar containers like external-provisioner and external-attacher that watch PVC and VolumeAttachment objects and translate them into gRPC calls against the actual driver) and a node-side component (a DaemonSet that mounts the resulting volume into the Pod on whichever node it lands on). The StorageClass is the configuration Kubernetes hands to that controller-side component whenever a PVC requests dynamic provisioning.

Dynamic vs static provisioning

• Dynamic: a PVC references a StorageClass; the CSI provisioner creates a matching PersistentVolume automatically. This is the default, recommended path, since it needs no manual admin step per volume and scales with the number of workloads.
• Static: an admin pre-creates PersistentVolumes ahead of time (for pre-existing storage, special hardware, or a resource that can't be provisioned on demand), and a PVC binds to one of them by matching capacity and access mode. No StorageClass is required for static binding, though a PVC can still be written to only match pre-provisioned volumes by leaving storageClassName unset or matching it explicitly.

Why WaitForFirstConsumer and Retain matter for multi-AZ clusters

• volumeBindingMode: WaitForFirstConsumer: with Immediate binding in a multi-zone cluster, the provisioner can create a volume in Zone A before the scheduler has decided where the Pod will actually run; if the scheduler then places the Pod in Zone B (because that's where capacity or other constraints point), the Pod is stuck Pending forever, since most block storage can't attach across zones. WaitForFirstConsumer fixes this by waiting until the Pod is scheduled first, then provisioning the volume in the same zone the Pod actually landed in.
• reclaimPolicy: Retain: in a multi-AZ, stateful-workload context, an accidental PVC deletion (a bad kubectl delete, a mistaken namespace teardown) with the default Delete policy destroys the underlying data immediately and irreversibly. Retain trades convenience for a manual recovery path: the storage and its data survive, and an admin can inspect, snapshot, or manually rebind it before anything is actually destroyed. Production stateful workloads (databases especially) commonly override the default to Retain for exactly this reason.

Small worked fragment

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-multiaz
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

When to mark a StorageClass as the cluster default

Mark a class as default only when it's the right choice for the majority of workloads that don't specify one explicitly, typically a general-purpose, moderate-cost, moderate-performance tier. Don't default to a niche, expensive, or retention-sensitive class (a high-IOPS tier, or anything using Retain) since Pods that forget to specify a class would silently get an unexpectedly costly or hard-to-clean-up volume. Critical workloads should reference their StorageClass explicitly regardless of what's marked default, so a later change to the cluster default can't silently change their storage tier out from under them.

Trade-offs and pitfalls

• Having more than one StorageClass marked default is a real, common misconfiguration; Kubernetes doesn't reliably pick one for you, and PVCs that omit storageClassName can end up on whichever one the admission logic happens to select.
• reclaimPolicy: Delete combined with a fast, careless cleanup script is the most common way production data gets destroyed accidentally; Retain is the safer default for anything holding data you can't regenerate.
• WaitForFirstConsumer delays seeing a bound PersistentVolume until a Pod actually exists, which can be confusing during manual troubleshooting if you expect to see an immediately-bound PV the moment you create a PVC.

Cloud and network-backed storage for Kubernetes splits into two families: block storage (AWS EBS, GCE PD, Azure Disk), which is fast and durable but normally attachable to only one node at a time, and network filesystems (AWS EFS, Azure Files, self-managed NFS), which are shareable across many nodes at once but pay a latency and throughput cost for that flexibility. Picking between them is really picking whether the workload needs raw single-writer performance or multi-node shared access.

Comparison

How to choose

• Single-writer, latency-sensitive, durable (a relational database's primary, a message queue's log): block storage (EBS, GCE PD, Azure Disk). Access mode ReadWriteOnce, sized and provisioned for the IOPS the workload actually needs.
• Shared, multi-reader-or-writer, latency-tolerant (shared configuration, static assets, a CI build cache used by many concurrent jobs): a managed network filesystem (EFS, Azure Files) if available on your cloud, or self-managed NFS if not, understanding that NFS's durability and availability are now your responsibility to engineer.
• Regional or multi-zone resilience for a block-storage workload: look at the provider's own cross-zone replication option (GCE's regional Persistent Disk is the clearest example) rather than assuming ordinary zonal block storage survives a zone failure; ordinary zonal EBS/PD/Azure Disk does not.

Worked example

A team needs (a) a primary Postgres (Postgres) volume and (b) a shared directory of report templates read by 20 replica Pods across multiple nodes.

• (a) is single-writer and latency-sensitive: provision an EBS/GCE PD/Azure Disk volume through a StorageClass with ReadWriteOnce, sized for the database's IOPS profile.
• (b) needs concurrent multi-node reads: provision an EFS/Azure Files/NFS volume through a StorageClass supporting ReadWriteMany, since a block-storage volume cannot satisfy that access pattern at all, regardless of performance tier.

Trade-offs and pitfalls

• Don't reach for a network filesystem by default "to be safe" for multi-node access; if the workload is genuinely single-writer, block storage's lower latency is the better fit and the shared-filesystem's variability is pure downside.
• Zonal block storage (the common case for EBS/GCE PD/Azure Disk) does not survive the loss of its Availability Zone; if that's a real requirement, either use the provider's cross-zone replicated variant where one exists, or handle replication at the application layer (e.g., a database's own streaming replication to a replica in another zone) rather than assuming the storage layer covers it.
• A "multi-writer" flag on a block-storage product is not the same guarantee as a real shared filesystem: it typically still requires a cluster-aware filesystem and application-level write coordination, so verify exactly what's supported for your disk type before relying on it, rather than assuming ReadWriteMany-equivalent behavior.