Kubernetes Architecture, Operations, and Troubleshooting Questions

A StorageClass is a cluster-scoped object that tells Kubernetes how to dynamically create storage for a PersistentVolumeClaim (PVC, a namespaced request for storage): which driver provisions it, what parameters to pass that driver, and what happens to the underlying volume when its claim is deleted.

The important fields

How the CSI model makes this work

The provisioner field names a CSI driver, the standard interface Kubernetes uses so any storage vendor can plug in a compliant driver without Kubernetes needing vendor-specific code built in. In practice, a CSI driver ships as a controller-side component (which handles create/delete/attach requests, typically via sidecar containers like external-provisioner and external-attacher that watch PVC and VolumeAttachment objects and translate them into gRPC calls against the actual driver) and a node-side component (a DaemonSet that mounts the resulting volume into the Pod on whichever node it lands on). The StorageClass is the configuration Kubernetes hands to that controller-side component whenever a PVC requests dynamic provisioning.

Dynamic vs static provisioning

• Dynamic: a PVC references a StorageClass; the CSI provisioner creates a matching PersistentVolume automatically. This is the default, recommended path, since it needs no manual admin step per volume and scales with the number of workloads.
• Static: an admin pre-creates PersistentVolumes ahead of time (for pre-existing storage, special hardware, or a resource that can't be provisioned on demand), and a PVC binds to one of them by matching capacity and access mode. No StorageClass is required for static binding, though a PVC can still be written to only match pre-provisioned volumes by leaving storageClassName unset or matching it explicitly.

Why WaitForFirstConsumer and Retain matter for multi-AZ clusters

• volumeBindingMode: WaitForFirstConsumer: with Immediate binding in a multi-zone cluster, the provisioner can create a volume in Zone A before the scheduler has decided where the Pod will actually run; if the scheduler then places the Pod in Zone B (because that's where capacity or other constraints point), the Pod is stuck Pending forever, since most block storage can't attach across zones. WaitForFirstConsumer fixes this by waiting until the Pod is scheduled first, then provisioning the volume in the same zone the Pod actually landed in.
• reclaimPolicy: Retain: in a multi-AZ, stateful-workload context, an accidental PVC deletion (a bad kubectl delete, a mistaken namespace teardown) with the default Delete policy destroys the underlying data immediately and irreversibly. Retain trades convenience for a manual recovery path: the storage and its data survive, and an admin can inspect, snapshot, or manually rebind it before anything is actually destroyed. Production stateful workloads (databases especially) commonly override the default to Retain for exactly this reason.

Small worked fragment

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-multiaz
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

When to mark a StorageClass as the cluster default

Mark a class as default only when it's the right choice for the majority of workloads that don't specify one explicitly, typically a general-purpose, moderate-cost, moderate-performance tier. Don't default to a niche, expensive, or retention-sensitive class (a high-IOPS tier, or anything using Retain) since Pods that forget to specify a class would silently get an unexpectedly costly or hard-to-clean-up volume. Critical workloads should reference their StorageClass explicitly regardless of what's marked default, so a later change to the cluster default can't silently change their storage tier out from under them.

Trade-offs and pitfalls

• Having more than one StorageClass marked default is a real, common misconfiguration; Kubernetes doesn't reliably pick one for you, and PVCs that omit storageClassName can end up on whichever one the admission logic happens to select.
• reclaimPolicy: Delete combined with a fast, careless cleanup script is the most common way production data gets destroyed accidentally; Retain is the safer default for anything holding data you can't regenerate.
• WaitForFirstConsumer delays seeing a bound PersistentVolume until a Pod actually exists, which can be confusing during manual troubleshooting if you expect to see an immediately-bound PV the moment you create a PVC.

The Horizontal Pod Autoscaler (HPA) never talks to Prometheus, your app, or a cloud queue directly: it only ever queries one of three Kubernetes metrics APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io), so scaling on queue length requires something that exposes the queue's depth through one of those APIs. The standard pipeline is: the application or a sidecar exposes the metric, a metrics system collects it, and a metrics adapter (most commonly prometheus-adapter, or a project like KEDA, Kubernetes Event-Driven Autoscaling, which ships its own adapter and is now the more common current choice specifically for external event sources like queues) is registered with the API server as an aggregated API and translates queries into that metric API's shape.

The three metrics APIs

A message queue's depth (say, a managed queue service or a Kafka consumer-group lag) is not a property of any Kubernetes object, so it belongs under external.metrics.k8s.io and the HPA metric type: External, not Pods or Object.

Required components and how the metric gets exposed

1. Instrumentation: the application (preferred) or a sidecar exporter publishes a metric, e.g. a Prometheus gauge myapp_queue_length{queue="orders"} on /metrics.
2. Prometheus scrapes it on a scrape job.
3. prometheus-adapter is deployed with rules mapping a PromQL query to an external metric name Kubernetes will expose, and it registers an APIService (apiregistration.k8s.io) so kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 returns real data. This registration needs its own RBAC: a ClusterRole granting the HPA controller's service account (system:kube-controller-manager reaching through the aggregation layer) permission to read the external metrics API, plus the adapter's own service account needing permission to read Prometheus.
4. The HPA references the metric by name:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric: {name: queue_length_orders}
        target: {type: AverageValue, averageValue: "100"}

(autoscaling/v2 has been the stable API since Kubernetes 1.23; the older v2beta2 was removed in 1.26, so any manifest or tooling still referencing it is out of date.)

Operational pitfalls

• A silently wrong zero is worse than a visible failure. If the adapter genuinely can't reach its data source, the HPA does not quietly treat that as "no load": it sets the ScalingActive condition to False with reason FailedGetExternalMetric and shows the metric's current value as <unknown> in kubectl describe hpa, holding replica count steady. The real danger is the opposite case: an adapter that, on a query error, returns a literal 0 instead of erroring. That looks healthy to the HPA and drives a real scale-down during an actual outage in the metrics path, which is why adapter error-handling is worth testing explicitly rather than assumed.
• Cardinality. High-cardinality labels on the underlying metric (one series per customer ID, for instance) can make Prometheus memory blow up long before the HPA ever sees a problem; keep the label set the adapter maps from small and stable.
• Flapping. A noisy queue-length signal causes replica oscillation; use behavior.scaleDown.stabilizationWindowSeconds (part of the autoscaling/v2 HPA behavior fields) or pre-aggregate with a Prometheus recording rule rather than reacting to raw noise.
• Latency in the chain. Scrape interval, adapter caching, and the HPA's own sync period all stack up between a real queue-depth change and a scaling action; a 15s scrape interval plus a slow adapter cache can easily add tens of seconds of lag, which matters for a bursty queue.
• Cost. Recomputing an expensive PromQL query on every HPA sync (default every 15 seconds) across many HPAs can meaningfully load a Prometheus instance; pre-aggregate with recording rules for anything non-trivial.

Validate end to end in staging with synthetic queue load before trusting this in production, watching the full chain (producer to queue to exporter to Prometheus to adapter to HPA) rather than any single hop in isolation.

Trade-off note

Hand-rolling prometheus-adapter rules gives full control over the PromQL mapping but is fiddly YAML to maintain; KEDA trades some of that flexibility for purpose-built scalers for dozens of common event sources (queues, streams, schedules) and is usually less operational overhead for exactly this "scale on queue depth" scenario, at the cost of being one more component to run alongside (or instead of) prometheus-adapter.

Run the database through a purpose-built operator (a controller that encodes that specific database's operational knowledge as a CRD, Custom Resource Definition) rather than a hand-rolled StatefulSet, back it with topology-aware block storage, protect quorum with a PodDisruptionBudget (PDB) and anti-affinity, and treat backup/restore and failure recovery as first-class design requirements, not afterthoughts. The hardest part is not storage or scaling, it's guaranteeing zero data loss without a split-brain during failover, which needs an explicit fencing mechanism the platform doesn't give you for free.

Requirements first

Pin down RPO/RTO (Recovery Point/Time Objective) targets, expected throughput, latency budget, number of Availability Zones (AZs), and whether managed-DB-as-a-service is politically or contractually off the table; these decide almost everything below.

Three-way comparison: plain StatefulSet vs operator vs managed DB

For production, an operator is almost always the right middle ground unless a managed service is available and acceptable; a bare StatefulSet without an operator means re-implementing failover, backup orchestration, and safe upgrades yourself, which is exactly the hard part.

Concrete examples of DB-specific operators: CloudNativePG or the Zalando operator for PostgreSQL, Strimzi for Kafka, ECK (Elastic Cloud on Kubernetes) for Elasticsearch. Worth noting for currency: Kafka 4.0 (released March 2025) dropped ZooKeeper support entirely and runs KRaft-only, so a current Strimzi-managed Kafka cluster is one StatefulSet-backed topology (the brokers, using the KRaft consensus protocol for metadata) rather than two separate stateful systems (brokers plus a ZooKeeper ensemble) as older deployments required.

Architecture

flowchart TB
    OP[Database Operator / CRD controller]
    SS[StatefulSet: ordered, stable Pod identities]
    PDB[PodDisruptionBudget: protects quorum]
    SC[Topology-aware StorageClass, CSI provisioner]
    BK[Backup: snapshots + WAL/binlog shipping to object storage]
    OP --> SS
    OP --> PDB
    SS --> SC
    OP --> BK
    SS -->|anti-affinity| AZ1[AZ 1: primary]
    SS -->|anti-affinity| AZ2[AZ 2: sync replica]
    SS -->|anti-affinity| AZ3[AZ 3: async replica / DR]

Storage layer

• CSI (Container Storage Interface)-backed StorageClass with volumeBindingMode: WaitForFirstConsumer, so the volume is provisioned in the same zone the scheduler actually places the Pod, avoiding an unschedulable Pod stuck waiting on a volume in the wrong zone.
• reclaimPolicy: Retain on the StorageClass: an accidental PVC (PersistentVolumeClaim) deletion must not silently destroy the primary's data; recovery should require a deliberate, human step.
• Block storage versus a distributed filesystem, explicitly. Zonal block storage (EBS/PD/Azure Disk-style volumes) gives the lowest latency per replica but is single-zone and single-writer; the database's own replication protocol is what provides cross-zone durability, not the storage layer. A distributed filesystem or storage system (Ceph, Portworx, and similar) instead replicates data itself across nodes/zones at the storage layer, trading some latency and operational complexity for a storage layer that doesn't rely entirely on the database's own replication being correctly configured. For a workload with a mature, well-tested internal replication protocol (most production RDBMS and Kafka), zonal block storage plus database-level replication is usually the better latency/complexity trade-off; a distributed storage layer is more attractive for workloads with weak or no built-in replication.

The zero-data-loss and fencing problem

This is the part a plain StatefulSet does not solve for you. If a primary becomes unreachable but is not actually dead (a network partition, not a crash), promoting a replica while the old primary is still accepting writes creates two primaries accepting conflicting writes, a split-brain, which is a correctness failure, not just an availability one. A correct design needs explicit fencing: before a new primary is promoted, the old one must be provably prevented from accepting further writes, whether by a storage-level fence (revoking its volume attachment), a network-level fence (an admission gate the operator controls), or a consensus-based lease/term mechanism where a promoted replica only becomes writable once it holds a lease the old primary cannot renew. Most mature operators build this in (Kafka's KRaft controller quorum and Postgres operators using a distributed lock like Patroni's approach are both fencing-aware); a hand-rolled StatefulSet failover script that just runs a promote command is not safe against a network partition unless it independently solves this same problem.

PodDisruptionBudget and placement

• PodDisruptionBudget with minAvailable set so voluntary disruptions (node drains, cluster upgrades) can never take out enough replicas to lose quorum, for example minAvailable: 2 on a 3-replica set.
• Required (hard) pod anti-affinity across zones so replicas never land on the same node or, ideally, the same zone, or a single zone failure takes out more than one replica at once.
• Readiness probes gate traffic during recovery; liveness probes tuned to the database's actual health semantics, not a generic TCP check.

3-AZ replication topology

A common shape for regional durability without full cross-region cost: synchronous replication between the primary and one replica in a second AZ (so a confirmed write survives losing either AZ, at the cost of added write latency for the round trip), plus asynchronous replication to a third AZ or a separate region for disaster recovery, accepting a small, bounded RPO there in exchange for avoiding synchronous replication's latency cost across a third leg. The trade-off is explicit: every synchronous replica added tightens RPO but adds write latency; asynchronous replicas protect against a wider blast radius (a whole-region event) but carry a real, non-zero RPO if the primary is lost before the async replica catches up.

Backup and restore

• Application-consistent snapshots via CSI Volume Snapshots for fast full backups.
• Continuous WAL (write-ahead log) or binlog shipping for point-in-time recovery, usually built into the operator.
• Periodic backups to durable object storage (S3/GCS/Azure Blob-equivalent) with encryption and a lifecycle/retention policy.
• Restore drills on a schedule, not just documented in a runbook; an untested restore path is not a real recovery capability.

Handling node failures and scaling

• Node failure: the operator promotes a healthy replica (using the fencing mechanism above), Kubernetes reschedules the failed Pod elsewhere, and the storage layer either reattaches the same volume (single-zone block storage on a healthy node) or the operator rebuilds a fresh replica from backup/replication if the volume itself is unavailable.
• Vertical scaling: usually requires a coordinated, operator-sequenced resize or a planned failover, not a live in-place change.
• Horizontal scaling: read scaling via additional replicas behind a read-only Service; write scaling (if needed at all) requires sharding at the application or a purpose-built layer (e.g. Citus for Postgres, Vitess for MySQL), which is a materially different architecture, not a StatefulSet replica-count change.

Trade-offs and pitfalls

• Treating "add a PodDisruptionBudget and anti-affinity" as sufficient without addressing fencing is the most dangerous gap: it protects against voluntary disruption and Pod placement risk but says nothing about split-brain during an involuntary, ambiguous failure like a network partition.
• Choosing a distributed storage layer for its own replication when the database already replicates well is redundant complexity and extra latency for no real durability gain; verify what layer is actually providing the durability guarantee you're relying on.
• An untested backup is not a backup; the failure mode where a restore procedure is discovered to be broken happens exactly once, during a real incident.
• Running a fully DIY StatefulSet without an operator is a legitimate choice only for genuinely low-stakes or learning environments; for anything production-grade, the missing failover and fencing logic is a correctness gap, not just an operational inconvenience.

Direct answer

Gatekeeper enforces policy as a validating admission webhook: a ConstraintTemplate defines reusable Rego (OPA's policy language) logic and the CRD (Custom Resource Definition) shape it is configured with, and a Constraint is an instance of that template scoped to specific kinds and namespaces. For "deny privileged pods or pods missing resource limits," the template checks spec.containers[].securityContext.privileged and resources.limits.cpu/resources.limits.memory across every container and returns one violation message per offending container. The safe way to ship it is audit-only first, then targeted enforcement, never enforce cluster-wide on day one against an unaudited cluster.

Structured elaboration

ConstraintTemplate: the reusable Rego logic

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredpodsecurityandresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredPodSecurityAndResources
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredpodsecurityandresources

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          c.securityContext.privileged == true
          msg := sprintf("container '%v' is privileged", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.cpu
          msg := sprintf("container '%v' is missing resources.limits.cpu", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.memory
          msg := sprintf("container '%v' is missing resources.limits.memory", [c.name])
        }

This is Rego v0 syntax, still Gatekeeper's default. Gatekeeper 3.19 and later also supports opt-in Rego v1, which requires an explicit if before each rule body, but v0 remains what ships by default and what most existing ConstraintTemplates use.

Constraint: applying the template

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredPodSecurityAndResources
metadata:
  name: deny-privileged-or-no-limits
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]

enforcementAction: dryrun records violations without blocking anything, the correct starting state for any new policy on an existing cluster.

Validating vs mutating admission, and when to reach for either over OpenAPI schema validation

Gatekeeper is a validating admission webhook: it can accept or reject an object but cannot change it. A mutating admission webhook runs earlier in the chain and can rewrite the object before it is persisted, for example a sidecar injector adding a container, or a default value filled in. Kubernetes ships built-in admission controllers doing exactly these two jobs without any webhook at all, and they are the closest analogues to what Gatekeeper does here: LimitRanger (mutating; injects default resource requests and limits when a pod omits them) and ResourceQuota (validating; rejects a request that would exceed a namespace's aggregate quota).

The choice between a webhook and plain OpenAPI schema validation on a CRD comes down to what the rule needs to know:

• If the rule is fully expressible as a shape constraint on one object in isolation (a field must be one of an enum, a string must match a pattern, a number must sit in a range), OpenAPI schema validation on the CRD costs nothing at admission time and needs no separate service running.
• Reach for a webhook only when the rule needs something schema validation cannot express: cross-field logic (a field is required only if another field has a certain value), cross-object lookups (checking a Secret exists, checking sibling objects against a quota), or a policy that must apply uniformly across many unrelated resource kinds, exactly the "any Pod, any namespace" shape of the privileged/no-limits policy here.

Safe rollout plan

1. Deploy in enforcementAction: dryrun; let it run against real traffic for a full deploy cycle while pulling violations from Gatekeeper's audit results.
2. Share violations with owning teams with the exact fix needed (add resources.limits.cpu/memory, remove privileged: true), not just "you're non-compliant."
3. Flip to enforcementAction: deny first in one low-risk namespace, watch for unexpected rejections, then expand namespace by namespace.
4. Keep excludedNamespaces narrow and explicit, system namespaces only; a broad exclusion list defeats the point of a cluster-wide policy.

Worked example

A pod with two containers: one declares resources.limits: {cpu: "500m", memory: "256Mi"} and passes cleanly; the other has no resources block at all. The template's second and third rules each fire once for the second container, producing two separate violation messages ("missing resources.limits.cpu" and "missing resources.limits.memory"). That per-container, per-field granularity is what makes the audit output actionable rather than a single opaque "pod rejected."

Trade-offs and pitfalls

• Rego policy is powerful but opaque to most application developers; ship it with plain-language violation messages as above, and do not expect teams to read Rego to understand why they were blocked.
• excludedNamespaces is a blunt instrument. Overusing it to unblock a team quickly erodes the policy's coverage silently; track exclusions the same way you would track a firewall exception.
• A validating webhook adds a synchronous hop to every matched request. Keep the Rego evaluation cheap, no external calls, so it does not become the very apiserver latency problem it would otherwise be diagnosing.

A safe restart uses kubectl rollout restart, which recreates pods through the normal RollingUpdate strategy rather than deleting them directly, so the same availability guarantees that protect a routine deployment protect the restart too.

Commands

Trigger and watch a rolling restart:

kubectl rollout restart deployment my-app -n prod
kubectl rollout status deployment my-app -n prod --watch

View rollout history and inspect a specific revision:

kubectl rollout history deployment my-app -n prod
kubectl rollout history deployment my-app -n prod --revision=3

Roll back:

kubectl rollout undo deployment my-app -n prod --to-revision=3
kubectl rollout status deployment my-app -n prod

Ship an image change with a recorded reason (the --record flag some older references use for this is deprecated; annotate explicitly instead):

kubectl set image deployment/my-app my-app=registry/app:1.2.3 -n prod
kubectl annotate deployment my-app kubernetes.io/change-cause="bump to 1.2.3, ticket OPS-441" --overwrite -n prod

What keeps a restart from becoming a cascading failure

• RollingUpdate parameters: maxUnavailable and maxSurge (both default to 25% of desired replicas) bound how many old pods can be down and how many extra new pods can exist at once. For a consumer-facing service, a conservative setting (for example maxUnavailable: 0, maxSurge: 1) never drops capacity below the current replica count during the restart, at the cost of briefly running more pods than the steady-state count.
• Readiness probes: a Service only sends traffic to pods that pass their readiness probe, so a newly restarted pod that's still initializing doesn't receive requests it can't yet handle. This is the single biggest lever against a restart-induced error spike; without a readiness probe, the rollout has no signal that a "new" pod is actually ready and can start routing traffic to it immediately.
• PodDisruptionBudget (PDB): guarantees a minimum number (or percentage) of replicas stay available throughout the restart, independent of the Deployment's own maxUnavailable setting, which matters when other voluntary disruptions (a node drain, a cluster upgrade) happen to overlap with the restart window.
• Graceful shutdown: a preStop hook plus a terminationGracePeriodSeconds long enough for in-flight requests to finish, combined with the Service removing the pod's endpoint before the container actually stops, avoids dropping requests that were already in progress when the restart began.
• Staged rollout for risk-sensitive services: restarting (or deploying) to a small subset first, watching error rate and latency, then proceeding, catches a bad new revision before it reaches full traffic; this is a general staged-rollout practice, not a specific traffic-splitting mechanism (traffic-splitting techniques like weighted canary routing are a load-balancing/ingress-layer concern, not something the Deployment object itself provides).

Trade-offs and pitfalls

• kubectl rollout restart only recreates pods; it does not change the Deployment's spec, so rollout history records it as a new revision with the same template, which is easy to forget when later trying to undo your way back past a restart that changed nothing.
• Setting maxUnavailable: 0 guarantees no capacity loss but requires enough spare cluster capacity for maxSurge extra pods to schedule; on a tightly packed cluster this can leave the rollout stuck Pending on the surge pods instead of proceeding.
• A rollback only restores the pod template (image, env, resource requests, and so on). If the Deployment reads a ConfigMap or Secret by a fixed name and that ConfigMap was edited in place rather than replaced with a new name or hash-suffixed name, rolling the Deployment back does not restore the old configuration content, only the old pod template pointing at the same (already-mutated) ConfigMap. This is the most common way a rollback fails to actually roll back.
• The same gap applies to a PersistentVolumeClaim (PVC): a Deployment's rollback restores the pod template's volume mount references, not the data on the volume itself. If the new version wrote a schema migration or otherwise mutated data in place on that volume, rolling the Deployment back gives you the old code pointing at already-changed data, not the old data. Anything stateful needs its own restore path (a volume snapshot or application-level backup) alongside the Deployment rollback, not instead of thinking about it separately.