Kubernetes Architecture, Operations, and Troubleshooting Questions

A PodDisruptionBudget (PDB) tells the cluster how much of a replicated workload is allowed to be taken down at once by a voluntary disruption, things a human or controller chooses to do, such as a node drain, a cluster upgrade, or the cluster autoscaler removing a node. It has no effect on involuntary disruptions: a node crashing, a container getting OOMKilled (killed by the kernel's out-of-memory killer for exceeding its memory limit), or hardware failure all bypass it entirely, because there's no eviction request for the PDB to block in those cases.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  selector:
    matchLabels: { app: web }
  minAvailable: 2

(policy/v1 is the stable, current API group and version for this object; older manifests using policy/v1beta1 are targeting a removed API and will fail to apply on a current cluster.)

How PDBs interact with each operation

• Rolling updates: the Deployment controller's own maxUnavailable/maxSurge settings already bound disruption during a rollout, and a PDB adds an independent floor on top that applies across all voluntary disruptions hitting that workload at once, not just the rollout in isolation. If a rollout and a node drain happen to overlap, the PDB is what prevents their combined effect from dropping available replicas too far.
• Cluster Autoscaler (CA) scale-down: before removing a node, CA evicts the pods on it through the eviction API, and that eviction is refused if it would violate a pod's PDB. CA then either finds another node to remove or skips that node until eviction becomes possible; it does not force through a PDB violation.
• Manual maintenance (kubectl drain, a kubeadm upgrade, a node reboot): these also go through the eviction API and are blocked the same way; an operator hitting a blocked drain either waits, adjusts the PDB, or in genuine emergencies deletes pods directly (bypassing the eviction API, which also bypasses the PDB, and should be a deliberate, audited exception rather than a routine workaround).

Problem scenarios and fixes

Trade-offs and pitfalls

• Prefer percentage-based values (maxUnavailable: 25%) over absolute counts for workloads whose replica count changes with load; an absolute minAvailable: 3 on a Deployment that autoscales down to 2 replicas becomes an unsatisfiable budget that blocks every voluntary disruption until it scales back up.
• A PDB with minAvailable: 100% (or an absolute count equal to current replicas) reads as "maximally safe" but actually means zero voluntary disruptions are ever permitted, which silently blocks every future drain and upgrade until someone notices and loosens it, often under time pressure during an incident.
• PDBs protect against voluntary disruption stacking up in ways a single controller's own settings can't see; they are not a substitute for redundancy. A workload with only one replica and a PDB requiring minAvailable: 1 gains nothing from the PDB, since that one pod being unavailable for any reason, voluntary or not, is already an outage.

For most teams, a managed control plane (Google Kubernetes Engine/GKE, Amazon Elastic Kubernetes Service/EKS, Azure Kubernetes Service/AKS) is the right default, because the provider absorbs control-plane high availability, etcd operations, and version patching that otherwise consume ongoing senior engineering time. Self-managed clusters on provisioned VMs earn their keep only when a concrete constraint, not a general preference for control, rules the managed option out: a data-residency or regulatory requirement the provider cannot satisfy, hardware or networking the managed offering does not support, or a genuinely large fleet where the provider's per-cluster fee becomes material against staff you would need to hire anyway.

Evaluation matrix

The security row matters more than it looks: managed Kubernetes does not remove the shared-responsibility split so much as move its boundary, the same shape as the shared-responsibility model on any IaaS. Workload-level security (RBAC, NetworkPolicy, secrets, supply-chain scanning) is the team's job either way.

A short decision checklist

Answer these before defaulting to self-managed:

1. Is there a regulatory or data-residency constraint that no available cloud provider's managed offering can satisfy (for example, a requirement that the control plane run inside a specific on-premises facility with no third-party operational access)?
2. Does the workload need node/kernel-level customization (specialized drivers, SR-IOV, custom kernel modules) that managed node pools do not allow?
3. Does the footprint include environments with no managed control-plane offering at all (deep edge sites, air-gapped networks, telecom points of presence)?
4. At current or planned scale, does the provider's control-plane fee plus its operational savings actually exceed what dedicated in-house Kubernetes operations staff would cost?

If none of these are "yes," managed is the safer default; if one or more are "yes," self-managed (or a lightweight distribution suited to the constraint) deserves serious evaluation.

Worked scenarios

• Compliance-driven case: a bank operating under a regulator that requires its control plane to run inside a specific on-premises data center, with no third-party (including the cloud provider's own control-plane operators) having operational access to cluster internals. This disqualifies every managed offering outright, regardless of cost or operational convenience; the decision is made by question 1 above before cost or overhead ever enters the comparison.
• Edge/scale-driven case: a telecom operator running Kubernetes across dozens of far-edge sites where no cloud provider offers a managed control plane at all. Self-managed, or a purpose-built lightweight distribution, is the only option that reaches the hardware; the comparison in the table above is largely academic here because there is no managed alternative on the table.

Trade-offs and pitfalls

• The most common wrong turn is choosing self-managed "for control" without budgeting the standing SRE headcount it requires; control-plane upgrades, etcd operations, and CVE patching do not stop after go-live, they become a permanent line item.
• Assuming managed Kubernetes removes all security burden is the second most common mistake; node OS hardening, RBAC design, NetworkPolicy, and secret management remain the team's responsibility under either model.
• Very large fleets sometimes justify self-managed purely on the provider fee, but that comparison is frequently done without pricing in the incident risk of an in-house etcd outage against a provider's SLA-backed uptime, which understates the true cost of self-managed at scale.
• Starting managed and only moving specific clusters to self-managed once a concrete constraint actually appears is lower risk than starting self-managed everywhere on a hypothetical future need.

kube-scheduler places a pod in two passes: filtering, which throws out every node that cannot possibly run the pod, and scoring, which ranks the survivors and picks the best one. Since Kubernetes 1.19 this two-pass model is implemented as the scheduling framework, a set of extension points (PreFilter, Filter, PreScore, Score, Reserve, Permit, Bind, and others) that built-in and custom plugins hook into; the older "predicates and priorities" terminology describes the same two passes from before the framework existed.

What the filter and score passes check

• Filter (hard constraints): available CPU/memory versus the pod's resource requests, node taints against the pod's tolerations, nodeSelector and nodeAffinity (required rules), volume topology and attach limits, and any PodAntiAffinity or topology spread constraint marked as required.
• Score (soft preferences): bin-packing versus spreading, preferred node affinity weights, image locality (a node that already has the image scores higher), and inter-pod affinity/anti-affinity preferences.

Taints, tolerations, and the two affinity mechanisms

A taint on a node (kubectl taint nodes node1 key=value:NoSchedule) repels pods unless they carry a matching toleration; this is how you reserve nodes (for example GPU nodes, or nodes mid-drain, which Kubernetes taints with node.kubernetes.io/unschedulable) for only the workloads that opt in. nodeSelector is a flat, exact-match label requirement: a pod either has a node with all the listed labels or it doesn't. nodeAffinity is strictly more expressive: it supports operators beyond equality (In, NotIn, Exists, Gt, Lt), and it splits into requiredDuringSchedulingIgnoredDuringExecution (a hard filter, same as nodeSelector but richer) and preferredDuringSchedulingIgnoredDuringExecution (a soft, weighted scoring input). Use nodeSelector for a simple single-label requirement and nodeAffinity once you need "any of these labels" logic or a graceful preference rather than a hard rule.

Preemption

If no node passes the filter pass for a pod, and that pod has a higher PriorityClass than pods already running, the scheduler can evict (preempt) lower-priority pods on a candidate node to make room, then schedule the higher-priority pod there. Preemption is deliberately a last resort: it only runs after normal scheduling fails, and it respects each evicted pod's PodDisruptionBudget (PDB, an object that caps how many replicas of a workload can be voluntarily disrupted at once) where possible, though a sufficiently high-priority pod can still violate a PDB if there is no other way to place it.

Debugging checklist for a pod stuck Pending

Start from the pod's own events, which is where the scheduler reports why it failed, then work outward:

$ kubectl describe pod checkout-7d9f6c-abcde -n prod
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  85s   default-scheduler   0/12 nodes are available: 4 Insufficient cpu,
                                                        3 node(s) didn't match Pod's node affinity/selector,
                                                        5 node(s) had untolerated taint {dedicated: gpu}.

That single event line already narrows the search: this pod failed on three distinct filters across the fleet. The checklist:

1. kubectl describe pod and read the FailedScheduling event message: it names the exact filters that eliminated nodes, and how many nodes each filter eliminated.
2. Compare requested CPU/memory against actual free capacity: kubectl describe nodes | grep -A5 Allocated or kubectl top nodes, and check whether requests (not limits) are what's exhausting the cluster.
3. Check the pod's nodeSelector / nodeAffinity labels against kubectl get nodes --show-labels for a typo or a label that no longer exists on any node.
4. Check node taints (kubectl describe node <n> under Taints:) against the pod's tolerations.
5. Check PersistentVolumeClaim (PVC) binding status if the pod mounts a volume (kubectl get pvc -n <ns>); an unbound claim or a StorageClass with no available capacity in the pod's zone blocks scheduling too.
6. Check PodAntiAffinity and topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule for an over-constrained placement rule (common after adding a third replica to a two-zone cluster).
7. Check whether the pod's PriorityClass is high enough that you'd expect preemption, and if so, why it didn't fire (often: the only feasible node candidates were also excluded by a hard filter, so there is nothing to preempt into).
8. If everything above looks clean, raise kube-scheduler's log verbosity (-v=10 on the scheduler component, or the equivalent in your managed offering) to see the plugin-by-plugin filter trace for that pod.
9. In a multi-scheduler setup, confirm the pod's spec.schedulerName actually matches the scheduler you're inspecting.

At-scale scheduler tuning

On very large clusters, kube-scheduler does not necessarily score every feasible node before choosing one; percentageOfNodesToScore in the scheduler configuration caps how many nodes it evaluates once a cluster passes roughly 100 nodes, trading a small chance of a marginally less optimal placement for scheduling throughput. If pods are scheduling slowly at scale (not stuck Pending, just slow to place), this setting, along with the number of scheduler profiles and any scheduler extenders in the pipeline, is where to look before assuming it's a resource-shortage problem.

Cold-start latency

A pod that schedules quickly can still take a long time to reach Running if the node has to pull a large image cold. Pre-pulling common images onto nodes (via a DaemonSet or node image baked into the AMI), keeping images small, and using an appropriate startupProbe so a slow-starting container isn't killed by liveness checks before it finishes initializing, are the standard levers; that's a related but distinct problem from scheduling itself, since a pod experiencing this has already left Pending.

Trade-offs and pitfalls

• Reading only the first FailedScheduling message and assuming a single cause is the most common debugging mistake; the message lists every filter that eliminated nodes, and a fix that addresses only one of them can still leave the pod Pending.
• Loosening nodeAffinity from required to preferred unblocks scheduling but silently removes a placement guarantee (for example zone isolation); confirm that's actually acceptable before doing it under incident pressure.
• Preemption is a blunt instrument: overusing high PriorityClass values on routine workloads causes churn as pods repeatedly evict each other rather than the cluster simply scaling out.

When CPU usage is genuinely above target but the Horizontal Pod Autoscaler (HPA) will not scale, the fault is almost always in the metrics plumbing between the container and the HPA controller (missing resource requests, a metrics-server or custom-metrics adapter that is not actually returning data, or a scaleTargetRef/selector mismatch) rather than in the scaling decision logic itself, which is a small, deterministic formula once it has correct inputs.

How the HPA actually decides (not scaling strategy, the mechanics)

The controller (autoscaling/v2 API, the current stable version; the earlier autoscaling/v2beta2 was removed in Kubernetes 1.26, so a cluster still authoring v2beta2 manifests is itself a currency problem worth checking for) recomputes replica count on a fixed sync period (default 15s) using:

with a tolerance band (default 10%, --horizontal-pod-autoscaler-tolerance) below which it intentionally does nothing, to avoid thrashing on noise. This tolerance and sync period explain most "it's above target but nothing happens" reports that turn out to be working as designed: a ratio of 1.05 (5% over target) is inside the default tolerance band and will not trigger a scale event.

Checklist, in order of how often each one is the actual cause

1. HPA state and events, the fastest signal:

kubectl describe hpa myapp-hpa -n myapp

Look for the Conditions block and events such as:

Warning  FailedGetResourceMetric  hpa-controller
  failed to get cpu utilization: unable to get metrics for resource cpu:
  missing request for cpu

That specific event means the fix is a resource request, not anything about metrics-server health.
2. CPU utilization requires requests, always. HPA's Utilization metric type is computed as usage divided by the container's CPU request, not its limit. If a container has no resources.requests.cpu, the HPA controller cannot compute a percentage at all and reports the event above, regardless of how busy the pod actually is.
3. Metrics API is actually serving data:

kubectl top pods -n myapp
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/myapp/pods" | jq

If kubectl top returns nothing, metrics-server itself is the suspect, not the HPA:

kubectl get pods -n kube-system -l k8s-app=metrics-server
kubectl logs -n kube-system <metrics-server-pod>

4. Custom or external metrics, if the HPA targets something other than CPU/memory:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq

An empty or erroring response here means the metrics adapter (commonly the Prometheus Adapter) is not registered correctly as an APIService, or the metric name in the HPA spec does not match the name the adapter actually exposes (a very common typo-class bug: the HPA references http_requests_per_second while the adapter exposes requests_per_second).
5. scaleTargetRef correctness: confirm .spec.scaleTargetRef.kind/.name in the HPA actually matches the Deployment being scaled; a stale reference (left over from a renamed Deployment) means the HPA is quietly watching nothing.
6. Pod readiness: the HPA only counts Ready pods toward currentReplicas and current metric averaging; a pod stuck failing its readiness probe is invisible to the calculation even though it is consuming CPU.
7. minReplicas/maxReplicas bounds: confirm maxReplicas is actually above the current replica count; a forgotten low ceiling silently caps scaling with no error at all.

Worked example

Suppose myapp-hpa targets 50% average CPU utilization, is currently running 4 replicas, and kubectl top pods shows the average pod is using CPU equal to 82% of its request. Plugging into the formula:

If the HPA is healthy, kubectl describe hpa should show current / target: 82% / 50% and a scaling event moving the Deployment toward 7 replicas. If instead the describe output shows no current value at all (a blank or <unknown>), the problem is upstream of this formula entirely: metrics are not reaching the controller, so there is nothing for the formula to compute against.

Trade-offs and pitfalls

• This checklist deliberately does not touch which metric to scale on or whether reactive CPU-based scaling is the right strategy for the workload; that is a scaling-strategy design decision, separate from why a correctly-configured HPA fails to act mechanically.
• A common wrong turn is restarting metrics-server or the HPA controller as a first move; that only helps if the actual event says the adapter is unresponsive, and it resets nothing if the real cause is a missing CPU request or a scaleTargetRef typo.
• Combining HPA with a Vertical Pod Autoscaler (VPA) on the same CPU metric can fight itself: VPA changing requests up or down changes the denominator HPA's utilization percentage is computed against, producing scaling behavior that looks erratic but is actually two controllers correctly reacting to each other.
• The tolerance band means "barely above target" is not a bug; check the actual ratio before assuming the controller is broken.

Treat this as an incident, not a debugging session: stop the failing controller from doing further damage first, restore a known-good control plane second, validate correctness third, and only then work backward on prevention. The part worth being precise about is the rollback step itself: "roll back the control plane" is not a single universal command, and what it actually means depends heavily on whether the cluster is managed or self-run.

flowchart TD
    A[Detect: mass pod failures after upgrade] --> B[Mitigate: stop the failing controller]
    B --> C[Roll back control plane]
    C --> D[Validate cluster integrity]
    D --> E[Root-cause + prevent recurrence]

1. Stop the bleeding

Scale the misbehaving controller to zero so it stops acting on objects while you work:

kubectl -n <ns> scale deploy/<controller> --replicas=0

If the controller registered an admission webhook that's now rejecting or mutating requests based on the new (incompatible) API shape, remove or disable that webhook configuration rather than leaving it live while the controller itself is down, since a stale webhook can independently block unrelated traffic. If pods are stuck Terminating because the controller's own finalizers can no longer complete their cleanup logic against the new API, that's a sign to investigate before force-removing finalizers, which is a last resort that can leave orphaned cloud resources behind.

2. Roll back the control plane, correctly

This is the step where an overly generic plan is dangerous. What "rollback" actually means splits by environment:

• Self-managed (kubeadm or similar). There is no rollback command. etcd (the cluster's backing datastore) does not support a minor-version downgrade once components have written data in the new version's format, so the only real path is: restore etcd from a snapshot taken before the upgrade, then reinstall the previous kubelet/kubeadm/control-plane component versions. If no pre-upgrade etcd snapshot exists, there is no clean rollback, only forward fixes; this is why taking that snapshot has to happen before every upgrade attempt, not after something breaks.
• Managed control planes vary and are changing quickly. Amazon EKS added a native version-rollback capability in 2026 that lets you revert to the immediately previous minor version, but only within a limited window (documented as 7 days) after the upgrade; past that window it is not available and you're in the same "restore from backup" position as self-managed. Google Kubernetes Engine (GKE) is the opposite case from EKS: outside one narrow exception, GKE does not let you downgrade a cluster's control plane to a previous minor version at all, and attempting it is rejected outright with an explicit error that the requested version is not newer than the current one. GKE does support patch-level downgrades within the same minor version, and it has a Preview-stage rollback that only works mid-way through a two-step control-plane minor upgrade: after the first step (the binary upgrade) and only until the second step (the emulated-version upgrade) completes; once that second step finishes, the previous minor version is no longer reachable either. Don't assume your specific managed provider or cluster version has this capability; check it explicitly rather than treating "it's managed, so it rolls back" as a given, because that assumption is exactly what a previous, now-corrected version of this plan got wrong.

3. Validate cluster integrity after rollback

• API server health: kubectl get --raw /livez?verbose and /readyz?verbose (prefer these over kubectl get componentstatuses, which is deprecated and unreliable on any control plane that isn't a single static node, since it hard-codes checks that don't reflect real high-availability setups).
• Workload state: kubectl get nodes,pods,deployments --all-namespaces, looking specifically for CrashLoopBackOff or stuck Pending pods that predate the rollback and shouldn't be blamed on it.
• The CRD-backed controller itself: once rolled back, restart it at low replica count first, watch its logs and a sample of its managed custom resources' status fields for a few reconcile cycles before restoring full replicas.
• Run an actual smoke test against the CRD: create/update/delete a throwaway custom resource and confirm the controller reconciles it correctly, rather than only checking that the controller process is running.

4. Prevent recurrence

• Require a pre-upgrade compatibility gate in the upgrade runbook: check the Kubernetes deprecated/removed API list for the target version against every CRD and controller in the cluster, not just built-in workload types.
• Maintain a staging cluster that mirrors production's actual CRD and controller versions (not a clean, controller-free cluster) and run the real upgrade against it first, including the CRD-backed controller's own test suite.
• Take an etcd snapshot immediately before every control-plane upgrade, unconditionally, regardless of whether the provider claims to support rollback.
• Add automated post-upgrade health gates (a scripted smoke test against critical CRDs and workloads) that must pass before an upgrade is considered complete, rather than relying on someone noticing pod failures.

Trade-offs and pitfalls

• Force-removing finalizers to unstick Terminating pods is fast but can leave real orphaned resources (cloud disks, external DNS records) behind; only do it once you've confirmed what that specific controller's finalizer was supposed to clean up.
• Restoring an etcd snapshot rolls back everything written since the snapshot, not just the CRD schema; any legitimate workload changes made in that window are lost too, which is the real cost of not having a native, narrower rollback available.
• The instinct to "wait and see if it recovers" after a bad upgrade is usually wrong for a CRD-backed controller actively causing mass pod failures; the controller should be stopped first, investigated second.