Kubernetes Architecture, Operations, and Troubleshooting Questions

A PodDisruptionBudget (PDB) tells the cluster how much of a replicated workload is allowed to be taken down at once by a voluntary disruption, things a human or controller chooses to do, such as a node drain, a cluster upgrade, or the cluster autoscaler removing a node. It has no effect on involuntary disruptions: a node crashing, a container getting OOMKilled (killed by the kernel's out-of-memory killer for exceeding its memory limit), or hardware failure all bypass it entirely, because there's no eviction request for the PDB to block in those cases.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  selector:
    matchLabels: { app: web }
  minAvailable: 2

(policy/v1 is the stable, current API group and version for this object; older manifests using policy/v1beta1 are targeting a removed API and will fail to apply on a current cluster.)

How PDBs interact with each operation

• Rolling updates: the Deployment controller's own maxUnavailable/maxSurge settings already bound disruption during a rollout, and a PDB adds an independent floor on top that applies across all voluntary disruptions hitting that workload at once, not just the rollout in isolation. If a rollout and a node drain happen to overlap, the PDB is what prevents their combined effect from dropping available replicas too far.
• Cluster Autoscaler (CA) scale-down: before removing a node, CA evicts the pods on it through the eviction API, and that eviction is refused if it would violate a pod's PDB. CA then either finds another node to remove or skips that node until eviction becomes possible; it does not force through a PDB violation.
• Manual maintenance (kubectl drain, a kubeadm upgrade, a node reboot): these also go through the eviction API and are blocked the same way; an operator hitting a blocked drain either waits, adjusts the PDB, or in genuine emergencies deletes pods directly (bypassing the eviction API, which also bypasses the PDB, and should be a deliberate, audited exception rather than a routine workaround).

Problem scenarios and fixes

Trade-offs and pitfalls

• Prefer percentage-based values (maxUnavailable: 25%) over absolute counts for workloads whose replica count changes with load; an absolute minAvailable: 3 on a Deployment that autoscales down to 2 replicas becomes an unsatisfiable budget that blocks every voluntary disruption until it scales back up.
• A PDB with minAvailable: 100% (or an absolute count equal to current replicas) reads as "maximally safe" but actually means zero voluntary disruptions are ever permitted, which silently blocks every future drain and upgrade until someone notices and loosens it, often under time pressure during an incident.
• PDBs protect against voluntary disruption stacking up in ways a single controller's own settings can't see; they are not a substitute for redundancy. A workload with only one replica and a PDB requiring minAvailable: 1 gains nothing from the PDB, since that one pod being unavailable for any reason, voluntary or not, is already an outage.

At 5,000 nodes and 50,000 pods the design sits close to Kubernetes' documented large-cluster envelope (guidance of up to 5,000 nodes, no more than 110 pods per node, and no more than 150,000 total pods per cluster), so the control plane has to be sized deliberately rather than by default assumptions: one etcd member per availability zone (AZ) for quorum-safe failure tolerance, several API server replicas per AZ behind per-AZ and regional load balancing, and leader-elected controller-manager/scheduler replicas, with the actual trade-off being etcd member count (3 versus 5) traded against write latency.

flowchart TB
    subgraph AZ1
        E1[etcd member 1]
        A1[API server replicas]
    end
    subgraph AZ2
        E2[etcd member 2]
        A2[API server replicas]
    end
    subgraph AZ3
        E3[etcd member 3]
        A3[API server replicas]
    end
    LB[Regional load balancer] --> A1
    LB --> A2
    LB --> A3
    A1 --- E1
    A2 --- E2
    A3 --- E3
    E1 --- E2
    E2 --- E3
    CM[controller-manager and scheduler, leader-elected] --> LB

Placement

• etcd: one member per AZ (3 members total for the baseline design), each on a dedicated host not shared with API server or scheduling workloads, so etcd's latency-sensitive disk writes are never contending with unrelated CPU/IO pressure.
• API servers: stateless, so scale horizontally within each AZ, for example 3 to 5 replicas per AZ (9 to 15 total), behind a per-AZ load balancer that prefers local-AZ API servers for kubelet traffic (reducing cross-AZ hops for the highest-volume traffic class) plus a regional load balancer presenting one endpoint for external clients.
• controller-manager and scheduler: run multiple replicas with Kubernetes' built-in leader election (via Lease objects); only the elected leader is active at any time, the rest are hot standbys that take over on leader failure.

Etcd sizing: the real trade-off

Etcd requires a strict majority (quorum) of members to agree before committing a write. For a cluster of $N$ members, quorum is:

With $N=3$: quorum $= \lfloor 3/2 \rfloor + 1 = 2$, tolerating $\lfloor (3-1)/2 \rfloor = 1$ failure, exactly one full AZ. With $N=5$: quorum $= \lfloor 5/2 \rfloor + 1 = 3$, tolerating $\lfloor (5-1)/2 \rfloor = 2$ failures, more than one AZ's worth. At 3 AZs specifically, a 5-member cluster cannot place members one-per-AZ evenly (it needs a 2-1-2 or similar split), which reintroduces asymmetric AZ dependence; 3 members, one per AZ, is the cleaner fit for a 3-AZ topology and is the standard recommendation unless a documented durability requirement justifies the extra latency of 5.

Avoiding split-brain

• Quorum arithmetic itself prevents split-brain by construction: a write only commits with agreement from a strict majority, so a network partition can produce at most one side with a majority, never two.
• Never run etcd with an even member count; an even split has no built-in majority-breaking property and is strictly worse than the next odd number down for the same failure tolerance.
• Etcd's default heartbeat interval (100ms) and election timeout (1000ms) are tuned assuming low, LAN-like latency; validate real inter-AZ round-trip time against these defaults before relying on them across AZs, since higher-than-assumed cross-AZ latency causes spurious leader elections that look like instability but are actually a timeout tuning gap.
• Keep etcd off nodes that also run general workloads; resource contention from an unrelated noisy pod can delay etcd's disk fsync enough to look like a network partition to the rest of the quorum.

Operational trade-offs accepted

• 3 versus 5 etcd members: the design above accepts 3 (tolerating one AZ loss) for lower write latency; this is a deliberate trade against the extra durability 5 would provide, appropriate because 3 AZs already provides physical fault isolation and losing two AZs simultaneously is a much rarer event than losing one.
• API server replica count versus etcd read load: more API server replicas absorb client load and reduce tail latency, but every replica also issues reads against etcd (mitigated by the API server's built-in watch cache, so most reads are served from memory rather than hitting etcd directly); sizing replicas too aggressively without accounting for this can still add unnecessary etcd read pressure.
• Scale ceiling: at 50,000 pods the cluster is well within documented limits, but is a meaningful fraction of the 150,000-pod ceiling; if growth continues, the accepted trade-off today (one large cluster) may need revisiting toward sharding workloads across multiple clusters before hitting the ceiling, rather than as an emergency reaction to it.
• Backups: frequent etcd snapshots plus a rehearsed, tested restore procedure are accepted as an ongoing operational cost, not a one-time setup task, since an untested restore path is not meaningfully different from having no backup at all.

Trade-offs and pitfalls

• Assuming "3 AZs" alone guarantees safety without validating actual inter-AZ latency against etcd's default timeouts is the most common design gap; the topology looks correct on a diagram and still produces spurious elections in practice.
• Over-provisioning API server replicas without watch-cache-aware capacity planning trades one bottleneck (client-facing latency) for another (etcd read amplification).
• Treating 5-member etcd as strictly "safer" than 3 ignores that it does not fit evenly across exactly 3 AZs and costs write latency on every single commit, not just during a failure.

For a global SaaS product with regional data residency and zero-downtime failover requirements, the default topology is one Kubernetes cluster per region, not a single cluster stretched across regions. Kubernetes has no native multi-region control plane: etcd's Raft consensus requires a majority of members to acknowledge every write within one election/heartbeat cycle, and cross-continent round-trip latency alone makes that unreliable at global distances. Each region's cluster keeps its own control plane and etcd quorum, owns its data locally to satisfy residency, and is tied together by a thin layer above the clusters: global traffic steering, GitOps for consistent config, and replicated (but region-scoped) secrets and policy.

Cluster-per-region vs. multiple clusters per region

Start with one cluster per region and add more clusters within a region only when a specific pressure appears, not by default:

Adding clusters within a region multiplies the GitOps, secret-distribution, and observability work described below, so it is a cost paid deliberately, not a default.

Global traffic management

• Geo-aware DNS with health checks (for example, weighted or latency-based routing) sends a client to its nearest healthy region by default and reweights away from a region that fails health checks.
• A global anycast (the same IP address announced from multiple physical points of presence at once, so network routing itself delivers a client's packets to whichever announcing location is topologically closest) entry point or global load balancer in front of the DNS layer absorbs volumetric attacks and terminates connections at the edge closest to the client before routing inward to a region.
• Where a jurisdiction requires data to never leave a specific region, routing must be policy-aware, not just latency-aware: a request from a covered user has to be pinned to its compliant region even if a closer region would otherwise be faster, which means the routing layer needs identity or geolocation context, not just liveness.
• Zero-downtime failover, in this context, means the traffic layer detects a region-health failure and reweights traffic away from it before users see errors. It does not by itself guarantee every in-flight workload survives with zero data loss; that second guarantee is a matter of replication design and failover orchestration for the specific stateful service, which is its own high-availability and disaster-recovery discipline and mostly outside the scope of the cluster-topology decision itself.

Cross-cluster service discovery

Two mechanisms are commonly combined, at different layers:

• Kubernetes-native multi-cluster service export/import: a Service in one cluster is explicitly exported and becomes resolvable, with its own DNS name, from other member clusters. This keeps cross-cluster calls to a short, explicit allow-list of shared services rather than opening the whole cluster network to every other cluster.
• Mesh federation (Istio's or Linkerd's multi-cluster modes, or a dedicated cross-cluster networking layer like Submariner): builds a flat, mutually authenticated network across clusters so workloads in different regions can reach each other's pod IPs directly, secured with mutual TLS (mTLS, where both sides present a certificate rather than only the server).

Favor the narrower, explicit-export model for anything crossing a residency or trust boundary, and reserve full mesh federation for trusted, high-fan-out internal calls where the operational cost of running and certifying a cross-cluster mesh is justified. The traffic-weighting and health-check behavior once a call reaches the mesh is an ingress/load-balancer-layer concern, not a cluster-topology one, and is out of scope here.

GitOps synchronization

• One Git repository holds a base configuration plus per-region overlays (Kustomize or Helm values per region), so region-specific differences (residency-driven storage classes, region-scoped secrets, capacity) are explicit and reviewable.
• A GitOps controller (Argo CD or Flux) runs inside each regional cluster and pulls only its own overlay; a controller outage or compromise in one region cannot push into another region's cluster, because there is no cross-region push path.
• Promotion between regions is pull-request-driven and staged: merge to a designated pilot region's overlay first, watch that region's health, then promote to the rest. The specific traffic-shifting mechanics used to validate a rollout region before promoting further belong to the ingress/load-balancer layer, not to the GitOps topology itself.

Policy and secret distribution

• Policy as code (OPA Gatekeeper or Kyverno) ships through the same GitOps pipeline as workloads, so every region enforces an identical baseline without a human re-applying policy per cluster.
• Secrets: a central secrets backend (HashiCorp Vault is a common choice) issues short-lived, per-region-scoped credentials rather than long-lived secrets replicated everywhere; a per-cluster agent (Vault Agent, or the External Secrets Operator syncing from Vault into native Kubernetes Secret objects) pulls only what that region is authorized to read.
• Residency enforcement for secrets is a scoping problem, not a replication problem: a secret that must never leave a region should have no path, human or automated, that copies it into another region's Vault namespace or cluster.

Worked example: why a single stretched control plane fails on physics alone

Take two regions roughly 6,000 km apart (for example, the U.S. East Coast and Western Europe), and assume light travels through fiber at about two-thirds the vacuum speed of light:

$c \approx 300{,}000\text{ km/s}$

$v_{\text{fiber}} \approx \frac{2}{3} \times 300{,}000 = 200{,}000\text{ km/s}$

One-way propagation delay alone is:

$t_{\text{one-way}} = \frac{6{,}000\text{ km}}{200{,}000\text{ km/s}} = 0.03\text{ s} = 30\text{ ms}$

A round trip is at least:

$t_{\text{RTT}} = 2 \times 30\text{ ms} = 60\text{ ms}$

This is a physical floor from propagation delay alone, before any queuing, processing, or retransmission. A single stretched etcd cluster needs the leader to replicate a write to a quorum of members and wait for their acknowledgments, at least one full round trip to the farthest quorum member, and in practice more once Raft's heartbeat and election-timeout margins (which must comfortably exceed that RTT to avoid false leader elections) are added. A control plane that needs low-tens-of-milliseconds write latency to stay responsive cannot tolerate a mandatory 60-millisecond-plus round trip on its consensus path; that floor is exactly why the architecture keeps etcd and the API server local to a region instead of spreading a single quorum across continents.

Trade-offs and pitfalls

• Mesh federation gives transparent cross-region calls but expands the trust and blast-radius boundary to every federated cluster; a compromised workload in one region can potentially reach services in another unless network policy and mTLS identity are scoped tightly per exported service.
• GitOps-per-cluster avoids a single push-based control point, but drifts silently if a region's controller falls behind (stuck sync, expired credentials); alert on GitOps sync staleness per cluster, not just on workload health.
• Centralizing secrets in one Vault deployment recreates a single point of failure for every region if that deployment is unavailable; run Vault itself as a highly available deployment, or accept that a Vault outage degrades new deployments (not running workloads) globally.
• The most common design mistake is treating "multi-cluster" as purely a networking problem and deferring the GitOps and secrets-scoping design until after the clusters exist; retrofitting per-region secret scoping onto a cluster that already replicates everything everywhere is far harder than designing it in from the start.

flowchart TD
    DNS[Geo-aware DNS + global LB] --> RA[Region A cluster]
    DNS --> RB[Region B cluster]
    RA --> ETCDA[(etcd A, region-local)]
    RB --> ETCDB[(etcd B, region-local)]
    GIT[Git: base config + per-region overlays] --> ArgoA[GitOps controller in A]
    GIT --> ArgoB[GitOps controller in B]
    ArgoA --> RA
    ArgoB --> RB
    Vault[Central Vault] -->|short-lived, region-scoped creds| RA
    Vault -->|short-lived, region-scoped creds| RB
    RA <-->|explicit service export, mTLS| RB

A node showing NotReady means kubelet stopped reporting a healthy heartbeat to the API server, and diagnosis means figuring out whether kubelet itself is broken, the network path between node and control plane is broken, or the underlying kernel/host is in trouble. Which of the three it is usually falls out of reading the node's own Conditions plus the right log source for each layer.

Node conditions to inspect

kubectl describe node <node>

Conditions:
  Type             Status  Reason
  MemoryPressure   False   KubeletHasSufficientMemory
  DiskPressure     False   KubeletHasNoDiskPressure
  PIDPressure      False   KubeletHasSufficientPID
  Ready            False   KubeletNotReady

Kubelet vs network vs kernel

Kubelet-level. Check on the node directly, since kubelet is a host-level service, not a pod you can kubectl logs:

systemctl status kubelet
journalctl -u kubelet -n 200
crictl ps      # confirms the container runtime kubelet talks to is actually up

Network-level. If kubelet is running but can't reach the API server:

ip a; ip route
curl -sk https://<api-server>:6443/healthz

Look for CNI plugin errors in journalctl -u kubelet or the CNI's own logs; a broken CNI on the node can prevent kubelet's own traffic to the control plane even though the process is alive.
Kernel/host-level.

dmesg | tail -n 200
journalctl -k
df -h; free -m

A kernel OOM killer that targets the kubelet or container-runtime process itself (not an application container) produces NotReady directly, which is a distinct case from the container-level OOMKilled covered separately: check dmesg for the OOM killer's target process name specifically, not just that an OOM event happened somewhere on the box.

Three linked scenarios worth naming explicitly

What happens to the node's existing pods. NotReady does not immediately delete anything. Kubernetes applies a node.kubernetes.io/not-ready:NoExecute taint automatically, and pods get a default tolerationSeconds of 300 (5 minutes) for that taint unless they specify otherwise. Only after that window expires are the pods actually evicted; if they're owned by a Deployment or ReplicaSet, a replacement gets scheduled elsewhere, which can itself go Pending if there isn't spare capacity, linking this scenario directly to the pod-scheduling case above.

Mass NotReady right after a DaemonSet rollout. If several nodes go NotReady at roughly the same time, the more likely cause is a shared node-level agent, not N independent hardware failures. A DaemonSet update to something like the CNI plugin, a log agent, or a security agent can crash-loop node-side or consume enough host resources to break kubelet's own health loop. Check:

kubectl rollout history daemonset/<name> -n kube-system

and correlate the timestamp of the NotReady onset against the DaemonSet's rollout time; kubectl rollout undo daemonset/<name> is usually the faster, safer move than debugging individual nodes one at a time.

Safe cordon and drain

Cordoning marks the node unschedulable, so the scheduler stops placing any new pods on it without touching what is already running there. Draining then evicts those existing pods, respecting each one's PodDisruptionBudget (PDB, an object that caps how many replicas of a workload can be voluntarily unavailable at once), so they get rescheduled onto healthy nodes before you take this one down for maintenance.

kubectl cordon <node>
kubectl get pdb -A                      # check PodDisruptionBudgets first
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

--delete-local-data is an old flag name that was deprecated and later removed; --delete-emptydir-data is the current one. After maintenance:

kubectl uncordon <node>

Trade-offs and pitfalls

• Don't drain immediately on every NotReady; if it's a transient network blip that self-heals in under the 300s toleration window, a drain is unnecessary churn. Give it a short grace period unless the node is clearly compromised (disk full, kernel panic).
• A single NotReady node is usually a node problem; several at once is usually a shared-dependency problem (DaemonSet, a common upstream network path, or a control-plane-side issue), and treating it as N separate incidents wastes time.

A pod is the smallest unit Kubernetes schedules and manages: one or more containers that always run together on the same node, sharing a network namespace and, optionally, storage volumes. It is the unit, rather than the individual container, because the kubelet, the scheduler, and every controller reason about placement, restarts, and networking at that granularity, not below it.

Why the pod, not the container, is the atomic unit

• Co-scheduling guarantee: every container in a pod is placed on the same node and started, stopped, and restarted as a unit by the kubelet.
• One IP per pod: regardless of how many containers it holds, a pod gets exactly one IP address; Services and DNS resolve to that pod IP, not to an individual container.
• Shared network namespace: containers in a pod talk to each other over localhost, and must not claim conflicting ports, because they share the same network namespace.
• Shared volumes: volumes are declared once at the pod spec level and can be mounted into more than one container, which is how a sidecar can read or write files the main container produces without a network hop.

When to run more than one container in a pod

• Sidecar pattern: a helper process that shares the fate and resources of the main container, most commonly a log shipper, a proxy, or a metrics exporter.
• Init containers: containers that run to completion, in order, before the main containers start, typically for one-time setup like a schema migration or config templating.
• Native sidecar containers: since the SidecarContainers feature became enabled by default in Kubernetes 1.29 (stable as of 1.33), you can declare a sidecar under initContainers with restartPolicy: Always. Kubernetes then starts it before the main container, keeps it running for the pod's whole life, and stops it after the main container on shutdown, which fixes the older ordering problem where a plain extra container might not be ready before the app started, or might outlive a completed Job's main container instead of shutting down with it.

A small example, an app container and a log-forwarding sidecar sharing a volume instead of a network call:

containers:
  - name: app
    image: myapp:1.4
    volumeMounts:
      - name: logs
        mountPath: /var/log/app
  - name: log-shipper
    image: fluent-bit:latest
    volumeMounts:
      - name: logs
        mountPath: /var/log/app
        readOnly: true
volumes:
  - name: logs
    emptyDir: {}

The app writes to /var/log/app; the sidecar tails the same path through the shared emptyDir volume, no network hop involved.

Worked example: what a failing sidecar looks like

If the log-shipper container above enters a crash loop while app keeps running fine, the two containers' fates are independent even though they share a pod: app keeps serving traffic, but the log pipeline drops. This is visible directly in the pod list, where the READY column reports containers-ready over containers-total, not a single number:

NAME                     READY   STATUS    RESTARTS      AGE
myapp-6d947f8db8-x2z1p   1/2     Running   4 (30s ago)   6m

Here 1/2 means only one of the pod's two containers is passing its readiness state, and the restart count of 4 belongs to the sidecar, not to app.

Trade-offs and pitfalls

• Co-location couples lifecycle: a sidecar cannot be scaled independently of the app the way a separate Deployment could be. Native sidecar containers loosen this slightly by giving the sidecar its own restart behavior, but it is still tied to the pod's schedule and node.
• Every container in the pod counts toward the pod's total resource footprint; forgetting the sidecar's own requests and limits when sizing the node is a common under-provisioning mistake.
• Splitting a component into a separate pod (plus a Service) regains independent scaling and blast-radius isolation, at the cost of a network hop and losing the localhost and shared-volume convenience. Reach for separate pods when the components genuinely differ in scaling or failure characteristics, for example a stateless API versus a shared cache, rather than defaulting to a sidecar for anything that happens to run alongside the main app.