Kubernetes Architecture, Operations, and Troubleshooting Questions

A Kubernetes Service is a stable virtual address in front of a changing set of pods. The four types differ in who can reach that address and how: ClusterIP is internal-only, NodePort adds a port opened on every node, LoadBalancer asks the cloud provider to provision an external load balancer in front of that, and ExternalName is different in kind: a DNS alias to something outside the cluster with no pod routing involved at all.

Service types

A fifth variant sits alongside these four rather than replacing any of them: a headless Service (clusterIP: None). It skips kube-proxy's virtual-IP rewriting entirely, so instead of one stable IP load-balancing across pods, CoreDNS returns the individual pod IPs behind the Service directly to the client (sourced from the same EndpointSlices kube-proxy would otherwise consume). This is why StatefulSets pair with a headless Service: each replica needs its own stable, individually addressable DNS name rather than one shared address in front of all of them.

kube-proxy: the mechanism underneath

kube-proxy is what actually implements the ClusterIP, NodePort, and LoadBalancer rewriting, and it has more than one implementation:

• iptables mode (the long-standing default): programs Linux netfilter rules to destination-NAT (DNAT, rewriting a packet's destination address) Service traffic to a pod IP. Simple, but rule evaluation is roughly linear in Service count, which shows up as latency at high Service counts.
• ipvs mode (IP Virtual Server): uses the kernel's IPVS load-balancing tables instead of a rule chain, giving better performance and more scheduling algorithm choices at scale.
• nftables mode: the newer replacement backend, built on the modern Linux nftables framework, which reached general availability in Kubernetes 1.33. It targets the same correctness as iptables mode with materially better performance at high Service counts, though iptables remains the cluster-wide default for now.
  Older material sometimes still mentions a userspace mode; it was deprecated years ago and fully removed in Kubernetes 1.26, so it should not appear in any current design.

Worked example: tracing one request through a LoadBalancer Service

flowchart LR
    Client --> ELB[Cloud load balancer]
    ELB --> NP[Node: NodePort]
    NP --> KP[kube-proxy DNAT rule]
    KP --> PodA[Pod A]
    KP --> PodB[Pod B]

The client hits the cloud load balancer's public address, which forwards to a NodePort on whichever node it picked, where kube-proxy's DNAT rule sends the packet to one of the pods listed in the Service's EndpointSlices. With externalTrafficPolicy left at its default Cluster, the node that first receives the packet can forward it on to a pod on a different node, an extra hop, but with even load spread across all pods. Setting it to Local skips that extra hop and preserves the client's real source IP, at the cost of only routing to pods that happen to already be on the node the load balancer chose, so an uneven pod distribution across nodes turns directly into uneven traffic.

Trade-offs and pitfalls

• ExternalName is often reached for as 'the DNS one' when someone actually wants a placeholder Service for pods that do not exist yet. Because it has no selector and no health checking, nothing about it verifies the external target is even up.
• NodePort's fixed 30000-32767 range and one-port-per-Service-per-node model does not scale past a modest number of exposed Services and puts every node's IP in the attack surface; it is a building block for an external load balancer, rarely the final production answer on its own.
• LoadBalancer provisions one cloud load balancer per Service by default, which gets expensive and slow to provision as the number of Services grows; that pressure is usually solved by putting an Ingress controller behind a single LoadBalancer Service, a platform decision distinct from Service type selection itself.
• kube-proxy's mode is a cluster-wide setting, not a per-Service choice; moving from iptables to ipvs or nftables needs a validated cluster-wide rollout plan, not a Service-by-service change.

A resource request is what the scheduler reserves for a pod when deciding node placement; a limit is the hard ceiling the kubelet enforces once the pod is already running. Kubernetes derives a Quality of Service (QoS) class purely from how requests and limits compare across every container in the pod: Guaranteed, Burstable, or BestEffort. That class, not the raw numbers, is what decides which pod gets evicted first when a node runs low on memory.

QoS classes

Two mechanisms are doing the work here, and they are easy to conflate:

• Scheduling and overcommit: the scheduler only sums container requests to decide whether a pod fits on a node; limits are invisible to it. That is deliberate: setting limits above requests lets a cluster pack pods more densely than their guaranteed usage, trading headroom for cost. It also means a node can be scheduled to 100% of its requested memory while still having every pod's limit sum to well over the node's actual capacity.
• Enforcement at runtime: CPU limits are enforced through the cgroup (Linux control group, the kernel mechanism containers use for resource accounting and isolation) CFS (Completely Fair Scheduler) quota, so exceeding a CPU limit throttles the container rather than killing it. Memory limits are a hard cgroup cap; exceeding one gets the container killed by the Linux OOM killer, reported on the pod as OOMKilled.
• Eviction: separately from OOM kills, the kubelet watches node-level pressure signals (conditions such as MemoryPressure) and proactively evicts pods, ranked by QoS class first and then by how far a pod's usage exceeds its own request within that class.

Worked example

Overcommit arithmetic. Suppose eight pods are each given a memory request of 256MiB and a limit of 512MiB, scheduled onto a node with 4GiB of allocatable memory:
$8 \times 256\text{ MiB} = 2048\text{ MiB} = 2\text{ GiB}$
The scheduler is satisfied at 2GiB of committed requests, well under the node's 4GiB. But if every pod actually used its full limit at once:
$8 \times 512\text{ MiB} = 4096\text{ MiB} = 4\text{ GiB}$
that consumes the entire node with zero headroom left for the OS and the kubelet itself, which is exactly the condition that triggers evictions or OOM kills. The gap between 2GiB and 4GiB is the overcommit the operator is deliberately accepting.

CPU throttling arithmetic. A container with a CPU limit of 0.5 (500 millicores) gets translated into a cgroup CFS quota against the kernel's default 100ms accounting period (cfs_period_us of 100000 microseconds):
$\text{cfs\_quota\_us} = \text{limit (CPUs)} \times \text{cfs\_period\_us}$
$0.5 \times 100000 = 50000$
So the container is allotted 50,000 microseconds of CPU time in every 100,000-microsecond window. Once it burns through that in a burst, the kernel throttles it until the next window opens, which shows up as added latency, not a crash or restart, and is invisible in dashboards that only track average CPU utilization.

Practical right-sizing guidance

• Measure real usage (p95/p99, steady-state and burst) under representative load before setting either number; guessing produces exactly the overcommit risk described above.
• Set requests near the workload's realistic steady-state usage so the scheduler's placement promise matches reality; set limits to cap runaway growth, keeping memory limits close to (or equal to) requests for anything where an OOM kill is worse than wasted capacity.
• Reserve Guaranteed QoS for workloads where losing the pod outright is worse than the density cost (primary databases, payment-critical services); use Burstable for typical stateless services with reasonable headroom; avoid BestEffort in production entirely.

Trade-offs and pitfalls

• Setting requests low purely to make a pod schedule more easily quietly turns Guaranteed-shaped intent into Burstable behavior and makes that pod an earlier eviction candidate than the team expects.
• All-Guaranteed clusters are simple to reason about but give up the density Kubernetes is meant to provide; treat Guaranteed as a scalpel, not a default.
• CPU throttling is easy to miss operationally because it does not appear as an error; a service that looks fine on average CPU utilization can still be throttled in short, latency-relevant bursts unless you scrape the cgroup throttling counters directly.
• Running the Vertical Pod Autoscaler (VPA) in Auto mode on the same metric a Horizontal Pod Autoscaler (HPA) reacts to causes the two to fight each other, since VPA resizing requests changes the utilization the HPA measures; keep VPA in recommendation-only mode, or scope it to memory while HPA drives scaling on CPU or a custom metric.

For 10,000 Pods with strict latency requirements, Cilium is the strongest default choice because its eBPF (extended Berkeley Packet Filter, a Linux kernel technology for running sandboxed programs directly in the kernel) data plane avoids the per-packet overhead that a rule-chain-based CNI (Container Network Interface, the plugin responsible for Pod networking) accumulates at scale, and it ships built-in flow-level observability you would otherwise have to bolt on separately.

Comparison

Data-plane architecture, briefly

• Calico: Felix (the per-node agent) programs either iptables rules or, in eBPF mode, kernel programs directly; BGP (via BIRD) or VXLAN handles the routing/overlay between nodes depending on configuration.
• Cilium: eBPF programs attach at multiple kernel hook points (the network device, and the socket layer for some paths) to forward and enforce policy without traversing the traditional netfilter/iptables stack; Hubble consumes the same eBPF-derived flow data for observability, so visibility isn't a separate tap on the traffic, it's the same data path instrumented.
• Flannel: a simple overlay, most commonly VXLAN, that encapsulates Pod traffic to move it between nodes; there's no independent policy or observability layer because that was never Flannel's design goal, it solves connectivity only.

Recommendation and reasoning for the given scenario

Choose Cilium:

• At 10k Pods, the volume of Service/endpoint churn and cross-node flows makes an eBPF data plane's lower per-packet cost and avoidance of large rule-chain lookups meaningfully better for tail latency than an iptables-heavy approach.
• Hubble gives per-flow latency and error visibility out of the box, which matters operationally at this scale: you need to find the noisy Pod or the failing dependency quickly, not reconstruct it from raw counters.
• Identity-based policy (as opposed to IP-based) holds up better as Pods churn constantly at this scale, since policy doesn't need to be rewritten every time an IP changes.

Caveats before committing

• Current Cilium releases require a reasonably modern Linux kernel (broadly, 5.10 or newer, or an enterprise-distro kernel with the equivalent backported features); this requirement has risen across Cilium versions, so check the specific release you plan to run against your node OS before deciding, not against a number memorized from an older version.
• Validate BPF map sizing (the fixed-capacity kernel tables eBPF programs use to track connections, policies, and endpoint identities) and control-plane behavior in a staging cluster at representative scale, since a smaller cluster won't surface map-limit issues (those tables filling up) or identity-churn (the rate at which Cilium creates and retires per-endpoint security identities as pods are created and removed), both of which only appear near 10k Pods.
• If the kernel requirement can't be met (older managed nodes, a locked-down OS image), Calico's eBPF mode is the fallback; avoid Flannel for a strict-latency requirement, since its overlay encapsulation is working against the goal from the start.

Trade-offs and pitfalls

• eBPF capability differs by kernel version and distribution; "supports eBPF" is not a single yes/no fact independent of the exact kernel you're running.
• Flannel's simplicity is a real advantage for small, low-stakes clusters where policy and observability aren't requirements; it is the wrong comparison baseline once either latency or policy enforcement matters.
• Migrating an existing cluster's CNI is disruptive (it typically requires per-node reconfiguration and often a rolling node replacement), so this decision is much cheaper to get right at cluster creation than to revisit later.

Kubernetes gives every Pod its own IP address on one flat, cluster-wide network where any Pod can reach any other Pod's IP directly, without port-mapping or NAT (Network Address Translation) in the middle. That single assumption is the whole model; everything else (Services, kube-proxy, the CNI plugin) exists to make that flat network real and to add load-balancing and service discovery on top of it.

The core assumption: IP-per-Pod, flat and routable

• Every Pod gets its own IP, allocated from the cluster's Pod network, not from the node's own network.
• Containers inside the same Pod share that IP and can reach each other over localhost.
• Any Pod's IP is expected to be reachable from any node in the cluster, as if the whole cluster were one big L3 (Layer 3, meaning IP-address-level) network, even though physically it spans many separate machines.

This is a deliberate simplification: application code never has to deal with port-mapping or think about which node it's running on to reach another Pod. The cost of that simplification is pushed down into the networking layer that has to make the "flat network" illusion actually true.

How Pod-to-pod traffic actually crosses nodes

Two Pods on the same node talk through a local bridge or virtual interface, no different in spirit from two processes on one machine. Two Pods on different nodes need their packets to physically cross the network between those nodes, and that's the job of the CNI (Container Network Interface) plugin, using one of a few common approaches:

• Overlay/encapsulation (for example VXLAN, used by Flannel by default): the CNI wraps the Pod packet inside another packet addressed node-to-node, then unwraps it on arrival. Simple to run, but every packet pays an encapsulation cost.
• Native routing (for example Calico's BGP mode): nodes advertise routes to each other's Pod IP ranges directly, so packets travel as ordinary routed IP traffic with no wrapping.
• In-kernel eBPF forwarding (for example Cilium): programs attached to the kernel's networking hooks forward Pod traffic with less per-packet overhead than either of the above.

Whichever mechanism is used, kube-proxy is not involved in raw Pod-to-Pod traffic; that's entirely the CNI's job. kube-proxy only comes into play for Service traffic, described next.

Where kube-proxy fits in

A Pod's IP is not stable: Pods get rescheduled, restarted, and replaced with new IPs constantly. A Service gives client code one stable address (a ClusterIP) that represents a group of Pods, and kube-proxy is the component that makes connections to that stable address actually land on one of the current, healthy backend Pods. It does this by programming the node's kernel with forwarding rules (commonly iptables or IPVS, an in-kernel load-balancing feature) built from the Service's list of ready backend Pods. Put simply: the CNI plugin makes any Pod reachable by its own IP; kube-proxy makes a stable Service IP resolve to whichever real Pod IP should currently handle the traffic.

Common limitations and implicit assumptions to watch for

• The underlying network must actually support it. Cloud VPC (Virtual Private Cloud) routing tables, security groups, or on-prem firewalls have to allow the CNI's chosen traffic pattern (raw routed IP, VXLAN-encapsulated UDP, or eBPF-forwarded packets) between every pair of nodes, or the "flat network" assumption breaks silently.
• IP address planning matters. The Pod CIDR (the address range Pods are allocated from) needs to be sized for cluster growth and must not overlap with the VPC's own address space, or routing becomes ambiguous.
• NetworkPolicy is opt-in. Without it, any Pod can reach any other Pod by default (east-west traffic is wide open); the flat-network model is about reachability, not isolation, and isolation has to be added deliberately.
• Encapsulation costs latency and throughput. An overlay CNI adds a real, if small, per-packet tax versus native routing or eBPF; this matters more as traffic volume grows.
• kube-proxy itself has a scaling ceiling. iptables-based rule chains get slower to evaluate as the number of Services and endpoints grows; IPVS or an eBPF-based CNI's own Service handling scales better at high counts.

Worked example

A Pod on Node A (IP 10.244.1.7) calling a Service backed by a Pod on Node B (IP 10.244.2.4):

1. The calling Pod connects to the Service's ClusterIP, say 10.96.10.20:80.
2. kube-proxy's node-local rules (built from the Service's endpoint list) DNAT that connection to 10.244.2.4:8080, the real backend Pod.
3. The CNI plugin now has to deliver a packet addressed to 10.244.2.4, a Pod IP on a different node, across the physical network: an overlay CNI wraps it in a VXLAN packet destined for Node B's real IP; a routed CNI simply routes it there directly using advertised Pod-subnet routes.
4. Node B unwraps (if needed) and delivers the packet to the backend Pod over its local bridge.

Trade-offs and pitfalls for a team new to this model

• Don't assume Pod IPs are stable enough to hardcode anywhere; always address other workloads through a Service name, never a Pod IP directly.
• Introducing a service mesh or NetworkPolicy changes this baseline model by adding sidecar proxies or enforcement points into the path described above; treat this explanation as the foundation those layers build on, not the final picture.
• Multi-cluster or hybrid-cloud setups often break the single-flat-network assumption outright (two clusters don't share one Pod CIDR space by default), which is exactly why solutions in that space exist to stitch separate flat networks back together.

A node showing NotReady means kubelet stopped reporting a healthy heartbeat to the API server, and diagnosis means figuring out whether kubelet itself is broken, the network path between node and control plane is broken, or the underlying kernel/host is in trouble. Which of the three it is usually falls out of reading the node's own Conditions plus the right log source for each layer.

Node conditions to inspect

kubectl describe node <node>

Conditions:
  Type             Status  Reason
  MemoryPressure   False   KubeletHasSufficientMemory
  DiskPressure     False   KubeletHasNoDiskPressure
  PIDPressure      False   KubeletHasSufficientPID
  Ready            False   KubeletNotReady

Kubelet vs network vs kernel

Kubelet-level. Check on the node directly, since kubelet is a host-level service, not a pod you can kubectl logs:

systemctl status kubelet
journalctl -u kubelet -n 200
crictl ps      # confirms the container runtime kubelet talks to is actually up

Network-level. If kubelet is running but can't reach the API server:

ip a; ip route
curl -sk https://<api-server>:6443/healthz

Look for CNI plugin errors in journalctl -u kubelet or the CNI's own logs; a broken CNI on the node can prevent kubelet's own traffic to the control plane even though the process is alive.
Kernel/host-level.

dmesg | tail -n 200
journalctl -k
df -h; free -m

A kernel OOM killer that targets the kubelet or container-runtime process itself (not an application container) produces NotReady directly, which is a distinct case from the container-level OOMKilled covered separately: check dmesg for the OOM killer's target process name specifically, not just that an OOM event happened somewhere on the box.

Three linked scenarios worth naming explicitly

What happens to the node's existing pods. NotReady does not immediately delete anything. Kubernetes applies a node.kubernetes.io/not-ready:NoExecute taint automatically, and pods get a default tolerationSeconds of 300 (5 minutes) for that taint unless they specify otherwise. Only after that window expires are the pods actually evicted; if they're owned by a Deployment or ReplicaSet, a replacement gets scheduled elsewhere, which can itself go Pending if there isn't spare capacity, linking this scenario directly to the pod-scheduling case above.

Mass NotReady right after a DaemonSet rollout. If several nodes go NotReady at roughly the same time, the more likely cause is a shared node-level agent, not N independent hardware failures. A DaemonSet update to something like the CNI plugin, a log agent, or a security agent can crash-loop node-side or consume enough host resources to break kubelet's own health loop. Check:

kubectl rollout history daemonset/<name> -n kube-system

and correlate the timestamp of the NotReady onset against the DaemonSet's rollout time; kubectl rollout undo daemonset/<name> is usually the faster, safer move than debugging individual nodes one at a time.

Safe cordon and drain

Cordoning marks the node unschedulable, so the scheduler stops placing any new pods on it without touching what is already running there. Draining then evicts those existing pods, respecting each one's PodDisruptionBudget (PDB, an object that caps how many replicas of a workload can be voluntarily unavailable at once), so they get rescheduled onto healthy nodes before you take this one down for maintenance.

kubectl cordon <node>
kubectl get pdb -A                      # check PodDisruptionBudgets first
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

--delete-local-data is an old flag name that was deprecated and later removed; --delete-emptydir-data is the current one. After maintenance:

kubectl uncordon <node>

Trade-offs and pitfalls

• Don't drain immediately on every NotReady; if it's a transient network blip that self-heals in under the 300s toleration window, a drain is unnecessary churn. Give it a short grace period unless the node is clearly compromised (disk full, kernel panic).
• A single NotReady node is usually a node problem; several at once is usually a shared-dependency problem (DaemonSet, a common upstream network path, or a control-plane-side issue), and treating it as N separate incidents wastes time.