Kubernetes Architecture, Operations, and Troubleshooting Questions

A Kubernetes Service is a stable virtual address in front of a changing set of pods. The four types differ in who can reach that address and how: ClusterIP is internal-only, NodePort adds a port opened on every node, LoadBalancer asks the cloud provider to provision an external load balancer in front of that, and ExternalName is different in kind: a DNS alias to something outside the cluster with no pod routing involved at all.

Service types

A fifth variant sits alongside these four rather than replacing any of them: a headless Service (clusterIP: None). It skips kube-proxy's virtual-IP rewriting entirely, so instead of one stable IP load-balancing across pods, CoreDNS returns the individual pod IPs behind the Service directly to the client (sourced from the same EndpointSlices kube-proxy would otherwise consume). This is why StatefulSets pair with a headless Service: each replica needs its own stable, individually addressable DNS name rather than one shared address in front of all of them.

kube-proxy: the mechanism underneath

kube-proxy is what actually implements the ClusterIP, NodePort, and LoadBalancer rewriting, and it has more than one implementation:

• iptables mode (the long-standing default): programs Linux netfilter rules to destination-NAT (DNAT, rewriting a packet's destination address) Service traffic to a pod IP. Simple, but rule evaluation is roughly linear in Service count, which shows up as latency at high Service counts.
• ipvs mode (IP Virtual Server): uses the kernel's IPVS load-balancing tables instead of a rule chain, giving better performance and more scheduling algorithm choices at scale.
• nftables mode: the newer replacement backend, built on the modern Linux nftables framework, which reached general availability in Kubernetes 1.33. It targets the same correctness as iptables mode with materially better performance at high Service counts, though iptables remains the cluster-wide default for now.
  Older material sometimes still mentions a userspace mode; it was deprecated years ago and fully removed in Kubernetes 1.26, so it should not appear in any current design.

Worked example: tracing one request through a LoadBalancer Service

flowchart LR
    Client --> ELB[Cloud load balancer]
    ELB --> NP[Node: NodePort]
    NP --> KP[kube-proxy DNAT rule]
    KP --> PodA[Pod A]
    KP --> PodB[Pod B]

The client hits the cloud load balancer's public address, which forwards to a NodePort on whichever node it picked, where kube-proxy's DNAT rule sends the packet to one of the pods listed in the Service's EndpointSlices. With externalTrafficPolicy left at its default Cluster, the node that first receives the packet can forward it on to a pod on a different node, an extra hop, but with even load spread across all pods. Setting it to Local skips that extra hop and preserves the client's real source IP, at the cost of only routing to pods that happen to already be on the node the load balancer chose, so an uneven pod distribution across nodes turns directly into uneven traffic.

Trade-offs and pitfalls

• ExternalName is often reached for as 'the DNS one' when someone actually wants a placeholder Service for pods that do not exist yet. Because it has no selector and no health checking, nothing about it verifies the external target is even up.
• NodePort's fixed 30000-32767 range and one-port-per-Service-per-node model does not scale past a modest number of exposed Services and puts every node's IP in the attack surface; it is a building block for an external load balancer, rarely the final production answer on its own.
• LoadBalancer provisions one cloud load balancer per Service by default, which gets expensive and slow to provision as the number of Services grows; that pressure is usually solved by putting an Ingress controller behind a single LoadBalancer Service, a platform decision distinct from Service type selection itself.
• kube-proxy's mode is a cluster-wide setting, not a per-Service choice; moving from iptables to ipvs or nftables needs a validated cluster-wide rollout plan, not a Service-by-service change.

Pods are the runtime unit; everything else here is a controller deciding how many pods to run and how to replace them. A ReplicaSet just keeps N identical pods alive. A Deployment wraps a ReplicaSet with declarative rolling updates and rollback history. A StatefulSet keeps stable per-replica identity and storage for workloads that care which replica they are. A DaemonSet guarantees exactly one pod per matching node. Job and CronJob run pods to completion, once or on a schedule, rather than keeping them running indefinitely.

Comparison

Worked example: rolling-update arithmetic

For a Deployment with replicas: 10, maxSurge: 25%, and maxUnavailable: 25%, Kubernetes rounds surge up and unavailable down:
$\text{surge} = \lceil 10 \times 0.25 \rceil = 3$
$\text{unavailable} = \lfloor 10 \times 0.25 \rfloor = 2$
So at the busiest point of the rollout the Deployment can briefly run up to 13 pods (10 plus 3 surge) while guaranteeing at least 8 stay available (10 minus 2 unavailable). This asymmetric rounding is why setting surge and unavailable to the same percentage still lets the total pod count grow slightly during a rollout instead of staying flat.

DaemonSets interact with node taints in a way that is easy to miss: they commonly carry an explicit toleration for the control-plane taint (conventionally node-role.kubernetes.io/control-plane:NoSchedule) so node-level agents like a CNI plugin or kube-proxy itself still run on control-plane nodes that ordinary application pods are excluded from. That toleration is a manifest choice the DaemonSet author makes, not automatic behavior.

Trade-offs and pitfalls

• Choosing Deployment for a workload that actually needs stable identity (a small quorum service, a primary-replica database) forces the team to rebuild that identity logic in application code; StatefulSet gives it for free, at the cost of slower, ordered rollouts.
• A stuck rollout is usually a readiness-probe failure hiding behind the surge and unavailable math: if new pods never pass readiness, the Deployment can stall indefinitely partway through, with old and new pods coexisting. Check pod events and readiness status before assuming a controller bug.
• A PodDisruptionBudget set too strictly (for example, minAvailable equal to the full replica count) can make a Deployment's own rolling update unable to proceed, because the update's own temporary unavailability trips the budget meant to protect against unrelated disruptions.
• DaemonSets bypass normal replica-count thinking: adding nodes silently adds pods and cost. Forgetting a DaemonSet exists across a large node pool is a common source of untracked resource consumption.

CrashLoopBackOff means the container keeps exiting and the kubelet is backing off its restart attempts with exponential delay (10s, 20s, 40s, up to a 5-minute cap). It is a symptom, not a root cause: something inside the container is exiting, or something outside it is killing the container, and the job is to figure out which.

Step-by-step diagnosis

1. Describe the pod first. This shows the Events list and the container's last termination reason before you even look at logs.

kubectl describe pod <pod> -n <ns>

2. Read current and previous logs. The -p (previous) flag is what most people forget, and it's the only way to see output from the crashed instance once it has already restarted.

kubectl logs <pod> -c <container> -n <ns>
kubectl logs <pod> -c <container> -p -n <ns>

3. Check the exit code, visible in describe's Last State block.

4. Separate "app crashed" from "kubelet killed it." A liveness probe failure produces its own event and a different signature than an application-level crash:

Containers:
  app:
    State:      Waiting
      Reason:   CrashLoopBackOff
    Last State: Terminated
      Reason:   Error
      Exit Code: 1
    Restart Count: 5
Events:
  Warning  Unhealthy  90s (x3 over 3m)   kubelet  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": dial tcp 10.1.2.3:8080: connect: connection refused
  Warning  BackOff    30s (x12 over 6m)  kubelet  Back-off restarting failed container

If the events show Liveness probe failed right before the restart, the app may actually be starting fine but too slowly, or the probe is pointed at the wrong port or path, and kubelet is killing a container that would otherwise have recovered.
5. Check image and config next, since a large share of CrashLoopBackOff cases are not code bugs: wrong environment variable, a ConfigMap/Secret key that doesn't exist, a missing volume mount, or an image tag that doesn't match what's expected.

Reproduce and test a fix safely

• Recreate the same image/env in a disposable pod without a restart policy fighting you:

kubectl run debug --image=<image> --restart=Never -n dev --env="KEY=value" --command -- sleep 3600
kubectl exec -it debug -n dev -- /bin/sh

• If you suspect the liveness probe is the culprit, don't leave it disabled permanently; temporarily loosen initialDelaySeconds or failureThreshold in a copy of the manifest in a non-prod namespace, confirm the app actually stabilizes, then fix the probe config for real rather than shipping it disabled.
• Once you have a fix, patch and roll it out normally rather than editing the live object by hand:

kubectl set image deploy/<d> <c>=<image>:<new-tag> -n <ns>
kubectl rollout status deploy/<d> -n <ns>

• Capture the failing pod's YAML before you touch anything (kubectl get pod <pod> -o yaml), since the evidence disappears once the pod is replaced.

Trade-offs and pitfalls

• Don't "fix" a crash loop by disabling the liveness probe or setting restartPolicy more leniently; that hides the symptom and turns a visible failure into a silent one.
• Exit code 137 is a strong hint but not proof of OOMKilled; always confirm via the Reason field in describe, since a probe-triggered kill or an external kill -9 produces the same code.
• If a pod never produces logs at all before crashing, the failure is likely before your application's logging even initializes (missing dependency, bad entrypoint, immediate segfault); at that point kubectl logs -p is empty and you have to fall back to the exit code, describe events, and possibly running the image manually to see stdout that never reached the log driver.

NetworkPolicy is a namespaced Kubernetes object that restricts which pods can talk to which other pods (and, via egress rules, which external destinations pods can reach), solving east-west traffic control inside the cluster: without it, any pod can reach any other pod by default, which is a large blast radius for a compromised or misconfigured workload.

The enforcement point that isn't the API server

Creating a NetworkPolicy object only stores intent in etcd via the API server, the same as any other Kubernetes object; the API server itself does not inspect or block a single packet. Enforcement is entirely the responsibility of the CNI (container network interface) plugin running on each node, and only if that plugin actually implements NetworkPolicy support. Calico, Cilium, and several others do; a purely overlay-focused CNI like plain Flannel historically does not. Applying a NetworkPolicy on a cluster whose CNI doesn't enforce it is a silent no-op: kubectl get networkpolicy shows the object as created, but traffic flows exactly as if it didn't exist. Confirming the CNI's NetworkPolicy support is a prerequisite, not an afterthought, before relying on this mechanism for anything security-relevant.

Expressing ingress and egress

A policy selects pods with podSelector, and, once selected, ingress and egress traffic is governed independently:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-policy
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector: { matchLabels: { role: frontend } }
    ports:
    - port: 80
  egress:
  - to:
    - namespaceSelector: { matchLabels: { env: services } }
    ports:
    - port: 5432

This says: pods labeled app: web accept ingress only from pods labeled role: frontend on port 80, and may only send egress to pods in namespaces labeled env: services on port 5432. Everything else in that direction is implicitly denied once any rule for that direction exists.

Selectors and namespaceSelector

• podSelector inside from/to matches pods by label within the same namespace as the rule's target namespace context.
• namespaceSelector matches whole namespaces by label; commonly combined with podSelector in the same from/to entry to mean "these specific pods, but only in namespaces with this label," rather than either alone.
• An empty podSelector: {} at the top level of the spec selects every pod in the namespace; an empty selector inside a from/to entry means "all pods" for that source/destination.

Default behavior

With no NetworkPolicy selecting a pod at all, every ingress and egress connection to and from it is allowed, no restriction whatsoever. The moment at least one policy selects a pod and declares a policyTypes direction (say, Ingress), that direction flips to default-deny for that pod: only traffic matching an explicit rule is allowed, and everything else in that direction is dropped. This is why a common, deliberate starting point is a namespace-wide default-deny policy (a policy selecting all pods with empty ingress/egress rules) followed by narrow allow rules added on top, rather than trying to enumerate every exception against an open-by-default baseline.

Trade-offs and pitfalls

• NetworkPolicies are additive across multiple policies selecting the same pod (a union of allowed traffic, not an intersection), which means adding a second, broader policy to a namespace can unintentionally widen access that a narrower first policy had restricted.
• Egress policies frequently get overlooked because ingress is the more obvious security concern, but an over-permissive default (no egress policy at all) lets a compromised pod exfiltrate data or reach internal services it has no business reaching; DNS resolution in particular is easy to break by an egress default-deny policy that forgets to allow port 53 to the cluster's DNS service.
• Because enforcement lives in the CNI, NetworkPolicy behavior (which selector combinations are supported, whether egress is supported at all, performance under many rules) varies by CNI choice; a policy that works correctly in one cluster is not guaranteed to behave identically after a CNI migration.

Kubernetes gives every Pod its own IP address on one flat, cluster-wide network where any Pod can reach any other Pod's IP directly, without port-mapping or NAT (Network Address Translation) in the middle. That single assumption is the whole model; everything else (Services, kube-proxy, the CNI plugin) exists to make that flat network real and to add load-balancing and service discovery on top of it.

The core assumption: IP-per-Pod, flat and routable

• Every Pod gets its own IP, allocated from the cluster's Pod network, not from the node's own network.
• Containers inside the same Pod share that IP and can reach each other over localhost.
• Any Pod's IP is expected to be reachable from any node in the cluster, as if the whole cluster were one big L3 (Layer 3, meaning IP-address-level) network, even though physically it spans many separate machines.

This is a deliberate simplification: application code never has to deal with port-mapping or think about which node it's running on to reach another Pod. The cost of that simplification is pushed down into the networking layer that has to make the "flat network" illusion actually true.

How Pod-to-pod traffic actually crosses nodes

Two Pods on the same node talk through a local bridge or virtual interface, no different in spirit from two processes on one machine. Two Pods on different nodes need their packets to physically cross the network between those nodes, and that's the job of the CNI (Container Network Interface) plugin, using one of a few common approaches:

• Overlay/encapsulation (for example VXLAN, used by Flannel by default): the CNI wraps the Pod packet inside another packet addressed node-to-node, then unwraps it on arrival. Simple to run, but every packet pays an encapsulation cost.
• Native routing (for example Calico's BGP mode): nodes advertise routes to each other's Pod IP ranges directly, so packets travel as ordinary routed IP traffic with no wrapping.
• In-kernel eBPF forwarding (for example Cilium): programs attached to the kernel's networking hooks forward Pod traffic with less per-packet overhead than either of the above.

Whichever mechanism is used, kube-proxy is not involved in raw Pod-to-Pod traffic; that's entirely the CNI's job. kube-proxy only comes into play for Service traffic, described next.

Where kube-proxy fits in

A Pod's IP is not stable: Pods get rescheduled, restarted, and replaced with new IPs constantly. A Service gives client code one stable address (a ClusterIP) that represents a group of Pods, and kube-proxy is the component that makes connections to that stable address actually land on one of the current, healthy backend Pods. It does this by programming the node's kernel with forwarding rules (commonly iptables or IPVS, an in-kernel load-balancing feature) built from the Service's list of ready backend Pods. Put simply: the CNI plugin makes any Pod reachable by its own IP; kube-proxy makes a stable Service IP resolve to whichever real Pod IP should currently handle the traffic.

Common limitations and implicit assumptions to watch for

• The underlying network must actually support it. Cloud VPC (Virtual Private Cloud) routing tables, security groups, or on-prem firewalls have to allow the CNI's chosen traffic pattern (raw routed IP, VXLAN-encapsulated UDP, or eBPF-forwarded packets) between every pair of nodes, or the "flat network" assumption breaks silently.
• IP address planning matters. The Pod CIDR (the address range Pods are allocated from) needs to be sized for cluster growth and must not overlap with the VPC's own address space, or routing becomes ambiguous.
• NetworkPolicy is opt-in. Without it, any Pod can reach any other Pod by default (east-west traffic is wide open); the flat-network model is about reachability, not isolation, and isolation has to be added deliberately.
• Encapsulation costs latency and throughput. An overlay CNI adds a real, if small, per-packet tax versus native routing or eBPF; this matters more as traffic volume grows.
• kube-proxy itself has a scaling ceiling. iptables-based rule chains get slower to evaluate as the number of Services and endpoints grows; IPVS or an eBPF-based CNI's own Service handling scales better at high counts.

Worked example

A Pod on Node A (IP 10.244.1.7) calling a Service backed by a Pod on Node B (IP 10.244.2.4):

1. The calling Pod connects to the Service's ClusterIP, say 10.96.10.20:80.
2. kube-proxy's node-local rules (built from the Service's endpoint list) DNAT that connection to 10.244.2.4:8080, the real backend Pod.
3. The CNI plugin now has to deliver a packet addressed to 10.244.2.4, a Pod IP on a different node, across the physical network: an overlay CNI wraps it in a VXLAN packet destined for Node B's real IP; a routed CNI simply routes it there directly using advertised Pod-subnet routes.
4. Node B unwraps (if needed) and delivers the packet to the backend Pod over its local bridge.

Trade-offs and pitfalls for a team new to this model

• Don't assume Pod IPs are stable enough to hardcode anywhere; always address other workloads through a Service name, never a Pod IP directly.
• Introducing a service mesh or NetworkPolicy changes this baseline model by adding sidecar proxies or enforcement points into the path described above; treat this explanation as the foundation those layers build on, not the final picture.
• Multi-cluster or hybrid-cloud setups often break the single-flat-network assumption outright (two clusters don't share one Pod CIDR space by default), which is exactly why solutions in that space exist to stitch separate flat networks back together.