Kubernetes Architecture, Operations, and Troubleshooting Questions

Direct answer

Every pod gets a routable IP address from the CNI (Container Network Interface) plugin, with no NAT (network address translation) between pods by default. A Service layers a stable virtual IP and DNS name on top of that, and kube-proxy (or an eBPF-based replacement like Cilium; eBPF lets Linux run sandboxed programs inside the kernel to process packets directly) programs the actual rewriting rules that turn a Service IP into a real pod IP. When two pods on different nodes cannot reach each other on a specific port, the fault sits in exactly one of four layers, checked in this order: the pod's own listener, the kube-proxy/Service plumbing, the CNI's cross-node routing, or a host/cloud firewall. Testing each layer directly beats guessing from symptoms.

Structured elaboration

Pod IP model

The CNI plugin assigns each pod a unique IP from the cluster's pod address range and is responsible for making that IP reachable from every node, either by programming host routes directly (an "underlay" approach, for example Calico in BGP mode, Border Gateway Protocol, the routing protocol routers use to exchange reachability information) or by encapsulating pod traffic in an overlay (VXLAN, a tunneling protocol that wraps one Ethernet frame inside a UDP packet, used by Flannel and by Calico's IP-in-IP mode).

Service virtual IPs and kube-proxy modes

A ClusterIP Service is a stable virtual IP that never corresponds to a real listener; kube-proxy rewrites traffic to it into a real pod IP and port.

Cilium and some other CNIs bypass kube-proxy entirely, attaching eBPF programs to kernel networking hooks to do the same rewriting job with lower per-packet overhead and without iptables' rule-count scaling problem.

CNI plugins in practice

• Calico: pure L3 routing via BGP (no overlay, lowest overhead, needs a BGP-capable network) or IP-in-IP/VXLAN encapsulation where BGP peering is not available.
• Flannel: a simple VXLAN overlay by default; fewer features than Calico or Cilium.
• Cilium: eBPF-based; can replace kube-proxy, adds L7-aware NetworkPolicy and traffic observability, at the cost of needing a reasonably current kernel.

flowchart LR
  A[Pod A on Node 1] -->|packet to Service ClusterIP| KP1[kube-proxy or eBPF dataplane, Node 1]
  KP1 -->|rewrite to real Pod IP| CNI1[CNI routing or overlay, Node 1]
  CNI1 -->|cross-node route or tunnel| CNI2[CNI routing or overlay, Node 2]
  CNI2 --> B[Pod B on Node 2]
  FW[Host firewall or cloud security group] -.may block specific ports.-> CNI2

Debugging cross-node connectivity, layer by layer

Reproduce and localize before changing anything:

$ kubectl get pod -o wide -n app
NAME      READY   STATUS    NODE     IP
pod-a     1/1     Running   node-1   10.244.1.7
pod-b     1/1     Running   node-2   10.244.2.9

$ kubectl exec -n app pod-a -- nc -vz 10.244.2.9 8080
nc: connect to 10.244.2.9 port 8080 (tcp) failed: Connection timed out

A timeout, not "connection refused," means the packet is being dropped somewhere in transit rather than rejected by pod-b's listener; that already rules out "the app isn't listening" and points at routing, the CNI, or a firewall.

1. Confirm the listener exists: kubectl exec -n app pod-b -- ss -tlnp should show something bound to 8080; if not, this was never a networking problem.
2. Check node-to-node routing for the pod's address range:

$ ip route get 10.244.2.9
10.244.2.9 via 10.0.1.2 dev eth0 src 10.0.1.5

A missing route, or one pointing at the wrong interface, means the CNI failed to program cross-node routing, usually because its daemonset is crashlooping:

$ kubectl get pods -n kube-system -l k8s-app=calico-node
NAME                READY   STATUS             RESTARTS
calico-node-x7k2p   0/1     CrashLoopBackOff   14

3. If the failure is specifically Service-IP traffic rather than direct pod-IP traffic, check kube-proxy's own health the same way (kubectl get pods -n kube-system -l k8s-app=kube-proxy) and inspect its programmed rules (sudo iptables -t nat -S | grep <service-name> or sudo ipvsadm -Ln, depending on mode).
4. If routing and the CNI both look healthy, check host-level and cloud firewall rules for the specific port. CNI and kube-proxy problems tend to affect all ports uniformly; a single-port failure with everything else in that layer healthy is a strong signal for a firewall rule scoped to that port.
5. Capture packets at the suspect boundary to confirm rather than infer: sudo tcpdump -n -i any host 10.244.2.9 and port 8080 on the destination node shows whether the packet arrives at the host at all, before deciding whether the problem is upstream (routing) or downstream (host firewall, pod network namespace).

Trade-offs and pitfalls

• iptables rule count grows linearly with Services and endpoints and becomes a real CPU cost past a few thousand Services; IPVS or nftables scale better, but changing kube-proxy mode cluster-wide is a disruptive change to plan and test, not to flip live.
• The distinction between "Connection timed out" and "Connection refused" is easy to skip past but changes where to look first: refused means the packet arrived and something rejected it (wrong port, no listener); timed out means the packet was dropped somewhere before that.
• MTU (maximum transmission unit) mismatches between overlay encapsulation and the underlying network are a classic cause of connections that work for small packets and hang for large ones. Confirm with a full-size, non-fragmenting ping rather than assuming a working small ping rules out MTU.

A Pod's DNS query for my-service.default.svc.cluster.local goes to CoreDNS, which answers from Service and EndpointSlice objects it watches from the API server; kube-proxy separately programs the node's packet-forwarding rules from those same EndpointSlice objects, so the DNS answer and the actual routing path are produced by two independent watchers reading the same underlying data. For a headless Service, CoreDNS skips the middle step entirely and hands back backend Pod IPs directly.

1) The Pod's DNS query

Every Pod's /etc/resolv.conf is set by the kubelet to point at the cluster DNS Service's ClusterIP (CoreDNS), plus a search-domain list (default.svc.cluster.local, svc.cluster.local, cluster.local) that lets short names like my-service resolve without the full suffix. The application's query goes out as a normal UDP (falling back to TCP for large responses) request to that ClusterIP on port 53.

2) CoreDNS: where the answer comes from

CoreDNS runs the kubernetes plugin, which watches the API server for Service and EndpointSlice objects (an EndpointSlice groups the ready backend Pod IPs and ports for a Service; it replaced the older single Endpoints object specifically so that large services don't require rewriting one giant object on every Pod change). Current CoreDNS releases watch EndpointSlices exclusively for this data; they do not fall back to the older Endpoints API, which matters if you're running a pre-1.21-era cluster still on v1beta1 EndpointSlices.

• Normal (ClusterIP) Service: CoreDNS returns a synthetic A record for the Service's stable ClusterIP, plus SRV records for named ports. The Pod's connection always lands on that one virtual IP.
• Headless Service (clusterIP: None): CoreDNS instead returns one A record per ready backend Pod IP, taken straight from the EndpointSlice. The client picks (or round-robins across) an actual Pod IP and connects to it directly. This is also how a StatefulSet's per-replica DNS names (pod-0.my-service.default.svc.cluster.local) resolve, since StatefulSets are built on a headless Service.

Because CoreDNS reacts to API watch events, a new or removed endpoint typically propagates into DNS within a second or two of the API server processing the change, not on a polling interval. Client-side and CoreDNS-side caching still add their own delay on top of that (see pitfalls below).

3) kube-proxy: turning EndpointSlice data into packet-forwarding rules

kube-proxy runs on every node, watches the same Service and EndpointSlice objects, and programs the node's kernel accordingly. It does not sit in the data path itself; it configures rules the kernel then executes per-packet.

Regardless of mode, the rule shape is the same idea: "packets to Service ClusterIP:port get DNAT'd (Destination Network Address Translation, rewriting the destination address) to one of the ready backend Pod IP:port pairs."

4) Runtime packet path

sequenceDiagram
    participant API as API Server
    participant DNS as CoreDNS
    participant App as Client Pod
    participant KP as Node kernel (kube-proxy rules)
    participant BE as Backend Pod
    API-->>DNS: watch Service + EndpointSlice (continuous)
    API-->>KP: watch Service + EndpointSlice (continuous)
    App->>DNS: query my-service.default.svc.cluster.local
    DNS-->>App: ClusterIP A record (or Pod IPs if headless)
    App->>KP: connect to ClusterIP:port
    KP->>BE: DNAT to a ready endpoint Pod IP:port
    BE-->>App: response

• If the chosen backend Pod is on the same node, the packet is delivered locally through the CNI's (Container Network Interface, the plugin that wires up Pod networking) bridge or veth pair without leaving the host.
• If it's on another node, the packet is routed (or encapsulated, depending on the CNI) across the node network, and some CNI/kube-proxy combinations perform source NAT on cross-node traffic so return packets route back correctly.
• For a headless Service, there's no DNAT step at all: CoreDNS already gave the client a real Pod IP, so kube-proxy is not involved in that connection.
• Only Pods that are Ready (passing their readiness probe) appear in the EndpointSlice at all, so an unready Pod is invisible to both DNS and kube-proxy simultaneously, not resolvable and not routed to.

Worked example: representative output

$ kubectl get endpointslices -l kubernetes.io/service-name=my-service
NAME               ADDRESSTYPE   PORTS   ENDPOINTS
my-service-x7f2q   IPv4          8080    10.244.1.12,10.244.2.9

$ dig +short my-service.default.svc.cluster.local
10.96.140.201

10.96.140.201 is the stable ClusterIP; 10.244.1.12 / 10.244.2.9 are the actual Pod IPs kube-proxy's rules will DNAT to. If the same Service were headless, the dig output would show 10.244.1.12 and 10.244.2.9 directly instead of the ClusterIP.

Trade-offs and pitfalls

• DNS record churn at scale: a Service with rapidly changing backends (frequent rollouts, aggressive autoscaling) generates a steady stream of EndpointSlice updates. CoreDNS itself answers from its watch cache correctly, but application-level and node-level DNS caching (and any fixed TTL a client library applies) can serve a stale backend IP after that Pod is gone; keep client-side DNS caching TTLs short for volatile Services rather than assuming CoreDNS's freshness is the only factor.
• Conntrack (connection tracking) table exhaustion or stale entries on a node can cause traffic to silently drop even though both DNS and the iptables/IPVS rules are correct; this is a distinct failure mode from DNS or kube-proxy misconfiguration and needs node-level conntrack metrics to diagnose.
• A Pod that resolves DNS successfully but still can't reach the backend usually means the DNS layer and the kube-proxy/CNI layer are fine but something else (NetworkPolicy, a firewall, or the backend Pod itself) is blocking the connection; treat DNS success as ruling out one layer, not all of them.

For 10,000 Pods with strict latency requirements, Cilium is the strongest default choice because its eBPF (extended Berkeley Packet Filter, a Linux kernel technology for running sandboxed programs directly in the kernel) data plane avoids the per-packet overhead that a rule-chain-based CNI (Container Network Interface, the plugin responsible for Pod networking) accumulates at scale, and it ships built-in flow-level observability you would otherwise have to bolt on separately.

Comparison

Data-plane architecture, briefly

• Calico: Felix (the per-node agent) programs either iptables rules or, in eBPF mode, kernel programs directly; BGP (via BIRD) or VXLAN handles the routing/overlay between nodes depending on configuration.
• Cilium: eBPF programs attach at multiple kernel hook points (the network device, and the socket layer for some paths) to forward and enforce policy without traversing the traditional netfilter/iptables stack; Hubble consumes the same eBPF-derived flow data for observability, so visibility isn't a separate tap on the traffic, it's the same data path instrumented.
• Flannel: a simple overlay, most commonly VXLAN, that encapsulates Pod traffic to move it between nodes; there's no independent policy or observability layer because that was never Flannel's design goal, it solves connectivity only.

Recommendation and reasoning for the given scenario

Choose Cilium:

• At 10k Pods, the volume of Service/endpoint churn and cross-node flows makes an eBPF data plane's lower per-packet cost and avoidance of large rule-chain lookups meaningfully better for tail latency than an iptables-heavy approach.
• Hubble gives per-flow latency and error visibility out of the box, which matters operationally at this scale: you need to find the noisy Pod or the failing dependency quickly, not reconstruct it from raw counters.
• Identity-based policy (as opposed to IP-based) holds up better as Pods churn constantly at this scale, since policy doesn't need to be rewritten every time an IP changes.

Caveats before committing

• Current Cilium releases require a reasonably modern Linux kernel (broadly, 5.10 or newer, or an enterprise-distro kernel with the equivalent backported features); this requirement has risen across Cilium versions, so check the specific release you plan to run against your node OS before deciding, not against a number memorized from an older version.
• Validate BPF map sizing (the fixed-capacity kernel tables eBPF programs use to track connections, policies, and endpoint identities) and control-plane behavior in a staging cluster at representative scale, since a smaller cluster won't surface map-limit issues (those tables filling up) or identity-churn (the rate at which Cilium creates and retires per-endpoint security identities as pods are created and removed), both of which only appear near 10k Pods.
• If the kernel requirement can't be met (older managed nodes, a locked-down OS image), Calico's eBPF mode is the fallback; avoid Flannel for a strict-latency requirement, since its overlay encapsulation is working against the goal from the start.

Trade-offs and pitfalls

• eBPF capability differs by kernel version and distribution; "supports eBPF" is not a single yes/no fact independent of the exact kernel you're running.
• Flannel's simplicity is a real advantage for small, low-stakes clusters where policy and observability aren't requirements; it is the wrong comparison baseline once either latency or policy enforcement matters.
• Migrating an existing cluster's CNI is disruptive (it typically requires per-node reconfiguration and often a rolling node replacement), so this decision is much cheaper to get right at cluster creation than to revisit later.

NetworkPolicy is a namespaced Kubernetes object that restricts which pods can talk to which other pods (and, via egress rules, which external destinations pods can reach), solving east-west traffic control inside the cluster: without it, any pod can reach any other pod by default, which is a large blast radius for a compromised or misconfigured workload.

The enforcement point that isn't the API server

Creating a NetworkPolicy object only stores intent in etcd via the API server, the same as any other Kubernetes object; the API server itself does not inspect or block a single packet. Enforcement is entirely the responsibility of the CNI (container network interface) plugin running on each node, and only if that plugin actually implements NetworkPolicy support. Calico, Cilium, and several others do; a purely overlay-focused CNI like plain Flannel historically does not. Applying a NetworkPolicy on a cluster whose CNI doesn't enforce it is a silent no-op: kubectl get networkpolicy shows the object as created, but traffic flows exactly as if it didn't exist. Confirming the CNI's NetworkPolicy support is a prerequisite, not an afterthought, before relying on this mechanism for anything security-relevant.

Expressing ingress and egress

A policy selects pods with podSelector, and, once selected, ingress and egress traffic is governed independently:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-policy
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector: { matchLabels: { role: frontend } }
    ports:
    - port: 80
  egress:
  - to:
    - namespaceSelector: { matchLabels: { env: services } }
    ports:
    - port: 5432

This says: pods labeled app: web accept ingress only from pods labeled role: frontend on port 80, and may only send egress to pods in namespaces labeled env: services on port 5432. Everything else in that direction is implicitly denied once any rule for that direction exists.

Selectors and namespaceSelector

• podSelector inside from/to matches pods by label within the same namespace as the rule's target namespace context.
• namespaceSelector matches whole namespaces by label; commonly combined with podSelector in the same from/to entry to mean "these specific pods, but only in namespaces with this label," rather than either alone.
• An empty podSelector: {} at the top level of the spec selects every pod in the namespace; an empty selector inside a from/to entry means "all pods" for that source/destination.

Default behavior

With no NetworkPolicy selecting a pod at all, every ingress and egress connection to and from it is allowed, no restriction whatsoever. The moment at least one policy selects a pod and declares a policyTypes direction (say, Ingress), that direction flips to default-deny for that pod: only traffic matching an explicit rule is allowed, and everything else in that direction is dropped. This is why a common, deliberate starting point is a namespace-wide default-deny policy (a policy selecting all pods with empty ingress/egress rules) followed by narrow allow rules added on top, rather than trying to enumerate every exception against an open-by-default baseline.

Trade-offs and pitfalls

• NetworkPolicies are additive across multiple policies selecting the same pod (a union of allowed traffic, not an intersection), which means adding a second, broader policy to a namespace can unintentionally widen access that a narrower first policy had restricted.
• Egress policies frequently get overlooked because ingress is the more obvious security concern, but an over-permissive default (no egress policy at all) lets a compromised pod exfiltrate data or reach internal services it has no business reaching; DNS resolution in particular is easy to break by an egress default-deny policy that forgets to allow port 53 to the cluster's DNS service.
• Because enforcement lives in the CNI, NetworkPolicy behavior (which selector combinations are supported, whether egress is supported at all, performance under many rules) varies by CNI choice; a policy that works correctly in one cluster is not guaranteed to behave identically after a CNI migration.

An Ingress object declares what HTTP(S) routing should happen (hostnames, path rules, which Service backs each path, which TLS secret to use); an Ingress Controller is the running component that watches Ingress objects and actually implements that routing on real infrastructure. Kubernetes splits these into two objects for the same reason it splits a Deployment's spec from the controller that reconciles it: the API object stays a stable, portable declaration, while the implementation (which varies enormously between environments) is free to be swapped out without changing what you wrote.

What the Ingress object itself declares

An Ingress is pure declaration, no execution: a list of host/path rules, each pointing at a backend Service and port, plus optional TLS configuration referencing a Secret containing a certificate and key. It has no opinion about how that routing gets enforced; by itself, an Ingress object sitting in the API server does nothing.

What the Ingress Controller does with it

The controller (for example nginx-ingress, Traefik, or a cloud provider's own controller such as GKE's or AGIC for Azure) watches for Ingress objects and translates the declared rules into a real, running configuration: it opens listener ports, terminates TLS using the referenced Secret, and applies the host/path routing logic, typically by configuring an underlying proxy (like nginx) or provisioning a cloud load balancer.

Why split the two, contrasted with a Service of type LoadBalancer

A Service of type LoadBalancer works at L4 (Layer 4, the TCP/UDP transport layer): it exposes one Service on one IP and port, with no visibility into HTTP hostnames or paths. Ingress works at L7 (Layer 7, the application layer): it can route many hostnames and paths to many different backend Services through a single entry point, but only because something understands HTTP well enough to look inside the request to make that decision, which is exactly the controller's job.

Why split the responsibility this way

• No single "correct" implementation exists. Different environments need genuinely different routing implementations (self-hosted nginx, a cloud provider's managed load balancer, a service mesh's gateway); a single hardcoded Ingress-to-infrastructure mapping baked into the API server couldn't serve all of them.
• The API object stays stable across environments. The same Ingress manifest can move between a local cluster running nginx-ingress and a cloud cluster running a managed controller, changing only which controller is installed, not the application team's YAML.
• It matches Kubernetes' general controller pattern. Objects declare desired state; a controller reconciles that state against reality. Ingress is just this pattern applied to L7 routing, the same separation Kubernetes already uses everywhere else (a Deployment declares desired replica state, the Deployment controller makes it real).

Trade-offs and pitfalls

• Because the controller does the real work, its feature set and its vendor-specific annotations determine what's actually possible; two clusters running different controllers can behave differently from the exact same Ingress YAML, which undermines the portability the split is meant to provide unless you stick to well-supported, portable fields.
• No Ingress Controller installed means Ingress objects sit inert; a common early mistake is writing correct Ingress rules and then wondering why nothing routes, when the real gap is a missing or misconfigured controller.
• Cross-cutting concerns that need to apply before traffic even reaches a specific controller instance (global rate limiting, a shared web application firewall) are often better handled at an edge layer in front of the cluster rather than pushed entirely into per-Ingress annotations.