Kubernetes Architecture, Operations, and Troubleshooting Questions

Kubernetes controllers that must have exactly one active instance, the scheduler, the controller-manager, and any custom controller or operator running multiple replicas for availability, coordinate through a Lease object in the coordination.k8s.io API rather than some separate consensus protocol of their own. Lease-based leader election has been the default resource lock since Kubernetes 1.20 (older ConfigMap- and Endpoints-based locks are deprecated); whichever replica successfully holds and keeps renewing a Lease is the leader, and every other replica watches that same Lease to know when to attempt a takeover.

The mechanics

• Acquire. A candidate that finds no current Lease, or one whose renewTime is already stale, writes itself in as holderIdentity with a fresh renewTime and a leaseDurationSeconds.
• Renew. The leader periodically updates renewTime well before leaseDurationSeconds elapses since the last successful renewal. The client-go leader-election defaults used by most controllers are a lease duration of 15 seconds, a renew deadline of 10 seconds (the leader must succeed a renewal within this window or it steps down voluntarily), and a retry period of 2 seconds (how often a non-leader checks whether it can acquire).
• Failover. If the leader fails to renew within leaseDurationSeconds of its last successful renewal, any other candidate observing that staleness is free to acquire the Lease and becomes the new leader.

Failure modes and mitigations

Lease TTL (time-to-live, the window after which a lease is considered stale) tuned too aggressively. A very short lease duration fails over faster but is also more likely to flap on a transient blip in API server latency; a longer duration is more stable but slower to fail over. This is a direct trade-off, not a bug to eliminate: pick the duration based on how expensive a few extra seconds without a leader actually is for that specific controller.

Clock drift between candidates. Each candidate compares its own local clock against the renewTime timestamp, which was written using the previous leader's clock, not a server-side timestamp from the API server. If a rival candidate's clock runs meaningfully ahead of the leader's, the rival perceives the lease as staler than it truly is and may attempt to acquire it before the leader's own renewal deadline has actually passed from the leader's perspective, risking a period where two replicas each believe they are leader. NTP (Network Time Protocol), through a client such as chrony, keeping all control-plane nodes tightly synchronized is the standard mitigation.

API server unavailability. If the leader cannot reach the API server to renew, it cannot tell whether it is still leader or has already failed over; well-behaved controllers treat a failed renewal as "step down," stopping leader-only work defensively rather than assuming they are still leader, which avoids a leader continuing to act after a rival has already taken over.

Network partition. Because the Lease itself lives in etcd behind a single API server, or a set of API servers backed by one etcd quorum, a genuine split-brain, two replicas both correctly holding a valid, currently renewed Lease at the same time, requires a partition inside etcd's own quorum, which etcd's own consensus protocol is designed to prevent by only allowing the quorum-holding side to make progress. A leader on the minority side of a network partition simply loses the ability to renew, falling into the API-server-unavailable case above, rather than continuing to hold a stale lease it can prove is still valid.

Worked example: how much clock skew is safe

With the client-go defaults above, the safety margin between when the leader must renew and when a rival is allowed to treat the lease as expired is:

$\text{margin} = \text{leaseDuration} - \text{renewDeadline} = 15\text{s} - 10\text{s} = 5\text{s}$

This margin exists to absorb clock skew between candidates: if a rival's clock reads more than 5 seconds ahead of the leader's, the rival could judge the lease stale and attempt a takeover before the leader's own 15-second window has genuinely elapsed by the leader's clock. Operationally, clock skew across every node running this controller needs to stay comfortably below that 5-second margin; a common operational target is well under one second, since ordinary NTP-synchronized nodes typically hold skew in the tens-of-milliseconds range, leaving most of the 5 seconds as headroom for the transient API server latency the renewal itself has to tolerate.

Trade-offs and pitfalls

• Shortening the lease duration to fail over faster directly trades away flapping resistance; a controller whose leader-only work is cheap to briefly duplicate (two replicas both running a reconcile loop for a few seconds) can tolerate a shorter duration better than one where duplicate execution causes real damage, such as both replicas issuing conflicting cloud-API calls.
• Custom controllers built with controller-runtime get this leader-election behavior largely for free, the same coordination.k8s.io Lease mechanism and the same client-go defaults, but a controller author still has to make its reconcile loop safe to briefly run twice during a failover window; leader election reduces the frequency of concurrent execution, it does not make concurrent execution impossible.
• Monitoring only whether a leader is elected misses flapping; watch the rate of leader transitions over time. A leader-election status metric changing repeatedly in a short window is the actual signal that something, usually clock skew or API server latency, is wrong.

sequenceDiagram
    participant L as Leader (replica A)
    participant R as Rival (replica B)
    participant API as API server / Lease
    L->>API: renew Lease (renewTime = T0)
    Note over L,API: renews every < renewDeadline (10s)
    R->>API: watch Lease
    L--xAPI: renewal fails (API server unreachable)
    Note over L: leader steps down (defensive)
    R->>API: check local_now - renewTime > leaseDuration (15s)
    R->>API: acquire Lease (holderIdentity = B)
    Note over R: becomes new leader

A single well-governed multi-tenant cluster is usually the right starting point for internal teams: namespaces provide the boundary, RBAC (Role-Based Access Control) scopes who can act inside it, ResourceQuota and LimitRange keep one team's usage from starving another, and NetworkPolicy restricts which pods can talk to which. Move a team to its own dedicated cluster only when a specific pressure, compliance, blast radius, or a competing upgrade cadence, makes shared governance the wrong trade, not as a default posture.

What a namespace actually scopes

This decides where every other control applies. A namespace is a logical partition inside one cluster used to scope names and access, not a security boundary by itself: pods in different namespaces still share the same kernel, kubelet, and node pool unless further isolation is added.

Getting this table wrong is a common onboarding mistake: teams write a NetworkPolicy assuming it also restricts node-level traffic, or expect a namespaced RBAC Role to grant node access, when Node is cluster-scoped and untouched by namespace-level RBAC.

Building the tenant boundary

1. Namespaces: one namespace per team plus separate namespaces for shared platform services (ingress controller, logging agents, CI/CD runners). Avoid namespace-per-project-per-team sprawl unless its lifecycle is also automated; an unmanaged namespace is an onboarding shortcut that becomes an offboarding liability.
2. RBAC: define a small set of reusable Role templates (namespace-admin, developer, read-only) and bind them via RoleBinding to groups from an identity provider (an external system, typically integrated via OpenID Connect, OIDC, that authenticates users and asserts group membership) rather than to individual users, so team-membership changes never require touching Kubernetes RBAC directly. Reserve ClusterRole grants for the platform team.
3. ResourceQuota and LimitRange: cap each namespace's aggregate CPU/memory (ResourceQuota) and set sane per-pod defaults and maximums (LimitRange) so one team cannot silently consume the whole node pool. The exact admission-time interaction between the two is its own deep mechanism; the summary a platform designer needs here is that LimitRange fills in and bounds individual pods, while ResourceQuota bounds the namespace's total.
4. NetworkPolicy: default-deny at the namespace boundary, then explicitly allow the flows a team actually needs (its own pods, plus named shared services like an internal registry or logging endpoint). This requires a CNI (Container Network Interface, the plugin layer responsible for pod networking) that enforces NetworkPolicy; Calico and Cilium both do, but not every CNI plugin does.
5. Cost allocation: label every workload with its owning team, scrape actual usage with kube-state-metrics into Prometheus, and attribute cost with a tool built for it (Kubecost or OpenCost are common choices) rather than hand-rolling chargeback from raw node-hour billing.

Worked example: proportional chargeback

Suppose a cluster's monthly infrastructure bill is $12,000, and three teams' namespaces show the following measured CPU-hour consumption over the month (measured usage, not their static quota, since actual usage is what should drive cost):

$\text{Payments' bill} = \$12{,}000 \times 0.50 = \$6{,}000$

$\text{Search's bill} = \$12{,}000 \times 0.25 = \$3{,}000$

$\text{Internal tools' bill} = \$12{,}000 \times 0.25 = \$3{,}000$

This proportional-usage model scales the same way whether the cluster has 3 tenants or 1,000: the mechanism, label-based usage scraped continuously and aggregated, does not change; only the reporting layer has to move from a handful of dashboards a human reads to an automated nightly rollup a finance system consumes, since nobody reviews a thousand individual namespace dashboards by hand.

Single-cluster vs. per-team clusters

The same axis reappears one level down inside a single cluster: many teams that keep dev, staging, and production as one shared multi-tenant cluster for environments still break production out into its own cluster regardless of how they handle team tenancy, because a shared production control-plane incident is categorically worse than a shared-namespace tenant issue in a lower environment.

Trade-offs and pitfalls

• Quotas without LimitRange defaults mean a team that forgets to set requests/limits on a Deployment can be rejected outright at pod creation rather than merely scheduled sub-optimally; this surprises teams the first time it happens.
• NetworkPolicy default-deny with no shared-services allowance breaks logging and monitoring silently: pods that used to reach a cluster-wide logging endpoint stop being able to, and the failure shows up as missing logs rather than an error in the app itself.
• "Start single-cluster, split later" only works if the platform team retains an actual migration path (namespace export, workload relabeling); treat that migration tooling as part of the initial design, not a someday problem.

Intermittent drops that correlate with packet size or specific hosts point at something breaking exactly when a packet needs to be fragmented or exceeds a link's MTU (Maximum Transmission Unit, the largest packet size a link will carry without fragmenting it). In an overlay-networked cluster this usually means one of two things: the pod-facing MTU is set too high for what the overlay's encapsulation overhead leaves available, or Path MTU Discovery (PMTUD, the mechanism that lets a sender learn the smallest MTU along a path and shrink its packets accordingly) is broken because something is blocking the ICMP message it depends on.

Diagnostic procedure

1. Reproduce the cutoff with ping and the do-not-fragment bit. Sending progressively smaller packets with the DF (do-not-fragment) flag set finds the exact size where things start working:

kubectl exec -it pod-a -- ping -M do -s 1472 pod-b-ip

The -s 1472 size is not arbitrary: a standard 1500-byte Ethernet MTU minus a 20-byte IPv4 header and an 8-byte ICMP header leaves exactly

$1500 - 20 - 8 = 1472$

bytes of payload, so this command sends the largest packet a normal, un-encapsulated 1500-byte link should be able to carry without fragmenting. If it fails at 1472 but succeeds well below that, the working link is smaller than a plain 1500-byte Ethernet MTU, pointing at overlay encapsulation eating the difference.

2. Check the overlay's actual overhead. VXLAN (Virtual Extensible LAN, the tunneling protocol most CNI, Container Network Interface, overlay networking uses to carry pod traffic across nodes) adds a fixed amount of header on top of every encapsulated packet: an outer Ethernet header, an outer IP header, an outer UDP header, and the VXLAN header itself:

$14\text{ (outer Ethernet)} + 20\text{ (outer IP)} + 8\text{ (outer UDP)} + 8\text{ (VXLAN)} = 50\text{ bytes}$

So on a host with a 1500-byte physical MTU, the pod-facing overlay interface has to advertise an MTU of at most

$1500 - 50 = 1450$

bytes, not 1500, or packets that fill a full 1500-byte pod-side MTU need fragmentation, or are dropped outright, once VXLAN wraps them. Confirm both values directly:

ip link show dev eth0     # physical/host MTU
ip link show dev vxlan0   # overlay interface MTU

If the overlay interface shows 1500 instead of 1450 on a host whose physical link is 1500, that mismatch alone explains large-packet drops without needing to look any further.

3. Capture traffic to see whether PMTUD is even getting a chance to work. When a packet is too large for a link and DF is set, a router along the path is supposed to send back an ICMP "fragmentation needed" message so the sender can shrink future packets:

tcpdump -i any icmp -w icmp.pcap

A capture that shows nothing at all on the affected flow, no ICMP error, no successful smaller retry, while large packets simply vanish, points at that ICMP message being filtered somewhere in the path (a security group, firewall, or a misconfigured NAT device dropping ICMP), which breaks PMTUD even though the MTU mismatch itself is correctly detected by the network. A capture that does show an ICMP type 3 code 4 ("fragmentation needed") message confirms PMTUD is at least attempting to work, and the investigation shifts to why the sender is not honoring it.

4. Confirm the kernel's PMTUD and MSS behavior on affected hosts.

sysctl net.ipv4.ip_no_pmtu_disc      # 0 = PMTUD enabled (expected default)
sysctl net.ipv4.tcp_mtu_probing      # fallback probing if PMTUD's ICMP path is blocked

Fixes, from least to most invasive

1. Align the overlay MTU with reality. Set the CNI's configured MTU to host MTU minus the overlay's actual overhead, 1450 for a 1500-byte host with VXLAN, following the arithmetic above, rather than leaving it at a default that assumes no encapsulation.
2. Clamp TCP's MSS (Maximum Segment Size) at the border, as a safety net. This forces TCP connections to negotiate a segment size that already accounts for the path's real MTU, so they never depend on PMTUD succeeding:

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

3. If ICMP genuinely cannot be allowed through, a security policy that blocks it entirely, enable TCP MTU probing as a fallback so affected connections can still discover a working size without relying on the ICMP message:

sysctl -w net.ipv4.tcp_mtu_probing=1

4. Longer-term, use jumbo frames on the physical network if the cloud provider or on-prem network supports a larger MTU (many cloud VPCs support jumbo frames on instances with the right networking mode, though the exact ceiling is provider- and instance-type-specific), which restores headroom for overlay overhead without shrinking the pod-facing MTU at all.

Trade-offs and pitfalls

• MSS clamping is a low-risk, immediate fix but only helps TCP; UDP-based traffic, including some service mesh data planes and certain database protocols, gets no benefit from it and still depends on a correct MTU or working PMTUD.
• Raising the physical network to jumbo frames without also raising the overlay's per-packet overhead accounting is a common half-fix: the arithmetic above still needs to subtract the overlay overhead from whatever the new physical MTU is; it does not make the overhead disappear.
• Changing a live cluster's CNI MTU setting typically requires restarting the CNI's node agent, and sometimes recreating existing pods, to pick up the new value, so treat it as a change with a blast radius, not a live config tweak.

flowchart TD
    A[Large-packet drops, correlated with size/host] --> B[Ping with DF bit, decreasing size]
    B -->|fails below 1500| C[Check host MTU vs overlay interface MTU]
    C -->|overlay MTU too high for overhead| D[Fix: lower CNI overlay MTU]
    B --> E[tcpdump for ICMP frag-needed]
    E -->|ICMP present, not honored| F[Enable tcp_mtu_probing]
    E -->|ICMP missing entirely| G[ICMP blocked in path: fix firewall, or clamp MSS]

An Ingress object declares what HTTP(S) routing should happen (hostnames, path rules, which Service backs each path, which TLS secret to use); an Ingress Controller is the running component that watches Ingress objects and actually implements that routing on real infrastructure. Kubernetes splits these into two objects for the same reason it splits a Deployment's spec from the controller that reconciles it: the API object stays a stable, portable declaration, while the implementation (which varies enormously between environments) is free to be swapped out without changing what you wrote.

What the Ingress object itself declares

An Ingress is pure declaration, no execution: a list of host/path rules, each pointing at a backend Service and port, plus optional TLS configuration referencing a Secret containing a certificate and key. It has no opinion about how that routing gets enforced; by itself, an Ingress object sitting in the API server does nothing.

What the Ingress Controller does with it

The controller (for example nginx-ingress, Traefik, or a cloud provider's own controller such as GKE's or AGIC for Azure) watches for Ingress objects and translates the declared rules into a real, running configuration: it opens listener ports, terminates TLS using the referenced Secret, and applies the host/path routing logic, typically by configuring an underlying proxy (like nginx) or provisioning a cloud load balancer.

Why split the two, contrasted with a Service of type LoadBalancer

A Service of type LoadBalancer works at L4 (Layer 4, the TCP/UDP transport layer): it exposes one Service on one IP and port, with no visibility into HTTP hostnames or paths. Ingress works at L7 (Layer 7, the application layer): it can route many hostnames and paths to many different backend Services through a single entry point, but only because something understands HTTP well enough to look inside the request to make that decision, which is exactly the controller's job.

Why split the responsibility this way

• No single "correct" implementation exists. Different environments need genuinely different routing implementations (self-hosted nginx, a cloud provider's managed load balancer, a service mesh's gateway); a single hardcoded Ingress-to-infrastructure mapping baked into the API server couldn't serve all of them.
• The API object stays stable across environments. The same Ingress manifest can move between a local cluster running nginx-ingress and a cloud cluster running a managed controller, changing only which controller is installed, not the application team's YAML.
• It matches Kubernetes' general controller pattern. Objects declare desired state; a controller reconciles that state against reality. Ingress is just this pattern applied to L7 routing, the same separation Kubernetes already uses everywhere else (a Deployment declares desired replica state, the Deployment controller makes it real).

Trade-offs and pitfalls

• Because the controller does the real work, its feature set and its vendor-specific annotations determine what's actually possible; two clusters running different controllers can behave differently from the exact same Ingress YAML, which undermines the portability the split is meant to provide unless you stick to well-supported, portable fields.
• No Ingress Controller installed means Ingress objects sit inert; a common early mistake is writing correct Ingress rules and then wondering why nothing routes, when the real gap is a missing or misconfigured controller.
• Cross-cutting concerns that need to apply before traffic even reaches a specific controller instance (global rate limiting, a shared web application firewall) are often better handled at an edge layer in front of the cluster rather than pushed entirely into per-Ingress annotations.

Direct answer

Every pod gets a routable IP address from the CNI (Container Network Interface) plugin, with no NAT (network address translation) between pods by default. A Service layers a stable virtual IP and DNS name on top of that, and kube-proxy (or an eBPF-based replacement like Cilium; eBPF lets Linux run sandboxed programs inside the kernel to process packets directly) programs the actual rewriting rules that turn a Service IP into a real pod IP. When two pods on different nodes cannot reach each other on a specific port, the fault sits in exactly one of four layers, checked in this order: the pod's own listener, the kube-proxy/Service plumbing, the CNI's cross-node routing, or a host/cloud firewall. Testing each layer directly beats guessing from symptoms.

Structured elaboration

Pod IP model

The CNI plugin assigns each pod a unique IP from the cluster's pod address range and is responsible for making that IP reachable from every node, either by programming host routes directly (an "underlay" approach, for example Calico in BGP mode, Border Gateway Protocol, the routing protocol routers use to exchange reachability information) or by encapsulating pod traffic in an overlay (VXLAN, a tunneling protocol that wraps one Ethernet frame inside a UDP packet, used by Flannel and by Calico's IP-in-IP mode).

Service virtual IPs and kube-proxy modes

A ClusterIP Service is a stable virtual IP that never corresponds to a real listener; kube-proxy rewrites traffic to it into a real pod IP and port.

Cilium and some other CNIs bypass kube-proxy entirely, attaching eBPF programs to kernel networking hooks to do the same rewriting job with lower per-packet overhead and without iptables' rule-count scaling problem.

CNI plugins in practice

• Calico: pure L3 routing via BGP (no overlay, lowest overhead, needs a BGP-capable network) or IP-in-IP/VXLAN encapsulation where BGP peering is not available.
• Flannel: a simple VXLAN overlay by default; fewer features than Calico or Cilium.
• Cilium: eBPF-based; can replace kube-proxy, adds L7-aware NetworkPolicy and traffic observability, at the cost of needing a reasonably current kernel.

flowchart LR
  A[Pod A on Node 1] -->|packet to Service ClusterIP| KP1[kube-proxy or eBPF dataplane, Node 1]
  KP1 -->|rewrite to real Pod IP| CNI1[CNI routing or overlay, Node 1]
  CNI1 -->|cross-node route or tunnel| CNI2[CNI routing or overlay, Node 2]
  CNI2 --> B[Pod B on Node 2]
  FW[Host firewall or cloud security group] -.may block specific ports.-> CNI2

Debugging cross-node connectivity, layer by layer

Reproduce and localize before changing anything:

$ kubectl get pod -o wide -n app
NAME      READY   STATUS    NODE     IP
pod-a     1/1     Running   node-1   10.244.1.7
pod-b     1/1     Running   node-2   10.244.2.9

$ kubectl exec -n app pod-a -- nc -vz 10.244.2.9 8080
nc: connect to 10.244.2.9 port 8080 (tcp) failed: Connection timed out

A timeout, not "connection refused," means the packet is being dropped somewhere in transit rather than rejected by pod-b's listener; that already rules out "the app isn't listening" and points at routing, the CNI, or a firewall.

1. Confirm the listener exists: kubectl exec -n app pod-b -- ss -tlnp should show something bound to 8080; if not, this was never a networking problem.
2. Check node-to-node routing for the pod's address range:

$ ip route get 10.244.2.9
10.244.2.9 via 10.0.1.2 dev eth0 src 10.0.1.5

A missing route, or one pointing at the wrong interface, means the CNI failed to program cross-node routing, usually because its daemonset is crashlooping:

$ kubectl get pods -n kube-system -l k8s-app=calico-node
NAME                READY   STATUS             RESTARTS
calico-node-x7k2p   0/1     CrashLoopBackOff   14

3. If the failure is specifically Service-IP traffic rather than direct pod-IP traffic, check kube-proxy's own health the same way (kubectl get pods -n kube-system -l k8s-app=kube-proxy) and inspect its programmed rules (sudo iptables -t nat -S | grep <service-name> or sudo ipvsadm -Ln, depending on mode).
4. If routing and the CNI both look healthy, check host-level and cloud firewall rules for the specific port. CNI and kube-proxy problems tend to affect all ports uniformly; a single-port failure with everything else in that layer healthy is a strong signal for a firewall rule scoped to that port.
5. Capture packets at the suspect boundary to confirm rather than infer: sudo tcpdump -n -i any host 10.244.2.9 and port 8080 on the destination node shows whether the packet arrives at the host at all, before deciding whether the problem is upstream (routing) or downstream (host firewall, pod network namespace).

Trade-offs and pitfalls

• iptables rule count grows linearly with Services and endpoints and becomes a real CPU cost past a few thousand Services; IPVS or nftables scale better, but changing kube-proxy mode cluster-wide is a disruptive change to plan and test, not to flip live.
• The distinction between "Connection timed out" and "Connection refused" is easy to skip past but changes where to look first: refused means the packet arrived and something rejected it (wrong port, no listener); timed out means the packet was dropped somewhere before that.
• MTU (maximum transmission unit) mismatches between overlay encapsulation and the underlying network are a classic cause of connections that work for small packets and hang for large ones. Confirm with a full-size, non-fragmenting ping rather than assuming a working small ping rules out MTU.