Kubernetes Architecture, Operations, and Troubleshooting Questions

The Horizontal Pod Autoscaler (HPA) never talks to Prometheus, your app, or a cloud queue directly: it only ever queries one of three Kubernetes metrics APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io), so scaling on queue length requires something that exposes the queue's depth through one of those APIs. The standard pipeline is: the application or a sidecar exposes the metric, a metrics system collects it, and a metrics adapter (most commonly prometheus-adapter, or a project like KEDA, Kubernetes Event-Driven Autoscaling, which ships its own adapter and is now the more common current choice specifically for external event sources like queues) is registered with the API server as an aggregated API and translates queries into that metric API's shape.

The three metrics APIs

A message queue's depth (say, a managed queue service or a Kafka consumer-group lag) is not a property of any Kubernetes object, so it belongs under external.metrics.k8s.io and the HPA metric type: External, not Pods or Object.

Required components and how the metric gets exposed

1. Instrumentation: the application (preferred) or a sidecar exporter publishes a metric, e.g. a Prometheus gauge myapp_queue_length{queue="orders"} on /metrics.
2. Prometheus scrapes it on a scrape job.
3. prometheus-adapter is deployed with rules mapping a PromQL query to an external metric name Kubernetes will expose, and it registers an APIService (apiregistration.k8s.io) so kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 returns real data. This registration needs its own RBAC: a ClusterRole granting the HPA controller's service account (system:kube-controller-manager reaching through the aggregation layer) permission to read the external metrics API, plus the adapter's own service account needing permission to read Prometheus.
4. The HPA references the metric by name:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric: {name: queue_length_orders}
        target: {type: AverageValue, averageValue: "100"}

(autoscaling/v2 has been the stable API since Kubernetes 1.23; the older v2beta2 was removed in 1.26, so any manifest or tooling still referencing it is out of date.)

Operational pitfalls

• A silently wrong zero is worse than a visible failure. If the adapter genuinely can't reach its data source, the HPA does not quietly treat that as "no load": it sets the ScalingActive condition to False with reason FailedGetExternalMetric and shows the metric's current value as <unknown> in kubectl describe hpa, holding replica count steady. The real danger is the opposite case: an adapter that, on a query error, returns a literal 0 instead of erroring. That looks healthy to the HPA and drives a real scale-down during an actual outage in the metrics path, which is why adapter error-handling is worth testing explicitly rather than assumed.
• Cardinality. High-cardinality labels on the underlying metric (one series per customer ID, for instance) can make Prometheus memory blow up long before the HPA ever sees a problem; keep the label set the adapter maps from small and stable.
• Flapping. A noisy queue-length signal causes replica oscillation; use behavior.scaleDown.stabilizationWindowSeconds (part of the autoscaling/v2 HPA behavior fields) or pre-aggregate with a Prometheus recording rule rather than reacting to raw noise.
• Latency in the chain. Scrape interval, adapter caching, and the HPA's own sync period all stack up between a real queue-depth change and a scaling action; a 15s scrape interval plus a slow adapter cache can easily add tens of seconds of lag, which matters for a bursty queue.
• Cost. Recomputing an expensive PromQL query on every HPA sync (default every 15 seconds) across many HPAs can meaningfully load a Prometheus instance; pre-aggregate with recording rules for anything non-trivial.

Validate end to end in staging with synthetic queue load before trusting this in production, watching the full chain (producer to queue to exporter to Prometheus to adapter to HPA) rather than any single hop in isolation.

Trade-off note

Hand-rolling prometheus-adapter rules gives full control over the PromQL mapping but is fiddly YAML to maintain; KEDA trades some of that flexibility for purpose-built scalers for dozens of common event sources (queues, streams, schedules) and is usually less operational overhead for exactly this "scale on queue depth" scenario, at the cost of being one more component to run alongside (or instead of) prometheus-adapter.

Direct answer

Every pod gets a routable IP address from the CNI (Container Network Interface) plugin, with no NAT (network address translation) between pods by default. A Service layers a stable virtual IP and DNS name on top of that, and kube-proxy (or an eBPF-based replacement like Cilium; eBPF lets Linux run sandboxed programs inside the kernel to process packets directly) programs the actual rewriting rules that turn a Service IP into a real pod IP. When two pods on different nodes cannot reach each other on a specific port, the fault sits in exactly one of four layers, checked in this order: the pod's own listener, the kube-proxy/Service plumbing, the CNI's cross-node routing, or a host/cloud firewall. Testing each layer directly beats guessing from symptoms.

Structured elaboration

Pod IP model

The CNI plugin assigns each pod a unique IP from the cluster's pod address range and is responsible for making that IP reachable from every node, either by programming host routes directly (an "underlay" approach, for example Calico in BGP mode, Border Gateway Protocol, the routing protocol routers use to exchange reachability information) or by encapsulating pod traffic in an overlay (VXLAN, a tunneling protocol that wraps one Ethernet frame inside a UDP packet, used by Flannel and by Calico's IP-in-IP mode).

Service virtual IPs and kube-proxy modes

A ClusterIP Service is a stable virtual IP that never corresponds to a real listener; kube-proxy rewrites traffic to it into a real pod IP and port.

Cilium and some other CNIs bypass kube-proxy entirely, attaching eBPF programs to kernel networking hooks to do the same rewriting job with lower per-packet overhead and without iptables' rule-count scaling problem.

CNI plugins in practice

• Calico: pure L3 routing via BGP (no overlay, lowest overhead, needs a BGP-capable network) or IP-in-IP/VXLAN encapsulation where BGP peering is not available.
• Flannel: a simple VXLAN overlay by default; fewer features than Calico or Cilium.
• Cilium: eBPF-based; can replace kube-proxy, adds L7-aware NetworkPolicy and traffic observability, at the cost of needing a reasonably current kernel.

flowchart LR
  A[Pod A on Node 1] -->|packet to Service ClusterIP| KP1[kube-proxy or eBPF dataplane, Node 1]
  KP1 -->|rewrite to real Pod IP| CNI1[CNI routing or overlay, Node 1]
  CNI1 -->|cross-node route or tunnel| CNI2[CNI routing or overlay, Node 2]
  CNI2 --> B[Pod B on Node 2]
  FW[Host firewall or cloud security group] -.may block specific ports.-> CNI2

Debugging cross-node connectivity, layer by layer

Reproduce and localize before changing anything:

$ kubectl get pod -o wide -n app
NAME      READY   STATUS    NODE     IP
pod-a     1/1     Running   node-1   10.244.1.7
pod-b     1/1     Running   node-2   10.244.2.9

$ kubectl exec -n app pod-a -- nc -vz 10.244.2.9 8080
nc: connect to 10.244.2.9 port 8080 (tcp) failed: Connection timed out

A timeout, not "connection refused," means the packet is being dropped somewhere in transit rather than rejected by pod-b's listener; that already rules out "the app isn't listening" and points at routing, the CNI, or a firewall.

1. Confirm the listener exists: kubectl exec -n app pod-b -- ss -tlnp should show something bound to 8080; if not, this was never a networking problem.
2. Check node-to-node routing for the pod's address range:

$ ip route get 10.244.2.9
10.244.2.9 via 10.0.1.2 dev eth0 src 10.0.1.5

A missing route, or one pointing at the wrong interface, means the CNI failed to program cross-node routing, usually because its daemonset is crashlooping:

$ kubectl get pods -n kube-system -l k8s-app=calico-node
NAME                READY   STATUS             RESTARTS
calico-node-x7k2p   0/1     CrashLoopBackOff   14

3. If the failure is specifically Service-IP traffic rather than direct pod-IP traffic, check kube-proxy's own health the same way (kubectl get pods -n kube-system -l k8s-app=kube-proxy) and inspect its programmed rules (sudo iptables -t nat -S | grep <service-name> or sudo ipvsadm -Ln, depending on mode).
4. If routing and the CNI both look healthy, check host-level and cloud firewall rules for the specific port. CNI and kube-proxy problems tend to affect all ports uniformly; a single-port failure with everything else in that layer healthy is a strong signal for a firewall rule scoped to that port.
5. Capture packets at the suspect boundary to confirm rather than infer: sudo tcpdump -n -i any host 10.244.2.9 and port 8080 on the destination node shows whether the packet arrives at the host at all, before deciding whether the problem is upstream (routing) or downstream (host firewall, pod network namespace).

Trade-offs and pitfalls

• iptables rule count grows linearly with Services and endpoints and becomes a real CPU cost past a few thousand Services; IPVS or nftables scale better, but changing kube-proxy mode cluster-wide is a disruptive change to plan and test, not to flip live.
• The distinction between "Connection timed out" and "Connection refused" is easy to skip past but changes where to look first: refused means the packet arrived and something rejected it (wrong port, no listener); timed out means the packet was dropped somewhere before that.
• MTU (maximum transmission unit) mismatches between overlay encapsulation and the underlying network are a classic cause of connections that work for small packets and hang for large ones. Confirm with a full-size, non-fragmenting ping rather than assuming a working small ping rules out MTU.

For most teams, a managed control plane (Google Kubernetes Engine/GKE, Amazon Elastic Kubernetes Service/EKS, Azure Kubernetes Service/AKS) is the right default, because the provider absorbs control-plane high availability, etcd operations, and version patching that otherwise consume ongoing senior engineering time. Self-managed clusters on provisioned VMs earn their keep only when a concrete constraint, not a general preference for control, rules the managed option out: a data-residency or regulatory requirement the provider cannot satisfy, hardware or networking the managed offering does not support, or a genuinely large fleet where the provider's per-cluster fee becomes material against staff you would need to hire anyway.

Evaluation matrix

The security row matters more than it looks: managed Kubernetes does not remove the shared-responsibility split so much as move its boundary, the same shape as the shared-responsibility model on any IaaS. Workload-level security (RBAC, NetworkPolicy, secrets, supply-chain scanning) is the team's job either way.

A short decision checklist

Answer these before defaulting to self-managed:

1. Is there a regulatory or data-residency constraint that no available cloud provider's managed offering can satisfy (for example, a requirement that the control plane run inside a specific on-premises facility with no third-party operational access)?
2. Does the workload need node/kernel-level customization (specialized drivers, SR-IOV, custom kernel modules) that managed node pools do not allow?
3. Does the footprint include environments with no managed control-plane offering at all (deep edge sites, air-gapped networks, telecom points of presence)?
4. At current or planned scale, does the provider's control-plane fee plus its operational savings actually exceed what dedicated in-house Kubernetes operations staff would cost?

If none of these are "yes," managed is the safer default; if one or more are "yes," self-managed (or a lightweight distribution suited to the constraint) deserves serious evaluation.

Worked scenarios

• Compliance-driven case: a bank operating under a regulator that requires its control plane to run inside a specific on-premises data center, with no third-party (including the cloud provider's own control-plane operators) having operational access to cluster internals. This disqualifies every managed offering outright, regardless of cost or operational convenience; the decision is made by question 1 above before cost or overhead ever enters the comparison.
• Edge/scale-driven case: a telecom operator running Kubernetes across dozens of far-edge sites where no cloud provider offers a managed control plane at all. Self-managed, or a purpose-built lightweight distribution, is the only option that reaches the hardware; the comparison in the table above is largely academic here because there is no managed alternative on the table.

Trade-offs and pitfalls

• The most common wrong turn is choosing self-managed "for control" without budgeting the standing SRE headcount it requires; control-plane upgrades, etcd operations, and CVE patching do not stop after go-live, they become a permanent line item.
• Assuming managed Kubernetes removes all security burden is the second most common mistake; node OS hardening, RBAC design, NetworkPolicy, and secret management remain the team's responsibility under either model.
• Very large fleets sometimes justify self-managed purely on the provider fee, but that comparison is frequently done without pricing in the incident risk of an in-house etcd outage against a provider's SLA-backed uptime, which understates the true cost of self-managed at scale.
• Starting managed and only moving specific clusters to self-managed once a concrete constraint actually appears is lower risk than starting self-managed everywhere on a hypothetical future need.

Pods are the runtime unit; everything else here is a controller deciding how many pods to run and how to replace them. A ReplicaSet just keeps N identical pods alive. A Deployment wraps a ReplicaSet with declarative rolling updates and rollback history. A StatefulSet keeps stable per-replica identity and storage for workloads that care which replica they are. A DaemonSet guarantees exactly one pod per matching node. Job and CronJob run pods to completion, once or on a schedule, rather than keeping them running indefinitely.

Comparison

Worked example: rolling-update arithmetic

For a Deployment with replicas: 10, maxSurge: 25%, and maxUnavailable: 25%, Kubernetes rounds surge up and unavailable down:
$\text{surge} = \lceil 10 \times 0.25 \rceil = 3$
$\text{unavailable} = \lfloor 10 \times 0.25 \rfloor = 2$
So at the busiest point of the rollout the Deployment can briefly run up to 13 pods (10 plus 3 surge) while guaranteeing at least 8 stay available (10 minus 2 unavailable). This asymmetric rounding is why setting surge and unavailable to the same percentage still lets the total pod count grow slightly during a rollout instead of staying flat.

DaemonSets interact with node taints in a way that is easy to miss: they commonly carry an explicit toleration for the control-plane taint (conventionally node-role.kubernetes.io/control-plane:NoSchedule) so node-level agents like a CNI plugin or kube-proxy itself still run on control-plane nodes that ordinary application pods are excluded from. That toleration is a manifest choice the DaemonSet author makes, not automatic behavior.

Trade-offs and pitfalls

• Choosing Deployment for a workload that actually needs stable identity (a small quorum service, a primary-replica database) forces the team to rebuild that identity logic in application code; StatefulSet gives it for free, at the cost of slower, ordered rollouts.
• A stuck rollout is usually a readiness-probe failure hiding behind the surge and unavailable math: if new pods never pass readiness, the Deployment can stall indefinitely partway through, with old and new pods coexisting. Check pod events and readiness status before assuming a controller bug.
• A PodDisruptionBudget set too strictly (for example, minAvailable equal to the full replica count) can make a Deployment's own rolling update unable to proceed, because the update's own temporary unavailability trips the budget meant to protect against unrelated disruptions.
• DaemonSets bypass normal replica-count thinking: adding nodes silently adds pods and cost. Forgetting a DaemonSet exists across a large node pool is a common source of untracked resource consumption.

Liveness, readiness, and startup probes all ask whether a container is okay, but each answer drives a different Kubernetes action: a failing liveness probe gets the container restarted, a failing readiness probe gets the pod pulled out of Service traffic without touching the container at all, and a startup probe simply delays the other two until the app has had time to boot.

What each probe gates

Worked example: sizing a startup budget by workload archetype

What 'booting' means differs a lot by workload, and the startup probe has to be sized for the actual archetype, not guessed at: a machine learning (ML) inference service loading model weights into memory might need several minutes; a batch worker doing asynchronous Java Virtual Machine (JVM) warmup, classloading, and connection-pool initialization for an extract-transform-load (ETL) job might need under a minute; a stateless HTTP handler might be ready in under a second. Whichever number applies, it has to be encoded as periodSeconds times failureThreshold. Budgeting 5 minutes of startup headroom with a 10-second check interval for the ML case:
$10 \times 30 = 300\text{s} = 5\text{ min}$
means periodSeconds: 10 and failureThreshold: 30. Too tight in this calculation and the startup probe itself kills a healthy-but-slow container before it ever gets a chance to serve; too loose, and a genuinely stuck container burns minutes before anything reacts.

Trade-offs and pitfalls

• Swapping liveness and readiness is the classic mistake: pointing liveness at a deep dependency check (database reachability) means a transient database blip restarts every application pod at once instead of simply pulling them from rotation, turning a recoverable dependency issue into a self-inflicted outage.
• Using liveness as a substitute for a startup probe on a slow-booting app causes a restart loop before the app ever finishes initializing, since the container never survives long enough to pass a liveness check tuned for steady-state behavior.
• A readiness probe that is too permissive, common with the JVM-async pattern where the process starts accepting connections before its dependency pools are actually warm, reports the pod as ready while real requests still fail; that failure mode never shows up as a probe failure at all, only as user-visible errors.