Kubernetes Networking and Services Questions

Covers the Kubernetes networking model and how services are exposed, discovered, secured and managed within and outside a cluster. Candidates should understand the flat networking model where each pod receives an Internet Protocol address, container network interface plugins and kube proxy forwarding modes. They should know service types including ClusterIP, NodePort, LoadBalancer and ExternalName, the concept of headless services, and service endpoint resolution. Explain Domain Name System based service discovery, CoreDNS roles, ingress controllers and ingress resources for external routing, and load balancer behavior and health checks. Include network policy concepts for controlling pod ingress and egress and service mesh fundamentals such as Istio and Linkerd for advanced routing, observability and mutual Transport Layer Security. Cover service management and operational practices including readiness and liveness probes, session affinity, external traffic policy, scaling strategies and annotations for cloud load balancers. Describe debugging and troubleshooting techniques such as verifying service endpoints, checking Domain Name System resolution, inspecting network policies, capturing packet traces and reviewing metrics, logs and distributed traces. Discuss design trade offs for multi cluster and cross namespace communication, security segmentation, performance and cost implications of different service exposure methods.

MediumTechnical

90 practiced

When and how should you use sessionAffinity on a Kubernetes Service? Explain kube-proxy's implementation of clientIP affinity, its configuration (timeout seconds), and implications for load balancing, scaling, and sticky-session behavior with Ingress/LoadBalancer frontends.

Sample Answer

**When to use sessionAffinity (ClientIP)**- Use when an application requires sticky sessions and cannot implement session state outside the pod (e.g., legacy apps storing session in-memory).- Prefer short-term affinity for session continuity (shopping cart, WebSocket) while moving to session stores (Redis) or stateless design for scale.

**How kube-proxy implements ClientIP affinity**- Enable by setting Service.spec.sessionAffinity = "ClientIP" and optionally Service.spec.sessionAffinityConfig.clientIP.timeoutSeconds.- Default timeoutSeconds is 10800 (3 hours) if not set.- kube-proxy enforces affinity differently by mode: - iptables mode: kube-proxy programs rules so that packets from the same source IP are NATed consistently to the same endpoint for the duration of timeoutSeconds. It uses its internal tracking and conntrack entries to keep that mapping. - ipvs mode: IPVS supports persistent connections/weighting and kube-proxy programs IPVS persistence entries keyed on client IP with the configured timeout.- The timeout defines how long the mapping is kept since the last observed packet from that client IP.

**Implications for load balancing, scaling, and frontends**- Load distribution: ClientIP affinity skews load — heavy clients can overload particular pods; use with care on high-variance traffic.- Scaling: When pods are added/removed, mappings may change; existing clients might be reassigned to different pods after timeout or on pod churn, causing session interruption.- Ingress / LoadBalancer frontends: - Many cloud LBs or Ingress controllers terminate client connections and forward with the LB’s source IP. ClientIP affinity at the kube-proxy level will then see the LB IP, not the original client — breaking stickiness. - Workarounds: preserve source IP (externalTrafficPolicy=Local for NodePort/LoadBalancer), enable proxy-protocol / X-Forwarded-For and use an L7 sticky mechanism (cookies) or implement sticky at the Ingress/controller layer.- Reliability & troubleshooting: affinity relies on source IP visibility — NATs, CGNAT, or many clients behind a proxy will reduce effectiveness; monitor per-pod load and conntrack table size.

**Recommendations**- Prefer stateless services or external session stores for scalable, reliable systems.- Use sessionAffinity.ClientIP only as a stopgap for legacy apps; if stickiness is required at L7, implement cookie-based affinity in the Ingress or app.- When using ClientIP with cloud LBs, ensure source IP is preserved (externalTrafficPolicy=Local) and test behaviour under scaling and failover.

MediumTechnical

68 practiced

Explain externalTrafficPolicy for a Service of type LoadBalancer or NodePort. Compare the behavior of externalTrafficPolicy: Cluster vs Local in terms of client IP preservation, source IPs seen by pods, and health-check interactions with cloud load balancers.

Sample Answer

**Brief answer (what it is)**externalTrafficPolicy is a Service field that controls how traffic from external clients (via LoadBalancer or NodePort) is routed into the cluster. Two values: Cluster and Local — they change client IP preservation, what source IP pods see, and how cloud LBs perform health checks.

**Behavior comparison**

- externalTrafficPolicy: Cluster - Client IP preservation: NOT preserved to pods by default. The packet may be DNATed and/or SNATed when routed across nodes, so the pod often sees the node IP or kube-proxy as source. - Source IPs seen by pods: typically the node’s IP (or NAT address), not the original client IP. - Health checks: cloud load balancer health checks to node ports will be forwarded by kube-proxy to pods even on nodes without endpoints, so the LB will mark all nodes healthy. This can lead to traffic blackholing if a node has no local pods (traffic forwarded over the network).

- externalTrafficPolicy: Local - Client IP preservation: PRESERVED. The original client IP is kept and seen by pods (no SNAT), enabling accurate logging and client-based policies. - Source IPs seen by pods: the real client IP. - Health checks: cloud LBs probe nodes’ NodePort; nodes that have no local endpoints will fail health checks and be removed from the LB pool. Only nodes with local endpoints are considered healthy, avoiding cross-node forwarding.

**Operational implications (DevOps view)**- Use Local when you need real client IPs (e.g., rate-limiting, geo IP, forensics). Expect less even distribution and need sufficient pods on many nodes to maintain capacity; may require DaemonSets or careful scheduling.- Use Cluster for even cluster-wide load distribution when client IPs are not required.- On cloud LBs (AWS ELB/ALB, GCP LB): ensure health-check ports and NodePort mapping are correct. With Local you must ensure enough nodes advertise healthy endpoints to the LB.- If you need client IP and also want to front with a cloud LB that supports PROXY protocol, consider enabling proxy-protocol or using X-Forwarded-For on HTTP LBs where applicable.

**Example**- AWS NLB + Service with externalTrafficPolicy: Local => client IP preserved; only nodes with pod endpoints pass NLB health checks.- Change to Cluster => NLB sees all nodes healthy; pods see node IP as source; logging shows node IPs rather than client IPs.

**Summary**Cluster: better distribution, loses client IP visibility. Local: preserves client IP, requires co-located pods on nodes and affects cloud LB health-check behavior so only nodes with endpoints are used.

HardSystem Design

73 practiced

Design a zero-downtime blue-green deployment strategy for a stateful service running on Kubernetes that requires leader election. Explain how you would manage service discovery, headless services, DNS caching, leader election handoff, and data migration to ensure no split-brain and minimal client impact.

Sample Answer

**Clarify goals & constraints**- Zero-downtime for clients- Stateful service with single active leader (leader election)- No split-brain- Kubernetes target, clients may cache DNS

**High-level approach**1. Run blue and green as separate StatefulSets (stateful workloads, independent PVCs or replicated storage).2. Use a controlled traffic cutover: provision green, sync data, promote green leader, switch traffic, drain blue.

**Service discovery and headless services**- Provide two services: - A headless Service (clusterIP: None) per color to allow pod-level DNS / EndpointSlices for internal discovery and diagnostics. - A stable ClusterIP Service (virtual service) that clients use. This ClusterIP is the single DNS name clients call; it points to the currently active color via label selector change or via ServiceMesh/Ingress traffic routing.- Clients always use the stable ClusterIP name; headless service used for health checks/replication internals.

**DNS caching**- Set low TTL for external DNS (if used) and rely on ClusterIP for in-cluster clients (no external DNS churn).- For external clients, set small TTL and implement client retry/backoff and short connection timeouts. Document that clients respect TTL; otherwise use a LoadBalancer/Ingress that can switch routing without relying on DNS.

**Leader election handoff**- Use Kubernetes Lease API for leader election (coordination.k8s.io/Lease). Implement graceful demotion: - New green pods start with leader-election disabled until data sync completes. - After sync, trigger leader handoff by: - Calling current leader's API to quiesce (stop accepting new writes), wait for in-flight writes to finish. - Current leader releases Lease (or sets a “handing-over” flag). - Green candidate attempts to acquire Lease; only after Lease acquired does it start accepting writes. - Implement preStop hook in pods to demote leader on termination and block SIGTERM until demotion completes.- Enforce fencing: leader only accepts writes when it holds Lease; other nodes reject writes even if they see data.

**Data migration / replication**- Prefer continuous replication (replica follower model) rather than full copy: - Stand up green replicas as read followers against blue leader (DB-level replication or app-level change streams). - Verify replication lag metrics are within threshold. - If using PVCs, use snapshot/clone: create volume snapshot of blue leader and clone for green pods; validate consistency.- Only cutover when green state is fully synchronized and green leader has lease.

**Traffic switch**- When green leader is ready and holding Lease: - Update the stable ClusterIP Service selector to include green pods and remove blue (atomic update). - Alternatively, update ServiceMesh routing (Envoy/Ingress) to shift traffic gradually (canary percentages).- Health checks: readinessProbe only marks pods ready when they are synced and hold (or can accept) leader/non-leader roles appropriately.

**Avoiding split-brain**- Single source of truth for leadership: Kubernetes Lease + app-level verification (leader token/fencing).- Demote on SIGTERM and only release Lease after demotion completes.- Ensure clients verify leader identity (e.g., respond to /whoami) and retry on mismatch.

**Operational controls & metrics**- Automation in CI/CD pipeline: orchestrate sync, promote, switch service, rollback.- Monitor Lease status, replication lag, readiness, error rates.- Maintain PodDisruptionBudget to prevent concurrency issues.

**Failure & rollback**- If green fails during cutover, revert ClusterIP selector back to blue; blue leader resumes (Lease reacquisition enforced).- Keep immutable snapshots and logs for safe rollback.

This plan minimizes client-visible disruptions by keeping a stable Service name, performing controlled leader handoff with fencing, synchronizing data ahead of cutover, and using Kubernetes primitives (Leases, readiness, preStop) plus service routing updates for an atomic traffic switch.

HardSystem Design

92 practiced

Design a multi-cluster service discovery and routing strategy between an on-premises cluster and a cloud cluster that must expose a microservice across both for regional failover. Discuss DNS federation, service mesh federation, latency, security, and failover requirements and trade-offs.

Sample Answer

**Clarify requirements & constraints**- Expose a microservice across on-prem and cloud for regional failover, low latency for local region, secure traffic, automated failover, health-aware routing, multi-cluster observability. RTO < 60s, RPO minimal.

**High-level architecture**- Two Kubernetes clusters (on‑prem, cloud) running same microservice image/version.- Control plane: CI/CD pushes images and manifests to both clusters; GitOps (ArgoCD) for sync.- Data: use regionally replicated DB or active-passive replication with failover orchestration.

**Service discovery & routing options**- DNS federation (global): - Use a low TTL global DNS (e.g., Route 53/Cloud DNS) with health checks. Primary strategy: weighted / latency-based records pointing to region ingress. - Pros: simple, widely available. Cons: DNS caching; propagation/TTL limits fine-grained failover.- Service mesh federation: - Deploy Istio/Linkerd in both clusters and use mesh federation (e.g., Istio multicluster, Service Mesh Hub) to share service registry and mTLS identities. - Mesh provides fine-grained traffic shifting, cross-cluster service discovery, observability, and secure east-west traffic. - Combine with a global Gateway in cloud plus local ingress for optimal egress.

**Latency & traffic shaping**- Use latency-based DNS routing for read-heavy, geo-sensitive traffic.- Prefer local ingress with mesh federation for cross-cluster calls; configure circuit breakers, retries, and timeout policies to avoid cascading failures.- For heavy stateful traffic, route to nearest read-replica; for writes use leader pinned to cloud or orchestrate leader failover.

**Security**- mTLS between workloads via service mesh; mutual TLS for ingress gateways.- Use mutual authentication at cluster peering (VPN or AWS Transit Gateway) + network policies restrict access.- Secrets via external KMS + sealed-secrets or HashiCorp Vault replicated securely.- RBAC and OIDC for control plane operations.

**Failover behavior**- Active-active for stateless service: mesh distributes; DB replication required. Use health probes + ServiceEntry updates to remove unhealthy endpoints.- Active-passive: DNS weighted 100/0 plus health checks; automated promotion via orchestration when primary fails.

**Trade-offs**- DNS-only: simpler, cheaper, slower failover, coarse control.- Mesh federation: richer control, security, observability, but higher operational complexity and networking/NAT/trust setup.- Latency vs consistency: local reads faster; global consistency requires replication and conflict handling.

**Operational considerations**- CI/CD pipeline automates deployments and can trigger failover playbooks.- Monitoring: global dashboards, distributed tracing, synthetic checks.- Test runbooks with chaos drills and DNS/mesh failover simulations.

This balances DNS for user-facing routing and mesh federation for secure, health-aware cross-cluster discovery—choose based on team ops maturity and RTO/RPO needs.

EasyTechnical

78 practiced

What is kube-proxy, what are its forwarding modes (userspace, iptables, ipvs), and how do these modes differ in how they perform service load balancing and packet handling? Explain operational impacts (performance, debugging) of switching modes in a production cluster.

Unlock Full Question Bank

Get access to hundreds of Kubernetes Networking and Services interview questions and detailed answers.

Join thousands of developers preparing for their dream job.