Kubernetes Architecture, Operations, and Troubleshooting Questions
How Kubernetes works, how to run it, and how to debug it. Covers control-plane and node components, the scheduler and API server, cluster design, high availability and multi-cluster topologies, and platform-level operations; the workload primitives (pods, deployments, services, controllers), cluster upgrades, and designing Kubernetes as an internal platform; and the operational depth inside a cluster including pod and service networking, ingress and the CNI model, service mesh, persistent volumes and storage classes, resource requests and limits, and systematically diagnosing scheduling, networking, and storage failures. The full architecture-through-day-two-operations span of Kubernetes.
Explain Kubernetes NetworkPolicy at a high level. What problems does it solve, how are ingress and egress policies expressed, how do selectors and namespaceSelectors work, and what is the default behavior when no policies are defined?
Sample Answer
NetworkPolicy is a namespaced Kubernetes object that restricts which pods can talk to which other pods (and, via egress rules, which external destinations pods can reach), solving east-west traffic control inside the cluster: without it, any pod can reach any other pod by default, which is a large blast radius for a compromised or misconfigured workload.
The enforcement point that isn't the API server
Creating a NetworkPolicy object only stores intent in etcd via the API server, the same as any other Kubernetes object; the API server itself does not inspect or block a single packet. Enforcement is entirely the responsibility of the CNI (container network interface) plugin running on each node, and only if that plugin actually implements NetworkPolicy support. Calico, Cilium, and several others do; a purely overlay-focused CNI like plain Flannel historically does not. Applying a NetworkPolicy on a cluster whose CNI doesn't enforce it is a silent no-op: kubectl get networkpolicy shows the object as created, but traffic flows exactly as if it didn't exist. Confirming the CNI's NetworkPolicy support is a prerequisite, not an afterthought, before relying on this mechanism for anything security-relevant.
Expressing ingress and egress
A policy selects pods with podSelector, and, once selected, ingress and egress traffic is governed independently:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-policy
spec:
podSelector:
matchLabels: { app: web }
policyTypes: [Ingress, Egress]
ingress:
- from:
- podSelector: { matchLabels: { role: frontend } }
ports:
- port: 80
egress:
- to:
- namespaceSelector: { matchLabels: { env: services } }
ports:
- port: 5432
This says: pods labeled app: web accept ingress only from pods labeled role: frontend on port 80, and may only send egress to pods in namespaces labeled env: services on port 5432. Everything else in that direction is implicitly denied once any rule for that direction exists.
Selectors and namespaceSelector
podSelectorinsidefrom/tomatches pods by label within the same namespace as the rule's target namespace context.namespaceSelectormatches whole namespaces by label; commonly combined withpodSelectorin the samefrom/toentry to mean "these specific pods, but only in namespaces with this label," rather than either alone.- An empty
podSelector: {}at the top level of the spec selects every pod in the namespace; an empty selector inside afrom/toentry means "all pods" for that source/destination.
Default behavior
With no NetworkPolicy selecting a pod at all, every ingress and egress connection to and from it is allowed, no restriction whatsoever. The moment at least one policy selects a pod and declares a policyTypes direction (say, Ingress), that direction flips to default-deny for that pod: only traffic matching an explicit rule is allowed, and everything else in that direction is dropped. This is why a common, deliberate starting point is a namespace-wide default-deny policy (a policy selecting all pods with empty ingress/egress rules) followed by narrow allow rules added on top, rather than trying to enumerate every exception against an open-by-default baseline.
Trade-offs and pitfalls
- NetworkPolicies are additive across multiple policies selecting the same pod (a union of allowed traffic, not an intersection), which means adding a second, broader policy to a namespace can unintentionally widen access that a narrower first policy had restricted.
- Egress policies frequently get overlooked because ingress is the more obvious security concern, but an over-permissive default (no egress policy at all) lets a compromised pod exfiltrate data or reach internal services it has no business reaching; DNS resolution in particular is easy to break by an egress default-deny policy that forgets to allow port 53 to the cluster's DNS service.
- Because enforcement lives in the CNI, NetworkPolicy behavior (which selector combinations are supported, whether egress is supported at all, performance under many rules) varies by CNI choice; a policy that works correctly in one cluster is not guaranteed to behave identically after a CNI migration.
Describe the Kubernetes pod lifecycle and common pod states (Pending, ContainerCreating, Running, Succeeded, Failed, Unknown, CrashLoopBackOff). For each state explain what it implies and list kubectl commands and API resources you would inspect to diagnose a pod that is not in Running state.
Sample Answer
A pod moves through Pending (accepted but not yet running), ContainerCreating (scheduled, kubelet is pulling images and mounting volumes), Running (at least one container is up), and finally either Succeeded or Failed for a pod that's meant to terminate, or a restart loop such as CrashLoopBackOff for one that keeps failing and coming back. Unknown is different in kind from the rest: it means the API server has simply lost contact with the node, not that anything about the pod itself is known to be wrong.
States, what each implies, and where to look
| State | What it implies | First place to look |
|---|---|---|
Pending | Not yet scheduled (capacity, affinity, or taint mismatch), or scheduled but the image can't be pulled yet | kubectl describe pod Events; look for FailedScheduling or ImagePullBackOff |
ContainerCreating | Scheduled; kubelet is pulling the image, attaching volumes, or setting up the pod's network namespace via the CNI (container network interface, the plugin that wires a pod into the cluster network) | kubectl describe pod; a stall here usually points to a slow registry pull or a PersistentVolumeClaim (PVC) that hasn't bound |
Running | At least one container is up (does not by itself mean the app is healthy or serving traffic; that's what readiness probes are for) | kubectl get pod -o yaml for status.conditions; kubectl logs |
Succeeded | All containers exited 0 (normal for a Job, unusual for a long-running Deployment pod) | kubectl logs <pod>; the owning Job's status for completion count |
Failed | A container exited non-zero and the pod's restartPolicy did not restart it | kubectl describe pod; kubectl logs --previous for the last container's output before it died |
CrashLoopBackOff | The container keeps exiting and Kubernetes is backing off between restart attempts with an increasing delay | kubectl logs --previous; kubectl describe pod for the exit code and reason under lastState |
Unknown | The API server can't get a status update from the node's kubelet | kubectl get nodes for NotReady; node/kubelet logs, not the pod itself |
A sample of what the Events section actually looks like for a crash-looping pod:
$ kubectl describe pod worker-6b9f-x2z4p -n prod
...
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Ready: False
Restart Count: 6
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 38s (x5 over 90s) kubelet Back-off restarting failed container
OOMKilled here means the kernel's out-of-memory killer terminated the container because it exceeded its memory limit, not that the application itself crashed; that distinction changes the fix (raise the memory limit or reduce the footprint, versus debug application logic).
Graceful termination, and how it relates to Failed vs a clean stop
When a pod is deleted (including during a rolling update), Kubernetes first removes it from Service endpoints so it stops receiving new traffic, then sends SIGTERM to the container, runs any preStop hook first if one is defined, and waits up to terminationGracePeriodSeconds (30 seconds by default) before sending SIGKILL. A container that ignores SIGTERM and needs the full grace period before being force-killed can look, from the outside, like a slow or stuck termination rather than a clean stop; tuning the grace period and handling SIGTERM in the application are what make a shutdown graceful instead of abrupt.
Trade-offs and pitfalls
Runningis frequently mistaken for "healthy." A container can beRunningwhile its readiness probe fails continuously, meaning it's alive but not receiving traffic, which shows up instatus.conditions(Ready: False), not in the pod phase.FailedversusCrashLoopBackOffis really aboutrestartPolicy: withAlways(the Deployment default) a failing container becomesCrashLoopBackOffbecause Kubernetes keeps retrying with backoff; withNeverorOnFailurein the wrong combination a similar failure surfaces asFailedinstead, which changes which command shows you the useful evidence (--previouslogs only apply once a restart has actually happened).
Explain the differences between a Pod, ReplicaSet, Deployment, StatefulSet, and DaemonSet in Kubernetes. For each resource, describe its primary use cases, how it manages lifecycle and scaling, how updates/rollbacks are handled, and provide concrete examples of when you'd choose each resource (stateless web service, per-node agent, stateful database, etc.).
Sample Answer
Pods are the runtime unit; everything else here is a controller deciding how many pods to run and how to replace them. A ReplicaSet just keeps N identical pods alive. A Deployment wraps a ReplicaSet with declarative rolling updates and rollback history. A StatefulSet keeps stable per-replica identity and storage for workloads that care which replica they are. A DaemonSet guarantees exactly one pod per matching node. Job and CronJob run pods to completion, once or on a schedule, rather than keeping them running indefinitely.
Comparison
| Object | Identity model | Storage | Update mechanics | Reach for it when |
|---|---|---|---|---|
| ReplicaSet | anonymous, interchangeable pods | none built in | no native rolling update | almost never directly; it is what a Deployment manages underneath |
| Deployment | anonymous | typically shared, external, or none | rolling update via surge and unavailable settings, with full revision history and rollback | stateless services: web tiers, API servers |
| StatefulSet | stable ordinal identity (pod-0, pod-1, ...) and hostname | one PersistentVolumeClaim per ordinal via volumeClaimTemplates | ordered rolling update by default, highest ordinal first | stateful systems that need to know which replica they are: databases, brokers |
| DaemonSet | one pod per matching node | node-local, if any | rolling update per node, bounded by maxUnavailable | node-level agents: log collectors, CNI (Container Network Interface) plugins, monitoring exporters |
| Job | runs to completion, tracked by completions, parallelism, backoffLimit | ephemeral | retried on failure up to backoffLimit, not a rolling concept | one-off or batch work |
| CronJob | creates a Job on a schedule | ephemeral | governed by concurrencyPolicy (Allow, Forbid, or Replace) | scheduled batch work: nightly reports, cleanup jobs |
Worked example: rolling-update arithmetic
For a Deployment with replicas: 10, maxSurge: 25%, and maxUnavailable: 25%, Kubernetes rounds surge up and unavailable down:
surge=⌈10×0.25⌉=3
unavailable=⌊10×0.25⌋=2
So at the busiest point of the rollout the Deployment can briefly run up to 13 pods (10 plus 3 surge) while guaranteeing at least 8 stay available (10 minus 2 unavailable). This asymmetric rounding is why setting surge and unavailable to the same percentage still lets the total pod count grow slightly during a rollout instead of staying flat.
DaemonSets interact with node taints in a way that is easy to miss: they commonly carry an explicit toleration for the control-plane taint (conventionally node-role.kubernetes.io/control-plane:NoSchedule) so node-level agents like a CNI plugin or kube-proxy itself still run on control-plane nodes that ordinary application pods are excluded from. That toleration is a manifest choice the DaemonSet author makes, not automatic behavior.
Trade-offs and pitfalls
- Choosing Deployment for a workload that actually needs stable identity (a small quorum service, a primary-replica database) forces the team to rebuild that identity logic in application code; StatefulSet gives it for free, at the cost of slower, ordered rollouts.
- A stuck rollout is usually a readiness-probe failure hiding behind the surge and unavailable math: if new pods never pass readiness, the Deployment can stall indefinitely partway through, with old and new pods coexisting. Check pod events and readiness status before assuming a controller bug.
- A PodDisruptionBudget set too strictly (for example,
minAvailableequal to the full replica count) can make a Deployment's own rolling update unable to proceed, because the update's own temporary unavailability trips the budget meant to protect against unrelated disruptions. - DaemonSets bypass normal replica-count thinking: adding nodes silently adds pods and cost. Forgetting a DaemonSet exists across a large node pool is a common source of untracked resource consumption.
Describe what a Pod is in Kubernetes and why it is considered the smallest deployable unit. Explain when you would run multiple containers in a single pod, how containers inside a pod share network and volumes, and trade-offs of co-locating containers such as sidecar patterns versus separate pods.
Sample Answer
A pod is the smallest unit Kubernetes schedules and manages: one or more containers that always run together on the same node, sharing a network namespace and, optionally, storage volumes. It is the unit, rather than the individual container, because the kubelet, the scheduler, and every controller reason about placement, restarts, and networking at that granularity, not below it.
Why the pod, not the container, is the atomic unit
- Co-scheduling guarantee: every container in a pod is placed on the same node and started, stopped, and restarted as a unit by the kubelet.
- One IP per pod: regardless of how many containers it holds, a pod gets exactly one IP address; Services and DNS resolve to that pod IP, not to an individual container.
- Shared network namespace: containers in a pod talk to each other over
localhost, and must not claim conflicting ports, because they share the same network namespace. - Shared volumes: volumes are declared once at the pod spec level and can be mounted into more than one container, which is how a sidecar can read or write files the main container produces without a network hop.
When to run more than one container in a pod
- Sidecar pattern: a helper process that shares the fate and resources of the main container, most commonly a log shipper, a proxy, or a metrics exporter.
- Init containers: containers that run to completion, in order, before the main containers start, typically for one-time setup like a schema migration or config templating.
- Native sidecar containers: since the
SidecarContainersfeature became enabled by default in Kubernetes 1.29 (stable as of 1.33), you can declare a sidecar underinitContainerswithrestartPolicy: Always. Kubernetes then starts it before the main container, keeps it running for the pod's whole life, and stops it after the main container on shutdown, which fixes the older ordering problem where a plain extra container might not be ready before the app started, or might outlive a completed Job's main container instead of shutting down with it.
A small example, an app container and a log-forwarding sidecar sharing a volume instead of a network call:
containers:
- name: app
image: myapp:1.4
volumeMounts:
- name: logs
mountPath: /var/log/app
- name: log-shipper
image: fluent-bit:latest
volumeMounts:
- name: logs
mountPath: /var/log/app
readOnly: true
volumes:
- name: logs
emptyDir: {}
The app writes to /var/log/app; the sidecar tails the same path through the shared emptyDir volume, no network hop involved.
Worked example: what a failing sidecar looks like
If the log-shipper container above enters a crash loop while app keeps running fine, the two containers' fates are independent even though they share a pod: app keeps serving traffic, but the log pipeline drops. This is visible directly in the pod list, where the READY column reports containers-ready over containers-total, not a single number:
NAME READY STATUS RESTARTS AGE
myapp-6d947f8db8-x2z1p 1/2 Running 4 (30s ago) 6m
Here 1/2 means only one of the pod's two containers is passing its readiness state, and the restart count of 4 belongs to the sidecar, not to app.
Trade-offs and pitfalls
- Co-location couples lifecycle: a sidecar cannot be scaled independently of the app the way a separate Deployment could be. Native sidecar containers loosen this slightly by giving the sidecar its own restart behavior, but it is still tied to the pod's schedule and node.
- Every container in the pod counts toward the pod's total resource footprint; forgetting the sidecar's own requests and limits when sizing the node is a common under-provisioning mistake.
- Splitting a component into a separate pod (plus a Service) regains independent scaling and blast-radius isolation, at the cost of a network hop and losing the localhost and shared-volume convenience. Reach for separate pods when the components genuinely differ in scaling or failure characteristics, for example a stateless API versus a shared cache, rather than defaulting to a sidecar for anything that happens to run alongside the main app.
Describe Kubernetes Service types (ClusterIP, NodePort, LoadBalancer, ExternalName). Explain how each type routes traffic to pods, typical cloud provider integrations, operational limitations, and how kube-proxy implementation affects packet forwarding behavior.
Sample Answer
A Kubernetes Service is a stable virtual address in front of a changing set of pods. The four types differ in who can reach that address and how: ClusterIP is internal-only, NodePort adds a port opened on every node, LoadBalancer asks the cloud provider to provision an external load balancer in front of that, and ExternalName is different in kind: a DNS alias to something outside the cluster with no pod routing involved at all.
Service types
| Type | Reachable from | How traffic reaches pods | Typical cloud integration |
|---|---|---|---|
| ClusterIP | inside the cluster only | kube-proxy rewrites the Service's virtual IP to a pod IP | none needed |
| NodePort | any node's IP, on a fixed port in the 30000-32767 range | same in-cluster rewrite as ClusterIP, reached via the node's IP first | often sits behind a hand-rolled external load balancer |
| LoadBalancer | the internet, or a private network, via a provisioned load balancer's address | the provider's load balancer forwards to node ports, which kube-proxy then routes to pods | automatic provisioning on AWS, GCP, or Azure through the cloud-controller-manager |
| ExternalName | resolves to an external DNS name; no cluster-internal routing at all | none; CoreDNS returns a CNAME and the client connects directly | giving a managed external service (e.g., a managed database) a cluster-local name |
A fifth variant sits alongside these four rather than replacing any of them: a headless Service (clusterIP: None). It skips kube-proxy's virtual-IP rewriting entirely, so instead of one stable IP load-balancing across pods, CoreDNS returns the individual pod IPs behind the Service directly to the client (sourced from the same EndpointSlices kube-proxy would otherwise consume). This is why StatefulSets pair with a headless Service: each replica needs its own stable, individually addressable DNS name rather than one shared address in front of all of them.
kube-proxy: the mechanism underneath
kube-proxy is what actually implements the ClusterIP, NodePort, and LoadBalancer rewriting, and it has more than one implementation:
- iptables mode (the long-standing default): programs Linux netfilter rules to destination-NAT (DNAT, rewriting a packet's destination address) Service traffic to a pod IP. Simple, but rule evaluation is roughly linear in Service count, which shows up as latency at high Service counts.
- ipvs mode (IP Virtual Server): uses the kernel's IPVS load-balancing tables instead of a rule chain, giving better performance and more scheduling algorithm choices at scale.
- nftables mode: the newer replacement backend, built on the modern Linux nftables framework, which reached general availability in Kubernetes 1.33. It targets the same correctness as iptables mode with materially better performance at high Service counts, though iptables remains the cluster-wide default for now.
Older material sometimes still mentions a userspace mode; it was deprecated years ago and fully removed in Kubernetes 1.26, so it should not appear in any current design.
Worked example: tracing one request through a LoadBalancer Service
flowchart LR
Client --> ELB[Cloud load balancer]
ELB --> NP[Node: NodePort]
NP --> KP[kube-proxy DNAT rule]
KP --> PodA[Pod A]
KP --> PodB[Pod B]
The client hits the cloud load balancer's public address, which forwards to a NodePort on whichever node it picked, where kube-proxy's DNAT rule sends the packet to one of the pods listed in the Service's EndpointSlices. With externalTrafficPolicy left at its default Cluster, the node that first receives the packet can forward it on to a pod on a different node, an extra hop, but with even load spread across all pods. Setting it to Local skips that extra hop and preserves the client's real source IP, at the cost of only routing to pods that happen to already be on the node the load balancer chose, so an uneven pod distribution across nodes turns directly into uneven traffic.
Trade-offs and pitfalls
- ExternalName is often reached for as 'the DNS one' when someone actually wants a placeholder Service for pods that do not exist yet. Because it has no selector and no health checking, nothing about it verifies the external target is even up.
- NodePort's fixed 30000-32767 range and one-port-per-Service-per-node model does not scale past a modest number of exposed Services and puts every node's IP in the attack surface; it is a building block for an external load balancer, rarely the final production answer on its own.
- LoadBalancer provisions one cloud load balancer per Service by default, which gets expensive and slow to provision as the number of Services grows; that pressure is usually solved by putting an Ingress controller behind a single LoadBalancer Service, a platform decision distinct from Service type selection itself.
- kube-proxy's mode is a cluster-wide setting, not a per-Service choice; moving from iptables to ipvs or nftables needs a validated cluster-wide rollout plan, not a Service-by-service change.
Unlock Full Question Bank
Get access to all 13 Kubernetes Architecture, Operations, and Troubleshooting interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.