Kubernetes Architecture, Operations, and Troubleshooting Questions
How Kubernetes works, how to run it, and how to debug it. Covers control-plane and node components, the scheduler and API server, cluster design, high availability and multi-cluster topologies, and platform-level operations; the workload primitives (pods, deployments, services, controllers), cluster upgrades, and designing Kubernetes as an internal platform; and the operational depth inside a cluster including pod and service networking, ingress and the CNI model, service mesh, persistent volumes and storage classes, resource requests and limits, and systematically diagnosing scheduling, networking, and storage failures. The full architecture-through-day-two-operations span of Kubernetes.
You are troubleshooting PersistentVolumeClaim (PVC) not binding to any PersistentVolume (PV). Describe the steps to debug PV/PVC binding: inspect storage class, volumeMode, accessModes, reclaimPolicy, CSI provisioner logs, and cloud provider storage quotas. Provide kubectl commands and sample outputs you would expect in common failure cases.
Sample Answer
A PersistentVolumeClaim (PVC, a request for storage made by a workload) that never binds means either no matching PersistentVolume (PV, the actual piece of storage) exists yet, the StorageClass's provisioner is failing to create one, or a cloud-side constraint (quota, permissions) is blocking provisioning; the fix is to work through those three in order rather than guess.
1. Inspect the PVC itself
kubectl get pvc -n myns my-pvc -o yaml
Read status.phase, spec.storageClassName, spec.volumeMode (Filesystem or Block), spec.accessModes (ReadWriteOnce, ReadWriteMany, etc.), and spec.resources.requests.storage. A very common, entirely benign-looking state:
status:
phase: Pending
Events:
Type Reason Message
---- ------ -------
Normal WaitForFirstConsumer waiting for first consumer to be created before binding
That event is not a failure by itself: with a StorageClass whose volumeBindingMode is WaitForFirstConsumer (the current default recommendation, since it lets the scheduler pick a zone/node before the volume is provisioned there), the PVC is expected to sit Pending until a pod that uses it is scheduled. If the pod itself is also Pending (check kubectl describe pod), the PVC will never move until the pod's own scheduling problem (node selector, taint, resource fit) is resolved first.
2. Check the StorageClass
kubectl get sc <name> -o yaml
Confirm the provisioner field names a real, currently-running Container Storage Interface (CSI, the standard plugin interface storage vendors implement) driver, and note volumeBindingMode and reclaimPolicy. A StorageClass pointing at a provisioner name with no matching CSI driver deployed in the cluster will never provision anything, and the PVC's own events are usually the first place that shows up (a "no volume plugin matched" or "provisioner not found" message).
3. List existing PVs for a manual-binding mismatch
kubectl get pv -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,CAP:.spec.capacity.storage,MODE:.spec.volumeMode,ACCESS:.spec.accessModes,CLAIM:.spec.claimRef.name
If you're using statically pre-created PVs rather than dynamic provisioning, the common mismatches are: no PV with the same storageClassName, a PV with less capacity than requested, a volumeMode mismatch (PV is Block, PVC wants Filesystem), or a PV whose claimRef already points at a different claim.
4. Read the CSI provisioner's own logs
kubectl -n kube-system logs deploy/<csi-driver-controller-deployment> -c <provisioner-container>
(the exact deployment and container name depend on which CSI driver is installed, e.g. an EBS, Persistent Disk, or Azure Disk CSI controller; find it with kubectl get pods -n kube-system | grep csi). This is where the real, specific failure usually surfaces even when the PVC's own events are vague: a resource-exhausted error from the cloud API, an IAM/permission denial on the provisioner's own service account, or a zone mismatch between the requested volume and where the pod was scheduled.
5. Cloud-side quota and permissions
If the provisioner log shows a quota or resource-limit error, check the cloud provider's own quota view for the specific volume type in the specific region, and separately confirm the CSI controller's service account or workload identity actually has permission to create volumes (a permissions error and a quota error can produce very similar-looking provisioner log lines, so read the exact error string rather than assuming which one it is).
Trade-offs and pitfalls
WaitForFirstConsumerbinding being mistaken for a stuck PVC is the single most common false alarm in this whole flow; always check whether a pod is even trying to consume the PVC before treating Pending as broken.- Static PV/PVC binding (hand-created PVs) trades away dynamic provisioning's convenience for exact control over which underlying disk a claim binds to, at the cost of every capacity or accessMode mismatch becoming a manual matching problem instead of the provisioner's job.
- A ReadWriteMany PVC (multiple pods, multiple nodes, one volume) is not supported by most cloud block-storage CSI drivers at all; that class of "PVC never binds" is a genuine design mismatch (need a shared-filesystem storage class like a managed NFS/Filestore-backed one) rather than a transient failure to retry.
Describe the kubectl commands and rollout strategies you would use to perform a safe rolling restart of a Deployment, view rollout history, and rollback to a previous revision. Include examples using kubectl and explain how you would avoid causing cascading failures during a restart of a consumer‑facing service.
Sample Answer
A safe restart uses kubectl rollout restart, which recreates pods through the normal RollingUpdate strategy rather than deleting them directly, so the same availability guarantees that protect a routine deployment protect the restart too.
Commands
Trigger and watch a rolling restart:
kubectl rollout restart deployment my-app -n prod
kubectl rollout status deployment my-app -n prod --watch
View rollout history and inspect a specific revision:
kubectl rollout history deployment my-app -n prod
kubectl rollout history deployment my-app -n prod --revision=3
Roll back:
kubectl rollout undo deployment my-app -n prod --to-revision=3
kubectl rollout status deployment my-app -n prod
Ship an image change with a recorded reason (the --record flag some older references use for this is deprecated; annotate explicitly instead):
kubectl set image deployment/my-app my-app=registry/app:1.2.3 -n prod
kubectl annotate deployment my-app kubernetes.io/change-cause="bump to 1.2.3, ticket OPS-441" --overwrite -n prod
What keeps a restart from becoming a cascading failure
- RollingUpdate parameters:
maxUnavailableandmaxSurge(both default to 25% of desired replicas) bound how many old pods can be down and how many extra new pods can exist at once. For a consumer-facing service, a conservative setting (for examplemaxUnavailable: 0, maxSurge: 1) never drops capacity below the current replica count during the restart, at the cost of briefly running more pods than the steady-state count. - Readiness probes: a Service only sends traffic to pods that pass their readiness probe, so a newly restarted pod that's still initializing doesn't receive requests it can't yet handle. This is the single biggest lever against a restart-induced error spike; without a readiness probe, the rollout has no signal that a "new" pod is actually ready and can start routing traffic to it immediately.
- PodDisruptionBudget (PDB): guarantees a minimum number (or percentage) of replicas stay available throughout the restart, independent of the Deployment's own
maxUnavailablesetting, which matters when other voluntary disruptions (a node drain, a cluster upgrade) happen to overlap with the restart window. - Graceful shutdown: a
preStophook plus aterminationGracePeriodSecondslong enough for in-flight requests to finish, combined with the Service removing the pod's endpoint before the container actually stops, avoids dropping requests that were already in progress when the restart began. - Staged rollout for risk-sensitive services: restarting (or deploying) to a small subset first, watching error rate and latency, then proceeding, catches a bad new revision before it reaches full traffic; this is a general staged-rollout practice, not a specific traffic-splitting mechanism (traffic-splitting techniques like weighted canary routing are a load-balancing/ingress-layer concern, not something the Deployment object itself provides).
Trade-offs and pitfalls
kubectl rollout restartonly recreates pods; it does not change the Deployment's spec, sorollout historyrecords it as a new revision with the same template, which is easy to forget when later trying toundoyour way back past a restart that changed nothing.- Setting
maxUnavailable: 0guarantees no capacity loss but requires enough spare cluster capacity formaxSurgeextra pods to schedule; on a tightly packed cluster this can leave the rollout stuck Pending on the surge pods instead of proceeding. - A rollback only restores the pod template (image, env, resource requests, and so on). If the Deployment reads a ConfigMap or Secret by a fixed name and that ConfigMap was edited in place rather than replaced with a new name or hash-suffixed name, rolling the Deployment back does not restore the old configuration content, only the old pod template pointing at the same (already-mutated) ConfigMap. This is the most common way a rollback fails to actually roll back.
- The same gap applies to a PersistentVolumeClaim (PVC): a Deployment's rollback restores the pod template's volume mount references, not the data on the volume itself. If the new version wrote a schema migration or otherwise mutated data in place on that volume, rolling the Deployment back gives you the old code pointing at already-changed data, not the old data. Anything stateful needs its own restore path (a volume snapshot or application-level backup) alongside the Deployment rollback, not instead of thinking about it separately.
Explain Kubernetes networking end-to-end: pod IP model, service virtual IPs, kube-proxy modes (iptables vs IPVS), CNI plugins and how they implement pod networking, and practical debugging steps when pods on different nodes cannot reach each other on specific TCP ports. Include commands and sources of truth to inspect.
Sample Answer
Direct answer
Every pod gets a routable IP address from the CNI (Container Network Interface) plugin, with no NAT (network address translation) between pods by default. A Service layers a stable virtual IP and DNS name on top of that, and kube-proxy (or an eBPF-based replacement like Cilium; eBPF lets Linux run sandboxed programs inside the kernel to process packets directly) programs the actual rewriting rules that turn a Service IP into a real pod IP. When two pods on different nodes cannot reach each other on a specific port, the fault sits in exactly one of four layers, checked in this order: the pod's own listener, the kube-proxy/Service plumbing, the CNI's cross-node routing, or a host/cloud firewall. Testing each layer directly beats guessing from symptoms.
Structured elaboration
Pod IP model
The CNI plugin assigns each pod a unique IP from the cluster's pod address range and is responsible for making that IP reachable from every node, either by programming host routes directly (an "underlay" approach, for example Calico in BGP mode, Border Gateway Protocol, the routing protocol routers use to exchange reachability information) or by encapsulating pod traffic in an overlay (VXLAN, a tunneling protocol that wraps one Ethernet frame inside a UDP packet, used by Flannel and by Calico's IP-in-IP mode).
Service virtual IPs and kube-proxy modes
A ClusterIP Service is a stable virtual IP that never corresponds to a real listener; kube-proxy rewrites traffic to it into a real pod IP and port.
| Mode | Mechanism | Status |
|---|---|---|
| iptables | Programs Linux netfilter/iptables NAT rules per Service and endpoint | Still the default |
| IPVS | Uses the kernel's IPVS load balancer, built for large numbers of Services | GA, scales better than iptables (near-constant-time lookup versus iptables' linear rule matching) |
| nftables | Programs nftables rules, the modern successor to iptables | Alpha in 1.29, beta in 1.31, GA in Kubernetes 1.33; not the default even where available, but the direction the project is moving, since the iptables backend is legacy code |
Cilium and some other CNIs bypass kube-proxy entirely, attaching eBPF programs to kernel networking hooks to do the same rewriting job with lower per-packet overhead and without iptables' rule-count scaling problem.
CNI plugins in practice
- Calico: pure L3 routing via BGP (no overlay, lowest overhead, needs a BGP-capable network) or IP-in-IP/VXLAN encapsulation where BGP peering is not available.
- Flannel: a simple VXLAN overlay by default; fewer features than Calico or Cilium.
- Cilium: eBPF-based; can replace kube-proxy, adds L7-aware NetworkPolicy and traffic observability, at the cost of needing a reasonably current kernel.
flowchart LR
A[Pod A on Node 1] -->|packet to Service ClusterIP| KP1[kube-proxy or eBPF dataplane, Node 1]
KP1 -->|rewrite to real Pod IP| CNI1[CNI routing or overlay, Node 1]
CNI1 -->|cross-node route or tunnel| CNI2[CNI routing or overlay, Node 2]
CNI2 --> B[Pod B on Node 2]
FW[Host firewall or cloud security group] -.may block specific ports.-> CNI2
Debugging cross-node connectivity, layer by layer
Reproduce and localize before changing anything:
$ kubectl get pod -o wide -n app
NAME READY STATUS NODE IP
pod-a 1/1 Running node-1 10.244.1.7
pod-b 1/1 Running node-2 10.244.2.9
$ kubectl exec -n app pod-a -- nc -vz 10.244.2.9 8080
nc: connect to 10.244.2.9 port 8080 (tcp) failed: Connection timed out
A timeout, not "connection refused," means the packet is being dropped somewhere in transit rather than rejected by pod-b's listener; that already rules out "the app isn't listening" and points at routing, the CNI, or a firewall.
- Confirm the listener exists:
kubectl exec -n app pod-b -- ss -tlnpshould show something bound to 8080; if not, this was never a networking problem. - Check node-to-node routing for the pod's address range:
$ ip route get 10.244.2.9
10.244.2.9 via 10.0.1.2 dev eth0 src 10.0.1.5
A missing route, or one pointing at the wrong interface, means the CNI failed to program cross-node routing, usually because its daemonset is crashlooping:
$ kubectl get pods -n kube-system -l k8s-app=calico-node
NAME READY STATUS RESTARTS
calico-node-x7k2p 0/1 CrashLoopBackOff 14
- If the failure is specifically Service-IP traffic rather than direct pod-IP traffic, check kube-proxy's own health the same way (
kubectl get pods -n kube-system -l k8s-app=kube-proxy) and inspect its programmed rules (sudo iptables -t nat -S | grep <service-name>orsudo ipvsadm -Ln, depending on mode). - If routing and the CNI both look healthy, check host-level and cloud firewall rules for the specific port. CNI and kube-proxy problems tend to affect all ports uniformly; a single-port failure with everything else in that layer healthy is a strong signal for a firewall rule scoped to that port.
- Capture packets at the suspect boundary to confirm rather than infer:
sudo tcpdump -n -i any host 10.244.2.9 and port 8080on the destination node shows whether the packet arrives at the host at all, before deciding whether the problem is upstream (routing) or downstream (host firewall, pod network namespace).
Trade-offs and pitfalls
- iptables rule count grows linearly with Services and endpoints and becomes a real CPU cost past a few thousand Services; IPVS or nftables scale better, but changing kube-proxy mode cluster-wide is a disruptive change to plan and test, not to flip live.
- The distinction between "Connection timed out" and "Connection refused" is easy to skip past but changes where to look first: refused means the packet arrived and something rejected it (wrong port, no listener); timed out means the packet was dropped somewhere before that.
- MTU (maximum transmission unit) mismatches between overlay encapsulation and the underlying network are a classic cause of connections that work for small packets and hang for large ones. Confirm with a full-size, non-fragmenting ping rather than assuming a working small ping rules out MTU.
Explain the differences between Deployment and StatefulSet. Provide a scenario (e.g., a database cluster) that requires StatefulSet semantics and explain which StatefulSet features (stable network IDs, ordinal indexes, persistent volumes) are necessary.
Sample Answer
Deployment and StatefulSet both manage a set of pods from one template, but a Deployment treats every pod as interchangeable, while a StatefulSet gives each pod a stable, ordinal identity and its own dedicated storage. Reach for StatefulSet only when the application itself needs to know which replica it is, not just how many replicas exist.
Direct comparison
| Deployment | StatefulSet | |
|---|---|---|
| Pod naming | random suffix, no ordering | ordinal suffix (pod-0, pod-1, ...), stable across restarts |
| Network identity | pod IP changes on recreation; normally fronted by a load-balancing Service | stable hostname per ordinal, usually via a headless Service (clusterIP: None), so pod-0.svc.namespace.svc.cluster.local always resolves to the same replica |
| Storage | usually shared, external, or none | one PersistentVolumeClaim (PVC, a request for storage) bound to a PersistentVolume (PV, the actual storage resource) per ordinal, via volumeClaimTemplates; deleting the StatefulSet does not delete these PVCs |
| Startup and scale order | no ordering guarantee | default OrderedReady policy starts and stops one ordinal at a time, lowest-to-highest on scale-up and highest-to-lowest on scale-down; a Parallel policy is available when the app does not need this ordering |
Worked example: a PostgreSQL primary-replica cluster
A database cluster with one primary and several read replicas needs exactly the guarantees a Deployment cannot give:
- Stable hostnames: replication configuration and client connection strings can target
postgres-0.postgres-svcby name, and that name always means the same replica, even across restarts. - Ordinal indexes: an init script can branch on the pod's ordinal (ordinal 0 initializes as primary, ordinals 1 and above join as replicas) and rely on ordered startup so replicas do not attempt to join before the primary exists.
- Per-ordinal persistent volumes:
volumeClaimTemplatesties each replica's data directory to its own ordinal, sopostgres-1always reattaches topostgres-1's data after a restart, never to another replica's.
Trade-offs and pitfalls
- Not everything with 'stateful' in its name needs a StatefulSet. Kafka brokers do: each broker owns local partition log segments on disk and needs a stable identity for replica assignment. Kafka Connect workers do not: a Connect worker's task state and offsets live in Kafka topics, not on local disk, so the worker pool is effectively stateless and runs fine as a plain Deployment. Confirm where the durable state actually lives before defaulting to StatefulSet.
- StatefulSet's ordered rollout is slower and more conservative than a Deployment's by design; if a stateful workload's replicas are genuinely independent (sharded, no leader election), that ordering guarantee adds rollout latency with no real benefit, and a
Parallelpod-management policy, or even a Deployment with per-shard volumes mounted individually, may fit better. - Deleting a StatefulSet leaves its PVCs behind on purpose, to protect data from an accidental delete. Teams that do not know this get surprised in both directions: unexpected data loss if they assumed cleanup was automatic, or orphaned storage cost if they never clean it up.
Explain Kubernetes resource requests, limits, and QoS classes (Guaranteed, Burstable, BestEffort). How do these settings affect scheduler decisions, eviction behavior under memory pressure, and node overcommit? Provide practical guidance for right-sizing microservice resource requests and limits.
Sample Answer
A resource request is what the scheduler reserves for a pod when deciding node placement; a limit is the hard ceiling the kubelet enforces once the pod is already running. Kubernetes derives a Quality of Service (QoS) class purely from how requests and limits compare across every container in the pod: Guaranteed, Burstable, or BestEffort. That class, not the raw numbers, is what decides which pod gets evicted first when a node runs low on memory.
QoS classes
| QoS class | How it is assigned | Scheduling weight | Eviction order under memory pressure |
|---|---|---|---|
| Guaranteed | every container sets request equal to limit, for both CPU and memory | highest | evicted last; exceeding its own limit instead triggers an immediate out-of-memory (OOM) kill of that container |
| Burstable | at least one container sets a request, but some request does not equal its limit | medium | evicted after BestEffort, ranked by how far actual usage exceeds the pod's memory request |
| BestEffort | no requests or limits set at all | lowest | evicted first |
Two mechanisms are doing the work here, and they are easy to conflate:
- Scheduling and overcommit: the scheduler only sums container requests to decide whether a pod fits on a node; limits are invisible to it. That is deliberate: setting limits above requests lets a cluster pack pods more densely than their guaranteed usage, trading headroom for cost. It also means a node can be scheduled to 100% of its requested memory while still having every pod's limit sum to well over the node's actual capacity.
- Enforcement at runtime: CPU limits are enforced through the cgroup (Linux control group, the kernel mechanism containers use for resource accounting and isolation) CFS (Completely Fair Scheduler) quota, so exceeding a CPU limit throttles the container rather than killing it. Memory limits are a hard cgroup cap; exceeding one gets the container killed by the Linux OOM killer, reported on the pod as
OOMKilled. - Eviction: separately from OOM kills, the kubelet watches node-level pressure signals (conditions such as
MemoryPressure) and proactively evicts pods, ranked by QoS class first and then by how far a pod's usage exceeds its own request within that class.
Worked example
Overcommit arithmetic. Suppose eight pods are each given a memory request of 256MiB and a limit of 512MiB, scheduled onto a node with 4GiB of allocatable memory:
8×256 MiB=2048 MiB=2 GiB
The scheduler is satisfied at 2GiB of committed requests, well under the node's 4GiB. But if every pod actually used its full limit at once:
8×512 MiB=4096 MiB=4 GiB
that consumes the entire node with zero headroom left for the OS and the kubelet itself, which is exactly the condition that triggers evictions or OOM kills. The gap between 2GiB and 4GiB is the overcommit the operator is deliberately accepting.
CPU throttling arithmetic. A container with a CPU limit of 0.5 (500 millicores) gets translated into a cgroup CFS quota against the kernel's default 100ms accounting period (cfs_period_us of 100000 microseconds):
cfs_quota_us=limit (CPUs)×cfs_period_us
0.5×100000=50000
So the container is allotted 50,000 microseconds of CPU time in every 100,000-microsecond window. Once it burns through that in a burst, the kernel throttles it until the next window opens, which shows up as added latency, not a crash or restart, and is invisible in dashboards that only track average CPU utilization.
Practical right-sizing guidance
- Measure real usage (p95/p99, steady-state and burst) under representative load before setting either number; guessing produces exactly the overcommit risk described above.
- Set requests near the workload's realistic steady-state usage so the scheduler's placement promise matches reality; set limits to cap runaway growth, keeping memory limits close to (or equal to) requests for anything where an OOM kill is worse than wasted capacity.
- Reserve Guaranteed QoS for workloads where losing the pod outright is worse than the density cost (primary databases, payment-critical services); use Burstable for typical stateless services with reasonable headroom; avoid BestEffort in production entirely.
Trade-offs and pitfalls
- Setting requests low purely to make a pod schedule more easily quietly turns Guaranteed-shaped intent into Burstable behavior and makes that pod an earlier eviction candidate than the team expects.
- All-Guaranteed clusters are simple to reason about but give up the density Kubernetes is meant to provide; treat Guaranteed as a scalpel, not a default.
- CPU throttling is easy to miss operationally because it does not appear as an error; a service that looks fine on average CPU utilization can still be throttled in short, latency-relevant bursts unless you scrape the cgroup throttling counters directly.
- Running the Vertical Pod Autoscaler (VPA) in Auto mode on the same metric a Horizontal Pod Autoscaler (HPA) reacts to causes the two to fight each other, since VPA resizing requests changes the utilization the HPA measures; keep VPA in recommendation-only mode, or scope it to memory while HPA drives scaling on CPU or a custom metric.
Unlock Full Question Bank
Get access to all Kubernetes Architecture, Operations, and Troubleshooting interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.