Kubernetes Troubleshooting Questions

Covers diagnosing and resolving failures in container orchestration with an emphasis on Kubernetes pod and deployment issues. Topics include Kubernetes architecture and components, pod lifecycle and states, common failure modes such as CrashLoopBackOff, ImagePullBackOff, out of memory conditions, resource limit and quota problems, and node or scheduling issues. Practical debugging skills include using command line inspection and control tools, reading and correlating logs and events, using exec into containers, describing resources, checking node and cluster health, and interpreting health checks, liveness and readiness probes. Also includes distinguishing platform or cluster issues from application level faults, analyzing networking and service discovery problems, using monitoring and observability data, understanding rolling updates and deployment strategies, and applying remediation and rollback techniques for reliable recovery.

MediumTechnical

96 practiced

Describe how to use 'kubectl debug' with nsenter or a privileged debug pod to inspect another container's process and mount namespaces for forensics (open files, /proc, network sockets). Provide the command patterns and explain security implications and necessary RBAC or admission controller changes to allow this in a hardened cluster.

Sample Answer

Approach: use kubectl debug to run a privileged ephemeral pod that either nsenters the target container PID/ns or runs tools inside the target container's mount and proc namespaces. This is useful for forensics: listing open files, /proc, net sockets.

Example 1 — use kubectl debug with nsenter helper (attaches to host PID namespace then nsenter into container PID ns):

bash

# create ephemeral debug container sharing target pod's PID and mount namespaces
kubectl debug -it pod/my-app --image=nicolaka/netshoot --target=container-name --share-processes
# inside debug shell, find target PID and nsenter
TARGET_PID=$(pgrep -f 'process-name' || ps aux | grep process-name)
nsenter -t $TARGET_PID -a  # requires nsenter in image
# now run forensics commands
lsof -p $TARGET_PID
ls -l /proc/$TARGET_PID/fd
ss -pan

Example 2 — privileged debug pod mounting target container rootfs:

bash

kubectl debug node/<node-name> -it --image=alpine --privileged --share-processes
# or: kubectl debug pod/my-app -it --image=busybox --privileged --copy-to=debug-my-app
# mount container filesystem if needed, then inspect /proc or /sys

Key points:- --target lets kubectl create a debug container in the same pod to join namespaces.- --privileged or CAP_SYS_ADMIN may be required to nsenter, mount, or read /proc of another PID.- Tools: lsof, ss, strace, gdb, auditd artifacts; ensure debug image contains nsenter.

Security implications and required changes:- Joining other containers' namespaces is highly privileged: it can access secrets, host network, and escalate to host. Hardened clusters often block privileged pods and hostPID/hostIPC.- RBAC: create a ClusterRole allowing pods/exec/create on pods for specific service accounts used for debugging; limit to a namespace or label selector.- Admission controllers: PodSecurityPolicy (deprecated) or PodSecurity admission and PSP replacement (e.g., OPA/Gatekeeper) must allow PodSpec fields: hostPID, hostIPC, privileged, allowed capabilities. Create scoped exceptions for a dedicated debugging SA, bounded by namespace and TTL.- Audit and approval: require an approval workflow (JIT access), log all kubectl debug actions, and restrict images to an approved registry.

Safe practice:- Use ephemeral debug pods, minimal time-to-live, restrict SA, use read-only images when possible, scan debug images, and require multi-person approval for escalation.

MediumTechnical

75 practiced

Pods on two nodes cannot reach each other by IP while host network between those nodes works. Outline a CNI troubleshooting plan including verifying CNI daemonset health, CNI config files, iptables/ipvs rules, kube-proxy mode, and how to capture and interpret packet traces to isolate whether the issue is overlay, encapsulation, or routing.

Sample Answer

Start with hypothesis: host networking works so underlying L2/L3 between nodes is OK; problem is CNI overlay/encap/routing or kube-proxy rules. Follow this ordered checklist.

1) Verify CNI daemonset & pods- kubectl get ds -n kube-system | grep -E "calico|flannel|weave|cilium"- kubectl get pods -n kube-system -o wide- kubectl logs -n kube-system <cni-pod> --tail=200Check CrashLoopBackOff, image pull, or BPF/xdp errors.

2) Inspect CNI config and plugin binaries- On nodes: ls /etc/cni/net.d; cat /etc/cni/net.d/10-calico.conflist (or .conf)- Verify plugin binaries in /opt/cni/bin and config (IPAM ranges, MTU, encapsulation mode)

3) Check node local routing and iptables/ipvs- kube-proxy mode: kubectl get configmap -n kube-system kube-proxy -o yaml | grep mode- On nodes: ip route show ip addr show sudo iptables -t nat -S | head sudo iptables -S FORWARD sudo ipvsadm -L -n (if IPVS)Ensure FORWARD policy ACCEPT and CNI-created chains present.

4) Reproduce and capture packets- From pod A (node1) ping pod B IP; capture on: - Host veth on node1: sudo tcpdump -i <veth0> -w /tmp/node1-veth.pcap - Host physical interface on node1: sudo tcpdump -i eth0 -w /tmp/node1-eth.pcap - On node2: same for veth and eth0Commands:

bash

# find veth for pod:
pid=$(kubectl exec -n default pod-a -- sh -c 'echo $PPID' ) # or docker inspect
sudo nsenter -t $pid -n ip a
# tcpdump
sudo tcpdump -i eth0 host <podB_IP> -w /tmp/eth-node1.pcap
sudo tcpdump -i vethXXX host <podB_IP> -w /tmp/veth-node1.pcap

5) Interpret traces- If packets leave veth and appear on eth0 as encapsulated (e.g., UDP VXLAN dst port 4789 or IP protocol 50 for IPIP): implies overlay encapsulation is used; check MTU, UDP checksums, and offload issues.- If packets appear on eth0 as pod IP (no encap) and reach node2 eth0, but not delivered to pod veth on node2: problem with node2 routing/bridge or iptables FORWARD or CNI programming.- If packets never leave node1 eth0: encapsulation failed or CNI failed to program neighbor/route.- If you see ARP failures for pod IPs on node2: check routes/arp_proxy settings and CNI route table.- If iptables/NAT rules drop or MASQUERADE missing, kube-proxy or CNI iptables chains likely misconfigured (check conntrack).

6) Fixes to try- Restart CNI pods on affected nodes; mount logs.- Verify MTU: lower pod MTU to avoid fragmentation across encapsulation.- If IPVS mode, ensure kube-proxy running and kernel modules loaded.- For encapsulation issues, check kernel offload: ethtool -K eth0 tx off sg off and test.

7) Verification- Re-run tcpdump and trace to confirm decapsulation then successful veth delivery and ICMP response. Monitor metrics/alerts.

This plan isolates whether failure is at overlay encapsulation, host routing, or kube-proxy/iptables programming.

MediumTechnical

69 practiced

An application inside a pod fails with 'permission denied' when writing to a mounted volume. Explain how pod securityContext (runAsUser, runAsGroup), fsGroup, and container user permissions interact with filesystem ownership. Describe safe manifest changes or initContainer patterns to fix permissions without granting broad host privileges.

Sample Answer

Short answer:- Pod-level securityContext.runAsUser/runAsGroup set the UID/GID the container processes run as.- Pod-level fsGroup instructs Kubelet to ensure the mounted volume’s group ownership includes that GID (and on supported volume plugins to recursively chown/chmod) so processes can write even if file owner differs.- Container image file permissions still matter: if files/dirs are owned by a different UID and mode 000/400, process UID (or its groups) must have write permission.- Safe fixes: prefer fsGroup or per-pod runAsUser over running containers as root; use a one-time initContainer that adjusts permissions as root (only in the pod), avoiding host-level privileged mounts.

Detailed explanation + examples:

How they interact- runAsUser/runAsGroup: the Linux UID/GID used by the process inside the container. If you set runAsUser: 1000, processes run as UID 1000.- fsGroup: Kubelet will change the mounted volume’s group ownership to fsGroup (and on many volume types will recursively chown/chmod) so the group has access. The container process is member of that supplemental group, allowing writes if group write bit is set.- Filesystem ownership + mode: kernel enforces owner/group/other bits. If the directory is root:root with mode 0755 and your process UID is non-root and not in the group, writes fail.

Safe manifest changes (preferred, non-privileged)1) Use fsGroup and set runAsUser to non-root:api example:

yaml

apiVersion: v1
kind: Pod
metadata: {name: app}
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - mountPath: /data
      name: data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: pvc-data

Why: Kubelet ensures group ownership and the process runs as UID 1000 with GID 1000; group write allows writes without root.

2) If fsGroup is not applied or volume plugin doesn’t support recursive chown (or you need finer control), use an initContainer to adjust permissions inside the pod rather than changing host:

yaml

initContainers:
- name: fix-perms
  image: busybox
  securityContext:
    runAsUser: 0            # root only inside the pod, not host-privileged
  command: ["sh","-c","chown -R 1000:1000 /data && chmod -R g+rwX /data"]
  volumeMounts:
  - name: data
    mountPath: /data
containers:
- name: app
  image: myapp:latest
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
  volumeMounts:
  - name: data
    mountPath: /data

Notes: running an initContainer as root inside the pod is common and does not require privileged or hostPath privileges. It changes only the files within the volume for that pod lifecycle.

Caveats and best practices- Some managed volume plugins (CSI, cloud disks) support fsGroup and will perform the chown as kubelet; check your environment. For NFS or custom CSI drivers, fsGroup may not be applied automatically.- Avoid making PVCs world-writable (chmod 777) unless you accept the security risk.- Avoid hostPath with privileged containers. If you must use hostPath, restrict scope and use PodSecurityPolicies/PSA to limit privileges.- Use immutable images that expect non-root, and bake ownership into the image where possible (chown in Dockerfile when using build-time UID mapping).- Audit: add SecurityContextConstraints/PodSecurityAdmission rules and RBAC to prevent unintended privilege escalation.

In short: prefer fsGroup + non-root runAsUser; when fsGroup isn’t sufficient, use a short-lived initContainer that runs as root only inside the pod to chown/chmod the mounted path. This fixes permissions without granting broad host privileges.

EasyTechnical

76 practiced

Explain the difference between resource requests and limits for CPU and memory in Kubernetes. Describe how the scheduler uses requests, how kubelet enforces limits, and examples of misconfiguration that can lead to CPU throttling or containers being OOMKilled. Include how QoS classes are derived from requests/limits.

Sample Answer

Resource request vs limit (CPU & Memory)- Request: the amount of resource the pod promises to need. The scheduler uses requests to decide which node has capacity.- Limit: the maximum the container is allowed to use. Kubelet enforces limits via cgroups.

How the scheduler uses requests- The scheduler sums pod requests per node and only places a pod on nodes where requested CPU/memory fit. Requests drive placement and capacity planning (they’re treated as reserved).

How kubelet enforces limits- CPU: enforced by cgroups (CFS). If a container tries to use more CPU than its limit it gets throttled (time slices reduced); processes are not killed.- Memory: enforced by cgroups. Exceeding memory limit causes the kernel OOM killer to kill the process -> container becomes OOMKilled.

Misconfig examples- Low request, high limit: Scheduler may overcommit the node (many pods placed). If many pods suddenly use CPU up to their limits and limits are low, they get throttled—seen as high throttling metrics (container_cpu_cfs_throttled_seconds_total). For memory, if limits are low and usage spikes above limit, pods are OOMKilled.- No requests specified: pods land on nodes until sum of requests appears small -> risk of node pressure and eviction.- Requests > limits (invalid): limits must be >= requests; some schedulers/validators reject this.- Identical request and limit (CPU): avoids surprise throttling but reduces bin-packing flexibility.

QoS classes derivation- Guaranteed: every container in the pod sets memory and cpu request == limit. Pod prioritized for eviction last.- Burstable: pod has requests set but at least one container has request < limit or some containers lack limits; intermediate eviction priority.- BestEffort: no requests specified for any container; highest eviction priority under memory pressure.

SRE recommendations- Set realistic requests from monitoring (P95 usage) to help scheduler decisions.- Set limits to prevent noisy neighbors; for memory, set limit reasonably above requests to avoid OOMs on short spikes.- Monitor throttling (cpu throttled metrics) and OOMKilled events, and iterate requests/limits based on real usage.

MediumTechnical

97 practiced

Write a POSIX-compliant bash one-liner that lists all pods across all namespaces currently in CrashLoopBackOff and prints a header 'NAMESPACE/POD - last 20 log lines' followed by those last 20 lines using only kubectl and basic shell tools (no jq). Explain any assumptions you make for performance at scale.

Unlock Full Question Bank

Get access to hundreds of Kubernetes Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.