Kubernetes Troubleshooting Questions

Covers diagnosing and resolving failures in container orchestration with an emphasis on Kubernetes pod and deployment issues. Topics include Kubernetes architecture and components, pod lifecycle and states, common failure modes such as CrashLoopBackOff, ImagePullBackOff, out of memory conditions, resource limit and quota problems, and node or scheduling issues. Practical debugging skills include using command line inspection and control tools, reading and correlating logs and events, using exec into containers, describing resources, checking node and cluster health, and interpreting health checks, liveness and readiness probes. Also includes distinguishing platform or cluster issues from application level faults, analyzing networking and service discovery problems, using monitoring and observability data, understanding rolling updates and deployment strategies, and applying remediation and rollback techniques for reliable recovery.

EasyTechnical

141 practiced

Explain the difference between liveness and readiness probes. Provide two concrete examples where a readiness probe should mark a pod NotReady without triggering a restart, and explain how misconfigured probes can cause outages.

HardTechnical

68 practiced

A service mesh sidecar injection recently caused some pods to fail at startup. Describe a methodical approach to isolate whether the sidecar or the app is at fault, including runtime checks, logs to inspect, and safe mitigations to bring services back without disabling the mesh cluster-wide.

Sample Answer

**Approach summary (methodical, low-risk)** 1) Reproduce & scope the blast radius. 2) Isolate sidecar vs app with targeted runtime checks and logs. 3) Apply safe mitigations (per-pod/namespace) to restore service. 4) Root cause: compare sidecar version/config vs working pods.

**Runtime checks to run**- Check pod status and events:

bash

kubectl get pods -n app-ns
kubectl describe pod <pod> -n app-ns
kubectl get events -n app-ns --sort-by='.lastTimestamp'

- Inspect containers and crash reasons:

bash

kubectl logs <pod> -c <app-container> -n app-ns
kubectl logs <pod> -c <sidecar-container> -n app-ns
kubectl logs <pod> -p -c <sidecar-container> -n app-ns   # previous crashed instance

- Check readiness/liveness probe failures:

bash

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].state}' -n app-ns

- Check sidecar-specific diagnostics: - Istio: `istioctl proxy-status`, `istioctl pc log <pod>` - Linkerd: `linkerd -n app-ns check`, `kubectl logs -c linkerd-proxy`

**What logs/configs to inspect**- Sidecar logs for startup errors, TLS certs, envoy bootstrap failures, plugin mismatches.- App logs for port bind failures, env var or config mount errors, crash stacks.- Pod spec: initContainers, volumeMounts, securityContext, ports (conflicts).- Resource events: OOMKill, ImagePullBackOff.- Mesh control-plane logs (pilot/istio-ingress, linkerd-controller) for recent config pushes.

**Isolation technique (safe, per-pod)**- Temporarily disable injection for an affected pod via annotation to run app without sidecar:

bash

kubectl annotate pod <pod> sidecar.istio.io/inject=false --overwrite -n app-ns
# or re-deploy with annotation on Deployment to opt-out for a single replica

- Create a clone Deployment with sidecar disabled (use same image, different name) and scale to 1 to verify app starts without sidecar.- For Istio: use `istioctl analyze` on the pod/namespace.

**Safe mitigations to restore service**- Roll back the last mesh config or sidecar proxy version for affected nodes only (avoid cluster-wide).- Deploy a non-meshed canary: annotate Deployment to opt-out, route a percentage of traffic to it.- Increase resource limits or probe timeouts if sidecar startup delays cause readiness failures.- If a fleet-wide sidecar rollout caused regression, pause the sidecar upgrade and roll back control-plane or proxy image on a small subset.

**Root-cause validation & remediation**- If app runs without sidecar → sidecar responsible. Compare sidecar env, bootstrap/config, mTLS, and proxy version. Fix config or pin working proxy image.- If app still fails → application issue (config, mount, port, dependency). Reproduce locally with same image/container runtime.

**Postmortem actions**- Add automated smoke tests (start-and-check readiness) for sidecar rollouts in CI/CD.- Use canary sidecar rollouts, health gating, and documented per-namespace opt-out procedures.

EasyTechnical

85 practiced

You observe a Deployment with multiple pods stuck in CrashLoopBackOff. Describe the immediate investigative steps you would take to identify the root cause and the specific commands you would run to capture evidence before taking remediation actions to restore service.

Sample Answer

**Approach (brief):**I would quickly gather observable state and logs to triage root cause, capture all evidence to immutable files, then only after evidence collection start remediation. Prioritize what affects availability (readiness/liveness, resource starvation, image/runtime errors).

Steps & commands to run (capture outputs to files):

1. Observe pods and recent events

bash

kubectl get pods -n myns -o wide > /tmp/pods.txt
kubectl get events -n myns --sort-by=.lastTimestamp > /tmp/events.txt

2. Describe a failing pod to see restart reasons, events, mounts, env, probes

bash

kubectl describe pod mypod-xxxx -n myns > /tmp/describe-mypod.txt

3. Fetch container logs (previous and current)

bash

kubectl logs mypod-xxxx -c app -n myns > /tmp/logs-mypod-current.txt
kubectl logs mypod-xxxx -c app -n myns --previous > /tmp/logs-mypod-previous.txt

4. If init containers or crashes before logging, dump pod YAML, env, and filesystem

bash

kubectl get pod mypod-xxxx -n myns -o yaml > /tmp/pod-mypod.yaml
kubectl exec -it mypod-xxxx -c app -n myns -- env > /tmp/env-mypod.txt
kubectl exec -it mypod-xxxx -c app -n myns -- ls -la /app > /tmp/ls-mypod.txt

5. Check node and resource issues

bash

kubectl get nodes -o wide > /tmp/nodes.txt
kubectl top pod -n myns > /tmp/top-pods.txt
kubectl top node > /tmp/top-nodes.txt

6. Inspect image and startup command if crash on exec

bash

kubectl get deployment mydeploy -n myns -o yaml > /tmp/deploy.yaml
kubectl rollout history deployment/mydeploy -n myns > /tmp/rollout.txt

7. Capture cluster-level logs/CRI if pod can't exec (node journal)

bash

ssh node-of-pod 'sudo journalctl -u kubelet --no-pager > /tmp/kubelet.log'

Why these steps:- describe/events show probe failures, OOMKills, CrashLoopBackOff reasons.- logs (--previous) capture the crash output.- pod YAML and env reveal misconfig, wrong args, secrets or mount issues.- node checks find resource pressure or disk issues.- Saving outputs preserves evidence for postmortem and safe rollback.

After evidence collection I'd coordinate with devs for fixes (config, image), consider scaling down/rollback, and if urgent, restart/replace pods or rollback deployment.

HardTechnical

93 practiced

A container's liveness probe calls an endpoint that under heavy load causes the process to terminate, producing CrashLoopBackOff. Describe how you would change probe configuration or application behavior to avoid false-positive restarts. Provide an example of probe configuration changes and explain trade-offs between responsiveness and stability.

Sample Answer

**Approach (brief)** When liveness calls a heavy endpoint and that endpoint causes the process to die under load, treat it as a false-positive liveness failure. Fix either probe configuration (make Kubernetes less aggressive) or app behavior (make the endpoint safe under probe load). Prefer combination: adjust probes, add readiness/startup probes, and make the probed endpoint lightweight or rate-limited.

**Probe configuration example**

yaml

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 60       # give app time to warm up
  periodSeconds: 15             # poll less frequently
  timeoutSeconds: 5
  failureThreshold: 6           # tolerate transient failures

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

# Optional startupProbe to avoid liveness hits during startup
startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 60
  periodSeconds: 10

**Application changes**- Expose a lightweight /healthz that only checks in-memory state.- Make probed endpoint non-blocking or use a separate thread/pool.- Implement rate-limiting, circuit breaker, or prioritize probe requests.- Increase resource limits (CPU/memory) or tune thread pools.

**Trade-offs: responsiveness vs stability**- More tolerant probes (longer timeouts, higher thresholds) reduce false restarts but slow detection of real failures.- Short aggressive probes detect hangs quickly but can cause CrashLoopBackOff under transient load.- Use readinessProbe to stop traffic while keeping liveness stricter for catastrophic failures.- startupProbe avoids killing pods during initialization.

Recommendation: create a minimal health endpoint for liveness, use readiness for load-sensitive checks, and tune initialDelay/thresholds to match app behavior.

HardTechnical

70 practiced

The cluster is showing high API-server request latency, causing kubectl to be slow and controllers to lag. Which metrics and logs would you inspect (etcd request latency, apiserver request-duration, audit logs), and how would you triage whether the bottleneck is etcd, API server, network, or noisy clients?

Sample Answer

**Overview / approach**Start by collecting evidence (metrics + logs), then triangulate: is latency originating in etcd, API server, network stack, or noisy clients? Use steady-state comparison (before vs during), and isolate components by checking timings, error rates, leader changes, and high-volume callers.

**Metrics & logs to inspect**- API server metrics: - apiserver_request_duration_seconds (by verb/resource/handler) — looks for long tails - apiserver_current_inflight_requests, apiserver_request_total (4xx/5xx) - audit logs (who/what IP/verb/resource and timestamps)- etcd metrics: - etcd_server_request_duration_seconds, etcd_disk_backend_commit_duration_seconds - etcd_disk_wal_fsync_duration_seconds, etcd_network_peer_round_trip_time_seconds - etcd_server_leader_changes_seen_total, etcd_mvcc_db_total_size_in_bytes- Node / network: - kubelet API latency, CNI plugin metrics, packet drops, interface errors - netstat/tcpdump between apiserver and etcd- Controller-manager/scheduler: reconcile duration, workqueue queue length- Logs: - apiserver logs (slow request traces), etcd logs (leader election, failed fsync), kube-apiserver audit logs showing repetitive clients

**Commands / queries**

bash

# Check apiserver latency by handler
kubectl get --raw /metrics | grep apiserver_request_duration_seconds

# PromQL examples
# 95th-percentile latency per verb/resource
histogram_quantile(0.95, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le,verb,resource))
# etcd 95th
histogram_quantile(0.95, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (le))

**Triage steps**1. Correlate timestamps: find high apiserver request-duration spikes and check etcd metrics at same time.2. If apiserver latency high but etcd request_duration low: - Inspect apiserver CPU/memory, GC pauses, threadpool saturation, kube-apiserver logs for auth/webhook timeouts. - Check network: tcpdump between apiserver and etcd; measure RTT (curl) to etcd endpoints.3. If etcd request_duration or disk fsync latency high: - Look for high disk I/O, fsync latency, leader changes; check etcd slow logs and underlying storage (cloud disk IOPS). - Mitigate by adding etcd resources, improving storage (provision faster disks), or scaling read-only proxies.4. If audit logs show a small set of IPs or service accounts issuing massive requests: - Identify noisy clients (controllers, operators, CI bots) and throttle or fix them (respect client-side watch/limit, use fieldSelector/labelSelector). - Use apiserver request throttling (client-go QPS/Burst) or API priority and fairness.5. If network issues (packet drops/RTT variance): - Fix CNI, MTU, host networking, route tables; consider colocating apiserver and etcd network-wise.

**Decision signals**- etcd is culprit: high etcd_disk_* latencies, leader changes, increased etcd_request_duration synchronized with apiserver spikes.- apiserver-only: high apiserver CPU/GC, long auth/webhook times, but healthy etcd metrics.- network: high RTT between apiserver <-> etcd, packet loss, intermittent spikes.- noisy clients: audit logs show massive request rates from few sources; apiserver_request_total high with specific user/agent.

**Remediations (short-term vs long-term)**- Short: throttle clients, restart/reduce load on apiserver components, increase etcd disk IOPS, temporarily scale control plane if managed.- Long: tune client-go QPS/Burst, enable API Priority and Fairness, move etcd to provisioned fast storage, add monitoring/alerts for these specific metrics.

This shows how I’d gather evidence, isolate the layer causing latency, and apply targeted mitigation while capturing root-cause data for a permanent fix.

Unlock Full Question Bank

Get access to hundreds of Kubernetes Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.