Kubernetes Troubleshooting and Observability Questions

Kubernetes specific observability and debugging skills for clusters and containerized workloads. Topics include kubectl debugging commands, interpreting pod and node events, container and node metrics, resource requests and limits, log aggregation for pods, interpreting scheduler and kubelet behavior, networking within clusters, operator and control plane issues, and integrating Prometheus, Fluentd, Loki, and tracing into Kubernetes. Emphasis on diagnosing pod failures, resource contention, networking problems, and cluster level observability at scale.

EasyTechnical

45 practiced

Describe the kubectl commands and rollout strategies you would use to perform a safe rolling restart of a Deployment, view rollout history, and rollback to a previous revision. Include examples using kubectl and explain how you would avoid causing cascading failures during a restart of a consumer‑facing service.

Sample Answer

Start with safe rollout configuration and verification, then perform a controlled restart, inspect history, and rollback if needed.

Commands/examples:- Trigger a rolling restart (recommended for config/env changes):

bash

kubectl rollout restart deployment my-app -n prod
kubectl rollout status deployment my-app -n prod --watch

- View rollout history and revisions:

bash

kubectl rollout history deployment my-app -n prod
kubectl rollout history deployment my-app --revision=3 -n prod

- Roll back to a revision:

bash

kubectl rollout undo deployment my-app --to-revision=3 -n prod
kubectl rollout status deployment my-app -n prod

- Force new ReplicaSet by changing image or annotation:

bash

kubectl set image deployment/my-app my-app=registry/app:1.2.3 -n prod --record
# or
kubectl annotate deployment my-app kubernetes.io/change-cause="restart $(date -u)" --overwrite -n prod

Rollout strategy / risk mitigation (how and why):- Ensure Deployment spec has graceful settings: strategy.type=RollingUpdate with conservative values (e.g., maxUnavailable: 1, maxSurge: 1) so capacity never drops dangerously.- Use readinessProbes so pods only receive traffic when healthy; liveness probes detect bad pods.- Use PodDisruptionBudget to guarantee minimum available replicas during restarts.- Stagger traffic and deploy to a canary subset or a separate namespace/blue-green environment when consumer-facing: route a small percentage of traffic to new pods first, monitor metrics (errors, latency, CPU, user-facing SLOs) and only proceed if stable.- Monitor during rollout: kubectl rollout status, Prometheus/Grafana, logs, and SLO alerts. If metrics degrade, rollback quickly with kubectl rollout undo.- Prevent cascading failures by limiting downstream concurrency (rate limiting, circuit breakers), short preStop hooks to let in-flight requests drain, and by testing health checks and connection draining before rollout.

Edge checks:- Validate revision exists before undo.- Confirm PDB and HPA interplay so autoscaler doesn’t reduce capacity during rollout.- Automate canary promotion only after observability thresholds pass.

MediumTechnical

39 practiced

Describe the key steps to instrument an existing microservice running in Kubernetes with OpenTelemetry traces so it appears in a distributed trace system (e.g., Jaeger/Grafana Tempo). Include code/library changes, Kubernetes deployment changes (sidecar vs collector), sampling strategy, and configuration needed to correlate traces with Prometheus metrics and pod logs.

Sample Answer

Situation: You’re adding OpenTelemetry tracing to an existing Kubernetes microservice so traces show up in Jaeger/Tempo and correlate with Prometheus metrics and pod logs.

1) Code / library changes- Add OpenTelemetry SDK and exporter (example in Python):

python

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider, sampling
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({"service.name":"orders","service.instance.id": os.getenv("POD_NAME")})
provider = TracerProvider(resource=resource, sampler=sampling.ParentBased(sampling.TraceIdRatioBased(0.1)))
trace.set_tracer_provider(provider)
otlp = OTLPSpanExporter(endpoint="otel-collector.otel.svc.cluster.local:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp))

- Instrument code paths (manual or auto-instrumentation) to create spans and add attributes (http.method, db.statement, pod metadata).

2) Kubernetes deployment changes- Preferred: central OpenTelemetry Collector (Deployment + horizontal scaling) receiving from services via OTLP (gRPC/HTTP). Collector then exports to Jaeger/Tempo and to Prometheus (metrics).- Alternative: sidecar collector per pod for low-latency or multi-tenancy; heavier resource cost.- Expose collector via ClusterIP service otel-collector. Use NetworkPolicy to restrict.

Example Service annotation (optional auto-injection):- If using an injector, annotate pods for auto-instrumentation.

3) Sampling strategy- Head-based: set client sampler (TraceIdRatioBased) for low overhead (e.g., 0.05–0.2). Good for scale.- Tail-based (recommended for accuracy): Collector evaluates spans and keeps interesting traces (errors, high latency) using policy-based sampling—requires storing partial traces; implement in collector pipeline.- Combine: low-rate head sampling + collector tail-sampling for errors.

4) Correlating traces, Prometheus metrics, and logs- Ensure Resource attributes: service.name, service.instance.id (POD_NAME), namespace, deployment.- Export metrics from Collector to Prometheus using the prometheus receiver/exporter or have apps expose OTLP metrics.- Add trace context to logs: configure logging to include W3C traceparent and trace_id. Example Python logging formatter:

python

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
trace_id = format(trace.get_current_span().get_context().trace_id, '032x')
logger.info("processing", extra={"trace_id": trace_id})

- Use Fluentd/Fluent Bit or Loki to parse trace_id field and link logs to traces in UI.- In Prometheus/Grafana, attach labels like service, instance, trace_id when possible (e.g., push gateway or metrics with exemplar support). Exemplar support requires instrumenting Histogram/Counter to attach trace IDs via Collector so Grafana can link metrics to traces.

5) Collector config (high-level)- Receivers: otlp- Processors: batch, tail_sampling (optional)- Exporters: jaeger/tempo, prometheus- Example pipeline: otlp -> batch -> tail_sampling -> jaeger/tempo

6) Validation & ops- Start with dev: 100% sampling or debug traces for a short window.- Verify traces in Jaeger/Tempo; check logs have trace_id; check Prometheus metrics show exemplars.- Monitor collector CPU/memory, tune batch sizes and sampling.

Key trade-offs:- Sidecar = isolation/low-latency but resource heavy; central collector = easier management and tail-sampling but single networking hop.- Tail-sampling improves signal for debugging at cost of complexity/state.

This approach gives end-to-end traces in Jaeger/Tempo, correlated logs (trace_id), and Prometheus metrics with exemplars for drill-down.

MediumTechnical

39 practiced

Write a Bash script (or Python) that fetches logs for all pods in a given Deployment namespace created in the last 15 minutes and aggregates them into a single file per pod. The script should accept arguments: namespace and deployment name. Outline error handling for Pods that crash immediately and for containers with no logs.

Sample Answer

Approach: use kubectl + jq to list pods owned by the deployment whose creationTimestamp is within the last 15 minutes, then for each pod iterate containers, fetch logs (current and fallback to --previous), and write an aggregate file per pod. Handle errors (no logs, immediate crash/terminated containers) and return non-zero exit codes where appropriate.

bash

#!/usr/bin/env bash
# Usage: ./fetch_deploy_logs.sh <namespace> <deployment>
set -eo pipefail

NAMESPACE=$1
DEPLOYMENT=$2
SINCE_MINUTES=15
OUTDIR="./logs_${DEPLOYMENT}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$OUTDIR"

if [[ -z "$NAMESPACE" || -z "$DEPLOYMENT" ]]; then
  echo "Usage: $0 <namespace> <deployment>"
  exit 2
fi

now_ts=$(date +%s)
cutoff_ts=$((now_ts - SINCE_MINUTES*60))

# get pods in namespace, filter by ownerReference to deployment and creationTimestamp
pods=$(kubectl get pods -n "$NAMESPACE" -o json | jq -r --arg dep "$DEPLOYMENT" --argjson cutoff "$cutoff_ts" '
  .items[]
  | select(.metadata.ownerReferences[]? .name == $dep)
  | select((.metadata.creationTimestamp | sub("Z$";"")) as $t
           | ( ($t | sub("T";" ") ) | strptime("%Y-%m-%d %H:%M:%S") | mktime ) >= $cutoff)
  | .metadata.name')

if [[ -z "$pods" ]]; then
  echo "No pods for deployment '$DEPLOYMENT' created in last ${SINCE_MINUTES}m"
  exit 0
fi

rc=0
for pod in $pods; do
  outfile="${OUTDIR}/${pod}.log"
  echo "=== Pod: $pod ===" > "$outfile"
  # list containers (including init containers)
  containers=$(kubectl get pod "$pod" -n "$NAMESPACE" -o json | jq -r '.spec.initContainers[]?.name, .spec.containers[]?.name' | sed '/^$/d' | uniq)
  if [[ -z "$containers" ]]; then
    echo "[WARN] No containers found for pod $pod" >> "$outfile"
    continue
  fi

  for c in $containers; do
    echo "--- Container: $c (current) ---" >> "$outfile"
    if kubectl logs "$pod" -c "$c" -n "$NAMESPACE" --since=${SINCE_MINUTES}m >> "$outfile" 2>>"$outfile"; then
      echo "[INFO] fetched current logs for $pod/$c" >> "$outfile"
    else
      echo "[WARN] failed to fetch current logs for $pod/$c; trying --previous" >> "$outfile"
      if kubectl logs "$pod" -c "$c" -n "$NAMESPACE" --previous >> "$outfile" 2>>"$outfile"; then
        echo "[INFO] fetched previous logs for $pod/$c" >> "$outfile"
      else
        echo "[ERROR] no logs available for $pod/$c (container may have crashed immediately or produced no logs)" >> "$outfile"
        rc=3
      fi
    fi
    echo "" >> "$outfile"
  done
done

echo "Logs aggregated under $OUTDIR"
exit $rc

Key points / error handling:- Filter by ownerReference to ensure pods belong to the Deployment.- Use creationTimestamp compared to current time to select pods created in last 15 minutes.- For containers that crash immediately, kubectl logs current will fail; script retries with --previous to capture terminated container logs.- If both current and --previous fail, record an [ERROR] note in the pod file and set non-zero exit code.- If a container has no logs, the file will include a clear message; caller can alert on non-zero exit code.- Requires jq and kubectl configured with proper kubeconfig/permissions.

EasyTechnical

39 practiced

Explain how labels and selectors affect service discovery and traffic routing in Kubernetes. Provide an example where a typo or wrong selector causes traffic to disappear from a service and list kubectl commands to validate and fix the issue safely.

Sample Answer

Labels are key/value metadata attached to objects; selectors (label selectors) are used by Services, ReplicaSets, Deployments, and other controllers to identify which Pods they should target. A Service routes traffic to the Pods whose labels match its selector. If labels and selectors mismatch (e.g., typo, different key or value), the Service’s endpoints list becomes empty and traffic “disappears” even though Pods are Running.

Example (typical failure):- Deployment creates Pods with label app: frontend- Service selector is app: fronted (typo) → no endpoints, 0 ready backends, requests to ClusterIP or LoadBalancer return connection refused or 5xx.

How to validate and diagnose (safe, non-disruptive checks):- List Service and its selector:

bash

kubectl get svc my-service -o yaml

- List Pods and their labels:

bash

kubectl get pods --show-labels
kubectl get pods -l app=frontend
kubectl describe pod <pod-name>  # inspect pod labels in metadata

- Check Endpoints / Endpointslices:

bash

kubectl get endpoints my-service
kubectl describe endpoints my-service
kubectl get endpointslices -l kubernetes.io/service-name=my-service

- Test connectivity inside cluster:

bash

kubectl run -it --rm debug --image=busybox --restart=Never -- sh
# inside shell:
wget -qO- http://my-service.namespace.svc.cluster.local:80

How to fix safely:1. Prefer changing Service selector only if you’re sure all relevant Pods should be matched. More often, fix Pod labels on the Deployment so other consumers aren’t impacted.2. To update Deployment labels safely (rollout), patch the Deployment template so new Pods come up with correct labels, then let old pods terminate:

bash

kubectl patch deployment my-deploy -p '{"spec":{"template":{"metadata":{"labels":{"app":"frontend"}}}}}'
kubectl rollout status deployment/my-deploy
kubectl get pods --show-labels
kubectl get endpoints my-service  # should now show addresses

3. If you must patch Service temporarily:

bash

kubectl patch svc my-service -p '{"spec":{"selector":{"app":"frontend"}}}'

Post-fix validation:- Confirm endpoints populate:

bash

kubectl get endpoints my-service
kubectl get pods -l app=frontend
kubectl exec -it <debug-pod> -- wget -qO- http://my-service:80

Notes and best practices:- Use consistent label keys/values (e.g., app, component, tier).- Avoid using labels that change frequently; prefer immutable selectors like app and release.- Use readiness probes so only healthy Pods appear as endpoints.- Make changes via Deployment updates (rolling) to avoid downtime and validate rollout status before changing Service selectors.

MediumTechnical

41 practiced

Write a kubectl-friendly debugging checklist and a small helper (pseudo-code or kubectl alias) that an SRE can use to collect a standardized troubleshooting bundle when a pod fails: include pod describe, events, logs (current and previous), node info, and top resource consumers. The helper should minimize manual steps for on-call use.

Sample Answer

Checklist — quick kubectl-friendly debugging steps (run this in order):1. Identify failing pod(s): kubectl get pods -n NAMESPACE -o wide | grep -E "CrashLoop|Error|Evicted|Completed|0/" 2. Pod describe: kubectl describe pod POD -n NAMESPACE (look for Events, imagePull, OOMKilled)3. Events (cluster-level): kubectl get events -n NAMESPACE --sort-by='.lastTimestamp'4. Logs: kubectl logs POD -c CONTAINER -n NAMESPACE (current) and kubectl logs POD -c CONTAINER -n NAMESPACE --previous (previous)5. Node info: kubectl describe node NODE and kubectl get pods -o wide --all-namespaces --field-selector spec.nodeName=NODE6. Resource usage: kubectl top pod POD -n NAMESPACE and kubectl top node NODE (if metrics-server unavailable, check kubelet or monitoring)7. Check replica/owner: kubectl get ownerreferences -o yaml OR kubectl rollout status deploy/DEPLOYMENT -n NAMESPACE8. Capture related PVCs, events, and recent kube-apiserver errors if needed.

Helper: bash script to collect standardized bundle (save as collect_pod_bundle.sh)

bash

#!/usr/bin/env bash
# Usage: ./collect_pod_bundle.sh <namespace> <pod> [container]
set -euo pipefail
NS=$1; POD=$2; CONTAINER=${3:-}
OUTDIR="${PWD}/trouble-${NS}-${POD}-$(date +%Y%m%dT%H%M%S)"
mkdir -p "$OUTDIR"

kubectl get pod "$POD" -n "$NS" -o yaml > "$OUTDIR/pod.yaml"
kubectl describe pod "$POD" -n "$NS" > "$OUTDIR/pod.describe.txt"
kubectl get events -n "$NS" --sort-by='.lastTimestamp' > "$OUTDIR/events.txt"

if [ -n "$CONTAINER" ]; then
  kubectl logs "$POD" -c "$CONTAINER" -n "$NS" > "$OUTDIR/logs.current.txt" || true
  kubectl logs "$POD" -c "$CONTAINER" -n "$NS" --previous > "$OUTDIR/logs.previous.txt" || true
else
  kubectl logs "$POD" -n "$NS" > "$OUTDIR/logs.current.txt" || true
  kubectl logs "$POD" -n "$NS" --previous > "$OUTDIR/logs.previous.txt" || true
fi

NODE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}')
kubectl describe node "$NODE" > "$OUTDIR/node.describe.txt"
kubectl get pods --all-namespaces --field-selector spec.nodeName="$NODE" -o wide > "$OUTDIR/node.pods.txt"

kubectl top pod "$POD" -n "$NS" > "$OUTDIR/top.pod.txt" 2>/dev/null || echo "kubectl top unavailable" > "$OUTDIR/top.pod.txt"
kubectl top node "$NODE" > "$OUTDIR/top.node.txt" 2>/dev/null || echo "kubectl top unavailable" > "$OUTDIR/top.node.txt"

tar -czf "${OUTDIR}.tar.gz" -C "$(dirname "$OUTDIR")" "$(basename "$OUTDIR")"
echo "Bundle created: ${OUTDIR}.tar.gz"

Alias: add to .bashrc for quick use:

bash

alias kcollect='~/collect_pod_bundle.sh'

Why this helps:- Standardized outputs reduce cognitive load for on-call and make post-mortems easier.- Captures current + previous logs, events, node context, and resource tops for triage.- Script uses sensible defaults and tolerates missing previous logs or metrics-server.

Unlock Full Question Bank

Get access to hundreds of Kubernetes Troubleshooting and Observability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.