Kubernetes Architecture, Operations, and Troubleshooting Questions

Liveness, readiness, and startup probes all ask whether a container is okay, but each answer drives a different Kubernetes action: a failing liveness probe gets the container restarted, a failing readiness probe gets the pod pulled out of Service traffic without touching the container at all, and a startup probe simply delays the other two until the app has had time to boot.

What each probe gates

Worked example: sizing a startup budget by workload archetype

What 'booting' means differs a lot by workload, and the startup probe has to be sized for the actual archetype, not guessed at: a machine learning (ML) inference service loading model weights into memory might need several minutes; a batch worker doing asynchronous Java Virtual Machine (JVM) warmup, classloading, and connection-pool initialization for an extract-transform-load (ETL) job might need under a minute; a stateless HTTP handler might be ready in under a second. Whichever number applies, it has to be encoded as periodSeconds times failureThreshold. Budgeting 5 minutes of startup headroom with a 10-second check interval for the ML case:
$10 \times 30 = 300\text{s} = 5\text{ min}$
means periodSeconds: 10 and failureThreshold: 30. Too tight in this calculation and the startup probe itself kills a healthy-but-slow container before it ever gets a chance to serve; too loose, and a genuinely stuck container burns minutes before anything reacts.

Trade-offs and pitfalls

• Swapping liveness and readiness is the classic mistake: pointing liveness at a deep dependency check (database reachability) means a transient database blip restarts every application pod at once instead of simply pulling them from rotation, turning a recoverable dependency issue into a self-inflicted outage.
• Using liveness as a substitute for a startup probe on a slow-booting app causes a restart loop before the app ever finishes initializing, since the container never survives long enough to pass a liveness check tuned for steady-state behavior.
• A readiness probe that is too permissive, common with the JVM-async pattern where the process starts accepting connections before its dependency pools are actually warm, reports the pod as ready while real requests still fail; that failure mode never shows up as a probe failure at all, only as user-visible errors.

ImagePullBackOff is the pod status shown when the kubelet has repeatedly failed to pull a container's image and is now backing off further attempts; like CrashLoopBackOff, it names the symptom, not the cause. The four usual causes are registry authentication, DNS resolution to the registry, a wrong image name or tag, and network reachability from the node to the registry.

Causes and their signatures

Representative event text you'd actually see:

Failed to pull image "myrepo/app:v2": rpc error: code = NotFound desc = failed to pull and unpack image "myrepo/app:v2": failed to resolve reference "myrepo/app:v2": myrepo/app:v2: not found

Failed to pull image "123456789012.dkr.ecr.us-east-1.amazonaws.com/app:v1": rpc error: code = Unknown desc = failed to authorize: 401 Unauthorized

Step-by-step: cluster, node, or registry?

1. Read events first.

kubectl describe pod <pod> -n <ns>
kubectl get events -n <ns> --sort-by=.metadata.creationTimestamp

2. Check the pull secret and service account actually being used.

kubectl get sa <sa> -n <ns> -o yaml
kubectl get secret <secret> -n <ns> -o yaml

3. Test from inside the cluster (isolates cluster-level DNS/NetworkPolicy problems from node-level ones):

kubectl run -i --rm debug --image=alpine --restart=Never -- sh
# inside: apk add --no-cache curl bind-tools; nslookup registry.example.com; curl -v https://registry.example.com/v2/

4. Test from the node itself. Since dockershim was removed in Kubernetes 1.24, the node-level container runtime on essentially every current cluster is containerd (or CRI-O), so the right tool is crictl, not docker. docker pull only applies if you're intentionally still running Mirantis cri-dockerd or looking at a pre-1.24 node.

kubectl get pod <pod> -o wide -n <ns>
# on the node:
sudo crictl pull <image>
sudo journalctl -u kubelet -e

5. Decide scope:
   • Registry-related: the error is 401/403/manifest unknown both from the debug pod and from the node directly.
   • Cluster-level: the in-cluster debug pod can't resolve or reach the registry, but the node itself can (curl succeeds from the node); points at CoreDNS or a NetworkPolicy blocking egress.
   • Node-level: only some nodes fail; others pull the same image fine. Points at a node-specific proxy, MTU, or firewall rule.

Two variants worth naming explicitly

Registry rate limiting across many pods at once. If ImagePullBackOff appears on many unrelated pods across the cluster at roughly the same time, rather than one pod, suspect the registry itself: unauthenticated pulls against a public registry are commonly rate-limited (429 Too Many Requests or an equivalent 401 once the anonymous quota is spent), and every node hitting that same registry trips it simultaneously. The fix is authenticated pulls, a pull-through cache/mirror, or spreading pulls out over time, not chasing one pod's config.

ECR token expiry as a time-correlated failure. On EKS Amazon Elastic Kubernetes Service, if pulls are authenticated with a static, manually-refreshed token stored in an imagePullSecret rather than through IRSA (IAM Roles for Service Accounts), that token is only valid for 12 hours. Pulls succeed for a while and then start failing cluster-wide at almost exactly the token's issuance time plus 12 hours, which is the tell that distinguishes this from a one-off misconfiguration: correlate the failure's onset timestamp against when the credential was last refreshed. The fix is to stop relying on a static Secret and either use IRSA so kubelet-side credential retrieval is automatic, or run a credential-refresh mechanism (such as the ECR credential helper or a scheduled Secret-rotation job) instead of a manually maintained token.

Trade-offs and pitfalls

• Don't assume "it's the registry" just because the error message is generic; a node-level DNS or proxy issue can produce an error that looks identical to a registry outage.
• A fix that works from a debug pod but not from the node (or vice versa) is itself diagnostic information: it tells you the failure is layer-specific, not systemic.
• Long-lived static pull secrets are an operational trap even when they work today; anything time-boxed (ECR tokens, expiring service-account tokens) will eventually fail in a way that looks like a random new bug unless the expiry is tracked explicitly.

A pod is the smallest unit Kubernetes schedules and manages: one or more containers that always run together on the same node, sharing a network namespace and, optionally, storage volumes. It is the unit, rather than the individual container, because the kubelet, the scheduler, and every controller reason about placement, restarts, and networking at that granularity, not below it.

Why the pod, not the container, is the atomic unit

• Co-scheduling guarantee: every container in a pod is placed on the same node and started, stopped, and restarted as a unit by the kubelet.
• One IP per pod: regardless of how many containers it holds, a pod gets exactly one IP address; Services and DNS resolve to that pod IP, not to an individual container.
• Shared network namespace: containers in a pod talk to each other over localhost, and must not claim conflicting ports, because they share the same network namespace.
• Shared volumes: volumes are declared once at the pod spec level and can be mounted into more than one container, which is how a sidecar can read or write files the main container produces without a network hop.

When to run more than one container in a pod

• Sidecar pattern: a helper process that shares the fate and resources of the main container, most commonly a log shipper, a proxy, or a metrics exporter.
• Init containers: containers that run to completion, in order, before the main containers start, typically for one-time setup like a schema migration or config templating.
• Native sidecar containers: since the SidecarContainers feature became enabled by default in Kubernetes 1.29 (stable as of 1.33), you can declare a sidecar under initContainers with restartPolicy: Always. Kubernetes then starts it before the main container, keeps it running for the pod's whole life, and stops it after the main container on shutdown, which fixes the older ordering problem where a plain extra container might not be ready before the app started, or might outlive a completed Job's main container instead of shutting down with it.

A small example, an app container and a log-forwarding sidecar sharing a volume instead of a network call:

containers:
  - name: app
    image: myapp:1.4
    volumeMounts:
      - name: logs
        mountPath: /var/log/app
  - name: log-shipper
    image: fluent-bit:latest
    volumeMounts:
      - name: logs
        mountPath: /var/log/app
        readOnly: true
volumes:
  - name: logs
    emptyDir: {}

The app writes to /var/log/app; the sidecar tails the same path through the shared emptyDir volume, no network hop involved.

Worked example: what a failing sidecar looks like

If the log-shipper container above enters a crash loop while app keeps running fine, the two containers' fates are independent even though they share a pod: app keeps serving traffic, but the log pipeline drops. This is visible directly in the pod list, where the READY column reports containers-ready over containers-total, not a single number:

NAME                     READY   STATUS    RESTARTS      AGE
myapp-6d947f8db8-x2z1p   1/2     Running   4 (30s ago)   6m

Here 1/2 means only one of the pod's two containers is passing its readiness state, and the restart count of 4 belongs to the sidecar, not to app.

Trade-offs and pitfalls

• Co-location couples lifecycle: a sidecar cannot be scaled independently of the app the way a separate Deployment could be. Native sidecar containers loosen this slightly by giving the sidecar its own restart behavior, but it is still tied to the pod's schedule and node.
• Every container in the pod counts toward the pod's total resource footprint; forgetting the sidecar's own requests and limits when sizing the node is a common under-provisioning mistake.
• Splitting a component into a separate pod (plus a Service) regains independent scaling and blast-radius isolation, at the cost of a network hop and losing the localhost and shared-volume convenience. Reach for separate pods when the components genuinely differ in scaling or failure characteristics, for example a stateless API versus a shared cache, rather than defaulting to a sidecar for anything that happens to run alongside the main app.

NetworkPolicy is a namespaced Kubernetes object that restricts which pods can talk to which other pods (and, via egress rules, which external destinations pods can reach), solving east-west traffic control inside the cluster: without it, any pod can reach any other pod by default, which is a large blast radius for a compromised or misconfigured workload.

The enforcement point that isn't the API server

Creating a NetworkPolicy object only stores intent in etcd via the API server, the same as any other Kubernetes object; the API server itself does not inspect or block a single packet. Enforcement is entirely the responsibility of the CNI (container network interface) plugin running on each node, and only if that plugin actually implements NetworkPolicy support. Calico, Cilium, and several others do; a purely overlay-focused CNI like plain Flannel historically does not. Applying a NetworkPolicy on a cluster whose CNI doesn't enforce it is a silent no-op: kubectl get networkpolicy shows the object as created, but traffic flows exactly as if it didn't exist. Confirming the CNI's NetworkPolicy support is a prerequisite, not an afterthought, before relying on this mechanism for anything security-relevant.

Expressing ingress and egress

A policy selects pods with podSelector, and, once selected, ingress and egress traffic is governed independently:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-policy
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector: { matchLabels: { role: frontend } }
    ports:
    - port: 80
  egress:
  - to:
    - namespaceSelector: { matchLabels: { env: services } }
    ports:
    - port: 5432

This says: pods labeled app: web accept ingress only from pods labeled role: frontend on port 80, and may only send egress to pods in namespaces labeled env: services on port 5432. Everything else in that direction is implicitly denied once any rule for that direction exists.

Selectors and namespaceSelector

• podSelector inside from/to matches pods by label within the same namespace as the rule's target namespace context.
• namespaceSelector matches whole namespaces by label; commonly combined with podSelector in the same from/to entry to mean "these specific pods, but only in namespaces with this label," rather than either alone.
• An empty podSelector: {} at the top level of the spec selects every pod in the namespace; an empty selector inside a from/to entry means "all pods" for that source/destination.

Default behavior

With no NetworkPolicy selecting a pod at all, every ingress and egress connection to and from it is allowed, no restriction whatsoever. The moment at least one policy selects a pod and declares a policyTypes direction (say, Ingress), that direction flips to default-deny for that pod: only traffic matching an explicit rule is allowed, and everything else in that direction is dropped. This is why a common, deliberate starting point is a namespace-wide default-deny policy (a policy selecting all pods with empty ingress/egress rules) followed by narrow allow rules added on top, rather than trying to enumerate every exception against an open-by-default baseline.

Trade-offs and pitfalls

• NetworkPolicies are additive across multiple policies selecting the same pod (a union of allowed traffic, not an intersection), which means adding a second, broader policy to a namespace can unintentionally widen access that a narrower first policy had restricted.
• Egress policies frequently get overlooked because ingress is the more obvious security concern, but an over-permissive default (no egress policy at all) lets a compromised pod exfiltrate data or reach internal services it has no business reaching; DNS resolution in particular is easy to break by an egress default-deny policy that forgets to allow port 53 to the cluster's DNS service.
• Because enforcement lives in the CNI, NetworkPolicy behavior (which selector combinations are supported, whether egress is supported at all, performance under many rules) varies by CNI choice; a policy that works correctly in one cluster is not guaranteed to behave identically after a CNI migration.

CrashLoopBackOff means the container keeps exiting and the kubelet is backing off its restart attempts with exponential delay (10s, 20s, 40s, up to a 5-minute cap). It is a symptom, not a root cause: something inside the container is exiting, or something outside it is killing the container, and the job is to figure out which.

Step-by-step diagnosis

1. Describe the pod first. This shows the Events list and the container's last termination reason before you even look at logs.

kubectl describe pod <pod> -n <ns>

2. Read current and previous logs. The -p (previous) flag is what most people forget, and it's the only way to see output from the crashed instance once it has already restarted.

kubectl logs <pod> -c <container> -n <ns>
kubectl logs <pod> -c <container> -p -n <ns>

3. Check the exit code, visible in describe's Last State block.

4. Separate "app crashed" from "kubelet killed it." A liveness probe failure produces its own event and a different signature than an application-level crash:

Containers:
  app:
    State:      Waiting
      Reason:   CrashLoopBackOff
    Last State: Terminated
      Reason:   Error
      Exit Code: 1
    Restart Count: 5
Events:
  Warning  Unhealthy  90s (x3 over 3m)   kubelet  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": dial tcp 10.1.2.3:8080: connect: connection refused
  Warning  BackOff    30s (x12 over 6m)  kubelet  Back-off restarting failed container

If the events show Liveness probe failed right before the restart, the app may actually be starting fine but too slowly, or the probe is pointed at the wrong port or path, and kubelet is killing a container that would otherwise have recovered.
5. Check image and config next, since a large share of CrashLoopBackOff cases are not code bugs: wrong environment variable, a ConfigMap/Secret key that doesn't exist, a missing volume mount, or an image tag that doesn't match what's expected.

Reproduce and test a fix safely

• Recreate the same image/env in a disposable pod without a restart policy fighting you:

kubectl run debug --image=<image> --restart=Never -n dev --env="KEY=value" --command -- sleep 3600
kubectl exec -it debug -n dev -- /bin/sh

• If you suspect the liveness probe is the culprit, don't leave it disabled permanently; temporarily loosen initialDelaySeconds or failureThreshold in a copy of the manifest in a non-prod namespace, confirm the app actually stabilizes, then fix the probe config for real rather than shipping it disabled.
• Once you have a fix, patch and roll it out normally rather than editing the live object by hand:

kubectl set image deploy/<d> <c>=<image>:<new-tag> -n <ns>
kubectl rollout status deploy/<d> -n <ns>

• Capture the failing pod's YAML before you touch anything (kubectl get pod <pod> -o yaml), since the evidence disappears once the pod is replaced.

Trade-offs and pitfalls

• Don't "fix" a crash loop by disabling the liveness probe or setting restartPolicy more leniently; that hides the symptom and turns a visible failure into a silent one.
• Exit code 137 is a strong hint but not proof of OOMKilled; always confirm via the Reason field in describe, since a probe-triggered kill or an external kill -9 produces the same code.
• If a pod never produces logs at all before crashing, the failure is likely before your application's logging even initializes (missing dependency, bad entrypoint, immediate segfault); at that point kubectl logs -p is empty and you have to fall back to the exit code, describe events, and possibly running the image manually to see stdout that never reached the log driver.