Kubernetes Architecture, Operations, and Troubleshooting Questions

For strong isolation and compliance requirements, choose cluster-per-tenant over namespace-based multi-tenancy whenever a tenant needs a genuinely separate blast radius (a compromised or noisy tenant must not be able to reach another tenant's control plane, kubelet, or node kernel) or needs to be independently certified against a specific compliance regime; use hardened namespace-based multi-tenancy for internally trusted tenants where utilization and onboarding speed matter more than a hard boundary. In practice, the strongest designs are hybrid: a small number of dedicated, hardened clusters for high-compliance tenants, plus a shared, heavily guarded cluster for everyone else.

Cluster-per-tenant vs. namespace-based

Implementing each control

Network isolation. Use a CNI (Container Network Interface, the plugin layer responsible for pod networking) that enforces NetworkPolicy, such as Calico or Cilium. Default-deny ingress and egress per tenant namespace, then explicitly allow the flows a tenant actually needs. Cilium's eBPF (extended Berkeley Packet Filter, a Linux kernel technology for programmable packet processing) data path additionally supports layer-7-aware policy, useful for restricting a tenant to specific HTTP paths or gRPC methods on a shared internal API, not just IP-and-port pairs.

RBAC boundaries. RBAC (Role-Based Access Control) is Kubernetes' native authorization model. Grant only namespaced Role/RoleBinding pairs to tenant users; block ClusterRole and cluster-admin bindings for tenant service accounts entirely, enforced by an admission policy (OPA Gatekeeper or Kyverno) rather than by convention, since convention alone does not survive a misconfigured pipeline.

Resource quotas. ResourceQuota per tenant namespace plus a LimitRange for per-pod defaults and maximums, as in the general multi-tenancy case. In a shared cluster, also set a PriorityClass per tenant tier so that, under real node pressure, preemption and the scheduler favor higher-tier tenants' pods over lower-tier ones instead of resolving contention arbitrarily. This is the fairness mechanism that matters even when every tenant is well inside its own quota: quotas cap each tenant individually, but they do nothing to arbitrate contention for genuinely scarce cluster-wide capacity during a spike, which is what PriorityClass-driven preemption is for.

Pod security. Enforce the Pod Security Admission controller (the built-in mechanism that replaced PodSecurityPolicy, which was deprecated in Kubernetes 1.21 and removed in 1.25) at the restricted level for tenant namespaces: this denies privileged containers, host namespaces, and most capabilities by default. Layer a curated seccomp (secure computing mode, a Linux kernel feature that filters which system calls a process may make) profile and an AppArmor profile (a Linux kernel security module that restricts, per program, which files, network access, and capabilities it may use via a loaded policy) on top for defense in depth beyond what Pod Security Admission alone checks.

Image scanning and supply chain. Require every image to come from a private registry, scanned in CI before it can be deployed, and signed. cosign (part of the sigstore project) is a common tool for signing; Notation is the current CLI under the CNCF Notary Project for the same purpose (the older "notary" v1 client is the legacy predecessor). An admission-time check, the built-in ImagePolicyWebhook controller, or a Gatekeeper/Kyverno policy, verifies the signature and blocks unsigned or unscanned images from ever being scheduled.

Runtime detection. Falco watches kernel syscalls, via eBPF or a kernel module, for suspicious behavior, such as a shell spawned inside a container that never spawns shells, or a write to a path that should be read-only, and can alert or block. Cilium Hubble provides complementary network-flow visibility if Cilium is the CNI.

Audit and logging. Enable the Kubernetes API server's audit log at a verbosity that captures at least every write and every RBAC-relevant read, ship it to a tamper-evident store (object storage with retention locking, or a SIEM, a Security Information and Event Management system) separate from the cluster itself, and tag every entry with tenant identity so a compliance review can reconstruct one tenant's activity without touching another's data.

Worked example: an image-scan policy gate

Suppose the CI pipeline's scanner returns two findings for a candidate image before it is allowed into the tenant cluster:

With a policy of "block on CVSS 7 or above," the first finding fails the build ($9.8 \geq 7$) and the image is never pushed to the registry the admission controller trusts; the second finding ($4.3 < 7$) does not block. This is the mechanism, not the specific numbers: the policy threshold, and what counts as "shipped" (a dev-only dependency that never reaches the runtime image should not gate a build the same way a runtime dependency does), are the actual design decisions a platform team has to make, and they should be written down as policy-as-code so the same rule applies whether one engineer or a thousand submit an image.

Doing this at scale, roughly 1,000 tenants

The mechanisms above do not change in kind as tenant count grows into the hundreds or low thousands; what changes is that every manual step becomes untenable. Chargeback reporting has to run as an automated nightly job against labeled usage rather than a person building a thousand dashboards; onboarding has to be a GitOps-templated namespace-plus-policy bundle rather than a runbook a human executes by hand; and audit-log volume at that scale needs its own retention and cost budget, since a SIEM ingesting per-tenant audit trails for 1,000 tenants is a meaningfully different cost line than for ten.

Trade-offs and pitfalls

• Namespace-based isolation can be hardened close to cluster-per-tenant strength, but "close to" is doing real work in that sentence: a kernel-level container escape still reaches every tenant on that node, a risk that simply does not exist in cluster-per-tenant.
• Runtime detection (Falco) and admission-time policy (Gatekeeper/Kyverno) are complementary, not substitutes: admission policy stops known-bad configurations before they run; runtime detection catches behavior that only manifests once a workload is executing, such as a legitimate-looking image that turns malicious after a dependency is compromised post-deployment.
• Treating "namespace vs. cluster per tenant" as a single cluster-wide decision misses that different tenants can warrant different answers; the hybrid model, a few hardened dedicated clusters plus one well-governed shared cluster, is usually a better fit than picking one model for every tenant.

flowchart LR
    Build[CI build] --> Scan[Image scan: CVSS gate]
    Scan -->|pass| Sign[Sign image: cosign / Notation]
    Scan -->|fail| Block[Build blocked]
    Sign --> Registry[Private registry]
    Registry --> Admission[Admission check: signature + policy]
    Admission -->|pass| Run[Pod scheduled to tenant namespace]
    Admission -->|fail| Reject[Pod creation rejected]
    Run --> Falco[Falco: runtime syscall monitoring]
    Falco --> Audit[Audit log + SIEM]

Liveness, readiness, and startup probes all ask whether a container is okay, but each answer drives a different Kubernetes action: a failing liveness probe gets the container restarted, a failing readiness probe gets the pod pulled out of Service traffic without touching the container at all, and a startup probe simply delays the other two until the app has had time to boot.

What each probe gates

Worked example: sizing a startup budget by workload archetype

What 'booting' means differs a lot by workload, and the startup probe has to be sized for the actual archetype, not guessed at: a machine learning (ML) inference service loading model weights into memory might need several minutes; a batch worker doing asynchronous Java Virtual Machine (JVM) warmup, classloading, and connection-pool initialization for an extract-transform-load (ETL) job might need under a minute; a stateless HTTP handler might be ready in under a second. Whichever number applies, it has to be encoded as periodSeconds times failureThreshold. Budgeting 5 minutes of startup headroom with a 10-second check interval for the ML case:
$10 \times 30 = 300\text{s} = 5\text{ min}$
means periodSeconds: 10 and failureThreshold: 30. Too tight in this calculation and the startup probe itself kills a healthy-but-slow container before it ever gets a chance to serve; too loose, and a genuinely stuck container burns minutes before anything reacts.

Trade-offs and pitfalls

• Swapping liveness and readiness is the classic mistake: pointing liveness at a deep dependency check (database reachability) means a transient database blip restarts every application pod at once instead of simply pulling them from rotation, turning a recoverable dependency issue into a self-inflicted outage.
• Using liveness as a substitute for a startup probe on a slow-booting app causes a restart loop before the app ever finishes initializing, since the container never survives long enough to pass a liveness check tuned for steady-state behavior.
• A readiness probe that is too permissive, common with the JVM-async pattern where the process starts accepting connections before its dependency pools are actually warm, reports the pod as ready while real requests still fail; that failure mode never shows up as a probe failure at all, only as user-visible errors.

Direct answer

Increased kube-apiserver latency almost always traces to one of three places: etcd itself (disk or network bound), a slow admission webhook sitting in the request path, or the apiserver process running short of CPU or memory (including its own request-concurrency limiting kicking in). The fastest way to tell them apart is to look at where time is spent inside a single slow request, etcd round trip versus admission call versus everything else, rather than guessing from symptoms alone.

Structured elaboration

Metrics to pull first

Triage order

1. Scope it: slice apiserver_request_duration_seconds by resource and verb. If only one resource type or client is slow, an etcd-wide or apiserver-wide problem is unlikely; look at that resource's admission webhooks first.
2. Check APF rejection reasons: apiserver_flowcontrol_rejected_requests_total labeled reason="queue-full" or reason="concurrency-limit" means the server is intentionally shedding load under its configured priority levels. The 429 responses clients see in that case are the mechanism doing its job, not a hidden bug; the real bottleneck is upstream (etcd or CPU), not APF itself.
3. If the slowdown is global, every resource, every client, check etcd's disk metrics and leader stability before touching the apiserver. A slow etcd backend shows up as apiserver latency because every write and every quorum read waits on it.
4. If etcd looks healthy but apiserver CPU or memory is pegged, it is apiserver resource pressure.

Remedial actions per root cause

• etcd bound: faster disks (etcd is fsync-latency sensitive, not throughput sensitive), a disk dedicated to etcd separate from other I/O, defragmentation, and checking etcd_mvcc_db_total_size_in_bytes (a bloated database slows every commit). Reduce write volume from noisy controllers before adding etcd members: more members raise the replication cost of every write, they do not spread load the way a read replica would.
• Admission webhook bound: tune webhook timeoutSeconds and failurePolicy carefully, scale the webhook backend, and reconsider whether the check belongs in a webhook at all versus static OpenAPI schema validation. A webhook earns its cost when the rule needs data outside the request object, cross-field logic, or an external lookup; anything expressible as a plain schema constraint should live in the CRD's (Custom Resource Definition's) OpenAPI validation instead, since that costs nothing at admission time.
• apiserver resource bound: unlike etcd, kube-apiserver is stateless, so horizontally scaling it (more apiserver replicas behind the control-plane load balancer) is a legitimate, common fix, not a workaround. Also check audit log verbosity and watch cardinality; a controller opening many broad watches is a frequent, overlooked CPU driver.

Worked example (concept, not a fabricated benchmark)

Suppose apiserver_request_duration_seconds p99 for PATCH pods is elevated but p99 for every other verb and resource is flat. That shape alone rules out an etcd-wide or apiserver-wide bottleneck, because both would show up across every resource type, and points at something specific to pod patches, almost always a mutating webhook registered on Pods (for example a sidecar injector). Confirming it takes one more step: check apiserver_admission_webhook_admission_duration_seconds filtered to that webhook's name. A rising p99 there, correlated with the PATCH pods latency, closes the loop without needing to touch etcd or CPU metrics at all.

Trade-offs and pitfalls

• Do not disable webhooks blind as a first move. failurePolicy: Ignore on a security-relevant mutating webhook (one that injects a sidecar or a required label, for example) can silently change what gets admitted, not just how fast.
• Adding etcd members to "spread the load" is a common but wrong instinct: every additional voting member adds replication overhead to every write. It improves fault tolerance, not throughput.
• 429 responses from APF are a symptom of an upstream bottleneck, not a target to eliminate by raising limits. Raising a priority level's concurrency share without fixing the underlying etcd or CPU constraint just moves where the queue backs up.

A safe restart uses kubectl rollout restart, which recreates pods through the normal RollingUpdate strategy rather than deleting them directly, so the same availability guarantees that protect a routine deployment protect the restart too.

Commands

Trigger and watch a rolling restart:

kubectl rollout restart deployment my-app -n prod
kubectl rollout status deployment my-app -n prod --watch

View rollout history and inspect a specific revision:

kubectl rollout history deployment my-app -n prod
kubectl rollout history deployment my-app -n prod --revision=3

Roll back:

kubectl rollout undo deployment my-app -n prod --to-revision=3
kubectl rollout status deployment my-app -n prod

Ship an image change with a recorded reason (the --record flag some older references use for this is deprecated; annotate explicitly instead):

kubectl set image deployment/my-app my-app=registry/app:1.2.3 -n prod
kubectl annotate deployment my-app kubernetes.io/change-cause="bump to 1.2.3, ticket OPS-441" --overwrite -n prod

What keeps a restart from becoming a cascading failure

• RollingUpdate parameters: maxUnavailable and maxSurge (both default to 25% of desired replicas) bound how many old pods can be down and how many extra new pods can exist at once. For a consumer-facing service, a conservative setting (for example maxUnavailable: 0, maxSurge: 1) never drops capacity below the current replica count during the restart, at the cost of briefly running more pods than the steady-state count.
• Readiness probes: a Service only sends traffic to pods that pass their readiness probe, so a newly restarted pod that's still initializing doesn't receive requests it can't yet handle. This is the single biggest lever against a restart-induced error spike; without a readiness probe, the rollout has no signal that a "new" pod is actually ready and can start routing traffic to it immediately.
• PodDisruptionBudget (PDB): guarantees a minimum number (or percentage) of replicas stay available throughout the restart, independent of the Deployment's own maxUnavailable setting, which matters when other voluntary disruptions (a node drain, a cluster upgrade) happen to overlap with the restart window.
• Graceful shutdown: a preStop hook plus a terminationGracePeriodSeconds long enough for in-flight requests to finish, combined with the Service removing the pod's endpoint before the container actually stops, avoids dropping requests that were already in progress when the restart began.
• Staged rollout for risk-sensitive services: restarting (or deploying) to a small subset first, watching error rate and latency, then proceeding, catches a bad new revision before it reaches full traffic; this is a general staged-rollout practice, not a specific traffic-splitting mechanism (traffic-splitting techniques like weighted canary routing are a load-balancing/ingress-layer concern, not something the Deployment object itself provides).

Trade-offs and pitfalls

• kubectl rollout restart only recreates pods; it does not change the Deployment's spec, so rollout history records it as a new revision with the same template, which is easy to forget when later trying to undo your way back past a restart that changed nothing.
• Setting maxUnavailable: 0 guarantees no capacity loss but requires enough spare cluster capacity for maxSurge extra pods to schedule; on a tightly packed cluster this can leave the rollout stuck Pending on the surge pods instead of proceeding.
• A rollback only restores the pod template (image, env, resource requests, and so on). If the Deployment reads a ConfigMap or Secret by a fixed name and that ConfigMap was edited in place rather than replaced with a new name or hash-suffixed name, rolling the Deployment back does not restore the old configuration content, only the old pod template pointing at the same (already-mutated) ConfigMap. This is the most common way a rollback fails to actually roll back.
• The same gap applies to a PersistentVolumeClaim (PVC): a Deployment's rollback restores the pod template's volume mount references, not the data on the volume itself. If the new version wrote a schema migration or otherwise mutated data in place on that volume, rolling the Deployment back gives you the old code pointing at already-changed data, not the old data. Anything stateful needs its own restore path (a volume snapshot or application-level backup) alongside the Deployment rollback, not instead of thinking about it separately.

Kubernetes splits cluster management into a control plane, which decides and records desired state, and worker nodes, which run it. The control plane's core pieces are the kube-apiserver (the front door that validates and serves every request), etcd (the single source of truth for cluster state), the scheduler (decides which node a new pod lands on), the controller-manager (a bundle of reconciliation loops that push actual state toward desired state), and, on cloud-hosted clusters, the cloud-controller-manager (the seam that keeps cloud-specific logic like load balancer provisioning out of core Kubernetes). Every one of these follows the same pattern: watch the API server for objects it cares about, and reconcile until observed state matches spec.

Control plane components

flowchart TD
    Client[kubectl / clients] --> API[kube-apiserver]
    API --> ETCD[(etcd)]
    Sched[kube-scheduler] -->|watch unscheduled pods, write bindings| API
    CM[controller-manager] -->|watch + reconcile| API
    CCM[cloud-controller-manager] --> API
    CCM -->|provision LB, routes| Cloud[Cloud provider API]
    API --> Kubelet[kubelet, per node]
    Kubelet --> CRI[container runtime, via CRI]

The API server's gatekeeping

Every request passes through three stages before it touches etcd: authentication (who are you: client certificate, bearer token, or an external identity provider via OIDC), authorization (are you allowed: almost always Role-Based Access Control, RBAC, checking your identity against Roles and RoleBindings), and admission (should this specific object be allowed or modified: built-in admission controllers plus optional mutating and validating admission webhooks, which is also the mechanism behind things like automatic sidecar injection). A gap in any one of the three shows up as a very different symptom: authentication failures look like connection refusals, authorization failures return a 403, and admission failures reject an otherwise well-formed object with a specific rejection reason from the webhook or controller.

Worker-node components: kubelet, kube-proxy, and the container runtime

The control plane decides; the node executes. The kubelet is the agent on every node that watches the API server for pods assigned to that node and drives the container lifecycle through the CRI (Container Runtime Interface), a plugin boundary that lets Kubernetes talk to any compliant runtime (containerd and CRI-O are the common choices today; Docker itself was removed as a supported CRI implementation in Kubernetes 1.24). kube-proxy implements the networking side of a Service on each node; the mechanics of that (iptables, IPVS, or the newer nftables backend) belong to Service and networking questions rather than control-plane architecture, but it is worth knowing kube-proxy is a node component, not a control-plane one.

Stepping back, the reason Kubernetes is built this way rather than as a single monolithic scheduler is the reconciliation model itself: every component only has to compare desired state to observed state and take one corrective step, repeatedly, which is what makes the system self-healing and declarative rather than a one-shot deployment tool.

Worked example: reasoning about an etcd quorum failure

etcd tolerates the loss of a minority of its members because it uses a majority-vote (Raft) protocol; for a cluster of $n$ members it needs:
$\text{quorum} = \lfloor n/2 \rfloor + 1$
For the common 5-member etcd cluster:
$\lfloor 5/2 \rfloor + 1 = 2 + 1 = 3$
so it tolerates 2 simultaneous member failures while still accepting writes. This is also why an operator should never round a fault-tolerance target up to an even member count: a 4-member cluster still only tolerates 1 failure (quorum is 3), the same as a 3-member cluster, but pays for a fourth voter with no extra fault tolerance. Two metrics tell you this is happening before it becomes an outage: a sustained rise in etcd_server_leader_changes_seen_total (frequent leader changes usually mean the disk cannot keep up with etcd's Raft heartbeat interval) and a simultaneous rise in apiserver_request_duration_seconds p99, since every write now waits on a less stable etcd leader.

Trade-offs and pitfalls

• Stacked etcd (co-located with control-plane nodes) is simpler to run but ties etcd's failure domain to the same nodes serving the API; an external etcd cluster isolates that blast radius at the cost of more infrastructure to operate.
• On managed clusters (Amazon Elastic Kubernetes Service, Google Kubernetes Engine, Azure Kubernetes Service) the provider hides and operates the control plane entirely; you cannot inspect etcd directly, so day-to-day monitoring shifts to the provider's exposed control-plane metrics and SLA rather than self-run dashboards.
• cloud-controller-manager problems are easy to misdiagnose as networking bugs: a Service stuck in <pending> for its external address is very often a cloud-controller-manager or cloud-API quota issue, not a kube-proxy or CNI (Container Network Interface) problem, so check its logs before chasing the wrong component.