Kubernetes Architecture, Operations, and Troubleshooting Questions

Direct answer

etcd is a CP (consistency-favoring) system built on Raft consensus: a write only commits once a strict majority of voting members acknowledge it. When a network partition splits the cluster, at most one side can hold a majority, and that side may keep serving; everything else must refuse writes to avoid split brain. The immediate job in this incident is not to restore write availability by any means necessary, it is to work out which side, if any, already holds a majority and point API servers only at that side. You touch cluster membership only if no side has a majority, and even then you rebuild from a snapshot rather than force a quorum change against live, disconnected data.

Structured elaboration

Step 1: read state before acting, on every reachable member

etcdctl endpoint status --cluster -w table
etcdctl member list -w table
etcdctl endpoint health --cluster

(etcdctl defaults to the v3 API since etcd 3.4, so exporting ETCDCTL_API=3 is no longer required, though a runbook that still sets it is harmless.)

Illustrative shape of the output you are reading for:

+----------------+------------------+---------+-----------+-----------+
|    ENDPOINT    |        ID        | VERSION | IS LEADER | RAFT TERM |
+----------------+------------------+---------+-----------+-----------+
| 10.0.1.10:2379 | 8211f1d0f64f3269 |  3.5.x  |   true    |    42     |
| 10.0.1.11:2379 | 91bc3c398fb3c146 |  3.5.x  |   false   |    42     |
| 10.0.2.10:2379 |   unreachable    |    -    |     -     |     -     |
+----------------+------------------+---------+-----------+-----------+

A member showing unreachable, or a raft term that keeps climbing between polls, marks the partition boundary and tells you whether the reachable side is stably elected or still thrashing.

Step 2: do the quorum arithmetic before deciding anything

For an n-member cluster, majority = floor(n/2) + 1. Compare the number of members reachable from a candidate side against that number.

• If one side already meets majority: that side is the live cluster. Repoint kube-apiserver's --etcd-servers at that side only. Do not touch membership. Let a few election cycles settle, then resume writes.
• If no side meets majority: the cluster is down for writes everywhere. Two paths remain: repair the network so the majority side becomes reachable again (fastest, no data risk), or, if the network cannot be fixed quickly, restore a new cluster from the most recent consistent snapshot taken from the member with the highest committed index.

Step 3: membership commands are for a live cluster, not a way to resurrect a dead one

etcdctl member remove / member add only make sense issued against a client already talking to a live majority. Running them from an isolated minority does not restore quorum, it creates a second, diverging cluster that will conflict with the real majority once the network heals. If you must rebuild, restore the snapshot into a brand-new cluster ID and repoint the API servers; do not try to vote in new members from a minority island.

Step 4: rejoin recovered nodes as learners, not straight back in as voters

Since etcd 3.4, etcdctl member add <name> --learner adds a non-voting member that replicates the log without changing the size of the quorum; once etcdctl member status shows it has caught up, etcdctl member promote <id> makes it a voter. This is also the mechanism kubeadm uses to safely rejoin a control-plane node's etcd member today. Adding a recovered node straight back in as a voter while it is still behind is what turns a partial outage into a full quorum loss, because the quorum denominator grows before the member can contribute a useful vote.

flowchart TB
  subgraph RegionA["Region A: 3 members, has majority"]
    A1[etcd-1]
    A2[etcd-2]
    A3[etcd-3]
  end
  subgraph RegionB["Region B: 2 members, no majority"]
    B1[etcd-4]
    B2[etcd-5]
  end
  RegionA -.network partition.- RegionB

Worked example (quorum arithmetic, not a timing claim)

Take the scenario as posed: a 5-member cluster split by a region-level partition, 3 members in Region A and 2 in Region B.

$\text{majority} = \lfloor 5/2 \rfloor + 1 = 3$

Region A has 3 reachable members, meeting the majority threshold, so Region A is the live cluster and can keep serving writes. Region B has only 2, below majority, and correctly refuses writes: those are the API servers "failing writes" in the prompt. The fix is to repoint them at Region A's endpoints, not to force anything in Region B.

Contrast this with the related but distinct failure the note asks to fold in: instead of a network partition, 2 of the 3 voting members in a 3-member cluster actually crash (disk failure, nodes terminated).

$\text{majority} = \lfloor 3/2 \rfloor + 1 = 2$

Only 1 member remains, below the majority of 2, so no side anywhere has quorum; every remaining node refuses writes. This is strictly worse than a partition with a majority on one side, because there is no "point the API server elsewhere" fix available. You must either bring back one of the failed members with its original data disk intact (fastest, no data loss) or restore the newest snapshot into a fresh cluster (which loses anything written after that snapshot).

Trade-offs and pitfalls

• The most common mistake under pressure is forcing member remove / member add from the minority side. It does not restore quorum; it manufactures a diverging cluster that corrupts state once connectivity returns.
• Restoring from snapshot is safe but lossy for anything committed after the snapshot; treat it as a last resort, not the default response to any quorum event.
• Long term, do not stretch one etcd cluster's voting membership across regions with materially different network reliability. Keep the voting majority co-located in the most reliable region (odd counts: 3, 5, 7) and use non-voting learners in remote regions as warm standbys, or run fully separate regional control planes joined by a higher-level coordination layer instead of one cross-region etcd.
• Automate and rehearse snapshot backups and restore drills. An incident is the wrong time to discover the only backup is stale or the restore runbook has never actually been run.

Pods are the runtime unit; everything else here is a controller deciding how many pods to run and how to replace them. A ReplicaSet just keeps N identical pods alive. A Deployment wraps a ReplicaSet with declarative rolling updates and rollback history. A StatefulSet keeps stable per-replica identity and storage for workloads that care which replica they are. A DaemonSet guarantees exactly one pod per matching node. Job and CronJob run pods to completion, once or on a schedule, rather than keeping them running indefinitely.

Comparison

Worked example: rolling-update arithmetic

For a Deployment with replicas: 10, maxSurge: 25%, and maxUnavailable: 25%, Kubernetes rounds surge up and unavailable down:
$\text{surge} = \lceil 10 \times 0.25 \rceil = 3$
$\text{unavailable} = \lfloor 10 \times 0.25 \rfloor = 2$
So at the busiest point of the rollout the Deployment can briefly run up to 13 pods (10 plus 3 surge) while guaranteeing at least 8 stay available (10 minus 2 unavailable). This asymmetric rounding is why setting surge and unavailable to the same percentage still lets the total pod count grow slightly during a rollout instead of staying flat.

DaemonSets interact with node taints in a way that is easy to miss: they commonly carry an explicit toleration for the control-plane taint (conventionally node-role.kubernetes.io/control-plane:NoSchedule) so node-level agents like a CNI plugin or kube-proxy itself still run on control-plane nodes that ordinary application pods are excluded from. That toleration is a manifest choice the DaemonSet author makes, not automatic behavior.

Trade-offs and pitfalls

• Choosing Deployment for a workload that actually needs stable identity (a small quorum service, a primary-replica database) forces the team to rebuild that identity logic in application code; StatefulSet gives it for free, at the cost of slower, ordered rollouts.
• A stuck rollout is usually a readiness-probe failure hiding behind the surge and unavailable math: if new pods never pass readiness, the Deployment can stall indefinitely partway through, with old and new pods coexisting. Check pod events and readiness status before assuming a controller bug.
• A PodDisruptionBudget set too strictly (for example, minAvailable equal to the full replica count) can make a Deployment's own rolling update unable to proceed, because the update's own temporary unavailability trips the budget meant to protect against unrelated disruptions.
• DaemonSets bypass normal replica-count thinking: adding nodes silently adds pods and cost. Forgetting a DaemonSet exists across a large node pool is a common source of untracked resource consumption.

A single well-governed multi-tenant cluster is usually the right starting point for internal teams: namespaces provide the boundary, RBAC (Role-Based Access Control) scopes who can act inside it, ResourceQuota and LimitRange keep one team's usage from starving another, and NetworkPolicy restricts which pods can talk to which. Move a team to its own dedicated cluster only when a specific pressure, compliance, blast radius, or a competing upgrade cadence, makes shared governance the wrong trade, not as a default posture.

What a namespace actually scopes

This decides where every other control applies. A namespace is a logical partition inside one cluster used to scope names and access, not a security boundary by itself: pods in different namespaces still share the same kernel, kubelet, and node pool unless further isolation is added.

Getting this table wrong is a common onboarding mistake: teams write a NetworkPolicy assuming it also restricts node-level traffic, or expect a namespaced RBAC Role to grant node access, when Node is cluster-scoped and untouched by namespace-level RBAC.

Building the tenant boundary

1. Namespaces: one namespace per team plus separate namespaces for shared platform services (ingress controller, logging agents, CI/CD runners). Avoid namespace-per-project-per-team sprawl unless its lifecycle is also automated; an unmanaged namespace is an onboarding shortcut that becomes an offboarding liability.
2. RBAC: define a small set of reusable Role templates (namespace-admin, developer, read-only) and bind them via RoleBinding to groups from an identity provider (an external system, typically integrated via OpenID Connect, OIDC, that authenticates users and asserts group membership) rather than to individual users, so team-membership changes never require touching Kubernetes RBAC directly. Reserve ClusterRole grants for the platform team.
3. ResourceQuota and LimitRange: cap each namespace's aggregate CPU/memory (ResourceQuota) and set sane per-pod defaults and maximums (LimitRange) so one team cannot silently consume the whole node pool. The exact admission-time interaction between the two is its own deep mechanism; the summary a platform designer needs here is that LimitRange fills in and bounds individual pods, while ResourceQuota bounds the namespace's total.
4. NetworkPolicy: default-deny at the namespace boundary, then explicitly allow the flows a team actually needs (its own pods, plus named shared services like an internal registry or logging endpoint). This requires a CNI (Container Network Interface, the plugin layer responsible for pod networking) that enforces NetworkPolicy; Calico and Cilium both do, but not every CNI plugin does.
5. Cost allocation: label every workload with its owning team, scrape actual usage with kube-state-metrics into Prometheus, and attribute cost with a tool built for it (Kubecost or OpenCost are common choices) rather than hand-rolling chargeback from raw node-hour billing.

Worked example: proportional chargeback

Suppose a cluster's monthly infrastructure bill is $12,000, and three teams' namespaces show the following measured CPU-hour consumption over the month (measured usage, not their static quota, since actual usage is what should drive cost):

$\text{Payments' bill} = \$12{,}000 \times 0.50 = \$6{,}000$

$\text{Search's bill} = \$12{,}000 \times 0.25 = \$3{,}000$

$\text{Internal tools' bill} = \$12{,}000 \times 0.25 = \$3{,}000$

This proportional-usage model scales the same way whether the cluster has 3 tenants or 1,000: the mechanism, label-based usage scraped continuously and aggregated, does not change; only the reporting layer has to move from a handful of dashboards a human reads to an automated nightly rollup a finance system consumes, since nobody reviews a thousand individual namespace dashboards by hand.

Single-cluster vs. per-team clusters

The same axis reappears one level down inside a single cluster: many teams that keep dev, staging, and production as one shared multi-tenant cluster for environments still break production out into its own cluster regardless of how they handle team tenancy, because a shared production control-plane incident is categorically worse than a shared-namespace tenant issue in a lower environment.

Trade-offs and pitfalls

• Quotas without LimitRange defaults mean a team that forgets to set requests/limits on a Deployment can be rejected outright at pod creation rather than merely scheduled sub-optimally; this surprises teams the first time it happens.
• NetworkPolicy default-deny with no shared-services allowance breaks logging and monitoring silently: pods that used to reach a cluster-wide logging endpoint stop being able to, and the failure shows up as missing logs rather than an error in the app itself.
• "Start single-cluster, split later" only works if the platform team retains an actual migration path (namespace export, workload relabeling); treat that migration tooling as part of the initial design, not a someday problem.

A breaking CRD (CustomResourceDefinition, the mechanism that registers a new custom object type into the Kubernetes API) schema change is handled by running two versions side by side rather than cutting over: add the new version as served alongside the old one, put a conversion webhook between them so every client keeps working regardless of which version it speaks, run a migration controller that rewrites persisted objects onto the new shape, and only flip the storage version (and eventually drop the old one) once nothing in the fleet still depends on it. The property that makes this safe is that served and storage are independent per-version flags: a version can stay served for old clients long after it has stopped being how objects are written to etcd.

1. CRD versioning

Kubernetes CRDs are apiextensions.k8s.io/v1 (the only supported API version; v1beta1 was removed in Kubernetes 1.22, so pin your tooling to v1 if you're touching anything older). Each entry in spec.versions can independently be served (clients can request it) and storage (this is the shape actually persisted in etcd; exactly one version must have storage: true).

spec:
  versions:
    - name: v1
      served: true
      storage: true    # existing shape, still the write-path today
    - name: v2
      served: true
      storage: false    # new shape, readable/writable via conversion
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service: {name: crd-converter, namespace: platform, path: /convert}
      conversionReviewVersions: ["v1"]

Use conversion.strategy: Webhook (not None) whenever the change is genuinely breaking. None only works when the two versions are structurally identical except for a rename, since it does no real transformation.

2. Conversion webhook

A small, stateless HTTPS service implementing the ConversionReview API: given an object in version A, return it in version B. It runs on every read, list, and watch that crosses versions, so it sits on the request hot path and needs its own availability (multiple replicas, readiness probes, low latency) since a webhook outage makes the CRD's non-storage versions unreadable. The detail most people miss: conversion must be lossless for fields the other version doesn't know about. If v2 adds a field that v1 has no place for, converting v2 to v1 has to stash that field somewhere recoverable (commonly an annotation) rather than silently dropping it, or a client that reads it back as v2 loses data it never touched.

3. Migration controller

The webhook converts on the fly; it does not rewrite what's stored in etcd. A separate migration controller (or the community kube-storage-version-migrator add-on, API group storagemigration.k8s.io/v1beta1, which requires a 1.30+ API server with the StorageVersionMigrator feature gate) performs the one-time job of listing every existing object and issuing a no-op update, which forces the API server to re-serialize it in whatever version is currently marked storage: true. Until that rewrite runs, old objects sit in etcd in the old format and are converted up to v2 on read; new writes go straight to v2. Run the migration rate-limited and namespace-by-namespace so a large object count doesn't overload the API server, and track per-object migration state so a partial run can resume.

4. Coordinating clients and controllers

Multiple controller versions can run against the same objects during B and C because every read comes back in whatever version the controller asked for; the risk is two controllers racing to write the same object in different versions and stomping each other's fields, which is why a canary-then-bulk migration (not "flip everything at once") matters.

5. Rollback and safety

Take an etcd snapshot before flipping the storage version. Rollback from phase C is just flipping storage back to v1 and leaving the webhook running; rollback from D (after v1 is dropped) is much harder, which is why D should only happen once request-metrics or audit logs confirm nothing is still asking for v1.

Worked scenario

For a CRD with roughly 40 known consumers, a realistic sequencing looks like: week 0, add v2 as served (storage stays v1), start the migration controller against a single low-traffic namespace as a canary and watch its metrics; week 1, if the canary is clean, run the migration across the rest of the objects in rate-limited batches; week 2, once 100% of objects report migrated: true in their status and the conversion webhook's request counts show v1 traffic has stopped growing, flip storage to v2; week 3+, after confirming no client has requested v1 in that window, drop v1 from served and retire the webhook. This is a schedule, not a measured result, and is meant to illustrate the ordering, not a fixed timeline for every CRD.

Trade-offs and pitfalls

• The conversion webhook is a new single point of failure on the CRD's read path; under-provisioning or forgetting its own rollout strategy turns a schema migration into an availability incident.
• Flipping the storage version before validating the bulk migration is fast but risky: if the new schema has a subtle bug, everything written after the flip is in the new (possibly wrong) shape.
• For a CRD with very few consumers, all of this machinery can be overkill: a coordinated breaking change with a short deprecation window and direct communication to the small client list may be cheaper than building conversion + migration infrastructure. Reach for the full multi-version path when you can't enumerate or coordinate every consumer directly.

A Kubernetes Service is a stable virtual address in front of a changing set of pods. The four types differ in who can reach that address and how: ClusterIP is internal-only, NodePort adds a port opened on every node, LoadBalancer asks the cloud provider to provision an external load balancer in front of that, and ExternalName is different in kind: a DNS alias to something outside the cluster with no pod routing involved at all.

Service types

A fifth variant sits alongside these four rather than replacing any of them: a headless Service (clusterIP: None). It skips kube-proxy's virtual-IP rewriting entirely, so instead of one stable IP load-balancing across pods, CoreDNS returns the individual pod IPs behind the Service directly to the client (sourced from the same EndpointSlices kube-proxy would otherwise consume). This is why StatefulSets pair with a headless Service: each replica needs its own stable, individually addressable DNS name rather than one shared address in front of all of them.

kube-proxy: the mechanism underneath

kube-proxy is what actually implements the ClusterIP, NodePort, and LoadBalancer rewriting, and it has more than one implementation:

• iptables mode (the long-standing default): programs Linux netfilter rules to destination-NAT (DNAT, rewriting a packet's destination address) Service traffic to a pod IP. Simple, but rule evaluation is roughly linear in Service count, which shows up as latency at high Service counts.
• ipvs mode (IP Virtual Server): uses the kernel's IPVS load-balancing tables instead of a rule chain, giving better performance and more scheduling algorithm choices at scale.
• nftables mode: the newer replacement backend, built on the modern Linux nftables framework, which reached general availability in Kubernetes 1.33. It targets the same correctness as iptables mode with materially better performance at high Service counts, though iptables remains the cluster-wide default for now.
  Older material sometimes still mentions a userspace mode; it was deprecated years ago and fully removed in Kubernetes 1.26, so it should not appear in any current design.

Worked example: tracing one request through a LoadBalancer Service

flowchart LR
    Client --> ELB[Cloud load balancer]
    ELB --> NP[Node: NodePort]
    NP --> KP[kube-proxy DNAT rule]
    KP --> PodA[Pod A]
    KP --> PodB[Pod B]

The client hits the cloud load balancer's public address, which forwards to a NodePort on whichever node it picked, where kube-proxy's DNAT rule sends the packet to one of the pods listed in the Service's EndpointSlices. With externalTrafficPolicy left at its default Cluster, the node that first receives the packet can forward it on to a pod on a different node, an extra hop, but with even load spread across all pods. Setting it to Local skips that extra hop and preserves the client's real source IP, at the cost of only routing to pods that happen to already be on the node the load balancer chose, so an uneven pod distribution across nodes turns directly into uneven traffic.

Trade-offs and pitfalls

• ExternalName is often reached for as 'the DNS one' when someone actually wants a placeholder Service for pods that do not exist yet. Because it has no selector and no health checking, nothing about it verifies the external target is even up.
• NodePort's fixed 30000-32767 range and one-port-per-Service-per-node model does not scale past a modest number of exposed Services and puts every node's IP in the attack surface; it is a building block for an external load balancer, rarely the final production answer on its own.
• LoadBalancer provisions one cloud load balancer per Service by default, which gets expensive and slow to provision as the number of Services grows; that pressure is usually solved by putting an Ingress controller behind a single LoadBalancer Service, a platform decision distinct from Service type selection itself.
• kube-proxy's mode is a cluster-wide setting, not a per-Service choice; moving from iptables to ipvs or nftables needs a validated cluster-wide rollout plan, not a Service-by-service change.