Kubernetes Architecture, Operations, and Troubleshooting Questions

Run the database through a purpose-built operator (a controller that encodes that specific database's operational knowledge as a CRD, Custom Resource Definition) rather than a hand-rolled StatefulSet, back it with topology-aware block storage, protect quorum with a PodDisruptionBudget (PDB) and anti-affinity, and treat backup/restore and failure recovery as first-class design requirements, not afterthoughts. The hardest part is not storage or scaling, it's guaranteeing zero data loss without a split-brain during failover, which needs an explicit fencing mechanism the platform doesn't give you for free.

Requirements first

Pin down RPO/RTO (Recovery Point/Time Objective) targets, expected throughput, latency budget, number of Availability Zones (AZs), and whether managed-DB-as-a-service is politically or contractually off the table; these decide almost everything below.

Three-way comparison: plain StatefulSet vs operator vs managed DB

For production, an operator is almost always the right middle ground unless a managed service is available and acceptable; a bare StatefulSet without an operator means re-implementing failover, backup orchestration, and safe upgrades yourself, which is exactly the hard part.

Concrete examples of DB-specific operators: CloudNativePG or the Zalando operator for PostgreSQL, Strimzi for Kafka, ECK (Elastic Cloud on Kubernetes) for Elasticsearch. Worth noting for currency: Kafka 4.0 (released March 2025) dropped ZooKeeper support entirely and runs KRaft-only, so a current Strimzi-managed Kafka cluster is one StatefulSet-backed topology (the brokers, using the KRaft consensus protocol for metadata) rather than two separate stateful systems (brokers plus a ZooKeeper ensemble) as older deployments required.

Architecture

flowchart TB
    OP[Database Operator / CRD controller]
    SS[StatefulSet: ordered, stable Pod identities]
    PDB[PodDisruptionBudget: protects quorum]
    SC[Topology-aware StorageClass, CSI provisioner]
    BK[Backup: snapshots + WAL/binlog shipping to object storage]
    OP --> SS
    OP --> PDB
    SS --> SC
    OP --> BK
    SS -->|anti-affinity| AZ1[AZ 1: primary]
    SS -->|anti-affinity| AZ2[AZ 2: sync replica]
    SS -->|anti-affinity| AZ3[AZ 3: async replica / DR]

Storage layer

• CSI (Container Storage Interface)-backed StorageClass with volumeBindingMode: WaitForFirstConsumer, so the volume is provisioned in the same zone the scheduler actually places the Pod, avoiding an unschedulable Pod stuck waiting on a volume in the wrong zone.
• reclaimPolicy: Retain on the StorageClass: an accidental PVC (PersistentVolumeClaim) deletion must not silently destroy the primary's data; recovery should require a deliberate, human step.
• Block storage versus a distributed filesystem, explicitly. Zonal block storage (EBS/PD/Azure Disk-style volumes) gives the lowest latency per replica but is single-zone and single-writer; the database's own replication protocol is what provides cross-zone durability, not the storage layer. A distributed filesystem or storage system (Ceph, Portworx, and similar) instead replicates data itself across nodes/zones at the storage layer, trading some latency and operational complexity for a storage layer that doesn't rely entirely on the database's own replication being correctly configured. For a workload with a mature, well-tested internal replication protocol (most production RDBMS and Kafka), zonal block storage plus database-level replication is usually the better latency/complexity trade-off; a distributed storage layer is more attractive for workloads with weak or no built-in replication.

The zero-data-loss and fencing problem

This is the part a plain StatefulSet does not solve for you. If a primary becomes unreachable but is not actually dead (a network partition, not a crash), promoting a replica while the old primary is still accepting writes creates two primaries accepting conflicting writes, a split-brain, which is a correctness failure, not just an availability one. A correct design needs explicit fencing: before a new primary is promoted, the old one must be provably prevented from accepting further writes, whether by a storage-level fence (revoking its volume attachment), a network-level fence (an admission gate the operator controls), or a consensus-based lease/term mechanism where a promoted replica only becomes writable once it holds a lease the old primary cannot renew. Most mature operators build this in (Kafka's KRaft controller quorum and Postgres operators using a distributed lock like Patroni's approach are both fencing-aware); a hand-rolled StatefulSet failover script that just runs a promote command is not safe against a network partition unless it independently solves this same problem.

PodDisruptionBudget and placement

• PodDisruptionBudget with minAvailable set so voluntary disruptions (node drains, cluster upgrades) can never take out enough replicas to lose quorum, for example minAvailable: 2 on a 3-replica set.
• Required (hard) pod anti-affinity across zones so replicas never land on the same node or, ideally, the same zone, or a single zone failure takes out more than one replica at once.
• Readiness probes gate traffic during recovery; liveness probes tuned to the database's actual health semantics, not a generic TCP check.

3-AZ replication topology

A common shape for regional durability without full cross-region cost: synchronous replication between the primary and one replica in a second AZ (so a confirmed write survives losing either AZ, at the cost of added write latency for the round trip), plus asynchronous replication to a third AZ or a separate region for disaster recovery, accepting a small, bounded RPO there in exchange for avoiding synchronous replication's latency cost across a third leg. The trade-off is explicit: every synchronous replica added tightens RPO but adds write latency; asynchronous replicas protect against a wider blast radius (a whole-region event) but carry a real, non-zero RPO if the primary is lost before the async replica catches up.

Backup and restore

• Application-consistent snapshots via CSI Volume Snapshots for fast full backups.
• Continuous WAL (write-ahead log) or binlog shipping for point-in-time recovery, usually built into the operator.
• Periodic backups to durable object storage (S3/GCS/Azure Blob-equivalent) with encryption and a lifecycle/retention policy.
• Restore drills on a schedule, not just documented in a runbook; an untested restore path is not a real recovery capability.

Handling node failures and scaling

• Node failure: the operator promotes a healthy replica (using the fencing mechanism above), Kubernetes reschedules the failed Pod elsewhere, and the storage layer either reattaches the same volume (single-zone block storage on a healthy node) or the operator rebuilds a fresh replica from backup/replication if the volume itself is unavailable.
• Vertical scaling: usually requires a coordinated, operator-sequenced resize or a planned failover, not a live in-place change.
• Horizontal scaling: read scaling via additional replicas behind a read-only Service; write scaling (if needed at all) requires sharding at the application or a purpose-built layer (e.g. Citus for Postgres, Vitess for MySQL), which is a materially different architecture, not a StatefulSet replica-count change.

Trade-offs and pitfalls

• Treating "add a PodDisruptionBudget and anti-affinity" as sufficient without addressing fencing is the most dangerous gap: it protects against voluntary disruption and Pod placement risk but says nothing about split-brain during an involuntary, ambiguous failure like a network partition.
• Choosing a distributed storage layer for its own replication when the database already replicates well is redundant complexity and extra latency for no real durability gain; verify what layer is actually providing the durability guarantee you're relying on.
• An untested backup is not a backup; the failure mode where a restore procedure is discovered to be broken happens exactly once, during a real incident.
• Running a fully DIY StatefulSet without an operator is a legitimate choice only for genuinely low-stakes or learning environments; for anything production-grade, the missing failover and fencing logic is a correctness gap, not just an operational inconvenience.

A Pod's DNS query for my-service.default.svc.cluster.local goes to CoreDNS, which answers from Service and EndpointSlice objects it watches from the API server; kube-proxy separately programs the node's packet-forwarding rules from those same EndpointSlice objects, so the DNS answer and the actual routing path are produced by two independent watchers reading the same underlying data. For a headless Service, CoreDNS skips the middle step entirely and hands back backend Pod IPs directly.

1) The Pod's DNS query

Every Pod's /etc/resolv.conf is set by the kubelet to point at the cluster DNS Service's ClusterIP (CoreDNS), plus a search-domain list (default.svc.cluster.local, svc.cluster.local, cluster.local) that lets short names like my-service resolve without the full suffix. The application's query goes out as a normal UDP (falling back to TCP for large responses) request to that ClusterIP on port 53.

2) CoreDNS: where the answer comes from

CoreDNS runs the kubernetes plugin, which watches the API server for Service and EndpointSlice objects (an EndpointSlice groups the ready backend Pod IPs and ports for a Service; it replaced the older single Endpoints object specifically so that large services don't require rewriting one giant object on every Pod change). Current CoreDNS releases watch EndpointSlices exclusively for this data; they do not fall back to the older Endpoints API, which matters if you're running a pre-1.21-era cluster still on v1beta1 EndpointSlices.

• Normal (ClusterIP) Service: CoreDNS returns a synthetic A record for the Service's stable ClusterIP, plus SRV records for named ports. The Pod's connection always lands on that one virtual IP.
• Headless Service (clusterIP: None): CoreDNS instead returns one A record per ready backend Pod IP, taken straight from the EndpointSlice. The client picks (or round-robins across) an actual Pod IP and connects to it directly. This is also how a StatefulSet's per-replica DNS names (pod-0.my-service.default.svc.cluster.local) resolve, since StatefulSets are built on a headless Service.

Because CoreDNS reacts to API watch events, a new or removed endpoint typically propagates into DNS within a second or two of the API server processing the change, not on a polling interval. Client-side and CoreDNS-side caching still add their own delay on top of that (see pitfalls below).

3) kube-proxy: turning EndpointSlice data into packet-forwarding rules

kube-proxy runs on every node, watches the same Service and EndpointSlice objects, and programs the node's kernel accordingly. It does not sit in the data path itself; it configures rules the kernel then executes per-packet.

Regardless of mode, the rule shape is the same idea: "packets to Service ClusterIP:port get DNAT'd (Destination Network Address Translation, rewriting the destination address) to one of the ready backend Pod IP:port pairs."

4) Runtime packet path

sequenceDiagram
    participant API as API Server
    participant DNS as CoreDNS
    participant App as Client Pod
    participant KP as Node kernel (kube-proxy rules)
    participant BE as Backend Pod
    API-->>DNS: watch Service + EndpointSlice (continuous)
    API-->>KP: watch Service + EndpointSlice (continuous)
    App->>DNS: query my-service.default.svc.cluster.local
    DNS-->>App: ClusterIP A record (or Pod IPs if headless)
    App->>KP: connect to ClusterIP:port
    KP->>BE: DNAT to a ready endpoint Pod IP:port
    BE-->>App: response

• If the chosen backend Pod is on the same node, the packet is delivered locally through the CNI's (Container Network Interface, the plugin that wires up Pod networking) bridge or veth pair without leaving the host.
• If it's on another node, the packet is routed (or encapsulated, depending on the CNI) across the node network, and some CNI/kube-proxy combinations perform source NAT on cross-node traffic so return packets route back correctly.
• For a headless Service, there's no DNAT step at all: CoreDNS already gave the client a real Pod IP, so kube-proxy is not involved in that connection.
• Only Pods that are Ready (passing their readiness probe) appear in the EndpointSlice at all, so an unready Pod is invisible to both DNS and kube-proxy simultaneously, not resolvable and not routed to.

Worked example: representative output

$ kubectl get endpointslices -l kubernetes.io/service-name=my-service
NAME               ADDRESSTYPE   PORTS   ENDPOINTS
my-service-x7f2q   IPv4          8080    10.244.1.12,10.244.2.9

$ dig +short my-service.default.svc.cluster.local
10.96.140.201

10.96.140.201 is the stable ClusterIP; 10.244.1.12 / 10.244.2.9 are the actual Pod IPs kube-proxy's rules will DNAT to. If the same Service were headless, the dig output would show 10.244.1.12 and 10.244.2.9 directly instead of the ClusterIP.

Trade-offs and pitfalls

• DNS record churn at scale: a Service with rapidly changing backends (frequent rollouts, aggressive autoscaling) generates a steady stream of EndpointSlice updates. CoreDNS itself answers from its watch cache correctly, but application-level and node-level DNS caching (and any fixed TTL a client library applies) can serve a stale backend IP after that Pod is gone; keep client-side DNS caching TTLs short for volatile Services rather than assuming CoreDNS's freshness is the only factor.
• Conntrack (connection tracking) table exhaustion or stale entries on a node can cause traffic to silently drop even though both DNS and the iptables/IPVS rules are correct; this is a distinct failure mode from DNS or kube-proxy misconfiguration and needs node-level conntrack metrics to diagnose.
• A Pod that resolves DNS successfully but still can't reach the backend usually means the DNS layer and the kube-proxy/CNI layer are fine but something else (NetworkPolicy, a firewall, or the backend Pod itself) is blocking the connection; treat DNS success as ruling out one layer, not all of them.

Kubernetes controllers that must have exactly one active instance, the scheduler, the controller-manager, and any custom controller or operator running multiple replicas for availability, coordinate through a Lease object in the coordination.k8s.io API rather than some separate consensus protocol of their own. Lease-based leader election has been the default resource lock since Kubernetes 1.20 (older ConfigMap- and Endpoints-based locks are deprecated); whichever replica successfully holds and keeps renewing a Lease is the leader, and every other replica watches that same Lease to know when to attempt a takeover.

The mechanics

• Acquire. A candidate that finds no current Lease, or one whose renewTime is already stale, writes itself in as holderIdentity with a fresh renewTime and a leaseDurationSeconds.
• Renew. The leader periodically updates renewTime well before leaseDurationSeconds elapses since the last successful renewal. The client-go leader-election defaults used by most controllers are a lease duration of 15 seconds, a renew deadline of 10 seconds (the leader must succeed a renewal within this window or it steps down voluntarily), and a retry period of 2 seconds (how often a non-leader checks whether it can acquire).
• Failover. If the leader fails to renew within leaseDurationSeconds of its last successful renewal, any other candidate observing that staleness is free to acquire the Lease and becomes the new leader.

Failure modes and mitigations

Lease TTL (time-to-live, the window after which a lease is considered stale) tuned too aggressively. A very short lease duration fails over faster but is also more likely to flap on a transient blip in API server latency; a longer duration is more stable but slower to fail over. This is a direct trade-off, not a bug to eliminate: pick the duration based on how expensive a few extra seconds without a leader actually is for that specific controller.

Clock drift between candidates. Each candidate compares its own local clock against the renewTime timestamp, which was written using the previous leader's clock, not a server-side timestamp from the API server. If a rival candidate's clock runs meaningfully ahead of the leader's, the rival perceives the lease as staler than it truly is and may attempt to acquire it before the leader's own renewal deadline has actually passed from the leader's perspective, risking a period where two replicas each believe they are leader. NTP (Network Time Protocol), through a client such as chrony, keeping all control-plane nodes tightly synchronized is the standard mitigation.

API server unavailability. If the leader cannot reach the API server to renew, it cannot tell whether it is still leader or has already failed over; well-behaved controllers treat a failed renewal as "step down," stopping leader-only work defensively rather than assuming they are still leader, which avoids a leader continuing to act after a rival has already taken over.

Network partition. Because the Lease itself lives in etcd behind a single API server, or a set of API servers backed by one etcd quorum, a genuine split-brain, two replicas both correctly holding a valid, currently renewed Lease at the same time, requires a partition inside etcd's own quorum, which etcd's own consensus protocol is designed to prevent by only allowing the quorum-holding side to make progress. A leader on the minority side of a network partition simply loses the ability to renew, falling into the API-server-unavailable case above, rather than continuing to hold a stale lease it can prove is still valid.

Worked example: how much clock skew is safe

With the client-go defaults above, the safety margin between when the leader must renew and when a rival is allowed to treat the lease as expired is:

$\text{margin} = \text{leaseDuration} - \text{renewDeadline} = 15\text{s} - 10\text{s} = 5\text{s}$

This margin exists to absorb clock skew between candidates: if a rival's clock reads more than 5 seconds ahead of the leader's, the rival could judge the lease stale and attempt a takeover before the leader's own 15-second window has genuinely elapsed by the leader's clock. Operationally, clock skew across every node running this controller needs to stay comfortably below that 5-second margin; a common operational target is well under one second, since ordinary NTP-synchronized nodes typically hold skew in the tens-of-milliseconds range, leaving most of the 5 seconds as headroom for the transient API server latency the renewal itself has to tolerate.

Trade-offs and pitfalls

• Shortening the lease duration to fail over faster directly trades away flapping resistance; a controller whose leader-only work is cheap to briefly duplicate (two replicas both running a reconcile loop for a few seconds) can tolerate a shorter duration better than one where duplicate execution causes real damage, such as both replicas issuing conflicting cloud-API calls.
• Custom controllers built with controller-runtime get this leader-election behavior largely for free, the same coordination.k8s.io Lease mechanism and the same client-go defaults, but a controller author still has to make its reconcile loop safe to briefly run twice during a failover window; leader election reduces the frequency of concurrent execution, it does not make concurrent execution impossible.
• Monitoring only whether a leader is elected misses flapping; watch the rate of leader transitions over time. A leader-election status metric changing repeatedly in a short window is the actual signal that something, usually clock skew or API server latency, is wrong.

sequenceDiagram
    participant L as Leader (replica A)
    participant R as Rival (replica B)
    participant API as API server / Lease
    L->>API: renew Lease (renewTime = T0)
    Note over L,API: renews every < renewDeadline (10s)
    R->>API: watch Lease
    L--xAPI: renewal fails (API server unreachable)
    Note over L: leader steps down (defensive)
    R->>API: check local_now - renewTime > leaseDuration (15s)
    R->>API: acquire Lease (holderIdentity = B)
    Note over R: becomes new leader

A pod is the smallest unit Kubernetes schedules and manages: one or more containers that always run together on the same node, sharing a network namespace and, optionally, storage volumes. It is the unit, rather than the individual container, because the kubelet, the scheduler, and every controller reason about placement, restarts, and networking at that granularity, not below it.

Why the pod, not the container, is the atomic unit

• Co-scheduling guarantee: every container in a pod is placed on the same node and started, stopped, and restarted as a unit by the kubelet.
• One IP per pod: regardless of how many containers it holds, a pod gets exactly one IP address; Services and DNS resolve to that pod IP, not to an individual container.
• Shared network namespace: containers in a pod talk to each other over localhost, and must not claim conflicting ports, because they share the same network namespace.
• Shared volumes: volumes are declared once at the pod spec level and can be mounted into more than one container, which is how a sidecar can read or write files the main container produces without a network hop.

When to run more than one container in a pod

• Sidecar pattern: a helper process that shares the fate and resources of the main container, most commonly a log shipper, a proxy, or a metrics exporter.
• Init containers: containers that run to completion, in order, before the main containers start, typically for one-time setup like a schema migration or config templating.
• Native sidecar containers: since the SidecarContainers feature became enabled by default in Kubernetes 1.29 (stable as of 1.33), you can declare a sidecar under initContainers with restartPolicy: Always. Kubernetes then starts it before the main container, keeps it running for the pod's whole life, and stops it after the main container on shutdown, which fixes the older ordering problem where a plain extra container might not be ready before the app started, or might outlive a completed Job's main container instead of shutting down with it.

A small example, an app container and a log-forwarding sidecar sharing a volume instead of a network call:

containers:
  - name: app
    image: myapp:1.4
    volumeMounts:
      - name: logs
        mountPath: /var/log/app
  - name: log-shipper
    image: fluent-bit:latest
    volumeMounts:
      - name: logs
        mountPath: /var/log/app
        readOnly: true
volumes:
  - name: logs
    emptyDir: {}

The app writes to /var/log/app; the sidecar tails the same path through the shared emptyDir volume, no network hop involved.

Worked example: what a failing sidecar looks like

If the log-shipper container above enters a crash loop while app keeps running fine, the two containers' fates are independent even though they share a pod: app keeps serving traffic, but the log pipeline drops. This is visible directly in the pod list, where the READY column reports containers-ready over containers-total, not a single number:

NAME                     READY   STATUS    RESTARTS      AGE
myapp-6d947f8db8-x2z1p   1/2     Running   4 (30s ago)   6m

Here 1/2 means only one of the pod's two containers is passing its readiness state, and the restart count of 4 belongs to the sidecar, not to app.

Trade-offs and pitfalls

• Co-location couples lifecycle: a sidecar cannot be scaled independently of the app the way a separate Deployment could be. Native sidecar containers loosen this slightly by giving the sidecar its own restart behavior, but it is still tied to the pod's schedule and node.
• Every container in the pod counts toward the pod's total resource footprint; forgetting the sidecar's own requests and limits when sizing the node is a common under-provisioning mistake.
• Splitting a component into a separate pod (plus a Service) regains independent scaling and blast-radius isolation, at the cost of a network hop and losing the localhost and shared-volume convenience. Reach for separate pods when the components genuinely differ in scaling or failure characteristics, for example a stateless API versus a shared cache, rather than defaulting to a sidecar for anything that happens to run alongside the main app.

At 5,000 nodes and 50,000 pods the design sits close to Kubernetes' documented large-cluster envelope (guidance of up to 5,000 nodes, no more than 110 pods per node, and no more than 150,000 total pods per cluster), so the control plane has to be sized deliberately rather than by default assumptions: one etcd member per availability zone (AZ) for quorum-safe failure tolerance, several API server replicas per AZ behind per-AZ and regional load balancing, and leader-elected controller-manager/scheduler replicas, with the actual trade-off being etcd member count (3 versus 5) traded against write latency.

flowchart TB
    subgraph AZ1
        E1[etcd member 1]
        A1[API server replicas]
    end
    subgraph AZ2
        E2[etcd member 2]
        A2[API server replicas]
    end
    subgraph AZ3
        E3[etcd member 3]
        A3[API server replicas]
    end
    LB[Regional load balancer] --> A1
    LB --> A2
    LB --> A3
    A1 --- E1
    A2 --- E2
    A3 --- E3
    E1 --- E2
    E2 --- E3
    CM[controller-manager and scheduler, leader-elected] --> LB

Placement

• etcd: one member per AZ (3 members total for the baseline design), each on a dedicated host not shared with API server or scheduling workloads, so etcd's latency-sensitive disk writes are never contending with unrelated CPU/IO pressure.
• API servers: stateless, so scale horizontally within each AZ, for example 3 to 5 replicas per AZ (9 to 15 total), behind a per-AZ load balancer that prefers local-AZ API servers for kubelet traffic (reducing cross-AZ hops for the highest-volume traffic class) plus a regional load balancer presenting one endpoint for external clients.
• controller-manager and scheduler: run multiple replicas with Kubernetes' built-in leader election (via Lease objects); only the elected leader is active at any time, the rest are hot standbys that take over on leader failure.

Etcd sizing: the real trade-off

Etcd requires a strict majority (quorum) of members to agree before committing a write. For a cluster of $N$ members, quorum is:

With $N=3$: quorum $= \lfloor 3/2 \rfloor + 1 = 2$, tolerating $\lfloor (3-1)/2 \rfloor = 1$ failure, exactly one full AZ. With $N=5$: quorum $= \lfloor 5/2 \rfloor + 1 = 3$, tolerating $\lfloor (5-1)/2 \rfloor = 2$ failures, more than one AZ's worth. At 3 AZs specifically, a 5-member cluster cannot place members one-per-AZ evenly (it needs a 2-1-2 or similar split), which reintroduces asymmetric AZ dependence; 3 members, one per AZ, is the cleaner fit for a 3-AZ topology and is the standard recommendation unless a documented durability requirement justifies the extra latency of 5.

Avoiding split-brain

• Quorum arithmetic itself prevents split-brain by construction: a write only commits with agreement from a strict majority, so a network partition can produce at most one side with a majority, never two.
• Never run etcd with an even member count; an even split has no built-in majority-breaking property and is strictly worse than the next odd number down for the same failure tolerance.
• Etcd's default heartbeat interval (100ms) and election timeout (1000ms) are tuned assuming low, LAN-like latency; validate real inter-AZ round-trip time against these defaults before relying on them across AZs, since higher-than-assumed cross-AZ latency causes spurious leader elections that look like instability but are actually a timeout tuning gap.
• Keep etcd off nodes that also run general workloads; resource contention from an unrelated noisy pod can delay etcd's disk fsync enough to look like a network partition to the rest of the quorum.

Operational trade-offs accepted

• 3 versus 5 etcd members: the design above accepts 3 (tolerating one AZ loss) for lower write latency; this is a deliberate trade against the extra durability 5 would provide, appropriate because 3 AZs already provides physical fault isolation and losing two AZs simultaneously is a much rarer event than losing one.
• API server replica count versus etcd read load: more API server replicas absorb client load and reduce tail latency, but every replica also issues reads against etcd (mitigated by the API server's built-in watch cache, so most reads are served from memory rather than hitting etcd directly); sizing replicas too aggressively without accounting for this can still add unnecessary etcd read pressure.
• Scale ceiling: at 50,000 pods the cluster is well within documented limits, but is a meaningful fraction of the 150,000-pod ceiling; if growth continues, the accepted trade-off today (one large cluster) may need revisiting toward sharding workloads across multiple clusters before hitting the ceiling, rather than as an emergency reaction to it.
• Backups: frequent etcd snapshots plus a rehearsed, tested restore procedure are accepted as an ongoing operational cost, not a one-time setup task, since an untested restore path is not meaningfully different from having no backup at all.

Trade-offs and pitfalls

• Assuming "3 AZs" alone guarantees safety without validating actual inter-AZ latency against etcd's default timeouts is the most common design gap; the topology looks correct on a diagram and still produces spurious elections in practice.
• Over-provisioning API server replicas without watch-cache-aware capacity planning trades one bottleneck (client-facing latency) for another (etcd read amplification).
• Treating 5-member etcd as strictly "safer" than 3 ignores that it does not fit evenly across exactly 3 AZs and costs write latency on every single commit, not just during a failure.