Consensus and Coordination Algorithms Questions

Short answer

Split-brain is when a replicated cluster that is supposed to have one leader (or one writable primary) ends up with two or more nodes that each believe they are in charge, usually because a network failure split the cluster into groups that cannot see each other. If both sides keep accepting writes, the data forks: the same record can be changed two different ways, and there is no longer one correct version. The standard prevention is to require a majority quorum (more than half the voting members) before anyone may act as leader, so at most one side of any split can qualify, and to fence a deposed leader so its late writes are rejected.

Typical causes

Consequences

For correctness:

• Conflicting updates to the same key (two different balances, two owners of one seat).
• Lost updates when one side's writes are later discarded during recovery.
• Violated invariants that a single leader would have enforced: uniqueness, "stock never below zero", "each job runs once".
• Duplicate side effects: both leaders send the same email, charge, or event.

For availability, the design choice is stark. During a partition a system can either let both sides serve (available but inconsistent, the split-brain outcome) or let only the majority side serve (consistent, but the minority side refuses writes). This is the CAP theorem trade-off (consistency, availability, partition tolerance: under a partition you must give up one of the first two) in concrete form. Quorum-based systems deliberately choose to make the minority unavailable.

Worked example: why node count matters

Majority means strictly more than half.

• 5 nodes, partition into 3 and 2. Majority is 3. The 3-node side elects a leader and keeps working; the 2-node side cannot, so it refuses writes. Exactly one leader.
• 4 nodes, partition into 2 and 2. Majority is 3. Neither side has it, so with a correct quorum rule neither side can write: no split-brain, but a full write outage. With a broken rule ("elect if you can reach half"), both sides elect: split-brain.
• 2 nodes (primary plus replica). Any partition is 1 and 1. There is no majority possible, so automatic failover between two nodes is inherently unsafe without a third party. This is why two-node high-availability setups add a witness (a small voting member, often in a third location, that stores no data but breaks ties).

This is why clusters are sized 3 or 5: they tolerate 1 or 2 failures respectively, and an even-sized cluster tolerates no more failures than the odd size just below it.

Prevention and mitigation in production

1. Majority quorum for leadership. Use a consensus protocol (Raft, Paxos, or Zab, the protocol inside the ZooKeeper coordination service) or a store built on one to decide who leads. A leader that loses contact with a majority must step down (Raft implementations call this CheckQuorum).
2. Odd number of voters across failure domains (independent zones, such as separate data centers or power feeds, that are unlikely to fail at the same time). For example 3 nodes in 3 zones, or 2 + 2 + 1 across three sites, so no single site failure leaves the survivors without a majority.
3. Fencing. Each new leader gets a higher epoch or term number (a counter that only ever goes up; whichever side shows the higher number is the more recent leader), and storage rejects writes with an older one. This handles the zombie leader that wakes up after a pause. Where storage cannot check epochs, STONITH ("shoot the other node in the head") forcibly powers off or isolates the old primary before the new one starts.
4. Lease discipline. If leaders use time-based leases, the leader must treat its lease as expired earlier than followers will, with a margin that covers clock drift.
5. Detection. Alert when more than one node reports itself as writable or leader. Split-brain that is detected in one minute is an incident; detected in a day it is a data-recovery project.
6. Testing. Inject partitions and pauses in staging (for example with a chaos tool: software that deliberately triggers failures, such as killing a process or cutting network access, to test how the system responds) and verify the old leader actually stops writing.

Trade-offs and pitfalls

• A longer failure-detection timeout makes false failovers rarer but does not prevent split-brain; only quorum plus fencing does. It also lengthens every real outage.
• Quorum means the minority side goes read-only or down during a partition. That is the price of correctness, and it should be a documented, accepted behaviour.
• Split-brain is not only a database problem: schedulers, lock services and leader-elected workers all have the same failure if their leadership is not quorum-backed and fenced.

Direct answer

Use an external consensus system, and prefer one you already run: on Kubernetes, the built-in Lease objects (a small Kubernetes API resource that one process can hold and periodically renew, giving the cluster a native way to say "exactly one owner, until it stops renewing") for leader election; otherwise etcd or Consul. A "lightweight custom coordinator" is either a single process (a single point of failure) or, once you make it highly available, a consensus implementation you now own, which is the hardest kind of code to get right and the hardest to test. The only case where custom wins is when you can build it on top of a strongly consistent store you already operate (a database row with a lease and a compare-and-set, an atomic "update this row only if it still holds the value I expect" operation), which is custom glue, not custom consensus.

What the control plane actually needs

A control-plane service (the part of a platform that decides configuration and assignments, as opposed to the data plane that serves traffic) typically needs:

1. Leader election: exactly one active instance makes decisions, and a replacement takes over when it dies.
2. Configuration propagation: every worker learns the current config version quickly and in order, and can resume after a disconnect without missing a change.
3. Fencing: a deposed leader that wakes up late (after a garbage-collection pause or a network stall) must not be able to apply stale decisions.

etcd and Consul provide the primitives: leases or sessions (grants that expire unless renewed), atomic compare-and-set transactions, watches (etcd's push notifications: subscribe to a key and get told the instant it changes) or blocking queries (Consul's version of the same idea: an HTTP request that just hangs open until the watched key changes or a timeout elapses), and a monotonically increasing revision (etcd's cluster-wide counter, which goes up by one on every write anywhere in the store; each key remembers the revision at which it was last changed) or index that doubles as a fencing token.

Evaluation

Two product details matter for the design:

• Consul sessions are tied to node health checks as well as an optional TTL (time-to-live: an expiry that renewals push back), and by default apply a lock-delay of 15 seconds after a session is invalidated, during which the lock cannot be re-acquired. That protects against a just-deposed holder still acting, at the cost of a longer failover. Tune it deliberately.
• etcd limits: requests default to at most 1.5 MiB and the storage quota defaults to 2 GiB. Config propagation is fine; shipping large artifacts through etcd is not (store them elsewhere and propagate a pointer and a hash).

Worked example: config propagation with watches

A control plane publishes routing config to 2,000 workers.

• The leader writes config/routing with a new version. etcd commits it at, say, revision 90412.
• Each worker holds a watch on config/ from the last revision it applied. The change arrives as one event per worker: 2,000 small notifications, fanned out by etcd members, not by the leader service.
• A worker that was disconnected reconnects with "watch from revision 90390" and receives every change it missed, in order, as long as that revision has not been compacted away (etcd periodically discards, or "compacts," old revision history to bound its own storage, so a watch trying to resume from a revision older than the compaction point can no longer be honoured). If it has, the worker does a full read of config/ and resumes watching from the returned revision.

The custom equivalent needs its own versioning, per-worker cursors, replay buffer and "you are too far behind, do a full resync" path: each is a place for a lost-update bug.

Fencing: the part neither option does for you

Leader election decides who should act; it cannot stop an old leader who does not know it has been replaced. So:

• The leader reads the revision of its election key (etcd) or the lock index (Consul) when it wins, and attaches it as a fencing token to every write it makes.
• Downstream systems store the highest token seen and reject writes with a lower one. In etcd you can go further and make every control-plane write a transaction conditioned on the leader key still existing with that revision.

When custom is reasonable

• Built on an existing strongly consistent store. A table row (role, holder, lease_expires_at, token) updated with a conditional write ("only if the row is expired or already mine", incrementing the token) in Postgres or DynamoDB gives correct leader election without running new infrastructure. You still owe fencing downstream and careful clock handling (use the database's clock, not each client's).
• Truly single-instance by design, where an operator restarts it and minutes of unavailability are acceptable: then you do not need election at all, just a health check and a runbook.

What would flip the recommendation toward building more: extreme scale of coordination state (millions of keys, high write rates) where etcd's size and single-leader limits do not fit. Even then the answer is usually a sharded consensus database, not a hand-written protocol.

Trade-offs and pitfalls

• Reusing Kubernetes' etcd directly for application data: use the Lease API, not raw keys, so you do not load the cluster's own control store.
• Election without fencing: two active leaders for the length of a pause, with either option.
• Treating watch delivery as exactly-once: handlers must be idempotent (applying the same config version twice is harmless) and must handle compaction.
• Underestimating the "lightweight" option's test burden: its correctness can only be shown with partition, pause and clock-skew testing, which usually costs more than running etcd.

Direct answer

Start from the targets. RPO (recovery point objective: how much acknowledged data you may lose) of zero means a write must be acknowledged by enough replicas, in enough independent failure domains, that any failure you plan for leaves a copy. RTO (recovery time objective: how long you may be unavailable) sets how quickly the survivors must take over, which bounds failure detection plus leader election. Then pick N replicas, a write quorum W and a read quorum R so that W + R > N (every read quorum overlaps every write quorum, so reads see the latest acknowledged write) and 2W > N (two conflicting writes cannot both succeed). For most systems that need RPO 0 across a zone or region loss, the answer is N = 5 with W = R = 3, placed 2/2/1 across three failure domains (independent units, such as availability zones or regions, that can fail together as one: shared power, network, or an entire data centre), reads routed to the leader or through a quorum.

The rules behind quorum choice

A quorum is the number of replicas that must respond for an operation to succeed.

Writes stay available while at most N - W replicas are down; reads while at most N - R are down. RPO 0 for losing a whole failure domain additionally requires that the W acknowledgements always include at least one replica outside any single domain. With 2/2/1 placement and W=3 this holds automatically: the largest domain holds only 2 of the 5 replicas, so any 3 acknowledgements can never come entirely from one domain, meaning at least one of the 3 is always in a different domain from the other two. That is exactly why 2/2/1 with W=3 is the placement recommended below; a placement like 3/1/1 would let the domain holding 3 replicas satisfy W=3 on its own, which does not guarantee RPO 0 for that domain's loss (the table below confirms this: placement (3, 1, 1) does not keep a majority after losing one zone).

CAP (the result that during a network partition a system must choose between consistency, every read seeing the latest write, and availability, every request getting a non-error response): quorums are how you make that choice explicit. A side of the partition that cannot assemble W replicas must refuse writes if you want consistency, or accept them on the replicas it can reach if you want availability.

Example configurations

This short program enumerates common configurations and checks which zone placements survive losing one zone:

def describe(n, w, r):
    return {
        "N": n, "W": w, "R": r,
        "read_sees_latest_write": w + r > n,
        "no_conflicting_writes": 2 * w > n,
        "writes_survive_down": n - w,
        "reads_survive_down": n - r,
    }

print(f"{'N':>2} {'W':>2} {'R':>2}  W+R>N   2W>N  writes-ok-with-down  reads-ok-with-down")
for n, w, r in [(3, 2, 2), (3, 3, 1), (3, 1, 3), (3, 1, 1), (5, 3, 3), (5, 4, 2), (5, 2, 4)]:
    d = describe(n, w, r)
    print(f"{n:>2} {w:>2} {r:>2}  {str(d['read_sees_latest_write']):>5}  {str(d['no_conflicting_writes']):>5}"
          f"  {d['writes_survive_down']:>19}  {d['reads_survive_down']:>18}")

# Which placements of 5 replicas over 3 zones keep a majority (3) after losing ANY one zone?
print()
for placement in [(2, 2, 1), (3, 1, 1), (3, 2, 0)]:
    worst = min(5 - z for z in placement)
    print(f"placement {placement}: worst-case survivors after one zone loss = {worst} -> majority kept: {worst >= 3}")

Output:

 N  W  R  W+R>N   2W>N  writes-ok-with-down  reads-ok-with-down
 3  2  2   True   True                    1                   1
 3  3  1   True   True                    0                   2
 3  1  3   True  False                    2                   0
 3  1  1  False  False                    2                   2
 5  3  3   True   True                    2                   2
 5  4  2   True   True                    1                   3
 5  2  4   True  False                    3                   1

placement (2, 2, 1): worst-case survivors after one zone loss = 3 -> majority kept: True
placement (3, 1, 1): worst-case survivors after one zone loss = 2 -> majority kept: False
placement (3, 2, 0): worst-case survivors after one zone loss = 2 -> majority kept: False

Reading the table against targets:

Configurations with 2W not greater than N (W=1 of 3, W=2 of 5) allow two concurrent writes to succeed on disjoint replicas, so they need conflict resolution (versions, merge rules) and are not a basis for "no lost updates".

Meeting an RTO

For a leader-based system (Raft or Paxos underneath), the outage after a leader dies is roughly detection + election + client retry. With etcd's documented defaults (100 ms heartbeat, a periodic "I'm still leader" message so followers don't start an election; 1000 ms election timeout, how long a follower waits without hearing from the leader before starting an election of its own), that is on the order of one to two seconds, plus however long clients take to retry against the new leader. If the RTO is 30 seconds, these defaults have large margin; if it is sub-second, you need lower timeouts, which requires a fast, stable network (etcd's guidance: election timeout at least 10 times the round-trip time) or else you trade outage length for spurious elections.

For a leaderless system (Dynamo-style quorums, after Amazon's Dynamo design, where any replica accepts writes and the client or a coordinator waits for W of them), there is no election: as long as W replicas are reachable, writes continue, so RTO for a single replica failure is effectively the client's timeout and retry.

Read and write routing

• Writes always go to the leader (leader-based) or to a coordinator (in a leaderless design, whichever node the client's request happens to land on, acting as a temporary go-between for that one request) that waits for W acknowledgements (leaderless).
• Linearizable reads (a read that reflects every write already acknowledged to a client, with no stale or out-of-order result): route to the leader, which confirms it is still leader with a quorum round or a lease (a time-limited grant from the other replicas that lets the leader answer without re-checking every time, until it expires) before answering, or read from R replicas and take the newest version.
• Local, possibly stale reads from the nearest replica for data where staleness is acceptable. This lowers latency and spreads load but gives up the W + R > N guarantee for those reads.
• Across regions, per-region quorums (such as Cassandra's LOCAL_QUORUM, a quorum computed only among the replicas in the caller's own region rather than across the whole cluster) keep latency local but mean a write acknowledged in one region may not exist in another yet: RPO for a region loss is no longer zero.

Practical implications

Latency. A write returns when the W-th fastest replica acknowledges. With replicas 2/2/1 across regions A, B, C and assumed round trips of 1 ms within A and 65 ms from A to B, a W=3 write from a coordinator in A waits for A's second replica (1 ms) and the first B replica (65 ms): about 65 ms. Raising W to 4 means waiting for the second-fastest remote replica, and the tail latency (the 99th percentile, p99) of that replica now drives yours.

Throughput. Every write is sent to all N replicas regardless of W; W only changes how many you wait for. Larger N multiplies network and disk work per write. A higher R multiplies read work.

Correctness. W + R > N alone does not make a leaderless store linearizable: concurrent writes, clock-based last-writer-wins (when two writes conflict, whichever carries the later timestamp wins and the other is discarded, which breaks down if clocks disagree between nodes), sloppy quorums (accepting writes on stand-in replicas during a partition, then handing them back later) and failed writes that partially landed can all produce anomalies. If you need linearizability, use a consensus protocol, or add versioning and read repair (when a read notices some replicas answered with an older version, write the newest version back to those stale replicas so they heal) and test it.

Trade-offs and pitfalls

• Even N (4 or 6) buys no extra fault tolerance for majority quorums and costs latency.
• Two failure domains cannot give RPO 0 and availability through the loss of either one: whichever domain holds the majority takes the system down when it fails.
• Counting replicas instead of failure domains: 5 replicas with 3 in one zone look robust and are not.
• Tuning quorums without writing the target first: say "RPO 0 for a zone loss, RTO under 60 seconds" and derive the numbers from it.

Direct answer

A replicated database keeps several copies of each piece of data, and those copies must agree on the order of writes. The simplest way to get one order is to let exactly one node, the leader, accept writes and tell the others (the followers) what to apply. Leader election is the protocol that picks that node and replaces it when it fails, and it must never let two nodes both believe they are leader at the same time. Raft and Paxos are the algorithms; etcd and ZooKeeper are systems that package them. The price is a short write outage every time the leader dies or gets cut off, plus the rule that only the side of a partition holding a majority may keep working.

Why a leader at all

Without a single writer, two replicas can accept conflicting writes to the same row at the same moment ("set balance to 80" on one node, "set balance to 50" on another), and the cluster ends up with copies that disagree. Resolving that afterwards is either impossible (money) or needs merge rules the application must supply. A leader serialises writes: it assigns each write a position in a log (entry 41, 42, 43...), and every follower applies the same log in the same order, so every copy ends in the same state. This pattern is called state machine replication.

Two terms appear in every leader-election protocol:

• Quorum (majority): more than half the voting members, for example 3 of 5. Any two majorities of the same group share at least one member, and that shared member is what stops two leaders from being elected for the same period.
• Term (Raft) / epoch (ZooKeeper) / ballot (Paxos): a number that increases with every election. Messages carry it, and a node that sees a higher number steps down. It is how the cluster tells the current leader from an old one.

The common algorithms and systems

Note the two layers. A database such as CockroachDB or TiKV (distributed SQL/key-value databases that split their data into many small pieces) embeds Raft and elects a leader per data range (one such piece: a contiguous slice of the keyspace, replicated and led independently of the other pieces). A service that is not itself replicated (say, one active scheduler among three instances) borrows an election from etcd or ZooKeeper instead of implementing consensus.

How leader election affects availability

While there is no leader, the cluster cannot accept writes, and strongly consistent reads (reads guaranteed to reflect every write acknowledged before them, rather than possibly returning slightly old data) also stall. The outage is roughly:

failure detection time + election time + client retry time.

Worked example with etcd's documented defaults (heartbeat 100 ms, election timeout 1000 ms): a leader crashes. Followers wait up to about 1 s without a heartbeat, one starts an election, and with a network round trip of a few milliseconds it wins almost immediately. Clients that were mid-request see an error or timeout and retry. So a leader crash costs on the order of one to two seconds of write unavailability, not zero. A 5-member cluster tolerates 2 simultaneous failures (3 of 5 is still a majority); a 3-member cluster tolerates 1.

Reads that can accept slightly old data may be served by any follower during the gap, which is a deliberate trade of freshness for availability.

What an architect must consider

Network partitions. Suppose 5 nodes split into groups of 3 and 2. Only the group of 3 can elect a leader and commit writes; the group of 2 must refuse writes. If the old leader was in the group of 2, it keeps thinking it is leader until it notices it cannot reach a majority. That is the split brain risk (two nodes both acting as leader). Raft prevents two committed histories because the minority leader cannot get majority acknowledgements, but the stale leader can still serve stale reads or keep doing side effects outside the database unless:

• reads go through the leader with a majority check or a leader lease (a period during which the leader knows no one else can be elected), and
• external side effects carry a fencing token (the term number, checked by the receiving system so it rejects anything from an older term).

Placement. Put members in at least three failure domains (availability zones or regions). With two zones, losing the zone holding the majority takes the cluster down.

Leader churn. Frequent re-elections cost more than one clean failover, because each election is a write outage. Common causes are an election timeout set too close to the real network round trip, slow disks (a leader stuck in a disk sync misses heartbeats), long garbage-collection pauses and overloaded CPUs. The fixes are tuning the timeout to at least 10 times the round trip time (etcd's own guidance), fast dedicated disks, and alerting on the rate of leader changes rather than just on "is there a leader".

Cluster size. Use an odd number of voters, usually 3 or 5. Going from 5 to 7 tolerates one more failure but every write now waits for 4 acknowledgements instead of 3, so writes get slower.

Trade-offs and pitfalls

• Consensus is not free availability. It trades availability during partitions for correctness. If the data can tolerate conflicts (a "likes" counter, a shopping cart), a leaderless eventually consistent design (every replica can accept writes independently, and copies converge to the same value over time rather than agreeing instantly) may be the better fit.
• Rolling your own election with a heartbeat and a timeout (whoever has not heard from the leader for 5 s takes over) produces two leaders the first time a network is slow rather than dead. Borrow a proven implementation.
• Believing the process that holds leadership is the only one acting. A leader paused by garbage collection can wake up and act after it has been replaced. Fencing tokens, not timeouts, make that safe.
• Watching only "leader exists". A cluster that re-elects every few minutes looks healthy on that metric while clients see repeated error spikes.

Direct answer

I would build it on Raft (a leader-based consensus protocol where one elected node orders every write and replicates it to a majority), using a mature library rather than a hand-rolled protocol, and run 5 voting members spread 2/2/1 across three regions so that losing any single region still leaves a quorum (a majority, here 3 of 5). Paxos and Raft give the same fault tolerance, but Raft specifies the parts a production service actually needs (leader election, log repair, membership change, snapshots) as one design, which is where Paxos-based builds historically lose months. Leader leases, partitions, compaction and bootstrap are then handled by explicit rules below, each of which protects one safety property.

Paxos vs Raft: the decision

Consensus means a group of nodes agreeing on one ordered sequence of values even when some nodes crash or messages are lost. Paxos is the classic single-value protocol; running it repeatedly with a stable leader is called Multi-Paxos. Raft was designed to be equivalent to Multi-Paxos in guarantees but easier to understand and implement.

Commit: Raft. A coordination service carries low write volume and needs correctness, operability and a staff that can debug it at 3 a.m. The "holes in the log" freedom of Multi-Paxos buys little here and costs a lot of reasoning.

What would flip it: if every region needed low-latency writes with no single leader region (a write-heavy, geo-symmetric workload), a leaderless variant such as EPaxos, or separate per-region groups, becomes worth its complexity. For leader election and config discovery it does not.

Architecture

flowchart LR
  C1[Clients us-east] --> A1
  C2[Clients eu-west] --> B1
  subgraph R1[us-east]
    A1[voter 1 leader] --- A2[voter 2]
  end
  subgraph R2[eu-west]
    B1[voter 3] --- B2[voter 4]
  end
  subgraph R3[us-west]
    D1[voter 5]
  end
  A1 --> B1
  A1 --> D1

Primitives exposed: a key-value store with revisions, compare-and-set transactions (a write that only succeeds if the key's current value or revision still matches what the caller expects, and otherwise fails harmlessly, so two callers racing to update the same key cannot both silently win), watches (a client subscribes to changes on a key prefix), and client leases (a key that vanishes if its owner stops renewing). Leader election for applications is a recipe on top: create a key under /election/<name> bound to your lease; the lowest creation revision wins.

Leader lease semantics (two different leases)

1. The Raft leader's read lease (internal). A term (also called an epoch) is a monotonically increasing election number; each term has at most one leader. To serve a read locally without a network round trip, the leader must be sure no newer leader exists. Rule: when a majority acknowledges a heartbeat the leader sent at local time t_send, followers have promised not to vote for anyone else until their election timeout expires, because granting that heartbeat is what resets a follower's own election timer in the first place (the same reset that happens on every valid heartbeat); a follower that has just reset its timer cannot fire it again, and so cannot vote for a rival, until that timer would next expire. The leader may therefore serve local reads until

where ρ is the maximum clock-rate drift and δ a safety margin. Worked numbers: election timeout 1000 ms, assumed drift bound 1%, margin 50 ms gives a lease of 1000 x 0.99 - 50 = 940 ms after each acknowledged heartbeat. This depends on bounded clock rate drift (not synchronized wall clocks). If you do not trust that bound, use ReadIndex instead: the leader records its commit index, confirms leadership with one heartbeat round, waits until it has applied up to that index, then answers. That costs one round trip per read batch but needs no clock assumption. My default is ReadIndex for a global deployment and lease reads only inside one region.

2. Client leases (external, what applications see). A client creates a lease with a TTL (time-to-live), say 10 s, and renews it every 3 s. Keys attached to it (its election key, its lock) are deleted when the lease expires. Expiry is decided by the leader and applied through the log, so every replica agrees on the moment of revocation. Two rules keep this safe:

• A new leader extends all outstanding lease deadlines by a full TTL after taking over, because it cannot know how recently the previous leader saw each renewal. Better a late expiry than a premature one.
• The client must treat its own lease as lost at last_ack + TTL - margin measured on its own clock, and stop acting as leader then, even while partitioned.

Leases do not prevent a paused process from acting late, so every grant also returns a fencing token (the key's creation revision, strictly increasing), which downstream systems compare to reject writes from a stale holder.

Network partitions

Clients in the minority region lose writes and linearizable reads (reads guaranteed to reflect every write that completed before they started, the strongest ordinary consistency guarantee). That is the price of never electing two leaders; state it to your consumers as an SLO (service-level objective) property, not a bug.

Log compaction and snapshotting

The Raft log grows forever unless compacted. Every node independently:

1. Takes a snapshot of its state machine (the in-memory data this cluster is actually coordinating, built by applying the log's commands in order) at applied index i, recording lastIncludedIndex = i, lastIncludedTerm, and the current membership configuration (forgetting it would make a restarted node not know who its peers are).
2. Writes it atomically (write to a temp file, fsync it, meaning force the write to complete on physical storage before continuing rather than trusting a cache that could lose it on a crash, then rename the temp file over the old one, so a crash mid-write never leaves a half-written snapshot), then discards log entries up to i, keeping a tail so slightly lagging followers can catch up from the log.
3. Snapshots are taken from a copy-on-write (new writes go to a fresh copy of the changed data instead of overwriting it in place, so a reader still following the old copy is unaffected) or MVCC (multi-version concurrency control) view so writes keep flowing while the snapshot streams to disk.

A follower too far behind receives InstallSnapshot from the leader: chunks of the snapshot, then it discards its conflicting log and resumes normal replication. Throttle snapshot transfers, because a 1 GiB snapshot pushed to several followers at once can saturate the leader's network and delay heartbeats enough to trigger elections.

Trigger compaction by log size (for example every 64 MiB of log) rather than a fixed entry count, since coordination entries vary from bytes to kilobytes.

Safe bootstrap and scaling

Bootstrap. A split-brain (two groups each believing they are the cluster) can be created at birth:

• Start every initial node with the same static initial membership list and a unique cluster ID; nodes reject messages carrying a different cluster ID. Never let two operators each "bootstrap a new cluster" with overlapping hosts.
• A node whose disk was wiped must never rejoin under its old identity. It has lost its record of currentTerm and votedFor, so it could vote twice in one term and elect two leaders. Remove the old member and add it back as a new member.

Scaling. Membership changes go through the log as configuration entries:

• Add a new node first as a learner (non-voting replica), let it catch up via snapshot plus log, then promote it to voter. Adding a lagging voter immediately can stall commits, because a 3-node cluster that becomes 4 needs 3 acks and the newcomer cannot give one yet.
• Change one voter at a time (single-server changes guarantee old and new majorities overlap) or use joint consensus (a transitional configuration requiring majorities of both old and new sets) for bigger moves.
• Keep voter counts odd: 4 voters tolerate the same 1 failure as 3 but need a bigger quorum. Scale reads with learners or non-voting replicas, not extra voters.

Worked example: write latency for the 2/2/1 layout

Illustrative round-trip times (not measurements): us-east to eu-west 75 ms, us-east to us-west 65 ms, eu-west to us-west 140 ms. With the leader in us-east, a commit needs 3 of 5 acknowledgements: the leader itself, the second us-east voter (about 1 ms), and the fastest remote voter (us-west at 65 ms). So a write commits in about 65 ms plus disk fsync, and a client in eu-west adds its own 75 ms round trip to reach the leader: about 140 ms end to end. A 1000 ms election timeout is over 10 times the 75 ms leader-to-follower round trip, which matches the etcd tuning guidance that election timeout be at least ten times the round-trip time; a failover then costs roughly 1 to 2 s of write unavailability.

Trade-offs and pitfalls

• Global quorum means every write pays a WAN round trip. Fine for elections and config (tens of writes per second), wrong for a hot data path.
• Lease reads trade a clock assumption for latency. If VMs can be paused or clocks jump, prefer ReadIndex.
• The service owns ordering, not the clients' actions. Without fencing tokens, a GC-paused former leader still corrupts downstream state after its lease expired.
• Snapshot and membership bugs are where home-built consensus fails, not the happy-path election. Run the library's own test suites, add fault injection (partitions, disk stalls, clock skew) and a linearizability checker (a tool, such as Jepsen or Porcupine, that replays a recorded history of operations and checks whether any valid ordering could explain it) in CI before trusting it.
• "Without a cloud-managed service" does not mean writing Raft yourself. Embedding a proven library, or running etcd itself, is still self-hosted and removes most of the risk.