Distributed Systems Fundamentals Questions

Direct answer

Keep exactly one path for reads that must be linearizable, meaning the read is guaranteed to see every write that completed before the read began, and give everything else a cheaper path that trades a bounded amount of staleness for a shorter round trip. The three techniques below differ only in how they let something other than a full leader round trip answer safely, without breaking the guarantee for the reads that actually ask for it.

Technique 1: bounded-staleness follower reads

Each replica tracks the highest write position (a log index or sequence number) it has applied. A client that can tolerate some staleness issues a "read as of no more than X positions old" request to its nearest replica; if that replica's applied position is within X of the leader's latest, it answers locally. A strict "give me the current value" request is never eligible for this path and always goes to the leader (or through one of the other two techniques below).

Technique 2: leader lease reads

The leader holds a time-bound lease, renewed through its normal heartbeat or replication round trip with followers, that certifies no other node could have become leader before the lease expires. While the lease is valid, the leader can answer a read from its own local state without running a fresh round of consensus for that specific read, because the lease itself is the proof no newer leader could have committed a write elsewhere in the meantime. This only removes the coordination round trip, not the trip to the leader itself, so it mainly helps latency when the leader happens to be close to the requester; it does not by itself let a remote follower answer.

Technique 3: read-index protocol (two-tier API)

Before answering a strong read, the leader (or a follower proxying to it) records the currently committed log index, the "read index", and confirms with a lightweight quorum check-in, not a full new log entry, that it is still the leader. It can then let any replica serve the read once that replica's own applied index has reached the read index, including a nearby follower, without paying for a full consensus round trip per read. Exposing this as two API surfaces, a ReadStrong() that always uses the read-index path and a ReadFast() that skips straight to the nearest replica's current state, lets the client declare which guarantee it actually needs.

Worked example: comparing index positions

A product's price is replicated with a monotonically increasing log index. The leader's last committed index is 582. A follower in a remote region has an applied index of 579, three entries behind due to ordinary asynchronous replication lag.

• Strict read request: the system requires the answering replica to show an applied index of at least 582 before answering. The remote follower, at 579, does not qualify, so this read is served either by the leader directly, or the follower must first catch up to 582 (via read-index or a direct proxy).
• Bounded-staleness read request, tolerance = 10 positions: the remote follower's applied index (579) is only 3 behind the leader's 582, well within the stated tolerance of 10, so it answers locally with no round trip to the leader at all.

This is the actual mechanism behind "reduce latency for reads that don't need the freshest value": comparing an explicit applied-index number against an explicit, stated tolerance, not a vague notion of "probably fresh enough."

Trade-offs & pitfalls

The most common pitfall across all three is exposing a fast path and a strong path as separate API calls and then trusting the client to pick correctly every time; a safer design escalates automatically, for example retrying as a strong read when a fast read's own returned version looks suspiciously far behind what was expected, rather than relying solely on the caller's judgment.

Direct Answer

Eventual consistency means that if a piece of data stops changing, every copy of it, spread across different machines, will eventually show the same value, but there's no promise about how quickly that happens. Right after something changes, different copies can briefly disagree, so different people, or even the same person on different devices, can see different things for a short window.

Three Concrete Things Users Will Notice

1. A change that looks like it disappeared or reverted. You update something, say your profile bio, and it saves fine, but a moment later, on a different device or after a refresh, you briefly see the old version again. This happens because that device happened to read from a copy of the data that hadn't caught up yet, not because your change was lost. The same effect shows up in less obviously social products too: right after a recommendation or personalization model is updated, some requests can still be served by a copy of the system using the old values for a short window, so two people who do the exact same thing a minute apart can get visibly different recommendations, purely because of which copy answered them.

2. Actions that look duplicated. If a user doesn't get quick feedback that their action went through (a like, a form submission), they often retry it. If the retry and the original attempt both eventually land, the user can end up seeing what looks like two of the same action. This isn't really an eventual-consistency artifact on its own; it becomes a real duplicate unless the system also deduplicates the underlying writes, not just the on-screen display.

3. Optimistic updates that hide the delay, until they don't. Many products make the delay invisible to the person taking the action by updating their own screen immediately, before the write has actually finished spreading to other copies. For example, when you post a comment, it appears in your own feed the instant you hit submit, even though the write is still propagating to the copies that other users' feeds are reading from. This makes the product feel instant for the person who acted, but it means other people may not see that comment for a moment, and if the underlying write ultimately fails, the app has to quietly roll back the comment it optimistically showed you.

A Concrete Trace

Say a comment-posting service has two copies of the feed data, one near user A and one near user B. User A posts "Great point!". Step 1: A's client shows the comment in A's own feed immediately, the optimistic update, while the actual write is sent to A's nearby copy. Step 2: User B, served by their own nearby copy, refreshes their feed before the write has replicated over to B's copy; B does not see the comment yet. Step 3: once the write has replicated to B's copy, B's next refresh does show the comment. Nothing was lost; B was simply reading from a copy that hadn't caught up at step 2.

Trade-offs and What to Plan For

• Eventual consistency is a deliberate trade for availability and responsiveness, not a bug, but it is the wrong choice for data where a stale answer is actively harmful, such as an account balance, the last unit of inventory, or a security permission change. Those flows are usually worth paying for stronger consistency even if it's slower.
• A common and cheap mitigation for the "did my own change disappear" complaint is guaranteeing read-your-writes (RYW): making sure the person who just made a change always sees their own latest write, typically by routing their own subsequent reads back to the copy that has it, even while other users' view of that same data is still catching up.
• A common wrong turn is treating optimistic UI as if it solves eventual consistency; it only hides the delay from the person who acted. It doesn't change how long the write actually takes to reach everyone else, and it adds its own failure case, rolling back a shown-then-failed action, that the product needs to handle gracefully.

Raft replicates a log by having the leader append every client command locally, then replicate it to followers via AppendEntries RPCs; once a majority, including the leader, has stored an entry, the leader marks it committed and applies it to its state machine, and followers catch up the same way once they learn the new commit index. A slow or long-disconnected follower gets caught up either by incrementally backfilling AppendEntries, or, if it has fallen too far behind, by receiving a full InstallSnapshot of compacted state. The reason two leaders can never commit conflicting entries at the same log position is not a single rule, it is two rules working together: how logs are kept consistent, the log-matching property, and who is even allowed to become leader, the election restriction.

Terms and leader election, brief recap

Each term has at most one leader, elected by majority vote (covered in depth elsewhere in this material); what matters for replication is just that every entry in the log is tagged with the term of the leader that created it.

AppendEntries and the commit rule

• The leader sends AppendEntries(prevLogIndex, prevLogTerm, entries, leaderCommit) to each follower. A follower only accepts the new entries if its own log already has an entry at prevLogIndex with term prevLogTerm, matching exactly. If it does not match, the follower rejects, and the leader backs off to an earlier index and retries; this is what repairs a follower whose log has diverged.
• An entry counts as committed once it, and transitively every entry before it, is stored on a majority of servers, and that entry was created in the leader's own current term. That second condition, from the leader's own term, is the subtle part; see the worked example below for why it is necessary.
• Once committed, the leader applies the entry to its state machine and advances leaderCommit, which followers learn on the next AppendEntries, including empty heartbeats, and apply locally too.

Log compaction via snapshotting

• When a follower has fallen far enough behind that the leader has already discarded, or compacted, the log entries it needs, the leader sends InstallSnapshot instead: a full compacted copy of state plus the index and term of the last entry included in it.
• The follower replaces its own log prefix with the snapshot and adjusts its own next-expected index accordingly, then resumes normal AppendEntries catch-up from there.

Why two leaders can never commit conflicting entries at the same position, in plain terms

Two facts, combined, make this impossible:

1. Log matching: if two servers' logs both contain an entry at the same index with the same term, every entry up to and including that index is identical on both. This holds because a leader never reorders or overwrites its own log during its term, it only appends, and every follower's AppendEntries acceptance check requires the previous entry to match exactly before accepting anything new.
2. Election restriction: a candidate can only win a vote from a node whose log is at least as up to date as the candidate's own, compared by last entry's term, then index. Since winning requires a majority of votes, and any already-committed entry was, by definition, stored on a majority, at least one voter in any winning majority must already hold that committed entry. Any two majorities out of the same n servers are guaranteed to overlap in at least one server:

$|Q_1| + |Q_2| > n \implies Q_1 \cap Q_2 \neq \emptyset$

That overlap is exactly why whoever wins an election has a log at least as up to date as a node that already holds any previously committed entry, so the new leader must already have it too. A new leader can extend the log with new entries, but it can never come into office missing, and therefore able to overwrite, something the cluster already agreed on.

The genuinely tricky part, and the reason committed equals majority replicated is not the whole rule, is entries a leader replicated to a majority before it finished committing them in its own term and then crashed. A later leader, elected without ever having seen that specific entry, could otherwise legally overwrite it, which is why Raft only counts an entry as truly committed once a majority holds an entry from the current leader's own term: committing that later entry transitively commits everything before it, by log matching, without ever needing to separately reason about whether an older, not-yet-current-term entry was safe on its own.

Worked example: why from the leader's own term matters, printed trace

Five servers, S1 through S5.

• Term 2: S1 is leader. It appends an entry at log index 2, call its value 'x', and replicates it to S1 and S2 only, 2 of 5 servers, not a majority, before S1 crashes.
• Term 3: S5 becomes leader; S5's log does not have 'x' yet, and that is fine, since 'x' was never committed and was never required for S5's election vote to be up to date. S5 overwrites index 2 with its own entry, 'y', replicates it to S5 and S3, 2 of 5, still not a majority, then S5 also crashes before committing it.
• Term 4: S1 restarts and is re-elected leader; its log, containing the old 'x' at index 2, is still eligible since 'y' was never committed to a majority either. S1 now replicates its old index-2 entry 'x' to a majority: S1, S2, and S3, 3 of 5. Index 2 now genuinely has 'x' on a majority of servers.

Here is the crux: if Raft's rule were simply committed equals majority replicated, index 2 (holding 'x') would now count as committed, purely because a majority happens to hold it, even though it was replicated under a different leader's term, term 2, than the one currently in power, term 4. Raft explicitly does not commit it on that basis alone. Only once S1, as term-4 leader, replicates a new entry from term 4 to a majority does index 2 become safely committed, as a side effect of committing that later entry, since log matching guarantees everything before it, including 'x' at index 2, is identical across that same majority too. Without this extra rule, a hypothetical term-5 leader that had never seen 'x' could still legally overwrite index 2 yet again, since nothing had actually locked it in, and that overwrite would silently undo something an operator might already have treated as durably committed.

sequenceDiagram
    participant L as Leader (term 4)
    participant F1 as Follower S2
    participant F2 as Follower S3
    L->>F1: AppendEntries(index=3, term=4)
    L->>F2: AppendEntries(index=3, term=4)
    F1-->>L: success
    F2-->>L: success
    Note over L: majority (3 of 5 incl. leader) now hold index 3 from term 4, so index 3 commits, and index 2 commits transitively

Trade-offs & pitfalls

• Common wrong turn: explaining commit as majority replication, full stop. That is necessary but not sufficient; the current-term condition above is exactly what closes the gap, and is the part people skip when reciting Raft from memory.
• Followers that reject an AppendEntries because prevLogIndex or prevLogTerm do not match force the leader to step backward and retry; a purely one-at-a-time backoff is simple but slow for a follower that has fallen far behind, which is part of why snapshotting exists as an alternative catch-up path rather than only incremental backfill.
• Operational note: watching how often followers need a full InstallSnapshot, versus incremental catch-up, is a useful signal that a follower, or the network path to it, is chronically slow. This applies just as well to a Raft-backed machine learning model-metadata store as to any other Raft-based service; the underlying signal is the same regardless of what the log stores.

Raft's leader election works by giving every follower a randomized election timeout; whichever follower's timeout fires first becomes a candidate, bumps the term number, votes for itself, and asks every other node for its vote. Randomizing the timeout is what keeps split votes rare: it makes it unlikely that two followers time out at nearly the same instant and both start competing candidacies in the same term. A short timeout detects a dead leader faster but risks more split votes and RPC churn; a long timeout is calmer but leaves the cluster leaderless longer after a real crash.

Approach

Implement the state machine directly: a follower's only action is to reset its timer whenever it hears from a leader or grants a vote. When the timer fires with no such contact, it becomes a candidate, increments its term, votes for itself, and fires RequestVote RPCs at every peer in parallel. It counts granted votes as they arrive; on a majority, it becomes leader. Any RPC response or incoming request carrying a higher term makes it step down to follower immediately; this term-monotonicity is what prevents two nodes from believing they are both leader for the same term.

type Server struct {
    mu            sync.Mutex
    id            int
    peers         []int
    currentTerm   int
    votedFor      *int
    state         string // "follower", "candidate", "leader"
    votes         int
    electionTimer *Timer
}

func (s *Server) onElectionTimeout() {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.state == "leader" {
        return // leaders don't run elections; they send heartbeats instead
    }
    s.startElectionLocked()
}

func (s *Server) startElectionLocked() {
    s.state = "candidate"
    s.currentTerm++
    s.votedFor = &s.id
    s.votes = 1 // vote for self
    term := s.currentTerm
    s.resetElectionTimer()

    for _, peer := range s.peers {
        go func(peer int) {
            resp := sendRequestVote(peer, s.id, term, s.lastLogIndex(), s.lastLogTerm())
            s.mu.Lock()
            defer s.mu.Unlock()
            if resp.Term > s.currentTerm {
                s.currentTerm = resp.Term
                s.becomeFollowerLocked()
                return
            }
            if s.state != "candidate" || term != s.currentTerm {
                return // stale response from a prior term, or we already moved on
            }
            if resp.VoteGranted {
                s.votes++
                if s.votes > (len(s.peers)+1)/2 {
                    s.becomeLeaderLocked()
                }
            }
        }(peer)
    }
}

func (s *Server) resetElectionTimer() {
    d := randomDuration(electionTimeoutMin, electionTimeoutMax)
    s.electionTimer.Reset(d)
}

func (s *Server) onAppendEntries(args AppendEntriesArgs) AppendEntriesReply {
    s.mu.Lock()
    defer s.mu.Unlock()
    if args.Term >= s.currentTerm {
        s.currentTerm = args.Term
        s.becomeFollowerLocked()
        s.resetElectionTimer() // heard from a valid leader, don't start a new election
    }
    // log-matching and append logic omitted here
    return AppendEntriesReply{Term: s.currentTerm}
}

Key points

• Persist currentTerm and votedFor to stable storage before replying to any RPC. A node that crashes and restarts without doing this could grant a second vote in a term it already voted in, which is a real safety violation, not just a liveness inconvenience.
• A candidate only votes for itself and only requests votes once it becomes a candidate; a follower that already voted for someone else this term must refuse any other RequestVote for that same term.
• Majority is computed against the full cluster size n, including the leader itself, not just the peers list:

$\text{majority} = \lfloor n/2 \rfloor + 1$

Complexity

• Message complexity: one election attempt sends O(n) RequestVote RPCs, one per peer, linear in cluster size per attempt.
• State per node is O(1) beyond the log itself: a handful of integers and a timer.
• The number of election attempts before success is not bounded in the worst case, repeated collisions are theoretically possible forever, but the randomized timeout range makes the expected number of retries small in practice, since each retry independently redraws a new random timeout, making the same two nodes colliding again increasingly unlikely round over round.

Worked example: split vote and recovery, printed trace

Three nodes N1, N2, N3, currently term 0, no leader. Each node draws its election timeout independently from the same range.

Case A, no collision, the common case: N2 draws the shortest timeout and fires first. It becomes a candidate for term 1, votes for itself, and sends RequestVote(term=1) to N1 and N3. Neither has timed out or voted this term, so both grant their vote. N2 now has 3 of 3 votes, itself plus two grants, exceeding the majority threshold of 2, so N2 becomes leader for term 1. N1 and N3 reset their timers on granting the vote, so neither starts its own candidacy.

Case B, a genuine split vote: suppose instead N1, N2, and N3 all draw timeouts close enough together that all three fire before any RequestVote arrives. Each becomes a candidate for term 1, votes for itself, and requests votes from the other two. Because each node already voted for itself in term 1 before any request from a peer arrived, each rejects the other two's requests, having already voted this term. No candidate reaches a majority: N1 has 1 vote, N2 has 1 vote, N3 has 1 vote. All three time out again and start a new election for term 2, each independently drawing a fresh random timeout. Because these new timeouts are drawn independently, the chance that all three collide again is much lower than the first time, and typically one of them fires meaningfully before the others and wins term 2 uncontested.

Edge cases

• A network partition splits the cluster into a majority side and a minority side: the minority side's nodes keep timing out and calling elections forever, incrementing the term each time, but can never reach a majority, so they correctly never elect a leader. This is deliberate: it is what prevents split-brain, at the cost of the minority side being unavailable for writes until the partition heals.
• A node that crashes and restarts must reload currentTerm and votedFor from stable storage before participating again; skipping this can let it vote twice in a term it already voted in before crashing.
• A slow node, a garbage-collection pause or a disk stall, is indistinguishable from a crashed node from its peers' point of view; both simply stop sending heartbeats, and the same election timeout mechanism reacts to both identically, which can trigger an unnecessary election when a node was merely paused, not dead.
• Election timeouts are purely local, measured intervals, a node's own clock counting elapsed time, so cross-machine clock skew, which matters a great deal for wall-clock timestamp ordering elsewhere in this topic, does not matter here.

Trade-off: short vs long election timeout

• A short timeout detects a genuinely dead leader faster, a shorter unavailability window, but increases both the chance of a split vote and the RPC and CPU overhead from repeated election attempts under network jitter.
• A long timeout keeps the cluster calmer, fewer accidental elections during transient network hiccups, at the direct cost of a longer window with no leader after a real crash, since followers simply wait longer before noticing.
• In practice the timeout range is chosen relative to the network's expected round-trip time: long enough that a heartbeat reliably beats the timeout under normal jitter, short enough that a real leader loss is noticed promptly. There is no universal constant; it has to be tuned to the deployment's actual network characteristics.

stateDiagram-v2
    Follower --> Candidate: election timeout fires
    Candidate --> Candidate: split vote, retry next term
    Candidate --> Leader: receives majority of votes
    Candidate --> Follower: sees higher term or valid leader
    Leader --> Follower: discovers higher term

Linearizability, sequential, causal, and eventual consistency are four progressively weaker guarantees about the order in which operations on shared data appear to happen. Linearizability makes every operation look instantaneous and match real, wall-clock time. Sequential consistency drops the real-time requirement but still gives every observer the same single global order. Causal consistency only orders operations that are actually cause-and-effect related, letting unrelated operations be seen in different orders on different replicas. Eventual consistency drops ordering guarantees almost entirely and only promises that replicas converge once writes stop. Each weaker model permits more anomalies than the one above it.

Worked example: why causal consistency prevents an anomaly eventual consistency allows

Consider a social feed. Two events happen, in this order, involving the same user's friend:

1. Event P: a user publishes Post P.
2. Event C: after reading Post P, the user's friend writes Comment C, which references Post P.

Because the friend read P before writing C, C causally depends on P: P happened-before C.

• Under causal consistency, any replica that delivers C to a reader must already have delivered P to that same reader. There is no way for a client to see Comment C replying to Post P without also being able to see Post P: the system enforces the happened-before relationship on delivery.
• Under eventual consistency alone, P and C might replicate along different paths (different shards, different network routes) with no ordering guarantee between them. A reader on a lagging replica could receive C's replication packet before P's, and briefly render a comment that references a post the reader's own client cannot find yet, an orphaned reply. That is exactly the anomaly eventual consistency does not rule out and causal consistency does.

Because eventual consistency only promises the base table converges, a secondary index or materialized view (for example, a 'comments by post' index used to render the feed) can lag the base write for an unbounded window: the index might still return zero comments for Post P for some time after Comment C has already durably landed on a majority of the base replicas, since building the index from the base table's write stream is itself an eventually-consistent process, not an atomic one.

Trade-offs & pitfalls

• Common wrong turn: treating eventual consistency as one well-defined guarantee. It is really the absence of a guarantee during the convergence window, so two systems both labeled eventually consistent can behave very differently depending on how long that window typically is, and what session-level guarantees (read-your-writes, monotonic reads) are layered on top.
• Sequential consistency is rarely offered as a named product feature; it mostly shows up as an accidental byproduct of serving reads from any replica of a system that internally agrees on a single write order, without adding a real-time freshness check on the read path.
• Causal consistency requires tracking dependencies, commonly via vector clocks or similar metadata, which costs storage and complicates garbage collection, the same trade-off logical clocks introduce elsewhere in this material.
• Senior answers name the actual anomaly each model still allows, not just that it is looser. An answer that only says eventual is looser than causal, without naming a concrete permitted anomaly, is incomplete.