Module 24 / 46 · Phase E — Distributed Reality · 50 min

Consensus &
the Raft
protocol.

Five nodes. One total order to agree on. Some of them will fail. The network will lie. And somehow the system must keep ticking, keep accepting writes, and never serve two different stories. Raft is how the modern world solved it.

// What you'll know by the end

  • Why consensus is the hardest problem in distributed systems
  • The three sub-problems Raft decomposes it into
  • How leader election + log replication actually work
  • Why etcd, Consul, and CockroachDB all chose Raft
§ 01 — The hardest problem in distributed systems

"Five nodes.
One truth."

You have five servers. A client sends a write. How do you make sure all five agree on whether — and in what order — that write happened? Sounds trivial until you remember: any server can crash mid-conversation, the network can drop or duplicate any message, and clocks can disagree by hundreds of milliseconds (M.23). Now multiply the uncertainty by every concurrent client, every concurrent write, every reorganization caused by a node dying. This problem has a name. The name is consensus. The literature calls it "the hardest problem in distributed systems" for a reason.

What makes consensus hard isn't just the failure modes — it's that the algorithms to solve it are extraordinarily hard to get right. Lamport published the seminal solution (Paxos) in 1989, and for the next two decades the dominant feedback from implementers was: "we tried to build this and we couldn't." There's a famous 2007 Google paper, Paxos Made Live, where the engineers explain they had to write thousands of lines of code to make a "proven correct" algorithm actually work — and even then, edge cases kept emerging in production. Distributed consensus wasn't impossible. It was just impossible to teach.

// THE LINEAGE OF CONSENSUS
YEAR
ALGORITHM
VERDICT
1985
FLP Impossibility (Fischer-Lynch-Paterson) proves async consensus with even one failing node is impossible without some form of timing assumption.
discouraging but foundational
1989
Paxos (Lamport). The first working consensus protocol. Solves the problem. Almost impossible for normal humans to implement correctly.
correct but cryptic
1998
"The Part-Time Parliament" — Lamport's allegorical Paxos paper, written in 1990, was so confusing it took the journal 8 years to publish.
the difficulty made famous
2007
"Paxos Made Live" (Google) — engineers detail how they actually built it. Concludes: even with a proven correct algorithm, there were thousands of subtle traps.
a cry for help
2014
Raft (Ongaro & Ousterhout, Stanford). Same correctness guarantees as Paxos. Explicitly designed for understandability. Subtitle of the paper: "In Search of an Understandable Consensus Algorithm."
finally teachable

Raft's revolution wasn't a new theoretical result. It was a pedagogical breakthrough. Ongaro & Ousterhout took the same problem Paxos solved — distributed consensus under crash failures — and decomposed it into pieces that human engineers could actually hold in their heads. Then they built a real implementation, ran user studies (literally — they tested how well students learned Raft vs Paxos), and showed Raft was dramatically more learnable. Within five years, virtually every new distributed system chose Raft over Paxos. etcd, Consul, CockroachDB, TiKV, ScyllaDB, Hashicorp Vault — the modern distributed world runs on Raft.

Raft's breakthrough wasn't a new theorem. It was making the theorem fit in a human head.

The rest of this module gives you Raft in the same decomposition the original paper uses: roles & terms (§02), the three sub-problems (§03), an interactive 5-node simulator that walks through normal operation and the failure modes (§04), and the safety properties Raft guarantees (§05). By the end you'll be able to read the etcd source code and recognize what every function is doing.

§ 02 — Roles & terms

Three roles.
One term clock.

Raft's first design decision is to give every node one of three roles at any moment: Follower, Candidate, or Leader. Most of the time, exactly one node is a Leader and all others are Followers. This single-leader design is what makes Raft so understandable — once you accept that a leader exists, the work it must do is comparatively simple. The complexity is concentrated in electing a new leader when the old one fails, and the role machinery makes that explicit.

// THE THREE ROLES · AND HOW NODES MOVE BETWEEN THEM

F
Follower
"I'm waiting for instructions."

The default state. Receives AppendEntries from the leader (carrying log entries and heartbeats), votes on requests from candidates, but never initiates anything itself. Most nodes are followers most of the time.

Candidate if election timeout fires with no leader heartbeat
C
Candidate
"I'm running for leader."

A temporary state during elections. Increments the term number, votes for itself, and sends RequestVote RPCs to all other nodes. If it gets votes from a majority, it becomes the leader. If it sees a higher term, it steps down. If election times out, restarts.

Leader if wins majority
Follower if sees newer term or another wins
L
Leader
"I drive the cluster."

Handles all client requests. Appends entries to its log and replicates them to followers via AppendEntries. Sends periodic heartbeats (also via AppendEntries, just with no new entries) to prevent followers from starting elections. Exactly one leader per term — guaranteed.

Follower if discovers a higher term anywhere

The other half of the role machinery is the term number. Every node tracks a current term — a monotonically increasing integer, starting at 0. Each election bumps the term by one. Terms are Raft's logical clock, in the Lamport sense from M.23: they let nodes detect stale information ("I'm hearing from a leader claiming term 5, but I know we're already on term 7 — that leader is from the past, ignore"). Every RPC carries a term number; receivers always compare it to their own. The rule is simple: higher term always wins, and on seeing a higher term, you immediately become a follower. This single rule eliminates entire classes of split-brain failures.

Higher term always wins. On seeing one, you become a follower. That single rule kills split-brain.

Worth pausing here: Raft is "leader-based" consensus. The leader is the bottleneck and the single point of decision. Some distributed systems philosophers find this offensive — surely true democracy means peer-to-peer with no leaders? — but the simplicity dividend is enormous. Raft solves the hard problem (agreement during failures) by reducing it, most of the time, to a one-machine problem: "the leader decides; everyone else just follows." Then the genuinely hard part is making sure a new leader emerges quickly and safely when the old one dies. That's where the meat of the algorithm lives, and that's what §03 unpacks.

§ 03 — The three sub-problems

Decompose
the elephant.

The Raft paper's central pedagogical move is decomposing the consensus problem into three nearly-independent sub-problems. You can understand each one separately. Combine them and you get the full algorithm. The same decomposition is in the original paper's table of contents — and that table of contents is the curriculum every Raft tutorial follows.

// THE THREE SUB-PROBLEMS · RAFT'S TABLE OF CONTENTS

1
Leader Election
"How does a new leader emerge when the old one fails?"

Followers start election timers with randomized timeouts (150-300ms). When a timer expires without hearing from a leader, that follower becomes a candidate, increments the term, votes for itself, and sends RequestVote RPCs to every other node. The randomization breaks ties — usually one candidate's timer fires noticeably earlier, and it wins the election uncontested. If majority votes arrive, it becomes leader. If split-vote occurs (rare but possible), all candidates time out, randomize again, retry.

2
Log Replication
"How does the leader keep all followers' logs in sync?"

The leader appends client commands to its own log, then sends AppendEntries RPCs to all followers with new entries. Once a majority of followers acknowledge, the leader commits the entry (applies it to the state machine and tells the client). The next round of AppendEntries tells followers the new commit index, and they apply too. If a follower has fallen behind, the leader sends it the missing entries — going backward until it finds a matching point. This is essentially log shipping with majority quorum.

3
Safety
"How do we ensure the cluster never serves two different truths?"

The hard one. Multiple invariants conspire to prevent the worst failures: election restriction (a candidate only wins if its log is at least as up-to-date as a majority), commit rule (a leader can only commit entries from its current term), log matching (if two logs agree on an index+term, they agree on everything before it). Together these guarantee that once an entry is committed, no future leader can ever lose or override it.

The Raft paper devotes about ten pages to each sub-problem. The simulator below walks you through three of those concretely — leader election from a cold start, log replication of a client write, and what happens when the leader fails. After you've stepped through all four scenarios, the algorithm in your head will match the algorithm in the paper. That's the whole goal of Raft's design.

§ 04 — Raft cluster simulator · interactive lab

Watch five nodes
find their leader.

Below: a 5-node Raft cluster. Pick a scenario. Step through the events one at a time. Watch nodes change state (F→C→L), terms increment, votes flow, log entries replicate. The right panel narrates what just happened and why. By the end of the four scenarios, the algorithm is concrete — not abstract.

RAFT_CLUSTER.SIM // m.24 lab
Step 0 / 0
// 5-NODE CLUSTER · live state
SCENARIO READY
Press Next to start
"Step through the scenario to watch Raft unfold."
Each scenario walks through a sequence of events on the cluster. Use the step controls above to advance.
// node legend
FOLLOWER — receiving from leader
CANDIDATE — running election
LEADER — driving cluster
DOWN — crashed or partitioned
// committed log · cluster-wide
No entries committed yet.
§ 05 — Safety, formally

Five invariants.
Never violated.

The mechanics of leader election and log replication are easy to grasp. The deeper achievement of Raft is its safety guarantees — formally stated, formally proven, never broken even across the worst combination of failures. These are the properties that let etcd, Consul, and CockroachDB build production databases on top of Raft without paranoia. If any of these were violated, the cluster could lose or duplicate writes. They never are.

// THE FIVE SAFETY PROPERTIES · FROM THE RAFT PAPER

// PROPERTY 1
Election Safety

At most one leader can be elected in a given term. Because each node only votes once per term and a candidate needs majority to win, two candidates in the same term cannot both succeed — there aren't enough votes for both.

// PROPERTY 2
Leader Append-Only

A leader never overwrites or deletes entries in its own log. It only appends new entries. This rules out an entire class of bugs where a confused leader could "edit history."

// PROPERTY 3
Log Matching

If two logs contain an entry with the same index and term, the logs are identical in all preceding entries. This is what makes AppendEntries's "previous entry check" work: matching just one index+term back proves all earlier history matches too.

// PROPERTY 4
Leader Completeness

If an entry is committed in term T, then all leaders of all higher terms contain that entry. Achieved by the election restriction: a candidate only wins if its log is at least as up-to-date as a majority. Committed entries are on the majority, so the next leader has them.

// PROPERTY 5
State Machine Safety

If a node has applied a log entry at a given index to its state machine, no other node will ever apply a different entry at that index. This is the "single truth" guarantee that consumers of Raft (etcd, etc.) rely on. Falls out from Properties 3 and 4.

The proof structure is worth knowing even if you'll never write it: Properties 1 and 2 are direct from the algorithm (votes are limited, leaders never delete). Properties 3, 4, and 5 build on each other: log matching enables leader completeness via the election restriction, and the two together give state machine safety. The whole stack is about two pages of proof in the original paper. Compare that to Paxos, where the equivalent proof spans dozens of pages across several papers. This is the second part of Raft's pedagogical win — not just that the algorithm is understandable, but that the safety arguments are too.

One thing Raft doesn't guarantee, worth flagging: liveness during arbitrary network conditions. If the network keeps dropping RequestVote messages in just the wrong way, the cluster can keep timing out, electing, timing out, electing, without ever stabilizing. In practice this is extraordinarily rare — the randomized timeouts make it nearly impossible — but in pathological networks, Raft (like all consensus protocols) trades liveness for safety. You'll never lose data; you might briefly be unable to make progress. That trade-off is correct, and it's the right one for the systems Raft serves.

§ 06 — Eight words for the consensus conversation

Vocabulary,
for the leader.

Every term that appears in the Raft paper and every conversation about etcd, Consul, or CockroachDB. Lock these in.

Consensus
/kənˈsen.səs/
The problem of getting multiple nodes to agree on a single value (or sequence of values) despite failures and network unreliability. Provably impossible in fully async networks (FLP) — solved with timing assumptions in practice.
Term
/tɜːrm/
A monotonically increasing integer marking each election in Raft. Acts as a logical clock for the cluster. Higher term always wins; a node seeing a higher term immediately becomes a follower.
RequestVote
/rɪˈkwest voʊt/
The RPC sent by a candidate to all other nodes asking for their vote in the current election. Includes the candidate's term and log info so receivers can validate freshness.
AppendEntries
/əˈpend ˈen.triz/
The RPC the leader uses for everything: replicating new log entries, syncing followers that are behind, and (when empty) acting as heartbeat to prevent elections. One RPC, three purposes.
Election Timeout
/ɪˈlek.ʃən ˈtaɪm.aʊt/
The random interval (typically 150-300ms) a follower waits without hearing from the leader before starting its own election. The randomization is crucial — it prevents simultaneous elections by spreading out timer expirations.
Quorum
/ˈkwɔː.rəm/
In Raft, a strict majority: floor(N/2) + 1. Needed both to win elections and to commit log entries. Two quorums always overlap, which is the mathematical foundation of safety.
Log Entry
/lɒɡ ˈen.tri/
A single replicated command in Raft's log: (term, index, command). Once committed (majority acknowledged), durable forever. The cluster's source of truth.
Commit Index
/kəˈmɪt ˈɪn.deks/
The highest log index known to be committed (majority-acked). The leader tracks this and passes it to followers via AppendEntries. Entries below the commit index are applied to the state machine; entries above wait for majority.
§ 07 — Knowledge check

Five questions.
Defend the leader.

Test the Raft instinct. Click an answer; explanation drops in instantly.

QUESTION 1 OF 5
Loading question...
Score: 0 / 5
5 / 5

Elected.

Perfect. Raft's algorithm is now concrete in your head. Next: the broader replication landscape — multi-leader, leaderless, and what trade-offs each makes.

§ 08 — The recap

Three ideas to
carry forward.

The consensus protocol that took over the distributed world. Every module forward will reference it.

i

Leader-based wins

Raft elects a single leader who makes all decisions. The cluster reduces to a one-machine problem most of the time. Simple to reason about — and the simplicity is the whole point.

ii

Terms kill split-brain

Every election bumps a logical term clock. Higher term always wins; nodes seeing it immediately follow. This single rule prevents the worst class of distributed bugs.

iii

Quorum + log matching = safety

Five safety properties. Two from the structure (one leader per term, append-only). Three from the math (log matching, leader completeness, state machine safety). The proof fits in two pages.

↓ UP NEXT

M.25 — Replication
patterns.

Raft is one approach to replication — single-leader, strongly consistent. There are others: multi-leader (every node accepts writes), leaderless (Dynamo-style quorum). Each makes different trade-offs against the CP/AP spectrum. The closing module of Phase E.

Continue to Module 25 →