The idea: one primary, one term at a time
A MongoDB replica set is a few mongod processes that hold the same data.
One of them is the primary: it takes every write and records it in its
oplog, a capped collection of entries with a timestamp ts and a
term. The others are secondaries: they pull the oplog and replay it.
When the primary fails, the secondaries elect a new one. Since MongoDB 3.2 the
election protocol is protocolVersion: 1, which is close to Raft:
every election starts a new term, each member votes at most once per term, and a member only votes for
a candidate whose oplog is at least as recent as its own.
The canvas shows a set of three members, A, B and C, with one vote and priority 1 each, and the app
(the driver) that sends insertOne to the primary. Each panel shows a member's state, its
term, its last vote, its election timer, its lastApplied optime and the
majority commit point it knows, and the end of its oplog. Green entries are
majority-committed; yellow ones are not yet. The same users collection as on
How MongoDB Runs a Command is used, so the two pages read as one story.
The clock only moves when you press Wait 12 s; everything else takes milliseconds.
Heartbeats and the election timeout
Every member sends a heartbeat (replSetHeartbeat) to every other member every
heartbeatIntervalMillis = 2 s. A heartbeat answer carries the sender's state, term, optime
and commit point. Each time a secondary hears the primary, it resets its election timer.
If the timer reaches electionTimeoutMillis (10 s by default) without word from a primary, the
secondary stands for election. MongoDB adds a random offset of up to 15 % of the
timeout, so two secondaries rarely start at the same moment. The page fixes the offsets (A 0.6 s,
B 1.2 s, C 0.3 s) so the demos repeat exactly. A dead member is not noticed at once: in
Demo: primary dies, election, A dies at t = 0, the heartbeats at t = 2, 4, 6, 8 and 10 s get no
reply, and C's timer fires first at t = 10.3 s.
Dry run, votes and the "at least as recent" rule
A candidate first runs a dry run: it sends replSetRequestVotes with
dryRun: true and its term + 1, without changing its own term. Only if a majority would
vote yes does it raise its term, vote for itself, and ask again for real. The real vote is written to
disk (local.replset.election), so a member that restarts cannot vote twice in one term.
The dry run matters for a member that is cut off: it keeps losing dry runs but never raises the term,
so it does not disturb the healthy side when it comes back.
A voter answers no when the candidate's term is older than its own, when it has already voted in that term, or when the candidate's last optime (term, then ts) is behind its own. That last rule is the safety rule. In the demo, C is slow and misses ts 4; the write is still majority-committed on A and B. When C asks B for a vote, B answers "your oplog (ts 3, t1) is behind mine (ts 4, t1)". C gets 1 of 3 votes, stays SECONDARY and keeps term 1. At t = 11.2 s B's timer fires, C votes for B, and B becomes PRIMARY of term 2.
Why this protects majority writes: a write acknowledged with w: "majority" is on 2 of 3
members. Any winner needs 2 of 3 votes. The two groups share at least one member, and that member
refuses any candidate that lacks the write. So every future primary has it. A new primary also writes a
no-op "new primary" entry in its own term, because the commit point only moves on to an
entry of the current term (the same rule as Raft's Figure 8).
The majority commit point and write concern
Secondaries pull with a getMore on a tailable cursor over the primary's oplog, apply the
batch and report their position with replSetUpdatePosition. When 2 of 3 members have an entry,
the primary moves the majority commit point to it and passes it on in the next batch and
heartbeat (Demo: w: majority write, commit point).
| Write concern | The app gets its ack when | After a failover |
|---|---|---|
w: 1 | the primary has applied the write | can be rolled back if no other member had it |
w: "majority" (default since 5.0) | the commit point has passed the write (journaled on 2 of 3) | never rolled back |
Rollback of the old primary
In Demo: w: 1 write rolled back, A acknowledges {_id: 11, name: "Kim"} as ts 4
with w: 1 and crashes before B and C fetch it. C wins term 2, writes its no-op as ts 5, and the
app writes Lea as ts 6. When A restarts, it hears C's heartbeat with term 2 and picks C as its sync
source. Its newest entry (ts 4, t1) is not in C's oplog. A walks back to the common point,
ts 3, the newest entry both oplogs share. It enters ROLLBACK, undoes ts 4, and writes the
undone document to a rollback file
(<dbpath>/rollback/<collection UUID>/removed.<time>.bson). Then it is
RECOVERING while it fetches ts 5 and 6, and becomes SECONDARY again. The app had been told
"acknowledged", yet Kim is gone from the data; only an operator reading the rollback file can bring it back.
Since MongoDB 4.0, rollback uses recover to a timestamp: the member restores its data to the
stable timestamp (its majority commit point) and replays its own oplog up to the common point, instead of
refetching each document. A member refuses to roll back more than rollbackTimeLimitSecs
(24 hours) of writes and needs a full resync instead. In Demo: same with w: majority, nothing lost,
ts 4 is on all three members before the ack, so C's oplog has it too, A's common point is its newest entry,
and there is nothing to roll back.
Priority and takeover
Each member has a priority (0 to 1000, default 1; 0 means "never primary") and
votes (0 or 1). Priority does not change the vote rule. Instead, a secondary that sees a
primary with a lower priority calls a priority takeover election, but
only once it is caught up (within priorityTakeoverFreshnessWindowSeconds, 2 s, of the
primary). In Demo: priority takeover, C is first behind by ts 4, so it fetches it; then it wins
term 2. A grants the vote, sees term 2 > 1 and steps down. The same step-down can be asked for by hand
with rs.stepDown(), which waits until a secondary is caught up before giving up the primary role.
A partitioned primary steps down
In Demo: partition, old primary steps down, the network around A is cut, but the app can
still reach A. A still believes it is primary and accepts an insertOne with
w: "majority" as ts 4, but cannot replicate it, so the app waits. A primary that has not heard
from a majority for electionTimeoutMillis steps down: at t = 10 s A becomes
SECONDARY and the waiting write fails with an error. At t = 10.3 s C wins term 2 on the majority side.
When the network heals, A rolls back ts 4, a write the app was never told had succeeded.
For a short time a cut-off primary and a new primary can both report PRIMARY (MongoDB does not use
leases by default). That is safe for majority writes: the old one cannot get a majority to acknowledge
anything, and reads with readConcern: "linearizable" check with a majority first. Only
w: 1 writes and "local" reads on the old primary can be lost or stale.
More on the trade-off: The CAP Theorem and
Primary–Replica Replication.
Settings on the page
| Setting | Default | On the page |
|---|---|---|
protocolVersion | 1 | terms, dry run, vote rule |
settings.heartbeatIntervalMillis | 2000 | heartbeat rounds every 2 s |
settings.electionTimeoutMillis | 10000 | timer 10 s + offset; primary step-down |
settings.catchUpTimeoutMillis | -1 (no limit) | not shown: a new primary catching up from a newer secondary |
members[n].priority | 1 | Priority 2 button |
members[n].votes | 1 | 1 for every member |
Other election protocols on this site: Raft on five servers, Raft vs Multi-Paxos, and the classic Bully and Ring algorithms.
What reads return during a failover, and how read concern decides it, is on MongoDB Read Concern and Causal Consistency.
What the page leaves out
- Arbiters, hidden and delayed members, and sets with more than 7 voting members.
- The new primary's catch-up phase (
catchUpTimeoutMillis) and catchup takeover. - Chained replication (a secondary syncing from another secondary) and sync source selection.
- Real timestamps (cluster time,
Timestamp(seconds, increment)), the journal and checkpoints (on the MongoDB page). - Random election offsets, network delays, and heartbeats that succeed (only failed ones are drawn).
- Reads: read preference,
readConcernlevels and the driver's server discovery and monitoring.