The idea: certify, don't lock across servers

Group Replication keeps several MySQL servers, the members of a group, in step without a lock manager that spans them. A transaction runs on one member, against its local InnoDB, like on any MySQL server. Only at COMMIT does the member talk to the group: it sends a short description of what the transaction changed. A group communication engine, XCom, puts every such message into one total order. Every member then checks each message in that order with the same rule, certification, and so every member reaches the same verdict: commit everywhere, or roll back. An InnoDB Cluster is such a group of (usually) three members plus MySQL Router in front of it and MySQL Shell to set it up.

Clients C1 and C2 connect to MySQL Router, which sends writes to member s1. s1 runs the transaction locally and, at COMMIT, sends its write set and snapshot to XCom. XCom orders the message in slot #41 and delivers it to s1, s2 and s3, which each certify it and commit or apply it as grp:11.
The transaction runs on one member; only its write set goes through XCom, and every member certifies the same sequence.

The canvas shows three members s1, s2 and s3 with a table t(id, v) of three rows, the clients and Router on top, and the XCom log with the certification info below. grp stands for the group's UUID (group_replication_group_name), so grp:1-10 is the set of GTIDs a member has executed.

Write sets and snapshots

While a transaction runs, the member records two things about it:

  • the write set: a hash (XXHASH64) of every primary key (and unique key) of every row it changed, such as #5e1a for t.id=1;
  • the snapshot version: the member's gtid_executed when the transaction ran, such as grp:1-10. It says which other transactions this one could see.

At COMMIT, before anything is written to the binary log, the member sends the write set, the snapshot and the row changes (the binlog events) to the group in one message. Nothing has changed for anybody else yet.

Total order and deterministic certification

XCom is a Paxos variant (see Paxos; Raft solves the same problem with a leader). A message is decided into a numbered slot once a majority of the members has accepted it. Every member delivers the slots in the same order, with no gaps.

Each member keeps the certification info: for each write-set key, the GTIDs of the last transaction that changed it. On delivery of transaction T:

  • if a transaction not in T's snapshot changed one of T's keys, someone else changed the row after T read it: conflict. T is rolled back on its member, and its client gets ERROR 3101 (HY000): Plugin instructed the server to rollback the current transaction. The other members just drop the message.
  • otherwise T passes: it gets the next GTID, its keys now point to that GTID, its member commits it and the others queue it on their group_replication_applier channel and apply it.

No vote is needed after the ordering. The input is the same, the order is the same and the rule is the same, so the verdict is the same on every member. Demo: multi-primary conflict (ERROR 3101) runs two UPDATEs of id 1 on s1 and s2 at the same time: both have snapshot grp:1-10; T1 comes first in the total order and gets grp:11; T2's key was changed by grp:11, which is not in its snapshot, so it fails on every member. Demo: multi-primary, different rows shows that two concurrent transactions on different keys both pass.

Time runs down. s1 and s2 both update id 1 with snapshot grp:1-10. XCom orders T1 in slot #41 and T2 in slot #42. Every member certifies T1: key #5e1a not changed after grp:1-10, pass, grp:11. Every member certifies T2: #5e1a was changed by grp:11, which is not in its snapshot, conflict, T2 rolled back on s2 with ERROR 3101.
The first transaction in the total order wins; every member sees the same order, so every member rejects the second one.

Single-primary and multi-primary

single-primary (default)multi-primary
Who takes writesthe primary; the others run with super_read_onlyevery member
Conflicting updatesserialized by InnoDB row locks on the primary, so certification practically never failson different members: the later one in the total order fails with ERROR 3101
When a member failsif it was the primary, the majority elects a new onethe others keep taking writes
RouterRW port 6446 → the primaryRW port → any member
Restrictionsnone to speak ofno SERIALIZABLE, care with foreign-key cascades, no concurrent DDL on the same objects

Demo: single-primary write shows the whole path: C1 → Router → s1, local execution, the write set through XCom, certification on all three members, commit on s1 and apply on s2 and s3, and the answer back through Router. The mode can be switched online with group_replication_switch_to_multi_primary_mode(); on the page the mode select starts a fresh group.

Flow control

Certification is cheap; applying is not. A member with a slow disk can fall behind: certified transactions pile up in its applier queue, and reads there return old data. Members exchange their queue sizes once a second. When a queue exceeds group_replication_flow_control_applier_threshold (25000 transactions by default; the certifier queue has its own threshold), the writers get a quota (group_replication_flow_control_mode = QUOTA) and their COMMITs wait until the slow member catches up. Demo: flow control (slow s3) uses a threshold of 2: after two writes s3 is two transactions behind, so the third COMMIT waits until s3 has applied them.

Quorum, expulsion and primary election

XCom decides a slot only with a majority of the current view: 2 of 3, 3 of 5. A member that is cut off from the others can still run transactions locally, but their COMMIT never gets a slot: it hangs. That is what keeps the two sides of a network partition from both accepting writes and drifting apart.

A network cut separates s3 from s1 and s2. Left, the majority side s1 and s2: 2 of 3, they expel s3 and form view s1 s2, where COMMITs are certified and succeed. Right, the minority side s3: 1 of 3, no majority, its COMMIT waits forever.
A partition: the side with a majority goes on as a smaller group; the minority can't commit anything.

Members check each other with heartbeats. A member silent for 5 seconds is UNREACHABLE. After group_replication_member_expel_timeout more seconds (5 in MySQL 8.0.21 and later) the majority agrees on a new view without it. The expelled member, when it notices, goes to ERROR state, rolls back what was waiting and tries to rejoin (group_replication_autorejoin_tries). Joining uses distributed recovery: a donor sends the transactions the joiner is missing over the group_replication_recovery channel (or a full copy with the clone plugin). Demo: partition, the minority blocks.

If the primary leaves, the members of the new view elect a new one with the same rule each: the lowest MySQL version, then the highest group_replication_member_weight, then the lowest server_uuid. The new primary applies its backlog before it accepts writes. Demo: primary crash, election, Router failover. If a failure leaves no majority (two members, one fails), the survivor can't expel anyone and blocks; an operator must form a new group with group_replication_force_members.

MySQL Router and InnoDB Cluster

Applications don't connect to members directly. MySQL Router keeps a metadata cache of the cluster (from the mysql_innodb_cluster_metadata schema and the members' performance_schema.replication_group_members) and offers a read-write port (6446) that goes to the primary and a read-only port (6447) that spreads reads over the secondaries. When the primary changes, Router sends new connections to the new one; connections to the old primary are closed and the application reconnects and retries. Compare plain asynchronous replication, where a failover is a separate tool's job and transactions can be lost: MySQL Replication and Database Replication.

See also How MySQL Runs a Query for what happens inside one member, and Binlog Group Commit for how the commit is written to the binary log.

What the page leaves out

XCom's real structure: each member owns its own slots in turn, and messages are batched and compressed; the Paxos phases themselves are one step here. The certification info holds GTID sets, is garbage-collected once every member has applied a transaction, and also covers unique and foreign keys; the snapshot check is a GTID-set subset test, not a number comparison. group_replication_consistency (EVENTUAL, BEFORE_ON_PRIMARY_FAILOVER, BEFORE, AFTER), message fragmentation, the parallel applier, the certifier queue's flow-control threshold, view-change log events, the clone plugin, ClusterSet and read replicas, and how a local transaction that holds a row lock is killed when a certified remote one needs that row. Timeouts are shown as steps, not seconds, and the flow-control threshold is 2 instead of 25000.