The idea: Kubernetes keeps its whole state in etcd
Every Pod, Deployment, Secret and Lease in a Kubernetes cluster is one key in etcd, for example /registry/pods/default/web-1. Only kube-apiserver talks to etcd. etcd is a small key-value store with two layers. Underneath, a Raft log keeps 3 or 5 members in agreement. On top, an MVCC store (multi-version concurrency control) keeps every version of every key, numbered by one global revision counter. Watches, leases and transactions are built on that counter. Controllers and kubelets use them to follow the cluster's state, and scheduler replicas use them to elect a leader.
| On the page | Meaning |
|---|---|
| CLIENT | kube-apiserver, or a controller or scheduler replica using it: the last request, the reply and its header.revision |
| S1, S2, S3 | etcd members. The colour is the role, as on the Raft page (green leader, grey follower, orange candidate, dark grey down). The cells are the last Raft log entries: index inside, term colour once committed, white while not committed. store revision is the member's own MVCC revision. |
| WATCH and LEASES | an informer watching /registry/pods/, the events it got, and the leases with the seconds they have left |
| MVCC STORE | the leader's keyIndex (key → revisions) and its bbolt rows (revision → key, value, create/mod revision, version, lease) |
Requests take no time on the page. The clock moves only with wait 3 s (for lease TTLs) and with an election (1 s).
A write's path
A Put goes to the leader (a follower would forward it). The leader appends it to its Raft log as a new entry and fsyncs it to its write-ahead log (WAL). Then it sends MsgApp to the followers, which append the entry and fsync it too, then ack. When 2 of 3 members have the entry, it is committed. Only then is it applied to the MVCC store: the revision goes up by one, a row goes into bbolt, and the leader answers with header.revision. The followers learn the new commit index with the next heartbeat and apply it too (Demo 1). Leader election and log repair are on the Raft page. This page only shows them as short steps.
The Raft index and the revision are different counters. Every proposal uses a Raft index, but only a write that changes keys gets a new revision. A lease grant, a compaction, a new leader's empty entry and a transaction whose compare failed all go through Raft without one.
Revisions, versions and tombstones
The store has one revision for everything. It starts at 1 and goes up by one per write transaction (several keys written in one transaction share the main revision and differ in a sub-revision). Every key records three numbers:
| Field | Meaning | Kubernetes name |
|---|---|---|
create_revision | revision of the put that created the key (since its last delete) | (none) |
mod_revision | revision of the last change to the key | metadata.resourceVersion |
version | number of puts since it was created; a delete resets it to 0 | (none) |
A put never overwrites anything. bbolt stores a new row under the key (revision, sub), and the in-memory keyIndex (a B-tree) adds that revision to the key's list. So Get --rev=2 can read the key as it was at revision 2: keyIndex finds the newest revision at or below 2, and bbolt returns that row. A delete is a write too. It adds a tombstone row and ends the key's generation; a later put starts a new generation with a new create_revision (Demo 2).
Compaction
Without cleanup the history grows forever. Compact(C) keeps, for each key, the newest revision at or below C and everything after it. A key whose newest revision at or below C is a tombstone goes completely. After that, a read below C fails with mvcc: required revision has been compacted (Demo 3). kube-apiserver compacts every 5 minutes (--etcd-compaction-interval). Compaction frees space inside the bbolt file but does not shrink it; etcdctl defrag does that.
Watches and informers
A watch asks for every change to a key or a prefix after a revision. Its guarantee is strong: events arrive in revision order, none is missed, none is repeated. Watching from an old revision first replays history from bbolt, then the stream goes live. If the connection drops, the watcher reconnects from last seen revision + 1 and gets exactly what it missed (Demo 4). That only fails if the revision has been compacted in the meantime: then the watch is cancelled, and Kubernetes clients see 410 Gone ("too old resource version") and must list the objects again.
This is how Kubernetes works without polling. An informer lists the objects once (getting the list's revision), then watches from that revision and keeps a local cache up to date. Controllers and kubelets react to these events. kube-apiserver keeps its own watch cache per resource, so thousands of informers share one etcd watch.
Leases
A lease is a timer with an ID. LeaseGrant(TTL) creates it (a Raft entry), and keys can be attached with Put --lease=ID. The owner sends LeaseKeepAlive to keep it alive. KeepAlive goes to the leader only and is not a Raft entry, because only the leader's lessor counts lease time. When the TTL runs out, the leader proposes LeaseRevoke through Raft, and every attached key is deleted in one revision. Watchers see the deletes. A new leader renews every lease to its full TTL, so a failover never expires a lease early. Kubernetes attaches Events to leases (--event-ttl, 1 hour by default), so old Events delete themselves.
Transactions: compare and swap
A txn is If(compares) Then(ops) Else(ops), run atomically in one Raft entry. Two patterns matter:
// optimistic update (how kube-apiserver writes every object) Txn(If(mod_revision(key) == resourceVersion)).Then(Put(key, new)).Else(Get(key)) // take a key only if nobody holds it (election / lock) Txn(If(create_revision(key) == 0)).Then(Put(key, me, lease)).Else(Get(key))
The first is optimistic concurrency. Two controllers read the same object at resourceVersion 2. The first update succeeds and makes it 3. The second fails its compare, gets the current object from the Else branch and retries; Kubernetes reports this as 409 Conflict (Demo 5). The second pattern combined with a lease is a leader key. node-a takes /election/leader, node-b's compare fails, and when node-a stops sending KeepAlive the lease expires, the key is deleted and node-b's next campaign wins (Demo 6). etcd's own lock and election recipes refine this: every candidate writes its own key under a prefix, and the lowest create_revision wins. A lock holder must still guard its writes with a fencing token; see Distributed Locks and Fencing Tokens. Kubernetes controllers actually elect their leader with a Lease object updated by the first pattern; it is not an etcd lease. Other election algorithms are on Leader Election: Bully vs Ring.
Linearizable and serializable reads
By default a read is linearizable: it sees every write that was acknowledged before it started. A leader cannot just read its own store, because it might have been replaced without knowing. With ReadIndex it notes its commit index, checks with one heartbeat round that a majority still follows it, then reads once it has applied that far. A follower asks the leader for that index first. A serializable read (--consistency=s) skips all of this and answers from the member's local store. It is cheaper and still works when the member is cut off, but it can be stale (Demo 7). The same trade-off with quorums instead of a leader: Quorum Reads and Writes.
Failures
Leader crash. In Demo 8 S1 appends a write and dies before sending it. After an election timeout S2 wins term 2 and appends an empty entry at the same index. The client's request times out (context deadline exceeded). It cannot tell whether the write happened, so it retries on the new leader. When S1 comes back, its uncommitted entry conflicts with S2's log and is replaced. Nothing that was acknowledged is ever lost. An entry that did reach a majority before the crash would have been committed by the new leader even though its client timed out, so a retried write must be safe to repeat: a txn on mod_revision does that.
Partition. A follower that is cut off stops getting heartbeats. With pre-vote (on by default since etcd 3.5) it first asks whether it could win an election. Nobody answers, so it does not raise its term and does not disturb the leader when the network comes back. The other two members still form a majority and keep committing. After the heal, the leader sends the missing entries and the follower catches up.
How kube-apiserver uses etcd
kube-apiserver is etcd's only client (--etcd-servers, with TLS client certificates). Objects are stored as protobuf under /registry/<resource>/<namespace>/<name>. Create is a txn with create_revision == 0; update and delete are txns on mod_revision == resourceVersion. A list's resourceVersion is the reply's header revision. Watches start from it. Clusters run 3 or 5 members: 3 survive one failure, 5 survive two. Every write waits for a majority to fsync, so etcd needs fast disks. The rest of the control plane is on Kubernetes Cluster Architecture and How the Control Plane and Nodes Talk. Multi-version storage inside a SQL database, with snapshots and isolation levels, is on MVCC and Isolation Levels.
What the page leaves out
Snapshots of the Raft log and bringing a new member up from one, membership changes and learners, defragmentation and the space quota (2 GiB by default, then the cluster refuses writes with an alarm), the 1.5 MiB request size limit, gRPC proxies and the client's endpoint balancer, authentication and TLS, range queries over a prefix with limits and pagination, watch progress notifications and fragmented watch responses, and lease checkpoints. Simplifications: requests take no time; the client always finds the leader (real followers forward proposals); the election takes one fixed step with timeouts that fire in the order S2, S3, S1; followers apply a write at once instead of at the next heartbeat; only one watcher, served by the leader; and one key per lease.