Copy-on-Write File Systems vs Journaling

The idea: overwrite, or write somewhere else

A file system has to change blocks on disk without ever leaving them half changed, even when the power fails in the middle. There are two families of answers. Journaling file systems such as ext4 and XFS overwrite blocks in place and first write a note of the change to a journal, so a crash can be cleaned up. Copy-on-write (CoW) file systems such as btrfs and ZFS never overwrite a live block: they write the new version somewhere free and then switch one pointer.

The canvas runs the same operations on both machines, ext4 first, then btrfs, on a 3-block file notes.txt. Pick an action and a block and press Do It, or run a demo.

ext4: overwrite in place, journal the metadata

ext4 keeps each block of a file at a fixed home address. In the default data=ordered mode an overwrite goes like this (Demo 1):

1. The new data is written straight over the old data in its home block.
2. jbd2 writes a transaction to the journal: a descriptor block and copies of the changed metadata (here the inode with its new mtime).
3. After a cache flush it writes the commit block. Only now does the transaction count.
4. Later, a checkpoint writes the metadata to its home blocks and the journal space is reused.

Five block writes for one block of data. With data=journal the data block goes through the journal too, so it is written twice (six writes). The page cache, writeback and the jbd2 journal are animated in more detail in How a File System Survives a Crash; the same log-first rule in a database is Write-Ahead Logging and ARIES Recovery.

Copy-on-write: write new, flip the root

btrfs and ZFS keep everything in a tree. The superblock (ZFS: uberblock) at a fixed place points to the root; the root points to leaves; leaves point to data. To change one data block (Demo 1):

1. Write the new data to a free block. The old block is not touched.
2. The leaf that pointed to the old block must change, so write a copy of it pointing to the new block. Then a copy of the root pointing to the new leaf.
3. Flush, then write the superblock with generation + 1 and the new root pointer. This single block write is the commit.
4. Blocks that nothing points to any more are freed.

Four writes here, with no journal and no second copy of the data. On a real system the tree is deeper and several trees change (in btrfs: the file system tree, the extent tree, the checksum tree), so one small write can rewrite many nodes. This is the "wandering tree" cost. CoW file systems pay it back by grouping thousands of changes into one transaction (btrfs commits every 30 s, ZFS a transaction group every 5 s), so the upper nodes are rewritten once per group, not once per write.

A power cut in the middle

Both machines lose power at the same point of the same write. What does the next mount see?

Power cut	ext4 data=ordered	ext4 data=journal	btrfs / ZFS
during the data write (Demo 2, 3)	home block torn (half B2, half B1); no transaction to replay; the file reads back wrong, no error ✗	the torn copy is in the journal without a commit: discarded, the file is the old version ✓	the torn block is new and unreachable; the file is the old version ✓
before the commit / superblock (Demo 4)	new data in place, journal entry discarded, inode mtime old	discarded: old version ✓	the new tree exists but the superblock still points to the old root: old version ✓
after the commit	journal replayed: new version ✓	replayed: new version ✓	superblock already points to the new root: new version ✓

The journal makes ext4's metadata crash-safe: directories, inodes and the block bitmap are always consistent after replay. In ordered mode the data of an overwrite is not covered: a block torn by the power cut stays torn, and ext4 has nothing to compare it with. Applications that care (databases) keep their own log or write a new file and rename() it over the old one. A CoW file system gets data atomicity for free, because the old version is never touched until the new one is committed.

Snapshots are kept old roots

After a CoW commit, the old root still describes a complete, consistent file system. A snapshot is just a second pointer that keeps that root alive (Demo 5). Taking it copies nothing. As the live tree moves on, blocks it shares with the snapshot are not freed; they turn lavender on the canvas. Only blocks changed since the snapshot cost space. Rolling back is one more commit whose root pointer is the snapshot's root (zfs rollback, or a btrfs subvolume swap). ext4 cannot do this on its own: the old versions were overwritten. LVM snapshots add it below the file system, by copying each old block before it is overwritten.

Checksums and scrub

Each pointer in a CoW tree stores the checksum of the block it points to (in ZFS the block pointer holds it; btrfs keeps data checksums in a separate checksum tree). The checksum of the root is in the superblock. So every block is checked against its parent, all the way up: a Merkle tree. A flipped bit, a misdirected write or a firmware bug shows up on the next read as a mismatch (Demo 6).

With a single copy the read fails with EIO: no data is better than wrong data. With a second copy (btrfs raid1 or dup, a ZFS mirror or copies=2) the file system reads the other copy, checks it, returns it and rewrites the bad one (Demo 7). Scrub (btrfs scrub start, zpool scrub) reads every allocated block in the background and does the same, so rot is found while a good copy still exists. ext4 has checksums on its metadata (metadata_csum) but not on file data: the same flipped bit is read back as if nothing happened.

What copy-on-write costs

	ext4 (journaling)	btrfs / ZFS (CoW)
overwrite in the middle of a file	in place: the file stays contiguous	new location: files that are rewritten at random (databases, VM images) fragment
data written twice	only with data=journal	never (ZFS's ZIL is a separate log for sync writes)
metadata written per change	journal copy + checkpoint	a copy of every node up to the root, once per transaction
free space	a write never needs new space	an overwrite needs free space first; "disk full" can block a delete
snapshots, checksums, self-healing	no (LVM, dm-integrity below)	built in

That is why databases on btrfs often turn CoW off for their files (chattr +C, nodatacow), giving up checksums for those files too. SSDs, which also never overwrite in place, are shown in How an SSD Reads and Writes Data, and the same copy-on-write trick for memory pages in Copy-on-Write fork().

What the page leaves out

Real trees are B-trees with hundreds of pointers per node and several levels. btrfs has many trees (root, extent, checksum, chunk, file system trees) and keeps three superblock copies; ZFS keeps a ring of 128 uberblocks and picks the newest valid one. Transactions group many writes, free space is managed with extent trees or space maps, freed blocks are kept a few generations before reuse, and ZFS has the ZIL for fast fsync(). ext4's journal also has revoke blocks, journal checksums and asynchronous commit, and its ordered mode mainly protects newly allocated blocks from showing stale data. Checksums are crc32c, xxhash, sha256 or fletcher4, not an 8-bit hash.

References

btrfs documentation: introduction, CoW, checksums, scrub

OpenZFS documentation: checksums and their use in ZFS

Linux kernel documentation: ext4 journal (jbd2)

OSTEP: Crash Consistency: FSCK and Journaling