RAID Visualization | Striping, Mirroring and XOR Parity

The idea: many disks that look like one

A RAID (redundant array of independent disks) takes several disks and shows the operating system one bigger disk. On Linux the md driver does it in software; a hardware RAID card does the same in firmware. The level decides where each logical block goes: spread over the disks for speed, copied for safety, or protected by a parity block that can rebuild any one lost block. The page uses 4 disks with 4 rows each, and 4-bit blocks so you can do the XOR by eye.

Four disks under four RAID levels. RAID 0: A B C D across the first row, no copies. RAID 1: blocks A to D on disk 0 and again on disk 1. RAID 5: each row holds 3 data blocks and one parity block P, and the parity moves from disk 3 to disk 0 row by row. RAID 10: A on disks 0 and 1, B on disks 2 and 3, and so on. — The same four disks hold 16, 4 (on two disks), 12 or 8 blocks, depending on how much space goes to copies or parity.

RAID 0: striping

Striping puts block A on disk 0, B on disk 1, C on disk 2, D on disk 3, then starts the next row, the next stripe. A large read keeps all four disks busy at once, and all the capacity is usable. But nothing is stored twice: when one disk dies, every fourth block is gone, and so is nearly every file. Four disks fail four times as often as one. (Demo: RAID 0 striping, then a disk dies)

RAID 1 and RAID 10: mirroring

RAID 1 writes every block to two disks. A read can come from either copy; a write costs two disk writes. When a disk dies, the other copy serves everything, and the rebuild simply copies the survivor onto the new disk. (Demo: RAID 1 mirror survives a failure)

RAID 10 stripes across mirror pairs: disks 0 + 1 are one pair, disks 2 + 3 another. It survives one dead disk in each pair, but not two in the same pair. (Demo: RAID 10 loses one disk per pair)

RAID 5: XOR parity that rotates

RAID 5 gives up one block per stripe to parity: P = A ⊕ B ⊕ C, the bitwise XOR. XOR has two handy properties: x ⊕ x = 0, and all four blocks of a stripe XOR to 0000. So any one missing block is the XOR of the other three, whether it held data or parity.

The parity block rotates: stripe 0 keeps it on disk 3, stripe 1 on disk 2, and so on. Otherwise one disk would take the parity write of every write in the array. The page uses Linux md's default left-symmetric layout, where the data of a stripe starts on the disk right after its parity.

Losing a disk: degraded mode and rebuild

When one disk dies the array is degraded, not down. A read of a block on the dead disk reads the other 3 blocks of its stripe and XORs them: 3 reads instead of 1. A new disk is then rebuilt stripe by stripe, which reads every block of every surviving disk. On real multi-terabyte disks that takes hours, and a second failure (or one unreadable sector) during that time loses data. That is why large arrays use RAID 6, which keeps two independent parity blocks per stripe; its second parity is not a plain XOR but Reed-Solomon arithmetic, shown on Reed-Solomon Erasure Coding. (Demo: RAID 5 disk failure, degraded read, rebuild)

The small-write penalty

Writing one block on RAID 5 also changes its stripe's parity. Reading the whole stripe would be costly, so md uses read-modify-write: read the old data and the old parity, compute P' = P ⊕ D ⊕ D' (XOR-ing the old data takes it out of the parity, XOR-ing the new data puts it in), then write the new data and the new parity. One logical write costs 4 disk I/Os. (Demo: RAID 5 small write = 4 disk I/Os)

Writing block E = 0000 on RAID 5. Step 1, two reads: old E = 1001 from disk 0 and old parity P1 = 0000 from disk 2. Step 2, compute: 0000 xor 1001 xor 0000 = 1001. Step 3, two writes: the new E to disk 0 and the new P1 = 1001 to disk 2. In total 4 disk I/Os for one small write. — A small RAID 5 write is two reads and two writes; the other data disks are not touched.

A full-stripe write (all 3 data blocks at once) avoids the reads: the parity is computed from the new data alone. That is why RAID 5 is fine for large sequential writes and slow for small random ones, such as a database's.

level	usable (4 disks)	survives	disk I/Os per small write
RAID 0	100%	no failure	1
RAID 1	50%	1 disk	2
RAID 10	50%	1 disk per pair	2
RAID 5	75%	1 disk	4 (2 reads + 2 writes)
RAID 6	50%	any 2 disks	6 (3 reads + 3 writes)

The write hole

The data write and the parity write go to two different disks, and they don't land at the same instant. If the power fails between them, the stripe's parity no longer matches its data. Nothing looks wrong: every block reads back fine. The damage shows only later, when a disk dies and md rebuilds a block from that stale parity: the result is wrong, and nothing reports it. This is the write hole. (Demo: RAID 5 write hole)

Time runs down. 1: stripe 0 holds A 0110, B 1010, C 0011, P0 1111, consistent. 2: A is written as 1101, then the power fails before P0 is updated; P0 is still 1111 and is now stale. 3: disk 1 fails. 4: B is rebuilt as A xor C xor P0 = 1101 xor 0011 xor 1111 = 0001, but the true B is 1010. — A power cut between the data and parity writes leaves stale parity, and a later rebuild silently returns the wrong block.

The fixes all make the two writes atomic or recoverable: a battery-backed cache on a hardware RAID card, a resync of every stripe after an unclean shutdown (md's write-intent bitmap limits it to recently written regions), md's journal device or partial parity log, or a copy-on-write file system such as ZFS RAID-Z that never overwrites a stripe in place. A mirror has a milder form of the same problem: after a crash the two copies can disagree.

For how a single disk serves these reads and writes, see How a Hard Disk Reads and Writes Data and How an SSD Reads and Writes Data.

What the page leaves out

Real arrays use chunks of 64–512 KiB, not single blocks, and many more stripes. md also chooses between read-modify-write and reconstruct-write (read the other data blocks instead) by which needs fewer reads, keeps a stripe cache, can use hot spares that start rebuilding on their own, runs periodic scrubs (check) that find stale parity and bad sectors before a disk fails, and lets you grow or reshape an array. RAID 6's second parity, nested levels like RAID 50 and 60, and the rebuild speed limits are left out too.