Redis Persistence Visualization | RDB Snapshots vs the AOF

The idea: data in RAM, two ways to the disk

Redis keeps the whole dataset in RAM. If the machine loses power, RAM is gone, so Redis offers two ways to put the data on disk. An RDB snapshot writes the whole dataset at one moment, from time to time. The append-only file (AOF) logs every write command as it happens. Both trade speed for safety: the question is always how many recent writes a power loss can take.

A write lands in the keyspace in RAM. The RDB path goes from a forked child process to temp-pid.rdb and through rename() to dump.rdb. The AOF path goes from aof_buf through write() into the kernel page cache and through fsync() to the incr file in appendonlydir — A write reaches the disk either in the next snapshot (RDB) or through aof_buf, write() and fsync() (AOF). Only what is on the disk survives a power loss.

The canvas shows one client, the Redis process with its keyspace (k1 to k4), the kernel's page cache and the disk. Write #n is SET ki vn. The writes strip at the bottom colours each write by where it lives: white in RAM only, yellow in the page cache, green on disk, red lost. The clock is discrete (1 tick = 1 second) and the numbers in redis.conf are scaled down so things happen in seconds, not hours.

RDB: a snapshot from a forked child

The save lines are save points: save 3600 1 300 100 60 10000 (the default) means "snapshot after an hour if 1 key changed, after 5 minutes if 100 changed, after 1 minute if 10000 changed". The page uses save 60 1 10 3 2 5. Ten times a second serverCron checks them against dirty (changes since the last save) and lastsave.

When one is due, Redis runs BGSAVE: it calls fork(). The child sees the keyspace frozen at that moment and writes it to temp-<pid>.rdb, calls fsync(), and then rename()s it over dump.rdb. The rename is atomic, so dump.rdb is always a whole snapshot, the old one or the new one. The parent keeps serving clients the whole time. (Demo 1: BGSAVE from a save point)

The child does not get a copy of the memory, only of the page table: parent and child share every page copy-on-write. When the parent changes a key during the save, the kernel first copies the page that holds it, so the child still writes the old value. Memory grows with the write rate during a save. Copy-on-write fork() shows this page by page.

A crash loses every write since the last snapshot. In Demo 2: crash with RDB only the snapshot holds writes #1 to #5, then #6 to #8 come in, too few for a save point, and the power fails: they are gone.

The AOF: aof_buf, write(), fsync()

With appendonly yes, every write command is also appended, in the RESP protocol, to aof_buf (feedAppendOnlyFile). At the end of the event-loop turn, just before the replies are sent, flushAppendOnlyFile calls write() once for the whole buffer. After write() the data is in the kernel's page cache, which is still RAM: it survives a crash of the Redis process, but not a power loss. Only fsync() makes the disk confirm it. appendfsync chooses when that happens:

appendfsync	Who calls fsync, when	Lost on power loss
`always`	the main thread, after every `write()`, before the reply	nothing that was acknowledged
`everysec` (default)	a background (bio) thread, once a second	about the last second
`no`	never; the kernel writes pages back when they are about 30 s old	up to about 30 s

Four timelines ending in a power loss. RDB only loses everything since the last snapshot; appendfsync no loses up to 30 seconds since the last kernel writeback; everysec loses up to the last second; always loses nothing that was acknowledged — What a power loss takes: the writes after the last point where the data was forced to disk.

Demo 3: AOF everysec, then a crash loses the two writes made after the last fsync. Demo 4: appendfsync always vs no runs both: with always nothing is lost; with no the kernel writes #1 and #2 back at t = 30 s, and #3 to #5, written 20 s before the crash, are lost. always is the safest and the slowest: every write waits for the disk.

AOF rewrite and the multi-part AOF

The AOF only grows: 1000 INCRs on one key are 1000 commands, but the key holds one number. BGREWRITEAOF forks a child that writes the current dataset, one entry per key. It also runs on its own when the AOF is bigger than auto-aof-rewrite-min-size (64mb; 8 entries on the page) and has grown by auto-aof-rewrite-percentage (100 %) since the last rewrite.

Since Redis 7 the AOF is several files in appendonlydir/: a base file (with aof-use-rdb-preamble yes it is in RDB format), one or more incr files with the commands since then, and a manifest that lists them in order. A rewrite works like this (Demo 5: AOF rewrite, multi-part AOF):

The parent opens a new incr file, appendonly.aof.2.incr.aof, and writes a new manifest that lists it. New writes go there.
It forks. The child writes the dataset to temp-rewriteaof-bg-<pid>.aof and fsyncs it.
When the child is done, the parent renames that file to appendonly.aof.2.base.rdb and writes a manifest that marks the old base and old incr files as history (type h).
The history files are deleted and the manifest is written once more.

Three manifests side by side. Before: base 1 and incr 1. During the rewrite: base 1, incr 1 and the new incr 2, while the child writes a temp file. After: base 2, the temp file renamed, and incr 2; base 1 and incr 1 deleted — A rewrite only adds files and switches the manifest, so a crash at any point leaves a manifest that lists complete files.

Before Redis 7 the parent kept the writes made during a rewrite in a rewrite buffer in memory and appended them to the new file at the end. The new incr file replaces that buffer.

Startup and a torn last command

With appendonly yes Redis starts from the AOF, even if a dump.rdb exists: it reads the manifest, loads the base file (it starts with REDIS, so it is read like an RDB file), then replays the incr files in order, command by command (Demo 6: restart: load base + incr). With appendonly no it loads dump.rdb.

A power loss in the middle of a write can leave half a command at the end of the last incr file. With aof-load-truncated yes (the default) Redis logs a warning, cuts the file after the last whole command and starts. With no it refuses to start, and redis-check-aof --fix repairs the file. The cut command was never acknowledged: with always the reply comes only after the fsync. (Demo 7: a torn AOF tail)

Which one to use

RDB only: compact files, fast restarts, good for backups; minutes of writes can be lost.
AOF everysec: at most about a second lost; bigger files and slower restarts (the rewrite and the RDB preamble keep both in check).
Both: common in production. Redis restarts from the AOF, and the RDB files are handy backups.

See also the Redis event loop for the turn of the loop that ends in beforeSleep, Redis replication (a replica is the other way to survive a dead machine), the other pages of this series: the Redis dict, Redis encodings, sorted sets, expiry and eviction, streams and transactions, plus Redis Cluster and Redis Sentinel. For the same ideas in other systems: copy-on-write fork(), the PostgreSQL WAL, WAL crash recovery, Kafka log segments and file systems.

What the page leaves out

Real sizes, timing and hz; RDB internals (encodings, LZF compression, the checksum, rdb-key-save-delay); SAVE (the blocking save) and the save on SHUTDOWN; a crash of only the Redis process (the page cache survives it, so only aof_buf is at risk); everysec postponing a write() for up to 2 s while an fsync is slow; no-appendfsync-on-rewrite; aof-timestamp-enabled; the rewrite-limit back-off after failed rewrites; replication, which forks for RDB too; stop-writes-on-bgsave-error; and the disk's own write cache.