The idea: a key in a table, the bytes in pieces on many disks
An object store such as Amazon S3, Google Cloud Storage, Azure Blob, MinIO or Ceph RGW keeps objects: a blob of bytes plus a little metadata, addressed by a bucket and a key, like photos/cat.txt. There are no directories and no in-place edits: you PUT a whole object, GET it back, or DELETE it. The page shows a generic design that most of these systems share in some form; it does not claim to show any vendor's actual internals.
Two separate parts do the work. The metadata service is a strongly consistent index from bucket/key (and each version id) to the object's size, checksum and placement: which node holds which piece. The storage nodes hold the pieces. A front end (the API servers) talks to both, so the data path never needs to know about keys and the index never stores bytes.
Erasure coding: 4 data shards + 2 parity shards
Keeping 3 full copies would survive 2 lost disks but cost 3× the space. Erasure coding gets the same protection for 1.5×. The front end cuts the body into k = 4 data shards and computes m = 2 parity shards, then puts each of the 6 shards on a different node (a different disk, rack or zone in a real system). Any 4 of the 6 shards are enough to get the body back, so any 2 can be lost.
The page uses a Reed–Solomon code small enough to check by hand. Each byte position is treated separately: the four data bytes are the values of a polynomial P of degree 3 at x = 1, 2, 3, 4, and the parity bytes are P(5) and P(6), all modulo 257. A degree-3 polynomial is fixed by any 4 of its points, so any 4 shards give back P and with it the missing bytes. Real systems do the same over GF(28) so every parity value is still one byte; the page uses arithmetic mod 257 because the numbers are easier to read. A separate page on Reed–Solomon codes goes into the field arithmetic; Hamming codes show the simpler single-error idea.
| scheme | disk used | losses survived | a read needs |
|---|---|---|---|
| 3 full copies | 3× | 2 | 1 node |
| RS 4 + 2 (this page) | 1.5× | any 2 | 4 nodes |
| RS 10 + 4 (a common wide code) | 1.4× | any 4 | 10 nodes |
A PUT: shards first, then the metadata commit
The front end encodes the body, sends the 6 shards to their nodes and waits for acks. Only after enough shards are safely on disk does it write the version record in the metadata service. That write is the commit point: a GET that comes before it still sees the old version (or nothing), a GET after it sees the new one, never half of it (Demo 1). If too few nodes are up the store refuses the write with 503 and commits nothing; the page uses a write quorum of k + 1 = 5 shards. With exactly one node down a PUT still succeeds and the record marks the unwritten shard as missing, for a repair job to fill in later (try Fail / Recover on one node, then PUT).
A GET: read 4 shards, rebuild what is missing
The front end looks the key up, then reads 4 shards in parallel. If d1 to d4 all answer, decoding is free: put them side by side (Demo 1). If a data shard's node is down, it reads a parity shard instead and solves for the lost bytes (Demo 2: one node down, Demo 3: two nodes down). With 3 of 6 shards gone there are only 3 points for a degree-3 polynomial, which is not enough: the GET fails with 503 (Demo 4). Nothing is corrupted; bringing one node back makes the same object readable again.
Versioning and delete markers
With versioning on, a PUT to an existing key does not overwrite it. It adds a new version with its own version id and its own shards; the old version becomes noncurrent but stays readable with ?versionId=v1 (Demo 5). A DELETE without a version id erases nothing: it adds a delete marker as the newest version, so a plain GET answers 404 while older versions are still there (Demo 6). Deleting a specific version id, or a lifecycle rule that expires noncurrent versions, is what really frees the shards.
| request | what changes | what a plain GET returns after |
|---|---|---|
| PUT (new key) | v1 with 6 shards | v1 |
| PUT (same key) | v2 added, v1 noncurrent | v2 |
| DELETE (no version id) | delete marker v3 added, no shard touched | 404 |
| GET ?versionId=v1 | nothing | (returns v1 directly) |
In real systems
Amazon S3 has been strongly consistent for reads after writes since December 2020 and documents its durability as 11 nines, but it does not publish its internal layout. MinIO erasure-codes every object across an erasure set of drives with a configurable parity level (EC:N) and keeps the metadata next to the shards in xl.meta. Ceph RGW stores objects in RADOS pools that can be replicated or erasure-coded (k, m in an erasure-code profile), with the bucket index in separate objects. Placement is usually by hashing (Ceph's CRUSH, consistent hashing) so the metadata only needs the set, not each disk.
See also Quorum Reads and Writes (replicating whole copies instead of shards), Merkle Trees for Anti-Entropy Repair (finding which pieces need repair) and Hamming Codes (error-correcting codes on a smaller scale).
What the page leaves out
Large objects and multipart upload (a real object is many stripes of shards, each MB-sized), checksums on every shard to detect bit rot, the background repair (healing) that rebuilds missing shards onto healthy nodes, how the metadata service itself is replicated (usually with consensus such as Raft or Paxos), placement across racks and zones, listing keys by prefix, access control, encryption, storage classes and lifecycle rules, and concurrent writers to the same key.