The idea: one map keeper, many byte keepers
The Google File System (GFS, 2003) and its open-source copy, the Hadoop Distributed File System (HDFS), store huge files on hundreds of cheap machines that fail all the time. They split the job in two. One NameNode (GFS: master) keeps only the metadata: which files exist and where their pieces are. Many DataNodes (GFS: chunkservers) keep the bytes. File data never passes through the NameNode.
The page has four DataNodes on two racks and one file, /logs/day1.log, 200 MB.
Blocks and replicas
A file is cut into blocks of dfs.blocksize = 128 MB (GFS used 64 MB chunks). The 200 MB file becomes two blocks: 128 MB and 72 MB. The last block only takes the space it needs. Big blocks keep the metadata small: the NameNode holds every file and block in RAM, about 150 bytes each, so one NameNode can track hundreds of millions of blocks.
Each block is stored dfs.replication = 3 times as replicas on different DataNodes. The NameNode's block map says which DataNodes hold each block. It is not saved to disk: DataNodes report what they hold when they start and after each new block (blockReceived), and the NameNode rebuilds the map from that.
Rack-aware placement
Machines in one rack share a switch and a power feed, so a whole rack can fail at once. The default placement policy puts the 1st replica on the writer's own node (or a random one if the writer is outside the cluster), the 2nd on a different rack, and the 3rd on the same rack as the 2nd, on another node. Each block then lives on two racks, but only one copy crosses between racks during the write.
Demo 5: a whole rack fails, no block is lost kills DN3 and DN4. Every block still has its copy on DN1, and the NameNode copies it to DN2. With only two live nodes the blocks stay at 2 of 3 replicas.
The write pipeline
A write is a few small RPCs to the NameNode and a big stream to the DataNodes (Demo 1: write a 200 MB file):
create: the NameNode adds the path, marked under construction, and gives the client a lease (one writer per file).addBlockfor each block: the NameNode picks the target DataNodes and returns a new block id.- The client streams the block as 64 KB packets to the first DataNode. Each node writes a packet to disk and forwards it to the next: a pipeline. Acks flow back the other way; a node acks only when it and everyone after it have the packet.
- When the block is full, each DataNode finalizes its replica and sends
blockReceived; the NameNode records the locations. complete: the file is closed and can be read by everyone.
A pipeline uses each node's network link once: the client sends the block one time, not three. Files are write-once: a second create on the same path fails with FileAlreadyExistsException (press Write File twice). HDFS can append, but never change bytes in the middle of a file.
Reads and locality
To read, the client calls getBlockLocations. The NameNode returns every block with its DataNodes sorted by network distance: the same node is 0, the same rack 2, another rack 4. The client then reads each block straight from the first DataNode on the list (Demo 2: read it back: both blocks come from DN1, the client's own node, with no network at all). This is why MapReduce and Spark try to run each task on a node that holds its block.
If a DataNode does not answer, the client puts it on the stream's dead-nodes list and tries the next location (Demo 4: read while a DataNode is down). Each replica also carries a CRC checksum per 512 bytes; a corrupt replica is reported to the NameNode and the client reads another one. With a client outside the cluster every DataNode is equally far (Demo 6).
Heartbeats and re-replication
Each DataNode sends a heartbeat every 3 s (dfs.heartbeat.interval). The NameNode answers it, and the answer is how it gives orders. A node that has been silent for 2 × dfs.namenode.heartbeat.recheck-interval (5 min) + 10 × 3 s = 630 s is declared dead. The wait is long on purpose: a node that restarts within a few minutes should not trigger terabytes of copying.
When a node is declared dead, its replicas leave the block map. Each block that now has fewer than its replication factor is under-replicated; the NameNode sends a DNA_TRANSFER command to a DataNode that still has it, and that node copies the block to another one. No client takes part.
| what fails | who notices | what happens |
|---|---|---|
| DataNode, between writes | NameNode, after 630 s | replicas copied from survivors |
| DataNode, during a read | client, at once | next location on the list |
| DataNode, during a write | pipeline, at once | pipeline recovery (below) |
| a whole rack | NameNode, after 630 s | copies from the other rack |
| every replica of a block | client | BlockMissingException |
Pipeline recovery and generation stamps
If a DataNode dies in the middle of a write, the node before it gets no ack and sends an error ack that names it (Demo 3: a DataNode dies in the write pipeline). The client removes it, asks the NameNode for a new generation stamp for the block, and resends the unacknowledged packets through the nodes that are left. The partial replica on the dead node keeps the old stamp, so it is stale: if that node comes back, the NameNode will see the old stamp and delete it.
With the DEFAULT replace-datanode-on-failure policy and replication 3, the client goes on with 2 nodes instead of waiting for a replacement; the NameNode adds the third copy later, after the heartbeat timeout. The next block's addBlock lists the failed node as excluded, so it is not chosen again.
GFS and HDFS
| GFS (2003) | HDFS | |
|---|---|---|
| metadata server | master | NameNode (plus a standby for HA) |
| storage server | chunkserver | DataNode |
| unit | 64 MB chunk | 128 MB block |
| who orders writes | a primary replica holding a lease | the single writer's pipeline |
| writes | random writes and record append | write-once, append only |
| successor | Colossus (sharded metadata) | Federation, Ozone |
Replicating whole blocks costs 3× the space. HDFS can also store cold data with erasure coding, as on Reed-Solomon Erasure Coding: RS(6,3) survives three lost shards at 1.5× the space. Compare Kafka replication, where a leader replica orders writes, and quorum reads and writes, where no single node keeps the map.
What the page leaves out
NameNode high availability (standby NameNode, JournalNodes, ZooKeeper failover), the fsimage and edit-log checkpointing, federation, block reports in full, packets and per-chunk checksums drawn one by one, hflush and hsync, appends, lease recovery when a writer dies, decommissioning, the balancer, short-circuit local reads, erasure-coded files, and the random choices inside the real placement policy (the page picks fixed nodes so the demos repeat).