The idea: two views, or one log you can read again

A site wants to know how many times each page was viewed. The views come as a stream of events. The answer must be fresh (include the last second) and right (include every old event, counted with the right rule). Two well-known designs give both. Lambda runs two jobs: a slow batch job over all the data and a fast stream job over the newest data, and adds their results. Kappa runs one stream job over a log that keeps every event, and reads the log again when the rule changes.

Two diagrams. Lambda: events go both to a master dataset and to a speed job. A batch job reads the master dataset and writes a batch view; the speed job writes a realtime view; the serving layer adds the batch view and the realtime view. Kappa: events go to one log; one stream job reads the log and writes a table; the serving layer reads the table. Lambda has two code paths, Kappa one
Lambda keeps two code paths and adds two views at query time. Kappa keeps one code path and one log it can replay.

On this page 12 events e1 … e12 are views of page A or page B. Three of them (e3, e8, e12) come from a bot. The job should not count bot views, so the right answer after all 12 is A = 5, B = 4. Code v1 has a bug: it counts bot views and would say A = 7, B = 5. Code v2 is the fix.

Lambda: batch layer, speed layer, serving layer

Nathan Marz named the Lambda architecture in 2011. It has three layers:

  • Batch layer. Every event is appended to the master dataset, for example files in HDFS. Nothing is ever changed or deleted. Now and then a batch job (MapReduce or Spark) reads all of it and computes the batch view from scratch. Here it runs after every 5th event; in real life it may run every night and take hours.
  • Speed layer. A stream job counts each event as it comes into a realtime view. It only covers the events since the last batch run. After a batch run it throws those counts away, because the batch view has them now.
  • Serving layer. A query adds the two: answer = batch view + realtime view.

In Demo: Lambda, batch view + realtime view at query time (code v2), the batch job runs twice, after e5 and e10. At the end the batch view covers e1–e10 with A = 4, B = 4, the realtime view covers e11–e12 with A = 1, B = 0, and the query gives A = 5, B = 4.

A bug fix in Lambda

The bug must be fixed twice: once in the batch job and once in the speed job. They are often written with different tools, so it is two pieces of code that must give the same numbers. In Demo: Lambda, a bug fix in both code paths, the fix ships after e8. At that moment the batch view (e1–e5, v1) says A = 3, B = 2 and the realtime view (e6–e8, v1) says A = 2, B = 1, so the query says A = 5 while the right A is 3. The fix changes nothing that is already counted. The query stays wrong until the batch job runs again after e10: it recomputes e1–e10 with v2 from the raw data and gives A = 4, B = 4. From then on the answer is right. Marz calls this human fault tolerance: since the raw data is never changed, rerunning the batch job repairs any bug.

If only the batch job is fixed, the two paths drift apart (Demo: Lambda, fix only the batch code: the paths drift). After the batch run at e10 the answer is right, but the speed job still runs v1 and counts the bot view e12. The query ends with B = 5 instead of 4, and it would jump back to 4 at the next batch run. Answers that change on their own like this are hard to debug.

Kappa: one log, replay to reprocess

Jay Kreps proposed the Kappa architecture in 2014 as a reply. Keep the raw events in a log such as a Kafka topic, with a long retention. One stream job reads the log in order, keeps a table, and commits its offset. There is no batch layer. In Demo: Kappa, one log and one stream job, job v2 reads offsets 0–11 and its table says A = 5, B = 4.

To fix a bug, replay the log (Demo: Kappa, a bug fix by replaying the log):

  1. Start job v2 in a new consumer group (counts-v2) at offset 0. It writes a new table, counts_v2. Job v1 keeps serving.
  2. Job v2 reads offsets 0–7 again (8 events) as fast as it can and builds A = 3, B = 3. Job v1 had A = 5, B = 3.
  3. When v2 has caught up (lag 0), switch the serving layer to counts_v2.
  4. Stop v1 and drop counts_v1. New events e9–e12 go to v2 only, and the end result is A = 5, B = 4.
A time line from top to bottom. Job v1 reads offsets 0 to 7 and serves table counts_v1 with A 5, B 3. Job v2 starts at offset 0, replays offsets 0 to 7 and builds counts_v2 with A 3, B 3. The serving layer switches to counts_v2. Job v1 is stopped and counts_v1 dropped. Events 8 to 11 go to job v2 only, which ends with A 5, B 4
Reprocessing in Kappa: run the new version beside the old one from offset 0, switch readers when it catches up, then drop the old one.

The replay only works if the log still has the old events. In Demo: Kappa, retention too short for the replay the topic keeps only the last 6 events (think of a short retention.ms). When the fix ships, offsets 0 and 1 are gone, so v2 starts at offset 2. It misses e1 (A) and e2 (B) and ends with A = 4, B = 3: wrong for good. Kappa needs retention as long as the history you may want to recompute, or a copy of the log in cheap storage. For a log that keeps only the newest value per key, see Kafka log compaction.

The cost of two code paths

LambdaKappa
raw data kept inmaster dataset (HDFS, S3)the log (Kafka topic, long retention)
code to write and test2: batch job + speed job, must agree1 stream job
systems to runbatch engine + stream engine + a store that merges viewslog + stream engine + a table
fixing a bugchange both jobs; history heals at the next batch runrun v2 from offset 0, switch, stop v1
what it needstime and machines for full recomputesa log that keeps everything; a job fast enough to replay it
good whenthe batch side needs work a stream job does badly (big joins, ML training)the logic fits a stream job; one team owns one code base

Today many teams run one engine for both (Spark Structured Streaming, Apache Flink, Apache Beam), so the "two code paths" can share code even in a Lambda setup. How a stream job groups events by time and handles late ones is on Stream Windows and Watermarks. The batch side is MapReduce or Spark stages. Keeping every event and rebuilding state from it is also the idea behind event sourcing.

What the page leaves out

Keys, partitions and many parallel tasks; time windows and late events. A real batch run takes hours, so new events arrive during it and the speed layer must keep the events since the batch run started. The realtime view is often approximate (sampling, sketches). Exactly-once output and idempotent writes during a replay (see Delivery Semantics). Tiered storage that moves old log segments to object storage, and replaying from a data lake instead of the log. Serving stores such as Druid, Cassandra or Pinot and how they swap tables.

On the mind map

Open the Distributed Systems mind mapDistributedSystemsReplication: copies of data · what reads seeReplicationPartitioning: split data over nodesPartitioningTime and membership: order events, know who is aliveTime and membershipConsensus: agree on one value or logConsensusTransactions: one change across many servicesTransactionsMessaging: send events between systemsMessagingKafka internals: a replicated append-only logKafka internalsRedis internals: one thread, data in memoryRedis internalsKubernetes: reconcile desired stateKubernetesResilience and speed: stay up and fast under loadResilience and speedKafka message pathKafka message pathbatch, partition, replicate, commit offsetStream Windows & WatermarksStream Windows & WatermarksWindow fires when watermark passes its endLambda vs Kappatwo code paths vs replay one logKafka rebalanceKafka rebalancepartitions move; duplicates appearKafka log segmentsKafka log segmentsfiles by base offset, sparse indexKafka replicationKafka replicationISR and high watermark; epochs truncateKafka log compactionKafka log compactionkeep the last value per keyKafka KRaftKafka KRaftRaft quorum stores cluster metadataKafka zero copyKafka zero copysendfile: page cache to socketOpen the Distributed Systems mind mapDistributed Systemsa replicated append-only logKafka internalsKafka message pathKafka message pathbatch, partition, replicate, commit offsetStream Windows & WatermarksStream Windows & WatermarksWindow fires when watermark passes its endLambda vs Kappatwo code paths vs replay one logKafka rebalanceKafka rebalancepartitions move; duplicates appearKafka log segmentsKafka log segmentsfiles by base offset, sparse indexKafka replicationKafka replicationISR and high watermark; epochs truncateKafka log compactionKafka log compactionkeep the last value per keyKafka KRaftKafka KRaftRaft quorum stores cluster metadataKafka zero copyKafka zero copysendfile: page cache to socket
This page on the Distributed Systems mind map · 88 pages in 10 branches · open it to see the whole map and test yourself in recall mode.