The idea: where the file system and the drivers live

Every kernel must run the file system and the device drivers somewhere. A monolithic kernel such as Linux puts them inside the kernel: one big program in one address space, running in kernel mode (ring 0). A microkernel such as seL4, MINIX 3 or QNX keeps only the minimum in kernel mode: messages between processes (IPC), scheduling, address spaces and interrupts. The file system and the drivers become ordinary user processes (ring 3). The page sends the same read(fd, buf, 4096) from cat to both and counts what it costs.

Left: a monolithic kernel. User space holds cat, bash and sshd. Kernel space holds one big box with syscall entry, VFS, ext4, the page cache and the NVMe driver, connected by function calls. Right: a microkernel. User space holds cat, the file server, the reincarnation server and the disk driver. Kernel space holds only a small box for IPC, scheduling, address spaces and interrupts; the processes talk through it with messages
The line between user and kernel mode is drawn in a different place: around everything on the left, around almost nothing on the right.

A mode switch is the CPU moving between user mode and kernel mode: a syscall or a fault goes in, sysret or the return from an exception goes out. The scoreboard under each lane counts the mode switches, the IPC messages and the function calls of the last read, and adds up its time.

One read() on Linux

cat runs the syscall instruction: mode switch 1. From there on everything is kernel code in the same address space. The syscall table calls vfs_read(), VFS calls ext4's read_iter(), and ext4 looks in the page cache. On a hit, copy_to_user() copies 4096 bytes into buf, the calls return, and sysret goes back to user mode: mode switch 2 (Demo 1: read() a cached page).

On a miss, ext4 calls submit_bio() and the block layer calls the NVMe driver: still function calls. The driver writes a command to the disk's queue and cat sleeps. When the disk has copied the block into memory (DMA) it raises an interrupt; the handler is more kernel code, so there is no extra mode switch while the CPU idles in kernel mode (Demo 2: read() that goes to the disk). See System Calls and Interrupt-Driven I/O for these two pieces on their own.

The same read() as messages

In the microkernel, read() is a library stub. It puts READ(fd 3, 4096) in a message and traps into the kernel. The kernel copies the message, switches to the file server's address space and returns to user mode there. That is one IPC message and two mode switches: every hop between components costs this much. The file server answers from its cache with a reply message, and the data goes into a buffer cat granted it.

On a miss the file server sends DEV_READ to the disk driver, another process. The disk's registers are mapped into the driver's address space, so it starts the disk itself, then waits in receive(). The kernel does not run driver code on an interrupt: it turns the interrupt into a notification message for the driver. Then the driver replies to the file server and the file server replies to cat.

A time-down sequence of the disk read on the microkernel. Four columns: cat, kernel, file server, disk driver. Five messages pass through the kernel column: READ from cat to the file server, DEV_READ from the file server to the driver, the interrupt notification to the driver, the reply to the file server and the reply to cat. Each message crosses the user-kernel line twice, ten mode switches in all
Every message goes down into the kernel and back up into another process: 5 messages, 10 mode switches for one disk read.

What it costs

The page's costs: 0.1 µs per mode switch, 0.2 µs of kernel work per message (copy it, switch address space), 1.0 µs of file-system work, 0.5 µs for the driver, 80 µs for the NVMe read. Function calls are counted but cost almost nothing.

read()monolithicmicrokernel
cached page2 switches, 0 messages, 2 calls: 1.2 µs4 switches, 2 messages: 1.8 µs (+50 %)
from the disk2 switches, 0 messages, 3 calls: 81.7 µs10 switches, 5 messages: 83.5 µs (+2 %)

The extra cost is per hop, not per byte. It hurts most on cheap, frequent calls, and nearly vanishes when a slow device dominates. A real switch also costs indirectly: cache and TLB entries of the other address space are cold. That is why microkernels live or die by the speed of their IPC: seL4 makes a call-and-reply in a few hundred cycles, and QNX passes messages without extra copies where it can. Compare with the copies of other IPC styles in IPC Mechanisms and the cost of switching between processes in Context Switch.

A driver crash

Demo 3: the disk driver crashes arms a one-shot bug: the driver's interrupt handler reads through a NULL pointer. The code is the same in both lanes; only where it runs differs.

monolithicmicrokernel
the NULL read happens inkernel mode, interrupt contexta user process
it becomesa kernel oops: "BUG: kernel NULL pointer dereference"an ordinary page fault
what stopseverything: "Kernel panic - not syncing: Fatal exception in interrupt"the driver process only
recoverya reboot, about 30 s; every process starts againthe reincarnation server starts a fresh driver (5 ms); the file server re-sends its request
the read()never returnsreturns 4096 after 5165.6 µs: 9 messages, 18 switches

In Demo 4: crash, then read again the left machine is still down, while the right one serves the next read in 1.8 µs. A bug in a monolithic kernel can also go unnoticed and silently corrupt another subsystem's memory, because they share one address space; in the microkernel, the driver can only touch its own memory and the device it was given.

Real kernels sit in between

  • Linux is monolithic but modular: drivers are loadable modules, still in ring 0. It also has user-space escapes: FUSE file systems, and VFIO or UIO drivers.
  • MINIX 3 runs drivers and servers as processes and restarts crashed drivers with its reincarnation server, as on the page.
  • seL4 is a formally verified microkernel of about 10 000 lines; QNX is a microkernel used in cars and medical devices.
  • Hybrid kernels (Windows NT, macOS XNU) started from microkernel ideas but put the file systems and most drivers back in kernel mode for speed.

What the page leaves out

Readahead and the block layer's queues, the scheduler picking what runs while cat sleeps, KPTI and other mitigations that make mode switches dearer, the cache and TLB cost of address-space switches, IOMMUs that keep a user-space driver's DMA inside its memory, how MINIX 3's grants and seL4's capabilities give access to memory, asynchronous IPC, and drivers that keep state a restart loses.