SO_REUSEPORT Visualization | Thundering Herd vs a Socket per Worker

The idea: many workers, one port

A server like nginx runs several worker processes, usually one per CPU, and all of them serve the same port 80. There are two ways to share that port on Linux. In the classic way the master calls listen() once and forks, so every worker holds the same listen socket with one accept queue. With SO_REUSEPORT each worker opens its own listen socket on the same port, and the kernel decides which socket each new connection goes to.

Left: one listen socket on port 80 with one accept queue, watched by four workers W1 to W4. Right: four listen sockets on port 80, each with its own accept queue and one worker; a hash of the 4-tuple picks the socket. — One shared queue that every worker watches, or one queue per worker chosen by a hash.

The thundering herd

Each worker waits in epoll_wait(), and each worker's epoll instance has an entry on the shared socket's wait queue. When a handshake completes, the kernel puts the connection in the accept queue and wakes the waiters. With a plain EPOLLIN entry, it wakes all of them. They all call accept(); one gets the connection and the others get EAGAIN and go back to sleep. That is the thundering herd: three of four wakeups do nothing but cost context switches and cache misses (Demo: shared socket, thundering herd: 12 wakeups, 9 wasted).

The demo also shows a second problem. The worker that went to sleep last tends to run first, so the same worker wins again and again. Here W4 gets all three connections.

EPOLLEXCLUSIVE

Linux 4.5 added the EPOLLEXCLUSIVE flag for epoll_ctl(). The kernel walks the wait queue in order and wakes only the first epoll that has a thread waiting in it. No herd: one connection, one wakeup (Demo: EPOLLEXCLUSIVE wakes one). But the order never changes, so W1 is woken for every connection while it is free, and the load stays uneven. A busy worker is skipped, and the next free one takes the connection at once (Demo: shared queue, busy worker is skipped).

SO_REUSEPORT and the 4-tuple hash

With SO_REUSEPORT (Linux 3.9) every worker sets the option and binds its own socket to :80. The sockets form a reuseport group. For each SYN the kernel computes a hash of the 4-tuple, the source and destination address and port: inet_ehashfn(), a jhash with a secret seed chosen at boot. Then reciprocal_scale(hash, n) = (hash × n) >> 32 picks one of the n sockets. Only the owner of that socket is woken, and different clients spread over the workers (Demo: SO_REUSEPORT spreads by hash: 8 connections, 8 wakeups, 2 · 3 · 1 · 2). The same 4-tuple always lands on the same socket.

The price: a busy worker's queue waits

The kernel picks the socket before any worker looks at the connection, and it does not know which worker is busy. If W2 is stuck on a slow request, the connections hashed to socket #2 wait in its queue while W1, W3 and W4 sleep (Demo: SO_REUSEPORT, a busy worker's queue waits: C1 waits 3 ticks). With one shared queue, any free worker would have taken them.

Two timelines over ticks 1 to 4 with W2 busy. Top, SO_REUSEPORT: C1 and C3 hash to socket 2 and wait until W2 is free at tick 4; C1 waits 3 ticks. Bottom, shared socket with EPOLLEXCLUSIVE: W1 takes C1 to C4 at once, no waiting. — Same clients, W2 busy: SO_REUSEPORT makes C1 wait 3 ticks; the shared queue serves everyone at once, all on W1.

In nginx

setting	what it does
`listen 80;`	one socket shared by all workers
`accept_mutex on`	workers take turns watching the socket (the old fix for the herd); off by default since 1.11.3
`EPOLLEXCLUSIVE`	used automatically on Linux 4.5+ when `accept_mutex` is off
`listen 80 reuseport;`	one socket per worker (nginx 1.9.1+)

Side by side

	shared, plain epoll	shared + EPOLLEXCLUSIVE	SO_REUSEPORT
wakeups per connection	every waiting worker	one	one
who gets it	whoever runs first	first free worker in the wait queue	the hash
load spread	uneven	uneven	even on average
busy worker	others take the connection	others take the connection	its connections wait

See also Linux epoll for the interest and ready lists, the socket backlog for the accept queue, the TCP handshake, how nginx uses the kernel and Prefork vs Threads vs Events.

What the page leaves out

Serving a normal request takes no time here, and a slow one takes exactly 4 ticks. The page leaves out the SYN queue and SYN cookies, the backlog limit and somaxconn, the epoll ready-list entry a busy worker still gets with EPOLLEXCLUSIVE, blocking accept() (already exclusive since Linux 2.6), BPF programs that choose the socket (SO_ATTACH_REUSEPORT_EBPF), connections lost from a socket's queue when its worker exits on reload, and threads instead of processes.