Linux Socket Backlog Visualization | SYN Queue and Accept Queue

The idea: a listening socket has two queues

A server calls listen(fd, backlog) and then accept() in a loop. Between the two, the kernel does all the work of setting up connections, and it keeps them in two queues on the listening socket. The SYN queue holds half-open handshakes: a SYN came in, the kernel sent a SYN-ACK, and it is waiting for the client's ACK. The accept queue holds connections whose handshake is finished; they wait there until the program calls accept(). Both queues have a limit. What the kernel does when one of them is full explains a lot of odd production symptoms: connects that take exactly 1 or 3 seconds, clients that think they are connected when the server has no socket, and SYN floods.

A SYN enters the SYN queue as a half-open request and a SYN-ACK goes back; the client's ACK moves it to the accept queue as an ESTABLISHED socket; accept() moves it to nginx as a new fd. Both queues are limited by max = min(backlog, somaxconn); the accept queue holds max + 1. — The SYN moves a connection into the SYN queue, the ACK into the accept queue, and only accept() hands it to the program.

On the canvas, six curl clients on the left connect to nginx on the right. The network takes no time; the clock only moves when you press Wait (next timer), which jumps to the next retry. Each client keeps one colour from its row to its slot in a queue and its fd in nginx's table.

The handshake fills the SYN queue, the ACK empties it

A SYN arrives for port 80. The kernel creates a small request socket in state SYN_RECV, puts it in the SYN queue and sends a SYN-ACK.
The client gets the SYN-ACK. For the client, the connection is now ESTABLISHED: connect() returns 0 and it sends the final ACK.
The ACK arrives. The kernel turns the request into a full socket (ESTABLISHED) and moves it to the accept queue.
accept() takes the oldest socket off the accept queue and returns a new fd for it.

nginx is not involved until step 4. A server that is too busy to call accept() still completes handshakes, until the accept queue is full (Demo: queues fill and drain).

backlog and somaxconn

The limit of both queues is max = min(backlog, net.core.somaxconn). If the program asks for more than somaxconn, the kernel lowers it silently: listen() still returns 0 (Demo: somaxconn clamps listen(fd, 4) to 2). To raise the limit you must raise both: the application's backlog and the sysctl.

Who	Default
`net.core.somaxconn`	4096 since Linux 5.4, 128 before
nginx `listen 80 backlog=…`	511 on Linux
Java `ServerSocket`	50
Tomcat `acceptCount`	100
Redis `tcp-backlog`	511

Two kernel details the canvas shows. The SYN queue is full when it holds max requests, but the accept queue is only full when it holds more than max, so it really fits max + 1 (the dashed "+1" slot). And ss -lnt shows a listening socket's queue: Recv-Q is the number of connections waiting in the accept queue, Send-Q is max.

A full accept queue drops SYNs, and clients wait 1 s, 3 s

When the accept queue is full, a new SYN is dropped. The kernel does not answer at all, not even with RST, because the server is only busy, not gone. The client hears nothing, so its kernel sends the SYN again after 1 s, then 3 s, 7 s, 15 s, 31 s and 63 s after the first try (tcp_syn_retries = 6, the wait doubles each time), and gives up with ETIMEDOUT after 127 s.

Time going down: at 0 s C4's SYN is dropped because the accept queue is full; at 1 s the retry is dropped too; nginx calls accept() at 2 s; at 3 s the retry gets a SYN-ACK and connect() returns after 3 s. — A slow accept() loop shows up on the client as connect times of exactly 1, 3 or 7 seconds.

That is why a server whose accept() loop falls behind shows up as connect latencies of exactly 1 s, 3 s or 7 s on the client, with no error anywhere (Demo: slow accept(): client waits 3 s). The counters TcpExtListenOverflows and TcpExtListenDrops go up on the server.

A full queue mid-handshake: the client thinks it is connected

The accept queue is checked again when the final ACK arrives. If several SYNs arrive close together, all of them can get into the SYN queue while there is room, and the queue fills before the last ACKs come in (Demo: final ACK dropped (queue full)). What happens to such an ACK depends on net.ipv4.tcp_abort_on_overflow:

`tcp_abort_on_overflow`	Server	Client
0 (default)	drops the ACK; the request stays in the SYN queue and the SYN-ACK is resent after 1 s, 3 s, …	already ESTABLISHED, sends its request and waits; the next ACK gets in once there is room ✓
1	frees the request and sends RST	its first `send()` or `recv()` fails with `ECONNRESET` ✗

The default is usually better: a short burst only costs a delay. Setting it to 1 makes overflows visible to clients as errors, which helps only when you want them to fail fast and go to another server (Demo: tcp_abort_on_overflow = 1: RST).

SYN floods and SYN cookies

In a SYN flood an attacker sends SYNs from fake source addresses. Each one takes a slot in the SYN queue, the SYN-ACKs go to addresses that never asked for them, and no ACK ever comes. Each request stays for 63 s (SYN-ACK resent at 1, 3, 7, 15 and 31 s, tcp_synack_retries = 5).

Left, tcp_syncookies = 0: the SYN queue is full of spoofed requests and the real client's SYN is dropped. Right, tcp_syncookies = 1: the real client gets a SYN-ACK whose sequence number is a cookie, nothing is stored, and its ACK carrying the cookie creates the socket straight in the accept queue. — With SYN cookies the server keeps no state for a SYN, so a full SYN queue no longer locks real clients out.

tcp_syncookies = 0: once the SYN queue is full, every new SYN is dropped, real clients included. They retry, and keep losing as long as the flood goes on (Demo: SYN flood, syncookies off).
tcp_syncookies = 1 (the default): once the SYN queue is full, the kernel answers with a SYN cookie. It encodes the connection (addresses, ports, time, MSS) in the SYN-ACK's sequence number and stores nothing. A real client's ACK carries the number back plus one; the kernel checks it and builds the socket straight into the accept queue (Demo: SYN flood, syncookies on). The kernel logs Possible SYN flooding on port 80. Sending cookies.

A cookie can't hold everything a normal request socket remembers, so some TCP options (like window scaling) are lost unless TCP timestamps are on. That is why cookies are only used while the queue is full.

Seeing and fixing it on a real machine

ss -lnt: Recv-Q close to Send-Q on a LISTEN socket means the accept queue is (nearly) full.
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtSyncookiesSent or netstat -s | grep -i listen: counters that only go up; watch how fast they grow.
dmesg: "Possible SYN flooding on port 80" means the SYN queue was full.

A full accept queue means the program does not call accept() fast enough. The real fix is there: more worker processes, an event loop that is not blocked (see Linux epoll), or SO_REUSEPORT so each worker gets its own listening socket and queue. A bigger backlog (both the application's and somaxconn) only absorbs short bursts; if the program is slower than the arrivals, a bigger queue just makes the wait longer.

See also TCP Three-Way Handshake and the Listen Queues for the handshake states and the classic BSD two-queue rule, A TCP Socket in Linux for one connection from socket() to close(), nginx Proxy Request and How Tomcat handles an HTTP request for the accept loop of a real server, and Backpressure for bounded queues in general. For a connected socket's data path, from NIC to recv(), see How Linux Receives a Packet.

What the page leaves out

Real round-trip times, the client's request riding on the final ACK and its retransmission, the tcp_max_syn_backlog rule that drops SYNs when cookies are off and the SYN queue is three-quarters full, the shorter SYN-ACK retries for old requests when the queue is busy, cookie ACKs that arrive when the accept queue is full, TCP_DEFER_ACCEPT, TCP Fast Open, per-worker SO_REUSEPORT queues, and IPv6.