The idea: four delays at every hop

A packet does not jump from sender to receiver. It crosses a chain of links and routers, and at every hop it loses time in four ways. Add them up over all hops and you get the end-to-end delay.

One hop from router R1 to router R2. Inside R1: processing 0.1 ms (read the header, pick the link), then queueing (wait behind packets ahead). Then transmission: 12,000 bits divided by 4 Mbit/s is 3.0 ms to push the bits onto the link. Then propagation: 1000 km divided by 200,000 km per second is 5.0 ms on the wire to R2
Processing and queueing happen inside the router; transmission puts the bits on the link; propagation carries them to the next router.
  • Processing dproc: the router reads the header, checks it and picks the output link. Microseconds in real routers; 0.1 ms here.
  • Queueing dqueue: the packet waits in the output queue while the link sends the packets ahead of it. Zero on an idle link, large on a busy one.
  • Transmission dtrans = L / R: the time to push all L bits onto a link of rate R. 12,000 bits at 12 Mbit/s take 1.0 ms.
  • Propagation dprop = d / s: the time for a bit to travel the wire of length d at signal speed s (about 2×108 m/s in fibre or copper, 5 µs per km).

The page sends packets of 1500 bytes (12,000 bits) from host A to host B over three links: 12 Mbit/s and 20 km, then a middle link you pick, then 12 Mbit/s and 20 km again. On the path, orange means the sender is still pushing the packet's bits out (transmission); blue means every bit is on the wire (propagation).

Transmission is not propagation

The two are easy to mix up. Transmission depends on the packet and the link rate, not the distance. Propagation depends on the distance, not the rate or the packet. A faster link shortens the first; only a shorter wire shortens the second.

Link 2dtransdpropone packet A → B
4 Mbit/s, 1000 km3.0 ms5.0 ms10.4 ms
4 Mbit/s, 20 km3.0 ms0.1 ms5.5 ms
40 Mbit/s, 1000 km0.3 ms5.0 ms7.7 ms

One packet over the default path takes 10.4 ms: transmission 1.0 + 3.0 + 1.0 = 5.0 ms, propagation 0.1 + 5.0 + 0.1 = 5.2 ms, processing 0.2 ms, no queueing (Demo 1: one packet, four kinds of delay). With a 20 km middle link, propagation drops to 0.3 ms and the packet arrives in 5.5 ms; now pushing bits out is almost all of the delay (Demo 2: short middle link).

Watch the shape of the packet on a link. On link 1 the transmission (1.0 ms) is ten times the propagation (0.1 ms), so the first bit reaches R1 long before the last one leaves A: the bits fill the whole wire. On link 2 the 3.0 ms packet covers 60% of the 5.0 ms wire. The number of bits a link holds at once is R × dprop, the bandwidth-delay product: 4 Mbit/s × 5 ms = 20,000 bits on link 2.

Store-and-forward

A router must have the whole packet before it can check it and send it on, so it stores the packet until the last bit has arrived, then forwards it. Each hop therefore adds a full transmission time. Sending one packet of L bits over N equal links of rate R takes N × L / R plus the propagation, not L / R.

Several packets still overlap, like a pipeline: while link 2 sends P1, link 1 already sends P2. Three packets arrive at 10.4, 13.4 and 16.4 ms, not at 10.4, 20.8 and 31.2 (Demo 3: three packets). But link 1 hands R1 a packet every 1.0 ms and link 2 needs 3.0 ms for each, so P2 waits 2.0 ms and P3 waits 4.0 ms in R1's queue: that is queueing delay.

Time runs down. Three columns A, R1, R2, B with slanted bands for packets P1, P2 and P3. On link 1 each packet takes 1 ms, on link 2 each takes 3 ms, so P2 and P3 wait at R1 for 2 and 4 ms. On link 3 the packets leave R2 3 ms apart and reach B at 10.4, 13.4 and 16.4 ms
Packets leave the slow link 3.0 ms apart, and that gap stays all the way to B.

The bottleneck link sets the throughput

Throughput is how many bits per second actually reach the receiver. With ten packets the queue at R1 grows to 6 packets, and packets reach B exactly one every 3.0 ms: 12,000 bits / 3.0 ms = 4.0 Mbit/s, the rate of link 2. The 12 Mbit/s links before and after it cannot make that faster (Demo 4: ten packets). The slowest link on the path is the bottleneck, and a long transfer gets throughput min(R1, R2, R3).

Make link 2 40 Mbit/s and the bottleneck moves: now link 1 is the slowest, packets arrive one every 1.0 ms (12.0 Mbit/s), no queue forms at R1, and the last packet arrives at 16.7 ms instead of 37.4 (Demo 5: a faster middle link). Upgrading a link that is not the bottleneck would change nothing.

For the whole transfer the average is lower than the bottleneck rate, because the first packet also pays the path delay: 120 kbit in 37.4 ms is 3.2 Mbit/s. The bigger the file, the closer the average gets to the bottleneck rate. On the real Internet the bottleneck is often the access link (home uplink, Wi-Fi) or a busy link shared with many other flows.

A full buffer means loss

A router's queue has room for a limited number of packets. When a packet arrives and the queue is full, the router drops it. With room for only 2 waiting packets, R1 drops P5, P6, P8 and P9; only 6 of the 10 packets reach B (Demo 6: a 2-packet buffer overflows). The packets that do arrive still come every 3.0 ms, because the bottleneck stays busy, but 40% of what A sent is gone and must be sent again by a protocol such as TCP. Which packet to drop, and how big the buffer should be, is the subject of router queues and AQM.

Queueing delay grows quickly as a link gets busy. If packets arrive at rate a and the link sends at rate R, the traffic intensity a / R tells the story: well below 1, queues are short; near 1, the wait grows without bound; above 1, as here (link 1 feeds link 2 three times faster than it can send), the queue grows until the buffer overflows. Latency vs throughput shows the same curve for a server.

Packet switching vs circuit switching

The telephone network used circuit switching: before talking, the call sets up a circuit, and every switch on the way reserves a share of its links (a time slot or a frequency) for this call alone. The page's SETUP message reserves 4 Mbit/s on all three links and the CONNECT reply comes back: 10.8 ms before any data moves. Then the bits flow as a steady stream at exactly 4 Mbit/s, with no store-and-forward, no queue and no loss; the last bit reaches B at 46.0 ms (Demo 7: circuit switching). Packet switching sent the same ten packets by 37.4 ms because it needed no setup.

Two timelines for ten packets on the same path. Packet switching: no setup, first packet at 10.4 ms, the rest one every 3 ms, done at 37.4 ms, a queue of up to 6 packets at R1. Circuit switching: 10.8 ms of SETUP and CONNECT, then a 4 Mbit/s stream, done at 46.0 ms, no queue and no loss, but the reserved rate stays idle when A is quiet
Circuit switching pays a setup round trip and keeps its share even when idle; packet switching starts at once and shares the links, at the cost of queues and loss.
Packet switchingCircuit switching
Before datanothing: send at oncesetup round trip (10.8 ms here)
Link capacityshared, used by whoever has packetsreserved per call, idle when the call is silent
Delayvaries with the queuesfixed after setup
Losspossible when a buffer is fullnone once the call is accepted (a call can be refused)
Each nodestore-and-forward per packetpasses bits on as they come

Why the Internet uses packet switching: most traffic comes in bursts. Say a 1 Mbit/s link and users who each need 100 kbit/s, but are active only 10% of the time. Circuit switching can admit 10 users. Packet switching can serve 35: the chance that more than 10 of them are active at the same moment is below 0.0004, so the link is almost never overloaded. Sharing (statistical multiplexing) wins when traffic is bursty.

In practice

  • ping shows the round trip; the minimum over many pings is about twice the propagation plus transmission, and the spread above it is queueing.
  • Traceroute shows the delay hop by hop; a jump between two hops usually means a long link (an ocean) or a busy queue.
  • Light in fibre covers about 200 km per millisecond: Hanoi to Ho Chi Minh City is about 1,140 km in a straight line, so a packet can never cross it in less than 5.7 ms one way, however fast the links are.
  • A speed test measures the bottleneck: a 1 Gbit/s LAN behind a 100 Mbit/s home uplink gives 100 Mbit/s to the Internet.

See also router queues and AQM, latency vs throughput on a server, traceroute, TCP congestion control, how a router forwards a packet and hubs, switches and routers.

See also BitTorrent and the DHT for why a P2P swarm spreads a file faster than one server as the number of peers grows.

What the page leaves out

Packets of different sizes and other flows sharing the links; retransmission of lost packets (TCP would resend P5, P6, P8 and P9 and slow down, see ARQ protocols); cut-through switches that forward before the whole frame has arrived; the ACKs on the way back; real processing times (micro- to nanoseconds in hardware); time-division and frequency-division slots of a real circuit, and calls being refused when no slot is free.

On the mind map

Open the Networking mind mapNetworkingLayers & links: frames on one local networkLayers & linksIP & routing: packets hop between networksIP & routingCore services: address, name and timeCore servicesTCP, UDP & sockets: end to end between two portsTCP, UDP & socketsHTTP & TLS: what a browser does per requestHTTP & TLSAPIs & push: how apps talk to serversAPIs & pushProxies & edge: boxes in the middle of the pathProxies & edgeContainers: virtual networks from Linux partsContainersReal-time media: voice and video over UDPReal-time mediaSecurity at the edge: filter traffic at the borderSecurity at the edgeRouter forwardingRouter forwardinglongest prefix match picks the next hopVRRP / HSRP failoverVRRP / HSRP failoverBackup takes over the gateway in 3.6 sSDN and OpenFlowSDN and OpenFlowcontroller writes rules, switch looks upTracerouteTracerouteTTL 1, 2, 3: each hop sends Time ExceededNATNATmaps private IP:port to one public IPIPv4 vs IPv6IPv4 vs IPv632 vs 128 bits; SLAAC, no NAT neededAnycast and BGPAnycast and BGPsame prefix announced from many sitesOSPF and BGP policyOSPF and BGP policyPolicy first, then path, then IGP costDistance vector vs link stateDistance vector vs link stateRIP shares distances; OSPF floods the mapSubnets and CIDRSubnets and CIDR/24 = mask: same subnet? ARP, else routerVLSM PlanningVLSM PlanningLargest first: no gaps, one summary routeRouter queues and AQMRouter queues and AQMFull buffers add delay; drop earlyDelay, loss, throughputSlowest link sets pace; full buffer dropsOpen the Networking mind mapNetworkingpackets hop between networksIP & routingRouter forwardingRouter forwardinglongest prefix match picks the next hopVRRP / HSRP failoverVRRP / HSRP failoverBackup takes over the gateway in 3.6 sSDN and OpenFlowSDN and OpenFlowcontroller writes rules, switch looks upTracerouteTracerouteTTL 1, 2, 3: each hop sends Time ExceededNATNATmaps private IP:port to one public IPIPv4 vs IPv6IPv4 vs IPv632 vs 128 bits; SLAAC, no NAT neededAnycast and BGPAnycast and BGPsame prefix announced from many sitesOSPF and BGP policyOSPF and BGP policyPolicy first, then path, then IGP costDistance vector vs link stateDistance vector vs link stateRIP shares distances; OSPF floods the mapSubnets and CIDRSubnets and CIDR/24 = mask: same subnet? ARP, else routerVLSM PlanningVLSM PlanningLargest first: no gaps, one summary routeRouter queues and AQMRouter queues and AQMFull buffers add delay; drop earlyDelay, loss, throughputSlowest link sets pace; full buffer drops
This page on the Networking mind map · 62 pages in 10 branches · open it to see the whole map and test yourself in recall mode.