The idea: many boxes come out, one box per object should stay

An object detector answers two questions for every object in an image: where is it, and what is it. The "where" is a bounding box. The detector does not output one box per object. It scores thousands of candidate boxes, and every candidate whose class score passes a score threshold comes out. A real object usually gets several boxes that are almost the same. The last step of the detector, non-maximum suppression (NMS), keeps the best box for each object and throws away the others.

Left: three grey discs G1, G2, G3 with eight candidate boxes D1 to D8, three on G1, two on G2, two on G3 and one on empty background. Right: after NMS at threshold 0.5 four boxes are left, D1 on G1, D2 on G2, D5 on G3 and D8 on the background
NMS turns a cloud of overlapping guesses into one box per object; it cannot remove a box that overlaps nothing, such as D8.

The page uses a synthetic image of 400 × 300 pixels with three objects of one class, "circle": G1 alone, and G2 and G3 standing close together. The 8 candidate boxes are numbered by score, so D1 (0.92) is the best and D8 (0.35) the worst. These are made-up boxes, not the output of a real network. How a CNN turns pixels into features is on Convolution in a CNN, and how to reuse a pre-trained backbone is on Transfer Learning.

Candidate boxes and scores

A box is written by its corners [x1, y1, x2, y2] in image pixels: x grows to the right and y grows down, so (x1, y1) is the top-left corner. Its area is (x2 − x1) × (y2 − y1). D1 is [48, 62, 138, 152], so its area is 90 × 90 = 8100. Each box also carries a score: how sure the detector is that the box holds a circle. In a real detector there is one score per class; the page has one class.

IoU, step by step

To compare two boxes, detection uses intersection over union (IoU): the area they share divided by the area they cover together. It is 1 for two identical boxes and 0 for boxes that do not touch.

  1. The intersection starts at the larger x1 and the larger y1, and ends at the smaller x2 and the smaller y2. If that end is not past that start, in x or in y, the boxes do not overlap and the intersection is 0.
  2. Union = area A + area B − intersection, so the shared part is counted once.
  3. IoU = intersection / union.
Box D1 from 48, 62 to 138, 152 with area 8100 and box D4 from 40, 66 to 132, 158 with area 8464. Their intersection from 48, 66 to 132, 152 is 84 by 86, area 7224. Union 8100 plus 8464 minus 7224 equals 9340. IoU 7224 over 9340 equals 0.773
IoU of D1 and D4: 7224 / 9340 = 0.773, two guesses for the same object (Demo 1).

For D2 and D5, two boxes on the two different objects G2 and G3, the intersection is 70 × 84 = 5880 and the union is 9800 + 9996 − 5880 = 13916, so IoU = 0.423 (Demo 2). The pair D1 and D2 does not touch at all: IoU 0.

Greedy non-maximum suppression

NMS needs only the boxes, their scores and one number, the IoU threshold:

sort the boxes by score, highest first
while boxes are left:
    take the first box and keep it
    remove every remaining box whose IoU with it is > threshold

This is what torchvision.ops.nms(boxes, scores, iou_threshold) does; it returns the indices of the kept boxes in score order. It is called "non-maximum" suppression because every box that is not the local maximum of the score gets suppressed. At threshold 0.5 the page keeps D1, D2, D5 and D8 (Demo 3): D1 removes D4 (0.773) and D6 (0.598), D2 removes D3 (0.639) but keeps D5 (0.423) and D7 (0.327), and D5 then removes D7 (0.774). NMS runs once per class: a box of a dog never suppresses a box of a cat.

The threshold trade-off

The threshold decides how much overlap counts as "the same object". Real objects can overlap too: people in a crowd, cars in a queue. Here G2 and G3 have IoU 0.423 with each other, so their correct boxes D2 and D5 also overlap by 0.423.

NMS thresholdkept boxeswhat goes wrong
0.3 (Demo 4)D1, D2, D8D5 overlaps D2 by 0.423 > 0.3 and is removed: G3 is missed ✗
0.5 (Demo 3)D1, D2, D5, D8one box per object ✓
0.7 (Demo 5)D1, D2, D3, D5, D6, D8D3 (0.639 with D2) and D6 (0.598 with D1) survive: duplicates ✗
Three panels of the two overlapping objects G2 and G3 with their boxes. At threshold 0.3 only D2 is left and G3 has no box. At 0.5 D2 and D5 are left, one per object. At 0.7 D2, D3 and D5 are left, G2 has two boxes
Too low a threshold merges neighbours, too high a threshold keeps duplicates.

Real defaults vary: 0.45 in the YOLOv5 detect script, 0.5 for the final boxes of torchvision's Faster R-CNN, and 0.7 inside its region proposal network, where duplicates are cheap because a second stage looks at every proposal again.

Scoring one class: TP, FP and average precision

To score a detector we compare its kept boxes with the ground-truth boxes drawn by people. Go through the boxes in score order. A box is a true positive (TP) if its best IoU with a ground-truth box is at least 0.5 and that object was not found yet. Otherwise it is a false positive (FP): a box on the background, a box that fits too badly, or a duplicate of an object that a higher box already found. An object that no box found is a miss (false negative). Precision and recall then follow as usual; what they mean is on Precision & Recall.

Precision and recall are recomputed after each box in score order. Average precision (AP) is the area under that precision–recall curve, with each precision replaced by the best precision at that recall or later (all-point interpolation, PASCAL VOC since 2010). With 3 objects every TP adds 1/3 of recall, so AP = the sum of 1/3 × that precision over the TPs.

runTPFPmissedprecisionrecallAP@0.5
no NMS (Demo 7)3500.3751.0000.867
NMS 0.32110.6670.6670.667
NMS 0.5 (Demo 6)3100.7501.0001.000 ✓
NMS 0.73300.5001.0000.917

Without NMS the ranking is D1 TP, D2 TP, D3 FP, D4 FP, D5 TP: when the last object is found precision is 3/5, so AP = 1/3·1 + 1/3·1 + 1/3·0.6 = 0.867. At 0.5 all three objects are found before the only FP (D8, score 0.35), so AP is 1.000 even though precision over all kept boxes is 0.75: AP only punishes an FP that ranks above a TP. At 0.3 the missed G3 caps recall at 2/3, and AP at 0.667 (Demo 8 fills the whole table). The AP of every class, averaged, is the mAP that papers report. ROC and AUC is a different way to sum up a ranking; detection uses AP because the number of true negatives (background boxes) is not defined.

"Non-maximum suppression" has a cousin in edge detection: Canny thins edges by keeping only the pixel with the largest gradient across an edge. Same idea, local maxima only, but on pixels instead of boxes. CNN vs Transformer compares the two kinds of backbone a detector can use.

See also Harris Corner Detection, which applies non-maximum suppression to pixels to keep one point per corner.

What the page leaves out

How the candidates are made (anchor boxes, region proposals, the box regression that refines them); more than one class; Soft-NMS, which lowers the scores of overlapping boxes instead of removing them; NMS-free detectors such as DETR; the score threshold before NMS and the cap on boxes per image; COCO's AP, which averages over IoU thresholds 0.50 to 0.95 and uses 101 recall points; and real images and networks.

On the mind map

Open the Machine Learning mind mapMachine LearningLearning setup: what feedback, how to scoreLearning setupLinear models: weighted sum w·x + bLinear modelsOther classic models: trees, margins, neighboursOther classic modelsUnsupervised: find structure without labelsUnsupervisedTraining: loss, gradient, small stepTrainingNeural networks: layers of σ(W·a + b)Neural networksArchitectures: how layers share informationArchitecturesLanguage models: predict the next tokenLanguage modelsLLM speed and tuning: cheaper to run, safer to useLLM speed and tuningRetrieval and agents: give the model data and toolsRetrieval and agentsKV cacheKV cachekeep past K and V, append one per tokenSpeculative decodingSpeculative decodingsmall model drafts, big one verifiesMixture of expertsMixture of expertsrouter sends each token to top-kLoRA and quantizationLoRA and quantizationlow-rank B·A update; fewer bitsRLHF vs DPORLHF vs DPOreward model + PPO vs direct lossMHA vs GQA vs MQAMHA vs GQA vs MQAfewer K/V heads, smaller KV cacheFlashAttentionFlashAttentiontiles in SRAM + online softmax, less HBMContinuous batchingContinuous batchingrequests join and leave every decode stepKnowledge distillationKnowledge distillationStudent learns teacher's soft probabilitiesTransfer learningTransfer learningFreeze backbone, train head, then fine-tuneObject detection NMSKeep top box, drop IoU > thresholdOpen the Machine Learning mind mapMachine Learningcheaper to run, safer to useLLM speed and tuningKV cacheKV cachekeep past K and V, append one per tokenSpeculative decodingSpeculative decodingsmall model drafts, big one verifiesMixture of expertsMixture of expertsrouter sends each token to top-kLoRA and quantizationLoRA and quantizationlow-rank B·A update; fewer bitsRLHF vs DPORLHF vs DPOreward model + PPO vs direct lossMHA vs GQA vs MQAMHA vs GQA vs MQAfewer K/V heads, smaller KV cacheFlashAttentionFlashAttentiontiles in SRAM + online softmax, less HBMContinuous batchingContinuous batchingrequests join and leave every decode stepKnowledge distillationKnowledge distillationStudent learns teacher's soft probabilitiesTransfer learningTransfer learningFreeze backbone, train head, then fine-tuneObject detection NMSKeep top box, drop IoU > threshold
This page on the Machine Learning mind map · 77 pages in 10 branches · open it to see the whole map and test yourself in recall mode.