The idea: one recipe, five ways to split the bits
A floating-point number is stored as three fields: a sign bit, an exponent that says which power of two the number is near, and a fraction (mantissa) that says where it is between that power and the next. Every format here uses the same recipe. They differ only in how many bits go to each field.
Machine learning moved from 32-bit to 16-bit and now 8-bit numbers because each halving doubles how many numbers fit in memory and how fast the hardware multiplies them. The price is range and precision, and the page shows exactly what is lost by storing one number in all five formats at once.
Sign, biased exponent, fraction
Write the number in binary scientific notation, x = ±1.f₂ × 2ᵉ. The sign bit is 0 for
plus, 1 for minus. The exponent field stores e + bias, with
bias = 2^(bits−1) − 1: 127 for an 8-bit exponent, 15 for 5 bits, 7 for 4 bits. That
keeps the field a plain unsigned number. The fraction field keeps the first m bits of
f; the leading 1 is not stored (the hidden bit).
For 0.1 in FP16: 0.1 = 1.1001100110011…₂ × 2⁻⁴. Sign 0, exponent −4 + 15 = 11 = 01011₂, fraction 1001100110 (the 11th bit onward is rounded to nearest, ties to even). The stored value is 0.0999755859375. (Demo 1: 0.1 in all five formats)
Exponent field 0 means a subnormal: no hidden 1, so the numbers go down evenly to zero but with fewer significant bits. Exponent all ones means ∞ or NaN in the IEEE formats.
Range vs precision
Exponent bits decide the range: how big and how small a number can be. Fraction bits decide the precision: how many significant digits survive, about (m + 1)·log₁₀2.
| Format | Sign · exp · frac | Bias | Largest | Smallest normal | Digits |
|---|---|---|---|---|---|
| FP32 | 1 · 8 · 23 | 127 | 3.4×10³⁸ | 1.2×10⁻³⁸ | 7.2 |
| FP16 | 1 · 5 · 10 | 15 | 65504 | 6.1×10⁻⁵ | 3.3 |
| BF16 | 1 · 8 · 7 | 127 | 3.4×10³⁸ | 1.2×10⁻³⁸ | 2.4 |
| FP8 E4M3 | 1 · 4 · 3 | 7 | 448 | 0.0156 | 1.2 |
| FP8 E5M2 | 1 · 5 · 2 | 15 | 57344 | 6.1×10⁻⁵ | 0.9 |
The spacing between neighbouring numbers doubles at every power of two. With 3 fraction bits there are 8 numbers per power of two; with 2 bits only 4. That is the whole difference between the two FP8 formats. (Demo 4: FP8 E4M3 vs E5M2 on 300 and 1000)
Overflow and underflow in FP16
FP16's largest number is 1.1111111111₂ × 2¹⁵ = 65504. Anything that rounds above it becomes ∞: 70000 does, and so does 65520, which lies exactly halfway to 65536 and rounds to the even pattern, 2¹⁶. (Demo 2, Demo 5) An activation or a sum of squares that passes 65504 turns into ∞, and the next subtraction makes it NaN.
At the other end, FP16's smallest subnormal is 2⁻²⁴ ≈ 6×10⁻⁸. Gradients in a deep network are
often smaller. A gradient of 2×10⁻⁸ rounds to 0 and the weight never moves. The fix in FP16
training is loss scaling: multiply the loss by a large factor (PyTorch's
GradScaler starts at 2¹⁶ = 65536), so every gradient is multiplied too, then divide
by it in FP32 before the update, and lower the factor whenever an ∞ appears.
(Demo 3: a tiny gradient underflows in FP16)
Why training uses BF16
BF16 (brain float, from Google Brain) is simply the top 16 bits of an FP32 number. It keeps all 8 exponent bits, so it has FP32's range: nothing overflows that would not overflow in FP32, and tiny gradients survive without loss scaling. It keeps only 7 fraction bits, but neural network training tolerates noisy arithmetic well. What it does not tolerate is ∞ and NaN. Converting FP32 to BF16 is also trivial: round and drop the low 16 bits.
In mixed precision training the matrix multiplies run in BF16 with FP32 accumulation, and an FP32 master copy of the weights takes the updates, because adding a small update to a weight needs more than 3 digits.
Why inference uses FP8
Inference is mostly reading weights from memory, so halving their size from 16 to 8 bits nearly doubles tokens per second, and recent GPUs (NVIDIA Hopper and Blackwell, AMD MI300) multiply FP8 at twice the BF16 rate. The OCP FP8 standard defines two formats. E4M3 has more precision and a range of only ±448; it gives up ∞ (the pattern S.1111.111 is its only NaN) to gain one more step at the top. E5M2 has the range of FP16 but only 2 fraction bits; it is used for gradients when training in FP8.
Because ±448 is so small, every FP8 tensor carries a scale: the values are divided by, for example, max|w| / 448 before the cast and multiplied back afterwards, so the tensor uses the whole range. Libraries convert with saturation (clamp to ±448), as the page does, rather than producing NaN.
See also Number Systems and IEEE 754 Floating Point for FP32 bit by bit and float addition, and LoRA and Quantization for int8 and int4, which store integers with a scale instead of floats.
What the page leaves out
The page converts one number at a time with round to nearest, ties to even. It does not do arithmetic in the formats (sums in BF16 lose small terms, which is why hardware accumulates in FP32), does not show NaN propagation or the signed zero, and skips the other rounding modes (stochastic rounding is used in some FP8 and BF16 training). It also leaves out FP64, TF32 (NVIDIA's 19-bit tensor-core format), the newer 4-bit MX and NVFP4 formats with a shared scale per block of 32 or 16 values, and how per-tensor scales are chosen from the history of max values.