Softmax & Cross-Entropy: from Logits to Loss and Gradient

The idea: ten scores in, one probability distribution out

The last layer of a classifier computes one number per class, a logit z_k. Logits can be any real numbers: negative, large, not adding up to anything. Softmax turns them into probabilities p_k that are positive and add up to 1. The cross-entropy loss then says how wrong the answer is: L = −log p_t, where t is the true class. Its gradient is the simplest in all of deep learning: p − y. On this page the 10 classes are the digits 0–9, as in Draw a Digit. Drag a logit bar up or down and every number is recomputed in your browser. Compute plays the steps one by one, and Gradient Step takes one learning step on the logits.

Softmax: exponentiate, then divide by the sum

pₖ = e^(zₖ) / Σⱼ e^(zⱼ)          k = 0 … 9

e^z is positive for every z, so every p_k is positive. Dividing by the sum makes them add up to 1. The largest logit gets the largest probability, and the order never changes. Because all classes share the sum, raising one logit lowers every other probability: the classes compete. The name says what it does: a "soft" version of argmax. A logit 1 higher than another gives a probability e ≈ 2.72 times larger.

Compare this with Draw a Digit, where each of the 10 outputs is a separate sigmoid σ(z_k) = 1 / (1 + e^−z_k). Each sigmoid is in 0 … 1, but they know nothing about each other: ten unrelated yes/no answers. Their sum can be anything from 0 to 10. The last column of the table shows σ(z_k) for the same logits, and the Σ row shows that it does not add up to 1.

Only differences matter: the stable form

Add the same constant c to every logit. Every e^z_k+c = e^z_k·e^c is multiplied by the same e^c, and so is the sum, so e^c cancels:

e^(zₖ + c) / Σⱼ e^(zⱼ + c) = e^c·e^(zₖ) / (e^c·Σⱼ e^(zⱼ)) = pₖ

So softmax only sees the differences between logits. That matters in practice. A double (a JavaScript number) can hold at most 1.8·10³⁰⁸ ≈ e^709.8; e¹⁰⁰⁰ is Infinity, and Infinity / Infinity is NaN (in 32-bit floats the limit is already e^88.7). Real code therefore picks c = −max z:

m  = max z
uₖ = zₖ − m              every uₖ ≤ 0, the largest is exactly 0
pₖ = e^(uₖ) / Σⱼ e^(uⱼ)  every e^(uₖ) is in (0, 1], and the sum is between 1 and 10

The table shows both forms side by side: the naive columns e^z_k and naive p_k, and the stable columns u_k, e^u_k and p_k. They agree to about 10⁻¹⁶ until the naive one overflows (Demo: add a constant, then overflow at z = 1000: adding 3 changes nothing; adding 1000 turns the naive column into Infinity and NaN, while the stable p is unchanged).

Cross-entropy: −log of the true class's probability

Write the true answer as a one-hot vector y: y_t = 1 for the true class t, 0 for the rest. The cross-entropy between y and p is

L = −Σₖ yₖ · log pₖ = −log pₜ

Only the true class's probability counts. L = 0 when p_t = 1; L = log 10 = 2.303 when the network spreads its bets evenly over 10 classes (Demo: all logits equal, L = log 10); and L grows without limit as p_t → 0. A confident wrong answer costs a lot: p_t = 0.001 gives L = 6.9 (Demo: confident and wrong: z₇ = 6, z₃ = −1, true class 3, L = 7.03). A confident right answer costs almost nothing (Demo: confident and right: p₃ = 0.969, L = 0.031).

The page computes L as log Σ_j e^u_j − u_t, which is the same number but never takes the log of 0, even when p_t underflows. Frameworks do the same: PyTorch's CrossEntropyLoss takes the logits, not the probabilities, and fuses softmax and log ("log-softmax").

Why −log? Over a training set, the total loss Σ_i −log p_ti is −log Π_i p_ti: minus the log of the probability the model gives to all the observed labels. Minimising cross-entropy is maximum likelihood. The log turns a product of many small probabilities into a sum, and it is also what makes the gradient so simple.

The gradient is p − y

Put softmax into L and simplify:

L = −log pₜ = −log( e^(zₜ) / Σⱼ e^(zⱼ) ) = −zₜ + log Σⱼ e^(zⱼ)

Differentiate with respect to one logit z_k. The first term gives −1 if k = t, else 0, that is −y_k. The second gives e^z_k / Σ_j e^z_j, which is p_k:

∂L/∂zₖ = pₖ − yₖ

The same result with the chain rule, through p_t. The derivative of softmax is ∂p_t/∂z_k = p_t(y_k − p_k) (with y_k = 1 only for k = t), and ∂L/∂p_t = −1/p_t. Multiply: ∂L/∂z_k = −1/p_t · p_t(y_k − p_k) = p_k − y_k. The p_t of the log cancels the p_t of the softmax derivative.

Read it as two rules. The true logit gets p_t − 1 ≤ 0: a gradient step pushes it up by 1 − p_t, the probability still missing. Every other logit gets +p_k: it is pushed down in proportion to its own probability, so the most dangerous wrong class is pushed hardest. The gradients add up to Σ p_k − 1 = 0. The arrows on the logit bars are these pushes, −∂L/∂z_k (40 px per unit). One Gradient Step z ← z − η·(p − y) with η = 1 moves every bar the way its arrow points, and the loss falls (from 0.774 to 0.481 at the start). The verification also checks each entry against a finite difference (L(z + h) − L(z − h)) / 2h.

In a full network the logits are z = W·a + b of the last layer, so the backpropagation of Backpropagation starts from δ = p − y: ∂L/∂b = p − y and ∂L/∂W = (p − y)·aᵀ.

Why it trains faster than sigmoid + squared error

Draw a Digit uses separate sigmoids and the squared error C = Σ (a − y)². For the output of the true class its gradient has an extra factor σ'(z) = a(1 − a):

sigmoid + squared error:   ∂C/∂zₜ = 2(aₜ − 1) · aₜ(1 − aₜ)
softmax + cross-entropy:   ∂L/∂zₜ = pₜ − 1

When the network is confidently wrong, a_t is near 0, and so is a_t(1 − a_t). The sigmoid has saturated: at a_t = 0.001 the push is only 0.002. The network learns slowest exactly when it is most wrong. With softmax and cross-entropy the −log cancels the flat part, there is no σ' factor, and the push is 0.999 (Demo: confident and wrong: one step lowers L from 7.03 to 5.11). Nielsen's book calls this the learning slowdown, and cross-entropy its cure. A single sigmoid with binary cross-entropy has the same property: ∂L/∂z = a − y.

Temperature

Dividing every logit by a temperature T before softmax changes how peaked p is, never the order:

pₖ = e^(zₖ/T) / Σⱼ e^(zⱼ/T)          ∂L/∂zₖ = (pₖ − yₖ) / T

T = 1 is plain softmax. A high T shrinks the differences, so p moves towards uniform (0.1 each); as T → ∞ it becomes exactly uniform. A low T stretches them, so p moves towards a one-hot vector on the largest logit; as T → 0 softmax becomes argmax. With the starting logits, T = 5 gives L = 1.889 and T = 0.5 gives p₃ = 0.797, L = 0.226 (Demo: temperature high vs low). Language models use T when they sample the next token (see How a Large Language Model Works), and knowledge distillation trains a small network on a big one's softened probabilities at T > 1.

A worked example

Three classes, logits z = (2, 1, 0.1), true class t = 0, T = 1:

Step	Class 0	Class 1	Class 2	Note
u = z − max z	0	−1	−1.9	max z = 2
eu	1	0.368	0.150	Σ = 1.517
ez (naive)	7.389	2.718	1.105	Σ = 11.212 = 1.517·e²
p	0.659	0.242	0.099	both forms; Σ p = 1
y	1	0	0	one-hot, t = 0
∂L/∂z = p − y	−0.341	+0.242	+0.099	adds up to 0
z after one step (η = 1)	2.341	0.758	0.001	p0 = 0.768

The loss is L = −log 0.659 = 0.417 before the step and −log 0.768 = 0.264 after it. The separate sigmoids of the same logits would be 0.881, 0.731 and 0.525, which add up to 2.14.

See also Neural Network Forward Pass (a whole network ending in softmax) and Gradient Descent (what one step z ← z − η·∇L does on a loss surface).

See also Decoding Strategies for temperature and top-p used to pick the next token.

See also Mixture of Experts: a router uses softmax to turn expert scores into gate weights.

What the page leaves out

The page moves the logits directly; a real network moves the weights that produce them, and averages the gradient over a mini-batch of examples. It uses one-hot labels; label smoothing (for example 0.91 on the true class and 0.01 on the others) is common and gives the gradient p − y with that soft y. Language models apply softmax over 50,000 or more tokens, and some training methods only sample part of that sum. The binary case (two classes) is a single sigmoid with binary cross-entropy, which this page does not draw separately.

References

Michael Nielsen — Neural Networks and Deep Learning, ch. 3: the cross-entropy cost function and softmax

Goodfellow, Bengio, Courville — Deep Learning, §6.2.2.3: Softmax units for multinoulli output distributions

PyTorch documentation — torch.nn.CrossEntropyLoss