The idea: a bend after every weighted sum
A neuron computes a weighted sum z = w·x + b and then passes it through an activation function: its output is a = f(z). Without f, every layer is a linear map, and any stack of linear maps is one linear map, so depth would buy nothing (see Neural Network Forward Pass). The page compares the four functions you meet most: sigmoid, tanh, ReLU and GELU. Each lane draws the function in blue and its slope f′(z) in orange, and all four lanes get the same numbers.
| Function | f(z) | f′(z) | Largest slope |
|---|---|---|---|
| Sigmoid | 1 / (1 + e^−z) | σ(z)·(1 − σ(z)) | 0.25 at z = 0 |
| tanh | (e^z − e^−z) / (e^z + e^−z) | 1 − tanh²(z) | 1 at z = 0 |
| ReLU | max(0, z) | 1 if z > 0, else 0 | 1 for every z > 0 |
| GELU | z·Φ(z) | Φ(z) + z·φ(z) | about 1.13 at z ≈ 1.4 |
Φ is the standard normal CDF and φ its density. Demo 5: probe the curves moves a dot along all four curves at once.
The derivative is what backprop sees
Training needs the gradient of the loss with respect to every weight. Backpropagation gets it with the chain rule, from the loss backwards. Each time the gradient passes back through a neuron, it is multiplied by that neuron's f′(z) (and by a weight). The page fixes every weight at 1 and uses a chain of 8 one-neuron layers, so the gradient reaching layer k is just the product f′(z_k) · f′(z_k+1) · … · f′(z_8). The bars show it on a log scale, from 1 down to 10⁻¹².
Vanishing gradients
If every factor is below 1, the product shrinks exponentially with depth. Sigmoid's slope is at most 0.25, so after 8 layers the gradient is at most 0.25⁸ ≈ 1.5·10⁻⁵. With z = 0.5 at every layer, layer 1 gets 9.3·10⁻⁶: its weights barely move, while the last layer's move normally. This is the vanishing gradient problem that kept deep sigmoid networks from training (Demo 1: z = 0.5 everywhere). tanh is better near 0 (slope up to 1) and ends at 0.146, but both S-curves go flat for large |z|. That is saturation: at z = 3 sigmoid's slope is 0.045 and tanh's 0.0099, and the gradient at layer 1 drops to 1.7·10⁻¹¹ and 9·10⁻¹⁷ (Demo 2: saturated z = 3).
ReLU: no shrinking, but dead neurons
ReLU's slope is exactly 1 for every positive z, so a path of active neurons passes the gradient unchanged, however deep the network. That, and being cheap to compute, made ReLU the default for deep networks. The price is the other half: for z < 0 the slope is exactly 0. One negative z on the path cuts the gradient to 0 for every layer below it (Demo 3: one negative z). If a neuron's z is negative for every input, for example after a big update pushed its bias far down, it outputs 0 and gets gradient 0 on every example, so its weights never change again. That is a dead ReLU (Demo 4: z = −1 everywhere). Leaky ReLU (max(0.01·z, z)) keeps a small slope on the negative side to avoid it.
GELU: a smooth ReLU
GELU multiplies z by Φ(z), the chance that a standard normal value is below z. For large positive z it is about z (slope about 1), for large negative z about 0, and in between it curves smoothly, dipping slightly below 0 near z = −0.75. Its slope at a negative z is small but not 0: at z = −0.4 it is 0.197, so in Demo 3 GELU still delivers 0.257 to layer 1 where ReLU delivers 0. It is not magic, though: at z = −1 its slope is −0.083, and eight of those leave 2.3·10⁻⁹. GELU is the activation in BERT and GPT-style transformers.
Which one to use
| Function | Good at | Watch out for |
|---|---|---|
| Sigmoid | Output layer for a yes/no probability (0 to 1) | Vanishing gradients in hidden layers; outputs not centred on 0 |
| tanh | Hidden state in RNNs and LSTMs; centred on 0 | Saturates for |z| above about 2 |
| ReLU | Default for deep feed-forward and convolutional nets; cheap | Dead neurons; needs a sensible learning rate and He initialization |
| GELU | Transformers; smooth, keeps a little gradient for z < 0 | Costs more to compute; still small for very negative z |
What the page leaves out
Real layers have many neurons and weights other than 1: the gradient is multiplied by weight matrices too, and initializations such as Xavier and He scale them so the product neither shrinks nor explodes. Normalization layers (batch norm, layer norm) and residual connections keep z in the useful range and give the gradient a path around the activations, which is how very deep networks train today. The page also skips leaky ReLU, ELU, SiLU/Swish, softmax for the output layer, and the tanh approximation of GELU that some libraries use.