The idea: one gradient, three ways to use it
Training a model means walking downhill on a loss L(w). At every point the gradient ∇L(w) says which way is uphill. All three optimizers on this page start at the same point and get exactly the same gradient. They differ only in what they do with it: SGD steps straight against it, momentum adds it to a running velocity, and Adam divides it by its own typical size, coordinate by coordinate. One Step gives each of them one update, so the race is fair: same surface, same start, same number of gradients.
SGD: straight downhill
w ← w − η·∇L. No memory: each step uses only the gradient at the current point. On a round bowl, where every direction curves the same, this is already the best direction, so SGD walks in along a straight line (Demo 1: 42 steps, the shortest path, 3.56). With a bigger η on the same bowl it is the fastest of the three, 16 steps (Demo 6). On real data "SGD" also means the gradient comes from a small random batch; here the gradient is exact, so the page isolates the update rule (see Gradient Descent for batch vs mini-batch).
Momentum: a heavy ball
v ← β·v + ∇L, w ← w − η·v with β = 0.9. The velocity remembers past gradients. Where the gradient keeps its sign, as along the floor of a valley, the velocity grows up to 1/(1 − β) = 10 times a single gradient. Where it flips sign, as across the valley, the parts cancel. On the narrow valley, L = ½(w1² + 50·w2²), momentum reaches the target in 87 steps while SGD needs 127 (Demo 2). It is not a smooth glide, though: the velocity also carries it across the valley and past the minimum, so its path is the longest (24.76 vs 9.64). On the round bowl it overshoots and circles in, and still finishes first, 34 steps against 42.
Adam: the same step size in every coordinate
Adam keeps two running averages per parameter: m, the mean of the gradient (β1 = 0.9), and s, the mean of its square (β2 = 0.999). Both start at 0, so they are divided by (1 − βᵗ) to remove that start-up bias, giving m̂ and ŝ. The step is −η·m̂ / (√ŝ + ε). When the gradient is steady, m̂ / √ŝ is about ±1, so every coordinate moves about η, whether its gradient is 50 or 0.0001. That is why Adam's first step is the same diagonal on every surface, and why it wins at the saddle point: starting at w2 = 0.01, just beside the ridge, the w2 gradient is tiny, and SGD and momentum need many steps for it to grow, but Adam scales it up to a full step and rolls off the ridge first (Demo 3: Adam 32, momentum 53, SGD 61). The price shows on the round bowl: m carries Adam past the minimum like a small momentum, and √ŝ still remembers the large gradients from the start, so its steps near the bottom are small. There it is the slowest, 75 steps.
When to pick which
| Situation | SGD | Momentum | Adam |
|---|---|---|---|
| Round bowl, η = 0.1 (Demo 1) | ✓ 42 steps, straight | ✓ 34, circles in | ✓ 75 |
| Narrow valley (condition number 50) (Demo 2) | ✓ 127, zig-zags then crawls | ✓ 87 | ✓ 95, smooth curve |
| Saddle point (Demo 3) | ✓ 61 | ✓ 53 | ✓ 32 |
| Valley, η × 1.2 (Demo 4) | ✗ diverges at step 17 | ✓ 83 | ✓ 85 |
| Valley, η × 2.5 (Demo 5) | ✗ step 2 | ✗ step 2 | ✓ 67 |
| Round bowl, η × 2.5 (Demo 6) | ✓ 16 | ✓ 21 | ✓ 56 |
| In practice | with a tuned schedule, often the best final accuracy (vision models) | the usual form of SGD in deep learning (β = 0.9) | default for transformers and quick experiments; least tuning |
No optimizer wins every race. The shape of the loss decides: a well-conditioned bowl rewards the simplest rule, a long valley rewards memory, and tiny or badly scaled gradients reward Adam's per-coordinate scaling.
Learning rate and stability
Along a direction where the loss curves with strength λ (50 across the valley), one SGD step multiplies the distance to the minimum by 1 − η·λ. SGD converges only if η·λ < 2. Momentum's limit is wider, η·λ < 2·(1 + β) = 3.8. With η × 1.2 = 0.042, η·λ = 2.1: SGD's zig-zag grows by 1.1 each step and leaves the map at step 17, while momentum still converges (Demo 4). With η × 2.5 = 0.0875, η·λ = 4.4 breaks both limits and both leave the map at step 2 (Demo 5). Adam's step stays near η whatever the gradient, so a too-large η cannot make it explode: it still finishes in 67 steps.
What to try
- Step through the first two updates and compare the three lane cards: same ∇L, three different steps.
- Race to the End on each surface, then switch η to × 1.2 and race again.
- Watch the loss chart: momentum's curve goes up and down as it swings; Adam's flattens out near the bottom.
See also Gradient Descent for one optimizer at a time (learning rate, zig-zag, batch vs mini-batch vs SGD) and Backpropagation for where the gradient of a neural network comes from.
What the page leaves out
The surfaces have two parameters and exact gradients; real training has millions of parameters and noisy mini-batch gradients, and the noise changes the picture (it can help escape saddles even for SGD). The valley lines up with the axes, which suits Adam's per-coordinate scaling; a rotated valley would help it less. The page leaves out Nesterov momentum, AdamW and weight decay, RMSProp and AdaGrad, learning-rate schedules and warm-up, gradient clipping, and generalization (which optimizer gives the better model on new data, not just the lower training loss). Adam's η here is 0.1 because the problem is tiny; for neural networks the usual default is 0.001.