CLIP Visualization | Contrastive Loss & Zero-Shot Labels

The idea: learn from pairs, not from labels

CLIP (Radford et al., OpenAI, 2021) was trained on 400 million photos with the text found next to them on the web. Nobody labelled them "dog" or "car". The only signal is which caption belongs to which photo. That is contrastive learning: in a batch, the true pairs should look alike and every other combination should look different.

Photos go into an image encoder and captions into a text encoder. Both outputs are normalised to unit vectors in one shared space. The cosines of every photo with every caption form a 4 by 4 matrix whose diagonal holds the true pairs. — Two encoders, one space: the N × N matrix of cosines is all the loss looks at.

The canvas trains the last layer of both encoders on 4 pairs (dog, cat, car, pizza) and shows every number the loss uses. Start with Demo: encode the batch, then Demo: one training step.

One shared space of unit vectors

CLIP has an image encoder (a ResNet or a Vision Transformer) and a text encoder (a Transformer). They read different things, but each ends in a linear projection to the same size, 512 numbers for ViT-B/32. Each output is then divided by its length (L2 normalisation), so every photo and every caption is a unit vector. On the page the space has 2 numbers, so the unit vectors sit on a circle: squares are photos, circles are captions, one colour per pair.

For unit vectors the dot product is the cosine of the angle between them: 1 for the same direction, 0 at 90°, −1 opposite. The paper's pseudo-code is short:

I_e = l2_normalize(image_encoder(I) · W_i)    # [N, d_e]
T_e = l2_normalize(text_encoder(T)  · W_t)    # [N, d_e]
logits = (I_e · T_eᵀ) * exp(t)                # [N, N], exp(t) = 1/τ
labels = [0, 1, …, N−1]
loss = (cross_entropy(logits, labels, axis=0) +
        cross_entropy(logits, labels, axis=1)) / 2

The page runs exactly this. The backbones are frozen and give 4 hand-made features per item (fur, whiskers, wheels, cheese); only W_img and W_txt, 2 × 4 each, are trained.

The N × N similarity matrix

With N pairs in a batch there are N × N cosines. The N cells on the diagonal are the true pairs. The other N² − N cells are negatives, and they cost nothing: every other caption in the batch is a wrong answer for this photo. CLIP used N = 32,768, so each photo was compared with 32,767 wrong captions at every step. The ✓ column on the canvas marks rows whose largest cell is their own caption.

Symmetric InfoNCE loss

Read a row as a multiple-choice question: which of the N captions goes with this photo? Divide the row by the temperature τ, take a softmax, and the loss is the cross-entropy with the diagonal as the right answer, −log p(own caption). That is the InfoNCE loss (van den Oord et al., 2018). CLIP asks the question both ways: rows (image → text) and columns (text → image), and averages the two.

The same 4 by 4 matrix twice. On the left, row 1 is highlighted and its softmax gives probabilities over the 4 captions. On the right, column 1 is highlighted and its softmax gives probabilities over the 4 photos. The loss is the mean of minus log of the diagonal probabilities in both directions. — Rows ask "which caption?", columns ask "which photo?"; the loss averages both cross-entropies.

Its gradient on a cell is simple: (p − 1) on the diagonal, p elsewhere, from both softmaxes. So each true pair is pulled up and each negative is pushed down in proportion to how much probability it still takes. Negatives the model already rules out get almost no push. On the canvas the pulled cells turn green (↑) and the pushed ones red (↓) (Demo: one training step). Untrained, the loss is 8.23; after 30 steps it is 0.06 and all 4 rows are ✓ (Demo: train 30 steps).

Direction	Softmax over	Question
image → text	a row	which caption describes this photo?
text → image	a column	which photo does this caption describe?

Temperature

Cosines can't leave [−1, 1]. Fed straight into a softmax, the best possible row for 4 pairs 90° apart is softmax(1, 0, −1, 0), which gives only 0.53 to the right caption, so the loss can't go below 0.63 (Demo: τ = 1 is too soft). Dividing by a temperature τ stretches the logits: τ = 0.07 multiplies them by 14.3 and the softmax becomes sharp. CLIP learns τ as a parameter, starting at 0.07 and clipping 1/τ at 100. A small τ also concentrates the gradient on the hard negatives, the wrong captions that are nearly as close as the right one.

Zero-shot classification

After training, CLIP can label photos into classes it never trained on as classes. Write one sentence per label, "a photo of a {label}", encode the sentences with the text encoder, encode the photo with the image encoder, and pick the sentence with the highest cosine. The prompt vectors act as the weights of a linear classifier, written in words. CLIP ViT-L/14 reached 76% top-1 on ImageNet this way, with none of ImageNet's 1.28 million training labels.

A puppy photo is encoded into one unit vector. Four prompts, a photo of a dog, a cat, a car and a pizza, are encoded into four unit vectors. The cosines are 1.00, 0.88, minus 0.98 and minus 0.60; after softmax with temperature 0.07 dog gets probability 0.84. — Zero-shot: the labels are sentences, and the answer is the nearest one.

On the canvas the puppy, kitten and truck photos were never in the batch and still land next to the right prompt (Demo: zero-shot a new photo). Before training the same question is close to a guess (Demo: zero-shot before training). The burger has no label, so it gets the nearest one, pizza: zero-shot can only pick among the prompts you write. The wording matters too; the paper gained 1.3 points on ImageNet from "a photo of a {label}" over the bare label, and more by averaging 80 prompt templates.

False negatives and batch size

The loss assumes that every off-diagonal cell is wrong. If a batch holds two dog photos with two dog captions, the second dog caption counts as a wrong answer for the first dog photo. In Demo: two dogs in one batch the two dog rows each split their probability about 50/50 and the loss stalls at 0.36, close to ln 2 / 2 = 0.35. With 400 million web pairs and random batches this is rare enough to ignore; with small curated datasets it is a real problem. Bigger batches give more negatives per step, which is why contrastive methods like large batches (CLIP: 32,768; SimCLR: 4,096).

The same idea, a shared embedding space searched by cosine, is what dense retrieval uses for text (see BM25 vs vector search), and word vectors were learned with a contrastive loss years earlier (see word2vec negative sampling).

This page vs real CLIP

Part	This page	CLIP ViT-B/32
Batch size N	4	32,768
Embedding size	2 (a circle)	512
Encoders	frozen 4 features, trained 2 × 4 projection	ViT image encoder and 12-layer Transformer text encoder, all trained
Temperature	fixed, picked from 0.07 / 0.5 / 1	learned, starts at 0.07, 1/τ clipped at 100
Optimiser	plain gradient steps	Adam with weight decay, cosine schedule, 32 epochs
Data	4 hand-written pairs	400 million (image, text) pairs from the web

What the page leaves out

Real CLIP trains the whole image and text encoders, not only the final projection, and learns the temperature. Its batch is sharded over hundreds of GPUs, and each GPU gathers the embeddings from all the others to build the full 32,768 × 32,768 matrix. Later work changes the loss (SigLIP uses one sigmoid per cell instead of a softmax per row, so no all-gather is needed), filters the web data, or adds prompt ensembles and linear probes. The page also skips how the text is tokenised and how the encoders work inside.