Vol. XXIX · Multimodal Models Plate 2 of 7CLIP, contrastive learning, InfoNCE, vision-language pretraining, zero-shot, negatives, batch sizeLE

Training to See.

Pull matching image and caption together, push everything else in the batch apart.

CLIP (Radford et al., 2021) trained an image encoder and a text encoder together on 400 million image–caption pairs from the web with a contrastive objective: in each batch of N pairs, the N matching image–text similarities should be high and the N² − N mismatched ones low — the symmetric InfoNCE loss. The result is a shared space where any caption can be compared with any image, which gives zero-shot classification (score an image against “a photo of a dog”, “a photo of a cat”, …) and became the standard vision encoder for vision–language models.

The workshop trains two linear encoders on a toy world of 8 shapes × 8 colours: an “image” is a fixed random projection of its shape and colour plus noise, a “caption” a projection of the two words. 16 of the 64 combinations are held out; after training, an image of an unseen combination must be matched to its caption among all 64. Batch size sets the number of negatives each step.

Below: zero-shot accuracy on held-out combinations during training for the chosen batch size; chance is 1 in 64.

What you will learn
Workshop
8 shapes × 8 colours; inputs 32-dimensional; linear encoders into 8 dimensions, normalised; temperature τ = 0.1; 400 steps of plain gradient descent (learning rate 0.05) on batches of distinct training pairs; image noise of the chosen size. Held out: the 16 combinations with (shape + 2 × colour) divisible by 4. Zero-shot accuracy: 4 noisy images of each held-out combination matched against all 64 captions.

Negatives teach the space

With noisy images, after 40 steps a batch of 4 matches unseen combinations 16% of the time and a batch of 32 81%; by the end, 70% and 95%, against chance of 1 in 64. None of the 16 test combinations was ever seen as a pair, so the encoders have learned shape and colour separately and compose them — the property that makes zero-shot classification with new captions work. Each step, a larger batch contrasts every pair with more wrong partners, giving a richer signal about what distinguishes them.

CLIP’s batches held 32 768 pairs for exactly this reason, and later work (SigLIP) replaced the batch-wide softmax with independent sigmoid terms so batches can grow without the softmax’s cost. Vision–language models are then usually trained in stages: a contrastively pretrained encoder, a projector trained on image–caption data with the language model frozen, and joint fine-tuning on interleaved and instruction data so the model can answer questions about images, not just match them.

Contrastive models learn what captions on the web say about images, including their biases and blind spots: text in images, counting and spatial relations were long weak points.

Terms

Contrastive learning
Training matching pairs to be similar and mismatched pairs dissimilar.
InfoNCE
The softmax cross-entropy over a batch that implements it.
Zero-shot classification
Classifying by comparing an image with captions for each class.
Negatives
The mismatched pairs in a batch.
Contrastive pretraining aligns images and text in one space; batch negatives drive it, and composition lets it name unseen combinations.

Sources

  1. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision” (CLIP), ICML 2021.
  2. van den Oord, Li and Vinyals, “Representation Learning with Contrastive Predictive Coding” (InfoNCE), arXiv:1807.03748, 2018.
  3. Zhai et al., “Sigmoid Loss for Language Image Pre-Training” (SigLIP), ICCV 2023.