Vol. XXIX · Multimodal Models Plate 1 of 7vision transformer, patches, image tokens, vision encoder, projection, LLaVA, ViTCE

Images as Tokens.

Cut the picture into squares; each square becomes a vector the model can attend to.

The Vision Transformer (Dosovitskiy et al., 2021) showed that the same architecture that reads text can read images if the image is made a sequence: split it into a grid of p × p patches, flatten each patch, project it to a vector with a learned matrix, add a position embedding, and run a Transformer over the result. Each patch is then one token’s worth of input. A 224 × 224 image in 16 × 16 patches is 196 tokens.

Most vision–language models attach such a vision encoder, pre-trained on images, to a language model with a small projector that maps the encoder’s output vectors into the language model’s embedding space; the projected vectors are then placed in the context like word embeddings. LLaVA-1.5 used a CLIP ViT-L/14 encoder at 336 pixels — 576 visual tokens per image — and a two-layer MLP projector. The workshop cuts a test image into patches of the chosen size and shows what each token covers.

Below: the test image, and the same image as patch tokens — each patch shown as its mean — for the chosen patch size; readouts give the token count at common resolutions.

What you will learn
Workshop
A 64 × 64 procedural test image (a disc, a square and a line on a gradient). Patch tokens: each p × p patch replaced by its mean grey level — a stand-in for the information one token carries before embedding, not for what a trained encoder extracts. Counts: (H/p)(W/p) without overlap.

Detail costs tokens quadratically

A 224-pixel image is 196 tokens in 16-pixel patches, the ViT standard; LLaVA-1.5’s 336-pixel input in 14-pixel patches is 576. Halving the patch size quadruples the count: a 1 024-pixel image is 4,096 tokens at 16 pixels and 5,476 at 14. Since attention cost grows with the square of sequence length (plate IV.7), high-resolution images are expensive, which is why systems tile large images, downscale them, or compress patch tokens before the language model sees them.

Smaller patches see finer detail — small text, thin lines — at that cost; larger ones blur it, as the patch view shows. The projector is the hinge of the design: a linear map or small MLP (LLaVA), cross-attention layers that let the language model read image features without putting them in the sequence (Flamingo), or a learned set of query vectors that summarises an image into a fixed number of tokens (BLIP-2’s Q-Former). Each trades how many tokens an image costs against how much of it the language model can use.

Reading text in images (documents, screenshots, charts) needs fine patches, which is why document-heavy models raise their image resolution and computer-use agents work hard on grounding (plate XXV.6).

Terms

Patch
A square block of pixels treated as one input token.
Vision encoder
A Transformer that turns patches into feature vectors.
Projector
The map from vision features into the language model’s embedding space.
Resolution
The pixel size an image is processed at, which sets its token count.
Images enter a Transformer as patch tokens; patch size and resolution set the token count, which grows with the square of resolution.

Sources

  1. Dosovitskiy et al., “An Image is Worth 16x16 Words”, ICLR 2021.
  2. Liu, Li, Wu and Lee, “Visual Instruction Tuning” (LLaVA), NeurIPS 2023; Liu, Li, Li and Lee, “Improved Baselines with Visual Instruction Tuning”, CVPR 2024.
  3. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning”, NeurIPS 2022.