Vol. XXIX · Multimodal Models Plate 4 of 7early fusion, native multimodal, Chameleon, interleaved data, image tokenisers, VQ, late fusionCE

One Model for Everything.

Make images tokens in the same vocabulary as words, and train on documents that mix them.

The designs of plate XXIX.1 are late fusion: a separately trained vision encoder feeds a language model through a projector, and images can be read but not written. Early fusion goes further. An image tokeniser — a VQ autoencoder — turns each image into discrete codes from a codebook, the codes join the text vocabulary, and a single Transformer is trained from the start on interleaved sequences of text and image tokens, predicting both. Chameleon (Meta, 2024) encodes a 512 × 512 image as 1,024 tokens from a codebook of 8,192, inside a vocabulary of 65,536.

The appeal is one model, one sequence and one objective for any mix of modalities, and generation of images by the same next-token prediction as text. The cost is length: an image is as many tokens as a long paragraph, and training must balance modalities whose losses behave very differently. The workshop computes the sequence lengths and attention cost of documents mixing text and images under early fusion and a projector-based late fusion.

Below: tokens per document and relative attention cost, for the chosen number of images per document, under early fusion and late fusion with a compressed projector.

What you will learn
Workshop
A document of 1 000 text tokens with the chosen number of images. Early fusion: 1 024 tokens per image (Chameleon’s 512 × 512 tokeniser). Late fusion: 576 tokens per image (LLaVA-1.5), or 64 with a query-based compressor. Attention cost relative to the text alone, ∝ length².

Everything is a token, so everything costs tokens

A thousand words with four images is 5,096 tokens under early fusion, 80% of them image tokens, and its attention costs about 26 times the text alone. That is the central trade: early fusion buys one uniform model that reads and writes images with the same machinery as text, at the price of long sequences and of images dominating the token count. The Chameleon report describes the other cost — keeping training stable when modalities with very different statistics share every parameter — and the architectural changes (query–key normalisation among them) needed to do it.

Production systems mix the approaches: native multimodal pretraining (Gemini was trained multimodal from the start), learned compression of image tokens, separate decoders for generating images or audio at high fidelity (plate XXIX.5), and resolution chosen per task. The direction of travel has been towards models trained on interleaved data from the beginning, because they handle mixed documents — slides, web pages, papers with figures — more naturally than an encoder bolted onto a finished text model.

An image tokeniser decides what an early-fusion model can see and draw: text inside images is hard for VQ tokenisers to reconstruct, which the Chameleon authors note limits OCR-heavy tasks.

Terms

Late fusion
A separate encoder feeding a language model through a projector.
Early fusion
Image and text tokens in one vocabulary and one model from the start.
Image tokeniser
A VQ autoencoder turning images into discrete codes.
Interleaved data
Documents where text and images alternate.
Early fusion makes images tokens in the text vocabulary; one model reads and writes both, at the cost of long sequences and harder training.

Sources

  1. Chameleon Team, “Chameleon: Mixed-Modal Early-Fusion Foundation Models”, arXiv:2405.09818, 2024.
  2. van den Oord, Vinyals and Kavukcuoglu, “Neural Discrete Representation Learning” (VQ-VAE), NeurIPS 2017.
  3. Li, Li, Savarese and Hoi, “BLIP-2”, ICML 2023.
  4. Team Gemini, “Gemini: A Family of Highly Capable Multimodal Models”, arXiv:2312.11805, 2023.