Drills: Multimodal.
Five questions at a time from the whole volume, until every modality comes with its token count.
This plate draws rounds of five from the questions and predictions of the six plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that every modality becomes a sequence: images as patches or codes, audio as frames or codebook indices, generation as tokens or denoising steps — and every choice is paid for in tokens.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Count tokens for images and audio.
- Explain contrastive pretraining.
- Contrast fusion and generation methods.
- Budget image-heavy work.
What to carry forward
Images enter as patch tokens whose number grows with the square of resolution (plate XXIX.1); contrastive pretraining aligns images and text and generalises compositionally (plate XXIX.2); audio becomes dense codec tokens through residual vector quantisation (plate XXIX.3).
Early fusion makes images tokens in one vocabulary at the cost of length (plate XXIX.4); diffusion generates in steps whose number sets quality and cost (plate XXIX.5); and every image is billed by its patches up to a cap (plate XXIX.6). Vol. XXX turns to the research frontier.
Terms
- Patch token
- One image patch as a token.
- InfoNCE
- The contrastive loss.
- RVQ
- Residual vector quantisation.
- Early fusion
- Images in the text vocabulary.