Vol. XXIX · Multimodal Models Plate 5 of 7image generation, diffusion models, autoregressive generation, denoising, sampling steps, scoreLE

Generating Images and Sound.

Predict the next token, or start from noise and remove it a little at a time.

A model can produce an image in two broadly different ways. Autoregressively: generate the image’s discrete tokens one after another, exactly as for text (plate XXIX.4) — a 1 024-token image is 1 024 sequential steps. Or by diffusion (Ho et al., 2020; Song et al., 2021): learn to undo a process that gradually adds noise, then generate by starting from pure noise and denoising in a sequence of steps, each a pass of the network over the whole image at once. Diffusion dominates image, video and much audio generation; language models connect to it by writing the conditioning the diffusion model draws from, or by sharing a backbone with it.

The workshop shows the diffusion mechanism exactly on a small case: data from 8 Gaussian clusters on a circle, for which the denoising direction (the score of the noised data) has a closed form. 400 samples start as wide Gaussian noise and follow the probability-flow ODE down to almost no noise in the chosen number of steps.

Below: the 400 generated points against the 8 data modes for the chosen number of denoising steps, and sample quality against steps.

What you will learn
Workshop
Data: 8 Gaussians (σ = 0.05) on the unit circle. Samples start from a Gaussian of σ = 3 and follow the probability-flow ODE with the exact score down to σ = 0.002, in the chosen number of Euler steps on a geometric schedule. Near: within 0.15 of a mode. Covered: modes receiving at least 4 of the 400 samples.

Steps buy quality

With a single step, none of the samples lands near a mode (0%): one jump from wide noise cannot follow the curved path to the data. Two steps already reach every mode (100% covered) but put only 4% of samples close; five steps put 69% near, ten 87%, fifty 97%. Each step is a full evaluation of the denoiser, so the number of steps is the cost of a sample — far fewer than an autoregressive model’s one step per token, but each step processes the whole image.

Real diffusion models learn the score with a network rather than knowing it, and much of the field’s progress has been reducing the steps needed — better ODE solvers, distillation into one or a few steps, consistency models — and moving diffusion into a compressed latent space so each step is cheaper. Autoregressive image generation remains attractive for models that also write text, and hybrid designs use a language model for layout and meaning and a diffusion decoder for pixels.

Diffusion has also been applied to text itself — generating all tokens of a passage in parallel and refining them — a direction plate XXX.4 returns to.

Terms

Diffusion model
A generator that learns to reverse a noising process.
Score
The gradient of the log-density, pointing towards more likely data.
Denoising step
One pass of the network that removes some noise.
Probability-flow ODE
A deterministic path from noise to data with the same marginals as diffusion.
Generation is token by token or noise to data in steps; for diffusion, the number of denoising steps sets both quality and cost.

Sources

  1. Ho, Jain and Abbeel, “Denoising Diffusion Probabilistic Models”, NeurIPS 2020.
  2. Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations”, ICLR 2021.
  3. Karras, Aittala, Aila and Laine, “Elucidating the Design Space of Diffusion-Based Generative Models”, NeurIPS 2022.
  4. Ramesh et al., “Zero-Shot Text-to-Image Generation” (DALL·E), ICML 2021 (autoregressive image tokens).