Vol. XXIX · Multimodal Models Plate 3 of 7audio tokens, speech, spectrograms, Whisper, neural codecs, EnCodec, residual vector quantisationCE

Audio and Speech.

Turn sound into a sequence the model can read — and, for speech out, one it can write.

There are two ways to give a language model audio. Speech in can go through an encoder, as in Whisper (Radford et al., 2023): 30 seconds of audio become an 80-channel log-Mel spectrogram of 3,000 frames, convolutions halve that to 1,500 positions, and a Transformer encodes them for a decoder to read. Speech out — and fully speech-native models — need audio as discrete tokens the model can generate, which neural codecs provide: an encoder compresses audio to a stream of vectors at a fixed frame rate, and residual vector quantisation (RVQ) turns each vector into a few codebook indices, each codebook quantising what the previous ones left.

EnCodec (Défossez et al., 2023) runs 24 kHz audio at 75 frames per second and covers 1.5 to 24 kbps by the number of codebooks used. The workshop applies RVQ directly to one second of a synthetic signal at 8 kHz — frames of 16 samples, codebooks of 64 entries learned by k-means one after another — and measures reconstruction quality against the number of codebooks.

Below: signal-to-noise ratio of the reconstruction against the number of codebooks, and the tokens per second each setting produces.

What you will learn
Workshop
One second of a synthetic signal at 8 kHz (two tones, a glide and noise), cut into frames of 16 samples (500 frames per second). Each codebook has 64 entries (6 bits), learned by 12 rounds of k-means on the residual left by the codebooks before it. SNR: signal power over reconstruction error power, in decibels. Real codecs quantise a learned encoder’s output, not raw samples, at far lower frame rates.

Each codebook refines the last

One codebook reconstructs the signal at 11.7 dB; each further one quantises what is left, and eight reach 34.8 dB — about 3.3 dB per codebook, a steady refinement because each works on a smaller residual. The cost is tokens: here every codebook adds 500 tokens per second. At EnCodec’s 75 frames per second, eight codebooks would be 600 tokens per second of speech, against about 50 encoder positions per second for Whisper’s input — audio is far denser than text, whose speech runs at a few tokens per second.

Speech-native models manage that density by predicting codebooks hierarchically — the first codebook for content, later ones for acoustic detail, often by a smaller model — or by delaying and interleaving codebook streams so they can be generated in parallel; Moshi generates text and audio tokens together for real-time dialogue. Speech in is cheaper because an encoder can summarise; speech out must produce every frame, which is why low frame rates and few codebooks are central to their design.

The same RVQ idea compresses images (residual-quantised VAEs) and even KV caches; quantising what the previous stage left over is a general way to spend bits where error remains.

Terms

Spectrogram
Frequency content of audio over time, the usual encoder input.
Neural codec
A learned encoder–decoder that compresses audio to discrete tokens.
Residual vector quantisation
Quantising a vector with a sequence of codebooks, each on the residual.
Frame rate
Codec vectors per second of audio.
Audio becomes tokens through codecs with residual vector quantisation; each codebook adds fidelity and tokens, so audio is far denser than text.

Sources

  1. Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision” (Whisper), ICML 2023.
  2. Défossez, Copet, Synnaeve and Adi, “High Fidelity Neural Audio Compression” (EnCodec), TMLR 2023.
  3. Zeghidour et al., “SoundStream: An End-to-End Neural Audio Codec”, IEEE/ACM TASLP 2022.
  4. Défossez et al., “Moshi: a speech-text foundation model for real-time dialogue”, arXiv:2410.00037, 2024.