Residual Vector Quantisation

A single codebook of 1024 vectors cannot faithfully represent a 128-dimensional continuous audio embedding - the reconstruction error is simply too large for perceptually transparent audio. Residual vector quantisation (RVQ) solves this by chaining several codebooks together, where each stage quantises what the previous stage got wrong. SoundStream (Zeghidour et al., 2021) showed that eight such codebook stages let a neural codec operate at 3 kbps while outperforming Opus at 12 kbps on perceptual quality scores.

From scalar to vector to residual quantisation

Scalar quantisation maps each dimension of a vector independently to the nearest bin. It is fast but ignores correlations between dimensions, so it wastes representation capacity.

Vector quantisation (VQ) treats the entire embedding as one point in a high-dimensional space and finds the nearest entry in a fixed lookup table called a codebook. During training the codebook is learnt via a commitment loss that pulls encoder outputs towards codebook entries:

L_vq = || sg[z_e] - e ||^2 + beta * || z_e - sg[e] ||^2

where z_e is the encoder output, e is the nearest codebook entry, sg[·] is stop-gradient, and beta (typically 0.25) controls how hard the encoder is pushed towards the codebook. The quantised vector e replaces z_e for decoding; gradients flow back through a straight-through estimator.

A codebook of size K can represent K distinct embeddings. For audio, even K = 8192 entries is inadequate if the embedding dimension is large - you quickly hit the curse of dimensionality where most codebook entries are never used (codebook collapse).

Residual VQ sidesteps this by decomposing the quantisation across N stages:

r_0 = z_e                        # start with encoder output
q_1 = nearest(r_0, C_1)         # quantise with codebook 1
r_1 = r_0 - q_1                 # residual after stage 1
q_2 = nearest(r_1, C_2)         # quantise the residual
r_2 = r_1 - q_2
...
z_q = q_1 + q_2 + ... + q_N     # sum of all codebook lookups

Each codebook need only represent the error of the previous stage, which is a much smaller signal. In practice, N stages of size K each give you the expressiveness of K^N combinations while transmitting only N log2(K) bits per frame.

Bitrate arithmetic

For audio encoded at 75 frames per second with N=8 codebook stages of size K=1024:

bits per second = 75 frames/s × 8 stages × log2(1024) bits/stage
               = 75 × 8 × 10
               = 6000 bps  (6 kbps)

Halving the number of stages halves the bitrate exactly. This is why RVQ codecs can operate at multiple bitrates from a single model - you just transmit fewer stages and instruct the decoder to ignore the rest. SoundStream achieves this with structured dropout during training: codebook prefixes of length 1 to N are sampled uniformly, so the decoder learns to reconstruct from any subset.

How the codebooks are trained

All N codebooks are trained jointly with the encoder and decoder. The commitment loss extends to a sum over stages:

L_commit = sum_{i=1}^{N} || sg[r_{i-1}] - q_i ||^2

Codebook vectors themselves are updated by exponential moving average (EMA) of the encoder outputs assigned to them - this is more stable than gradient descent directly on the codebook:

N_i <- decay * N_i + (1 - decay) * count_i
m_i <- decay * m_i + (1 - decay) * sum_of_assigned_z_e
e_i <- m_i / N_i

Low-usage entries are periodically re-initialised from random encoder outputs to prevent collapse. EnCodec (Defossez et al., 2022) adds an entropy-based regulariser that explicitly penalises uneven codebook usage, pushing each entry to carry equal probability mass.

Why RVQ matters for codec language models

Once audio is discretised by RVQ, a language model can treat each frame as a small sequence of token IDs - one per codebook stage. VALL-E (Wang et al., 2023) exploits this structure directly: the first codebook stage is modelled autoregressively (capturing coarse prosody and speaker identity), while later stages are predicted in parallel via non-autoregressive modelling (filling in fine acoustic detail). This two-tier strategy reflects a real property of RVQ: stage 1 codes carry the most perceptual information; later stages correct increasingly subtle distortions.

Dropping stages 4-8 gives a recognisable but slightly hollow reconstruction; dropping stage 1 gives unintelligible noise - which is exactly why language models treat the stages asymmetrically.

When it falls down

Codebook collapse is the most common training failure. If the encoder learns a very low-variance representation, many codebook entries are never assigned and a few attract almost all inputs. The model then effectively uses far fewer bits than its nominal bitrate. EMA updates and re-initialisation reduce but do not eliminate this risk; monitoring per-entry usage during training is essential.

Mismatch between training and inference bitrates can degrade quality when structured dropout is used. A model trained to also function at N=1 may sacrifice some top-bitrate quality in favour of robustness at low bitrates. If the deployment target is always full bitrate, training without dropout is marginally better.

Temporal resolution is fixed at encoding. The strided convolution that produces frames is chosen once; it cannot adapt to locally easy or hard regions of audio. Silence consumes as many bits as a complex transient. Learned variable-rate coding (e.g., using a Gaussian entropy model to skip low-information frames) is an active research direction but not yet mainstream.

Out-of-distribution audio. Codebooks optimised for speech generalise poorly to music or environmental sounds unless trained jointly. Codebook entries cluster around speech-typical activations; music activations land in sparse regions and the nearest-neighbour lookup produces large residuals that even eight stages cannot cover.

Streaming and causality constraints. Standard RVQ uses non-causal convolutions. Real-time streaming codecs must replace these with causal layers, introducing an architectural penalty (larger receptive fields for the same quality). SoundStream addresses this with a causal variant at the cost of slightly higher bitrate for equivalent MUSHRA scores.

Further reading

• SoundStream: An End-to-End Neural Audio Codec (Zeghidour et al., 2021)
• High Fidelity Neural Audio Compression - EnCodec (Defossez et al., 2022)
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers - VALL-E (Wang et al., 2023)