EMNLP Findings2024
PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain
Jianyi Chen, Zheqi Dai, Zhen Ye, Xu Tan, Qifeng Liu, Yike Guo, Wei Xue
Abstract
Generating well-structured long music compositions, spanning several minutes, remains a challenge due to inefficient representation and the lack of structured representation. In this paper, we propose PyramidCodec, a hierarchical discrete representation of audio, for long audio-domain music generation. Specifically, we employ residual vector quantization on different levels of features to obtain the hierarchical discrete representation. The highest level of features has the largest hop size, resulting in the most compact token sequence. The quantized higher-level representation is upsampled and combined with lower-level features to apply residual vector quantization and obtain lower-level discrete representations. Furthermore, we design a hierarchical training strategy to ensure that the details are gradually added with more levels of tokens. By performing hierarchical tokenization, the overall token sequence represents information at various scales, facilitating long-context modeling in music and enabling the generation of well-structured compositions. The experimental results demonstrate that our proposed Pyra-midCodec achieves competitive performance in terms of reconstruction quality and token per second (TPS). By enabling ultra-long music modeling at the lowest level, the proposed approach facilitates training a language model that can generate well-structured long-form music for up to 3 minutes, whose quality is further demonstrated by subjective and objective evaluations. The samples can be found at https://pyramidcodec.github.io/.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding cites High-Fidelity Audio Compression with Improved RVQGAN
- Simple and Controllable Music Generation cites High-Fidelity Audio Compression with Improved RVQGAN
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction cites Finite Scalar Quantization: VQ-VAE Made Simple
Cited by top-tier papers1
Builds on15
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale PredictionKeyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng et al.NeurIPS 2024 · 1,199 citations
- High-Fidelity Audio Compression with Improved RVQGANRithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar et al.NeurIPS 2023 · 910 citations
- Simple and Controllable Music GenerationJade Copet, Felix Kreuk, Itai Gat, Tal Remez et al.NeurIPS 2023 · 843 citations
- Finite Scalar Quantization: VQ-VAE Made SimpleFabian Mentzer, David Minnen, Eirikur Agustsson, Michael TschannenICLR 2024 · 442 citations
- MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised TrainingYizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma et al.ICLR 2024 · 277 citations
Related papers
- MuCodec: Ultra Low-Bitrate Music Codec for Music GenerationYaoxun Xu, Hangting Chen, Jianwei Yu, Wei Tan et al.ACM MM 2025 · 4 citations
- Language-Codec: Bridging Discrete Codec Representations and Speech Language ModelsShengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang et al.ACL 2025
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language ModelingShengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen et al.ICLR 2025
- Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music GenerationBoyu Cao, Lekai Qian, Dehan Li, Haoyu Gu et al.ACL Findings 2026
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-SpeechJaehyeon Kim, Keon Lee, Seungjun Chung, Jaewoong ChoICLR 2024 · 67 citations
