Lune

ACL Findings2021

MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training

Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, Tie-Yan Liu

Abstract

Symbolic music understanding, which refers to the understanding of music from the symbolic data (e.g., MIDI format, but not audio), covers many music applications such as genre classification, emotion classification, and music pieces matching. While good music representations are beneficial for these applications, the lack of training data hinders representation learning. Inspired by the success of pre-training models in natural language processing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music understanding. To this end, we construct a large-scale symbolic music corpus that contains more than 1 million music songs. Since symbolic music contains more structural (e.g., bar, position) and diverse information (e.g., tempo, instrument, and pitch), simply adopting the pre-training techniques from NLP to symbolic music only brings marginal gains. Therefore, we design several mechanisms, including OctupleMIDI encoding and bar-level masking strategy, to enhance pre-training with symbolic music data. Experiments demonstrate the advantages of MusicBERT on four music understanding tasks, including melody completion, accompaniment suggestion, genre classification, and style classification. Ablation studies also verify the effectiveness of our designs of OctupleMIDI encoding and barlevel masking strategy in MusicBERT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

12 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Contrastive Learning with Positive-Negative Frame Mask for Music Representation cites PopMAG: Pop Music Accompaniment Generation
  • Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization cites Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
  • Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization cites PopMAG: Pop Music Accompaniment Generation
  • Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization cites Byte Pair Encoding for Symbolic Music
  • Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs cites Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
  • Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation cites Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs
  • Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation cites Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
  • Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation cites PopMAG: Pop Music Accompaniment Generation
  • PopMAG: Pop Music Accompaniment Generation cites Language Models are Few-Shot Learners
  • PopMAG: Pop Music Accompaniment Generation cites Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
  • Byte Pair Encoding for Symbolic Music cites Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs
  • Byte Pair Encoding for Symbolic Music cites Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
  • Byte Pair Encoding for Symbolic Music cites PopMAG: Pop Music Accompaniment Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers12

  1. Museformer: Transformer with Fine- and Coarse-Grained Attention for Music GenerationBotao Yu, Peiling Lu, Rui Wang, Wei Hu et al.NeurIPS 2022 · 104 citations
  2. Contrastive Learning with Positive-Negative Frame Mask for Music RepresentationDong Yao, Zhou Zhao, Shengyu Zhang, Jieming Zhu et al.WWW 2022 · 26 citations
  3. Exploring the Needs of Practising Musicians in Co-Creative AI Through Co-DesignStephen James Krol, Maria Teresa Llano Rodriguez, Miguel Loor ParedesCHI 2025 · 13 citations
  4. Byte Pair Encoding for Symbolic MusicNathan Fradet, Nicolas Gutowski, Fabien Chhel, Jean-Pierre BriotEMNLP 2023 · 8 citations
  5. Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured TokenizationLongshen Ou, Jingwei Zhao, Ziyu Wang, Gus Xia et al.NeurIPS 2025 · 5 citations

Builds on7

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
  3. Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano CompositionsYu-Siang Huang, Yi-Hsuan YangACM MM 2020 · 265 citations
  4. Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed HypergraphsWen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, Yi-Hsuan YangAAAI 2021 · 242 citations
  5. PopMAG: Pop Music Accompaniment GenerationYi Ren, Jinzheng He, Xu Tan, Tao Qin et al.ACM MM 2020 · 91 citations

Related papers

  1. Let the Model Learn to Feel: Mode-Guided Tonality Injection for Symbolic Music Emotion RecognitionHaiying Xia, Zhongyi Huang, Yumei Tan, Shuxiang SongAAAI 2026
  2. N-gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music UnderstandingJinhao Tian, Zuchao Li, Jiajia Li, Ping WangAAAI 2024
  3. MusicBERT: A Self-supervised Learning of Music RepresentationHongyuan Zhu, Ye Niu, Di Fu, Hao WangACM MM 2021 · 17 citations
  4. MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised TrainingYizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma et al.ICLR 2024 · 277 citations
  5. MuPT: A Generative Symbolic Music Pretrained TransformerXingwei Qu, Yuelin Bai, Yinghao Ma, Ziya Zhou et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines