Lune

ACL Findings2024

The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models

Jiajia Li, Lu Yang, Mingni Tang, Cong Chen, Zuchao Li, Ping Wang, Hai Zhao

Abstract

Benchmark plays a pivotal role in assessing the advancements of large language models (LLMs). While numerous benchmarks have been proposed to evaluate LLMs' capabilities, there is a notable absence of a dedicated benchmark for assessing their musical abilities. To address this gap, we present ZIQI-Eval, a comprehensive and large-scale music benchmark specifically designed to evaluate the musicrelated capabilities of LLMs. ZIQI-Eval encompasses a wide range of questions, covering 10 major categories and 56 subcategories, resulting in over 14,000 meticulously curated data entries. By leveraging ZIQI-Eval, we conduct a comprehensive evaluation over 16 LLMs to evaluate and analyze LLMs' performance in the domain of music. Results indicate that all LLMs perform poorly on the ZIQI-Eval benchmark, suggesting significant room for improvement in their musical capabilities. With ZIQI-Eval, we aim to provide a standardized and robust evaluation framework that facilitates a comprehensive assessment of LLMs' musicrelated abilities. The dataset is available at GitHub 1 and HuggingFace 2 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores cites Measuring Massive Multitask Language Understanding
  • SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away cites Simple and Controllable Music Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical ScoresCongren Dai, Yue Yang, Krinos Li, Huichi Zhou et al.ACL 2026 · 4 citations
  2. SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years AwayJiliang Hu, Jiajia Li, Ziyi Pan, Chong Chen et al.AAAI 2025 · 3 citations
  3. BoYaEval: Evaluating Multimodal Large Language Models on Understanding Ancient Chinese Musical ScoresJiajia Li, Weizhi Xue, Yao Yao, Qiwei Li et al.ACL 2026

Builds on15

  1. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. DiffWave: A Versatile Diffusion Model for Audio SynthesisZhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao et al.ICLR 2021 · 1,902 citations
  4. Simple and Controllable Music GenerationJade Copet, Felix Kreuk, Itai Gat, Tal Remez et al.NeurIPS 2023 · 843 citations
  5. Semantics-Aware BERT for Language UnderstandingZhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li et al.AAAI 2020 · 396 citations

Related papers

  1. Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge EvaluationZhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang et al.AAAI 2024 · 82 citations
  2. Large Language Models for Predictive Analysis: How Far Are They?Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang ShiACL Findings 2025 · 2 citations
  3. AgriEval: A Comprehensive Chinese Agricultural Benchmark for Large Language ModelsLian Yan, Haotian Wang, Chen Tang, Haifeng Liu et al.AAAI 2026 · 3 citations
  4. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code RepositoriesJia Li, Ge Li, Yunfei Zhao, Yongmin Li et al.ACL Findings 2024 · 44 citations
  5. DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code GenerationQiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin et al.AAAI 2025 · 25 citations
Dusk over the sea between two cliffs drawn in fine vertical lines