Lune

ACL Findings2024

X-ACE: Explainable and Multi-factor Audio Captioning Evaluation

Qian Wang, Jia-Chen Gu, Zhen-Hua Ling

Abstract

Automated audio captioning (AAC) aims to generate descriptions based on audio input, attracting exploration of emerging audio language models (ALMs). However, current evaluation metrics only provide a single score to assess the overall quality of captions without characterizing the nuanced difference by systematically going through an evaluation checklist. To this end, we propose the explainable and multi-factor audio captioning evaluation (X-ACE) paradigm. X-ACE identifies four main factors that constitute the majority of audio features, specifically sound event, source, attribute and relation. To assess a given caption from an ALM, it is firstly transformed into an audio graph, where each node denotes an entity in the caption and corresponds to a factor. On the one hand, graph matching is conducted from part to whole for a holistic assessment. On the other hand, the nodes contained within each factor are aggregated to measure the factor-level performance. The pros and cons of an ALM can be explicitly and clearly demonstrated through X-ACE, pointing out the direction for further improvements. Experiments show that X-ACE 1 exhibits better correlation with human perception and can detect mismatches sensitively.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Evaluating Object Hallucination in Large Vision-Language Models cites Visual Instruction Tuning
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. Pengi: An Audio Language Model for Audio TasksSoham Deshmukh, Benjamin Elizalde, Rita Singh, Huaming WangNeurIPS 2023 · 352 citations
  5. Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations

Related papers

  1. Learning to See through Sound: From VggCaps to Multi2Cap for Richer Automated Audio CaptioningSangyeon Cho, Mingi Kim, Jinkwon Hwang, Jaehoon Go et al.EMNLP 2025
  2. ADIFF: Explaining audio difference using natural languageSoham Deshmukh, Shuo Han, Rita Singh, Bhiksha RajICLR 2025
  3. How Users Experience Closed Captions on Live Television: Quality Metrics Remain a ChallengeMariana Arroyo Chavez, Molly Feanny, Matthew Seita, Bernard Thompson et al.CHI 2024 · 15 citations
  4. AudioGenX: Explainability on Text-to-Audio Generative ModelsHyunju Kang, Geonhee Han, Yoonjae Jeong, Hogun ParkAAAI 2025
  5. CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image GenerationHyunwoo Oh, SeungJu Cha, Kwanyoung Lee, Si-Woo Kim et al.ACM MM 2025
Dusk over the sea between two cliffs drawn in fine vertical lines