Lune

ACL Findings2024

Rethinking Efficient Multilingual Text Summarization Meta-Evaluation

Rilyn Han, Jiawen Chen, Yixin Liu, Arman Cohan

Abstract

Evaluating multilingual summarization evaluation metrics, i.e., meta-evaluation, is challenging because of the difficulty of human annotation collection. Therefore, we investigate an efficient multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual meta-evaluation dataset into multilingual versions. To this end, we introduce a statistical test to verify the transformed dataset quality by checking the metaevaluation result consistency on the original dataset and back-translated dataset. With this quality verification method, we transform an existing English summarization meta-evaluation dataset, RoSE, into 30 languages, and conduct a multilingual meta-evaluation of several representative automatic evaluation metrics. In our meta-evaluation, we find that metric performance varies in different languages and neural metrics generally outperform classical textmatching-based metrics in non-English languages. Moreover, we identify a two-stage evaluation method with superior performance, which first translates multilingual texts into English and then performs evaluation. We make the transformed datasets publicly available to facilitate future research.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization cites XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
  • Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive SummarizationItai Mondshine, Tzuf Paz-Argaman, Reut TsarfatyACL 2025 · 6 citations
  2. Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian LanguagesAmir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri KotoACL 2026

Builds on11

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 LanguagesTahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin Mubasshir et al.ACL Findings 2021 · 228 citations
  3. WikiLingua: A New Benchmark Dataset for Multilingual Abstractive SummarizationFaisal Ladhak, Esin Durmus, Claire Cardie, Kathleen R. McKeownEMNLP Findings 2020 · 151 citations
  4. Multilingual Translation from Denoising Pre-TrainingYuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen et al.ACL Findings 2021 · 99 citations
  5. What Have We Achieved on Text Summarization?Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao et al.EMNLP 2020 · 72 citations

Related papers

  1. Evaluating the Efficacy of Summarization Evaluation across LanguagesFajri Koto, Jey Han Lau, Timothy BaldwinACL Findings 2021 · 6 citations
  2. SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization EvaluationElizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez et al.EMNLP 2023 · 6 citations
  3. CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1, 500+ Language PairsAbhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li et al.ACL 2023 · 23 citations
  4. MultiSumm: Towards a Unified Model for Multi-Lingual Abstractive SummarizationYue Cao, Xiaojun Wan, Jin-ge Yao, Dian YuAAAI 2020 · 28 citations
  5. Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human EvaluationYixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao et al.ACL 2023 · 50 citations
Dusk over the sea between two cliffs drawn in fine vertical lines