ACL Findings2021
Evaluating the Efficacy of Summarization Evaluation across Languages
Fajri Koto, Jey Han Lau, Timothy Baldwin
Abstract
While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization cites Unsupervised Cross-lingual Representation Learning at Scale
- Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites Unsupervised Cross-lingual Representation Learning at Scale
- Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
- Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation cites Unsupervised Cross-lingual Representation Learning at Scale
Cited by top-tier papers5
- SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive SummarizationMathieu Ravaut, Shafiq R. Joty, Nancy F. ChenACL 2022 · 116 citations
- Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual GenerationTu Vu, Aditya Barua, Brian Lester, Daniel Cer et al.EMNLP 2022 · 18 citations
- Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive SummarizationItai Mondshine, Tzuf Paz-Argaman, Reut TsarfatyACL 2025 · 6 citations
- CaLMQA: Exploring culturally specific long-form question answering across 23 languagesShane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee et al.ACL 2025
- Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian LanguagesAmir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri KotoACL 2026
Builds on4
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation MetricsNitika Mathur, Timothy Baldwin, Trevor CohnACL 2020 · 14 citations
- Unsupervised Opinion Summarization with Noising and DenoisingReinald Kim Amplayo, Mirella LapataACL 2020 · 8 citations
- Re-evaluating Evaluation in Text SummarizationManik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu et al.EMNLP 2020 · 3 citations
Related papers
- Rethinking Efficient Multilingual Text Summarization Meta-EvaluationRilyn Han, Jiawen Chen, Yixin Liu, Arman CohanACL Findings 2024 · 3 citations
- Multilingual Definition ModelingEdison Marrese-Taylor, Erica K. Shimomoto, Alfredo Solano, Enrique ReidACL Findings 2025
- MultiSumm: Towards a Unified Model for Multi-Lingual Abstractive SummarizationYue Cao, Xiaojun Wan, Jin-ge Yao, Dian YuAAAI 2020 · 28 citations
- SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization EvaluationElizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez et al.EMNLP 2023 · 6 citations
- Cross-Lingual Transfer in Zero-Shot Cross-Language Entity LinkingElliot Schumacher, James Mayfield, Mark DredzeACL Findings 2021 · 4 citations
