Lune

ACL Findings2021

Evaluating the Efficacy of Summarization Evaluation across Languages

Fajri Koto, Jey Han Lau, Timothy Baldwin

Abstract

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization cites Unsupervised Cross-lingual Representation Learning at Scale
  • Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites Unsupervised Cross-lingual Representation Learning at Scale
  • Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
  • Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation cites Unsupervised Cross-lingual Representation Learning at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive SummarizationMathieu Ravaut, Shafiq R. Joty, Nancy F. ChenACL 2022 · 116 citations
  2. Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual GenerationTu Vu, Aditya Barua, Brian Lester, Daniel Cer et al.EMNLP 2022 · 18 citations
  3. Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive SummarizationItai Mondshine, Tzuf Paz-Argaman, Reut TsarfatyACL 2025 · 6 citations
  4. CaLMQA: Exploring culturally specific long-form question answering across 23 languagesShane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee et al.ACL 2025
  5. Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian LanguagesAmir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri KotoACL 2026

Builds on4

  1. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  2. Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation MetricsNitika Mathur, Timothy Baldwin, Trevor CohnACL 2020 · 14 citations
  3. Unsupervised Opinion Summarization with Noising and DenoisingReinald Kim Amplayo, Mirella LapataACL 2020 · 8 citations
  4. Re-evaluating Evaluation in Text SummarizationManik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu et al.EMNLP 2020 · 3 citations

Related papers

  1. Rethinking Efficient Multilingual Text Summarization Meta-EvaluationRilyn Han, Jiawen Chen, Yixin Liu, Arman CohanACL Findings 2024 · 3 citations
  2. Multilingual Definition ModelingEdison Marrese-Taylor, Erica K. Shimomoto, Alfredo Solano, Enrique ReidACL Findings 2025
  3. MultiSumm: Towards a Unified Model for Multi-Lingual Abstractive SummarizationYue Cao, Xiaojun Wan, Jin-ge Yao, Dian YuAAAI 2020 · 28 citations
  4. SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization EvaluationElizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez et al.EMNLP 2023 · 6 citations
  5. Cross-Lingual Transfer in Zero-Shot Cross-Language Entity LinkingElliot Schumacher, James Mayfield, Mark DredzeACL Findings 2021 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines