Lune

EMNLP Findings2023

EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics

Daniil Larionov, Jens Grünwald, Christoph Leiter, Steffen Eger

Abstract

Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics. Our approach involves replacing computation-intensive transformers with lighter alternatives and employing linear and quadratic approximations for alignment algorithms on top of LLM representations. We evaluate six (reference-free and referencebased) metrics across three MT datasets and examine 16 lightweight transformers. In addition, we look into the training efficiency of metrics like COMET by utilizing adapters. Our results indicate that (a) TinyBERT provides the optimal balance between quality and efficiency, (b) CPU speed-ups are more substantial than those on GPU; (c) WMD approximations yield no efficiency gains while reducing quality and (d) adapters enhance training efficiency (regarding backward pass speed and memory requirements) as well as, in some cases, metric quality. These findings can help to strike a balance between evaluation speed and quality, which is essential for effective NLG systems. Furthermore, our research contributes to the ongoing efforts to optimize NLG evaluation metrics with minimal impact on performance. To our knowledge, ours is the most comprehensive analysis of different aspects of efficiency for MT metrics conducted so far.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  • xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics cites BERTScore: Evaluating Text Generation with BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Interpretable Text Embeddings and Text Similarity Explanation: A SurveyJuri Opitz, Lucas Möller, Andrianos Michail, Sebastian Padó et al.EMNLP 2025 · 3 citations
  2. xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation MetricsDaniil Larionov, Mikhail Seleznyov, Vasiliy Viskov, Alexander Panchenko et al.EMNLP 2024 · 2 citations

Builds on24

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  3. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained TransformersWenhui Wang, Furu Wei, Li Dong, Hangbo Bao et al.NeurIPS 2020 · 2,727 citations
  4. MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
  5. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations

Related papers

  1. SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental ImpactsNghiem Thanh Pham, Tung Kieu, Duc-Manh Nguyen, Ha Xuan Son et al.EMNLP Findings 2025 · 7 citations
  2. Refining Salience-Aware Sparse Fine-Tuning Strategies for Language ModelsXinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng et al.ACL 2025 · 3 citations
  3. Effective Pretraining Objectives for Transformer-based AutoencodersLuca Di Liello, Matteo Gabburo, Alessandro MoschittiEMNLP Findings 2022 · 5 citations
  4. Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer ModelsDeepak Narayanan, Keshav Santhanam, Peter Henderson, Rishi Bommasani et al.NeurIPS 2023 · 14 citations
  5. Identifying Reliable Evaluation Metrics for Scientific Text RevisionLéane Jourdan, Nicolas Hernandez, Florian Boudin, Richard DufourACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines