ACL Findings2024
URG: A Unified Ranking and Generation Method for Ensembling Language Models
Bo Lv, Chen Tang, Yanan Zhang, Xin Liu, Ping Luo, Yue Yu
Abstract
Prior research endeavors of the ensemble Large Language Models (LLMs) achieved great success by employing an individual language model (LM) rank before the text generation. However, the use of an individual LM ranker faces two primary challenges: (1) The timeintensive nature of the ranking process, stemming from the comparisons between models;
(2) The issue of error propagation arising from the separate ranking and generation models within the framework. In order to overcome these challenges, we propose a novel ensemble framework, namely Unified Ranking and Generation (URG). URG represents an end-toend framework that jointly ranks the outputs of LLMs and generates fine-grained fusion results, via utilizing a dedicated cross-attention-based module and noise mitigation training against irrelevant information stemming from bad ranking results. Through extensive experimentation and evaluation, we demonstrate the efficiency and effectiveness of our framework in both the ranking and generation tasks. With the close coordination of the ranking and generation modules, our end-to-end framework achieves the state-of-the-art (SOTA) performance on these tasks, and exhibits substantial enhancements to any of the ensembled models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback cites BARTScore: Evaluating Generated Text as Text Generation
- SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback cites BERTScore: Evaluating Text Generation with BERT
- BARTScore: Evaluating Generated Text as Text Generation cites BERTScore: Evaluating Text Generation with BERT
- SARA: Salience-Aware Reinforced Adaptive Decoding for Large Language Models in Abstractive Summarization cites BERTScore: Evaluating Text Generation with BERT
- Masked Language Model Scoring cites BERTScore: Evaluating Text Generation with BERT
- SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization cites BARTScore: Evaluating Generated Text as Text Generation
- SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization cites BERTScore: Evaluating Text Generation with BERT
Cited by top-tier papers6
- Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRYulong Zhang, Tianyi Liang, Erfei Cui, Guoqing Wang et al.CVPR 2026 · 14 citations
- SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online FeedbackBo Lv, Nayu Liu, Chen Tang, Xin Liu et al.NeurIPS 2025 · 7 citations
- SARA: Salience-Aware Reinforced Adaptive Decoding for Large Language Models in Abstractive SummarizationNayu Liu, Junnan Zhu, Yiming Ma, Zhicong Lu et al.ACL 2025 · 7 citations
- Lookahead Routing for Large Language ModelsCanbin Huang, Tianyuan Shi, Yuhua Zhu, Ruijun Chen et al.NeurIPS 2025 · 5 citations
- Best-of-Infinity: Asymptotic Performance of Test-Time LLM EnsemblingJunpei Komiyama, Daisuke Oba, Masafumi OyamadaICLR 2026 · 2 citations
Builds on17
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
- BARTScore: Evaluating Generated Text as Text GenerationWeizhe Yuan, Graham Neubig, Pengfei LiuNeurIPS 2021 · 1,143 citations
- Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, Katrin KirchhoffACL 2020 · 167 citations
- SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive SummarizationMathieu Ravaut, Shafiq R. Joty, Nancy F. ChenACL 2022 · 116 citations
Related papers
- LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative FusionDongfu Jiang, Xiang Ren, Bill Yuchen LinACL 2023 · 95 citations
- OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMsJintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen et al.EMNLP Findings 2024 · 3 citations
- MOSAIC: Multiple Observers Spotting AI ContentMatthieu Dubois, François Yvon, Pablo PiantanidaACL Findings 2025
- EAGER-LLM: Enhancing Large Language Models as Recommenders through Exogenous Behavior-Semantic IntegrationMinjie Hong, Yan Xia, Zehan Wang, Jieming Zhu et al.WWW 2025 · 30 citations
- Rethinking LLM-Based Recommendations: A Personalized Query-Driven Parallel IntegrationDonghee Han, Hwanjun Song, Mun Yong YiEMNLP Findings 2025 · 1 citation
