EMNLP Findings2025
Self-Ensemble: Mitigating Confidence Distortion for Large Language Models
Zicheng Xu, Guanchu Wang, Guangyao Zheng, Yu-Neng Chuang, Alex Szalay, Xia Hu, Vladimir Braverman
Abstract
Although Large Language Models (LLMs) perform well in general fields, they exhibit a confidence distortion problem on multichoice question-answering (MCQA), particularly as the number of answer choices increases. Specifically, on MCQA with many choices, LLMs suffer from under-confidence in correct predictions and over-confidence in incorrect ones, leading to a substantially degraded performance. To solve this problem, we propose Self-Ensemble in this work. Our method splits the choices into several groups and ensembles LLM predictions across these groups to reach a final decision. The advantage of Self-Ensemble is its plug-and-play nature, where it can be integrated into existing LLM architecture based on a designed attention mask and positional encoding, without requiring labeled datasets for parameter tuning. Experimental results on three LLMs and datasets demonstrate that Self-Ensemble comprehensively addresses the confidence distortion problem of LLMs, outperforming standard inference as well as baseline methods. The source code is available at https://github. com/ZichengXu/Self-Ensemble.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudication cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Measuring Massive Multitask Language Understanding cites QASC: A Dataset for Question Answering via Sentence Composition
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
Cited by top-tier papers3
- HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL ConferencesYusuke Sakai, Hidetaka Kamigaito, Taro WatanabeACL 2026 · 18 citations
- DTS: Enhancing Large Reasoning Models via Decoding Tree SketchingZicheng Xu, Xiuyi Lou, Guanchu Wang, Yu-Neng Chuang et al.ICML 2026 · 5 citations
- Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary AdjudicationLeon Chlon, Ahmed Karim, MarcAntonio AwadaICML 2026 · 2 citations
Builds on7
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV CacheZirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong et al.ICML 2024 · 436 citations
- QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations
Related papers
- Multicalibration for Confidence Scoring in LLMsGianluca Detommaso, Martin Bertran Lopez, Riccardo Fogliato, Aaron RothICML 2024 · 39 citations
- Mirror-Consistency: Harnessing Inconsistency in Majority VotingSiyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng et al.EMNLP Findings 2024 · 1 citation
- Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language ModelsQi Cao, Takeshi Kojima, Andrew Gambardella, Helinyi Peng et al.ACL Findings 2026
- Learning to Route LLMs with Confidence TokensYu-Neng Chuang, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio et al.ICML 2025
- Test-Time Self-Adaptive Small Language Models for Question AnsweringSoyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang et al.EMNLP Findings 2023 · 5 citations
