ACL Findings2025
Confidence Improves Self-Consistency in LLMs
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, Gal Yona
摘要
Self-consistency decoding enhances LLMs' performance on reasoning tasks by sampling diverse reasoning paths and selecting the most frequent answer. However, it is computationally expensive, as sampling many of these (lengthy) paths is required to increase the chances that the correct answer emerges as the most frequent one. To address this, we introduce Confidence-Informed Self-Consistency (CISC). CISC performs a weighted majority vote based on confidence scores obtained directly from the model. By prioritizing highconfidence paths, it can identify the correct answer with a significantly smaller sample size. When tested on nine models and four datasets, CISC outperforms self-consistency in nearly all configurations, reducing the required number of reasoning paths by over 40% on average. In addition, we introduce the notion of withinquestion confidence evaluation, after showing that standard evaluation methods are poor predictors of success in distinguishing correct and incorrect answers to the same question. In fact, the most calibrated confidence method proved to be the least effective for CISC. Lastly, beyond these practical implications, our results and analyses show that LLMs can effectively judge the correctness of their own outputs, contributing to the ongoing debate on this topic.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 34 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models 引用了 Self-Refine: Iterative Refinement with Self-Feedback
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Self-Refine: Iterative Refinement with Self-Feedback 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning 引用了 Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Let's Verify Step by Step 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
引用它的顶会 Paper34
- Beyond Binary Rewards: Training LMs to Reason About Their UncertaintyMehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld 等ICLR 2026 · 116 次引用
- LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet ArenaQingchuan Yang, Simon Mahns, Sida Li, Anri Gu 等ICLR 2026 · 36 次引用
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM ReasoningZhi Zhou, Tan Yuhao, Zenan Li, Yuan Yao 等NeurIPS 2025 · 17 次引用
- Calibrating Verbalized Confidence with Self-Generated DistractorsVictor Wang, Elias Stengel-EskinICLR 2026 · 15 次引用
- How Far Are We from Optimal Reasoning Efficiency?Jiaxuan Gao, Shu Yan, Qixin Tan, Lu Yang 等NeurIPS 2025 · 12 次引用
它引用的 Paper25
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma 等NeurIPS 2022 · 22,562 次引用
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou 等ICLR 2021 · 7,905 次引用
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran 等NeurIPS 2023 · 5,068 次引用
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan 等NeurIPS 2023 · 4,972 次引用
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards 等ICLR 2024 · 3,045 次引用
相关 Paper
- Mirror-Consistency: Harnessing Inconsistency in Majority VotingSiyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng 等EMNLP Findings 2024 · 1 次引用
- VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer SelectionJames Petullo, Sonny George, Dylan Cashman, Nianwen XueACL Findings 2026 · 1 次引用
- Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer ReasoningJungsuk Oh, Jay-Yoon LeeAAAI 2026 · 2 次引用
- Self-Para-Consistency: Improving Reasoning Tasks at Low Cost for Large Language ModelsWenqing Chen, Weicheng Wang, Zhixuan Chu, Kui Ren 等ACL Findings 2024 · 5 次引用
- Slim-SC: Thought Pruning for Efficient Scaling with Self-ConsistencyColin Hong, Xu Guo, Anand Chaanan Singh, Esha Choukse 等EMNLP 2025
