Lune

ACL Findings2025

Confidence Improves Self-Consistency in LLMs

Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, Gal Yona

Abstract

Self-consistency decoding enhances LLMs' performance on reasoning tasks by sampling diverse reasoning paths and selecting the most frequent answer. However, it is computationally expensive, as sampling many of these (lengthy) paths is required to increase the chances that the correct answer emerges as the most frequent one. To address this, we introduce Confidence-Informed Self-Consistency (CISC). CISC performs a weighted majority vote based on confidence scores obtained directly from the model. By prioritizing highconfidence paths, it can identify the correct answer with a significantly smaller sample size. When tested on nine models and four datasets, CISC outperforms self-consistency in nearly all configurations, reducing the required number of reasoning paths by over 40% on average. In addition, we introduce the notion of withinquestion confidence evaluation, after showing that standard evaluation methods are poor predictors of success in distinguishing correct and incorrect answers to the same question. In fact, the most calibrated confidence method proved to be the least effective for CISC. Lastly, beyond these practical implications, our results and analyses show that LLMs can effectively judge the correctness of their own outputs, contributing to the ongoing debate on this topic.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

34 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Refine: Iterative Refinement with Self-Feedback
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  • A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Let's Verify Step by Step cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers34

  1. Beyond Binary Rewards: Training LMs to Reason About Their UncertaintyMehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld et al.ICLR 2026 · 116 citations
  2. LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet ArenaQingchuan Yang, Simon Mahns, Sida Li, Anri Gu et al.ICLR 2026 · 36 citations
  3. A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM ReasoningZhi Zhou, Tan Yuhao, Zenan Li, Yuan Yao et al.NeurIPS 2025 · 17 citations
  4. Calibrating Verbalized Confidence with Self-Generated DistractorsVictor Wang, Elias Stengel-EskinICLR 2026 · 15 citations
  5. How Far Are We from Optimal Reasoning Efficiency?Jiaxuan Gao, Shu Yan, Qixin Tan, Lu Yang et al.NeurIPS 2025 · 12 citations

Builds on25

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
  4. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  5. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations

Related papers

  1. Mirror-Consistency: Harnessing Inconsistency in Majority VotingSiyuan Huang, Zhiyuan Ma, Jintao Du, Changhua Meng et al.EMNLP Findings 2024 · 1 citation
  2. VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer SelectionJames Petullo, Sonny George, Dylan Cashman, Nianwen XueACL Findings 2026 · 1 citation
  3. Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer ReasoningJungsuk Oh, Jay-Yoon LeeAAAI 2026 · 2 citations
  4. Self-Para-Consistency: Improving Reasoning Tasks at Low Cost for Large Language ModelsWenqing Chen, Weicheng Wang, Zhixuan Chu, Kui Ren et al.ACL Findings 2024 · 5 citations
  5. Slim-SC: Thought Pruning for Efficient Scaling with Self-ConsistencyColin Hong, Xu Guo, Anand Chaanan Singh, Esha Choukse et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines