ACL Findings2024
Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios
Lei Lin, Jia-Yi Fu, Pengli Liu, Qingyang Li, Yan Gong, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai
Abstract
Although chain-of-thought (CoT) prompting combined with language models has achieved encouraging results on complex reasoning tasks, the naive greedy decoding used in CoT prompting usually causes the repetitiveness and local optimality. To address this shortcoming, ensemble-optimization tries to obtain multiple reasoning paths to get the final answer assembly. However, current ensemble-optimization methods either simply employ rule-based postprocessing such as self-consistency, or train an additional model based on several task-related human annotations to select the best one among multiple reasoning paths, yet fail to generalize to realistic settings where the type of input questions is unknown or the answer format of reasoning paths is unknown. To avoid their limitations, we propose Self-Agreement, a generalizable ensemble-optimization method applying in almost all scenarios where the type of input questions and the answer format of reasoning paths may be known or unknown. Self-agreement firstly samples from language model's decoder to generate a diverse set of reasoning paths, and subsequently prompts the language model one more time to determine the optimal answer by selecting the most agreed answer among the sampled reasoning paths. Selfagreement simultaneously achieves remarkable performance on six public reasoning benchmarks and superior generalization capabilities.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies cites Large Language Models are Zero-Shot Reasoners
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Solving Quantitative Reasoning Problems with Language Models cites Language Models are Few-Shot Learners
- Solving Quantitative Reasoning Problems with Language Models cites The Curious Case of Neural Text Degeneration
- Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs cites The Curious Case of Neural Text Degeneration
- Calibrate Before Use: Improving Few-shot Performance of Language Models cites Language Models are Few-Shot Learners
- Calibrate Before Use: Improving Few-shot Performance of Language Models cites The Curious Case of Neural Text Degeneration
Cited by top-tier papers3
- Multi-Agent Design: Optimizing Agents with Better Prompts and TopologiesHan Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi et al.ICLR 2026 · 127 citations
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMsXueyan Li, Guinan Su, Mrinmaya Sachan, Jonas GeipingICLR 2026 · 5 citations
- Beyond Surface Simplicity: Revealing Hidden Reasoning Attributes for Precise Commonsense DiagnosisHuijun Lian, Zekai Sun, Keqi Chen, Yingming Gao et al.ACL 2025
Builds on22
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
- Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
Related papers
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
- Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM ReasoningRaman Saparkhan, Majd Hawasly, Md. Rizwan Parvez, Mohammad RazaACL Findings 2026
- GCoT-Decoding: Unlocking Deep Reasoning Paths for Universal Question AnsweringGuanran Luo, Wentao Qiu, Zhongquan Jian, Meihong Wang et al.ACL Findings 2026
- Self-Para-Consistency: Improving Reasoning Tasks at Low Cost for Large Language ModelsWenqing Chen, Weicheng Wang, Zhixuan Chu, Kui Ren et al.ACL Findings 2024 · 5 citations
- Chain-of-Thought Reasoning Without PromptingXuezhi Wang, Denny ZhouNeurIPS 2024 · 305 citations
