EMNLP Findings2022
Natural Language Deduction through Search over Statement Compositions
Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, Greg Durrett
Abstract
In settings from fact-checking to question answering, we frequently want to know whether a collection of evidence (premises) entails a hypothesis. Existing methods primarily focus on the end-to-end discriminative version of this task, but less work has treated the generative version in which a model searches over the space of statements entailed by the premises to constructively derive the hypothesis. We propose a system for doing this kind of deductive reasoning in natural language by decomposing the task into separate steps coordinated by a search procedure, producing a tree of intermediate conclusions that faithfully reflects the system's reasoning process.
Our experiments on the EntailmentBank dataset (Dalvi et al., 2021) demonstrate that the proposed system can successfully prove true statements while rejecting false ones. Moreover, it produces natural language explanations with a 17% absolute higher step validity than those produced by an end-to-end T5 model.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
19 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance cites Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Deductive Verification of Chain-of-Thought Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning cites Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
- MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering cites The Curious Case of Neural Text Degeneration
Cited by top-tier papers19
- Deductive Verification of Chain-of-Thought ReasoningZhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang et al.NeurIPS 2023 · 234 citations
- MuSR: Testing the Limits of Chain-of-thought with Multistep Soft ReasoningZayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri et al.ICLR 2024 · 172 citations
- Learning Deductive Reasoning from Synthetic Corpus based on Formal LogicTerufumi Morishita, Gaku Morio, Atsuki Yamaguchi, Yasuhiro SogawaICML 2023 · 45 citations
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-ThoughtAbulhair Saparov, He HeICLR 2023 · 38 citations
- Rainier: Reinforced Knowledge Introspector for Commonsense Question AnsweringJiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He et al.EMNLP 2022 · 31 citations
Builds on13
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team PerformanceGagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok et al.CHI 2021 · 713 citations
- Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Peter Hase, Mohit BansalACL 2020 · 216 citations
Related papers
- Generating Natural Language Proofs with Verifier-Guided SearchKaiyu Yang, Jia Deng, Danqi ChenEMNLP 2022 · 22 citations
- Explaining Answers with Entailment TreesBhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie et al.EMNLP 2021 · 6 citations
- Natural Language Deduction with Incomplete InformationZayne Sprague, Kaj Bostrom, Swarat Chaudhuri, Greg DurrettEMNLP 2022 · 8 citations
- FRVA: Fact-Retrieval and Verification Augmented Entailment Tree Generation for Explainable Question AnsweringYue Fan, Hu Zhang, Ru Li, Yujie Wang et al.ACL Findings 2024
- RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment TreesTengxiao Liu, Qipeng Guo, Xiangkun Hu, Yue Zhang et al.EMNLP 2022 · 8 citations
