Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning

Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like supposition following or chain construction. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field. Problem: Evaluate whether the given conclusion logically follows from the premises below: 1. There is a blue marble if and only if there is a white marble. 2. Either there is a white marble or else there is a red marble, but not both. 3. There is a red marble if and only if there is a pink marble. Conclusion: If there is a blue marble then there is a pink marble. Problem 1: Statements: 1. Either there is a white marble in the box or else there is a black marble in the box, but not both. 2. Either there is a black marble in the box or else there is a pink marble in the box, but not both. 3. There is a pink marble in the box if and only if there is a gray marble in the box. Conclusion: If there is a white marble in the box then there is a gray marble in the box. Response: Sure, I'd be happy to help! Here's my reasoning process for this problem: 1. First, I'll start by analyzing the given statements: From Statement 1, we know that the box contains either a white marble or a black marble, but not both. From Statement 2, we know that the box contains either a black marble or a pink marble, but not both. From Statement 3, we know that if there is a pink marble in the box, then there must be a gray marble in the box. 2. Now, let's consider the conclusion: If there is a white marble in the box then there is a gray marble in the box. Can we logically deduce this conclusion from the given statements? Well, if there is a white marble in the box, then by Statement 1, there cannot be a black marble in the box. If there is no black marble in the box, then by Statement 2, there must be a pink marble in the box. But if there is a pink marble in the box, then by Statement 3, there must be a gray marble in the box. So, yes, we can logically deduce the conclusion that if there is a white marble in the box, then there is a gray marble in the box. Conclusion: True