Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models

As Large Language Models (LLMs) continue to exhibit remarkable performance in natural language understanding tasks, there is a crucial need to measure their ability for human-like multi-step logical reasoning. Existing logical reasoning evaluation benchmarks often focus primarily on simplistic single-step or multistep reasoning with a limited set of inference rules. Furthermore, the lack of datasets for evaluating non-monotonic reasoning represents a crucial gap since it aligns more closely with human-like reasoning. To address these limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset encompassing multistep logical reasoning with various inference rules and depths. Multi-LogiEval covers three logic types -propositional, first-order, and non-monotonic consisting of more than 30 inference rules and more than 60 of their combinations with various depths. Leveraging this dataset, we conduct evaluations on a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral, employing a zero-shot chain-of-thought. Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ∼ 68% at depth-1 to ∼ 43% at depth-5). We further conduct a thorough investigation of reasoning chains generated by LLMs which reveals several important findings. We believe that Multi-LogiEval facilitates future research for evaluating and enhancing the logical reasoning ability of LLMs 1 . Rule Combination Context and Question PL Rules: MT, DS Propositions: p: Capture shots in golden hours. q: Photo wins awards. r: Focus on rare wildlife. Context: In wildlife photography, Olivia was certain that if she captured shots in the golden hours, her photos would win awards. However, opportunities varied each day. It was evident that she either captured shots during the golden hours or she focused on rare wildlife, or both. Olivia's latest photos did not win any awards. Question: Is it true that she focused on rare wildlife?