Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimate

This paper systematically addresses the challenges of rule retrieval, a crucial yet underexplored area. Vanilla retrieval methods using sparse or dense retrievers to directly search for relevant rules to support downstream reasoning, often suffer from low accuracy. This is primarily due to a significant semantic gap between the instantiated facts in the queries and the abstract representations of the rules. Such misalignment results in suboptimal retrieval quality, which in turn negatively impacts reasoning performance. To overcome these challenges, we propose Self-Induction Augmented Retrieval (SIAR), a novel approach that utilizes Large Language Models (LLMs) to induce potential inferential rules that might offer benefits for reasoning by abstracting the underlying knowledge and logical structure in queries. These induced rules are then used for query augmentation to improve retrieval effectiveness. Additionally, we introduce Rule Relevance ReEstimate (R 3 ), a method that re-estimates the relevance of retrieved rules by assessing whether the abstract knowledge they contain can be instantiated to align with the facts in the queries and the helpfulness for reasoning. Extensive experiments across various settings demonstrate the effectiveness and versatility of our proposed methods. Limitations Currently, the rule libraries we discussed remain quite limited in size, as seen in datasets like Clutrr, ULogic, and CAIL2018, which contain only 1,048, 830, and 166 rules, respectively. Compared to the vast number of articles in traditional passage retrieval, the rule bases we retrieved are still relatively small. However, even with these small datasets, traditional retrieval methods have shown a decline in reasoning performance, underscoring the need for deeper exploration in rule retrieval. The smaller number of rules reduces the difficulty of the benchmark. In future work, we aim to introduce more irrelevant rules to explore additional challenges in rule retrieval.