ACL Findings2023
IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning
Abstract
In the field of machine reading comprehension (MRC), existing systems have surpassed the average performance of human beings in many tasks like SQuAD. However, there is still a long way to go when it comes to logical reasoning. Although some methods for it have been put forward, they either are designed in a quite complicated way or rely too much on external structures. In this paper, we proposed IDOL (InDicator-Oriented Logic Pre-training), an easy-to-understand but highly effective further pre-training task which logically strengthens the pre-trained models with the help of 6 types of logical indicators and a logically rich dataset LoGic Pre-training (LGP). IDOL achieves state-of-the-art performance on Re-Clor and LogiQA, the two most representative benchmarks in logical reasoning MRC, and is proven to be capable of generalizing to different pre-trained models and other types of MRC benchmarks like RACE and SQuAD 2.0 while keeping competitive general language understanding ability through testing on tasks in GLUE. Besides, at the beginning of the era of large language models, we take several of them like ChatGPT into comparison and find that IDOL still shows its advantage. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Deberta: decoding-Enhanced Bert with Disentangled Attention cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Revisiting Pre-Trained Models for Chinese Natural Language Processing cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- CLOMO: Counterfactual Logical Modification with Large Language Models cites ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Cited by top-tier papers1
Builds on12
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
- Revisiting Pre-Trained Models for Chinese Natural Language ProcessingYiming Cui, Wanxiang Che, Ting Liu, Bing Qin et al.EMNLP Findings 2020 · 866 citations
- ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningWeihao Yu, Zihang Jiang, Yanfei Dong, Jiashi FengICLR 2020 · 325 citations
Related papers
- APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical ReasoningSoumya Sanyal, Yichong Xu, Shuohang Wang, Ziyi Yang et al.ACL 2023 · 4 citations
- REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-trainingFangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji et al.ACL Findings 2021 · 8 citations
- Span Selection Pre-training for Question AnsweringMichael R. Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto et al.ACL 2020 · 9 citations
- Reasoning Like Program ExecutorsXinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi et al.EMNLP 2022 · 32 citations
- RLP: Reinforcement as a Pretraining ObjectiveAli Hatamizadeh, Syeda Nahida Akter, Shrimai Prabhumoye, Jan Kautz et al.ICLR 2026 · 26 citations
