Lune

ACL Findings2023

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron Halfaker, Liqun Shao, Daniel McDuff, Ahmed Hassan Awadallah, Dragomir Radev, Jianfeng Gao

摘要

Natural language contains rich logical structures and logical information, and correctly detecting and accurately understanding these logical structures and information underlying natural language texts is very crucial for NLP models' performance on many important NLU and NLG tasks. Existing pre-trained language models based on the transformer architecture mostly adopt a classical design for constructing their input embeddings that ignores the logical structures underlying natural language texts, thus limiting their ability to better capture and encode key logical information in the input sequences. To overcome such limitations, in this paper we first propose a novel approach to construct logic-aware input embeddings for transformer language models through a combination of logic detection, logic mapping and hierarchical logical projections, and then develop a corresponding new modeling paradigm that can upgrade existing transformer language models into logical transformers to boost their performance on different NLU and NLG tasks. Our empirical experiments on four important and challenging NLU and NLG tasks demonstrate that our proposed logical transformer language models can achieve superior performance over their baseline transformer models through a deeper understanding of the logical structures of texts.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper9

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel 等ICLR 2020 · 7,418 次引用
  2. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 次引用
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad 等ACL 2020 · 1,224 次引用
  4. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsKevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. ManningICLR 2020 · 541 次引用
  5. ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningWeihao Yu, Zihang Jiang, Yanfei Dong, Jiashi FengICLR 2020 · 325 次引用

相关 Paper

  1. PLOG: Table-to-Logic Pretraining for Logical Table-to-Text GenerationAo Liu, Haoyu Dong, Naoaki Okazaki, Shi Han 等EMNLP 2022 · 15 次引用
  2. Learning Language Representations with Logical Inductive BiasJianshu ChenICLR 2023
  3. Do LLMs Really Struggle at NL-FOL Translation? Revealing Their Strengths via a Novel Benchmarking StrategyAndrea Brunello, Luca Geatti, Michele Mignani, Angelo Montanari 等AAAI 2026
  4. Probing Linguistic Information for Logical Inference in Pre-trained Language ModelsZeming Chen, Qiyue GaoAAAI 2022 · 11 次引用
  5. Not all quantifiers are equal: Probing Transformer-based language models' understanding of generalised quantifiersTharindu Madusanka, Iqra Zahid, Hao Li, Ian Pratt-Hartmann 等EMNLP 2023 · 2 次引用
黄昏的海面,两侧是细线勾勒的悬崖