Lune

ICLR2022Top-tier venue

GreaseLM: Graph REASoning Enhanced Language Models

Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec

Abstract

Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GREASELM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GREASELM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8× larger. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

31 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs cites Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings
  • Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning cites Can Language Models Solve Graph Problems in Natural Language?
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers31

  1. Reasoning on Graphs: Faithful and Interpretable Large Language Model ReasoningLinhao Luo, Yuan-Fang Li, Gholamreza Haffari, Shirui PanICLR 2024 · 499 citations
  2. Can Language Models Solve Graph Problems in Natural Language?Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan et al.NeurIPS 2023 · 420 citations
  3. Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation LearningXiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold et al.ICLR 2024 · 151 citations
  4. Semantic Probabilistic Layers for Neuro-Symbolic LearningKareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck et al.NeurIPS 2022 · 133 citations
  5. Editing Large Language Models: Problems, Methods, and OpportunitiesYunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng et al.EMNLP 2023 · 83 citations

Builds on13

  1. UnifiedQA: Crossing Format Boundaries With a Single QA SystemDaniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal et al.EMNLP Findings 2020 · 514 citations
  2. (Comet-) Atomic 2020: On Symbolic and Neural Commonsense Knowledge GraphsJena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da et al.AAAI 2021 · 458 citations
  3. Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box EmbeddingsHongyu Ren, Weihua Hu, Jure LeskovecICLR 2020 · 355 citations
  4. Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge GraphsHongyu Ren, Jure LeskovecNeurIPS 2020 · 267 citations
  5. Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question AnsweringShangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang et al.AAAI 2020 · 224 citations

Related papers

  1. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question AnsweringYanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang et al.EMNLP 2020 · 207 citations
  2. Graph Reasoning Transformers for Knowledge-Aware Question AnsweringRuilin Zhao, Feng Zhao, Liang Hu, Guandong XuAAAI 2024 · 10 citations
  3. Deep Bidirectional Language-Knowledge Graph PretrainingMichihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang et al.NeurIPS 2022 · 294 citations
  4. Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question AnsweringZiniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang et al.EMNLP 2022 · 32 citations
  5. Relation-Aware Language-Graph Transformer for Question AnsweringJinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeon-Jin Park et al.AAAI 2023 · 16 citations
Dusk over the sea between two cliffs drawn in fine vertical lines