Deep Bidirectional Language-Knowledge Graph Pretraining

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised method to pretrain a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves strong performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon . Glamping GNN 1st Layer Glamping GNN Final Layer camp trip beach school beach RoBERTa: A. camp (✗) GreaseLM: C. camp (✗) Glamping: B. school (✓) school trip Glamping GNN 1st Layer Glamping GNN Final Layer garage trip beach school beach garage movie Glamping GNN 1st Layer Glamping GNN Final Layer entertain ment station movie Glamping GNN 1st Layer Glamping GNN Final Layer live movie movie entertain ment station live Model Prediction RoBERTa: B. school (✗) GreaseLM: B. school (✗) Glamping: C. beach (✓) Model Prediction B. movie theater (✓) GreaseLM: B. movie theater (✓) Glamping: B. movie theater (✓) Model Prediction B. movie theater (✗) GreaseLM: B. movie theater (✗)