ACL Findings2022
Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models
Simran Arora, Sen Wu, Enci Liu, Christopher Ré
Abstract
Popular language models (LMs) struggle to capture knowledge about rare tail facts and entities. Since widely used systems such as search and personal-assistants must support the long tail of entities that users ask about, there has been significant effort towards enhancing these base LMs with factual knowledge. We observe proposed methods typically start with a base LM and data that has been annotated with entity metadata, then change the model, by modifying the architecture or introducing auxiliary loss terms to better capture entity knowledge. In this work, we question this typical process and ask to what extent can we match the quality of model modifications, with a simple alternative: using a base LM and only changing the data. We propose metadata shaping, a method which inserts substrings corresponding to the readily available entity metadata, e.g. types and descriptions, into examples at train and inference time based on mutual information. Despite its simplicity, metadata shaping is quite effective. On standard evaluation benchmarks for knowledge-enhanced LMs, the method exceeds the base-LM baseline by an average of 4.3 F1 points and achieves state-of-the-art results. We further show the gains are on average 4.4x larger for the slice of examples containing tail vs. popular entities.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction cites LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
- KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding cites LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
- KILM: Knowledge Injection into Encoder-Decoder Language Models cites LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
- Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding cites K-BERT: Enabling Language Representation with Knowledge Graph
Cited by top-tier papers3
- Knowledge Prompting in Pre-trained Language Model for Natural Language UnderstandingJianing Wang, Wenkang Huang, Minghui Qiu, Qiuhui Shi et al.EMNLP 2022 · 26 citations
- KILM: Knowledge Injection into Encoder-Decoder Language ModelsYan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar et al.ACL 2023 · 16 citations
- KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document UnderstandingShangbin Feng, Zhaoxuan Tan, Wenqian Zhang, Zhenyu Lei et al.ACL 2023 · 5 citations
Builds on10
- Unsupervised Data Augmentation for Consistency TrainingQizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong et al.NeurIPS 2020 · 2,774 citations
- K-BERT: Enabling Language Representation with Knowledge GraphWeijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang et al.AAAI 2020 · 898 citations
- Long Range Arena : A Benchmark for Efficient TransformersYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen et al.ICLR 2021 · 881 citations
- LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentionIkuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda et al.EMNLP 2020 · 562 citations
- KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation ExtractionXiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng et al.WWW 2022 · 488 citations
Related papers
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric MemoriesAlex Mallen, Akari Asai, Victor Zhong, Rajarshi Das et al.ACL 2023 · 233 citations
- Open Knowledge Enrichment for Long-tail EntitiesErmei Cao, Difeng Wang, Jiacheng Huang, Wei HuWWW 2020 · 51 citations
- Entity-aware Transformers for Entity SearchEmma J. Gerritse, Faegheh Hasibi, Arjen P. de VriesSIGIR 2022 · 27 citations
- Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected KnowledgeYasumasa Onoe, Michael J. Q. Zhang, Shankar Padmanabhan, Greg Durrett et al.ACL 2023 · 24 citations
- Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language ModelWenhan Xiong, Jingfei Du, William Yang Wang, Veselin StoyanovICLR 2020 · 215 citations
