EMNLP Findings2022
PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion
Jianhao Shen, Chenguang Wang, Ye Yuan, Jiawei Han, Heng Ji, Koushik Sen, Ming Zhang, Dawn Song
Abstract
This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a "fill-in-the-blank" task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-theart approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
- DeepStruct: Pretraining of Language Models for Structure Prediction cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- DeepStruct: Pretraining of Language Models for Structure Prediction cites Language Models are Few-Shot Learners
- Measuring Vision-Language STEM Skills of Neural Models cites Language Models are Few-Shot Learners
- Measuring Vision-Language STEM Skills of Neural Models cites DeepStruct: Pretraining of Language Models for Structure Prediction
Cited by top-tier papers1
Builds on10
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
- Learning Hierarchy-Aware Knowledge Graph Embeddings for Link PredictionZhanqiu Zhang, Jianyu Cai, Yongdong Zhang, Jie WangAAAI 2020 · 481 citations
- DeepStruct: Pretraining of Language Models for Structure PredictionChenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong et al.ACL Findings 2022 · 78 citations
Related papers
- Pretrain-KGE: Learning Knowledge Representation from Pretrained Language ModelsZhiyuan Zhang, Xiaoqian Liu, Yi Zhang, Qi Su et al.EMNLP Findings 2020 · 52 citations
- Subgraph-Aware Training of Language Models for Knowledge Graph Completion Using Structure-Aware Contrastive LearningYoumin Ko, Hyemin Yang, Taeuk Kim, Hyunjoon KimWWW 2025 · 10 citations
- Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph CompletionCunhang Fan, Yujie Chen, Jun Xue, Yonghui Kong et al.AAAI 2024 · 6 citations
- Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
- Commonsense Knowledge Base Completion with Structural and Semantic ContextChaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, Yejin ChoiAAAI 2020 · 155 citations
