Lune

EMNLP Findings2022

PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion

Jianhao Shen, Chenguang Wang, Ye Yuan, Jiawei Han, Heng Ji, Koushik Sen, Ming Zhang, Dawn Song

Abstract

This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a "fill-in-the-blank" task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-theart approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
  • DeepStruct: Pretraining of Language Models for Structure Prediction cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • DeepStruct: Pretraining of Language Models for Structure Prediction cites Language Models are Few-Shot Learners
  • Measuring Vision-Language STEM Skills of Neural Models cites Language Models are Few-Shot Learners
  • Measuring Vision-Language STEM Skills of Neural Models cites DeepStruct: Pretraining of Language Models for Structure Prediction
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Measuring Vision-Language STEM Skills of Neural ModelsJianhao Shen, Ye Yuan, Srbuhi Mirzoyan, Ming Zhang et al.ICLR 2024 · 14 citations

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  4. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link PredictionZhanqiu Zhang, Jianyu Cai, Yongdong Zhang, Jie WangAAAI 2020 · 481 citations
  5. DeepStruct: Pretraining of Language Models for Structure PredictionChenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong et al.ACL Findings 2022 · 78 citations

Related papers

  1. Pretrain-KGE: Learning Knowledge Representation from Pretrained Language ModelsZhiyuan Zhang, Xiaoqian Liu, Yi Zhang, Qi Su et al.EMNLP Findings 2020 · 52 citations
  2. Subgraph-Aware Training of Language Models for Knowledge Graph Completion Using Structure-Aware Contrastive LearningYoumin Ko, Hyemin Yang, Taeuk Kim, Hyunjoon KimWWW 2025 · 10 citations
  3. Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph CompletionCunhang Fan, Yujie Chen, Jun Xue, Yonghui Kong et al.AAAI 2024 · 6 citations
  4. Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
  5. Commonsense Knowledge Base Completion with Structural and Semantic ContextChaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, Yejin ChoiAAAI 2020 · 155 citations
Dusk over the sea between two cliffs drawn in fine vertical lines