Lune

EMNLP Findings2022

Self-Distillation with Meta Learning for Knowledge Graph Completion

Yunshui Li, Junhao Liu, Min Yang, Chengming Li

Abstract

In this paper, we propose a self-distillation framework with meta learning (MetaSD) for knowledge graph completion with dynamic pruning, which aims to learn compressed graph embeddings and tackle the long-tail samples. Specifically, we first propose a dynamic pruning technique to obtain a small pruned model from a large source model, where the pruning mask of the pruned model could be updated adaptively per epoch after the model weights are updated. The pruned model is supposed to be more sensitive to difficult-to-memorize samples (e.g., long-tail samples) than the source model. Then, we propose a one-step meta selfdistillation method for distilling comprehensive knowledge from the source model to the pruned model, where the two models co-evolve in a dynamic manner during training. In particular, we exploit the performance of the pruned model, which is trained alongside the source model in one iteration, to improve the source model's knowledge transfer ability for the next iteration via meta learning. Extensive experiments show that MetaSD achieves competitive performance compared to strong baselines, while being 10x smaller than baselines.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. DuetGraph: Coarse-to-Fine Knowledge Graph Reasoning with Dual-Pathway Global-Local FusionJin Li, Zezhong Ding, Xike XieNeurIPS 2025 · 5 citations
  2. Dark Side of Modalities: Reinforced Multimodal Distillation for Multimodal Knowledge Graph ReasoningYu Zhao, Ying Zhang, Xuhui Sui, Baohang Zhou et al.ACM MM 2025

Builds on3

  1. Adaptive Attentional Network for Few-Shot Knowledge Graph CompletionJiawei Sheng, Shu Guo, Zhenyu Chen, Juwei Yue et al.EMNLP 2020 · 117 citations
  2. Duality-Induced Regularizer for Tensor Factorization Based Knowledge Graph CompletionZhanqiu Zhang, Jianyu Cai, Jie WangNeurIPS 2020 · 64 citations
  3. Faster Meta Update Strategy for Noise-Robust Deep LearningYoujiang Xu, Linchao Zhu, Lu Jiang, Yi YangCVPR 2021

Related papers

  1. Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token PruningChang Liu, Chongyang Tao, Jianxin Liang, Jiazhan Feng et al.EMNLP Findings 2023
  2. Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph CompletionCunhang Fan, Yujie Chen, Jun Xue, Yonghui Kong et al.AAAI 2024 · 6 citations
  3. IterDE: An Iterative Knowledge Distillation Framework for Knowledge Graph EmbeddingsJiajun Liu, Peng Wang, Ziyu Shang, Chenxiao WuAAAI 2023 · 26 citations
  4. EPSD: Early Pruning with Self-Distillation for Efficient Model CompressionDong Chen, Ning Liu, Yichen Zhu, Zhengping Che et al.AAAI 2024 · 9 citations
  5. Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across DomainsHaojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang et al.ACL 2021
Dusk over the sea between two cliffs drawn in fine vertical lines