Lune

ACL Findings2024

PEK: A Parameter-Efficient Framework for Knowledge-Grounded Dialogue Generation

Pan Yang, Dandan Song, Zhijing Wu, Yanru Zhou

Abstract

Pre-trained language models (PLMs) have shown great dialogue generation capability in different scenarios. However, the huge VRAM consumption when fine-tuning them is one of their drawbacks. PEFT approaches can significantly reduce the number of trainable parameters, which enables us to fine-tune larger dialogue generation models. However, the reduction in parameter quantity can diminish a PLM's expressive capacity and affect the PLM's learning from certain specific examples like knowledge-related conversations. Previous works have demonstrated that injecting external knowledge into dialogue generation models can improve the model's performance in knowledge-related conversations. Nonetheless, these methods are designed for the scenario where most parameters of the entire framework are trainable. In this paper, we propose PEK, a parameter-efficient framework for knowledgeenhanced dialogue generation. It enables PLMs to leverage external knowledge documents and knowledge graphs to enhance its generation capabilities with an acceptable number of trainable parameters. Evaluation results on the Wizard of Wikipedia and CMU_DoG datasets show that our approach outperforms baseline methods on multiple evaluation metrics, which validates the effectiveness of our approach.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Knowledge-Grounded Dialogue Generation with Pre-trained Language Models cites Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
  • Retrieval Augmentation Reduces Hallucination in Conversation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Retrieval Augmentation Reduces Hallucination in Conversation cites Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
  • Retrieval Augmentation Reduces Hallucination in Conversation cites Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. Retrieval Augmentation Reduces Hallucination in ConversationKurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela et al.EMNLP Findings 2021 · 579 citations
  4. Sequential Latent Knowledge Selection for Knowledge-Grounded DialogueByeongchang Kim, Jaewoo Ahn, Gunhee KimICLR 2020 · 179 citations
  5. Knowledge-Grounded Dialogue Generation with Pre-trained Language ModelsXueliang Zhao, Wei Wu, Can Xu, Chongyang Tao et al.EMNLP 2020 · 153 citations

Related papers

  1. Multi-Stage Prompting for Knowledgeable Dialogue GenerationZihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye et al.ACL Findings 2022 · 45 citations
  2. TegTok: Augmenting Text Generation via Task-specific and Open-world KnowledgeChao-Hong Tan, Jia-Chen Gu, Chongyang Tao, Zhen-Hua Ling et al.ACL Findings 2022 · 14 citations
  3. KG-Adapter: Enabling Knowledge Graph Integration in Large Language Models through Parameter-Efficient Fine-TuningShiyu Tian, Yangyang Luo, Tianze Xu, Caixia Yuan et al.ACL Findings 2024 · 24 citations
  4. Towards Informative Open-ended Text Generation with Dynamic Knowledge TriplesZixuan Ren, Yang Zhao, Chengqing ZongEMNLP Findings 2023 · 5 citations
  5. KPT: Keyword-Guided Pre-training for Grounded Dialog GenerationQi Zhu, Fei Mi, Zheng Zhang, Yasheng Wang et al.AAAI 2023 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines