EMNLP Findings2021
K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
Song Xu, Haoran Li, Peng Yuan, Yujia Wang, Youzheng Wu, Xiaodong He, Ying Liu, Bowen Zhou
Abstract
Existing pre-trained language models (PLMs) have demonstrated the effectiveness of selfsupervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledgeinjected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge. Specifically, we propose five knowledgeaware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domainspecific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities. K-PLUG achieves new state-of-the-art results on a suite of domain-specific NLP tasks, including product knowledge base completion, abstractive product summarization, and multiturn dialogue, significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks. Our code is available at https:// github.com/xu-song/k-plug.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- Plug-and-Play Knowledge Injection for Pre-trained Language Models cites K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Cited by top-tier papers3
- EcomGPT: Instruction-Tuning Large Language Models with Chain-of-Task Tasks for E-commerceYangning Li, Shirong Ma, Xiaobin Wang, Shen Huang et al.AAAI 2024 · 85 citations
- Plug-and-Play Knowledge Injection for Pre-trained Language ModelsZhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang et al.ACL 2023 · 10 citations
- JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and SummarizationNan Zhao, Haoran Li, Youzheng Wu, Xiaodong HeEMNLP 2022 · 6 citations
Builds on11
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- K-Adapter: Infusing Knowledge into Pre-Trained Models with AdaptersRuize Wang, Duyu Tang, Nan Duan, Zhongyu Wei et al.ACL Findings 2021 · 473 citations
- ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingWeizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu et al.EMNLP Findings 2020 · 385 citations
- SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment AnalysisHao Tian, Can Gao, Xinyan Xiao, Hao Liu et al.ACL 2020 · 265 citations
Related papers
- Knowledge Perceived Multi-modal Pretraining in E-commerceYushan Zhu, Huaixiao Zhao, Wen Zhang, Ganqiang Ye et al.ACM MM 2021 · 21 citations
- KnowLog: Knowledge Enhanced Pre-trained Language Model for Log UnderstandingLipeng Ma, Weidong Yang, Bo Xu, Sihang Jiang et al.ICSE 2024 · 26 citations
- Automating Code-Related Tasks Through Transformers: The Impact of Pre-trainingRosalia Tufano, Luca Pascarella, Gabriele BavotaICSE 2023 · 15 citations
- Knowledge Prompting in Pre-trained Language Model for Natural Language UnderstandingJianing Wang, Wenkang Huang, Minghui Qiu, Qiuhui Shi et al.EMNLP 2022 · 26 citations
- Injecting Domain Knowledge in Language Models for Task-oriented Dialogue SystemsDenis Emelin, Daniele Bonadiman, Sawsan Alqahtani, Yi Zhang et al.EMNLP 2022 · 9 citations
