Lune

EMNLP Findings2021

Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models

Qiuhao Lu, Dejing Dou, Thien Huu Nguyen

Abstract

Domain-specific pre-trained language models (PLMs) have achieved great success over various downstream tasks in different domains. However, existing domain-specific PLMs mostly rely on self-supervised learning over large amounts of domain text, without explicitly integrating domain-specific knowledge, which can be essential in many domains. Moreover, in knowledge-sensitive areas such as the biomedical domain, knowledge is stored in multiple sources and formats, and existing biomedical PLMs either neglect them or utilize them in a limited manner. In this work, we introduce an architecture to integrate domain knowledge from diverse sources into PLMs in a parameter-efficient way. More specifically, we propose to encode domain knowledge via adapters, which are small bottleneck feed-forward networks inserted between intermediate transformer layers in PLMs. These knowledge adapters are pre-trained for individual domain knowledge sources and integrated via an attention-based knowledge controller to enrich PLMs. Taking the biomedical domain as a case study, we explore three knowledge-specific adapters for PLMs based on the UMLS Metathesaurus graph, the Wikipedia articles for diseases, and the semantic grouping information for biomedical concepts. Extensive experiments on different biomedical NLP tasks and datasets demonstrate the benefits of the proposed architecture and the knowledge-specific adapters across multiple PLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters cites E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
  • E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT cites K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  3. K-Adapter: Infusing Knowledge into Pre-Trained Models with AdaptersRuize Wang, Duyu Tang, Nan Duan, Zhongyu Wei et al.ACL Findings 2021 · 473 citations
  4. E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTNina Pörner, Ulli Waltinger, Hinrich SchützeEMNLP Findings 2020 · 146 citations
  5. Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name RecognitionYun He, Ziwei Zhu, Yin Zhang, Qin Chen et al.EMNLP 2020 · 103 citations

Related papers

  1. Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language ModelsZaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su et al.ACL 2022 · 36 citations
  2. Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models' MemoriesShizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang et al.ACL 2023 · 17 citations
  3. DS-TOD: Efficient Domain Specialization for Task-Oriented DialogChia-Chien Hung, Anne Lauscher, Simone Paolo Ponzetto, Goran GlavasACL Findings 2022 · 15 citations
  4. Injecting Domain Knowledge in Language Models for Task-oriented Dialogue SystemsDenis Emelin, Daniele Bonadiman, Sawsan Alqahtani, Yi Zhang et al.EMNLP 2022 · 9 citations
  5. SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text MiningTaolin Zhang, Zerui Cai, Chengyu Wang, Minghui Qiu et al.ACL 2021
Dusk over the sea between two cliffs drawn in fine vertical lines