EMNLP Findings2022
ClinicalT5: A Generative Language Model for Clinical Text
Qiuhao Lu, Dejing Dou, Thien Huu Nguyen
Abstract
In the past few years, large pre-trained language models (PLMs) have been widely adopted in different areas and have made fundamental improvements over a variety of downstream tasks in natural language processing (NLP). Meanwhile, domain-specific variants of PLMs are being proposed to address the needs of domains that demonstrate a specific pattern of writing and vocabulary, e.g., BioBERT for the biomedical domain and ClinicalBERT for the clinical domain. Recently, generative language models like BART and T5 are gaining popularity with their competitive performance on text generation as well as on tasks cast as generative problems. However, in the clinical domain, such domain-specific generative variants are still underexplored. To address this need, our work introduces a T5-based text-to-text transformer model pre-trained on clinical text, i.e., Clini-calT5. We evaluate the proposed model both intrinsically and extrinsically over a diverse set of tasks across multiple datasets, and show that ClinicalT5 dramatically outperforms T5 in the domain-specific tasks and compares favorably with its close baselines. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Subjective Behaviors and Preferences in LLM: Language of Browsing cites Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
- Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models cites Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
- Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models cites Language Models are Few-Shot Learners
Cited by top-tier papers2
- Leveraging Knowledge Graph-Enhanced LLMs for Context-Aware Medical ConsultationSu-Hyeong Park, Ho-Beom Kim, Seong-Jin Park, Dinara Aliyeva et al.EMNLP 2025
- Subjective Behaviors and Preferences in LLM: Language of BrowsingSai Sundaresan, Harshita Chopra, Atanu R. Sinha, Koustava Goswami et al.EMNLP 2025
Builds on6
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Autoregressive Entity RetrievalNicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio PetroniICLR 2021 · 200 citations
- Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name RecognitionYun He, Ziwei Zhu, Yin Zhang, Qin Chen et al.EMNLP 2020 · 103 citations
- Learning Conceptual-Contextual Embeddings for Medical TextXiao Zhang, Dejing Dou, Ji WuAAAI 2020 · 17 citations
- Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language ModelsQiuhao Lu, Dejing Dou, Thien Huu NguyenEMNLP Findings 2021 · 15 citations
Related papers
- Incorporating medical knowledge in BERT for clinical relation extractionArpita Roy, Shimei PanEMNLP 2021 · 56 citations
- DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsYanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier et al.ACL 2023 · 19 citations
- Exploring the Impact of Corpus Diversity on Financial Pretrained Language ModelsJaeyoung Choe, Keonwoong Noh, Nayeon Kim, Seyun Ahn et al.EMNLP Findings 2023 · 6 citations
- Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language ModelsRan Xu, Hejie Cui, Yue Yu, Xuan Kan et al.ACL Findings 2024 · 24 citations
- Large language models are few-shot clinical information extractorsMonica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim et al.EMNLP 2022 · 285 citations
