Lune

EMNLP Findings2022

ClinicalT5: A Generative Language Model for Clinical Text

Qiuhao Lu, Dejing Dou, Thien Huu Nguyen

Abstract

In the past few years, large pre-trained language models (PLMs) have been widely adopted in different areas and have made fundamental improvements over a variety of downstream tasks in natural language processing (NLP). Meanwhile, domain-specific variants of PLMs are being proposed to address the needs of domains that demonstrate a specific pattern of writing and vocabulary, e.g., BioBERT for the biomedical domain and ClinicalBERT for the clinical domain. Recently, generative language models like BART and T5 are gaining popularity with their competitive performance on text generation as well as on tasks cast as generative problems. However, in the clinical domain, such domain-specific generative variants are still underexplored. To address this need, our work introduces a T5-based text-to-text transformer model pre-trained on clinical text, i.e., Clini-calT5. We evaluate the proposed model both intrinsically and extrinsically over a diverse set of tasks across multiple datasets, and show that ClinicalT5 dramatically outperforms T5 in the domain-specific tasks and compares favorably with its close baselines. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Subjective Behaviors and Preferences in LLM: Language of Browsing cites Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
  • Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models cites Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
  • Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Leveraging Knowledge Graph-Enhanced LLMs for Context-Aware Medical ConsultationSu-Hyeong Park, Ho-Beom Kim, Seong-Jin Park, Dinara Aliyeva et al.EMNLP 2025
  2. Subjective Behaviors and Preferences in LLM: Language of BrowsingSai Sundaresan, Harshita Chopra, Atanu R. Sinha, Koustava Goswami et al.EMNLP 2025

Builds on6

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Autoregressive Entity RetrievalNicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio PetroniICLR 2021 · 200 citations
  3. Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name RecognitionYun He, Ziwei Zhu, Yin Zhang, Qin Chen et al.EMNLP 2020 · 103 citations
  4. Learning Conceptual-Contextual Embeddings for Medical TextXiao Zhang, Dejing Dou, Ji WuAAAI 2020 · 17 citations
  5. Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language ModelsQiuhao Lu, Dejing Dou, Thien Huu NguyenEMNLP Findings 2021 · 15 citations

Related papers

  1. Incorporating medical knowledge in BERT for clinical relation extractionArpita Roy, Shimei PanEMNLP 2021 · 56 citations
  2. DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsYanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier et al.ACL 2023 · 19 citations
  3. Exploring the Impact of Corpus Diversity on Financial Pretrained Language ModelsJaeyoung Choe, Keonwoong Noh, Nayeon Kim, Seyun Ahn et al.EMNLP Findings 2023 · 6 citations
  4. Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language ModelsRan Xu, Hejie Cui, Yue Yu, Xuan Kan et al.ACL Findings 2024 · 24 citations
  5. Large language models are few-shot clinical information extractorsMonica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim et al.EMNLP 2022 · 285 citations
Dusk over the sea between two cliffs drawn in fine vertical lines