ACL Findings2021
Probing Pre-Trained Language Models for Disease Knowledge
Abstract
Pre-trained language models such as Clini-calBERT have achieved impressive results on tasks such as medical Natural Language Inference. At first glance, this may suggest that these models are able to perform medical reasoning tasks, such as mapping symptoms to diseases. However, we find that standard benchmarks such as MedNLI contain relatively few examples that require such forms of reasoning. To better understand the medical reasoning capabilities of existing language models, in this paper we introduce DisKnE, a new benchmark for Disease Knowledge Evaluation. To construct this benchmark, we annotated each positive MedNLI example with the types of medical reasoning that are needed. We then created negative examples by corrupting these positive examples in an adversarial way. Furthermore, we define training-test splits per disease, ensuring that no knowledge about test diseases can be learned from the training data, and we canonicalize the formulation of the hypotheses to avoid the presence of artefacts. This leads to a number of binary classification problems, one for each type of reasoning and each disease. When analysing pre-trained models for the clinical/biomedical domain on the proposed benchmark, we find that their performance drops considerably.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Inducing Relational Knowledge from BERT
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites Inducing Relational Knowledge from BERT
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Cited by top-tier papers1
Builds on4
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
- Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
- Inducing Relational Knowledge from BERTZied Bouraoui, José Camacho-Collados, Steven SchockaertAAAI 2020 · 183 citations
- Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name RecognitionYun He, Ziwei Zhu, Yin Zhang, Qin Chen et al.EMNLP 2020 · 103 citations
Related papers
- Benchmarking LLMs on Authentic Cases from Medical JournalsWanlong Liu, Junying Chen, Yunjin Yang, Prayag Tiwari et al.ACL Findings 2026 · 1 citation
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language ModelsYan Cai, Linlin Wang, Ye Wang, Gerard de Melo et al.AAAI 2024 · 42 citations
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning BenchmarksMiao Jing, Mengting Jia, Junling Lin, Zhongxia Shen et al.ICLR 2026 · 4 citations
- DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language ModelsYakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang et al.ACL Findings 2026 · 3 citations
- DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsYanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier et al.ACL 2023 · 19 citations
