Lune

ACL Findings2021

Probing Pre-Trained Language Models for Disease Knowledge

Israa Alghanmi, Luis Espinosa Anke, Steven Schockaert

Abstract

Pre-trained language models such as Clini-calBERT have achieved impressive results on tasks such as medical Natural Language Inference. At first glance, this may suggest that these models are able to perform medical reasoning tasks, such as mapping symptoms to diseases. However, we find that standard benchmarks such as MedNLI contain relatively few examples that require such forms of reasoning. To better understand the medical reasoning capabilities of existing language models, in this paper we introduce DisKnE, a new benchmark for Disease Knowledge Evaluation. To construct this benchmark, we annotated each positive MedNLI example with the types of medical reasoning that are needed. We then created negative examples by corrupting these positive examples in an adversarial way. Furthermore, we define training-test splits per disease, ensuring that no knowledge about test diseases can be learned from the training data, and we canonicalize the formulation of the hypotheses to avoid the presence of artefacts. This leads to a number of binary classification problems, one for each type of reasoning and each disease. When analysing pre-trained models for the clinical/biomedical domain on the proposed benchmark, we find that their performance drops considerably.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Inducing Relational Knowledge from BERT
  • Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites Inducing Relational Knowledge from BERT
  • Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Cross-Lingual Consistency of Factual Knowledge in Multilingual Language ModelsJirui Qi, Raquel Fernández, Arianna BisazzaEMNLP 2023 · 9 citations

Builds on4

  1. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  2. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  3. Inducing Relational Knowledge from BERTZied Bouraoui, José Camacho-Collados, Steven SchockaertAAAI 2020 · 183 citations
  4. Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name RecognitionYun He, Ziwei Zhu, Yin Zhang, Qin Chen et al.EMNLP 2020 · 103 citations

Related papers

  1. Benchmarking LLMs on Authentic Cases from Medical JournalsWanlong Liu, Junying Chen, Yunjin Yang, Prayag Tiwari et al.ACL Findings 2026 · 1 citation
  2. MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language ModelsYan Cai, Linlin Wang, Ye Wang, Gerard de Melo et al.AAAI 2024 · 42 citations
  3. Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning BenchmarksMiao Jing, Mengting Jia, Junling Lin, Zhongxia Shen et al.ICLR 2026 · 4 citations
  4. DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language ModelsYakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang et al.ACL Findings 2026 · 3 citations
  5. DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsYanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier et al.ACL 2023 · 19 citations
Dusk over the sea between two cliffs drawn in fine vertical lines