Lune

EMNLP Findings2022

BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples

Mohaddeseh Bastan, Mihai Surdeanu, Niranjan Balasubramanian

Abstract

Natural language inference (NLI) is critical for complex decision-making in biomedical domain. One key question, for example, is whether a given biomedical mechanism is supported by experimental evidence. This can be seen as an NLI problem but there are no directly usable datasets to address this. The main challenge is that manually creating informative negative examples for this task is difficult and expensive. We introduce a novel semi-supervised procedure that bootstraps an NLI dataset from existing biomedical dataset that pairs mechanisms with experimental evidence in abstracts. We generate a range of negative examples using nine strategies that manipulate the structure of the underlying mechanisms both with rules, e.g., flip the roles of the entities in the interaction, and, more importantly, as perturbations via logical constraints in a neuro-logical decoding system (Lu et al., 2021b).

We use this procedure to create a novel dataset for NLI in the biomedical domain, called BioNLI and benchmark two state-of-the-art biomedical classifiers. The best result we obtain is around mid 70s in F1, suggesting the difficulty of the task. Critically, the performance on the different classes of negative examples varies widely, from 97% F1 on the simple role change negative examples, to barely better than chance on the negative examples generated using neuro-logic decoding. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Adversarial NLI: A New Benchmark for Natural Language Understanding cites Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. MED-COREASONER: Reducing Language Disparities in Medical Reasoning via Language-Informed Co-ReasoningFan Gao, Sherry T. Tong, Jiwoong Sohn, Jiahao Huang et al.ACL 2026 · 1 citation
  2. Efficient Domain Continual pretraining by Mitigating the Stability GapYiduo Guo, Jie Fu, Huishuai Zhang, Dongyan ZhaoACL 2025

Builds on10

  1. Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
  2. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  3. LinkBERT: Pretraining Language Models with Document LinksMichihiro Yasunaga, Jure Leskovec, Percy LiangACL 2022 · 463 citations
  4. Controlled Text Generation as Continuous Optimization with Multiple ConstraintsSachin Kumar, Eric Malmi, Aliaksei Severyn, Yulia TsvetkovNeurIPS 2021 · 91 citations
  5. Evaluating the Factual Consistency of Abstractive Text SummarizationWojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard SocherEMNLP 2020 · 67 citations

Related papers

  1. NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial ReportsMaël Jullien, Marco Valentino, Hannah Frost, Paul O'Regan et al.EMNLP 2023 · 6 citations
  2. Conditional Natural Language InferenceYoungwoo Kim, Razieh Rahimi, James AllanEMNLP Findings 2023 · 1 citation
  3. Learning to Infer from Unlabeled Data: A Semi-supervised Learning Approach for Robust Natural Language InferenceMobashir Sadat, Cornelia CarageaEMNLP Findings 2022 · 1 citation
  4. Transitive self-consistency evaluation of NLI models without gold labelsWei Wu, Mark LastEMNLP 2025
  5. IMPLI: Investigating NLI Models' Performance on Figurative LanguageKevin Stowe, Prasetya Ajie Utama, Iryna GurevychACL 2022 · 52 citations
Dusk over the sea between two cliffs drawn in fine vertical lines