An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Negation is underrepresented in existing natural language inference benchmarks. Additionally, one can often ignore the few negations in existing benchmarks and still make the right inference judgments. In this paper, we present a new benchmark for natural language inference in which negation plays an important role. We also show that state-of-the-art transformers struggle making inference judgments with the new pairs. Original pair New pair w/ negation RTE T: Tropical Storm Debby is blamed for several deaths across the Caribbean. T neg : Tropical Storm Debby is not blamed for several deaths across the Caribbean. H: A tropical storm has caused loss of life. H neg : A tropical storm has not caused loss of life.