EMNLP Findings2023
BLM-s/lE: A structured dataset of English spray-load verb alternations for testing generalization in LLMs
Giuseppe Samo, Vivi Nastase, Chunyang Jiang, Paola Merlo
Abstract
Current NLP models appear to be achieving performance comparable to human capabilities on well-established benchmarks. New benchmarks are now necessary to test deeper layers of understanding of natural languages by these models.
Ask Lune
Ask anything. The answer quotes the paper.
Lune reads all of it and the papers around it, then shows you the passages behind every answer.
Builds on2
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsKevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. ManningICLR 2020 · 541 citations
- Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Can Large Language Models pass the test?Paola MerloEMNLP Findings 2023 · 2 citations
Related papers
- Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
- AwarenessBench: Assessing Cognitive Capabilities of Language ModelsXiaojian Li, Rongwu Xu, Tianyun Zhang, Yue Wang et al.ACL 2026
- In Benchmarks We Trust ... Or Not?Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens et al.EMNLP 2025 · 1 citation
- What's the Best Place for an AI Conference, Vancouver or _______: Why Completing Comparative Questions is DifficultAvishai Zagoury, Einat Minkov, Idan Szpektor, William W. CohenAAAI 2021 · 6 citations
- Do LLMs Really Struggle at NL-FOL Translation? Revealing Their Strengths via a Novel Benchmarking StrategyAndrea Brunello, Luca Geatti, Michele Mignani, Angelo Montanari et al.AAAI 2026
