EMNLP Findings2024
Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition
Abstract
Verbs form the backbone of language, providing the structure and meaning to sentences. Yet, their intricate semantic nuances pose a longstanding challenge. Understanding verb relations through the concept of lexical entailment is crucial for comprehending sentence meanings and grasping verb dynamics. This work investigates the capabilities of eight Large Language Models in recognizing lexical entailment relations among verbs through differently devised prompting strategies and zero-/few-shot settings over verb pairs from two lexical databases, namely WordNet and Hy-perLex. Our findings unveil that the models can tackle the lexical entailment recognition task with moderately good performance, although at varying degree of effectiveness and under different conditions. Also, utilizing fewshot prompting can enhance the models' performance. However, perfectly solving the task arises as an unmet challenge for all examined LLMs, which raises an emergence for further research developments on this topic.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on5
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMsMiao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li et al.ICLR 2024 · 867 citations
- TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic TasksViktor Moskvoretskii, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko et al.ACL 2024 · 11 citations
- This is not a Dataset: A Large Negation Benchmark to Challenge Large Language ModelsIker García-Ferrero, Begoña Altuna, Javier Álvez, Itziar Gonzalez-Dios et al.EMNLP 2023 · 8 citations
- WordNet Is All You Need: A Surprisingly Effective Unsupervised Method for Graded Lexical EntailmentJoseph Renner, Pascal Denis, Rémi GilleronEMNLP Findings 2023 · 1 citation
Related papers
- No clues good clues: out of context Lexical Relation ClassificationLucia Pitarch, Jordi Bernad, Lacramioara Dranca, Carlos Bobed Lisbona et al.ACL 2023 · 4 citations
- Pre-trained Language Models Can be Fully Zero-Shot LearnersXuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu et al.ACL 2023 · 22 citations
- Benchmarking Query-Conditioned Natural Language InferenceMarc E. Canby, Xinchi Chen, Xing Niu, Jifan Chen et al.ACL Findings 2025 · 1 citation
- Exploring the Effectiveness of Prompt Engineering for Legal Reasoning TasksFangyi Yu, Lee Quartey, Frank SchilderACL Findings 2023 · 44 citations
- Prompting Language Models for Linguistic StructureTerra Blevins, Hila Gonen, Luke ZettlemoyerACL 2023 · 15 citations
