ACL Findings2025
Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elements
Metehan Oguz, Yavuz Faruk Bakman, Duygu Nur Yaldiz
Abstract
Large Language Models (LLMs) have demonstrated impressive performances in tasks related to coreference resolution. However, previous studies mostly assessed LLM performance on coreference resolution with nouns and third person pronouns. This study evaluates LLM performance on coreference resolution with indexicals like I, you, here and tomorrow, which come with unique challenges due to their linguistic properties. We present the first study examining how LLMs interpret indexicals in English, releasing the English Indexical Dataset with 1600 multiple-choice questions. We evaluate pioneering LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek V3. Our results reveal that LLMs exhibit an impressive performance with some indexicals (I), while struggling with others (you, here, tomorrow), and that syntactic cues (e.g. quotation) contribute to LLM performance with some indexicals, while they reduce performance with others.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on3
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Large language models are few-shot clinical information extractorsMonica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim et al.EMNLP 2022 · 285 citations
- MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMsYavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao et al.ACL 2024 · 6 citations
Related papers
- Correct-Detect: Balancing Performance and Ambiguity Through the Lens of Coreference Resolution in LLMsAmber Shore, Russell Scheinberg, Ameeta Agrawal, So Young LeeEMNLP 2025
- ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in InstructionsXingwei He, Qianru Zhang, Pengfei Chen, Guanhua Chen et al.AAAI 2026 · 2 citations
- Measuring Intent Comprehension in LLMsNadav Kunievsky, James EvansICML 2026 · 1 citation
- Information Extraction from Legal Wills: How Well Does GPT-4 Do?Alice Saebom Kwak, Cheonkam Jeong, Gaetano Forte, Derek E. Bambauer et al.EMNLP Findings 2023 · 11 citations
- ExpliCa: Evaluating Explicit Causal Reasoning in Large Language ModelsMartina Miliani, Serena Auriemma, Alessandro Bondielli, Emmanuele Chersoni et al.ACL Findings 2025 · 4 citations
