EMNLP Findings2023
MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
Damien Sileo, Antoine Lernould
Abstract
Theory of Mind (ToM) is a critical component of intelligence but its assessment remains the subject of heated debates. Prior research applied human ToM assessments to natural language processing models using either humancreated standardized tests or rule-based templates. However, these methods primarily focus on simplistic reasoning and require further validation. Here, we leverage dynamic epistemic logic to isolate a particular component of ToM and to generate controlled problems. We also introduce new verbalization techniques to express these problems in English natural language. Our findings indicate that some language model scaling (from 70M to 6B and 350M to 174B) does not consistently yield results better than random chance. While GPT-4 demonstrates superior epistemic reasoning capabilities, there is still room for improvement. Our code and datasets are publicly available 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers4
- Logical forms complement probability in understanding language model (and human) performanceYixuan Wang, Freda ShiACL 2025 · 2 citations
- Revealing the Seen, Imagining the Beyond: A Survey of Image-Grounded Chain-of-Thought Reasoning in Multimodal LLMsQihua Dong, Yitian Zhang, Huimin Zeng, Yizhou Wang et al.ACL 2026
- CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language ModelsHaibo Tong, Zeyang Yue, Feifei Zhao, Erliang Lin et al.ACL 2026
- Theory of Mind in Large Language Models: Assessment and EnhancementRuirui Chen, Weifeng Jiang, Chengwei Qin, Cheston TanACL 2025
Builds on7
- Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingPengcheng He, Jianfeng Gao, Weizhu ChenICLR 2023 · 394 citations
- Perturbation Augmentation for Fairer NLPRebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith et al.EMNLP 2022 · 54 citations
- Pushing the Limits of Rule Reasoning in Transformers through Natural Language SatisfiabilityKyle Richardson, Ashish SabharwalAAAI 2022 · 29 citations
- MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative TasksCristian-Paul Bara, Sky CH-Wang, Joyce ChaiEMNLP 2021 · 28 citations
Related papers
- ToMBench: Benchmarking Theory of Mind in Large Language ModelsZhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen et al.ACL 2024 · 6 citations
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMsMaarten Sap, Ronan Le Bras, Daniel Fried, Yejin ChoiEMNLP 2022 · 92 citations
- Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language ModelsYufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea et al.EMNLP Findings 2023 · 8 citations
- MMToM-QA: Multimodal Theory of Mind Question AnsweringChuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang et al.ACL 2024 · 8 citations
- AutoToM: Scaling Model-based Mental Inference via Automated Agent ModelingZhining Zhang, Chuanyang Jin, Mung Yao Jia, Shunchi Zhang et al.NeurIPS 2025 · 30 citations
