GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations
Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, Kaidi Xu
Abstract
As Large Language Models (LLMs) are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial. This paper evaluates LLMs' reasoning abilities in competitive environments through game-theoretic tasks, e.g., board and card games that require pure logic and strategic reasoning to compete with opponents. We first propose GTBENCH, a languagedriven environment composing 10 widely-recognized tasks, across a comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios. Then, we ➊ Characterize the game-theoretic reasoning of LLMs; and ➋ Perform LLM-vs.-LLM competitions as reasoning evaluation. We observe that ➊ LLMs have distinct behaviors regarding various gaming scenarios; for example, LLMs fail in complete and deterministic games yet they are competitive in probabilistic gaming scenarios; ➋ Most open-source LLMs, e.g., CodeLlama-34b-Instruct and Llama-2-70b-chat, are less competitive than commercial LLMs, e.g., GPT-4, in complex games, yet the recently released Llama-3-70b-Instruct makes up for this shortcoming. In addition, code-pretraining greatly benefits strategic reasoning, while advanced reasoning methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) do not always help. We further characterize the game-theoretic properties of LLMs, such as equilibrium and Pareto Efficiency in repeated games. Detailed error profiles are provided for a better understanding of LLMs' behavior. We hope our research provides standardized protocols and serves as a foundation to spur further explorations in the strategic reasoning of LLMs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
13 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Bayesian Social Deduction with Graph-Informed Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Evaluating Language Models' Evaluations of Games cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers13
- Code World Models for General Game PlayingWolfgang Lehrach, Daniel Hennes, Miguel Lazaro-Gredilla, Xinghua Lou et al.ICLR 2026 · 27 citations
- How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool UseMinhua Lin, Enyan Dai, Hui Liu, Xianfeng Tang et al.ICLR 2026 · 9 citations
- Evaluating Language Models' Evaluations of GamesKatherine M. Collins, Cedegao E. Zhang, Graham Todd, Lance Ying et al.ICLR 2026 · 5 citations
- MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM GamesYunfei Xie, Kevin Wang, Bobby Cheng, Jianzhu Yao et al.ICML 2026 · 4 citations
- Bayesian Social Deduction with Graph-Informed Language ModelsShahab Rahimirad, Guven Gergerli, Lucia Romero, Angela Qian et al.ACL 2026 · 4 citations
Builds on21
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
Related papers
- AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
- LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language ModelsMihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura et al.ACL 2024
- PokerBench: Training Large Language Models to Become Professional Poker PlayersRichard Zhuang, Akshat Gupta, Richard Yang, Aniket Rahane et al.AAAI 2025 · 12 citations
- GAMEBoT: Transparent Assessment of LLM Reasoning in GamesWenye Lin, Jonathan Roberts, Yunhan Yang, Samuel Albanie et al.ACL 2025 · 12 citations
- GameArena: Evaluating LLM Reasoning through Live Computer GamesLanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang et al.ICLR 2025
