Lune

ACL Findings2024

Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games

Dekun Wu, Haochen Shi, Zhiyuan Sun, Bang Liu

Abstract

In this study, we explore the application of Large Language Models (LLMs) in Jubensha, a Chinese detective role-playing game and a novel area in Artificial Intelligence (AI) driven gaming. We introduce the first dataset specifically for Jubensha, including character scripts and game rules, to foster AI agent development in this complex narrative environment. Our work also presents a unique multi-agent interaction framework using LLMs, allowing AI agents to autonomously engage in this game. To evaluate the gaming performance of these AI agents, we developed novel methods measuring their mastery of case information and reasoning skills. Furthermore, we incorporated the latest advancements in in-context learning to improve the agents' performance in information gathering, murderer identification, and logical reasoning. The experimental results validate the effectiveness of our proposed methods. This work aims to offer a novel perspective on understanding LLM capabilities and establish a new benchmark for evaluating large language model-based agents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model ServingYuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray et al.SIGCOMM 2024 · 111 citations
  2. Defending Against Social Engineering Attacks in the Age of LLMsLin Ai, Tharindu Kumarage, Amrita Bhattacharjee, Zizhou Liu et al.EMNLP 2024 · 12 citations
  3. PRISON: Unmasking the Criminal Potential of Large Language ModelsXinyi Wu, Geng Hong, Pei Chen, Yueyue Chen et al.ICLR 2026 · 3 citations
  4. MultiMind: Enhancing Werewolf Agents with Multimodal Reasoning and Theory of MindZheng Zhang, Nuoqian Xiao, Qi Chai, Deheng Ye et al.ACM MM 2025 · 1 citation
  5. Investigating Prosocial Behavior Theory in LLM Agents Under Policy-Induced InequitiesYujia Zhou, Hexi Wang, Qingyao Ai, Zhen Wu et al.AAAI 2026 · 1 citation

Related papers

  1. TurnaboutLLM: A Deductive Reasoning Benchmark from Detective GamesYuan Yuan, Muyu He, Muhammad Adil Shahid, Ziyang Li et al.EMNLP 2025
  2. Large Language Models Are Neurosymbolic ReasonersMeng Fang, Shilong Deng, Yudi Zhang, Zijing Shi et al.AAAI 2024 · 53 citations
  3. Learning to Play Like Humans: A Framework for LLM Adaptation in Interactive Fiction GamesJinming Zhang, Yunfei LongACL Findings 2025
  4. Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video GamesDongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim et al.ICLR 2026 · 30 citations
  5. CharacterCraft: Bridging the Literature-Reality Dialogue Gap for Practical Role-Playing AgentsXuyan Yin, Xinran Yang, Zihao Li, Lixin Zou et al.EMNLP Findings 2025
Dusk over the sea between two cliffs drawn in fine vertical lines