ACL Findings2024
Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World
Guande Wu, Chen Zhao, Cláudio T. Silva, He He
Abstract
Language agents that interact with the world on their own have great potential for automating digital tasks. While large language model (LLM) agents have made progress in understanding and executing tasks such as textual games and webpage control, many real-world tasks also require collaboration with humans or other LLMs in equal roles, which involves intent understanding, task coordination, and communication. To test LLM's ability to collaborate, we design a blocks-world environment, where two agents, each having unique goals and skills, build a target structure together. To complete the goals, they can act in the world and communicate in natural language. Under this environment, we design increasingly challenging settings to evaluate different collaboration perspectives, from independent to more complex, dependent tasks. We further adopt chain-of-thought prompts that include intermediate reasoning steps to model the partner's state and identify and correct execution errors. Both human-machine and machine-machine experiments show that LLM agents have strong grounding capacities, and our approach significantly improves the evaluation metric.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- Satori 悟り: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers2
- MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical ReasoningShuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen et al.NeurIPS 2024 · 215 citations
- Satori 悟り: Towards Proactive AR Assistant with Belief-Desire-Intention User ModelingChenyi Li, Guande Wu, Gromit Yeuk-Yin Chan, Dishita G. Turakhia et al.CHI 2025 · 49 citations
Builds on13
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
- WebArena: A Realistic Web Environment for Building Autonomous AgentsShuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou et al.ICLR 2024 · 1,197 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
Related papers
- Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on MinecraftKranti Chalamalasetti, Sherzod Hakimov, David SchlangenEMNLP Findings 2024 · 2 citations
- Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement LearningWenjin Liu, Haoran Luo, Xueyuan Lin, Haoming Liu et al.ACL Findings 2026 · 1 citation
- Theory of Mind for Multi-Agent Collaboration via Large Language ModelsHuao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell et al.EMNLP 2023 · 57 citations
- Symbolic Planning and Code Generation for Grounded DialogueJustin T. Chiu, Wenting Zhao, Derek Chen, Saujas Vaduguru et al.EMNLP 2023
- Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative AgentsHaochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren et al.EMNLP 2025
