ACL Findings2024
Towards Objectively Benchmarking Social Intelligence of Language Agents at the Action Level
Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang
Abstract
for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures. Our code is available at https://github.com/wcx21/ Social-Tasks-in-Sandbox-Simulation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems cites Training language models to follow instructions with human feedback
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers1
Builds on22
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
Related papers
- BLM-s/lE: A structured dataset of English spray-load verb alternations for testing generalization in LLMsGiuseppe Samo, Vivi Nastase, Chunyang Jiang, Paola MerloEMNLP Findings 2023 · 1 citation
- Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET BenchmarkMinje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu et al.EMNLP 2023 · 36 citations
- Among Us: A Sandbox for Measuring and Detecting Agentic DeceptionSatvik Golechha, Adrià Garriga-AlonsoNeurIPS 2025 · 27 citations
- SocialBench: Sociality Evaluation of Role-Playing Conversational AgentsHongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu et al.ACL Findings 2024 · 16 citations
- LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social SimulationsViet Thanh Pham, Lizhen Qu, Thuy-Trang Vu, Gholamreza Haffari et al.ACL 2026
