EMNLP Findings2024
Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues
Deuksin Kwon, Emily Weiss, Tara Kulshrestha, Kushal Chawla, Gale M. Lucas, Jonathan Gratch
Abstract
A successful negotiation requires a range of capabilities, including comprehension of the conversation context, Theory-of-Mind (ToM) skills to infer the partner's motives, strategic reasoning, and effective communication, making it challenging for automated systems.Despite the remarkable performance of LLMs in various NLP tasks, there is no systematic evaluation of their capabilities in negotiation.Such an evaluation is critical for advancing AI negotiation agents and negotiation research, ranging from designing dialogue systems to providing pedagogical feedback and scaling up data collection practices.This work aims to systematically analyze the multifaceted capabilities of LLMs across diverse dialogue scenarios throughout the stages of a typical negotiation interaction.Our analysis highlights GPT-4's superior performance in many tasks while identifying specific challenges, such as making subjective assessments and generating contextually appropriate, strategically advantageous responses.The code is available at https://github.com/DSincerity/SysEval- NegoLLMs
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MERIT Feedback Elicits Better Bargaining in LLM Negotiators cites ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning via Tool-integrated Action for Dynamic Offer Optimization
- CoSToM: Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language Models cites Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User Simulation
- Towards Reasoning in Large Language Models: A Survey cites Large Language Models are Zero-Shot Reasoners
- Towards Reasoning in Large Language Models: A Survey cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers5
- ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning via Tool-integrated Action for Dynamic Offer OptimizationDeuksin Kwon, Jiwon Hae, Emma Clift, Daniel Shamsoddini et al.EMNLP 2025 · 1 citation
- Strength Lies in Differences! Improving Strategy Planning for Non-collaborative Dialogues via Diversified User SimulationTong Zhang, Chen Huang, Yang Deng, Hongru Liang et al.EMNLP 2024 · 1 citation
- MERIT Feedback Elicits Better Bargaining in LLM NegotiatorsJihwan Oh, Murad Aghazada, Yooju Shin, Se-Young Yun et al.ACL 2026 · 1 citation
- Estimating the Empowerment of Language Model AgentsJinyeop Song, Jeff Gore, Max Kleiman-WeinerICML 2026
- CoSToM: Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language ModelsMengfan Li, Xuanhua Shi, Yang DengACL 2026
Builds on7
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
- Towards Reasoning in Large Language Models: A SurveyJie Huang, Kevin Chen-Chuan ChangACL Findings 2023 · 428 citations
Related papers
- NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation SurroundingChunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng et al.EMNLP Findings 2024 · 4 citations
- How Well Can LLMs Negotiate? NegotiationArena Platform and AnalysisFederico Bianchi, Patrick John Chia, Mert Yüksekgönül, Jacopo Tagliabue et al.ICML 2024 · 90 citations
- ToMBench: Benchmarking Theory of Mind in Large Language ModelsZhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen et al.ACL 2024 · 6 citations
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language AgentsXuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang et al.ICLR 2024 · 288 citations
- Evaluating Language Model Agency Through NegotiationsTim R. Davidson, Veniamin Veselovsky, Michal Kosinski, Robert WestICLR 2024 · 52 citations
