ACL Findings2024
Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method
Tian Xia, Zhiwei He, Tong Ren, Yibo Miao, Zhuosheng Zhang, Yang Yang, Rui Wang
Abstract
Bargaining is an important and unique part of negotiation between humans. As LLM-driven agents learn to negotiate and act like real humans, how to evaluate agents' bargaining abilities remains an open problem. For the first time, we formally described the Bargaining task as an asymmetric incomplete information game, defining the gains of the Buyer and Seller in multiple bargaining processes. It allows us to quantitatively assess an agent's performance in the Bargain task. We collected a real product price dataset, AmazonHistoryPrice, and conducted evaluations of various LLM agents' bargaining abilities 1 . We find that playing a Buyer is much harder than a Seller, and increasing model size can not effectively improve the Buyer's performance. To address the challenge, we propose a novel approach called OG-Narrator that integrates a deterministic Offer Generator to control the price range of Buyer's offers, and an LLM Narrator to create natural language sentences for generated offers. Experimental results show that OG-Narrator improves the buyer's deal rates from 26.67% to 88.88% and brings a ten times multiplication of profits on all baselines, even a model that has not been aligned.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation cites Generative Agents: Interactive Simulacra of Human Behavior
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation cites Large Language Models are Zero-Shot Reasoners
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated Negotiation cites Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy
- Generative Agents: Interactive Simulacra of Human Behavior cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers8
- Richelieu: Self-Evolving LLM-Based Agents for AI DiplomacyZhenyu Guan, Xiangyu Kong, Fangwei Zhong, Yizhou WangNeurIPS 2024 · 48 citations
- Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language ModelsZhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu et al.ACL 2024 · 17 citations
- PoliCon: Evaluating LLMs on Achieving Diverse Political Consensus ObjectivesZhaowei Zhang, Xiaobo Wang, Minghua Yi, Mengmeng Wang et al.ICLR 2026 · 4 citations
- Scaling Inference-Time Computation via Opponent Simulation: Enabling Online Strategic Adaptation in Repeated NegotiationXiangyu Liu, Di Wang, Zhe Feng, Aranyak MehtaICML 2026 · 2 citations
- ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning via Tool-integrated Action for Dynamic Offer OptimizationDeuksin Kwon, Jiwon Hae, Emma Clift, Daniel Shamsoddini et al.EMNLP 2025 · 1 citation
Builds on9
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
- Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 citations
Related papers
- Evaluating Language Model Agency Through NegotiationsTim R. Davidson, Veniamin Veselovsky, Michal Kosinski, Robert WestICLR 2024 · 52 citations
- MERIT Feedback Elicits Better Bargaining in LLM NegotiatorsJihwan Oh, Murad Aghazada, Yooju Shin, Se-Young Yun et al.ACL 2026 · 1 citation
- Language of BargainingMourad Heddaya, Solomon Dworkin, Chenhao Tan, Rob Voigt et al.ACL 2023 · 1 citation
- Market-Bench: Benchmarking Large Language Models on Economic and Trade CompetitionYushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu et al.ACL 2026 · 1 citation
- Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation DialoguesDeuksin Kwon, Emily Weiss, Tara Kulshrestha, Kushal Chawla et al.EMNLP Findings 2024 · 17 citations
