Lune

ICML2024Top-tier venue

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su

Abstract

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

91 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  • PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites Language Models are Few-Shot Learners
  • PlanGenLLMs: A Modern Survey of LLM Planning Capabilities cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Agent Learning via Early Experience cites Reflexion: language agents with verbal reinforcement learning
  • Agent Learning via Early Experience cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reflexion: language agents with verbal reinforcement learning cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers91

  1. MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP ServersZhenting Wang, Qi Chang, Hemani Patel, Shashank Biju et al.ICLR 2026 · 109 citations
  2. Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous ThoughtHanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao et al.NeurIPS 2025 · 86 citations
  3. PlanGenLLMs: A Modern Survey of LLM Planning CapabilitiesHui Wei, Zihao Zhang, Shenghua He, Tian Xia et al.ACL 2025 · 78 citations
  4. The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMsAkshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab et al.ICLR 2026 · 64 citations
  5. Agent Learning via Early ExperienceKai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue et al.ICML 2026 · 59 citations

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
  4. Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
  5. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations

Related papers

  1. Beyond Itinerary Planning - A Real-World Benchmark for Multi-Turn and Tool-Using Travel TasksXiang Cheng, Yulan Hu, Xiangwen Zhang, Lu Xu et al.ACL 2026 · 4 citations
  2. ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel PlanningJihye Choi, Jinsung Yoon, Jiefeng Chen, Somesh Jha et al.ICLR 2026 · 18 citations
  3. CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use AgentsJiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong et al.ACL 2026 · 19 citations
  4. Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex ScenariosShijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu et al.ACL Findings 2024 · 15 citations
  5. TripTailor: A Real-World Benchmark for Personalized Travel PlanningKaimin Wang, Yuanzhe Shen, Changze Lv, Xiaoqing Zheng et al.ACL Findings 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines