Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges

Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose DialogTool, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) tool creation; 2) tool utilization: tool awareness, tool selection, tool execution; and 3) role-consistent response: response generation and role play. Furthermore, we build VirtualMobile -an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs 1 . Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct openand closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons. "Hi, could you get me a restaurant booking on the 8th please?" "Any preference on the restaurant, location and time?" "Could you get me a reservation at P.f. Chang's in Corte Madera at afternoon 12?" "Please confirm your reservation at P.f. Chang's in Corte Madera at 12 pm for 2 on March 8th.