Lune

ACL Findings2026

SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures

Jiaming Wang, Zhe Tang, Zehao Jin, Hefei Chen, Yilin Jin, Peng Ding, Xiaoyu Li, Xuezhi Cao

Abstract

As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have opensourced our work on: https://github.com/ meituan-longcat/SOP-Maze.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following cites InFoBench: Evaluating Instruction Following Ability in Large Language Models
  • Self-Instruct: Aligning Language Models with Self-Generated Instructions cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  2. Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu et al.ACL 2023 · 540 citations
  3. COLLIE: Systematic Construction of Constrained Text Generation TasksShunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang et al.ICLR 2024 · 65 citations
  4. InFoBench: Evaluating Instruction Following Ability in Large Language ModelsYiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao et al.ACL Findings 2024 · 26 citations
  5. StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction FollowingJinnan Li, Jinzhe Li, Yue Wang, Yi Chang et al.ACL Findings 2025 · 3 citations

Related papers

  1. SupChain-Bench: Benchmarking Large Language Models for Real-World Supply Chain ManagementShengyue Guan, Yihao Liu, Lang CaoACL Findings 2026
  2. EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise SettingsShiva Malay, Perampalli Shravan Nayak, Sagar Davasam, Srinivas Sunkara et al.ICML 2026 · 8 citations
  3. OrchestrationBench: LLM-Driven Agentic Planning and Tool Use in Multi-Domain ScenariosAelim Ahn, Sooyeon Lee, Hyosun Wang, Chiwan Park et al.ICLR 2026
  4. AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
  5. EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language ModelsTao Zou, Xinghua Zhang, Haiyang Yu, Minzheng Wang et al.EMNLP 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines