ACL Findings2026
ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Yining Zheng
摘要
The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of realworld engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from opensource repositories. Distinct from previous evaluations, ABC-Bench requires the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external endto-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
它引用的 Paper8
- SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun 等NeurIPS 2024 · 1,586 次引用
- SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based AgentsYifu Guo, Jiaye Lin, Huacan Wang, Yuzhen Han 等NeurIPS 2025 · 73 次引用
- RepairAgent: An Autonomous, LLM-Based Agent for Program RepairIslem Bouzenia, Premkumar T. Devanbu, Michael PradelICSE 2025 · 54 次引用
- HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation TaskZhaojian Yu, Yilun Zhao, Arman Cohan, Xiaoping ZhangACL Findings 2025 · 13 次引用
- NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User QueriesShudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng 等ACL Findings 2024 · 11 次引用
相关 Paper
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding AgentsJingzhe Ding, Shengda Long, Changxin Pu, Ge Zhang 等ICML 2026 · 37 次引用
- BaxBench: Can LLMs Generate Correct and Secure Backends?Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev 等ICML 2025
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security TasksHwiwon Lee, Ziqi Zhang, Hanxiao Lu, Lingming ZhangNeurIPS 2025 · 86 次引用
- FeatureBench: Benchmarking Agentic Coding for Complex Feature DevelopmentQixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao 等ICLR 2026 · 30 次引用
- AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World ContextsKeyu Li, Junhao Shi, Yang Xiao, Mohan Jiang 等ACL 2026 · 14 次引用
