ACL Findings2026
Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming
Sama Hadhoud, Alaa Elsetohy, Frederikus Hudi, Jan Christian Blaise Cruz, Steven Halim, Alham Fikri Aji
Abstract
Large Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic reasoning with code-level implementation. We argue that competitive programming is fundamentally a problem-solving task and propose centering natural-language editorials in both solution generation and evaluation. Generating an editorial prior to code improves solve rates for some LLMs, with substantially larger gains when using expertly written gold editorials. However, even with gold editorials, models continue to struggle with implementation, while the gap between generated and gold editorials reveals a persistent problem-solving bottleneck in specifying correct and complete algorithms. Beyond pass/fail metrics, we diagnose reasoning errors by comparing model-generated editorials to gold standards using expert annotations and validate an LLM-as-a-judge protocol for scalable evaluation. We introduce a dataset of 83 ICPC-style problems with gold editorials and full test suites, and evaluate 19 LLMs, arguing that future benchmarks should explicitly separate problem solving from implementation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- MapCoder: Multi-Agent Code Generation for Competitive Problem Solving cites CodeT: Code Generation with Generated Tests
- MapCoder: Multi-Agent Code Generation for Competitive Problem Solving cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- CodeContests+: High-Quality Test Case Generation for Competitive Programming cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Builds on6
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
- CodeT: Code Generation with Generated TestsBei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan et al.ICLR 2023 · 64 citations
- MapCoder: Multi-Agent Code Generation for Competitive Problem SolvingMd. Ashraful Islam, Mohammed Eunus Ali, Md. Rizwan ParvezACL 2024 · 29 citations
- ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive ProgrammingXinwei Yang, Zhaofeng Liu, Chen Huang, Jiashuai Zhang et al.ACL 2025 · 15 citations
- CodeContests+: High-Quality Test Case Generation for Competitive ProgrammingZihan Wang, Siyao Liu, Yang Sun, Ming Ding et al.EMNLP Findings 2025 · 7 citations
Related papers
- AetherCode: Evaluating LLMs’ Ability to Win In Premier Programming CompetitionsZihan Wang, Jiaze Chen, Zhicheng Liu, Haojie Pan et al.ICLR 2026 · 11 citations
- AutoCode: LLMs as Problem Setters for Competitive ProgrammingShang Zhou, Zihan Zheng, Kaiyuan Liu, Zeyu Shen et al.ICLR 2026 · 12 citations
- InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed InformationJiaze Li, Aocheng Shen, Bing Liu, Boyu Zhang et al.ICML 2026
- Self-Edit: Fault-Aware Code Editor for Code GenerationKechi Zhang, Zhuo Li, Jia Li, Ge Li et al.ACL 2023 · 42 citations
- Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive ProgrammingTingqiang Xu, Hangrui Zhou, Tianle Cai, Alex Gu et al.ICML 2026 · 1 citation
