AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms
Haoyu Zhao, Ziran Yang, Jiawei Li, Deyuan Mike He, Zenan Li, Chi Jin, Venugopal Veeravalli, Aarti Gupta, Sanjeev Arora
摘要
Vericoding refers to the generation of formally verified code from rigorous specifications. Recent AI models show promise in vericoding, but a unified methodology for cross-paradigm evaluation is lacking. Existing benchmarks test only individual languages/tools (e.g., Dafny, Verus, and Lean) and each covers very different tasks, so the performance numbers are not directly comparable. We address this gap with ALGOVERI, a benchmark that evaluates vericoding of 77 classical algorithms in Dafny, Verus, and Lean. By enforcing identical functional contracts, ALGOV-ERI reveals critical capability gaps in verification systems. While frontier models achieve tractable success in Dafny (40.3% for Gemini-3 Flash), where high-level abstractions and SMT automation simplify the workflow, performance collapses under the systems-level memory constraints of Verus (24.7%) and the explicit proof construction required by Lean (7.8%). Beyond aggregate metrics, we uncover a sharp divergence in test-time compute dynamics: Gemini-3 effectively utilizes iterative repair to boost performance (e.g., tripling pass rates in Dafny), whereas GPT-OSS saturates early. Finally, our error analysis shows that language design affects the refinement trajectory: while Dafny allows models to focus on logical correctness, Verus and Lean trap models in persistent syntactic and semantic barriers. All data and evaluation code can be found at https://gi thub.com/haoyuzhao123/algoveri .
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper13
- Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng 等ICLR 2024 · 被引用 858 次
- Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code ContributionsHammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt 等S&P 2022 · 被引用 725 次
- Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-CorrectionYong Lin, Shange Tang, Bohan Lyu, Ziran Yang 等ICLR 2026 · 被引用 160 次
- Baldur: Whole-Proof Generation and Repair with Large Language ModelsEmily First, Markus N. Rabe, Talia Ringer, Yuriy BrunFSE 2023 · 被引用 89 次
- Verus: Verifying Rust Programs using Linear Ghost TypesAndrea Lattuada, Travis Hance, Chanhee Cho, Matthias Brun 等OOPSLA 2023 · 被引用 86 次
相关 Paper
- Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal VerificationXu Xu, Xin Li, Xingwei Qu, Jie Fu 等ICLR 2026 · 被引用 9 次
- VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable CodeLingfei Zeng, Fengdi Che, Xuhan Huang, Fei Ye 等ICLR 2026 · 被引用 8 次
- VERINA: Benchmarking Verifiable Code GenerationZhe Ye, Zhengxu Yan, Jingxuan He, Timothe Kasriel 等ICLR 2026 · 被引用 34 次
- ALGO: Synthesizing Algorithmic Programs with Generated Oracle VerifiersKexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang 等NeurIPS 2023 · 被引用 68 次
- AutoVerus: Automated Proof Generation for Rust CodeChenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Jianan Yao 等OOPSLA 2025 · 被引用 11 次
