ACL Findings2024
OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, Dacheng Tao
Abstract
Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favour of functional programming (FP), e.g., HumanEval and MBPP. To address this, ❶ our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. ❷ We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k metric. ❸ Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field. Our benchmark and scripts are publicly released at: https: //github.com/alphadl/OOP-eval.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models cites Execution-Based Evaluation for Open-Domain Code Generation
- JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
- AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation cites JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models
- Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions cites CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions cites Efficient Memory Management for Large Language Model Serving with PagedAttention
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions cites Execution-Based Evaluation for Open-Domain Code Generation
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
- Execution-Based Evaluation for Open-Domain Code Generation cites CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation cites CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct cites CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Cited by top-tier papers6
- The Power of Noise: Redefining Retrieval for RAG SystemsFlorin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice et al.SIGIR 2024 · 212 citations
- JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language ModelsJialun Cao, Zhiyong Chen, Jiarong Wu, Shing-Chi Cheung et al.ASE 2024 · 9 citations
- Code2Bench: Scaling Source and Rigor for Dynamic Benchmark ConstructionZhe Zhang, Runlin Liu, Aishan Liu, Xingyu Liu et al.ICLR 2026 · 4 citations
- AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet AdaptationTanghaoran Zhang, Xinjun Mao, Shangwen Wang, Yuxin Zhao et al.ASE 2025 · 1 citation
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex InstructionsTerry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu et al.ICLR 2025
Builds on5
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program SynthesisErik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu et al.ICLR 2023 · 234 citations
- Execution-Based Evaluation for Open-Domain Code GenerationZhiruo Wang, Shuyan Zhou, Daniel Fried, Graham NeubigEMNLP Findings 2023 · 20 citations
Related papers
- PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMsAnkit Yadav, Himanshu Beniwal, Mayank SinghEMNLP Findings 2024 · 12 citations
- AutoCodeBench: Large Language Models are Automatic Code Benchmark GeneratorsChangzhi Zhou, Ao Liu, Yuchi Deng, Zhiying Zeng et al.ICLR 2026 · 27 citations
- CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and GenerationWeixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li et al.ACL 2024
- Evaluating Large Language Models in Class-Level Code GenerationXueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang et al.ICSE 2024 · 118 citations
- McEval: Massively Multilingual Code EvaluationLinzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin et al.ICLR 2025 · 1 citation
