EMNLP Findings2024
One-to-many testing for code generation from (just) natural language
Mansi Uniyal, Mukul Singh, Gust Verbruggen, Sumit Gulwani, Vu Le
Abstract
MBPP is a popular dataset for evaluating the task of code generation from natural language. Despite its popularity, there are three problems:
(1) it relies on providing test cases to generate the right signature, (2) there is poor alignment between instruction and evaluation test cases, and (3) contamination of the exact phrasing being present in training datasets. We adapt MBPP to emphasize on generating code from just natural language by (1) removing ambiguity about the semantics of the task from the descriptions, and (2) evaluating generated code on multiple sets of assertions to account for ambiguity in the syntax. We compare popular open and closed weight models on the original (MBPP) and adapted (MBUPP) datasets.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
- PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMsAnkit Yadav, Himanshu Beniwal, Mayank SinghEMNLP Findings 2024 · 12 citations
- On Leakage of Code Generation Evaluation DatasetsAlexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone et al.EMNLP Findings 2024 · 12 citations
- NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User QueriesShudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng et al.ACL Findings 2024 · 11 citations
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language ModelsMartin Riddell, Ansong Ni, Arman CohanACL 2024
Related papers
- When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task DescriptionsMaya Larbi, Amal Akli, Mike Papadakis, Rihab Bouyousfi et al.ICSE 2026
- Multi-lingual Evaluation of Code Generation ModelsBen Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li et al.ICLR 2023 · 28 citations
- DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code GenerationWenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang et al.ACL Findings 2025 · 6 citations
- ReCode: Robustness Evaluation of Code Generation ModelsShiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang et al.ACL 2023 · 32 citations
- CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement LearningHung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese et al.NeurIPS 2022 · 571 citations
