ACL Findings2025
CODEMENV: Benchmarking Large Language Models on Code Migration
Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang
Abstract
Large language models (LLMs) have shown remarkable capabilities across various software engineering tasks; however, their effectiveness in code migration-adapting code to run in different environments-remains insufficiently studied. In this work, we introduce CODEMENV: Code Migration Across Environment, a new benchmark specifically designed to assess LLMs' abilities in code migration scenarios. CODEMENV consists of 922 examples spanning 19 Python and Java packages, and covers three core tasks:
(1) identifying functions incompatible with specific versions, (2) detecting changes in function definitions, and (3) adapting code to target environments. Experimental evaluation with seven LLMs on CODEMENV yield an average pass@1 rate of 26.50%, with GPT-4O achieving the highest score at 43.84%. Key findings include: (i) LLMs tend to be more proficient with newer function versions, which aids in migrating legacy code, and (ii) LLMs sometimes exhibit logical inconsistencies by identifying function changes irrelevant to the intended migration environment. The datasets are available at https://github.com/xdshen-ai/ Benchmark-of-Code-Migration.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation cites CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Cited by top-tier papers3
- Predicting LLM Output Length via Entropy-Guided RepresentationsHuanyi Xie, Yubin Chen, Liangyu Wang, Lijie Hu et al.ICLR 2026 · 12 citations
- GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version IncompatibilitiesDiganta Misra, Nizar Islah, Victor May, Brice Rauby et al.ACL 2026 · 8 citations
- Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical FallaciesXudong Shen, Li Yuan, Ye Chen, Xin Wu et al.ACL 2026
Builds on10
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 citations
- CodeT5+: Open Code Large Language Models for Code Understanding and GenerationYue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui et al.EMNLP 2023 · 339 citations
- Evaluating Large Language Models in Class-Level Code GenerationXueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang et al.ICSE 2024 · 118 citations
- Mass-Editing Memory in a TransformerKevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov et al.ICLR 2023 · 52 citations
Related papers
- An Empirical Study of Python Library Migration Using Large Language ModelsMohayeminul Islam, Ajay Kumar Jha, May Mahmoud, Ildar Akhmetov et al.ASE 2025 · 2 citations
- DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code RepositoriesJia Li, Ge Li, Yunfei Zhao, Yongmin Li et al.ACL Findings 2024 · 44 citations
- DSCodeBench: A Realistic Benchmark for Data Science Code GenerationShuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun et al.AAAI 2026 · 10 citations
- Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'Shanchao Liang, Nan Jiang, Yiran Hu, Lin TanACL 2025 · 9 citations
- RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code GenerationXiao Yu, Haoxuan Chen, Lei Liu, Xing Hu et al.ASE 2025
