Lune

ACL Findings2025

CODEMENV: Benchmarking Large Language Models on Code Migration

Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang

Abstract

Large language models (LLMs) have shown remarkable capabilities across various software engineering tasks; however, their effectiveness in code migration-adapting code to run in different environments-remains insufficiently studied. In this work, we introduce CODEMENV: Code Migration Across Environment, a new benchmark specifically designed to assess LLMs' abilities in code migration scenarios. CODEMENV consists of 922 examples spanning 19 Python and Java packages, and covers three core tasks:

(1) identifying functions incompatible with specific versions, (2) detecting changes in function definitions, and (3) adapting code to target environments. Experimental evaluation with seven LLMs on CODEMENV yield an average pass@1 rate of 26.50%, with GPT-4O achieving the highest score at 43.84%. Key findings include: (i) LLMs tend to be more proficient with newer function versions, which aids in migrating legacy code, and (ii) LLMs sometimes exhibit logical inconsistencies by identifying function changes irrelevant to the intended migration environment. The datasets are available at https://github.com/xdshen-ai/ Benchmark-of-Code-Migration.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • CodeT5+: Open Code Large Language Models for Code Understanding and Generation cites CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Predicting LLM Output Length via Entropy-Guided RepresentationsHuanyi Xie, Yubin Chen, Liangyu Wang, Lijie Hu et al.ICLR 2026 · 12 citations
  2. GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version IncompatibilitiesDiganta Misra, Nizar Islah, Victor May, Brice Rauby et al.ACL 2026 · 8 citations
  3. Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical FallaciesXudong Shen, Li Yuan, Ye Chen, Xin Wu et al.ACL 2026

Builds on10

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 citations
  3. CodeT5+: Open Code Large Language Models for Code Understanding and GenerationYue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui et al.EMNLP 2023 · 339 citations
  4. Evaluating Large Language Models in Class-Level Code GenerationXueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang et al.ICSE 2024 · 118 citations
  5. Mass-Editing Memory in a TransformerKevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov et al.ICLR 2023 · 52 citations

Related papers

  1. An Empirical Study of Python Library Migration Using Large Language ModelsMohayeminul Islam, Ajay Kumar Jha, May Mahmoud, Ildar Akhmetov et al.ASE 2025 · 2 citations
  2. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code RepositoriesJia Li, Ge Li, Yunfei Zhao, Yongmin Li et al.ACL Findings 2024 · 44 citations
  3. DSCodeBench: A Realistic Benchmark for Data Science Code GenerationShuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun et al.AAAI 2026 · 10 citations
  4. Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'Shanchao Liang, Nan Jiang, Yiran Hu, Lin TanACL 2025 · 9 citations
  5. RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code GenerationXiao Yu, Haoxuan Chen, Lei Liu, Xing Hu et al.ASE 2025
Dusk over the sea between two cliffs drawn in fine vertical lines