Lune

ACL Findings2025

CODEMENV: Benchmarking Large Language Models on Code Migration

Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang

摘要

Large language models (LLMs) have shown remarkable capabilities across various software engineering tasks; however, their effectiveness in code migration-adapting code to run in different environments-remains insufficiently studied. In this work, we introduce CODEMENV: Code Migration Across Environment, a new benchmark specifically designed to assess LLMs' abilities in code migration scenarios. CODEMENV consists of 922 examples spanning 19 Python and Java packages, and covers three core tasks:

(1) identifying functions incompatible with specific versions, (2) detecting changes in function definitions, and (3) adapting code to target environments. Experimental evaluation with seven LLMs on CODEMENV yield an average pass@1 rate of 26.50%, with GPT-4O achieving the highest score at 43.84%. Key findings include: (i) LLMs tend to be more proficient with newer function versions, which aids in migrating legacy code, and (ii) LLMs sometimes exhibit logical inconsistencies by identifying function changes irrelevant to the intended migration environment. The datasets are available at https://github.com/xdshen-ai/ Benchmark-of-Code-Migration.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 3 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities 引用了 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • CodeT5+: Open Code Large Language Models for Code Understanding and Generation 引用了 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper3

  1. Predicting LLM Output Length via Entropy-Guided RepresentationsHuanyi Xie, Yubin Chen, Liangyu Wang, Lijie Hu 等ICLR 2026 · 12 次引用
  2. GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version IncompatibilitiesDiganta Misra, Nizar Islah, Victor May, Brice Rauby 等ACL 2026 · 8 次引用
  3. Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical FallaciesXudong Shen, Li Yuan, Ye Chen, Xin Wu 等ACL 2026

它引用的 Paper10

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni 等NeurIPS 2020 · 19,162 次引用
  2. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 次引用
  3. CodeT5+: Open Code Large Language Models for Code Understanding and GenerationYue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui 等EMNLP 2023 · 339 次引用
  4. Evaluating Large Language Models in Class-Level Code GenerationXueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang 等ICSE 2024 · 118 次引用
  5. Mass-Editing Memory in a TransformerKevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov 等ICLR 2023 · 52 次引用

相关 Paper

  1. An Empirical Study of Python Library Migration Using Large Language ModelsMohayeminul Islam, Ajay Kumar Jha, May Mahmoud, Ildar Akhmetov 等ASE 2025 · 2 次引用
  2. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code RepositoriesJia Li, Ge Li, Yunfei Zhao, Yongmin Li 等ACL Findings 2024 · 44 次引用
  3. DSCodeBench: A Realistic Benchmark for Data Science Code GenerationShuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun 等AAAI 2026 · 10 次引用
  4. Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'Shanchao Liang, Nan Jiang, Yiran Hu, Lin TanACL 2025 · 9 次引用
  5. RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code GenerationXiao Yu, Haoxuan Chen, Lei Liu, Xing Hu 等ASE 2025
黄昏的海面,两侧是细线勾勒的悬崖