Lune

ASE2025Top-tier venue

An Empirical Study of Python Library Migration Using Large Language Models

Mohayeminul Islam, Ajay Kumar Jha, May Mahmoud, Ildar Akhmetov, Sarah Nadi

Abstract

Library migration is the process of replacing one library with another library that provides similar functionality. Manual library migration is time consuming and error prone, as it requires developers to understand the APIs of both libraries, map them, and perform the necessary code transformations. Large Language Models (LLMs) are shown to be effective at generating and transforming code as well as finding similar code, which are necessary upstream tasks for library migration. Such capabilities suggest that LLMs may be suitable for library migration. Accordingly, this paper investigates the effectiveness of LLMs for migration between Python libraries. We evaluate three LLMs, LLama 3.1, GPT-4o mini, and GPT-4o on PyMigBench, where we migrate 321 real-world library migrations that include 2,989 migration-related code changes. To measure correctness, we (1) compare the LLM’s migrated code with the developers’ migrated code in the benchmark and (2) run the unit tests available in the client repositories. We find that LLama 3.1, GPT-4o mini, and GPT-4o correctly migrate 89%, 89%, and 94% of the migration-related code changes, respectively. We also find that 36%, 52% and 64% of the LLama 3.1, GPT-4o mini, and GPT-4o migrations pass the same tests that passed in the developer’s migration. To ensure the LLMs are not reciting the migrations, we also evaluate them on 10 new repositories where the migration never happened. Overall, our results suggest that LLMs can be effective in migrating code between libraries, but we also identify some open challenges.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair cites Automated Program Repair in the Era of Large Pre-trained Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. Automated Program Repair in the Era of Large Pre-trained Language ModelsChunqiu Steven Xia, Yuxiang Wei, Lingming ZhangICSE 2023 · 321 citations
  2. Using an LLM to Help With Code UnderstandingDaye Nam, Andrew Macvean, Vincent J. Hellendoorn, Bogdan Vasilescu et al.ICSE 2024 · 264 citations
  3. Keep me Updated: An Empirical Study of Third-Party Library Updatability on AndroidErik Derr, Sven Bugiel, Sascha Fahl, Yasemin Acar et al.CCS 2017 · 196 citations
  4. Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program RepairYuxiang Wei, Chunqiu Steven Xia, Lingming ZhangFSE 2023 · 111 citations
  5. SOAR: A Synthesis Approach for Data Science API RefactoringAnsong Ni, Daniel Ramos, Aidan Z. H. Yang, Inês Lynce et al.ICSE 2021 · 27 citations

Related papers

  1. Pig: Leveraging Large Language Models for Python Library MigrationsMiryeong Kang, Wonseok Oh, Gabin An, Hakjoo OhFSE 2026
  2. CODEMENV: Benchmarking Large Language Models on Code MigrationKeyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang et al.ACL Findings 2025 · 5 citations
  3. Characterizing Python Library MigrationsMohayeminul Islam, Ajay Kumar Jha, Ildar Akhmetov, Sarah NadiFSE 2024 · 3 citations
  4. LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code CompletionChong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng et al.ICSE 2025 · 3 citations
  5. TransLibEval: Demystify Large Language Models' Capability in Third-Party Library-Targeted Code TranslationPengyu Xue, Kunwu Zheng, Zhen Yang, Yifei Pei et al.FSE 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines