Beyond Language Boundaries: Uncovering Programming Language Families for Code Language Models
Shangbo Yun, Xiaodong Gu, Jianghong Huang, Beijun Shen
Abstract
The rapid proliferation of diverse programming languages presents both opportunities and challenges for developing multilingual code LLMs. While existing techniques often train code LLMs by simply aggregating multilingual code data, few explore the deeper relationships between programming languages and how such relationships can be utilized to optimize the training and inference of code LLMs. In this work, we investigate two fundamental questions: (1) What are the deep linguistic relationships among programming languages? and (2) How can these relationships be leveraged to improve multilingual code LLMs? We propose an embedding-based framework to uncover the latent families of programming languages. Our approach begins by defining 21 primary linguistic features of programming languages, such as variable definition, control structures, and method declarations, and then employs LLMs to generate feature-aligned code samples across multiple languages. By embedding these semantically parallel code snippets from 19 languages, we construct a similarity matrix and perform hierarchical clustering to uncover inherent language relationships. Our analysis reveals clear hierarchical structures among programming languages. Closely related languages form well-defined clusters (e.g., C, C++, Java, and Swift group together), while Go exhibits as a central language with the highest cross-language similarity. Building on the uncovered language families, we propose three strategies to enhance multilingual LLM training: transfer learning across linguistically related languages, linguistic proximity-guided curriculum learning, and centroid-based intermediary code translation. Experiments on four code intelligence tasks demonstrate that our methods significantly improve multilingual LLM performance. This work offers a universal perspective on programming languages and advances more effective strategies for multilingual code LLM training.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on12
- Unsupervised Translation of Programming LanguagesBaptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, Guillaume LampleNeurIPS 2020 · 606 citations
- Cross-Lingual Ability of Multilingual BERT: An Empirical StudyKarthikeyan K, Zihan Wang, Stephen Mayhew, Dan RothICLR 2020 · 378 citations
- Cross-Domain Deep Code Search with Meta LearningYitian Chai, Hongyu Zhang, Beijun Shen, Xiaodong GuICSE 2022 · 42 citations
- Syntax and Domain Aware Model for Unsupervised Program TranslationFang Liu, Jia Li, Li ZhangICSE 2023 · 25 citations
- MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation LearningWeiguo Pian, Hanyu Peng, Xunzhu Tang, Tiezhu Sun et al.AAAI 2023 · 18 citations
Related papers
- Neuron-Guided Interpretation of Code LLMs: Where, Why, and How?Zhe Yin, Xiaodong Gu, Beijun ShenFSE 2026
- Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating CodeRangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar et al.ICSE 2024 · 96 citations
- How Programming Concepts and Neurons Are Shared in Code Language ModelsAmir Hossein Kargaran, Yihong Liu, François Yvon, Hinrich SchützeACL Findings 2025
- The Struggles of LLMs in Cross-Lingual Code Clone DetectionMicheline Bénédicte Moumoula, Abdoul Kader Kaboré, Jacques Klein, Tegawendé F. BissyandéFSE 2025 · 4 citations
- Qwen2.5-xCoder: Multi-Agent Collaboration for Multilingual Code Instruction TuningJian Yang, Wei Zhang, Yibo Miao, Shanghaoran Quan et al.ACL 2025 · 4 citations
