EMNLP Findings2025
DIPLomA: Efficient Adaptation of Instructed LLMs to Low-Resource Languages via Post-Training Delta Merging
Abstract
This paper investigates how open-weight instruction-tuned large language models (LLMs) can be efficiently adapted to low-resource languages without requiring costly large-scale post-training. We introduce DIPLomA (Decoupled Instruction-Preserving Language Adaptation), a lightweight delta-based transfer strategy that provides a practical and effective solution for this scenario. DIPLomA decouples language adaptation from post-training alignment by first continually pre-training a foundational LLM on a modest amount of monolingual target-language data while anchoring on English replay, and then injecting instruction-following capabilities via delta-based weight merging from the instructed counterpart of the base LLM. We evaluate DIPLomA on Basque and validate its generality on Welsh and Swahili, demonstrating consistent and substantial gains in instruction-following, linguistic proficiency, and safety. Compared to strong baselines, our method achieves average relative improvements of 50 points in Basque, 63 in Welsh, and 51 in Swahili, while preserving the original model's multilingual performance. These results highlight DIPLomA as an effective, resource-efficient strategy for bringing high-quality instruction alignment to underrepresented languages at scale.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on6
- ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations
- Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New LanguagesShih-Cheng Huang, Pin-Zu Li, Yu-Chi Hsu, Kuang-Ming Chen et al.ACL 2024 · 6 citations
- Latxa: An Open Language Model and Evaluation Suite for BasqueJulen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe et al.ACL 2024 · 5 citations
- Knowledge Fusion By Evolving Weights of Language ModelsGuodong Du, Jing Li, Hanting Liu, Runhua Jiang et al.ACL Findings 2024 · 5 citations
- ParamΔ for Direct Mixing: Post-Train Large Language Model At Zero CostSheng Cao, Mingrui Wu, Karthik Prasad, Yuandong Tian et al.ICLR 2025
Related papers
- SAFT: Safety-Preserving Adaptation via Fine-Tuning Transfer for Large Language ModelsZhiwen Ruan, Yan Yang, Zhuocheng Liang, Yun Chen et al.KDD 2026
- Mitigating Catastrophic Forgetting in Language Transfer via Model MergingAnton Alexandrov, Veselin Raychev, Mark Niklas Müller, Ce Zhang et al.EMNLP Findings 2024 · 6 citations
- GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language ModelsZhiwen Ruan, Yichao Du, Jianjie Zheng, Longyue Wang et al.ACL 2026
- Multilingual Instruction Tuning With Just a Pinch of MultilingualityUri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor et al.ACL Findings 2024 · 10 citations
- Exploring Two-Phase Continual Instruction Fine-tuning for Multilingual Adaptation in Large Language ModelsDivyanshu Aggarwal, Sankarshan Damle, Navin Goyal, Satya Lokam et al.ACL Findings 2026
