MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages
Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, Fei Yuan
Abstract
Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively 2 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering cites Long Chain-of-Thought Reasoning Across Languages
- Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality cites MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
- Long Chain-of-Thought Reasoning Across Languages cites LoRA: Low-Rank Adaptation of Large Language Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Training language models to follow instructions with human feedback
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Training language models to follow instructions with human feedback
Cited by top-tier papers7
- Long Chain-of-Thought Reasoning Across LanguagesJosh Barua, Seun Eisape, Kayo Yin, Alane SuhrICLR 2026 · 21 citations
- Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation EngineeringQiming Li, Xiaocheng Feng, Yixuan Ma, Ruihan Chen et al.ACL 2026 · 4 citations
- Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language UnderstandingKhanh-Tung Tran, Barry O'Sullivan, Hoang D. NguyenAAAI 2026 · 4 citations
- Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible MultilingualityMengyu Bu, Yang FengACL 2026 · 2 citations
- Toward Robust Multilingual Adaptation of LLMs for Low-Resource LanguagesHaolin Li, Haipeng Zhang, Mang Li, Yaohua Wang et al.ICML 2026
Builds on19
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
Related papers
- Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language ModelsLucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou et al.ICLR 2025
- Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?Yijie Hu, Zihao Zhou, Kaizhu Huang, Xiaowei Huang et al.NeurIPS 2025 · 3 citations
- LangBridge: Multilingual Reasoning Without Multilingual SupervisionDongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim et al.ACL 2024
- When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual ReasonersWeixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu et al.NeurIPS 2025 · 20 citations
- Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?Deokhyung Kang, Seonjeong Hwang, Daehui Kim, Hyounghun Kim et al.ACL Findings 2026
