Lune

NeurIPS2024Top-tier venue

MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages

Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, Fei Yuan

Abstract

Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively 2 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering cites Long Chain-of-Thought Reasoning Across Languages
  • Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality cites MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
  • Long Chain-of-Thought Reasoning Across Languages cites LoRA: Low-Rank Adaptation of Large Language Models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Training language models to follow instructions with human feedback
  • MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Long Chain-of-Thought Reasoning Across LanguagesJosh Barua, Seun Eisape, Kayo Yin, Alane SuhrICLR 2026 · 21 citations
  2. Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation EngineeringQiming Li, Xiaocheng Feng, Yixuan Ma, Ruihan Chen et al.ACL 2026 · 4 citations
  3. Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language UnderstandingKhanh-Tung Tran, Barry O'Sullivan, Hoang D. NguyenAAAI 2026 · 4 citations
  4. Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible MultilingualityMengyu Bu, Yang FengACL 2026 · 2 citations
  5. Toward Robust Multilingual Adaptation of LLMs for Low-Resource LanguagesHaolin Li, Haipeng Zhang, Mang Li, Yaohua Wang et al.ICML 2026

Builds on19

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  5. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations

Related papers

  1. Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language ModelsLucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou et al.ICLR 2025
  2. Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?Yijie Hu, Zihao Zhou, Kaizhu Huang, Xiaowei Huang et al.NeurIPS 2025 · 3 citations
  3. LangBridge: Multilingual Reasoning Without Multilingual SupervisionDongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim et al.ACL 2024
  4. When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual ReasonersWeixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu et al.NeurIPS 2025 · 20 citations
  5. Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?Deokhyung Kang, Seonjeong Hwang, Daehui Kim, Hyounghun Kim et al.ACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines