LangBridge: Multilingual Reasoning Without Multilingual Supervision

We introduce LANGBRIDGE, a zero-shot approach to adapt language models for multilingual reasoning tasks without multilingual supervision. LANGBRIDGE operates by "bridging" two models, each specialized in different aspects: (1) one specialized in understanding multiple languages (e.g., mT5 encoder) and ( 2 ) one specialized in reasoning (e.g., Orca 2). LANG-BRIDGE connects the two models by introducing minimal trainable parameters between them. Despite utilizing only English data for training, LANGBRIDGE considerably enhances the performance of language models on low-resource languages across mathematical reasoning, coding, and logical reasoning. Our analysis suggests that the efficacy of LANGBRIDGE stems from the language-agnostic characteristics of multilingual representations. We publicly release our code and models. 1 2023) and Orca 2 (Mitra et al., 2023), which have 040 undergone continuous domain-specific adaptation 041 from Llama 2 (Touvron et al., 2023b). These spe-042 cialized, domain-specific datasets are typically in 043 English, complicating multilingual support for the 044 underlying LM.