X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale

Large language models (LLMs) have achieved remarkable success across various NLP tasks with a focus on English due to English-centric pre-training and limited multilingual data. In this work, we focus on the problem of translation, and while some multilingual LLMs claim to support for hundreds of languages, models often fail to provide high-quality responses for mid-and low-resource languages, leading to imbalanced performance heavily skewed in favor of highresource languages. We introduce X-ALMA, a model designed to ensure toptier performance across 50 diverse languages, regardless of their resource levels. X-ALMA surpasses state-of-the-art open-source multilingual LLMs, such as Aya-101 ( Üstün et al., 2024) and Aya-23 (Aryabumi et al., 2024), in every single translation direction on the FLORES-200 and WMT'23 test datasets according to COMET-22. This is achieved by plug-and-play language-specific module architecture to prevent language conflicts during training and a carefully designed training regimen with novel optimization methods to maximize the translation performance. After the final stage of training regimen, our proposed Adaptive-Rejection Preference Optimization (ARPO) surpasses existing preference optimization methods in translation tasks. 1 ‡ Work done at Microsoft. 1 Code is released at https://github.com/fe1ixxu/ALMA . Models and Dataset are released at https://huggingface/X-ALMA .