Order Matters in the Presence of Dataset Imbalance for Multilingual Learning

In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-training on highresource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits showing that it achieves consistent improvements relative to the performance tradeoff profile of standard static weighting. We analyze under what data regimes this method is applicable and show its improvements empirically in neural machine translation (NMT) and multi-lingual language modeling. * Equal contribution † Work done as a student researcher at Google. 37th Conference on Neural Information Processing Systems (NeurIPS 2023).