EMNLP Findings2023
Multilingual Lottery Tickets to Pretrain Language Models
Abstract
The curse of multilinguality in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages, especially when the capacity is limited. While increasing the capacity may appear intuitive for overcoming this curse, it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping the capacity per each language more or less the same. Specifically, we first scale the model to reduce interference, then search for a perlanguage subnetwork, or a lottery ticket, with comparable performance to the full model. According to lottery ticket hypothesis, this scalethen-find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters to keep the parameters reduced. Finally, to avoid the cost of multiple retraining for searching multilingual tickets, we explore zeroshot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the inference cost of models for each languages, while boosting the performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is publicly available.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Neural Architecture Search without Training cites Pruning neural networks without any data by iteratively conserving synaptic flow
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation cites Unsupervised Cross-lingual Representation Learning at Scale
- IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators cites Unsupervised Cross-lingual Representation Learning at Scale
Cited by top-tier papers1
Builds on22
- Gradient Surgery for Multi-Task LearningTianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine et al.NeurIPS 2020 · 2,261 citations
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralisationJunjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig et al.ICML 2020 · 1,132 citations
- Pruning neural networks without any data by iteratively conserving synaptic flowHidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, Surya GanguliNeurIPS 2020 · 884 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Neural Architecture Search without TrainingJoe Mellor, Jack Turner, Amos Storkey, Elliot J. CrowleyICML 2021 · 477 citations
Related papers
- Discovering Language-neutral Sub-networks in Multilingual Language ModelsNegar Foroutan, Mohammadreza Banaei, Rémi Lebret, Antoine Bosselut et al.EMNLP 2022 · 9 citations
- Learning Language Specific Sub-network for Multilingual Machine TranslationZehui Lin, Liwei Wu, Mingxuan Wang, Lei LiACL 2021
- Finding the Dominant Winning Ticket in Pre-Trained Language ModelsZhuocheng Gong, Di He, Yelong Shen, Tie-Yan Liu et al.ACL Findings 2022 · 6 citations
- Composable Sparse Fine-Tuning for Cross-Lingual TransferAlan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan VulicACL 2022
- Exploring Lottery Prompts for Pre-trained Language ModelsYulin Chen, Ning Ding, Xiaobin Wang, Shengding Hu et al.ACL 2023 · 4 citations
