Lune

ACL Findings2021

More Parameters? No Thanks!

Zeeshan Khan, Kartheek Akella, Vinay P. Namboodiri, C. V. Jawahar

Abstract

This work studies the long-standing problems of model capacity and negative interference in multilingual neural machine translation (MNMT). We use network pruning techniques and observe that pruning 50-70% of the parameters from a trained MNMT model results only in a 0.29-1.98 drop in the BLEU score. Suggesting that there exist large redundancies in MNMT models. These observations motivate us to use the redundant parameters and counter the interference problem efficiently. We propose a novel adaptation strategy, where we iteratively prune and retrain the redundant parameters of an MNMT to improve bilingual representations while retaining the multilinguality. Negative interference severely affects high resource languages, and our method alleviates it without any additional adapter modules. Hence, we call it parameterfree adaptation strategy, paving way for the efficient adaptation of MNMT. We demonstrate the effectiveness of our method on a 9 language MNMT trained on TED talks, and report an average improvement of +1.36 on high resource pairs. Code will be released here.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on1

  1. Improving Massively Multilingual Neural Machine Translation and Zero-Shot TranslationBiao Zhang, Philip Williams, Ivan Titov, Rico SennrichACL 2020 · 213 citations

Related papers

  1. Gradient-based Gradual Pruning for Language-Specific Multilingual Neural Machine TranslationDan He, Minh-Quang Pham, Thanh-Le Ha, Marco TurchiEMNLP 2023 · 2 citations
  2. Causes and Cures for Interference in Multilingual TranslationUri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy et al.ACL 2023 · 6 citations
  3. Multilingual Machine Translation with Hyper-AdaptersChristos Baziotis, Mikel Artetxe, James Cross, Shruti BhosaleEMNLP 2022 · 20 citations
  4. Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation ModelYeskendir Koishekenov, Alexandre Berard, Vassilina NikoulinaACL 2023 · 11 citations
  5. Finding Sparse Structures for Domain Specific Neural Machine TranslationJianze Liang, Chengqi Zhao, Mingxuan Wang, Xipeng Qiu et al.AAAI 2021 · 33 citations
Dusk over the sea between two cliffs drawn in fine vertical lines