ACL Findings2021
Minimax and Neyman-Pearson Meta-Learning for Outlier Languages
Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivastava, Siva Reddy, Anders Søgaard
Abstract
Model-agnostic meta-learning (MAML) has been recently put forth as a strategy to learn resource-poor languages in a sample-efficient fashion. Nevertheless, the properties of these languages are often not well represented by those available during training. Hence, we argue that the i.i.d. assumption ingrained in MAML makes it ill-suited for cross-lingual NLP. In fact, under a decision-theoretic framework, MAML can be interpreted as minimising the expected risk across training languages (with a uniform prior), which is known as Bayes criterion. To increase its robustness to outlier languages, we create two variants of MAML based on alternative criteria: Minimax MAML reduces the maximum risk across languages, while Neyman-Pearson MAML constrains the risk in each language to a maximum threshold. Both criteria constitute fully differentiable two-player games. In light of this, we propose a new adaptive optimiser solving for a local approximation to their Nash equilibrium. We evaluate both model variants on two popular NLP tasks, part-of-speech tagging and question answering. We report gains for their average and minimum performance across low-resource languages in zeroand few-shot settings, compared to joint multisource transfer and vanilla MAML. The code for our experiments is available at https:// github.com/rahular/robust-maml.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages cites XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages cites XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
Cited by top-tier papers5
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and LanguagesEmanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy et al.ICML 2022 · 71 citations
- Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic LanguagesSumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal et al.ACL 2023 · 41 citations
- Empowering Cross-lingual Behavioral Testing of NLP Models with Typological FeaturesEster Hlavnova, Sebastian RuderACL 2023 · 2 citations
- Challenges and Strategies in Cross-Cultural NLPDaniel Hershcovich, Stella Frank, Heather C. Lent, Miryam de Lhoneux et al.ACL 2022
- In-context Mixing (ICM): Code-mixed Prompts for Multilingual LLMsBhavani Shankar, Preethi Jyothi, Pushpak BhattacharyyaACL 2024
Builds on9
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralisationJunjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig et al.ICML 2020 · 1,132 citations
- What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?Chi Jin, Praneeth Netrapalli, Michael I. JordanICML 2020 · 381 citations
- Balancing Training for Multilingual Neural Machine TranslationXinyi Wang, Yulia Tsvetkov, Graham NeubigACL 2020 · 74 citations
- Enhanced Meta-Learning for Cross-Lingual Named Entity Recognition with Minimal ResourcesQianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen et al.AAAI 2020 · 72 citations
- Task-Robust Model-Agnostic Meta-LearningLiam Collins, Aryan Mokhtari, Sanjay ShakkottaiNeurIPS 2020 · 66 citations
Related papers
- Meta-Learning for Fast Cross-Lingual Adaptation in Dependency ParsingAnna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos et al.ACL 2022 · 17 citations
- Multilingual Meta-Distillation Alignment for Semantic RetrievalMeryem M'hamdi, Jonathan May, Franck Dernoncourt, Trung Bui et al.SIGIR 2024
- Zero-Shot Cross-Lingual Transfer with Meta LearningFarhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, Isabelle AugensteinEMNLP 2020 · 12 citations
- A Nested Bi-level Optimization Framework for Robust Few Shot LearningKrishnaTeja Killamsetty, Changbin Li, Chen Zhao, Feng Chen et al.AAAI 2022 · 12 citations
- Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource LanguagesLinjuan Wu, Zongyi Guo, Baoliang Cui, Haihong Tang et al.EMNLP Findings 2023 · 1 citation
