EMNLP Findings2020
Extending Multilingual BERT to Low-Resource Languages
Zihan Wang, Karthikeyan K, Stephen Mayhew, Dan Roth
Abstract
Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. However, this success is focused only on the top 104 languages in Wikipedia it was trained on. In this paper, we propose a simple but effective approach to extend M-BERT (E-MBERT) so it can benefit any new language, and show that our approach aids languages that are already in M-BERT as well. We perform an extensive set of experiments with Named Entity Recognition (NER) on 27 languages, only 16 of which are in M-BERT, and show an average increase of about 6% F 1 on M-BERT languages and 23% F 1 increase on new languages. We release models and code at 1 .
Ask Lune
Ask anything. The answer quotes the paper.
Lune reads all of it and the papers around it, then shows you the passages behind every answer.
Lune answers from
the full paper · 22 top-tier papers that cite it · related work
Free to start. No credit card required.
Cited by top-tier papers22
- Expanding Pretrained Models to Thousands More Languages via Lexicon-based AdaptationXinyi Wang, Sebastian Ruder, Graham NeubigACL 2022 · 73 citations
- Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AIHoujiang Liu, Anubrata Das, Alexander Boltz, Didi Zhou et al.CSCW 2024 · 23 citations
- BLOOM+1: Adding Language Support to BLOOM for Zero-Shot PromptingZheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji et al.ACL 2023 · 20 citations
- Improving Low-Resource Languages in Pre-Trained Multilingual Language ModelsViktor Hangya, Hossain Shaikh Saadi, Alexander FraserEMNLP 2022 · 17 citations
- One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual TokenizersDiana Abagyan, Alejandro Salamanca, Andrés Felipe Cruz-Salinas, Kris Cao et al.ACL 2026 · 11 citations
Builds on1
Related papers
- Cross-Lingual Transfer in Zero-Shot Cross-Language Entity LinkingElliot Schumacher, James Mayfield, Mark DredzeACL Findings 2021 · 4 citations
- Cross-lingual Alignment Methods for Multilingual BERT: A Comparative StudySaurabh Kulshreshtha, José Luis Redondo García, Ching-Yun ChangEMNLP Findings 2020 · 37 citations
- Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine TranslationAditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Ari et al.AAAI 2020 · 74 citations
- Sources of Transfer in Multilingual Named Entity RecognitionDavid Mueller, Nicholas Andrews, Mark DredzeACL 2020 · 2 citations
- Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained EncodersGuanhua Chen, Shuming Ma, Yun Chen, Li Dong et al.EMNLP 2021 · 30 citations
