ACL Findings2025
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries
Haruki Sakajo, Yusuke Ide, Justin Vasselli, Yusuke Sakai, Yingtao Tian, Hidetaka Kamigaito, Taro Watanabe
Abstract
Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including lowresource languages. Existing approaches that utilize monolingual or parallel corpora face challenges when applied to languages with limited resources. In this work, we propose a simple yet effective vocabulary transfer method that utilizes bilingual dictionaries, which are available for many languages, thanks to descriptive linguists. Our proposed method leverages a property of BPE tokenizers where removing a subword from the vocabulary causes a fallback to shorter subwords. The embeddings of target subwords are estimated iteratively by progressively removing them from the tokenizer. The experimental results show that our approach outperforms existing methods for low-resource languages, demonstrating the effectiveness of a dictionary-based approach for cross-lingual vocabulary transfer 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Language Model Tokenizers Introduce Unfairness Between Languages cites Unsupervised Cross-lingual Representation Learning at Scale
- An Experimental Study on the Influence of Culture on Cross-Lingual Sentiment Transfer cites Unsupervised Cross-lingual Representation Learning at Scale
Cited by top-tier papers1
Builds on13
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Language Model Tokenizers Introduce Unfairness Between LanguagesAleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, Adel BibiNeurIPS 2023 · 301 citations
- Better & Faster Large Language Models via Multi-token PredictionFabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz et al.ICML 2024 · 286 citations
- Extending Multilingual BERT to Low-Resource LanguagesZihan Wang, Karthikeyan K, Stephen Mayhew, Dan RothEMNLP Findings 2020 · 90 citations
Related papers
- Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related LanguagesVaidehi Patil, Partha P. Talukdar, Sunita SarawagiACL 2022 · 39 citations
- BERT for Monolingual and Cross-Lingual Reverse DictionaryHang Yan, Xiaonan Li, Xipeng Qiu, Bocao DengEMNLP Findings 2020 · 18 citations
- Improving Low-Resource Languages in Pre-Trained Multilingual Language ModelsViktor Hangya, Hossain Shaikh Saadi, Alexander FraserEMNLP 2022 · 17 citations
- Subword Evenness (SuE) as a Predictor of Cross-lingual Transfer to Low-resource LanguagesOlga Pelloni, Anastassia Shaitarova, Tanja SamardzicEMNLP 2022 · 4 citations
- Improving Target-side Lexical Transfer in Multilingual Neural Machine TranslationLuyu Gao, Xinyi Wang, Graham NeubigEMNLP Findings 2020 · 4 citations
