ACL Findings2021
Multi-Granularity Contrasting for Cross-Lingual Pre-Training
Shicheng Li, Pengcheng Yang, Fuli Luo, Jun Xie
Abstract
Cross-lingual pre-training aims at providing effective prior representations for the inputs from multiple languages. With the modeling of bidirectional contexts, recently prevalent language modeling approaches such as XLM achieve better performance than traditional methods based on embedding alignment, which strives to assign similar vector representations to semantic-equivalent units. However, such approaches like XLM capture cross-lingual information based solely on shared BPE vocabulary, resulting in the absence of fine-grained supervision induced by embedding alignment. Inheriting the advantages of the above two paradigms, this work presents a multi-granularity contrasting framework, namely MGC, to learn languageuniversal representations. While predicting the masked words based on bidirectional contexts, the proposal also encodes semantic equivalents from different languages into similar representations to introduce more finegrained and explicit cross-lingual information. Two effective contrasting strategies are further proposed, which can be built upon semantic units of multiple granularities covering words, span, and sentences. Extensive experiments demonstrate that our approach can achieve significant performance gains in various downstream tasks, including machine translation and cross-lingual language understanding.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on3
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Cross-Lingual Natural Language Generation via Pre-TrainingZewen Chi, Li Dong, Furu Wei, Wenhui Wang et al.AAAI 2020 · 142 citations
- Alternating Language Modeling for Cross-Lingual Pre-TrainingJian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu et al.AAAI 2020 · 94 citations
Related papers
- On Learning Universal Representations Across LanguagesXiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing et al.ICLR 2021 · 93 citations
- Structural Contrastive Pretraining for Cross-Lingual ComprehensionNuo Chen, Linjun Shou, Tengtao Song, Ming Gong et al.ACL Findings 2023 · 2 citations
- Multilingual Pre-training with Self-supervision from Global Co-occurrence InformationXi Ai, Bin FangACL Findings 2023 · 1 citation
- M2-VLP: Enhancing Multilingual Vision-Language Pre-Training via Multi-Grained AlignmentAhtamjan Ahmat, Lei Wang, Yating Yang, Bo Ma et al.WWW 2025 · 2 citations
- VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and GenerationFuli Luo, Wei Wang, Jiahao Liu, Yijia Liu et al.ACL 2021
