Lune

ACL Findings2024

Zero-shot Cross-lingual Alignment for Embedding Initialization

Xi Ai, Zhiyong Huang

Abstract

For multilingual training, we present CrossInit, an initialization method that initializes embeddings into similar geometrical structures across languages in an unsupervised manner. CrossInit leverages a common cognitive linguistic mechanism called Zipf's law, which indicates that similar concepts across languages have similar word ranks or frequencies in their monolingual corpora. Instead of considering point-to-point alignments based on ranks, CrossInit considers the same span of consecutive ranks in each language as the Positive pairs for alignment, while others out of the span are used as Negative pairs. CrossInit then employs Contrastive Learning to iteratively refine randomly initialized embeddings for similar geometrical structures across languages. Our experiments on Unsupervised NMT, XNLI, and MLQA showed substantial gains in lowresource and dissimilar languages after applying CrossInit 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Emerging Cross-lingual Structure in Pretrained Language Models cites Cross-Lingual Ability of Multilingual BERT: An Empirical Study
  • Unsupervised Cross-lingual Representation Learning at Scale cites Emerging Cross-lingual Structure in Pretrained Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  3. Cross-Lingual Ability of Multilingual BERT: An Empirical StudyKarthikeyan K, Zihan Wang, Stephen Mayhew, Dan RothICLR 2020 · 378 citations
  4. Emerging Cross-lingual Structure in Pretrained Language ModelsAlexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer et al.ACL 2020 · 210 citations
  5. Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for LittleKoustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau et al.EMNLP 2021 · 177 citations

Related papers

  1. Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context AnchoringAitor Ormazabal, Mikel Artetxe, Aitor Soroa, Gorka Labaka et al.ACL 2021
  2. FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual ModelsKonstantin Dobler, Gerard de MeloEMNLP 2023 · 4 citations
  3. English Contrastive Learning Can Learn Universal Cross-lingual Sentence EmbeddingsYau-Shian Wang, Ashley Wu, Graham NeubigEMNLP 2022 · 18 citations
  4. Linear Cross-Lingual Mapping of Sentence EmbeddingsOleg V. Vasilyev, Fumika Isono, John BohannonACL Findings 2024 · 1 citation
  5. Normalization of Language Embeddings for Cross-Lingual AlignmentPrince Osei Aboagye, Yan Zheng, Chin-Chia Michael Yeh, Junpeng Wang et al.ICLR 2022 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines