EMNLP Findings2021
Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings
Sawsan Alqahtani, Garima Lalwani, Yi Zhang, Salvatore Romeo, Saab Mansour
Abstract
Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during finetuning to further improve multilingual contextualized representations for downstream crosslingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Concept Heterogeneity-aware Representation Steering cites Event Process Typing via Hierarchical Optimal Transport
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation cites Unsupervised Cross-lingual Representation Learning at Scale
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation cites On the Cross-lingual Transferability of Monolingual Representations
- A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT cites Multilingual Alignment of Contextual Word Representations
- Unbalanced Optimal Transport for Unbalanced Word Alignment cites A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
- On the Cross-lingual Transferability of Monolingual Representations cites Multilingual Alignment of Contextual Word Representations
Cited by top-tier papers6
- Pre-training for Speech Translation: CTC Meets Optimal TransportPhuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino et al.ICML 2023 · 33 citations
- SAKE: Steering Activations for Knowledge EditingMarco Scialanga, Thibault Laugel, Vincent Grari, Marcin DetynieckiACL 2025 · 6 citations
- Event Process Typing via Hierarchical Optimal TransportBo Zhou, Yubo Chen, Kang Liu, Jun ZhaoAAAI 2023 · 3 citations
- Unbalanced Optimal Transport for Unbalanced Word AlignmentYuki Arase, Han Bao, Sho YokoiACL 2023 · 1 citation
- Concept Heterogeneity-aware Representation SteeringLaziz Abdullaev, Noelle Y. L. Wong, Ryan Lee, Shiqi Jiang et al.ICML 2026
Builds on8
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralisationJunjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig et al.ICML 2020 · 1,132 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Multilingual Alignment of Contextual Word RepresentationsSteven Cao, Nikita Kitaev, Dan KleinICLR 2020 · 211 citations
- On the Cross-lingual Transferability of Monolingual RepresentationsMikel Artetxe, Sebastian Ruder, Dani YogatamaACL 2020 · 57 citations
- A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERTMasaaki Nagata, Katsuki Chousa, Masaaki NishinoEMNLP 2020 · 38 citations
Related papers
- Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual TransformersFélix Gaschi, Patricio Cerda, Parisa Rastin, Yannick ToussaintACL Findings 2023 · 2 citations
- On the Language Neutrality of Pre-trained Multilingual RepresentationsJindrich Libovický, Rudolf Rosa, Alexander FraserEMNLP Findings 2020 · 61 citations
- Improving Low-Resource Languages in Pre-Trained Multilingual Language ModelsViktor Hangya, Hossain Shaikh Saadi, Alexander FraserEMNLP 2022 · 17 citations
- Combining Static and Contextualised Multilingual EmbeddingsKatharina Hämmerl, Jindrich Libovický, Alexander FraserACL Findings 2022 · 11 citations
- Bilingual alignment transfers to multilingual alignment for unsupervised parallel text miningChih-chan Tien, Shane Steinert-ThrelkeldACL 2022 · 10 citations
