ACL Findings2026
Cross-lingual Matryoshka Representation Learning across Speech and Text
Yaya Sy, Dioula Doucouré, Christophe Cerisara, Irina Illina
Abstract
Speakers of under-represented languages face both a language barrier, as most online knowledge is in a few dominant languages, and a modality barrier, since information is largely text-based while many languages are primarily oral. We address this for French-Wolof by training the first bilingual speechtext Matryoshka embedding model, enabling efficient retrieval of French text from Wolof speech queries without relying on a costly ASR-translation pipelines. We introduce largescale data curation pipelines and new benchmarks, compare modeling strategies, and show that modality fusion within a frozen text Matryoshka model performs best. Although trained only for retrieval, the model generalizes well to other tasks, such as speech intent detection, indicating the learning of general semantic representations. Finally, we analyze cost-accuracy trade-offs across Matryoshka dimensions and ranks, showing that information is concentrated only in a few components, suggesting potential for efficiency improvements. The code, the models, and the datasets will be released: https://github. com/soynade-research/oolel-embed
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Matryoshka Representation Learning cites Learning Transferable Visual Models From Natural Language Supervision
- Matryoshka Query Transformer for Large Vision-Language Models cites Visual Instruction Tuning
- Matryoshka Query Transformer for Large Vision-Language Models cites Matryoshka Representation Learning
- Matryoshka Query Transformer for Large Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
Builds on9
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Matryoshka Representation LearningAditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford et al.NeurIPS 2022 · 364 citations
- Matryoshka Query Transformer for Large Vision-Language ModelsWenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath et al.NeurIPS 2024 · 58 citations
Related papers
- ML-Embed: Inclusive and Efficient Embeddings for a Multilingual WorldZiyin Zhang, Zihan Liao, Hang Yu, Peng Di et al.ICML 2026
- Towards Text-Image Interleaved RetrievalXin Zhang, Ziqi Dai, Yongqi Li, Yanzhao Zhang et al.ACL 2025 · 1 citation
- PromptST: Abstract Prompt Learning for End-to-End Speech TranslationTengfei Yu, Liang Ding, Xuebo Liu, Kehai Chen et al.EMNLP 2023 · 3 citations
- WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMChangli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang et al.ICLR 2026 · 9 citations
- Beyond Contrastive Learning: A Variational Generative Model for Multilingual RetrievalJohn Wieting, Jonathan H. Clark, William W. Cohen, Graham Neubig et al.ACL 2023 · 3 citations
