Lune

ACL Findings2026

Cross-lingual Matryoshka Representation Learning across Speech and Text

Yaya Sy, Dioula Doucouré, Christophe Cerisara, Irina Illina

摘要

Speakers of under-represented languages face both a language barrier, as most online knowledge is in a few dominant languages, and a modality barrier, since information is largely text-based while many languages are primarily oral. We address this for French-Wolof by training the first bilingual speechtext Matryoshka embedding model, enabling efficient retrieval of French text from Wolof speech queries without relying on a costly ASR-translation pipelines. We introduce largescale data curation pipelines and new benchmarks, compare modeling strategies, and show that modality fusion within a frozen text Matryoshka model performs best. Although trained only for retrieval, the model generalizes well to other tasks, such as speech intent detection, indicating the learning of general semantic representations. Finally, we analyze cost-accuracy trade-offs across Matryoshka dimensions and ranks, showing that information is concentrated only in a few components, suggesting potential for efficiency improvements. The code, the models, and the datasets will be released: https://github. com/soynade-research/oolel-embed

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Visual Instruction Tuning 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • Matryoshka Representation Learning 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • Matryoshka Query Transformer for Large Vision-Language Models 引用了 Visual Instruction Tuning
  • Matryoshka Query Transformer for Large Vision-Language Models 引用了 Matryoshka Representation Learning
  • Matryoshka Query Transformer for Large Vision-Language Models 引用了 Learning Transferable Visual Models From Natural Language Supervision
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper9

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh 等ICML 2021 · 47,906 次引用
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
  3. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary 等ACL 2020 · 539 次引用
  4. Matryoshka Representation LearningAditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford 等NeurIPS 2022 · 364 次引用
  5. Matryoshka Query Transformer for Large Vision-Language ModelsWenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath 等NeurIPS 2024 · 58 次引用

相关 Paper

  1. ML-Embed: Inclusive and Efficient Embeddings for a Multilingual WorldZiyin Zhang, Zihan Liao, Hang Yu, Peng Di 等ICML 2026
  2. Towards Text-Image Interleaved RetrievalXin Zhang, Ziqi Dai, Yongqi Li, Yanzhao Zhang 等ACL 2025 · 1 次引用
  3. PromptST: Abstract Prompt Learning for End-to-End Speech TranslationTengfei Yu, Liang Ding, Xuebo Liu, Kehai Chen 等EMNLP 2023 · 3 次引用
  4. WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLMChangli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang 等ICLR 2026 · 9 次引用
  5. Beyond Contrastive Learning: A Variational Generative Model for Multilingual RetrievalJohn Wieting, Jonathan H. Clark, William W. Cohen, Graham Neubig 等ACL 2023 · 3 次引用
黄昏的海面,两侧是细线勾勒的悬崖