EMNLP Findings2025
Evaluating Large Language Models for Cross-Lingual Retrieval
Longfei Zuo, Pingjun Hong, Oliver Kraus, Barbara Plank, Robert Litschko
Abstract
Multi-stage information retrieval (IR) has become a widely-adopted paradigm in search. While Large Language Models (LLMs) have been extensively evaluated as second-stage reranking models for monolingual IR, a systematic large-scale comparison is still lacking for cross-lingual IR (CLIR). Moreover, while prior work shows that LLM-based rerankers improve CLIR performance, their evaluation setup relies on lexical retrieval with machine translation (MT) for the first stage. This is not only prohibitively expensive but also prone to error propagation across stages. Our evaluation on passage-level and document-level CLIR reveals that further gains can be achieved with multilingual bi-encoders as first-stage retrievers and that the benefits of translation diminishes with stronger reranking models. We further show that pairwise rerankers based on instructiontuned LLMs perform competitively with listwise rerankers. To the best of our knowledge, we are the first to study the interaction between retrievers and rerankers in two-stage CLIR with LLMs. Our findings reveal that, without MT, current state-of-the-art rerankers fall severely short when directly applied in CLIR.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation cites Unsupervised Cross-lingual Representation Learning at Scale
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents cites Document Ranking with a Pretrained Sequence-to-Sequence Model
Cited by top-tier papers1
Builds on12
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge DistillationJianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo et al.ACL Findings 2024 · 762 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 citations
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking AgentsWeiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang et al.EMNLP 2023 · 182 citations
Related papers
- FIRST: Faster Improved Listwise Reranking with Single Token DecodingRevanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md. Arafat Sultan et al.EMNLP 2024 · 14 citations
- Query in Your Tongue: Reinforce Large Language Models with Retrievers for Cross-lingual Search Generative ExperiencePing Guo, Yue Hu, Yanan Cao, Yubing Ren et al.WWW 2024 · 4 citations
- CoRanking: Collaborative Ranking with Small and Large Ranking AgentsWenhan Liu, Xinyu Ma, Yutao Zhu, Lixin Su et al.EMNLP Findings 2025
- FreeRet: MLLMs as Training-Free RetrieversYuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li et al.ICML 2026 · 5 citations
- All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAGDan Wang, Guozhao Mo, Yafei Shi, Cheng Zhang et al.ACL 2026
