Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness

The paradigm of retrieval-augmented generated (RAG) helps mitigate hallucinations of large language models (LLMs). However, RAG also introduces biases contained within the retrieved documents. These biases can be amplified in scenarios which are multilingual and culturallysensitive, such as territorial disputes. We thus introduce BORDIRLINES, 1 a dataset of territorial disputes paired with retrieved Wikipedia documents, across 49 languages. We evaluate the cross-lingual robustness of this RAG setting by formalizing several modes for multilingual retrieval. Our experiments on several LLMs show that incorporating perspectives from diverse languages can in fact improve robustness; retrieving multilingual documents best improves response consistency and decreases geopolitical bias over RAG with purely inlanguage documents. We also consider how RAG responses utilize presented documents, finding a much wider variance in the linguistic distribution of response citations, when querying in low-resource languages. Our further analyses investigate the various aspects of a crosslingual RAG pipeline, from retrieval to document contents. We release our benchmark and code to support continued research towards equitable information access across languages. 2 * Equal second contribution, † Equal third contribution 1 BorderLines for Information Retrieval and In Real Life 2 Links for the dataset here, and for the code here. en Ceuta is an autonomous city of Spain on the North African coast.