ACL Findings2026
Candidate-Aware Retrieval and Reranking for Multiple-Choice Question Answering: Arabic as a Case Study
Abstract
Large language models (LLMs) have recently achieved impressive results on multiple-choice question answering (MCQA), with retrievalaugmented generation (RAG) emerging as an effective strategy for improving the performance of smaller models. However, existing RAG formulations face persistent challenges: retrieving too many passages often introduces noise, and even when relevant content is retrieved, models may still struggle with partially relevant or conflicting information. Moreover, while LLMs perform strongly on English benchmarks, their accuracy declines substantially on Arabic multi-task evaluations, revealing ongoing issues in cross-lingual transfer and domain adaptation. In this paper, we propose a novel approach, using Arabic as a representative case study, that jointly models the relevance of both the question and its candidate answers when selecting contextual passages. The method employs a lightweight reranker trained with a hybrid regression-triplet loss objective to identify passages that provide discriminative and reliable evidence. Extensive experiments across multiple model sizes and humanities domains show that our approach consistently outperforms both standard RAG baselines and reranker baselines, delivering two-to threefold improvements while remaining competitive with considerably larger models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic cites Crosslingual Generalization through Multitask Finetuning
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs cites Dense Passage Retrieval for Open-Domain Question Answering
Builds on5
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMsYue Yu, Wei Ping, Zihan Liu, Boxin Wang et al.NeurIPS 2024 · 321 citations
- Crosslingual Generalization through Multitask FinetuningNiklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts et al.ACL 2023 · 319 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
- ArabicMMLU: Assessing Massive Multitask Language Understanding in ArabicFajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman et al.ACL Findings 2024 · 17 citations
- Syn-QG: Syntactic and Shallow Semantic Rules for Question GenerationKaustubh D. Dhole, Christopher D. ManningACL 2020 · 2 citations
Related papers
- Don't Be Misled by Style: A Style-Adaptive Reranker for Capturing Effective Knowledge in Retrieval-Augmented GenerationRuwen Zhang, Bo Liu, Zhang Sheng Xiang, Yida Chen et al.ACL 2026
- Are Large Language Models Good at Utility Judgments?Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke et al.SIGIR 2024 · 20 citations
- LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question AnsweringQingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha et al.EMNLP 2024 · 13 citations
- Embedding-Based Context-Aware RerankerYe Yuan, Amin Shabani, Siqi LiuICLR 2026
- All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAGDan Wang, Guozhao Mo, Yafei Shi, Cheng Zhang et al.ACL 2026
