ACL Findings2024
Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
Zhuo Chen, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Kewei Tu
Abstract
In the era of large language models, applying techniques such as Retrieval Augmented Generation can better address Open-Domain Question-Answering problems. Due to constraints including model sizes and computing resources, the length of context is often limited, and it becomes challenging to empower the model to cover overlong contexts while answering questions from open domains. This paper proposes a general and convenient method to cover longer contexts in Open-Domain Question-Answering tasks. It leverages a small encoder and cross-attention mechanism and effectively encodes contexts. With our method, the original language models can cover several times longer contexts while keeping the computing requirements close to the baseline. Our experiments demonstrate that after finetuning, there is improved performance across two held-in datasets, four held-out datasets, and also in two In Context Learning settings. Our code will be released at https://github. com/Alibaba-NLP/Vec-RA-ODQA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Dense Passage Retrieval for Open-Domain Question Answering cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks cites Dense Passage Retrieval for Open-Domain Question Answering
- Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference cites Language Models are Few-Shot Learners
- Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Language Models are Few-Shot Learners cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks cites Generalization through Memorization: Nearest Neighbor Language Models
Cited by top-tier papers1
Builds on9
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Generalization through Memorization: Nearest Neighbor Language ModelsUrvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2020 · 1,038 citations
- Crosslingual Generalization through Multitask FinetuningNiklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts et al.ACL 2023 · 319 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
Related papers
- QPaug: Question and Passage Augmentation for Open-Domain Question Answering of LLMsMinsang Kim, Cheoneum Park, Seung BaekEMNLP Findings 2024 · 5 citations
- Q-RAG: Long Context Multi‑Step Retrieval via Value‑Based Embedder TrainingArtyom Y. Sorokin, Nazar Buzun, Alexander Anokhin, Egor Vedernikov et al.ICLR 2026 · 4 citations
- MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval AugmentationHongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao et al.WWW 2025 · 92 citations
- ConvX: A Lightweight Converter to Bridge Indexed Dense Representations and Large Language Models for Retrieval-Augmented GenerationBonggeun Choi, Keunha Kim, Junho Han, Youngjoong KoACL Findings 2026
- LLoCO: Learning Long Contexts OfflineSijun Tan, Xiuyu Li, Shishir G. Patil, Ziyang Wu et al.EMNLP 2024 · 3 citations
