Query-as-context Pre-training for Dense Passage Retrieval
Abstract
Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training. These methods simply consider two passages from the same document to be relevant, without taking into account the potential negative impacts of weakly correlated pairs. Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue. Query-as-context pre-training assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervised pre-training. The pre-trained models are evaluated on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. Experimental results show that query-as-context pre-training brings considerable gains for retrieval performances, demonstrating its effectiveness and efficiency.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites Pre-training Tasks for Embedding-based Large-scale Retrieval
- NExtLong: Toward Effective Long-Context Training without Long Documents cites Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model
- NExtLong: Toward Effective Long-Context Training without Long Documents cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- NExtLong: Toward Effective Long-Context Training without Long Documents cites Optimizing Dense Retrieval Model Training with Hard Negatives
- Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- QAEncoder: Towards Aligned Representation Learning in Question Answering Systems cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- QAEncoder: Towards Aligned Representation Learning in Question Answering Systems cites Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
- Optimizing Dense Retrieval Model Training with Hard Negatives cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Optimizing Dense Retrieval Model Training with Hard Negatives cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Cited by top-tier papers4
- Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage RetrievalGuangyuan Ma, Xing Wu, Zijia Lin, Songlin HuSIGIR 2024 · 5 citations
- QAEncoder: Towards Aligned Representation Learning in Question Answering SystemsZhengren Wang, Qinhan Yu, Shida Wei, Zhiyu Li et al.ACL 2025 · 5 citations
- Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language ModelChaochen Gao, Xing Wu, Qi Fu, Songlin HuICLR 2025
- NExtLong: Toward Effective Long-Context Training without Long DocumentsChaochen Gao, Xing Wu, Zijia Lin, Debing Zhang et al.ICML 2025
Builds on17
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
- Pre-training Tasks for Embedding-based Large-scale RetrievalWei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang et al.ICLR 2020 · 325 citations
- Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware SamplingSebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin et al.SIGIR 2021 · 297 citations
- Optimizing Dense Retrieval Model Training with Hard NegativesJingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo et al.SIGIR 2021 · 242 citations
Related papers
- ConTextual Masked Auto-Encoder for Dense Passage RetrievalXing Wu, Guangyuan Ma, Meng Lin, Zijia Lin et al.AAAI 2023 · 34 citations
- Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question AnsweringJiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo et al.ACL 2022 · 23 citations
- PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage RetrievalRuiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu et al.ACL Findings 2021 · 59 citations
- Improving Passage Retrieval with Zero-Shot Question GenerationDevendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan et al.EMNLP 2022 · 69 citations
- SimLM: Pre-training with Representation Bottleneck for Dense Passage RetrievalLiang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao et al.ACL 2023 · 41 citations
