ACL Findings2021
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval
Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, Ji-Rong Wen
Abstract
Recently, dense passage retrieval has become a mainstream approach to finding relevant information in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder retriever. In order to capture more comprehensive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To implement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. Extensive experiments show that our approach significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions datasets 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
30 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Adversarial Retriever-Ranker for Dense Text Retrieval cites ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
- Adversarial Retriever-Ranker for Dense Text Retrieval cites Distilling Knowledge from Reader to Retriever for Question Answering
- Adversarial Retriever-Ranker for Dense Text Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking cites ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking cites Distilling Knowledge from Reader to Retriever for Question Answering
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking cites Scalable Zero-shot Entity Linking with Dense Entity Retrieval
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Promptagator: Few-shot Dense Retrieval From 8 Examples cites Adversarial Retriever-Ranker for Dense Text Retrieval
- Promptagator: Few-shot Dense Retrieval From 8 Examples cites RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
- Promptagator: Few-shot Dense Retrieval From 8 Examples cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval cites RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
- SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Multi-View Document Representation Learning for Open-Domain Dense Retrieval cites Adversarial Retriever-Ranker for Dense Text Retrieval
- Multi-View Document Representation Learning for Open-Domain Dense Retrieval cites RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
- Multi-View Document Representation Learning for Open-Domain Dense Retrieval cites Distilling Knowledge from Reader to Retriever for Question Answering
- Multi-View Document Representation Learning for Open-Domain Dense Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Cited by top-tier papers30
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingRuiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao et al.EMNLP 2021 · 147 citations
- Adversarial Retriever-Ranker for Dense Text RetrievalHang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv et al.ICLR 2022 · 137 citations
- Multi-View Document Representation Learning for Open-Domain Dense RetrievalShunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang et al.ACL 2022 · 80 citations
- Promptagator: Few-shot Dense Retrieval From 8 ExamplesZhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan et al.ICLR 2023 · 46 citations
- SimLM: Pre-training with Representation Bottleneck for Dense Passage RetrievalLiang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao et al.ACL 2023 · 41 citations
Builds on8
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutesYang You, Jing Li, Sashank J. Reddi, Jonathan Hseu et al.ICLR 2020 · 1,170 citations
- ERNIE 2.0: A Continual Pre-Training Framework for Language UnderstandingYu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng et al.AAAI 2020 · 885 citations
- Scalable Zero-shot Entity Linking with Dense Entity RetrievalLedell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel et al.EMNLP 2020 · 336 citations
- Distilling Knowledge from Reader to Retriever for Question AnsweringGautier Izacard, Edouard GraveICLR 2021 · 317 citations
Related papers
- Query-as-context Pre-training for Dense Passage RetrievalXing Wu, Guangyuan Ma, Wanhui Qian, Zijia Lin et al.EMNLP 2023 · 2 citations
- GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage RetrievalJiduan Liu, Jiahao Liu, Yang Yang, Jingang Wang et al.EMNLP Findings 2022 · 5 citations
- xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question AnsweringNan Yang, Furu Wei, Binxing Jiao, Daxing Jiang et al.ACL 2021
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
- ConTextual Masked Auto-Encoder for Dense Passage RetrievalXing Wu, Guangyuan Ma, Meng Lin, Zijia Lin et al.AAAI 2023 · 34 citations
