Lune

ACL Findings2022

LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval

Canwen Xu, Daya Guo, Nan Duan, Julian J. McAuley

Abstract

In this paper, we propose LaPraDoR, a pretrained dual-tower dense retriever that does not require any supervised data for training. Specifically, we first present Iterative Contrastive Learning (ICoL) that iteratively trains the query and document encoders with a cache mechanism. ICoL not only enlarges the number of negative instances but also keeps representations of cached examples in the same hidden space. We then propose Lexicon-Enhanced Dense Retrieval (LEDR) as a simple yet effective way to enhance dense retrieval with lexical matching. We evaluate LaPraDoR on the recently proposed BEIR benchmark, including 18 datasets of 9 zeroshot text retrieval tasks. Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models, and further analysis reveals the effectiveness of our training strategy and objectives. Compared to re-ranking, our lexiconenhanced approach can be run in milliseconds (22.5× faster) while achieving superior performance. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  • Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites Pre-training Tasks for Embedding-based Large-scale Retrieval
  • COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning cites MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  • COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
  • COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  • LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
  • LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  • RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
  • RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder cites Pre-training Tasks for Embedding-based Large-scale Retrieval
  • I3: Intent-Introspective Retrieval Conditioned on Instructions cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
  • I3: Intent-Introspective Retrieval Conditioned on Instructions cites COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
  • I3: Intent-Introspective Retrieval Conditioned on Instructions cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-EncoderShitao Xiao, Zheng Liu, Yingxia Shao, Zhao CaoEMNLP 2022 · 63 citations
  2. LED: Lexicon-Enlightened Dense Retriever for Large-Scale RetrievalKai Zhang, Chongyang Tao, Tao Shen, Can Xu et al.WWW 2023 · 27 citations
  3. COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust LearningYue Yu, Chenyan Xiong, Si Sun, Chao Zhang et al.EMNLP 2022 · 21 citations
  4. I3: Intent-Introspective Retrieval Conditioned on InstructionsKaihang Pan, Juncheng Li, Wenjie Wang, Hao Fei et al.SIGIR 2024 · 7 citations
  5. Rule By Example: Harnessing Logical Rules for Explainable Hate Speech DetectionChristopher Clarke, Matthew Hall, Gaurav Mittal, Ye Yu et al.ACL 2023 · 7 citations

Builds on18

  1. Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
  2. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained TransformersWenhui Wang, Furu Wei, Li Dong, Hangbo Bao et al.NeurIPS 2020 · 2,727 citations
  3. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
  4. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
  5. Pre-training Tasks for Embedding-based Large-scale RetrievalWei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang et al.ICLR 2020 · 325 citations

Related papers

  1. How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense RetrievalSheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz et al.EMNLP Findings 2023 · 49 citations
  2. Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-TrainingYibin Lei, Liang Ding, Yu Cao, Changtong Zan et al.ACL Findings 2023 · 26 citations
  3. Boot and Switch: Alternating Distillation for Zero-Shot Dense RetrievalFan Jiang, Qiongkai Xu, Tom Drummond, Trevor CohnEMNLP Findings 2023
  4. Unsupervised Corpus Aware Language Model Pre-training for Dense Passage RetrievalLuyu Gao, Jamie CallanACL 2022
  5. PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document RetrievalShengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin et al.EMNLP 2024 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines