Lune

EMNLP Findings2023

How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen

Abstract

Various techniques have been developed in recent years to improve dense retrieval (DR), such as unsupervised contrastive learning and pseudo-query generation. Existing DRs, however, often suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, which some argue was due to the limited model capacity. We contradict this hypothesis and show that a generalizable DR can be trained to achieve high accuracy in both supervised and zero-shot retrieval without increasing model size. In particular, we systematically examine the contrastive learning of DRs, under the framework of Data Augmentation (DA). Our study shows that common DA practices such as query augmentation with generative models and pseudo-relevance label creation using a cross-encoder, are often inefficient and suboptimal. We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR. As a result, DRAGON, 1 our Dense Retriever trained with diverse AuGmentatiON, is the first BERTbase-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations and even competes with models using more complex late interaction.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

37 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token cites RA-DIT: Retrieval-Augmented Dual Instruction Tuning
  • RA-DIT: Retrieval-Augmented Dual Instruction Tuning cites When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  • RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs cites When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  • RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs cites RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers37

  1. RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMsYue Yu, Wei Ping, Zihan Liu, Boxin Wang et al.NeurIPS 2024 · 321 citations
  2. Chain of Agents: Large Language Models Collaborating on Long-Context TasksYusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister et al.NeurIPS 2024 · 297 citations
  3. RA-DIT: Retrieval-Augmented Dual Instruction TuningXi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi et al.ICLR 2024 · 229 citations
  4. InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context MemoryChaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao et al.NeurIPS 2024 · 223 citations
  5. xRAG: Extreme Context Compression for Retrieval-augmented Generation with One TokenXin Cheng, Xun Wang, Xingxing Zhang, Tao Ge et al.NeurIPS 2024 · 156 citations

Builds on18

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Accelerating Large-Scale Inference with Anisotropic Vector QuantizationRuiqi Guo, Philip Sun, Erik Lindgren, Quan Geng et al.ICML 2020 · 539 citations
  3. Pre-training Tasks for Embedding-based Large-scale RetrievalWei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang et al.ICLR 2020 · 325 citations
  4. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware SamplingSebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin et al.SIGIR 2021 · 297 citations
  5. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric MemoriesAlex Mallen, Akari Asai, Victor Zhong, Rajarshi Das et al.ACL 2023 · 233 citations

Related papers

  1. COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust LearningYue Yu, Chenyan Xiong, Si Sun, Chao Zhang et al.EMNLP 2022 · 21 citations
  2. Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-TrainingYibin Lei, Liang Ding, Yu Cao, Changtong Zan et al.ACL Findings 2023 · 26 citations
  3. LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text RetrievalCanwen Xu, Daya Guo, Nan Duan, Julian J. McAuleyACL Findings 2022 · 26 citations
  4. A Thorough Examination on Zero-shot Dense RetrievalRuiyang Ren, Yingqi Qu, Jing Liu, Xin Zhao et al.EMNLP Findings 2023 · 27 citations
  5. Query-as-context Pre-training for Dense Passage RetrievalXing Wu, Guangyuan Ma, Wanhui Qian, Zijia Lin et al.EMNLP 2023 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines