Lune

EMNLP Findings2024

Dense Passage Retrieval: Is it Retrieving?

Benjamin Z. Reichman, Larry Heck

Abstract

Large Language Models (LLMs) internally store repositories of knowledge. However, their access to this repository is imprecise and they frequently hallucinate information that is not true or does not exist. A paradigm called Retrieval Augmented Generation (RAG) promises to fix these issues. Dense passage retrieval (DPR) is the first step in this paradigm. In this paper, we analyze the role of DPR finetuning and how it affects the model being trained. DPR fine-tunes pre-trained networks to enhance the alignment of the embeddings between queries and relevant textual data. We explore DPR-trained models mechanistically by using a combination of probing, layer activation analysis, and model editing. Our experiments show that DPR training decentralizes how knowledge is stored in the network, creating multiple access pathways to the same information. We also uncover a limitation in this training style: the internal knowledge of the pre-trained model bounds what the retrieval model can retrieve. These findings suggest a few possible directions for dense retrieval: (1) expose the DPR training process to more knowledge so more can be decentralized, (2) inject facts as decentralized representations, (3) model and incorporate knowledge uncertainty in the retrieval process, and (4) directly map internal model knowledge to a knowledge base.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Eliciting Implicit Acoustic Styles from Open-domain Instructions to Facilitate Fine-grained Controllable Generation of Speech cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Re³: Relevance & Recency Retrieval for Mitigating Temporal Hallucination cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Eliciting Implicit Acoustic Styles from Open-domain Instructions to Facilitate Fine-grained Controllable Generation of SpeechJianxing Yu, Zihao Gou, Chen Li, Zhisheng Wang et al.EMNLP 2025
  2. Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAGIlias Triantafyllopoulos, Renyi Qu, Salvatore Giorgi, Brenda Curtis et al.ACL 2026
  3. Re³: Relevance & Recency Retrieval for Mitigating Temporal HallucinationJiawei Cao, Jie Ouyang, Mingyue Cheng, Zhaomeng Zhou et al.ACL 2026
  4. RPDR: A Round-trip Prediction-Based Data Augmentation Framework for Long-Tail Question AnsweringYiming Zhang, Siyue Zhang, Junbo Zhao, Chen ZhaoEMNLP 2025
  5. ALEX: A Light Editing-knowledge ExtractorMinghu Wang, Shuliang Zhao, Yuanyuan Zhao, Hongxia XuAAAI 2026

Builds on26

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
  3. Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations
  4. Transformer Memory as a Differentiable Search IndexYi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni et al.NeurIPS 2022 · 506 citations
  5. Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 citations

Related papers

  1. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMsOded Ovadia, Menachem Brief, Moshik Mishaeli, Oren ElishaEMNLP 2024 · 89 citations
  2. The Power of Noise: Redefining Retrieval for RAG SystemsFlorin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice et al.SIGIR 2024 · 212 citations
  3. Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented GenerationGuanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang et al.WWW 2025 · 44 citations
  4. Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGBowen Jin, Jinsung Yoon, Jiawei Han, Sercan Ö. ArikICLR 2025
  5. Fine-grained Knowledge Enhancement for Retrieval-Augmented GenerationJingxuan Han, Zhendong Mao, Yi Liu, Yexuan Che et al.ACL Findings 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines