Lune

ACL Findings2024

Distillation Enhanced Generative Retrieval

Yongqi Li, Zhen Zhang, Wenjie Wang, Liqiang Nie, Wenjie Li, Tat-Seng Chua

Abstract

Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generative language models, distinct from traditional sparse or dense retrieval methods. In this work, we identify a viable direction to further enhance generative retrieval via distillation and propose a feasible framework, named DGR. DGR utilizes sophisticated ranking models, such as the cross-encoder, in a teacher role to supply a passage rank list, which captures the varying relevance degrees of passages instead of binary hard labels; subsequently, DGR employs a specially designed distilled RankNet loss to optimize the generative retrieval model, considering the passage rank order provided by the teacher model as labels. This framework only requires an additional distillation step to enhance current generative retrieval systems and does not add any burden to the inference stage. We conduct experiments on four public datasets, and the results indicate that DGR achieves state-of-the-art performance among the generative retrieval methods. Additionally, DGR demonstrates exceptional robustness and generalizability with various teacher models and distillation losses. The code is released at https://github.com/liyongqi67/DGR.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Autoregressive Search Engines: Generating Substrings as Document Identifiers cites Language Models are Few-Shot Learners
  • Autoregressive Search Engines: Generating Substrings as Document Identifiers cites Transformer Memory as a Differentiable Search Index
  • Language Models are Few-Shot Learners cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • OneSug: The Unified End-to-End Generative Framework for E-commerce Query Suggestion cites Autoregressive Search Engines: Generating Substrings as Document Identifiers
  • OneSug: The Unified End-to-End Generative Framework for E-commerce Query Suggestion cites Transformer Memory as a Differentiable Search Index
  • Transformer Memory as a Differentiable Search Index cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. OneSug: The Unified End-to-End Generative Framework for E-commerce Query SuggestionXian Guo, Ben Chen, Siyuan Wang, Ying Yang et al.AAAI 2026 · 15 citations

Builds on14

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. Transformer Memory as a Differentiable Search IndexYi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni et al.NeurIPS 2022 · 506 citations
  4. Pre-training Tasks for Embedding-based Large-scale RetrievalWei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang et al.ICLR 2020 · 325 citations
  5. Autoregressive Search Engines: Generating Substrings as Document IdentifiersMichele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih et al.NeurIPS 2022 · 242 citations

Related papers

  1. Learning to Rank in Generative RetrievalYongqi Li, Nan Yang, Liang Wang, Furu Wei et al.AAAI 2024 · 83 citations
  2. DOGR: Leveraging Document-Oriented Contrastive Learning in Generative RetrievalPenghao Lu, Xin Dong, Yuansheng Zhou, Lei Cheng et al.AAAI 2025
  3. Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense RetrievalHouxing Ren, Linjun Shou, Ning Wu, Ming Gong et al.EMNLP 2022 · 6 citations
  4. Multiview Identifiers Enhanced Generative RetrievalYongqi Li, Nan Yang, Liang Wang, Furu Wei et al.ACL 2023 · 30 citations
  5. Killing Two Birds with One Stone: Unifying Retrieval and Ranking with a Single Generative Recommendation ModelLuankang Zhang, Kenan Song, Yi Quan Lee, Wei Guo et al.SIGIR 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines