Lune

EMNLP Findings2023

Auto Search Indexer for End-to-End Document Retrieval

Tianchi Yang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang

Abstract

Generative retrieval, which is a new advanced paradigm for document retrieval, has recently attracted research interests, since it encodes all documents into the model and directly generates the retrieved documents. However, its power is still underutilized since it heavily relies on the "preprocessed" document identifiers (docids), thus limiting its retrieval performance and ability to retrieve new documents. In this paper, we propose a novel fully end-to-end retrieval paradigm. It can not only end-to-end learn the best docids for existing and new documents automatically via a semantic indexing module, but also perform end-to-end document retrieval via an encoder-decoder-based generative model, namely Auto Search Indexer (ASI). Besides, we design a reparameterization mechanism to combine the above two modules into a joint optimization framework. Extensive experimental results demonstrate the superiority of our model over advanced baselines on both public and industrial datasets and also verify the ability to deal with new documents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

13 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Autoregressive Search Engines: Generating Substrings as Document Identifiers cites Transformer Memory as a Differentiable Search Index
  • Self-Retrieval: End-to-End Information Retrieval with One Large Language Model cites Autoregressive Search Engines: Generating Substrings as Document Identifiers
  • Self-Retrieval: End-to-End Information Retrieval with One Large Language Model cites Transformer Memory as a Differentiable Search Index
  • Self-Retrieval: End-to-End Information Retrieval with One Large Language Model cites A Neural Corpus Indexer for Document Retrieval
  • Generative Retrieval as Multi-Vector Dense Retrieval cites Autoregressive Search Engines: Generating Substrings as Document Identifiers
  • Generative Retrieval as Multi-Vector Dense Retrieval cites Transformer Memory as a Differentiable Search Index
  • Generative Retrieval as Multi-Vector Dense Retrieval cites A Neural Corpus Indexer for Document Retrieval
  • OneSug: The Unified End-to-End Generative Framework for E-commerce Query Suggestion cites Autoregressive Search Engines: Generating Substrings as Document Identifiers
  • OneSug: The Unified End-to-End Generative Framework for E-commerce Query Suggestion cites Transformer Memory as a Differentiable Search Index
  • Transformer Memory as a Differentiable Search Index cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • A Neural Corpus Indexer for Document Retrieval cites Autoregressive Search Engines: Generating Substrings as Document Identifiers
  • A Neural Corpus Indexer for Document Retrieval cites Transformer Memory as a Differentiable Search Index
  • CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks cites Autoregressive Search Engines: Generating Substrings as Document Identifiers
  • CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks cites Transformer Memory as a Differentiable Search Index
  • CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks cites A Neural Corpus Indexer for Document Retrieval
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers13

  1. CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive TasksXiaoxi Li, Zhicheng Dou, Yujia Zhou, Fangchao LiuSIGIR 2024 · 16 citations
  2. OneSug: The Unified End-to-End Generative Framework for E-commerce Query SuggestionXian Guo, Ben Chen, Siyuan Wang, Ying Yang et al.AAAI 2026 · 15 citations
  3. Self-Retrieval: End-to-End Information Retrieval with One Large Language ModelQiaoyu Tang, Jiawei Chen, Zhuoqun Li, Bowen Yu et al.NeurIPS 2024 · 14 citations
  4. Generative Retrieval as Multi-Vector Dense RetrievalShiguang Wu, Wenda Wei, Mengqi Zhang, Zhumin Chen et al.SIGIR 2024 · 14 citations
  5. Generative Recommender with End-to-End Learnable Item TokenizationEnze Liu, Bowen Zheng, Cheng Ling, Lantao Hu et al.SIGIR 2025 · 11 citations

Builds on11

  1. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  2. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  3. Transformer Memory as a Differentiable Search IndexYi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni et al.NeurIPS 2022 · 506 citations
  4. Autoregressive Search Engines: Generating Substrings as Document IdentifiersMichele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih et al.NeurIPS 2022 · 242 citations
  5. A Neural Corpus Indexer for Document RetrievalYujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao et al.NeurIPS 2022 · 242 citations

Related papers

  1. Descriptive and Discriminative Document Identifiers for Generative RetrievalJiehan Cheng, Zhicheng Dou, Yutao Zhu, Xiaoxi LiAAAI 2025 · 4 citations
  2. Learning to Tokenize for Generative RetrievalWeiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang et al.NeurIPS 2023 · 151 citations
  3. Enhancing Generative Retrieval with Reinforcement Learning from Relevance FeedbackYujia Zhou, Zhicheng Dou, Ji-Rong WenEMNLP 2023 · 14 citations
  4. DiffusionRet: Diffusion-Enhanced Generative Retriever using Constrained DecodingShanbao Qiao, Xuebing Liu, Seung-Hoon NaEMNLP Findings 2023 · 2 citations
  5. Multi-level Relevance Document Identifier Learning for Generative RetrievalFuwei Zhang, Xiaoyu Liu, Xinyu Jia, Yingfei Zhang et al.ACL 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines