Lune

ICLR2026Top-tier venue

Learning Retrieval Models with Sparse Autoencoders

Thibault Formal, Maxime Louis, Hervé Déjean, Stéphane Clinchant

Abstract

Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. By leveraging recently released open-source SAEs, we show that their latent features can serve as effective indexing units for representing documents and queries for sparse retrieval. Our experiments demonstrate that SAE-based LSR models consistently outperform their vocabulary-based counterparts in multilingual and out-of-domain settings. Finally, we introduce SPLARE, a 7B-parameter multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieving top results on MMTEB’s multilingual and English retrieval tasks. We also release a more efficient 2B-parameter variant, offering strong performance with a significantly lighter footprint.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites A Simple Framework for Contrastive Learning of Visual Representations
  • Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. From Tokens to Concepts: Leveraging SAE for SPLADEYuxuan Zong, Mathias Vast, Basile Van Cooten, Laure Soulier et al.SIGIR 2026

Builds on26

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  4. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
  5. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations

Related papers

  1. Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense RetrievalSeongwan Park, Taeklim Kim, Youngjoong KoEMNLP 2025
  2. LSAR: Sparse Lexical Representation Learning for Efficient and Interpretable Audio RetrievalHaoyue Li, Yuzhe Bai, Li NiuKDD 2026
  3. Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?Hengyi Feng, Zeang Sheng, Meiyi Qiang, Yang Li et al.ACL Findings 2026
  4. On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted RemedyJingyi Cui, Qi Zhang, Yifei Wang, Yisen WangICLR 2026 · 16 citations
  5. Sparse Latents Steer Retrieval-Augmented GenerationChunlei Xin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang et al.ACL 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines