Lune

SIGIR2026Top-tier venue

Adaptive Sparsity Optimization with Learnable Soft Top-K and Per-Term Thresholding for Efficient Retrieval

Wentai Xie, Parker Carlson, Shanxiu He, Tao Yang

Abstract

Recent work on neural sparse retrieval has demonstrated strong relevance by leveraging Large Language Models (LLMs) for semantic term expansion. However, learned models paired with previous sparsification techniques still yield overly long document and query vectors partly due to a large LLM vocabulary, imposing a serious challenge to retrieval time and space efficiency. This paper proposes a scheme for optimizing model sparsity through a synergy of adaptive strategies, including learnable soft top-??, per-term thresholding, and FLOPs regularization to increase the sparsity of query and document vectors. Experimental results with Lion-SP model on the MS MARCO and BEIR datasets demonstrate that the proposed scheme can outperform the baselines by significantly reducing the average query and document lengths. Our scheme can achieve much shorter retrieval latency and lower storage cost while maintaining highly competitive relevance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval cites Minimizing FLOPs to Learn Efficient Sparse Representations
  • LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. Sparse Autoencoders Find Highly Interpretable Features in Language ModelsRobert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart et al.ICLR 2024 · 1,072 citations
  3. Minimizing FLOPs to Learn Efficient Sparse RepresentationsBiswajit Paria, Chih-Kuan Yeh, Ian En-Hsu Yen, Ning Xu et al.ICLR 2020 · 85 citations
  4. Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse RepresentationsSebastian Bruch, Franco Maria Nardini, Cosimo Rulli, Rossano VenturiniSIGIR 2024 · 47 citations
  5. LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale RetrievalTao Shen, Xiubo Geng, Chongyang Tao, Can Xu et al.ICLR 2023 · 14 citations

Related papers

  1. Efficient Sparse Retrieval with Lightweight Superblock PruningParker Carlson, Wentai Xie, Rohil Shah, Tao YangSIGIR 2026 · 2 citations
  2. Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text RetrievalKyoungrok Jang, Junmo Kang, Giwon Hong, Sung-Hyon Myaeng et al.EMNLP 2021 · 13 citations
  3. Accelerating Inference of Retrieval-Augmented Generation via Sparse Context SelectionYun Zhu, Jia-Chen Gu, Caitlin Sikora, Ho Ko et al.ICLR 2025
  4. Learning Retrieval Models with Sparse AutoencodersThibault Formal, Maxime Louis, Hervé Déjean, Stéphane ClinchantICLR 2026 · 9 citations
  5. Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query ExpansionLingyuan Liu, Mengxiang ZhangACL Findings 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines