Lune

EMNLP Findings2025

Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings

Liyan Xu, Zhenlin Su, Mo Yu, Jiangnan Li, Fandong Meng, Jie Zhou

Abstract

This work stems from an observed limitation of text encoders: embeddings may not be able to recognize fine-grained entities or events within encoded semantics, resulting in failed retrieval even in simple cases. To examine such behaviors, we first introduce a new evaluation dataset, CapRetrieval, in which passages are image captions and queries are phrases targeting entity or event concepts in diverse forms. Zero-shot evaluation suggests that encoders often struggle with these fine-grained matching, regardless of training sources or model size. Aiming for enhancement, we proceed to finetune encoders with our proposed data generation strategies, enabling a small 0.1B encoder to outperform the state-of-the-art 7B model. Within this process, we further uncover the granularity dilemma, a challenge for embeddings to capture fine-grained salience while aligning with overall semantics. Our dataset, code and models in this work are publicly released at https: //github.com/lxucs/CapRetrieval.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  • SimCSE: Simple Contrastive Learning of Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  4. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
  5. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge DistillationJianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo et al.ACL Findings 2024 · 762 citations

Related papers

  1. UFineBench: Towards Text-based Person Retrieval with Ultra-fine GranularityJialong Zuo, Hanyu Zhou, Ying Nie, Feng Zhang et al.CVPR 2024 · 45 citations
  2. Rethinking Benchmarks for Cross-modal Image-text RetrievalWeijing Chen, Linli Yao, Qin JinSIGIR 2023 · 25 citations
  3. Retrieval-Enhanced Contrastive Vision-Text ModelsAhmet Iscen, Mathilde Caron, Alireza Fathi, Cordelia SchmidICLR 2024 · 44 citations
  4. NERetrieve: Dataset for Next Generation Named Entity Recognition and RetrievalUri Katz, Matan Vetzler, Amir David Nissan Cohen, Yoav GoldbergEMNLP Findings 2023 · 7 citations
  5. FLAIR: VLM with Fine-grained Language-informed Image RepresentationsRui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines