EMNLP Findings2024
HyQE: Ranking Contexts with Hypothetical Query Embeddings
Weichao Zhou, Jiaxin Zhang, Hilaf Hasson, Anu Singh, Wenchao Li
Abstract
In retrieval-augmented systems, context ranking techniques are commonly employed to reorder the retrieved contexts based on their relevance to a user query. A standard approach is to measure this relevance through the similarity between contexts and queries in the embedding space. However, such similarity often fails to capture the relevance. Alternatively, large language models (LLMs) have been used for ranking contexts. However, they can encounter scalability issues when the number of candidate contexts grows and the context window sizes of the LLMs remain constrained. Additionally, these approaches require fine-tuning LLMs with domain-specific data. In this work, we introduce a scalable ranking framework that combines embedding similarity and LLM capabilities without requiring LLM fine-tuning. Our framework uses a pre-trained LLM to hypothesize the user query based on the retrieved contexts and ranks the context based on the similarity between the hypothesized queries and the user query. Our framework is efficient at inference time and is compatible with many other retrieval and ranking techniques. Experimental results show that our method improves the ranking performance across multiple benchmarks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Large Language Models Can Be Easily Distracted by Irrelevant Context
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Builds on11
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
- Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations
- Are Emergent Abilities of Large Language Models a Mirage?Rylan Schaeffer, Brando Miranda, Sanmi KoyejoNeurIPS 2023 · 796 citations
Related papers
- Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document RankingShengyao Zhuang, Bing Liu, Bevan Koopman, Guido ZucconEMNLP Findings 2023 · 36 citations
- PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual AdapterHaoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang et al.EMNLP 2023 · 16 citations
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMsYue Yu, Wei Ping, Zihan Liu, Boxin Wang et al.NeurIPS 2024 · 321 citations
- Bridging the Preference Gap between Retrievers and LLMsZixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang et al.ACL 2024 · 8 citations
- Beyond Factual Queries: A Novel Predictive Retrieval-Augmented GenerationDebo Cheng, Jianfeng Deng, Qingfeng Chen, Jinyi Jie et al.WWW 2026
