ACL Findings2024
A Two-Stage Adaptation of Large Language Models for Text Ranking
Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Min Zhang
Abstract
Text ranking is a critical task in information retrieval.Recent advances in pre-trained language models (PLMs), especially large language models (LLMs), present new opportunities for applying them to text ranking.While supervised fine-tuning (SFT) with ranking data has been widely explored to better align PLMs with text ranking goals, previous studies have focused primarily on encoder-only and encoder-decoder PLMs.Research on leveraging decoder-only LLMs for text ranking remains scarce.An exception to this is Ran-kLLaMA (Ma et al., 2023a), which uses direct SFT to explore LLaMA's potential for text ranking.In this work, we propose a two-stage progressive paradigm to better adapt LLMs to text ranking.First, we conduct continual pretraining (CPT) of LLMs on a large weaklysupervised corpus.Second, we perform SFT, and propose an improved optimization strategy building upon RankLLaMA.Our experimental results on multiple benchmarks show that our approach outperforms previous methods in both in-domain and out-domain scenarios.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Passage Retrieval with Zero-Shot Question Generation cites Document Ranking with a Pretrained Sequence-to-Sequence Model
- Improving Passage Retrieval with Zero-Shot Question Generation cites Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
- REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking cites Document Ranking with a Pretrained Sequence-to-Sequence Model
- REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking cites UL2: Unifying Language Learning Paradigms
Cited by top-tier papers4
- ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text RerankingYuzheng Cai, Yanzhao Zhang, Dingkun Long, Mingxin Li et al.AAAI 2026 · 6 citations
- CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data PartitionsJun Rao, Xuebo Liu, Lian Lian, Shengjun Cheng et al.EMNLP 2024 · 2 citations
- Multi-Modality Expansion and Retention for LLMs through Parameter Merging and DecouplingJunlin Li, Guodong Du, Jing Li, Sim Kuan Goh et al.ACL 2025
- REALM: Recursive Relevance Modeling for LLM-based Document Re-RankingPinhuan Wang, Zhiqiu Xia, Chunhua Liao, Feiyi Wang et al.EMNLP 2025
Builds on8
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 citations
- UL2: Unifying Language Learning ParadigmsYi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia et al.ICLR 2023 · 97 citations
- Don't Stop Pretraining: Adapt Language Models to Domains and TasksSuchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo et al.ACL 2020 · 93 citations
- Improving Passage Retrieval with Zero-Shot Question GenerationDevendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan et al.EMNLP 2022 · 69 citations
Related papers
- Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document RankingShengyao Zhuang, Bing Liu, Bevan Koopman, Guido ZucconEMNLP Findings 2023 · 36 citations
- MatRank: Text Re-ranking by Latent Preference MatrixJinwen Luo, Jiuding Yang, Weidong Guo, Chenglin Li et al.EMNLP Findings 2022 · 1 citation
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking AgentsWeiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang et al.EMNLP 2023 · 182 citations
- HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text ClassificationZihan Wang, Peiyi Wang, Tianyu Liu, Binghuai Lin et al.EMNLP 2022 · 42 citations
- K-order Ranking Preference Optimization for Large Language ModelsShihao Cai, Chongming Gao, Yang Zhang, Wentao Shi et al.ACL Findings 2025 · 1 citation
