Lune

ACL Findings2024

A Two-Stage Adaptation of Large Language Models for Text Ranking

Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Min Zhang

Abstract

Text ranking is a critical task in information retrieval.Recent advances in pre-trained language models (PLMs), especially large language models (LLMs), present new opportunities for applying them to text ranking.While supervised fine-tuning (SFT) with ranking data has been widely explored to better align PLMs with text ranking goals, previous studies have focused primarily on encoder-only and encoder-decoder PLMs.Research on leveraging decoder-only LLMs for text ranking remains scarce.An exception to this is Ran-kLLaMA (Ma et al., 2023a), which uses direct SFT to explore LLaMA's potential for text ranking.In this work, we propose a two-stage progressive paradigm to better adapt LLMs to text ranking.First, we conduct continual pretraining (CPT) of LLMs on a large weaklysupervised corpus.Second, we perform SFT, and propose an improved optimization strategy building upon RankLLaMA.Our experimental results on multiple benchmarks show that our approach outperforms previous methods in both in-domain and out-domain scenarios.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Improving Passage Retrieval with Zero-Shot Question Generation cites Document Ranking with a Pretrained Sequence-to-Sequence Model
  • Improving Passage Retrieval with Zero-Shot Question Generation cites Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
  • REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking cites Document Ranking with a Pretrained Sequence-to-Sequence Model
  • REALM: Recursive Relevance Modeling for LLM-based Document Re-Ranking cites UL2: Unifying Language Learning Paradigms
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text RerankingYuzheng Cai, Yanzhao Zhang, Dingkun Long, Mingxin Li et al.AAAI 2026 · 6 citations
  2. CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data PartitionsJun Rao, Xuebo Liu, Lian Lian, Shengjun Cheng et al.EMNLP 2024 · 2 citations
  3. Multi-Modality Expansion and Retention for LLMs through Parameter Merging and DecouplingJunlin Li, Guodong Du, Jing Li, Sim Kuan Goh et al.ACL 2025
  4. REALM: Recursive Relevance Modeling for LLM-based Document Re-RankingPinhuan Wang, Zhiqiu Xia, Chunhua Liao, Feiyi Wang et al.EMNLP 2025

Builds on8

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 citations
  3. UL2: Unifying Language Learning ParadigmsYi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia et al.ICLR 2023 · 97 citations
  4. Don't Stop Pretraining: Adapt Language Models to Domains and TasksSuchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo et al.ACL 2020 · 93 citations
  5. Improving Passage Retrieval with Zero-Shot Question GenerationDevendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan et al.EMNLP 2022 · 69 citations

Related papers

  1. Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document RankingShengyao Zhuang, Bing Liu, Bevan Koopman, Guido ZucconEMNLP Findings 2023 · 36 citations
  2. MatRank: Text Re-ranking by Latent Preference MatrixJinwen Luo, Jiuding Yang, Weidong Guo, Chenglin Li et al.EMNLP Findings 2022 · 1 citation
  3. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking AgentsWeiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang et al.EMNLP 2023 · 182 citations
  4. HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text ClassificationZihan Wang, Peiyi Wang, Tianyu Liu, Binghuai Lin et al.EMNLP 2022 · 42 citations
  5. K-order Ranking Preference Optimization for Large Language ModelsShihao Cai, Chongming Gao, Yang Zhang, Wentao Shi et al.ACL Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines