FIRST: Faster Improved Listwise Reranking with Single Token Decoding

Large Language Models (LLMs) have significantly advanced the field of information retrieval, particularly for reranking. Listwise LLM rerankers typically showcase superior performance and generalizability over conventional supervised approaches. However, existing LLM rerankers can be inefficient as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. Further, they are trained using the standard language modeling objective, which treats all ranking errors uniformly, potentially at the cost of misranking highly relevant passages. Addressing these limitations, we introduce FIRST 1 , a novel listwise LLM reranking approach that leverages the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. We further utilize a learning-to-rank loss for this model, which prioritizes ranking accuracy for the more relevant passages. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining robust ranking performance, with gains across the BEIR benchmark. Finally, to illustrate the practical effectiveness of listwise LLM rerankers, we investigate their application in providing relevance feedback for retrievers during inference. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.