Reason-to-Rank: Distilling Direct and Comparative Reasoning from Large Language Models for Document Reranking

Reranking documents in information retrieval often relies on black-box models that improve effectiveness but lack explainability. We introduce Reason-to-Rank (R2R), a novel framework that separates direct relevance reasoning from comparison reasoning to provide both direct and comparitive explanations. We first prompt a large language model to produce comprehensive rationales and a ranking order; then we distill both the ranking decisions and textual explanations into a smaller, open-source student model. Our approach not only improves retrieval performance, as demonstrated in MSMARCO, BEIR, and BRIGHT, but also provides interpretable justifications for why one document outranks another. We report NDCG@5 (and NDCG@10) for direct comparisons with prior work, and show that the distilled student model achieves competitive results while significantly reducing computational overhead. By unifying direct and comparative reasoning in a single pipeline, R2R bridges the gap between transparency and effectiveness in modern reranking systems.