RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems

Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity. We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with fivelevel self-assessed familiarity ratings collected via a large-scale email survey, yielding 1,055 expert-annotated paper-reviewer-score annotations. We further propose RATE, a reviewercentric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling the matching of each manuscript against a reviewer profile directly. Across the LR-bench and the CMU gold-standard dataset, our approach consistently achieves state-ofthe-art performance, outperforming strong embedding baselines by a clear margin. We release LR-bench at https://huggingface.co/ datasets/Gnociew/LR-bench , and an github repository at https://github.com/Gnociew/ RATE-Reviewer-Assignment .