Are Key-Phrases All That Reviewers Care About? A Comprehensive Benchmarking of Reviewer Matchmaking Systems

Reviewer Matchmaking (RM) is a pivotal process in academic publishing that aligns manuscripts with appropriate reviewers based on their expertise and prior publications. The demand for an automated RM system has escalated with the significant surge in submissions over the past decade. State-of-the-art (SOTA) RM models are document-representation-based (DR-RM) and match the manuscript and reviewer's past publication using a similarity method defined on a high-dimensional vector space. However, they are far from accurate despite their largescale usage. In this paper, we establish that conventional RM evaluation measures are unreliable and instead emphasize that standard correlation measures are adequate. For the first time, we compare the performance of six SOTA DR-RM models with those of fourteen SOTA Key-phrase Extraction-based RM (KPE-RM) models -an alternate unexplored approach. We observe that KPE-RM models show comparable results in many cases, with the new best model being PatternRank-RMa KPE-RM model beating the best DR-RM model SPECTER2-RM (Pearson: 0.004+, Spearman: 0.006+, Kendall: 0.043+). We conclude that KPE-RM models must be contextualized to the RM task and cannot be used as plug-n-play.

Academic journals and conferences are crucial platforms for researchers to share their work and receive expert feedback. The process of optimally assigning appropriate manuscripts to reviewers based on the reviewers' area of interest, expertise, and availability such that every manuscript is assigned the minimum required reviewer and, at the same time, no reviewer is overburdened by more than the maximum manuscripts is called Reviewer Allocation (RA) (Stelmakh, Shah, and Singh 2019;Cousins, Payan, and Zick 2023;Aziz, Micha, and Shah 2023). A necessary step for RA is Reviewer Matchmaking (RM), also known as paper-reviewer matching, where a set of "most suitable" reviewers are matched, thereby providing a score to every manuscript-reviewer pair.

There has been a significant surge in the number of manuscript submissions received in the past decade, rendering the manually curated, careful RM process unrealistic. Even with a bidding system, the