ProvBench: A Benchmark of Legal Provision Recommendation for Contract Auto-Reviewing

Contract review is a critical process to protect the rights and interests of the parties involved. However, this process is time-consuming, laborintensive, and costly, especially when a contract faces multiple rounds of review. To accelerate the contract review and promote the completion of transactions, this paper introduces a novel benchmark of legal provision recommendation and conflict detection for contract auto-reviewing (PROVBENCH), which aims to recommend the legal provisions related to contract clauses and detect possible legal conflicts. Specifically, we construct the first Legal Provision Recommendation Dataset: PROVDATA, which covers 8 common contract types. In addition, we conduct extensive experiments to evaluate PROVBENCH on various state-of-theart models. Experimental results validate the feasibility of PROVBENCH and demonstrate the effectiveness of PROVDATA. Finally, we identify potential challenges in the PROVBENCH and advocate for further investigation 1 .