Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision

Collaborative review and revision of textual documents is the core of knowledge work and a promising target for empirical analysis and NLP assistance. Yet, a holistic framework that would allow modeling complex relationships between document revisions, reviews and author responses is lacking. To address this gap, we introduce Re3, a framework for joint analysis of collaborative document revision. We instantiate this framework in the scholarly domain, and present Re3-Sci, a large corpus of aligned scientific paper revisions manually labeled according to their action and intent, and supplemented with the respective peer reviews and human-written edit summaries. We use the new data to provide first empirical insights into collaborative document revision in the academic domain, and to assess the capabilities of state-of-the-art LLMs at automating edit analysis and facilitating text-based collaboration. We make our annotation environment and protocols, the resulting data and experimental code publicly available. 1 2022; Jiang et al., 2022), reviews and original doc-041 uments (Dycke et al., 2023), reviews and revisions 042 (Kuznetsov et al., 2022; D'Arcy et al., 2023), and 043 reviews and responses (Gao et al., 2019; Cheng 044 et al., 2020) -no prior frameworks allow jointly 045 modeling all three components of text-based col-046 laboration. Yet, such joint modeling is important 047 as it provides deeper insights into the processes 048 involved in text work, and opens new opportunities 049 for NLP applications. Important tasks that involve 050 reviews, revisions and responses such as edit sum-051 marization thus remain underexplored. 052 Comprehensive analysis of document-level re-053 visions poses additional challenges. Contrary to 054 sentence-level analysis, hierarchically structured 055 documents (Ruan et al., 2022) bring distinct levels 056 of granularity into editing. Individuals execute re-057 visions at various granularity levels, with a range 058 of actions and a spectrum of intents, reflecting the 059 what, how, and why of the revisions (Figure 1 and 060 §3.2). Realistic modeling of document revision 061 full-scope reviewrevision revisionresponse reviewresponse F1000RD (2022) no yes no no NLPeer (2023) no yes no no ARIES (2023) no yes no no Re3-Sci (ours) yes yes yes yes (b) Comparison of review-revision-response datasets. Presented are presence of full-scope revision annotations, and interactions between the documents. Table 1: Related datasets comparison. Document revision datasets. Research on text 123 revision originates in studies on Wikipedia (Dax-124