RST Discourse Parsing with Second-Stage EDU-Level Pre-training

Pre-trained language models (PLMs) have shown great potentials in natural language processing (NLP) including rhetorical structure theory (RST) discourse parsing. Current PLMs are obtained by sentence-level pre-training, which is different from the basic processing unit, i.e. element discourse unit (EDU). To this end, we propose a second-stage EDU-level pretraining approach in this work, which presents two novel tasks to learn effective EDU representations continually based on well pre-trained language models. Concretely, the two tasks are (1) next EDU prediction (NEP) and ( 2 ) discourse marker prediction (DMP). We take a state-of-the-art transition-based neural parser as baseline, and adopt it with a light bi-gram EDU modification to effectively explore the EDU-level pre-trained EDU representation. Experimental results on a benckmark dataset show that our method is highly effective, leading a 2.1-point improvement in F1-score. All codes and pre-trained models will be released publicly to facilitate future studies. 1 * Corresponding author. 1 http://github.com/yunan4nlp/ E-NNRSTParser 2 In this study, we focus on the tree construction task, assuming the gold standard EDU as inputs. e 1 [CNW Corp. said] e 2 [the final step in the acquisition of the company has been completed with the merger of CNW with a subsidiary of Chicago & North Western Holdings Corp.] e 3 [As reported,] e 4 [CNW agreed to be acquired by a group of investors] e 5 [led by Blackstone Capital Partners