Improving Large-scale Paraphrase Acquisition and Generation

This paper addresses the quality issues in existing Twitter-based paraphrase datasets, and discusses the necessity of using two separate definitions of paraphrase for identification and generation tasks. We present a new Multi-Topic Paraphrase in Twitter (MULTIPIT) corpus that consists of a total of 130k sentence pairs with crowdsoursing (MULTIPIT CROWD ) and expert (MULTIPIT EXPERT ) annotations using two different paraphrase definitions for paraphrase identification, in addition to a multi-reference test set (MULTIPIT NMR ) and a large automatically constructed training set (MULTIPIT AUTO ) for paraphrase generation. With improved data annotation quality and task-specific paraphrase definition, the best pre-trained language model fine-tuned on our dataset achieves the stateof-the-art performance of 84.2 F 1 for automatic paraphrase identification. Furthermore, our empirical results also demonstrate that the paraphrase generation models trained on MUL-TIPIT AUTO generate more diverse and highquality paraphrases compared to their counterparts fine-tuned on other corpora such as Quora, MSCOCO, and ParaNMT.