ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction, entity typing and question answering, especially under low-resource settings. 1 * Corresponding author. 1 Our code and data are publicly available at https:// github.com/thunlp/ERICA . [1] Culiacán is a city in northwestern Mexico. [2] Culiacán is the capital of the state of Sinaloa. [3] Culiacán is also the seat of Culiacán Municipality. [4] It had an urban population of 785,800 in 2015 while 905,660 lived in the entire municipality. [5] While Culiacán Municipality has a total area of 4,758 k! ! , Culiacán itself is considerably smaller, measuring only. [6] Culiacán is a rail junction and is located on the Panamerican Highway that runs south to Guadalajara and Mexico City. [7] Culiacán is connected to the north with Los Mochis, and to the south with Mazatlán, Tepic.