Entity Enhanced BERT Pre-training for Chinese NER

Character-level BERT pre-trained in Chinese suffers a limitation of lacking lexicon information, which shows effectiveness for Chinese NER. To integrate the lexicon into pre-trained LMs for Chinese NER, we investigate a semisupervised entity enhanced BERT pre-training method. In particular, we first extract an entity lexicon from the relevant raw text using a newword discovery method. We then integrate the entity information into BERT using Char-Entity-Transformer, which augments the selfattention using a combination of character and entity representations. In addition, an entity classification task helps inject the entity information into model parameters in pre-training. The pre-trained models are used for NER finetuning. Experiments on a news dataset and two datasets annotated by ourselves for NER in long-text show that our method is highly effective and achieves the best results.