Lune

ACL Findings2021

Toward Fully Exploiting Heterogeneous Corpus: A Decoupled Named Entity Recognition Model with Two-stage Training

Yun Hu, Yeshuang Zhu, Jinchao Zhang, Changwen Zheng, Jie Zhou

Abstract

Named Entity Recognition (NER) is a fundamental and widely used task in natural language processing (NLP), which is generally trained on the human-annotated corpus.However, data annotation is costly and timeconsuming, which restricts its scale and further leads to the performance bottleneck of NER models.In reality, we can conveniently collect large-scale entity dictionaries and distantly supervised data.However, the collected dictionaries are lack of semantic context and the distantly supervised training instances contain large noise, which will bring uncertain effects to NER models when directly incorporated into the high-quality training set.To address the above issue, we propose a BERT-based decoupled NER model with two-stage training to appropriately take advantage of the heterogeneous corpus, including dictionaries, distantly supervised instances, and human-annotated instances.Our decoupled model consists of a Mention-BERT and a Context-BERT to respectively learn from the context-deficient dictionaries and noised distantly supervised instances at the pre-training stage.At the unifiedtraining stage, the two BERTs are trained together on human-annotated data to predict the correct labels for candidate regions.Empirical studies on three Chinese NER datasets demonstrate that our method achieves significant improvements against several baselines, establishing the new state-of-the-art performance.* Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc. Work was done when Yun Hu was intern at WeChat AI.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  3. A Unified MRC Framework for Named Entity RecognitionXiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han et al.ACL 2020 · 617 citations
  4. Simplify the Usage of Lexicon in Chinese NERRuotian Ma, Minlong Peng, Qi Zhang, Zhongyu Wei et al.ACL 2020 · 286 citations
  5. Don't Stop Pretraining: Adapt Language Models to Domains and TasksSuchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo et al.ACL 2020 · 93 citations

Related papers

  1. Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-TrainingYu Meng, Yunyi Zhang, Jiaxin Huang, Xuan Wang et al.EMNLP 2021 · 50 citations
  2. Entity Enhanced BERT Pre-training for Chinese NERChen Jia, Yuefeng Shi, Qinrong Yang, Yue ZhangEMNLP 2020 · 59 citations
  3. BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant SupervisionChen Liang, Yue Yu, Haoming Jiang, Siawpeng Er et al.KDD 2020 · 118 citations
  4. SENCR: A Span Enhanced Two-Stage Network with Counterfactual Rethinking for Chinese NERHang Zheng, Qingsong Li, Shen Chen, Yuxuan Liang et al.AAAI 2024 · 8 citations
  5. Coarse-to-Fine Pre-training for Named Entity RecognitionMengge Xue, Bowen Yu, Zhenyu Zhang, Tingwen Liu et al.EMNLP 2020 · 49 citations
Dusk over the sea between two cliffs drawn in fine vertical lines