Lune

EMNLP Findings2021

UnClE: Explicitly Leveraging Semantic Similarity to Reduce the Parameters of Word Embeddings

Zhi Li, Yuchen Zhai, Chengyu Wang, Minghui Qiu, Kailiang Li, Yin Zhang

摘要

Natural language processing (NLP) models often require a massive number of parameters for word embeddings, which limits their application on mobile devices. Researchers have employed many approaches, e.g. adaptive inputs, to reduce the parameters of word embeddings. However, existing methods rarely pay attention to semantic information. In this paper, we propose a novel method called Unique and Class Embeddings (UnClE), which explicitly leverages semantic similarity with weight sharing to reduce the dimensionality of word embeddings. Inspired by the fact that words with similar semantic can share a part of weights, we divide the embeddings of words into two parts: unique embedding and class embedding. The former is one-to-one mapping like traditional embedding, while the latter is many-toone mapping and learn the representation of class information. Our method is suitable for both word-level and sub-word level models and can be used to reduce both input and output embeddings. Experimental results on the standard WMT 2014 English-German dataset show that our method is able to reduce the parameters of word embeddings by more than 11x, with about 93% performance retaining in BLEU metrics. For language modeling task, our model can reduce word embeddings by 6x or 11x on PTB/WT2 dataset at the cost of a certain degree of performance degradation.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper3

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel 等ICLR 2020 · 7,418 次引用
  2. DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence ModelingSachin Mehta, Rik Koncel-Kedziorski, Mohammad Rastegari, Hannaneh HajishirziICLR 2020 · 28 次引用
  3. Adaptive Compression of Word EmbeddingsYeachan Kim, Kang-Min Kim, SangKeun LeeACL 2020 · 20 次引用

相关 Paper

  1. All Word Embeddings from One EmbeddingSho Takase, Sosuke KobayashiNeurIPS 2020 · 16 次引用
  2. Block-wise Word Embedding Compression Revisited: Better Weighting and StructuringJong-Ryul Lee, Yong-Ju Lee, Yong-Hyuk MoonEMNLP Findings 2021 · 3 次引用
  3. word2ket: Space-efficient Word Embeddings inspired by Quantum EntanglementAliakbar Panahi, Seyran Saeedi, Tomasz ArodzICLR 2020 · 44 次引用
  4. Rethinking Embedding Coupling in Pre-trained Language ModelsHyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson 等ICLR 2021 · 11 次引用
  5. Accelerating Neural Machine Translation with Partial Word Embedding CompressionFan Zhang, Mei Tu, Jinyao YanAAAI 2021 · 3 次引用
黄昏的海面,两侧是细线勾勒的悬崖