EMNLP Findings2021
UnClE: Explicitly Leveraging Semantic Similarity to Reduce the Parameters of Word Embeddings
Zhi Li, Yuchen Zhai, Chengyu Wang, Minghui Qiu, Kailiang Li, Yin Zhang
Abstract
Natural language processing (NLP) models often require a massive number of parameters for word embeddings, which limits their application on mobile devices. Researchers have employed many approaches, e.g. adaptive inputs, to reduce the parameters of word embeddings. However, existing methods rarely pay attention to semantic information. In this paper, we propose a novel method called Unique and Class Embeddings (UnClE), which explicitly leverages semantic similarity with weight sharing to reduce the dimensionality of word embeddings. Inspired by the fact that words with similar semantic can share a part of weights, we divide the embeddings of words into two parts: unique embedding and class embedding. The former is one-to-one mapping like traditional embedding, while the latter is many-toone mapping and learn the representation of class information. Our method is suitable for both word-level and sub-word level models and can be used to reduce both input and output embeddings. Experimental results on the standard WMT 2014 English-German dataset show that our method is able to reduce the parameters of word embeddings by more than 11x, with about 93% performance retaining in BLEU metrics. For language modeling task, our model can reduce word embeddings by 6x or 11x on PTB/WT2 dataset at the cost of a certain degree of performance degradation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on3
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- DeFINE: Deep Factorized Input Token Embeddings for Neural Sequence ModelingSachin Mehta, Rik Koncel-Kedziorski, Mohammad Rastegari, Hannaneh HajishirziICLR 2020 · 28 citations
- Adaptive Compression of Word EmbeddingsYeachan Kim, Kang-Min Kim, SangKeun LeeACL 2020 · 20 citations
Related papers
- All Word Embeddings from One EmbeddingSho Takase, Sosuke KobayashiNeurIPS 2020 · 16 citations
- Block-wise Word Embedding Compression Revisited: Better Weighting and StructuringJong-Ryul Lee, Yong-Ju Lee, Yong-Hyuk MoonEMNLP Findings 2021 · 3 citations
- word2ket: Space-efficient Word Embeddings inspired by Quantum EntanglementAliakbar Panahi, Seyran Saeedi, Tomasz ArodzICLR 2020 · 44 citations
- Rethinking Embedding Coupling in Pre-trained Language ModelsHyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson et al.ICLR 2021 · 11 citations
- Accelerating Neural Machine Translation with Partial Word Embedding CompressionFan Zhang, Mei Tu, Jinyao YanAAAI 2021 · 3 citations
