EMNLP Findings2020
Tensorized Embedding Layers
Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, Ivan V. Oseledets
Abstract
The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parameterizing embedding layers based on the Tensor Train decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- EL-Rec: Efficient Large-Scale Recommendation Model Training via Tensor-Train Embedding Table cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Shapeshifter: a Parameter-efficient Transformer using Factorized Reshaped Matrices cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Exploring extreme parameter compression for pre-trained language models cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Cited by top-tier papers10
- Exploring extreme parameter compression for pre-trained language modelsBenyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang et al.ICLR 2022 · 23 citations
- Experimental Analysis of Large-scale Learnable Vector Storage CompressionHailin Zhang, Penghao Zhao, Xupeng Miao, Yingxia Shao et al.VLDB 2024 · 20 citations
- Few-bit Backward: Quantized Gradients of Activation Functions for Memory Footprint ReductionGeorgii Sergeevich Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov et al.ICML 2023 · 19 citations
- Shapeshifter: a Parameter-efficient Transformer using Factorized Reshaped MatricesAliakbar Panahi, Seyran Saeedi, Tom ArodzNeurIPS 2021 · 19 citations
- EL-Rec: Efficient Large-Scale Recommendation Model Training via Tensor-Train Embedding TableZheng Wang, Yuke Wang, Boyuan Feng, Dheevatsa Mudigere et al.SC 2022 · 15 citations
Builds on1
Related papers
- Improving Word Embedding Factorization for Compression using Distilled Nonlinear Neural DecompositionVasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md. Akmal Haidar et al.EMNLP Findings 2020 · 8 citations
- Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model CompressionJingcun Wang, Yu-Guang Chen, Ing-Chao Lin, Bing Li et al.ICLR 2025
- Nimble GNN Embedding with Tensor-Train DecompositionChunxing Yin, Da Zheng, Israt Nisa, Christos Faloutsos et al.KDD 2022 · 14 citations
- Differentiable Product Quantization for End-to-End Embedding CompressionTing Chen, Lala Li, Yizhou SunICML 2020 · 81 citations
- Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine TranslationSunzhu Li, Peng Zhang, Guobing Gan, Xiuqing Lv et al.EMNLP 2022 · 3 citations
