Lune

ICLR2021Top-tier venue

Learnable Embedding sizes for Recommender Systems

Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, Yong Li

Abstract

The embedding-based representation learning is commonly used in deep learning recommendation models to map the raw sparse features to dense vectors. The traditional embedding manner that assigns a uniform size to all features has two issues. First, the numerous features inevitably lead to a gigantic embedding table that causes a high memory usage cost. Second, it is likely to cause the over-fitting problem for those features that do not require too large representation capacity. Existing works that try to address the problem always cause a significant drop in recommendation performance or suffers from the limitation of unaffordable training time cost. In this paper, we proposed a novel approach, named PEP (short for Plug-in Embedding Pruning), to reduce the size of the embedding table while obviating a drop in accuracy and computational optimization. PEP prunes embedding parameter where the pruning threshold(s) can be adaptively learned from data. Therefore we can automatically obtain a mixed-dimension embedding-scheme by pruning redundant parameters for each feature. PEP is a general framework that can plug in various base recommendation models. Extensive experiments demonstrate it can efficiently cut down embedding parameters and boost the base model's performance. Specifically, it achieves strong recommendation performance while reducing 97-99% parameters. As for the computation cost, PEP only brings an additional 20-30% time cost compared with base models. Codes are available at https://github.com/ssui-liu/learnable-embed-sizes-for-RecSys.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

18 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
  • Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models cites Learning to Embed Categorical Features without Embedding Tables for Recommendation
  • Learning to Embed Categorical Features without Embedding Tables for Recommendation cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
  • Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
  • DreamShard: Generalizable Embedding Table Placement for Recommender Systems cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
  • DreamShard: Generalizable Embedding Table Placement for Recommender Systems cites Learning to Embed Categorical Features without Embedding Tables for Recommendation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers18

  1. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language ModelsXin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen et al.ACL 2026 · 57 citations
  2. Learning to Embed Categorical Features without Embedding Tables for RecommendationWang-Cheng Kang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi et al.KDD 2021 · 46 citations
  3. DreamShard: Generalizable Embedding Table Placement for Recommender SystemsDaochen Zha, Louis Feng, Qiaoyu Tan, Zirui Liu et al.NeurIPS 2022 · 37 citations
  4. Adaptive Low-Precision Training for Embeddings in Click-Through Rate PredictionShiwei Li, Huifeng Guo, Lu Hou, Wei Zhang et al.AAAI 2023 · 27 citations
  5. AdaEmbed: Adaptive Embedding for Large-Scale Recommendation ModelsFan Lai, Wei Zhang, Rui Liu, William Tsai et al.OSDI 2023 · 23 citations

Builds on2

  1. Soft Threshold Weight Reparameterization for Learnable SparsityAditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman et al.ICML 2020 · 266 citations
  2. Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation SystemsHao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, Jiyan YangKDD 2020 · 88 citations

Related papers

  1. Single-shot Embedding Dimension Search in Recommender SystemLiang Qu, Yonghong Ye, Ningzhi Tang, Lixin Zhang et al.SIGIR 2022 · 23 citations
  2. Hybrid Embedding Framework for Memory-Efficient Recommendation SystemsSeung Jin Yang, Hyuk-Jae Lee, Chae-Eun RheeDAC 2025
  3. Experimental Analysis of Large-scale Learnable Vector Storage CompressionHailin Zhang, Penghao Zhao, Xupeng Miao, Yingxia Shao et al.VLDB 2024 · 20 citations
  4. Clustered Embedding Learning for Recommender SystemsYizhou Chen, Guangda Huzhang, Anxiang Zeng, Qingtao Yu et al.WWW 2023 · 13 citations
  5. EL-Rec: Efficient Large-Scale Recommendation Model Training via Tensor-Train Embedding TableZheng Wang, Yuke Wang, Boyuan Feng, Dheevatsa Mudigere et al.SC 2022 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines