Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
Shiwei Li, Huifeng Guo, Lu Hou, Wei Zhang, Xing Tang, Ruiming Tang, Rui Zhang, Ruixuan Li
Abstract
Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their embedding tables at the training stage. To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed lowprecision training (LPT). Also, we provide theoretical analysis on its convergence. The results show that stochastic weight quantization has a faster convergence rate and a smaller convergence error than deterministic weight quantization in LPT. Further, to reduce the accuracy degradation, we propose adaptive low-precision training (ALPT) that learns the step size (i.e., the quantization resolution) through gradient descent. Experiments on two real-world datasets confirm our analysis and show that ALPT can significantly improve the prediction accuracy, especially at extremely low bit widths. For the first time in CTR models, we successfully train 8bit embeddings without sacrificing prediction accuracy. The code of ALPT is publicly available 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Experimental Analysis of Large-scale Learnable Vector Storage Compression cites Next Point-of-Interest Recommendation on Resource-Constrained Mobile Devices
- Experimental Analysis of Large-scale Learnable Vector Storage Compression cites Learnable Embedding sizes for Recommender Systems
- Experimental Analysis of Large-scale Learnable Vector Storage Compression cites Learned Step Size quantization
- Experimental Analysis of Large-scale Learnable Vector Storage Compression cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation cites FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
- FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models cites Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
- Learnable Embedding sizes for Recommender Systems cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
- CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models cites Experimental Analysis of Large-scale Learnable Vector Storage Compression
- CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models cites Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
- Up or Down? Adaptive Rounding for Post-Training Quantization cites Learned Step Size quantization
Cited by top-tier papers11
- FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language ModelsJintao Tong, Wenwei Jin, Pengda Qin, Anqi Li et al.NeurIPS 2025 · 31 citations
- Experimental Analysis of Large-scale Learnable Vector Storage CompressionHailin Zhang, Penghao Zhao, Xupeng Miao, Yingxia Shao et al.VLDB 2024 · 20 citations
- Semantic Retrieval Augmented Contrastive Learning for Sequential RecommendationZiqiang Cui, Yunpeng Weng, Xing Tang, Xiaokun Zhang et al.NeurIPS 2025 · 17 citations
- CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation ModelsHailin Zhang, Zirui Liu, Boxuan Chen, Yikai Zhao et al.SIGMOD 2024 · 15 citations
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank AdaptationShiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang et al.NeurIPS 2025 · 10 citations
Builds on9
- Learned Step Size quantizationSteven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy et al.ICLR 2020 · 1,037 citations
- Up or Down? Adaptive Rounding for Post-Training QuantizationMarkus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos et al.ICML 2020 · 816 citations
- Next Point-of-Interest Recommendation on Resource-Constrained Mobile DevicesQinyong Wang, Hongzhi Yin, Tong Chen, Zi Huang et al.WWW 2020 · 116 citations
- Learnable Embedding sizes for Recommender SystemsSiyi Liu, Chen Gao, Yihong Chen, Depeng Jin et al.ICLR 2021 · 97 citations
- Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation SystemsHao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, Jiyan YangKDD 2020 · 88 citations
Related papers
- CowClip: Reducing CTR Prediction Model Training Time from 12 Hours to 10 Minutes on 1 GPUZangwei Zheng, Pengtai Xu, Xuan Zou, Da Tang et al.AAAI 2023 · 9 citations
- Agile and Accurate CTR Prediction Model Training for Massive-Scale Online Advertising SystemsZhiqiang Xu, Dong Li, Weijie Zhao, Xing Shen et al.SIGMOD 2021 · 38 citations
- Fixed-Point Back-Propagation TrainingXishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu et al.CVPR 2020
- LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector QuantizationHaoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang et al.ICML 2026 · 1 citation
- AMPA: Adaptive Mixed Precision Allocation for Low-Bit Integer TrainingLi Ding, Wen Fei, Yuyang Huang, Shuangrui Ding et al.ICML 2024 · 5 citations
