Fast and Slow Gradient Approximation for Binary Neural Network Optimization
Xinquan Chen, Junqi Gao, Biqing Qi, Dong Li, Yiang Luo, Fangyuan Li, Pengfei Li
Abstract
Binary Neural Networks (BNNs) have garnered significant attention due to their immense potential for deployment on edge devices. However, the non-differentiability of the quantization function poses a challenge for the optimization of BNNs, as its derivative cannot be backpropagated. To address this issue, hypernetwork based methods, which utilize neural networks to learn the gradients of non-differentiable quantization functions, have emerged as a promising approach due to their adaptive learning capabilities to reduce estimation errors. However, existing hypernetwork based methods typically rely solely on current gradient information, neglecting the influence of historical gradients. This oversight can lead to accumulated gradient errors when calculating gradient momentum during optimization. To incorporate historical gradient information, we design a Historical Gradient Storage (HGS) module, which models the historical gradient sequence to generate the first-order momentum required for optimization. To further enhance gradient generation in hypernetworks, we propose a Fast and Slow Gradient Generation (FSG) method. Additionally, to produce more precise gradients, we introduce Layer Recognition Embeddings (LRE) into the hypernetwork, facilitating the generation of layer-specific fine gradients. Extensive comparative experiments on the CIFAR-10 and CIFAR-100 datasets demonstrate that our method achieves faster convergence and lower loss values, outperforming existing baselines.Code is available at github.com/FSG.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model cites Efficiently Modeling Long Sequences with Structured State Spaces
- Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training cites Learning Frequency Domain Approximation for Binary Neural Networks
- Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training cites ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting
Builds on7
- Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space ModelLianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang et al.ICML 2024 · 1,725 citations
- ResRep: Lossless CNN Pruning via Decoupling Remembering and ForgettingXiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu et al.ICCV 2021 · 202 citations
- Learning Frequency Domain Approximation for Binary Neural NetworksYixing Xu, Kai Han, Chang Xu, Yehui Tang et al.NeurIPS 2021 · 64 citations
- Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks TrainingXiao-Ming Wu, Dian Zheng, Zuhao Liu, Wei-Shi ZhengICCV 2023 · 28 citations
Related papers
- Towards Accurate Binary Spiking Neural Networks: Learning with Adaptive Gradient Modulation MechanismYu Liang, Wenjie Wei, Ammar Belatreche, Honglin Cao et al.AAAI 2025 · 10 citations
- FF-INT8: Efficient Forward-Forward DNN Training on Edge Devices with INT8 PrecisionJingxiao Ma, Priyadarshini Panda, Sherief RedaDAC 2025 · 1 citation
- BEP: A Binary Error Propagation Algorithm for Binary Neural Networks TrainingLuca Colombo, Fabrizio Pittorino, Daniele Zambon, Carlo Baldassi et al.ICLR 2026
- Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural NetworksYikai Wang, Yi Yang, Fuchun Sun, Anbang YaoICCV 2021 · 18 citations
- Memory-efficient training of binarized neural networks on the edgeMikail Yayla, Jian-Jia ChenDAC 2022 · 5 citations
