Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training

Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency on neuromorphic hardware due to their sparse activation patterns. However, conventional training methods based on surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under limited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates of intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substructures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self-generated knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable components, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and Im-ageNet demonstrate that our method reduces training complexity while achieving high-performance SNN training. Our code is available at https://github.com/Intelli- Chip-Lab/enhanced-self-distillation-framework-for-snn.

• We establish a mapping between the firing rates of intermediate layers and the ANN branch, optimizing the gradient errors of the intermediate layers under rate-based backpropagation. This results in outstanding performance with extremely low training cost.

• We analyze the reliability of teacher signals in the self-distillation process and propose a novel decoupling strategy that separates reliable and unreliable components, thereby constructing a more stable and effective self-distillation framework.

• We conduct empirical validation on standard datasets, including CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet, and perform ablation studies on various model components.

Our results demonstrate that the proposed framework effectively balances training efficiency and high performance, offering clear advantages over existing methods.

2 Related Work

The primary training approaches for Spiking Neural Networks (SNNs) include conversion-based methods and direct training methods. The former establishes a connection between SNNs and Artificial Neural Networks (ANNs) via an equivalent closed-form mapping, converting a pre-trained ANN into the target SNN model. This avoids the challenge of training SNNs from scratch. Although