EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature

Spiking neural networks (SNNs) have gained more and more interest as one of the energy-efficient alternatives of conventional artificial neural networks (ANNs). They exchange 0/1 spikes for processing information, thus most of the multiplications in networks can be replaced by additions. However, binary spike feature maps will limit the expressiveness of the SNN and result in unsatisfactory performance compared with ANNs. It is shown that a rich output feature representation (i.e., the feature vector before classifier) is beneficial to training an accurate model in ANNs for classification. We wonder if it also does for SNNs and how to improve the feature representation of the SNN. To this end, we materialize this idea in two special designed methods for SNNs. First, inspired by some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN with light modification to homogeneous SNN can obtain a well-performed SNN, we use rich information of the weight parameters from the trained ANN counterpart to guide the feature representation learning of the SNN. In particular, we present the SNN's and ANN's feature representation from the same input to ANN's classifier to product SNN's and ANN's outputs respectively and then align the feature with the KL-divergence loss as in knowledge distillation methods, called L AF loss. It can be seen as a novel and effective knowledge distillation method specially designed for the SNN that comes from both the knowledge distillation and ANN-SNN methods. Second, we replace the last Leaky Integrate-and-Fire (LIF) activation layer as the ReLU activation layer to generate the output feature, thus a more powerful SNN with full-precision feature representation can be achieved but with only a little extra computation. Experimental results show that our method consistently outperforms the current state-of-the-art algorithms on both popular non-spiking static and neuromorphic datasets. Recently, convolutional neural networks (CNNs) have become extremely popular due to their performing more and more well in diverse fields including pattern recognition He et al. (2016); Simonyan & Zisserman (2014), object detection Girshick (2015); Ren et al. (2016); Ming et al. (2023), language processing Chen et al. (2016), robotics Levine et al. ( 2015 ), and so on. However, these full-precision CNN models make them power-hungry and tedious for real-world deployment. The spiking neural network (SNN), which aims to mimic the behavior of the human brain, has become a promising energy-efficient architecture to substitute for the CNN in some specific scenarios Guo et al. (2022c);