Efficient Transformer Attention for SNNs via Hadamard Simplification

Spiking Neural Networks (SNNs) enable low-power, event-driven computation, but Transformer-based SNNs remain difficult to deploy on neuromorphic hardware due to dense operations and communication overhead. We propose two simplified attention mechanisms, Simplified Spiking Attention (SSA) and Ultra-Simplified Spiking Attention (USSA), which replace matrix multiplication with Hadamard products and eliminate hardware-unfriendly components such as multi-head attention and scaling. We show that consecutive masking is redundant and analyze a spiking-order effect in which early spiking contributes more temporal information to attention modulation. On CIFAR-10, CIFAR-100, and DVS-Gesture, SSA achieves accuracies of 96.38%, 79.45%, and 97.56%, respectively, while reducing computational complexity from $\mathcal{O}(N^2D)$ to $\mathcal{O}(ND)$ and communication complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(ND)$. USSA further reduces communication complexity to $\mathcal{O}(N)$ with only marginal accuracy degradation. On ImageNet-1K, SSA and USSA achieve 76.91% and 77.27% accuracy, respectively, demonstrating scalability to large-scale classification.