ReSprop: Reuse Sparsified Backpropagation

The success of Convolutional Neural Networks (CNNs) in various applications is accompanied by a significant increase in computation and training time. In this work, we focus on accelerating training by observing that about 90% of gradients are reusable during training. Leveraging this observation, we propose a new algorithm, Reuse-Sparse-Backprop (ReSprop), as a method to sparsify gradient vectors during CNN training. ReSprop maintains stateof-the-art accuracy on CIFAR-10, CIFAR-100, and Ima-geNet datasets with less than 1.1% accuracy loss while enabling a reduction in back-propagation computations by a factor of 10× resulting in a 2.7× overall speedup in training. As the computation reduction introduced by Re-Sprop is accomplished by introducing fine-grained sparsity that reduces computation efficiency on GPUs, we introduce a generic sparse convolution neural network accelerator (GSCN), which is designed to accelerate sparse backpropagation convolutions. When combined with ReSprop, GSCN achieves 8.0× and 7.2× speedup in the backward pass on ResNet34 and VGG16 versus a GTX 1080 Ti GPU.