ResQ: Residual Quantization for Video Perception

This paper accelerates video perception, such as segmentation and human pose estimation, by levering cross-frame redundancies. Unlike the existing approaches, which avoid redundant computations by warping the past features using optical-flow or by performing sparse convolutions on frame differences, we approach the problem from a different perspective: low-bit quantization. We observe that residuals, as the difference in network activations between two neighboring frames, exhibit properties that make them highly quantizable. Based on this observation, we propose a novel quantization scheme for video networks coined as Res idual Quantization. ResQ extends the standard, frame-by-frame, quantization scheme by incorporating temporal dependencies that lead to better performance in terms of accuracy vs. bit-width. Furthermore, we extend our model to dynamically adjust the bit-width proportionally to the amount of changes in the video. We showcase the superiority of our model, against the standard quantization and existing efficient video perception models, using various architectures on semantic segmentation, video object segmentation and human pose estimation benchmarks.