Convolutional Tensor-Train LSTM for Spatio-Temporal Learning

Learning from spatio-temporal data has numerous applications such as humanbehavior analysis, object tracking, video compression, and physics simulation. However, existing methods still perform poorly on challenging video tasks such as long-term forecasting. This is because these kinds of challenging tasks require learning long-term spatio-temporal correlations in the video sequence. In this paper, we propose a higher-order convolutional LSTM model that can efficiently learn these correlations, along with a succinct representations of the history. This is accomplished through a novel tensor-train module that performs prediction by combining convolutional features across time. To make this feasible in terms of computation and memory requirements, we propose a novel convolutional tensortrain decomposition of the higher-order model. This decomposition reduces the model complexity by jointly approximating a sequence of convolutional kernels as a low-rank tensor-train factorization. As a result, our model outperforms existing approaches, but uses only a fraction of parameters, including the baseline models. Our results achieve state-of-the-art performance in a wide range of applications and datasets, including the multi-steps video prediction on the Moving-MNIST-2 and KTH action datasets as well as early activity recognition on the Something-Something V2 dataset. Learning from (video) sequences. Most state-of-the-art video models are based on recurrent neural networks (RNNs), typically some variations of Convolutional LSTM (ConvLSTM) where spatiotemporal information is encoded explicitly in each cell [4] [5] [6] [7] . These RNNs are first-order Markovian * Equal contribution † This work was done while the author was an intern at NVIDIA. Project page: https://sites.google.com/nvidia.com/conv-tt-lstm Preprint. Under review.